About the project
Clinical decision-support systems are often built by manually gathering, formalising and implementing specialist knowledge. Therefore, they are limited by existing human knowledge concerning modelling clinical conditions, diagnosis and therapy, and can be inaccurate because of variations and complexity inherent to medical data. In order to circumvent these limitations, following an obvious growth of publicly available collections of medical-radiology diagnostic images, machine learning has become an irreplaceable tool for solving various problems concerning radiology image analysis, as a part of decision support systems in the clinic. As a complement to the transfer learning paradigm for a more efficient construction of new predictive models, the popularity of so-called foundation models has recently grown, primarily due to their ability to easily integrate multimodal data sources, generalize without domain adaptation, and easily adapt to new tasks.
The goal of this project is to use the power of modern knowledge achieved in the field of machine learning to build multimodal foundation models for medical radiology, which will have the ability to adapt to numerous and diverse complex tasks, such as identifying key tokens in narrative diagnoses and connecting them with visual concepts, generating narrative diagnoses based on given image inputs and vice versa, retrieving the most similar exam cases according to the given query, etc. The aforementioned will be realized by using the existing archive collection of 20 million medical radiology images taken from CHC Rijeka.
The work will be conducted by an interdisciplinary team (computer science, physics and medical science) of researchers from the University of Rijeka Faculty of Medicine and Faculty of Engineering, Specialty Hospital Medico and the University of Ljubljana Faculty of Mathematics and Physics.
Activities
- An analysis of the available multimodal medical radiology data collection and an overview of the current state of the field
- Implementation of the programming framework for the implementation of the experiment, including learning and comprehensive evaluation of the model
- Selection and implementation of multimodal machine learning methods from image and text
- Iterative learning and quantitative evaluation of the RADNET model, with optimization of hyperparameter values
- Examination of confidential data traces in RADNET models
- Selection and implementation of downstream diagnostic and other tasks for publicly available data collections
- Quantitative comparative evaluation of the RADNET model for solving selected tasks
- Qualitative comparative evaluation of learned models using saliency methods
- Investigating the possibility of using federated learning for secure handling of confidential data
- Examining the interest of other clinical centers for inclusion in the research
- Preparation of documentation for the project application of a larger consortium
Associates
prof. Ivan Štajduhar 1 – principal investigator
prof. Damir Miletić, M.D., radiology specialist 2
prof. Matija Milanič 3
Franko Hržić, Ph.D. 1
Tin Nadarević, Ph.D., M.D., radiology specialist 2
Mihaela Mamula Saračević, M.D., radiology specialist 4
Teo Manojlović, mag. ing. comp. 1
Mateja Napravnik, mag. ing. comp. 1
Mária Krajčí, mag. ing. comp. 1
Mateo Mikulić, mag. ing. comp. 1
Dominik Vičević, mag. ing. comp. 1
1 University of Rijeka, Faculty of Engineering
2 University of Rijeka, Faculty of Medicine / Clinical Hospital Centre Rijeka
3 University of Ljubljana, Faculty of Mathematics and Physics
4 Special Hospital Medico, Rijeka
Financing
UNIRI projects for materially demanding research, uniri-mzi-25-17
Funded by the European Union – NextGenerationEU. The views and opinions expressed are solely those of the author and do not necessarily reflect the official views of the European Union or the European Commission. Neither the European Union nor the European Commission can be held responsible for them.