Classifications

• • G—PHYSICS
  • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
  • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
  • G16H30/00—ICT specially adapted for the handling or processing of medical images
  • G16H30/20—ICT specially adapted for the handling or processing of medical images for handling medical images, e.g. DICOM, HL7 or PACS
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T7/00—Image analysis
  • G06T7/0002—Inspection of images, e.g. flaw detection
• • G—PHYSICS
  • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
  • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
  • G16H30/00—ICT specially adapted for the handling or processing of medical images
  • G16H30/40—ICT specially adapted for the handling or processing of medical images for processing medical images, e.g. editing
• • G—PHYSICS
  • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
  • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
  • G16H50/00—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics
  • G16H50/20—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics for computer-aided diagnosis, e.g. based on medical expert systems
• • G—PHYSICS
  • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
  • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
  • G16H50/00—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics
  • G16H50/70—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics for mining of medical data, e.g. analysing previous cases of other patients
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/30—Subject of image; Context of image processing
  • G06T2207/30168—Image quality inspection

Definitions

• the present invention generally relates to image processing, and in particular to an imaging system, a method of image processing, a computer program element, and a computer readable medium.
• the operator (referred to herein as “the user”) is responsible for performing a set of worksteps throughout the examination, including for example, depending on the modality and the specifics of equipment:
• Al Artificial intelligence
• Al systems are computer-implemented systems. They are based on machine learning algorithms that have been pre-trained on training data to perform a task, such as assisting the user during the examination. For training the machine learning algorithms, the training data has to be prepared for the development of the Al algorithm, which implies a high effort in the development process. Additionally, Al algorithms that have been trained in a different facility may not always fit the needs and the standards of a different facility, which makes it very difficult to obtain a sustainable architecture that can be used to train Al algorithms for different applications based on customer needs.
• an imaging system comprising: a medical imaging apparatus configured to use a set of pre-image settings for acquiring an image of a patient in an imaging session; a user interface configured to receive a user annotation in relation to (i) the acquired image of the patient and/or (ii) the set of pre-image settings used by the medical imaging apparatus for acquiring the image of the patient; at least one training database configured to store (i) the acquired image of the patient and the received user annotation and/or (ii) the set of pre-image settings and the received user annotation; and a training module configured to train at least one data-driven model with training data obtained from the at least one training database.
• Al has shown high potential to enhance the clinical workflow. Therefore, there are many attempts to develop Al algorithms that can be used in the clinical workflow. However, training of these algorithms is very challenging, as it requires well-annotated clinical data. Typically, this implies that a big effort has to be spent to find the suitable data and to annotate it.
• an imaging system comprises a medical imaging system (e.g. x-ray, CT or MRI scanner) for acquiring an image of a patient.
• a medical imaging system e.g. x-ray, CT or MRI scanner
• the image acquired by the medical imaging apparatus is then displayed and the user has the possibility to annotate the image via a user interface.
• the user interface may receive a user annotation in relation to an image of the patient acquired by a medical imaging apparatus (e.g. x-ray, CT or MRI scanner).
• a medical imaging apparatus e.g. x-ray, CT or MRI scanner.
• the user annotation may include, but are not limited to, an indication of image quality and a clinical finding.
• the user interface may be configured to receive the acquired image of the patient in real-time, e.g. right after image acquisition at the imaging system. The acquired image and the user annotation are then stored in a training database, thereby creating a training database for training of the Al algorithm.
• the user interface may receive a user annotation in relation to the set of pre-image settings used by the medical imaging apparatus for acquiring the image of the patient.
• the user annotation could be collimation settings, the exposure time settings, the tube voltage settings, the focal spot size settings, the selection of the X-ray sensitive areas for an X-ray imaging system to apply the correct dose to the patient, etc.
• the set of preimage settings and the user annotation are then stored in a training database, thereby creating a training database for training of the Al algorithm.
• the images and the sets of pre-image settings can be directly chosen from the clinical workflow and there is no need to select the images and/or the sets of pre-image settings and transfer them somewhere else (e.g. from other facilities) for the development.
• the parameters of the Al algorithm which has been trained using the training data from the training database, are adapted to fit the needs and standards of a particular facility, which makes it possible to obtain a sustainable architecture that can be used to train Al algorithms for different applications based on the customer’s needs.
• the training module may be implemented in a processor circuitry configured for parallel computing, for instance a multicore processor, a GPU or parts thereof.
• the training module may be included in a system-on-chip (SoC) circuitry.
• SoC system-on-chip
• the user interface may be part of a handheld device including one or more of: a mobile phone, a laptop computing device, and a tablet computer.
• the user interface may be part of the medical imaging apparatus.
• the medical imaging apparatus, the training module, and the user interface may have a wired connection (e.g. USB, coaxial or optical cable, etc.) and/or wireless connection (e.g. Bluetooth, NFC, WUAN, etc.).
• a wired connection e.g. USB, coaxial or optical cable, etc.
• wireless connection e.g. Bluetooth, NFC, WUAN, etc.
• a network may communicatively couple the medical imaging apparatus, the training module, and the user interface.
• the network may be the internet.
• the network may alternatively be any other type and number of networks.
• the network may be implemented by several local area networks connected to a wide area network.
• the network may comprise any combination of wired networks, wireless networks, wide area networks, local area networks, etc.
• the at least one data- driven model comprises one or more of: a data-driven model to analyze the acquired image of the patient to compute medical decision support information; and a data-driven model to analyze a camera image of the patient to compute a set of preimage settings, wherein the camera image is generated on the basis of sensor data obtained from a sensor arrangement, which has a field of view including at least part of an area, where the patient is positioned for imaging.
• the medical decision support information may comprise e.g. a recommended workflow in relation to the patient, an indication of an image quality in relation to the acquired image, an indication on a medical finding, and a priority information representing urgency of a medical finding.
• the camera image may be in the form of a depth image or an RGB image.
• the camera image may be acquired when the patient is positioned for the imaging examination, e.g., by lying or standing with the field of view of the imaging system.
• the camera image does not necessarily have to include the whole body surface of the patient; it may relate to only part of the body surface of the patient, which is relevant for the imaging examination. For example, if an anatomy of interest is a neck of the patient, only the measurement image of the upper body of the patient may be captured by the sensor arrangement.
• the medical imaging apparatus comprises the user interface.
• the medical imaging apparatus may comprise a touch screen that allows the user to input the user annotation.
• the imaging system further comprises a mobile annotation device that comprises: an input channel configured to receive (i) the acquired image of the patient and/or (ii) the set of pre-image settings used by the medical imaging apparatus for acquiring the image of the patient; a display configured to display (i) the acquired image of the patient and/or (ii) the set of pre-image settings used by the medical imaging apparatus for acquiring the image of the patient; the user interface configured to receive a user annotation in relation to (i) the acquired image of the patient and/or (ii) the set of pre-image settings used by the medical imaging apparatus for acquiring the image of the patient; and an output channel configured to provide (i) the acquired image of the patient and the received user annotation and/or (ii) the set of pre-image settings and the received user annotation to the at least one training database.
• a mobile annotation device that comprises: an input channel configured to receive (i) the acquired image of the patient and/or (ii) the set of pre-image settings used by the medical imaging apparatus for acquiring the image of the patient;
• the mobile annotation device is a handheld device including one or more of: a mobile phone, a laptop computing device, and a tablet computer.
• the mobile annotation device may be placed close to the radiologist’s reading monitor as shown in Fig. 4. Each time that a radiologists opens an image, the image may be additionally displayed on the mobile annotation device 14. On the mobile annotation device, the radiologist has the possibility to annotate the image with respect to e.g. image quality, clinical findings, etc.
• the training module is configured to repeatedly train the at least one data-driven model.
• the data-driven model may be trained on the fly on a new instance of images acquired by the medical imaging apparatus.
• the training mode may be continued in repeated training phases in the deployment phase.
• the training module of the imageprocessing device may be configured to train the data-driven model on the fly (i.e. during normal use of the data-driven model in the deployment phase) on the acquired image of the patient and the user annotation provided by the mobile annotation device.
• each newly acquired image and the respective user annotation provided by the mobile annotation device may be directly sent to the training module for updating the parameters of the data-driven model.
• the data-driven model is continuously trained, such that the data-driven model fits the needs and standards of a particular facility.
• the data-driven model is not trained on the fly, but after a certain number of images with annotation have been collected.
• the user may choose a new annotation for the image.
• the database is then enriched not only by this image but by all possible variations of the image (such as the same image but slightly cut-off). This data enrichment step may allow to increase the weight of this image that was wrongly annotated by the algorithm, as the algorithm is then retrained using the new image and all the newly generated image variations.
• the training module is configured to generate a user annotation on a random basis for starting training the data-driven model.
• the initial model is a naive model - that is to say that the first image feedback is generated on a random basis. This means that the model can be trained from scratch.
• the data-driven model is configured to provide a suggestion based on the computed medical decision support information for allowing a user to actively accept or reject the provided suggestion.
• a further possibility is that the model provides only a suggestion and the user needs to actively accept or reject the suggestion, as opposed to the possibility where the output user input can be a number of predefined tags (e.g. different diseases).
• the medical imaging system comprises: a first group of medical imaging apparatuses and a second group of medical imaging apparatuses, which is different from the first group of medical imaging apparatuses; wherein the user interface is configured to receive: a first user annotation in relation to (i) an image of a patient acquired by a medical imaging apparatus in the first group and/or (ii) a set of pre-image settings used by a medical imaging apparatus in the first group; and a second user annotation in relation to (i) an image of a patient acquired by a medical imaging apparatus in the second group and/or (ii) a set of pre-image settings used by a medical imaging apparatus in the second group; wherein the at least one training database (16) comprises: a first training database for storing (i) the image of a patient acquired by a medical imaging apparatus in the first group and the received user annotation and/or (ii) the set of preimage settings used by a medical imaging apparatus in the first group and the received user annotation; and a second training database for (
• the images and/or pre-image settings for training purposes are directly taken from the clinical workflow, the training can be performed according to the standards of each group (e.g. user group, or facility).
• the proposed IT infrastructure may offer a method to integrate training of Al algorithms into the clinical workflow and to make the training group-specific.
• the first group of medical imaging apparatuses and the second group of medical imaging apparatuses are from different facilities and/or different user groups.
• the user annotation comprises one or more of: an indication of image quality; a clinical finding; and an indication of a set of desired pre-image settings.
• the indication on image quality includes an indication of one any one or more of: a) patient positioning, b) collimator setting, c) contrast, d) resolution, e) noise, and f) artifact.
• clinical finding may also be referred to as medical finding.
• the indication of a set of desired pre-image settings e.g. for CT may include collimation settings, the exposure time settings, the tube voltage settings, the focal spot size settings, and/or the selection of the X-ray sensitive areas for an X-ray imaging system.
• the decision support information includes one or more of: a recommended workflow in relation to the patient; an indication of an image quality in relation to the acquired image; an indication on a medical finding; and a priority information representing urgency of a medical finding.
• the recommend workflow may be in variance to a previously defined workflow envisaged for the said patient.
• the mobile annotation device is a handheld device including one or more of: a mobile phone, a laptop computing device, and a tablet computer.
• a method of image processing comprising: acquiring, by a medical imaging apparatus using a set of pre-image settings, an image of a patient in an imaging session; receiving, via a user interface, a user annotation in relation to (i) the acquired image of the patient and/or (ii) the set of pre-image settings used by the medical imaging apparatus for acquiring the image of the patient; storing, in at least one training database, (i) the acquired image of the patient and the received user annotation and/or (ii) the set of pre-image settings and the received user annotation; and training, by a training module, at least one data-driven model with training data obtained from the at least one training database.
• a computer program element configured, which, when being executed by at least one processing unit, is adapted to cause the imaging system according to the first aspect and any associated example to perform the method according to the second aspect and any associated example.
• a computer readable medium comprising the computer program element.
• the term “user” refers to medical personnel at least partly involved in an administrative or organizational manner in the imaging procedure.
• the term “patient” refers to a person, or in veterinary settings, an animal (in particular a mammal), who is being imaged.
• machine-learning refers to the field of the computer sciences that studies the design of computer programs able to induce patterns, regularities, or rules from past experiences to develop an appropriate response to future data, or describe the data in some meaningful way.
• the term “learning” in the context of machine learning refers to the identification and training of suitable algorithms to accomplish tasks of interest.
• the learning that is, the performance of the machine learning of a task measurable by a performance metric, generally improves with the training data.
• the term “data-driven model” in the context of machine learning refers to a suitable algorithm that is trained on the basis of appropriate training data.
• a neural -network model is shown as may be used in examples. However, other machine learning techniques such as support vector machines, decision trees or other may be used instead of neural networks.
• An exemplary data-driven model is illustrated in Fig. 8.
• module may refer to, be part of, or include an Application Specific Integrated Circuit (ASIC), an electronic circuit, a processor (shared, dedicated, or group) and/or memory (shared, dedicated, or group) that execute one or more software or firmware programs, a combinational logical circuit, and/or other suitable components that provide the described functionality.
• ASIC Application Specific Integrated Circuit
• processor shared, dedicated, or group
• memory shared, dedicated, or group
• Fig. 1 shows a schematic block diagram of an exemplary medical imaging system.
• Fig. 2 shows a schematic block diagram of an exemplary imaging processing device.
• Fig. 3 shows a schematic block diagram of an exemplary mobile annotation device.
• Fig. 4 shows an example of implementing the mobile annotation device for a radiologist.
• Fig. 5 shows a schematic block diagram of a further exemplary medical imaging system.
• Fig. 6 shows a schematic block diagram of a further exemplary imaging-processing device.
• Fig. 7 shows a flow chart of an exemplary method of image processing.
• Fig. 8 shows a schematic diagram of an exemplary data-driven model.
• Fig. 1 shows a schematic block diagram of an exemplary medical imaging system 100.
• the medical imaging system 100 comprises a medical imaging apparatus 10, an image-processing device 12, a mobile annotation device 14, a training database 16, an image database 18a, 18b, and a display arrangement 20.
• the medical imaging apparatus 10 is configured for acquiring an image 22 of a patient in an imaging session.
• the medical imaging apparatus 10 may be of any modality such as transmission or emission imaging.
• Transmission imaging includes for instance x-ray based imaging carried out with a CT scanner or other.
• Magnetic resonance imaging MRI is also envisaged and so is ultrasound imaging.
• Emission imaging includes PET/SPECT and other nuclear medicine modalities.
• images 22 are acquired of the patient.
• the images 20 are preferably in digital form and may assist a physician in diagnosis.
• the image-processing device 12 comprises an image analyzer 26 configured to apply a data-driven model to analyze the acquired image of the patient to compute medical decision support information.
• the image-processing device 14 is a mobile device, such as, but not limited to, a mobile phone, a laptop computing device, or a tablet computer.
• the imageprocessing device 12 is a server that provides the computing service.
• the imaging processing device 12 is a workstation with the display arrangement 18.
• Fig. 2 shows a schematic block diagram of an exemplary image-processing device 12.
• the image-processing device 12 comprises an input channel 24, an image analyzer 26, and an output channel 28.
• the input channel 24 may be configured to receive the acquired image 22 of a patient in an imaging session.
• the input channel 12 may be, in an example, implemented as an Ethernet interface, a USB (TM) interface, a wireless interface such as a Wi-Fi (TM) or Bluetooth (TM) or any comparable data transfer interface enabling data transfer between input peripherals and the image analyzer 26.
• the input channel may access data over a network, such as the internet or any combination of wired networks, wireless networks, wide area networks, local area networks, etc.
• the image analyzer 26 of the image-processing device 12 may be driven by artificial intelligence.
• the image analyzer 26 may be included as a pre-trained data-driven model.
• the image analyzer may be run on a processing unit of image-processing device 12.
• the processing unit may include general purpose circuity and/or dedicated computing circuitry such as a GPU or may be a dedicated core of a multi -core multi-processor.
• the processing unit is configured for parallel computing. This is in particular advantageous if the underlying machine-learning model is a neural network such as a convolutional network.
• Such types of machine learning models can be efficiently implemented by vector, matrix or tensor multiplications. Such types of computations can be accelerated in a parallel computing infrastructure.
• the decision support information computed by the image analyzer 26 may hence include any one or more of the following: image quality, medical finding and/or in associated priority level.
• the image quality analysis may include an assessment of patient positioning, collimator setting (if any), contrast, resolution, image noise or artifacts. Some or all of these factors may be considered and represented as a single image quality score in a suitable metric or each factor is measured by a separate score in a different metric.
• the image may be sent to a different image database 18a.
• image may be sent to the image database 18b. Additionally, a suggestive graphical indication may be given when the image quality is deemed optimal. For instance, a suggestive “tick” symbol may be displayed in an apt coloring scheme, such as green or otherwise. If the image quality is found to be inadequate, this is also indicated on the onboard display in suggestive symbology such as a red cross or otherwise.
• a medical condition is found, this is indicated by a suitable textual or other symbol on the onboard display of the mobile display device.
• a related workflow may be suggested. This suggested workflow may be different from the currently assigned plan.
• a retake may be proposed, optionally with a suggestion for updated imaging parameters. The user may then accept the retake using the user interface, and a suitably formatted message is sent to the operating console to adjust the imaging parameters and/or initiate the image retake.
• the computing of the decision support information is done in a two-stage sequential processing flow.
• a first stage the image quality is established. If the image quality is found to be sufficient, only then is the imagery analyzed for a medical finding and/or workflow suggestions.
• the workflow computed based on the analyzed image may differ from a workflow originally associated with the patient at check-in for instance. This change in workflow may be required for instance if an unexpected medical condition is detected in the image that was not previously envisaged by the original workflow. For instance, if the patient is to receive a cancer treatment of a certain organ, such as the liver, a certain workflow is envisaged. However, if the analysis of the copy image accidentally reveals that the patient is in fact suffering from pneumonia, the workflow needs to be changed to first treat pneumonia before proceeding with the cancer treatment.
• the image-processing device 12 may further comprise a training module 30 fortraining the data-driven module.
• a training module 30 fortraining the data-driven module.
• training data is collected from the training database 16.
• the output channel 28 is configured to provide the decision support information , e.g. to a display arrangement 20 (for example, a built-in screen, a connected monitor or projector) or to a file storage (for example, a hard drive or a solid state drive), such as image database 18a, 18b.
• Fig. 3 shows a schematic block diagram of an exemplary mobile annotation device 14.
• the mobile annotation device 14 may include any type of wireless device such as consumer electronics devices, smart phones, tablet personal computers, wearable computing devices, personal digital assistants (PDAs), laptop computers, and/or any other like physical computing device.
• PDAs personal digital assistants
• a single mobile annotation device 14 is illustrated.
• two or more users may have the possibility to add their user annotations to the training database. This may provide comprehensive annotations of the acquired image and minimize opinion bias. In this way, the quality of the training data may be improved.
• the mobile annotation device 14 comprises an input channel 14a, a display 14b, a user interface 14c, and an output channel 14d.
• the input channel 14a is configured to receive the acquired image of the patient.
• the input channel 14a may be a universal interface and affords interoperability with a range of different medical imaging apparatuses, even of different modalities.
• the input channel 14a is configured to receive the acquired image based on direct imaging (“image of-image”) of the displayed image.
• image of-image the input channel 14a is arranged as NFC or Bluetooth, if imaging apparatus is so equipped.
• the display 14b is configured to display the acquired image of the patient. As the acquired image is displayed on the mobile annotation device 14, the user has the possibility to annotate the image.
• the user interface 14c is configured to receive a user annotation in relation to the acquired image of the patient.
• the term “user interface” may refer to an interface between a human user or operator and one or more devices that enables communication between the user and the device (s). Further examples of user interfaces that may be employed in various implementations of the present disclosure include, but are not limited to, switches, potentiometers, buttons, dials, sliders, track balls, display screens, various types of graphical user interfaces (GUIs), touch screens, microphones, and other types of sensors that may receive some form of human-generated stimulus and generate a signal in response thereto.
• GUIs graphical user interfaces
• the use may have the possibility to annotate the image with predefined parameters, e.g. predefined tags (e.g. for different diseases).
• the user may use the mobile annotation device 14, e.g. a client, such as a web-based thin client, to access the tailored content for annotating the image.
• the mobile annotation device 14 may be used with the thin-client as an app to access the content for annotating the image.
• the content may be stored on a database.
• the content may be customizable, e.g. by an administrator.
• the content for annotating the image may comprise a list of image qualities to be selected, such as “optimal”, “suboptimal”, and “bad”, a list of clinical findings in relation to a particular disease e.g. in patients with chronic complex regional pain syndrome, etc.
• the “annotation” could look like is discussed on the example of image quality for knee imaging.
• the user may choose the option “optimal” on the mobile annotation device 14.
• the user may choose “suboptimal” on the mobile annotation device 14.
• the input is added to the training database 16 for the data-driven model.
• the user annotation may comprise a clinical finding. For example, if a radiologist sees a pneumothorax on the image, he may choose “pneumothorax” on the mobile annotation device 14 and the image may be automatically added to the corresponding training database 16 for the data-driven model.
• This workflow may have the advantage that the radiologist can choose which images he or she wants to add - i.e. he can exclude the images where he or she is not sure about the findings.
• the data-driven model may provide only a suggestion and the user may need to actively accept or reject the suggestion, as opposed to the possibility where the output user input can be a number of predefined tags (e.g. for different diseases).
• the mobile annotation device 14 may be placed close to the radiologist’s reading monitor 18 as shown in Fig. 4. Each time that a radiologists opens an image, the image may be additionally displayed on the mobile annotation device 14. On the mobile annotation device 14, the radiologist has the possibility to annotate the image with respect to e.g. image quality, clinical findings, etc.
• the output channel 14d is configured to provide the acquired image of the patient and the user annotation to the training database 16 for training the data-driven model.
• the data-driven model may be used to automatically evaluate e.g. the image quality of a newly acquired image and/or clinical findings.
• the training can be performed according to the standards of the institution. Which persons are experienced enough to annotate the data, it can be determined by the institution which person is experienced enough for annotating the data.
• the proposed IT infrastructure may offer a method to integrate training of Al algorithms into the clinical workflow and to make the training user-specific.
• the training module 30 may be configured to train the data-driven model using the following steps of receiving the training data, and applying the data-driven model to the training data in one or more iterations. As a result of this application the pre-trained model is then obtained which can then be used in deployment. In deployment, new data, e.g. a new image not from the training set, can be applied to the pre-trained model to obtain the desired decision support information for this new data.
• the data-driven model is operable in two modes: “training mode/phase” and “deployment mode/phase”.
• training mode an initial model of the data-driven model is trained based on a set of training data to produce a trained data-driven model.
• deployment mode the pre -trained data-driven model is fed with non-training, newly acquired image data, to operate during normal use.
• the data-driven model is operable in a hybrid mode, where the system is running in an already trained manner, however, if a user sees an image where he/she disagrees with the feedback he can “overrule” the system.
• the mobile annotation device 14 may receive and display the medical decision support information computed by the image-processing device 12. If the user disagrees with the computed medical decision support information, the user may provide his/her feedback to overrule the system.
• the initial model of the data-driven model may be a naive model - that is to say that the first image feedback is generated on a random basis.
• the training module may be configured to generate a user annotation on a random basis for starting training the data-driven model and the data-driven model may be trained from scratch.
• the training mode may be continued in repeated training phases in the deployment phase.
• the training module 30 of the image-processing device 12 may be configured to train the data-driven model on the fly (i.e. during normal use of the data-driven model) on each newly acquired image of the patient and the user annotation provided by the mobile annotation device 14. For example, each newly acquired image and the respective user annotation provided by the mobile annotation device 14 may be directly sent to the training module for updating the parameters of the data-driven model. In this way, the data-driven model is continuously trained, such that the parameters of the data-driven model is continuously adapted for fitting the needs and standards of a particular facility.
• the data-driven model is not trained on the fly, but after a certain number of images with annotation have been collected.
• Fig. 5 shows a further example of the medical imaging system 100.
• the medical imaging system 100 comprises a first group 50a of medical imaging apparatuses and a second group 50b of medical imaging apparatuses, which is different from the first group 50a of medical imaging apparatuses.
• first and second groups of medical imaging apparatuses may represent different facilities. In an example, the first and second groups of medical imaging apparatus may represent different user groups.
• Each group may comprise one or more medical imaging apparatuses.
• the first group 50a of medical imaging apparatuses comprises three medical imaging apparatuses, while the second group 50b of medical imaging apparatuses comprises two medical imaging apparatuses.
• the image analyzer 26 of the image-processing device 12 is configured to apply a first data-driven model to analyze an image acquired by the first group of medical imaging apparatuses and to apply a second data-driven model to analyze an image acquired by the second group of medical imaging apparatuses.
• Fig. 6 shows a schematic block diagram of an exemplary imaging processing device 12.
• the image analyzer 26 is configured to apply a first data-driven model to analyze an image acquired by the first group 50a of medical imaging apparatuses.
• the image analyzer 26 is configured to apply a second data-driven model to analyze an image acquired by the second group 50b of medical imaging apparatuses.
• the first and second data-driven models may use the same neural networks, e.g. CNN.
• the first and second data-driven models may use different neural networks, such as CNN and recurrent NNs.
• the first data-driven model may use neural networks and the second data-driven model may use other machine learning techniques such as support vector machines, decision trees, etc.
• the training data for the first group 50a of medical imaging apparatuses and the second group 50b of medical imaging apparatuses is collected differently.
• the mobile annotation device 14 is configured to provide the image acquired by the first group 50a of medical imaging apparatuses together with the user annotation to a first training database 16a for training the first data-driven model.
• the mobile image annotation apparatus 14 is configured to provide the image acquired by the second group 50b of medical imaging apparatuses together with the user annotation to a second training database 16b for training the second data-driven model.
• the second training database 16b is different from the first training database 16a.
• the metadata of each acquired image may comprise a group identifier.
• the mobile annotation device 14 may send the acquired image and user annotation to the corresponding training database 16a, 6b according to the group identifier.
• the metadata of each acquired image may comprise an identifier for the medical imaging apparatus.
• the mobile annotation device 14 may check a lookup table using the identifier for the medical imaging apparatus to find the corresponding training database 16a, 16b.
• the images for training purposes are directly taken from the clinical workflow of each group, the training can be performed according to the standards of each group (e.g. user group, or facility).
• the proposed IT infrastructure may offer a method to integrate training of Al algorithms into the clinical workflow and to make the training group-specific.
• the exemplary imaging system in Figs. 1-6 illustrates a user interface in a mobile annotation device 14.
• the user interface may be part of the mobile imaging system.
• the medical imaging apparatus e.g. x-ray, MRI, CT, etc.
• the medical imaging apparatus may comprises a display with a touch screen configured to receive the user annotation.
• Fig. 7 shows a flow chart of a method of image processing that relates to the system described above.
• Fig. 7 shows a flow chart of a method of image processing that relates to the system described above.
• a medical imaging apparatus acquires an image of a patient in an imaging session.
• a user interface receives a user annotation in relation to the acquired image of the patient.
• the user interface receives the set of pre-image settings used by the medical imaging apparatus for acquiring the image of the patient.
• the acquired image of the patient and the received user annotation are stored in at least one training database.
• the user annotation may comprise an indication of image quality.
• the user annotation may comprise a clinical finding.
• the set of pre-image settings and the received user annotation are stored in at least one training database.
• a training module trains at least one data-driven model with training data obtained from the at least one training database.
• an initial model of a data-driven model is trained based on a set of training data to produce a trained data-driven model in a training mode.
• the pre -trained data- driven model is fed with non-training, newly acquired image data, to operate during normal use.
• the data-driven model is operable in a hybrid mode, where the system is running in an already trained manner, however, if a user sees an image where he/she disagrees with the feedback he can “overrule” the system.
• the mobile annotation device 14 may receive and display the medical decision support information computed by the image-processing device 12. If the user disagrees with the computed medical decision support information, the user may provide his/her feedback to overrule the system.
• first group of medical imaging apparatuses and a second group of medical imaging apparatuses, which is different from the first group of medical imaging apparatuses.
• Each group may comprise one or more medical imaging apparatuses.
• the first group of medical imaging apparatuses and the second group of medical imaging apparatuses may be from different facilities and/or different user groups.
• the method 200 may further comprises the step of providing, by the mobile annotation device, the image acquired by the first group of medical imaging apparatuses together with the user annotation to a first training database for training the first data-driven model for analyzing an image acquired by the first group of medical imaging apparatuses.
• the method 200 may further comprise the step of providing, by the mobile annotation device, the image acquired by the second group of medical imaging apparatuses and the user annotation to a second training database for training the first data-driven model for analyzing an image acquired by the first group of medical imaging apparatuses.
• the second training database is different from the first training database.
• the images fortraining purposes are directly taken from the clinical workflow, the training can be performed according to the standards of each group (e.g. user group, or facility).
• the proposed IT infrastructure may offer a method to integrate training of Al algorithms into the clinical workflow and to make the training group-specific.
• a neural-network model is shown as an exemplary data-driven model.
• other machine learning techniques such as support vector machines, decision trees or other may be used instead of neural networks.
• neural networks in particular convolutional networks, have been found to be of particular benefit especially in relation to image data.
• Fig. 8 is a schematic diagram of a convolutional neural -network (CNN).
• a fully configured NN as obtained after training may be thought as representation of an approximation of a latent mapping between two spaces, the images and the space of any one or more of image quality metrics, medical findings and treatment plans. These spaces can be represented as points in a potentially high dimensional space, such as an image being a matrix of Ax A, with A being the number of pixels.
• the image quality metrics, the medical findings and treatment plane may be similarly encoded as vectors, matrices or tensors.
• a workflow may be implemented as a matrix or vector structure, with each entry representing a workflow step.
• the learning task may be one or more of classification and/or regression.
• the input space of images may include 4D matrices to represent a time series of matrices, and hence a video sequence.
• a suitable trained machine-learning model or component attempts to approximate this mapping.
• the approximation may be achieved in a learning or training process where parameters, itself forming a high dimensional space, are adjusted in an optimization scheme based on training data.
• the machine-learning component may be realized as neural-network (“NN”), in particular a convolutional neuro-network (“CNN”). With continued reference to Fig. 11, this shows in more detail a CNN architecture as envisaged herein in embodiments.
• the CNN is operable in two modes: “training mode/phase” and “deployment mode/phase”.
• training mode an initial model of the CNN is trained based on a set of training data to produce a trained CNN model.
• deployment mode the pre-trained CNN model is fed with non-training, new data, to operate during normal use.
• the training mode may be a one-off operation or this is continued in repeated training phases to enhance performance. All that has been said so far in relation to the two modes is applicable to any kind of machine learning algorithms and is not restricted to CNNs or, for that matter, NNs.
• the CNN comprises a set of interconnected nodes organized in layers.
• the CNN includes an output layer OL and an input layer IL.
• the input layer IL may be a matrix whose size (rows and columns) matches that of the training input image.
• the output layer OL may be a vector or matrix with size matching the size chosen for the image quality metrics, medical findings and treatment plans.
• the CNN has preferably a deep learning architecture, that is, in between the OL and IL there is at least one, preferably two or more, hidden layers.
• Hidden layers may include one or more convolutional layers CL1, CL2 (“CL”) and/or one or more pooling layers PL1, PL2 (“PL”) and/or one or more fully connected layer FL1, FL2 (“FL”). CLs are not fully connected and/or connections from CL to a next layer may vary but are in generally fixed in FLs.
• Nodes are associated with numbers, called “weights”, which represent how the node responds to input from earlier nodes in a preceding layer.
• the set of all weights defines a configuration of the CNN.
• an initial configuration is adjusted based on the training data using a learning algorithm such as forward-backward (’7'7f '(-propagation or other optimization schemes, or other gradient descent methods. Gradients are taken with respect of the parameters of the objective function.
• the training mode is preferably supervised, that is, is based on annotated training data.
• one item is the training input data and the other item is target training data known a priori to be correctly associated with its training input data item.
• This association defines the annotation and is preferably provided by a human expert.
• the training pair includes imagery as training input data, and associated with each training image, is target of label for any one or more of: image quality indication, indication of medical finding represented by that image, indication of a priority level, indication of workflow step(s) called for the given image.
• annotated training data includes pairs or training data items directly taken from the clinical workflow. Therefore, the training can be performed according to the standards of the institution. It can be determined by the institution which person is experienced enough for annotating the data.
• the above-proposed IT infrastructure may offer a method to integrate training of Al algorithms into the clinical workflow and to make the training user-specific.
• the training can be performed according to the standards of the user group.
• the proposed IT infrastructure may offer a method to integrate training of Al algorithms into the clinical workflow and to make the training group-specific.
• the output is in general different from the target.
• the initial configuration is readjusted so as to achieve a good match between input training data and their respective target for all pairs.
• the match is measured by way of a similarity measure, which can be formulated in terms of on objective function, or cost function.
• the aim is to adjust the parameters to incur low cost, that is, a good match.
• the input training data items are applied to the input layer (IL) and passed through a cascaded group(s) of convolutional layers CL1, CL2 and possibly one or more pooling layers PL1, PL2, and are finally passed to one or more fully connected layers.
• the convolutional module is responsible for feature based learning (e.g. identifying features in the patient characteristics and context data, etc.), while the fully connected layers are responsible for more abstract learning, for instance, the impact of the features on the treatment.
• the output layer OL includes the output data that represents the estimates for the respective targets.
• the exact grouping and order of the layers as per Fig. 8 is but one exemplary embodiment, and other groupings and order of layers are also envisaged in different embodiments. Also, the number of layers of each type (that is, any one of CL, FL, PL) may differ from the arrangement shown in Fig 8. The depth of the CNN may also differ from the one shown in Fig. 8. All that has been said above is of equal application to other NNs envisaged herein, such as fully connected classical perceptron type NN, deep or not, and recurrent NNs, or others.
• the annotated (labelled) training data may need to be reformatted into structured form.
• the annotated training data may be arranged as vectors or matrices or tensor (arrays of dimension higher than 2). This reformatting may be done by a data pre-processor module (not shown), such as scripting program or filter that runs through patient records of the HIS of the current facility to pull up a set of patient characteristics.
• the training data sets are applied to the initially configured CNN and is then processed according to a learning algorithm such as the FB-propagation algorithm as mentioned before.
• a learning algorithm such as the FB-propagation algorithm as mentioned before.
• the so pre-trained CNN may then be used in deployment phase to compute the decision support information for new data, that is, newly acquired copy images not present in the training data.
• the data-driven model may be trained on the fly (i.e. during normal use of the data-driven model in the deployment phase) on the acquired image of the patient and the user annotation provided by the mobile annotation device 14. For example, each newly acquired image and the respective user annotation provided by the mobile annotation device 14 may be directly sent to the training module for updating the parameters of the data-driven model. In this way, the data-driven model is continuously trained, such that the data-driven model fits the needs and standards of a particular facility.
• Some or all of the above-mentioned steps may be implemented in hardware, in software or in a combination thereof.
• Implementation in hardware may include a suitably programmed FPGA (field-programmable-gate-array) or a hardwired IC chip.
• FPGA field-programmable-gate-array
• a hardwired IC chip For good responsiveness and high throughput, multi-core processors such as GPU or TPU or similar may be used to implement the above described training and deployment of the machine learning model, in particular for NNs.
• Circuitry may include discrete and/or integrated circuitry, application specific integrated circuitry (ASIC), a system-on- a-chip (SOC), and combinations thereof, a machine, a computer system, a processor and memory, a computer program.
• ASIC application specific integrated circuitry
• SOC system-on- a-chip
• Figs. 1-8 the approach is described in relation with a data-driven model to analyze the acquired image of the patient to compute medical decision support information. It will be appreciated that the above-described approach may be adapted to any data-driven model.
• the data-driven model is a data-driven model for analyzing a camera image of the patient to compute the set of pre-image settings.
• a data-driven model correlates one or more features in the camera image with set of pre-image settings.
• the set of pre-image settings for an x-ray imaging system may include at least one of a tube voltage, a tube current, a grid, a collimation window, and a geometry parameter of a collimator.
• the set of pre-image settings for a CT imaging system comprises at least one of a power supply level, a tube current, a dose modulation, a scan planning parameter, and a reconstruction parameter.
• the input of the data-driven model may include non-image patient data.
• the non-image patient data of the patient may comprise complementary information to the measurement image captured by the sensor arrangement.
• the size of the lung is known to be correlated to the patient weight, age and gender and may be influenced by certain diseases like COPD.
• the data-driven model may be trained to better model the relation between the patient and the scan configuration, e.g. collimation settings, to be used in the exam preparation and/or in the imaging procedure.
• the data-driven model may comprise at least one of: an artificial neural network, and a classification tree using at least one of Haar-like, scale -invariant feature transform (SIFT), and speed up robust feature (SURF) image features.
• SIFT scale -invariant feature transform
• SURF speed up robust feature
• the user interface 14c shown in Fig. 1 is configured to receive a user annotation in relation to the set of pre-image settings used by the medical imaging apparatus for acquiring the image of the patient.
• the training database 16 is configured to store the set of pre-image settings and the received user annotation.
• the training module 30 is configured to train the exemplary data-driven model with training data obtained from the at least one training database.
• the phrase “at least one,” in reference to a list of one or more elements, should be understood to mean at least one element selected from any one or more of the elements in the list of elements, but not necessarily including at least one of each and every element specifically listed within the list of elements and not excluding any combinations of elements in the list of elements.
• This definition also allows that elements may optionally be present other than the elements specifically identified within the list of elements to which the phrase “at least one” refers, whether related or unrelated to those elements specifically identified.
• the computer program element might therefore be stored on a computer unit, which might also be part of an embodiment of the present invention.
• This computing unit may be adapted to perform or induce a performing of the steps of the method described above. Moreover, it may be adapted to operate the components of the above described apparatus.
• the computing unit can be adapted to operate automatically and/or to execute the orders of a user.
• a computer program may be loaded into a working memory of a data processor.
• the data processor may thus be equipped to carry out the method of the invention.
• This exemplary embodiment of the invention covers both, a computer program that right from the beginning uses the invention and a computer program that by means of an up-date turns an existing program into a program that uses the invention.
• the computer program element might be able to provide all necessary steps to fulfil the procedure of an exemplary embodiment of the method as described above.
• a computer readable medium such as a CD-ROM
• the computer readable medium has a computer program element stored on it which computer program element is described by the preceding section.
• a computer program may be stored and/or distributed on a suitable medium, such as an optical storage medium or a solid state medium supplied together with or as part of other hardware, but may also be distributed in other forms, such as via the internet or other wired or wireless telecommunication systems.
• a suitable medium such as an optical storage medium or a solid state medium supplied together with or as part of other hardware, but may also be distributed in other forms, such as via the internet or other wired or wireless telecommunication systems.
• the computer program may also be presented over a network like the World Wide Web and can be downloaded into the working memory of a data processor from such a network.
• a medium for making a computer program element available for downloading is provided, which computer program element is arranged to perform a method according to one of the previously described embodiments of the invention.
• inventive embodiments are presented by way of example only and that, within the scope of the appended claims and equivalents thereto, inventive embodiments may be practiced otherwise than as specifically described and claimed.
• inventive embodiments of the present disclosure are directed to each individual feature, system, article, material, kit, and/or method described herein.

Landscapes

• Engineering & Computer Science (AREA)
• Health & Medical Sciences (AREA)
• Medical Informatics (AREA)
• Public Health (AREA)
• Epidemiology (AREA)
• General Health & Medical Sciences (AREA)
• Primary Health Care (AREA)
• Biomedical Technology (AREA)
• Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
• Radiology & Medical Imaging (AREA)
• Data Mining & Analysis (AREA)
• Databases & Information Systems (AREA)
• Pathology (AREA)
• Physics & Mathematics (AREA)
• General Physics & Mathematics (AREA)
• Theoretical Computer Science (AREA)
• Computer Vision & Pattern Recognition (AREA)
• Quality & Reliability (AREA)
• Medical Treatment And Welfare Office Work (AREA)
• Measuring And Recording Apparatus For Diagnosis (AREA)
• Apparatus For Radiation Diagnosis (AREA)

Priority Applications (4)

Applications Claiming Priority (2)

Publications (1)

Family

ID=72644118

Family Applications (1)

Country Status (5)

Cited By (1)

Citations (3)

Family Cites Families (5)

• 2020
  • 2020-09-24 EP EP20197985.3A patent/EP3975194A1/en not_active Withdrawn
• 2021
  • 2021-09-16 EP EP21777331.6A patent/EP4218024A1/en active Pending
  • 2021-09-16 US US18/027,947 patent/US20230368895A1/en active Pending
  • 2021-09-16 WO PCT/EP2021/075484 patent/WO2022063675A1/en not_active Ceased
  • 2021-09-16 CN CN202180065603.8A patent/CN116194999A/zh active Pending
  • 2021-09-16 JP JP2023518466A patent/JP2023543191A/ja active Pending

Patent Citations (3)

Also Published As

Similar Documents

Legal Events