WO2024133367A1 - Procédés et systèmes d'interrogation basée sur une image pour des caractéristiques radiographiques similaires - Google Patents

Procédés et systèmes d'interrogation basée sur une image pour des caractéristiques radiographiques similaires Download PDF

Info

Publication number
WO2024133367A1
WO2024133367A1 PCT/EP2023/086792 EP2023086792W WO2024133367A1 WO 2024133367 A1 WO2024133367 A1 WO 2024133367A1 EP 2023086792 W EP2023086792 W EP 2023086792W WO 2024133367 A1 WO2024133367 A1 WO 2024133367A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
search
documents
sentences
trained
Prior art date
Application number
PCT/EP2023/086792
Other languages
English (en)
Inventor
Xin Wang
Original Assignee
Koninklijke Philips N.V.
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Koninklijke Philips N.V. filed Critical Koninklijke Philips N.V.
Publication of WO2024133367A1 publication Critical patent/WO2024133367A1/fr

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/40—Information retrieval; Database structures therefor; File system structures therefor of multimedia data, e.g. slideshows comprising image and additional audio data
    • G06F16/43—Querying
    • G06F16/432—Query formulation
    • G06F16/434—Query formulation using image data, e.g. images, photos, pictures taken by a user
    • G—PHYSICS
    • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
    • G16H30/00—ICT specially adapted for the handling or processing of medical images
    • G16H30/20—ICT specially adapted for the handling or processing of medical images for handling medical images, e.g. DICOM, HL7 or PACS
    • G—PHYSICS
    • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
    • G16H30/00—ICT specially adapted for the handling or processing of medical images
    • G16H30/40—ICT specially adapted for the handling or processing of medical images for processing medical images, e.g. editing

Definitions

  • the present disclosure is directed generally to methods and systems for image -based querying.
  • a radiologist performing an examination of a patient, or reviewing the results of an exam may want to compare an obtained image or a region of interest (ROI) in an image to similar images. This is true regardless of the type of imaging modality utilized by the radiologist for the exam. For example, the radiologist may want to identify or diagnose a feature in an image, and may need or desire that comparison to similar features and radiology reports to increase confidence in the identification of the feature/condition and/or diagnosis of the patient.
  • ROI region of interest
  • the radiologist may also find, in an image or ROI, a radiographic feature that is new or rare because it is a new type of disease or a very rare disease.
  • the radiologist may find a feature that is familiar or pronounced of previous features, but cannot recall an identification or diagnosis of the feature.
  • the radiologist may know a specific identification or diagnosis for a condition, but the radiographic feature found during the exam may appear different than in previous or similar exams. In each of these situations, the radiologist may desire to compare the feature to similar features and radiology reports.
  • Various embodiments and implementations are directed to a method and system for image -based querying using an image-based query system.
  • the image -based query system receives a search image and provides the search image to a trained search model which is trained to identify one or more sentences in one or more documents of a report database using the search image.
  • the image-based query system then provides the identified sentences and documents to the user of the image-based query system.
  • the image is obtained by a medical imaging modality and the report database is a database of reports related to that medical imaging modality.
  • a method for image -based querying includes: (i) receiving, by an image-based query system, a search image; (ii) providing the received search image to a trained search model of the image -based query system, wherein the trained search model is trained to pair a search image with one or more sentences from one or more documents in a report database, each of the documents in the report database comprising one or more images and one or more sentences related to the one or more images, and wherein pairing a search image with one or more sentences from one or more documents in a report database comprises generating an association score between the search image and the one or more sentences, the association score being a measure of an association between the search image and the one or more sentences; (iii) receiving, from the trained search model, a plurality of generated association scores for the search image and a plurality of documents, each of the plurality of received documents associated with a respective one of the plurality of generated association scores; and (iv) providing via a user interface of the
  • the search image is a region of interest (ROI) of a medical image, the ROI selected by a user of a medical imaging system.
  • ROI region of interest
  • providing comprises providing a plurality of documents from the report database, and wherein the plurality of documents are ranked based on their respective association score.
  • providing comprises displaying one or more images from the provided at least one document.
  • providing at least one document from the report database comprises a predetermined association score threshold.
  • the method further includes filtering, based on the received plurality of generated association scores for the search image, the plurality of received documents using one or more limiting parameters.
  • the method further includes training the search model of the image -based query system.
  • Training the search model of the image -based query system includes: (i) receiving a plurality of reports, each of the plurality of reports comprising a report image and one or more report sentences; (ii) iterating, for each of the plurality of reports: processing, by a trained image embedding model, the report image to generate an image embedding; processing, by a trained natural language processing (NLP) transformer model, each of the one or more report sentences to generate a text embedding for each of the sentences; and analyzing, by a neural network, the image embedding and the text embedding for each of the sentences to generate a binary association classification, where a binary association classification of “1” indicates an association between the image embedding and the text embedding, and wherein a binary association classification of “0” indicates no association between the image embedding and the text embedding; and (ii) training, using the image embedding, the text
  • the method further includes storing the trained search model.
  • the NLP transformer model is a bidirectional encoder representations from transformers (BERT) model or a BERT-like model.
  • a system for image -based querying includes: a document database comprising a plurality of documents, each of the documents comprising one or more images and one or more sentences related to the one or more images; a trained search model, wherein the trained search model is trained to pair a search image with one or more sentences from one or more documents in the document database, wherein pairing a search image with one or more sentences comprises generating an association score between the search image and the one or more sentences, the association score being a measure of an association between the search image and the one or more sentences; a user interface configured to receive the search image and further configured to provide an output of the trained search model to a user; and a processor configured to: (i) provide the received search image to the trained search model; (ii) receive, from the trained search model, a plurality of generated association scores for the search image and a plurality of documents, each of the plurality of received documents associated with a respective one of the plurality of generated association scores; and (iii)
  • FIG. 1 is a flowchart of a method for image -based querying, in accordance with an embodiment.
  • FIG. 2 is a schematic representation of an image-based query system, in accordance with an embodiment.
  • FIG. 3 is a flowchart of a method for training a search model, in accordance with an embodiment.
  • FIG. 4 is a flowchart of a method for training a search model, in accordance with an embodiment.
  • the present disclosure describes various embodiments of a system and method configured to identify and provide documents, such as reports, from a database using an image query. More generally, Applicant has recognized and appreciated that it would be beneficial to provide a method and system to enable the rapid and efficient searching of an enormous database of documents using an image as an input query.
  • An image -based query system receives a search image and provides the search image to a trained search model which is trained to identify one or more sentences in one or more documents of a report database using the search image. The imagebased query system then provides the identified sentences and documents to the user of the image - based query system.
  • the image is obtained by a medical imaging modality and the report database is a database of reports related to that medical imaging modality.
  • the embodiments and implementations disclosed or otherwise envisioned herein can be utilized with any system that may utilize or benefit from image -based searching.
  • one application of the embodiments and implementations disclosed or otherwise envisioned herein is health imaging, analysis, and similar systems.
  • one application is to improve the functionality of the Philips® Vue Picture Archiving and Communication System (PACS) (manufactured by Koninklijke Philips, N.V.), among other products.
  • PACS Philips® Vue Picture Archiving and Communication System
  • Another application of the embodiments and implementations herein is to improve medical monitoring systems such as, e.g., a Philips Patient Monitoring system such as the Philips IntelliSpace® Precision Medicine products (manufactured by Koninklijke Philips, N.V.), among many other products.
  • the disclosure is not limited to these devices or systems, and thus disclosure and embodiments disclosed herein can encompass any system that may utilize or benefit from image-based searching.
  • FIG. 1 in one embodiment, is a flowchart of a method 100 for image -based querying using an image -based query system.
  • the methods described in connection with the figures are provided as examples only, and shall be understood not to limit the scope of the disclosure.
  • the image -based query system can be any of the systems described or otherwise envisioned herein.
  • the image -based query system can be a single system or multiple different systems.
  • an image -based query system 200 is provided.
  • the system comprises one or more of a processor 220, memory 230, user interface 240, communications interface 250, and storage 260, interconnected via one or more system buses 212.
  • FIG. 2 constitutes, in some respects, an abstraction and that the actual organization of the components of the system 200 may be different and more complex than illustrated.
  • image-based query system 200 can be any of the systems described or otherwise envisioned herein. Other elements and components of the image -based query system 200 are disclosed and/or envisioned elsewhere herein.
  • the image-based query system 200 comprises or is in direct or indirect communication with an imaging device and/or image database 270.
  • the imaging device may be any imaging device, and may obtain one or more images using any imaging modality.
  • the most common forms of imaging modality are X-ray, magnetic resonance imaging (MRI), ultrasound, computed tomography scan (CT scan), and nuclear imaging such as Positron Emission Tomography (PET), although many other types of health- or medicine -based imaging modalities are possible.
  • the one or more images obtained using the imaging modality may be obtained from a patient or other individual.
  • an image database may be a database utilized to store one or more images of a patient, obtained by an imaging device.
  • the imaging device and/or image database 270 may be local to the image -based query system, and may optionally be a component of the image-based query system.
  • the imaging device and/or image database 270 may alternatively be remote to the image-based query system, and thus is in direct or indirection communication with the image-based query system.
  • the image-based query system 200 comprises or is in direct or indirect communication with a report or document database 280.
  • the report or document database 280 is a database of 1000s or more of reports relevant to at least the imaging modality used by the radiologist, physician, clinician, researcher or other healthcare individual during an examination.
  • the report or document database 280 preferably comprises at least 100s or 1000s of reports or documents of prior MRI examinations.
  • the report or document database 280 preferably comprises at least 100s or 1000s of reports or documents of prior X-ray examinations.
  • the report or document database 280 may comprise a corpus of reports/documents for a plurality of different imaging modalities.
  • a report or document in the report or document database 280 comprises information about a prior examination.
  • a report may comprise information about an ultrasound of a patient, and thus the report may comprise one or more sentences about the patient and the examination, including one or more sentences or descriptors of a feature found within one or more images of the report.
  • a sentence or descriptor of the feature may describe a parameter of the feature, an appearance of the feature, a diagnosis of the feature, or may describe or define any other aspect of the feature.
  • the report or document database 280 may be local to the image -based query system, and may optionally be a component of the image -based query system.
  • the report or document database 280 may alternatively be remote to the image -based query system, and thus is in direct or indirection communication with the image -based query system.
  • the image-based query system receives, retrieves, or otherwise obtains a search image.
  • This received search image will be utilized to search the report or document database 280.
  • the received search image may be received from an imaging device 270 during an examination of a patient.
  • the imaging device may comprise a user interface via which a healthcare professional can submit the image to the image -based query system for searching.
  • the submission may be an entire medical image generated via the imaging device, or may be a region of interest (ROI) selected by a user of a medical imaging system.
  • the image -based query system may comprise a user interface via which the healthcare professional can submit the image or the ROI for searching as described below.
  • the user such as a healthcare professional indicates to the system that a search is desired or needed. This may be accomplished via a voice command, mouse click, screen touch, or any other method of providing information to the system, such as a user interface.
  • the system may prompt the user to provide the search information. Among other input, this may include a selection of an image and/or a ROI for searching.
  • the received search image is provided to a trained search model 262 of the image-based query system.
  • the trained search model 262 can be any algorithm, classifier, or model capable of creating the output, including but not limited to machine learning algorithms, classifiers, and other algorithms.
  • the trained algorithm is a unique algorithm based on the training data used to train the algorithm.
  • the trained search model 262 can be utilized or deployed immediately, or it may be stored in local and/or remote memory for future use and/or deployment.
  • the system comprises a trained search model 262 configured to identify search results as described or otherwise envisioned herein.
  • the search model 262 of the image -based query system is trained to pair a search image with one or more sentences from one or more documents in the report or document database 280.
  • some or all of the documents in the database can each comprise one or more images, such as images obtained using the same imaging modality as the search image, as well as one or more sentences related to the one or more images.
  • pairing a search image with one or more sentences from one or more documents in a report database, by the trained search model 262 of the image -based query system comprises generating an association score between the search image and the one or more sentences in each of one or more documents.
  • this association score is a measure of an association between the search image and the one or more sentences. Association methods and scores are further described herein.
  • the input to the trained search model 262 is therefore a search image or image ROI
  • the output from the trained search model 262 is a plurality of association scores between the search image and a plurality of documents, where each of the plurality of received documents is associated with a respective one of the plurality of generated association scores.
  • the number of association scores and/or the number of documents returned by the search model is predetermined, such as by a programmed threshold, limitation, or other indicator of a number of association scores and/or number of documents to be returned by the search model.
  • the number of association scores and/or the number of documents returned by the search model is determined by a user via input to the system. Other methods for determining the number of association scores and/or the number of documents returned by the search model are possible.
  • FIG. 3 in one embodiment, is a flowchart of a method 300 for training the trained search model 262 of the image -based query system.
  • This method may be performed by the image -based query system, or may be performed by another system such as a search model training system.
  • the system receives a training data set comprising a plurality of reports or similar documents, most or all of the documents each comprising one or more images obtained by one or more different imaging modalities, as well as a plurality of sentences, one or more of these plurality of sentences related to the one or more images.
  • a training data set comprising a plurality of reports or similar documents, most or all of the documents each comprising one or more images obtained by one or more different imaging modalities, as well as a plurality of sentences, one or more of these plurality of sentences related to the one or more images.
  • one or more sentences in the document may describe the image or an ROI of the image, such as a feature in the image.
  • the description may be an identification, diagnosis, or other reference to or description of the feature.
  • This corpus of documents may be stored in a database or system or component of the image -based query system or a training system, or may be stored in a database or system that is in direct or indirect communication with the image -based query system or a training system.
  • the image -based query system 200 comprises or is in direct or indirect communication with a report or document database 280 which comprises some or all of the training data set.
  • the training system may comprise a data pre-processor or similar component or algorithm configured to process the received training data.
  • the data pre-processor analyzes the training data to remove noise, bias, errors, and other potential issues.
  • the data pre-processor may also analyze the input data to remove low quality data. Many other forms of data pre-processing or data point identification and/or extraction are possible.
  • step 320 of the method the system iterates the following steps (i.e., steps 330 through
  • the system uses a trained image embedding model to process one or more images in the document to generate an image embedding for each image.
  • image embedding is a technique utilized to reduce the dimensionality of an image, which enables more efficient utilization of the image.
  • the trained image embedding model can be trained, for example, to generate as an output a vector that represents the image that is input into the model.
  • the model generates a lowerdimensional representation of the image by creating a vector representation of the image.
  • the trained image embedding model can be any algorithm, classifier, or model capable of creating the output, including but not limited to machine learning algorithms, classifiers, and other algorithms.
  • the trained algorithm is a unique algorithm based on the training data used to train the algorithm. Once generated, the trained image embedding model can be utilized or deployed immediately, or it may be stored in local and/or remote memory for future use and/or deployment.
  • the training system comprises a trained image embedding model configured to generate an image embedding for an input image.
  • the image embedding can be utilized immediately, or may be stored in local and/or remote storage for downstream use in the method.
  • FIG. 4 in one embodiment, is a block diagram representing a flowchart 400 for generating a training data set.
  • the training system receives or identifies a document/report from the training data set.
  • the system identifies an image or image ROI, such as comprising an image feature, and at step 430 of the method, the trained image embedding model generates an image embedding 440 for the image.
  • the system uses a trained text embedding model to process at least one sentence in the document to generate a text embedding for the at least one sentence.
  • text embedding is a natural language processing (NLP) technique utilized to generate a vector representation of the text.
  • the trained text embedding model can be trained, for example, to generate as an output a vector that represents the text that is input into the model.
  • the trained text embedding model can be an NLP transformer model, and can be for example a bidirectional encoder representations from transformers (BERT) model or a BERT- like model, among other models.
  • the model generates a lower-dimensional representation of the text by creating a vector representation of the text.
  • the trained text embedding model can be any algorithm, classifier, or model capable of creating the output, including but not limited to machine learning algorithms, classifiers, and other algorithms.
  • the trained algorithm is a unique algorithm based on the training data used to train the algorithm. Once generated, the trained text embedding model can be utilized or deployed immediately, or it may be stored in local and/or remote memory for future use and/or deployment.
  • the training system comprises a trained text embedding model configured to generate a text embedding for an input sentence.
  • the text embedding can be utilized immediately, or may be stored in local and/or remote storage for downstream use in the method.
  • the system is analyzing the same document as was analyzed above to generate the image embedding.
  • the system identifies a sentence in the document, and at step 432 of the method, the trained text embedding model generates a text embedding 442 for the sentence.
  • the pair of image embedding for an image/ROI and text embedding for a sentence is fed to a trained neural network configured to generate an association score between the image embedding and the text embedding.
  • the trained neural network can be any algorithm, classifier, or model capable of creating the output, including but not limited to machine learning algorithms, classifiers, and other algorithms.
  • the trained neural network is a unique neural network based on the training data used to train the neural network. Once generated, the trained neural network can be utilized or deployed immediately, or it may be stored in local and/or remote memory for future use and/or deployment.
  • the training system comprises a trained neural network configured to generate an association score between the image embedding and the text embedding.
  • the neural network is trained to generate a binary association classification (i.e., either “0” or “1”) between the image embedding and the text embedding, where a binary association classification of “1” indicates an association between the image embedding and the text embedding, and wherein a binary association classification of “0” indicates no association between the image embedding and the text embedding.
  • a binary association classification i.e., either “0” or “1
  • a report issuing from an MRI of a patient’s cervical spine may comprise or be associated in memory with an image of the patient’s cervical spine, and may comprise at least the following sentences: “Clinical information: patient is a florist,” and “Impression: multilevel degenerative disease and facet arthropathy, most severe from C4-5 through C6-7.”
  • the image of the patient’s cervical spine is reduced to an image embedding by the trained image embedding model.
  • each of the two sentences is reduced to an individual text embedding by the trained text embedding model.
  • the image embedding and each of the individual text embeddings are fed into the trained neural network configured to generate an association score between the image embedding and the text embedding.
  • the trained neural network may, for example, generate an association score of “0” indicating that there is no association between the image embedding and the text embedding for the first sentence (i.e., there is no association between the image of the patient’s cervical spine, and the sentence “Clinical information: patient is a florist.”).
  • the trained neural network may, for example, generate an association score of “1” indicating that there is an association between the image embedding and the text embedding for the second sentence (i.e., there is association between the image of the patient’s cervical spine, and the sentence “Impression: multilevel degenerative disease and facet arthropathy, most severe from C4-5 through C6-7.”).
  • association score may be stored in local and/or remote storage for downstream use in the method.
  • a document may be associated with numerous association scores, since a document may comprise a plurality of sentences and/or images.
  • the association score (in this example, “0” or “1”) is generated. This is repeated multiple times for a document, and the process is repeated for a large plurality of documents (preferably 100s or 1000s or many 1000s of documents/reports).
  • a training large data set that can be utilized to train a search model.
  • the search model of the image -based query system is trained using the training data set comprising, for each of the plurality of training documents/reports, an image embedding, text embedding(s), and a generated association score between the image embedding and each of the one or more text embedding(s).
  • the search model is trained to utilize this input to pair an image with one or more sentences from a document, comprising generating an association score between the image and the one or more sentences, the association score being a measure of an association between the image and the one or more sentences.
  • the model is trained using the training data set according to known methods for training a model. Following training, the system comprises a trained search model 262.
  • the trained search model 262 is stored for future use.
  • the trained search model 262 may be stored in local or remote storage.
  • a received search image is provided to a trained search model 262 of the image -based query system.
  • the input to the trained search model 262 is therefore a search image or image ROI
  • the output from the trained search model 262 is a plurality of association scores between the search image and a plurality of documents, where each of the plurality of received documents is associated with a respective one of the plurality of generated association scores.
  • the image -based query system receives from the trained search model 262 of the image -based query system a plurality of generated association scores for the search image and a plurality of documents.
  • Each of the plurality of generated association scores is associated with a respective one of the plurality of documents.
  • the number of association scores and/or the number of documents returned by the search model is predetermined, such as by a programmed threshold, limitation, or other indicator of a number of association scores and/or number of documents to be returned by the search model.
  • the number of association scores and/or the number of documents returned by the search model is determined by a user via input to the system.
  • the received plurality of generated association scores and/or the corresponding plurality of documents may be ranked by the respective association score, may be randomly ordered, or may otherwise be organized.
  • the plurality of generated association scores and the corresponding plurality of documents are received by the image -based query system, they can be utilized immediately, or may be stored in local and/or remote storage for downstream use in the method.
  • the image -based query system filters the received plurality of generated association scores and the corresponding plurality of documents according to one or more limiting parameters.
  • the system may receive additional information, before or after a search is conducted, that further limits the scores and documents received or returned by the system.
  • An example of a limiting parameter is a demographic parameter of a document, such as “biological female” or “age 26-30,” among many other parameters. Examples of other limiting parameters including imaging modality, date, medical history, treatment, or diagnosis, and many other limiting parameters.
  • This additional information can be provided to the image -based query system by a user such as a healthcare professional via a user interface, before and/or after a search is conducted.
  • the image-based query system can limit searching to only those documents that already satisfy the limiting parameter(s).
  • the system may organize documents according to one or more limiting parameters, and thus the system can optionally only search documents that already satisfy the limiting parameters.
  • step 142 there will be a plurality of generated association scores and a corresponding plurality of documents returned by the image -based query system, but these will satisfy both the search image query and the limiting parameters provided by the user.
  • This plurality of generated association scores and corresponding plurality of documents can be utilized immediately, or may be stored in local and/or remote storage for downstream use in the method.
  • the image -based query system provides the generated and returned or received output to a user via a user interface.
  • providing the output comprises providing at least one of the plurality of received documents, based on the generated association scores.
  • the provided document preferably includes the one or more sentences with the highest generated association score.
  • the provided document also preferably includes an image corresponding to the one or more sentences, which can be displayed or otherwise provided to the user.
  • the output can be provided to the user in any format that might be useful to that user, including but not limited to a report, image display such as a single image or a collage of two or more images, highlighted sentences, identified ROIs, and/or other formats.
  • the output comprises a plurality of documents from the report database, based on the generated association scores for each of those documents.
  • the provided documents may be provided because the association score with one or more sentences in the document exceed a predetermined or user-supplied association score threshold for display.
  • the plurality of documents can be ranked based on their respective association score.
  • FIG. 2 is a schematic representation of an image -based query system 200.
  • System 200 may be any of the systems described or otherwise envisioned herein, and may comprise any of the components described or otherwise envisioned herein. It will be understood that FIG. 2 constitutes, in some respects, an abstraction and that the actual organization of the components of the system 200 may be different and more complex than illustrated.
  • system 200 comprises a processor 220 capable of executing instructions stored in memory 230 or storage 260 or otherwise processing data to, for example, perform one or more steps of the method.
  • Processor 220 may be formed of one or multiple modules.
  • Processor 220 may take any suitable form, including but not limited to a microprocessor, microcontroller, multiple microcontrollers, circuitry, field programmable gate array (FPGA), application-specific integrated circuit (ASIC), a single processor, or plural processors.
  • FPGA field programmable gate array
  • ASIC application-specific integrated circuit
  • Memory 230 can take any suitable form, including a non-volatile memory and/or RAM.
  • the memory 230 may include various memories such as, for example LI, L2, or L3 cache or system memory.
  • the memory 230 may include static random access memory (SRAM), dynamic RAM (DRAM), flash memory, read only memory (ROM), or other similar memory devices.
  • SRAM static random access memory
  • DRAM dynamic RAM
  • ROM read only memory
  • the memory can store, among other things, an operating system.
  • the RAM is used by the processor for the temporary storage of data.
  • an operating system may contain code which, when executed by the processor, controls operation of one or more components of system 200. It will be apparent that, in embodiments where the processor implements one or more of the functions described herein in hardware, the software described as corresponding to such functionality in other embodiments may be omitted.
  • User interface 240 may include one or more devices for enabling communication with a user.
  • the user interface can be any device or system that allows information to be conveyed and/or received, and may include a display, a mouse, and/or a keyboard for receiving user commands.
  • user interface 240 may include a command line interface or graphical user interface that may be presented to a remote terminal via communication interface 250.
  • the user interface may be located with one or more other components of the system, or may located remote from the system and in communication via a wired and/or wireless communications network.
  • Communication interface 250 may include one or more devices for enabling communication with other hardware devices.
  • communication interface 250 may include a network interface card (NIC) configured to communicate according to the Ethernet protocol.
  • NIC network interface card
  • communication interface 250 may implement a TCP/IP stack for communication according to the TCP/IP protocols.
  • TCP/IP protocols Various alternative or additional hardware or configurations for communication interface 250 will be apparent.
  • Storage 260 may include one or more machine -readable storage media such as readonly memory (ROM), random-access memory (RAM), magnetic disk storage media, optical storage media, flash-memory devices, or similar storage media.
  • ROM readonly memory
  • RAM random-access memory
  • storage 260 may store instructions for execution by processor 220 or data upon which processor 220 may operate.
  • storage 260 may store an operating system 261 for controlling various operations of system 200.
  • memory 230 may also be considered to constitute a storage device and storage 260 may be considered a memory.
  • memory 230 and storage 260 may both be considered to be non-transitory machine -readable media.
  • non-transitory will be understood to exclude transitory signals but to include all forms of storage, including both volatile and non-volatile memories.
  • processor 220 may include multiple microprocessors that are configured to independently execute the methods described herein or are configured to perform steps or subroutines of the methods described herein such that the multiple processors cooperate to achieve the functionality described herein.
  • processor 220 may include a first processor in a first server and a second processor in a second server. Many other variations and configurations are possible.
  • system 200 comprises or is in direct or indirect communication with an imaging device and/or image database 270.
  • the imaging device may be any imaging device, and may obtain one or more images using any imaging modality.
  • the most common forms of imaging modality are X-ray, magnetic resonance imaging (MRI), ultrasound, computed tomography scan (CT scan), and nuclear imaging such as Positron Emission Tomography (PET), although many other types of health- or medicine -based imaging modalities are possible.
  • the one or more images obtained using the imaging modality may be obtained from a patient or other individual.
  • an image database may be a database utilized to store one or more images of a patient, obtained by an imaging device.
  • a radiologist, physician, clinician, researcher or other healthcare individual may review those stores one or more images as part of an examination.
  • the imaging device and/or image database 270 may be local to the image -based query system, and may optionally be a component of the image-based query system.
  • the imaging device and/or image database 270 may alternatively be remote to the image-based query system, and thus is in direct or indirection communication with the image -based query system.
  • system 200 comprises or is in direct or indirect communication with a report or document database 280.
  • the report or document database 280 is a database of 1000s or more of reports relevant to at least the imaging modality used by the radiologist, physician, clinician, researcher or other healthcare individual during an examination.
  • the report or document database 280 preferably comprises at least 100s or 1000s of reports or documents of prior MRI examinations.
  • the report or document database 280 preferably comprises at least 100s or 1000s of reports or documents of prior X-ray examinations.
  • the report or document database 280 may comprise a corpus of reports/documents for a plurality of different imaging modalities.
  • the report or document database 280 may be local to the imagebased query system, and may optionally be a component of the image -based query system.
  • the report or document database 280 may alternatively be remote to the image -based query system, and thus is in direct or indirection communication with the image -based query system.
  • storage 260 of system 200 may store one or more algorithms, modules, and/or instructions to carry out one or more functions or steps of the methods described or otherwise envisioned herein.
  • storage 260 may comprise, among other instructions or data, a trained search model 262, training instructions 263, and/or reporting instructions 264.
  • the trained search model 262 of the image -based query system is trained to pair a search image with one or more sentences from one or more documents in the report or document database 280.
  • the trained search model 262 can be any algorithm, classifier, or model capable of creating the output, including but not limited to machine learning algorithms, classifiers, and other algorithms.
  • the trained algorithm is a unique algorithm based on the training data used to train the algorithm. Once generated, the trained search model 262 can be utilized or deployed immediately, or it may be stored in local and/or remote memory for future use and/or deployment.
  • the system comprises a trained search model 262 configured to identify search results as described or otherwise envisioned herein.
  • training instructions 263 direct the system to train a search model 262 of the image-based query system.
  • the instructions direct the system to: at step 310 of the method 300 in FIG. 3, for example, retrieve, obtain, or receive a training data set comprising a plurality of reports or similar documents, most or all of the documents each comprising one or more images obtained by one or more different imaging modalities, as well as a plurality of sentences, one or more of these plurality of sentences related to the one or more images.
  • the system iterates the following steps (i.e., steps 330 through 350) many times for each of a plurality of the documents in the training data set.
  • the system uses a trained image embedding model to process one or more images in the document to generate an image embedding for each image.
  • the system uses a trained text embedding model to process at least one sentence in the document to generate a text embedding for the at least one sentence.
  • the pair of image embedding for an image/ROI and text embedding for a sentence is fed to a trained neural network configured to generate an association score between the image embedding and the text embedding.
  • a trained neural network configured to generate an association score between the image embedding and the text embedding.
  • the search model of the image-based query system is trained using the training data set comprising, for each of the plurality of training documents/reports, an image embedding, text embedding(s), and a generated association score between the image embedding and each of the one or more text embedding(s).
  • the trained search model 262 is stored for future use.
  • reporting instructions 264 direct the system to provide the output of the trained search model to a user via a user interface.
  • the provided output can be any of the information as described or otherwise envisioned herein.
  • the system may provide the information to a user via any mechanism, including but not limited to a visual display, an audible notification, a text message, an email, a page, or any other method of notification.
  • the information may be communicated by wired and/or wireless communication to another device.
  • the system may communicate the information to a mobile phone, computer, laptop, wearable device, and/or any other device configured to allow display and/or other communication of the information.
  • the image -based query system 200 is configured to process many thousands or millions of datapoints in the input data used to train the search model 262, such as via the training instructions 263.
  • generating a functional and skilled trained search model from a corpus of training data requires processing of millions of datapoints from input data and generated features. This can require millions or billions of calculations to generate a novel trained search model from those millions of datapoints and millions or billions of calculations.
  • each trained search model is novel and distinct based on the input data and parameters of the machine learning algorithm, and thus improves the functioning of the image- based query system.
  • Generating a functional and skilled trained search model comprises a process with a volume of calculation and analysis that a human brain cannot accomplish in a lifetime, or multiple lifetimes.
  • the novel image -based query system compares the search image to sentences in documents/reports in the search database using a novel trained search model, and identifies associations between the search image and one or more sentences in one or more reports of the search database. Comparing a search image to text, or a representation of text, is significantly faster and more computationally efficient than comparing a search image to an image (or a representation of an image). When multiplied by 100s or 1000s of comparisons, this increase in speed and computational efficiency results in significant improvement to the functionality of the image -based query systems. As a user’s time is valuable, and even as this search may occur during the middle of a real-time examination in some examples, saving time can result in better patient care and outcomes compared to slow and computationally inefficient prior art systems.
  • the phrase “at least one,” in reference to a list of one or more elements, should be understood to mean at least one element selected from any one or more of the elements in the list of elements, but not necessarily including at least one of each and every element specifically listed within the list of elements and not excluding any combinations of elements in the list of elements.
  • This definition also allows that elements may optionally be present other than the elements specifically identified within the list of elements to which the phrase “at least one” refers, whether related or unrelated to those elements specifically identified.
  • inventive embodiments are presented by way of example only and that, within the scope of the appended claims and equivalents thereto, inventive embodiments may be practiced otherwise than as specifically described and claimed.
  • inventive embodiments of the present disclosure are directed to each individual feature, system, article, material, kit, and/or method described herein.

Landscapes

  • Engineering & Computer Science (AREA)
  • Health & Medical Sciences (AREA)
  • Public Health (AREA)
  • Physics & Mathematics (AREA)
  • Epidemiology (AREA)
  • General Health & Medical Sciences (AREA)
  • Medical Informatics (AREA)
  • Primary Health Care (AREA)
  • Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
  • Radiology & Medical Imaging (AREA)
  • Theoretical Computer Science (AREA)
  • Multimedia (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Mathematical Physics (AREA)
  • Medical Treatment And Welfare Office Work (AREA)

Abstract

L'invention concerne un procédé (100) d'interrogation basée sur une image, comprenant : la réception (120) d'une image de recherche; la fourniture (130) de l'image de recherche reçue à un modèle de recherche entraîné (262) configuré pour apparier une image de recherche avec une ou plusieurs phrases à partir d'un ou de plusieurs documents dans une base de données de rapports, l'appariement d'une image de recherche avec une ou plusieurs phrases à partir d'un ou de plusieurs documents dans une base de données de rapports comprenant la génération d'un score d'association entre l'image de recherche et la ou les phrases; la réception (140), en provenance du modèle de recherche entraîné, d'une pluralité de scores d'association générés pour l'image de recherche et une pluralité de documents, chacun de la pluralité de documents reçus étant associé à un score respectif de la pluralité de scores d'association générés; et la fourniture (150) d'au moins l'un de la pluralité de documents reçus, comprenant la ou les phrases ayant le score d'association généré le plus élevé.
PCT/EP2023/086792 2022-12-22 2023-12-20 Procédés et systèmes d'interrogation basée sur une image pour des caractéristiques radiographiques similaires WO2024133367A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US202263434484P 2022-12-22 2022-12-22
US63/434,484 2022-12-22

Publications (1)

Publication Number Publication Date
WO2024133367A1 true WO2024133367A1 (fr) 2024-06-27

Family

ID=89508938

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/EP2023/086792 WO2024133367A1 (fr) 2022-12-22 2023-12-20 Procédés et systèmes d'interrogation basée sur une image pour des caractéristiques radiographiques similaires

Country Status (1)

Country Link
WO (1) WO2024133367A1 (fr)

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20120191720A1 (en) * 2009-10-01 2012-07-26 Koninklijke Philips Electronics N.V. Retrieving radiological studies using an image-based query
US20130259350A1 (en) * 2011-08-04 2013-10-03 Panasonic Corporation Similar case searching apparatus and similar case searching method
US20200019617A1 (en) * 2018-07-11 2020-01-16 Google Llc Similar Image Search for Radiology

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20120191720A1 (en) * 2009-10-01 2012-07-26 Koninklijke Philips Electronics N.V. Retrieving radiological studies using an image-based query
US20130259350A1 (en) * 2011-08-04 2013-10-03 Panasonic Corporation Similar case searching apparatus and similar case searching method
US20200019617A1 (en) * 2018-07-11 2020-01-16 Google Llc Similar Image Search for Radiology

Similar Documents

Publication Publication Date Title
Qayyum et al. Medical image retrieval using deep convolutional neural network
US11176188B2 (en) Visualization framework based on document representation learning
JP6749835B2 (ja) コンテキスト依存医学データ入力システム
Avni et al. X-ray categorization and retrieval on the organ and pathology level, using patch-based visual words
JP5056695B2 (ja) 類似画像提示装置及びプログラム
US9792681B2 (en) System and method for medical image analysis and probabilistic diagnosis
US10474742B2 (en) Automatic creation of a finding centric longitudinal view of patient findings
JP5736007B2 (ja) 検査報告書を生成する装置、システム、方法およびプログラム
US20110317892A1 (en) Method and system of classifying medical images
López-Úbeda et al. Automatic medical protocol classification using machine learning approaches
US11699508B2 (en) Method and apparatus for selecting radiology reports for image labeling by modality and anatomical region of interest
US20170185713A1 (en) Automatic detection of medical image acquisition protocol
Depeursinge et al. Comparative performance analysis of state-of-the-art classification algorithms applied to lung tissue categorization
US20180060487A1 (en) Method for automatic visual annotation of radiological images from patient clinical data
CA3060599C (fr) Explication de la recherche semantique
Tirilly et al. On modality classification and its use in text-based image retrieval in medical databases
Agarwal et al. Weakly-supervised lesion segmentation on CT scans using co-segmentation
CN111199801B (zh) 一种用于识别病历的疾病类型的模型的构建方法及应用
WO2024133367A1 (fr) Procédés et systèmes d'interrogation basée sur une image pour des caractéristiques radiographiques similaires
Murugan et al. Efficient clustering of unlabeled brain DICOM images based on similarity
CN115293163A (zh) 一种画像方法、装置、设备和存储介质
US11676702B2 (en) Method for automatic visual annotation of radiological images from patient clinical data
Xu et al. Texture classification of normal tissues in computed tomography
Wang et al. Iterated large-margin discriminant analysis for feature dimensionality reduction in medical image retrieval
US20240177818A1 (en) Methods and systems for summarizing densely annotated medical reports