US20190220978A1 - Method for integrating image analysis, longitudinal tracking of a region of interest and updating of a knowledge representation - Google Patents
Method for integrating image analysis, longitudinal tracking of a region of interest and updating of a knowledge representation Download PDFInfo
- Publication number
- US20190220978A1 US20190220978A1 US16/363,032 US201916363032A US2019220978A1 US 20190220978 A1 US20190220978 A1 US 20190220978A1 US 201916363032 A US201916363032 A US 201916363032A US 2019220978 A1 US2019220978 A1 US 2019220978A1
- Authority
- US
- United States
- Prior art keywords
- interest
- region
- image
- database
- knowledge representation
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 85
- 238000010191 image analysis Methods 0.000 title claims abstract description 11
- 238000003384 imaging method Methods 0.000 claims abstract description 39
- 238000013507 mapping Methods 0.000 claims abstract description 22
- 238000012544 monitoring process Methods 0.000 claims abstract description 13
- 210000000056 organ Anatomy 0.000 claims description 29
- 238000011282 treatment Methods 0.000 claims description 22
- 210000003484 anatomy Anatomy 0.000 claims description 11
- 238000012549 training Methods 0.000 claims description 11
- 238000007792 addition Methods 0.000 claims description 9
- 238000010801 machine learning Methods 0.000 claims description 4
- 238000004422 calculation algorithm Methods 0.000 description 79
- 210000000481 breast Anatomy 0.000 description 24
- 230000000875 corresponding effect Effects 0.000 description 18
- 230000005856 abnormality Effects 0.000 description 15
- 238000004891 communication Methods 0.000 description 10
- 238000002591 computed tomography Methods 0.000 description 10
- 230000003902 lesion Effects 0.000 description 10
- 230000008569 process Effects 0.000 description 9
- 238000003745 diagnosis Methods 0.000 description 7
- 208000004434 Calcinosis Diseases 0.000 description 6
- 238000012986 modification Methods 0.000 description 6
- 230000004048 modification Effects 0.000 description 6
- 238000010586 diagram Methods 0.000 description 5
- 239000003814 drug Substances 0.000 description 5
- 238000009607 mammography Methods 0.000 description 5
- 230000015654 memory Effects 0.000 description 5
- 210000002445 nipple Anatomy 0.000 description 5
- 230000004044 response Effects 0.000 description 5
- 206010047571 Visual impairment Diseases 0.000 description 4
- 229940079593 drug Drugs 0.000 description 4
- 238000002594 fluoroscopy Methods 0.000 description 4
- 230000002068 genetic effect Effects 0.000 description 4
- 230000036541 health Effects 0.000 description 4
- 230000007170 pathology Effects 0.000 description 4
- 238000002604 ultrasonography Methods 0.000 description 4
- 206010028980 Neoplasm Diseases 0.000 description 3
- 230000003542 behavioural effect Effects 0.000 description 3
- 201000011510 cancer Diseases 0.000 description 3
- 208000037265 diseases, disorders, signs and symptoms Diseases 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 210000003205 muscle Anatomy 0.000 description 3
- 210000000062 pectoralis major Anatomy 0.000 description 3
- 210000001519 tissue Anatomy 0.000 description 3
- 230000009471 action Effects 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 2
- 238000013475 authorization Methods 0.000 description 2
- 230000008901 benefit Effects 0.000 description 2
- 210000000988 bone and bone Anatomy 0.000 description 2
- 230000002308 calcification Effects 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 238000010276 construction Methods 0.000 description 2
- 238000013527 convolutional neural network Methods 0.000 description 2
- 238000013479 data entry Methods 0.000 description 2
- 238000002059 diagnostic imaging Methods 0.000 description 2
- 201000010099 disease Diseases 0.000 description 2
- 230000002452 interceptive effect Effects 0.000 description 2
- 238000005259 measurement Methods 0.000 description 2
- 230000007246 mechanism Effects 0.000 description 2
- 235000016709 nutrition Nutrition 0.000 description 2
- 230000037081 physical activity Effects 0.000 description 2
- 238000001454 recorded image Methods 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- 210000004872 soft tissue Anatomy 0.000 description 2
- 238000012546 transfer Methods 0.000 description 2
- 238000011269 treatment regimen Methods 0.000 description 2
- 230000000007 visual effect Effects 0.000 description 2
- 206010006187 Breast cancer Diseases 0.000 description 1
- 206010006272 Breast mass Diseases 0.000 description 1
- 208000026310 Breast neoplasm Diseases 0.000 description 1
- 206010061818 Disease progression Diseases 0.000 description 1
- 208000034826 Genetic Predisposition to Disease Diseases 0.000 description 1
- 206010054107 Nodule Diseases 0.000 description 1
- 208000025865 Ulcer Diseases 0.000 description 1
- 230000002159 abnormal effect Effects 0.000 description 1
- 238000010521 absorption reaction Methods 0.000 description 1
- 230000004075 alteration Effects 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000001574 biopsy Methods 0.000 description 1
- 238000004159 blood analysis Methods 0.000 description 1
- 238000012512 characterization method Methods 0.000 description 1
- 238000007635 classification algorithm Methods 0.000 description 1
- 150000001875 compounds Chemical class 0.000 description 1
- 238000004590 computer program Methods 0.000 description 1
- 238000004195 computer-aided diagnosis Methods 0.000 description 1
- 230000002596 correlated effect Effects 0.000 description 1
- 238000013480 data collection Methods 0.000 description 1
- 238000003066 decision tree Methods 0.000 description 1
- 230000007812 deficiency Effects 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 238000002405 diagnostic procedure Methods 0.000 description 1
- 230000005750 disease progression Effects 0.000 description 1
- 208000035475 disorder Diseases 0.000 description 1
- 239000000446 fuel Substances 0.000 description 1
- 238000003709 image segmentation Methods 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 210000004072 lung Anatomy 0.000 description 1
- 230000036210 malignancy Effects 0.000 description 1
- 210000005075 mammary gland Anatomy 0.000 description 1
- 244000005700 microbiome Species 0.000 description 1
- 230000035764 nutrition Effects 0.000 description 1
- 238000003909 pattern recognition Methods 0.000 description 1
- 230000000144 pharmacologic effect Effects 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 238000000275 quality assurance Methods 0.000 description 1
- 238000012552 review Methods 0.000 description 1
- 238000012216 screening Methods 0.000 description 1
- 230000000391 smoking effect Effects 0.000 description 1
- 238000001356 surgical procedure Methods 0.000 description 1
- 208000024891 symptom Diseases 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
- 230000036269 ulceration Effects 0.000 description 1
- 230000001755 vocal effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
- G06T7/0012—Biomedical image inspection
- G06T7/0014—Biomedical image inspection using an image reference approach
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G16H15/00—ICT specially adapted for medical reports, e.g. generation or transmission thereof
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G16H30/00—ICT specially adapted for the handling or processing of medical images
- G16H30/20—ICT specially adapted for the handling or processing of medical images for handling medical images, e.g. DICOM, HL7 or PACS
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G16H50/00—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics
- G16H50/20—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics for computer-aided diagnosis, e.g. based on medical expert systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10004—Still image; Photographic image
- G06T2207/10008—Still image; Photographic image from scanner, fax or copier
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10072—Tomographic images
- G06T2207/10081—Computed x-ray tomography [CT]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10116—X-ray image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30004—Biomedical image processing
- G06T2207/30068—Mammography; Breast
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G16H50/00—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics
- G16H50/70—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics for mining of medical data, e.g. analysing previous cases of other patients
Landscapes
- Engineering & Computer Science (AREA)
- Health & Medical Sciences (AREA)
- Medical Informatics (AREA)
- General Health & Medical Sciences (AREA)
- Public Health (AREA)
- Epidemiology (AREA)
- Primary Health Care (AREA)
- Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
- Radiology & Medical Imaging (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Quality & Reliability (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Biomedical Technology (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Pathology (AREA)
- Measuring And Recording Apparatus For Diagnosis (AREA)
- Apparatus For Radiation Diagnosis (AREA)
- Medical Treatment And Welfare Office Work (AREA)
Abstract
Description
- This patent application claims the benefit of priority to U.S. Utility patent application Ser. No. 14/093,470 filed Nov. 30, 2013 which was published as us 2014/0219500, which in turn claims priority to U.S. Utility patent application Ser. No. 13/188,415 filed Jul. 21, 2011 and issued as U.S. Pat. No. 9,014,485 on Apr. 21, 2015, which in turn claims priority to U.S. provisional patent application No. 61/366,492 filed Jul. 21, 2010, each above-identified application is incorporated by reference in its entirety.
- The present disclosure generally relates to image interpretation, and more particularly, to systems and methods for generating image reports.
- In current image interpretation practice, such as diagnostic radiology, a specialist trained in interpreting images and recognizing abnormalities may look at an image or an image sequence on an image display and report any visual findings by dictating or typing the findings into a report template. The dictating or typing usually includes a narration of the finding, a description about the location of the visual phenomena, abnormality, or region of interest within the images being reported on. The recipient of the report is often left to further analyze the contents of the narrative text report without having easy access to the underlying image. More particularly, in current reporting practice, there is no link between the specific location in the image and the finding associated with the visual phenomena, abnormality, or region of interest, in the image. A specialist also may have to compare a current image with an image and report previously done. This leaves the interpreter to refer back and forth between the image and the report.
- Computer-aided detection (CAD) systems are known in the art and are usually confined to detecting and classifying conspicuous structures in the image data. Computer-aided diagnosis (CAD) systems are used in mammography to highlight micro calcification clusters and hyperdense structures in the soft tissue. Computer-aided simple triage (CAST) is another type of CAD, which performs a fully automatic initial interpretation and triage of studies into some meaningful categories (e.g. negative and positive). Unfortunately, these prior art systems are limited to describing the location of the visual phenomena within the image file. By manner of illustration, the coordinate system provided by the CAD system cannot be used to guide a biopsy needle because it fails to identify the relative position within the organ or sample structure.
- While such inconveniences may pose a seemingly insignificant risk of error, a typical specialist must interpret a substantial amount of such images in short periods of time, which further compounds the specialist's fatigue and vulnerability to oversights. This is especially critical when the images to be interpreted are medical images of patients with their health being at risk.
- General articulation and narration of an image interpretation may be facilitated with reference to structured reporting templates or knowledge representations. One example of a knowledge representation in the form of a semantic network is the Systematized Nomenclature of Medicine-Clinical Terms (SNOMED-CT), which is a systematically organized and computer processable collection of medical terminology covering most areas of clinical information, such as diseases, findings, procedures, microorganisms, pharmaceuticals, and the like. SNOMED-CT provides a consistent way to index, store, retrieve, and aggregate clinical data across various specialties and sites of care. SNOMED-CT also helps in organizing the content of medical records, and in reducing the inconsistencies in the way data is captured, communicated, encoded, and used for clinical care of patients and research.
- Another example is the Breast Imaging-Reporting and Data System (BI-RADS), which is a quality assurance tool originally designed for use with mammography. Yet another example is RadLex, a lexicon for uniform indexing and retrieval of radiology information resources, which currently includes more than 30,000 terms. Applications include radiology decision support, reporting tools and search applications for radiology research and education. Reporting templates developed by the Radiological Society of North America (RSNA) Reporting Committee use RadLex terms in their content. Reports using RadLex terms are clearer and more consistent, reducing the potential for error and confusion. RadLex includes other lexicons and semantic networks, such as SNOMED-CT, BI-RADS, as well as any other system or combination of systems developed to help structure and standardize reporting. Richer forms of semantic networks in terms of knowledge representation are ontologies. Knowledge representations may also include probability models and identifying characteristics from image data generated by image segmentation and classification algorithms. Ontologies are encoded using ontology languages and commonly include the following components: instances (the basic or “ground level” objects), classes (sets, collections, concepts, classes in programming, types of objects, or kinds of things), attributes (aspects, properties, features, characteristics, or parameters that objects), relations (ways in which classes and individuals can be related to one another), function terms (complex structures formed from certain relations that can be used in place of an individual term in a statement), restrictions (formally stated descriptions of what must be true in order for some assertion to be accepted as input), rules (statements in the form of an if-then sentence that describe the logical inferences that can be drawn from an assertion in a particular form, axioms (assertions, including rules, in a logical form that together comprise the overall theory that the ontology describes in its domain of application), and events (the changing of attributes or relations).
- Currently existing image reporting mechanisms do not take full advantage of knowledge representations to assist interpretation while automating reporting. In particular, currently existing systems are not fully integrated with knowledge representations to provide seamless and effortless reference to knowledge representations during articulation of findings. Additionally, in order for such a knowledge representation interface to be effective, there must be a brokering service between the various forms of standards and knowledge representations that constantly evolve. While there is a general lack of such brokering service between the entities of most domains, there is an even greater deficiency in the available means to promote common agreements between terminologies, especially in image reporting applications. Furthermore, due to the lack of more streamlined agreements (alignment) between knowledge representations in image reporting, currently existing systems also lack means for automatically tracking the development of specific and related cases for inconsistencies or errors so that the knowledge representations may be updated to provide more accurate information in subsequent cases. Such tracking means provide the basis for a probability model for knowledge representations.
- In light of the foregoing, there is a need for an improved system and method for integrating image analysis, longitudinal tracking of a region of interest and updating of a knowledge representation.
- Disclosed is a method for integrating image analysis, longitudinal tracking of a region of interest and updating of a knowledge representation, said method comprising the steps of:
- retrieving an image representation of a sample structure from an image database;
- automatically selecting a generic structure from a database containing a plurality of generic structures based on an imaging modality of the sample structure, at least one knowledge representation stored in a second database, said knowledge representation associated with said selected generic structure, the knowledge representation being specific to the imaging modality;
- mapping the selected generic structure to the sample structure;
- automatically determining at least one region of interest within the sample structure or allowing the user to select a region of interest;
- automatically selecting at least one diagnostic finding or allowing the user to select at least one diagnostic finding from a focused set knowledge representations;
- retrievably storing the at least one diagnostic finding in an electronic record; and
- monitoring the electronic record for changes to the at least one diagnostic finding or new diagnostic findings and using such changes or new diagnostic findings to update the knowledge representation in the second database.
- The aforementioned method wherein the step of allowing the user to select at least one diagnostic finding from the focused set of knowledge representation includes allowing the user to enter the at least one diagnostic finding using free-form text.
- The aforementioned method wherein the selected generic structure is related to the sample structure by imaging modality and one or more attributes selected from the group (size, dimensions, area, shape, volume, weight, density, location, anatomical organ, and orientation).
- The aforementioned method further, wherein the selected generic structure has coordinate data defined therein.
- The aforementioned method, further comprising:
- using the coordinate data to generate natural language statements describing a location of the region of interest in the anatomy;
- automatically generating a diagnostic report based on the at least one diagnostic finding, and including the natural language statements describing the location in the anatomy of the region of interest; and
- storing the diagnostic report in the electronic record.
- The aforementioned method, wherein the knowledge representation is specific to an anatomical organ in which the region of interest is located and the imaging modality.
- The aforementioned method further comprising:
- automatically generating a diagnostic report based on the selections or free-form text entries, and including natural language statements describing the location in the anatomy of the region of interest; and storing the diagnostic report in the electronic record.
- The aforementioned method, wherein the step of automatically selecting a generic structure from among a plurality of generic structures is based on the imaging modality and a comparison of content of the sample structure to the content of the generic structure.
- The aforementioned method further comprising:
- for each at least one region of interest automatically selecting follow-up care or prompting the user to select from a focused set of follow-up care options; and storing the selected follow-up care in the electronic record.
- The aforementioned method, wherein the step of monitoring the electronic record includes checking for changes to the selected follow-up care and using such changes to update the knowledge representation in the second database.
- The aforementioned method, wherein the step of monitoring the electronic record includes checking for changes to treatment outcome and using such changes to update the knowledge representation in the second database.
- Also disclosed is a method for integrating image analysis, longitudinal tracking of a region of interest and updating of a knowledge representation, comprising the steps of:
- retrieving an image representation of sample structure depicting at least a portion of an anatomical organ from an image database;
- determining at least one region of interest within the sample structure or allowing the user to select a region of interest;
- automatically selecting at least one diagnostic finding or allowing the user to select at least one diagnostic finding from a focused set of knowledge representations specific to the anatomical organ and an imaging modality used to capture the image representation;
- retrievably storing the at least one diagnostic finding in the electronic record;
- monitoring the electronic record for changes and/or additions to the at least one diagnostic finding and updating the knowledge representation to reflect the changes and/or additions to the at least one diagnostic finding.
- Also disclosed is a method for integrating image analysis, longitudinal tracking of a region of interest and updating of a knowledge representation, the method comprising the steps of:
- recording at least one diagnostic finding for a given region of interest in an image database;
- monitoring the electronic record for changes to the at least one diagnostic finding for the region of interest; and
- automatically updating a knowledge representation stored in a database to reflect the changes to the at least one diagnostic finding for the region of interest.
- The aforementioned method, further comprising:
- retrieving an image representation of sample structure depicting at least a portion of an anatomical organ from an image database;
- automatically determining at least one region of interest within the sample structure or allowing the user to select a region of interest;
- automatically selecting at least one diagnostic finding or allowing the user to select at least one diagnostic finding from a focused set of knowledge representations specific to at least one of the anatomical organ and an imaging modality used to capture the image representation; and
- retrievably storing the at least one diagnostic finding in the electronic record.
- These and other aspects of this disclosure will become more readily apparent upon reading the following detailed description when taken in conjunction with the accompanying drawings.
-
FIG. 1 is a diagrammatic view of an exemplary system for supporting an image reporting method; -
FIG. 2 is a schematic view of an exemplary image reporting device constructed in accordance with the teachings of the disclosure; -
FIG. 3 is a diagrammatic view of an exemplary algorithm for image reporting; -
FIGS. 4A-4B are diagrammatic views of a sample structure; -
FIG. 5 is a diagrammatic view of a three-dimensional mapping technique; -
FIG. 6 is a diagrammatic view of a related three-dimensional mapping technique; -
FIGS. 7A-7C are diagrammatic views of a generic structure; -
FIGS. 8A-8D are illustrative views of a warping process; -
FIGS. 9A-9B are diagrammatic views of another sample structure; -
FIGS. 10A-10B are diagrammatic views of yet another sample structure; -
FIGS. 11A-11C are diagrammatic views of exemplary image reports; -
FIG. 12 is a schematic view of an exemplary image reporting system integrated with knowledge representation systems; -
FIGS. 13A and 13B is another diagrammatic view of a sample structure also illustrating knowledge representations; -
FIG. 14 is a diagrammatic view of showing the user being prompted to select from one of the guidelines or enter a user specified instruction for follow-up care; -
FIG. 15 is a diagrammatic view of a knowledge base or ontology showing recommended treatments for each of a plurality of diagnoses; -
FIG. 16 is a diagrammatic view of a sample structure showing a region of interest; -
FIG. 17 is a table providing both current and historical information regarding a region of interest; -
FIGS. 18-20 are flow diagrams of a method for longitudinally tracking changes to diagnostic findings; and -
FIGS. 21-22 are diagrams depicting the utilization of neural networks. - While the present disclosure is susceptible to various modifications and alternative constructions, certain illustrative embodiments thereof have been shown in the drawings and will be described below in detail. It should be understood, however, that there is no intention to limit the present invention to the specific forms disclosed, but on the contrary, the intention is to cover all modifications, alternative constructions, and equivalents falling with the spirit and scope of the present invention.
- Referring now to
FIG. 1 , anexemplary system 100 within which an image interpretation and reporting method may be integrated is provided. As shown, thesystem 100 may include acentral network 102 by which different components of thesystem 100 may communicate. For example, thenetwork 102 may take the form of a wired and/or wireless local area network (LAN), a wide area network (WAN), such as the Internet, a wireless local area network (WLAN), a storage or server area network (SAN), and the like. Thesystem 100 may also includeimage capture devices 104 configured to capture or generate two-dimensional and/or three-dimensional images. In medical imaging, for example, theimage capture devices 104 may include one or more of a mammography device, a computed tomography (CT) device, an ultrasound device, an X-ray device, a fluoroscopy device, a film printer, a film digitizer, and the like. One or more images of a sample structure captured by theimage capture devices 104 may be transmitted to animage server 106 and/or animage database 108 directly or through anetwork 102. - The
image server 106,image database 108 and/ornetwork 102 ofFIG. 1 may be configured to manage the overall storage, retrieval and transfer of images, as in Picture Archiving and Communication System (PACS) in accordance with Digital Imaging and Communications in Medicine (DICOM) standards, for example. In medical applications, each medical image stored in the DICOM database may include, for instance, a header containing relevant information, such as the patient name, the patient identification number, the image type, the scan type, or any other classification type by which the image may be retrieved. Based on the classification type, theserver 106 may determine where and how specific images are stored, associate the images with any additional information required for recalling the images, sort the images according to relevant categories and manage user access to those images. In further alternatives, the storage, retrieval and transfer of images may be managed and maintained within thenetwork 102 itself so as to enable services, for example, in an open source platform for individual users from any node with access thenetwork 102. In an application related to medical imaging, for example, each medical image may be tied to a particular patient, physician, symptom, diagnosis, or the like. The stored images may then be selectively recalled or retrieved at ahost 110. - As shown in
FIG. 1 , one ormore hosts 110 may be provided within thesystem 100 and configured to communicate with other nodes of thesystem 100 via thenetwork 102. Specifically, users with appropriate authorization may connect to theimage server 106 and/orimage database 108 via thenetwork 102 to access the images stored within theimage database 108. In medical applications, for example, ahost 110 may be used by a physician, a patient, a radiologist, or any other user granted access thereto. In alternative embodiments, thesystem 100 may be incorporated into a more localized configuration wherein thehost 110 may be in direct communication with one or moreimage capture devices 104 and/or animage database 108. - Turning now to
FIG. 2 , one exemplaryimage reporting device 200 as applied at ahost 110 is provided. Theimage reporting device 200 may essentially include acomputational device 202 and auser interface 204 providing user access to thecomputational device 202. Theuser interface 204 may include at least oneinput device 206 which provides, for example, one or more of a keypad, a keyboard, a pointing device, a microphone, a camera, a touch screen, or any other suitable device for receiving user input. Theuser interface 204 may further include at least one output orviewing device 208, such as a monitor, screen, projector, touch screen, printer, or any other suitable device for outputting information to a user. Each of theinput device 206 and theviewing device 208 may be configured to communicate with thecomputational device 202. - In the particular
image reporting device 200 ofFIG. 2 , thecomputational device 202 may include at least one controller ormicroprocessor 210 and a storage device ormemory 212 configured to perform image interpretation and/or reporting. More specifically, thememory 212 may be configured to at least one algorithm for performing the image reporting function, while themicroprocessor 210 may be configured to execute computations and actions for performing according to the stored algorithm. In alternative embodiments, themicroprocessor 210 may include on-board memory 213 similarly capable of storing the algorithm and allowing themicroprocessor 210 access thereto. The algorithm may also be provided on a removable computer-readable medium 214 in the form of a computer program product. Specifically, the algorithm may be stored on theremovable medium 214 as control logic or a set of program codes which configure thecomputational device 202 to perform according to the algorithm. Theremovable medium 214 may be provided as, for example, a compact disc (CD), a floppy, a removable hard drive, a universal serial bus (USB) drive, a flash drive, or any other form of computer-readable removable storage. - Still referring to
FIG. 2 , theimage reporting device 200 may be configured such that thecomputational device 202 is in communication with at least oneimage source 216. Theimage source 216 may include, for example, animage capture device 104 and/or a database of retrievable images, as shown inFIG. 1 . In a localized configuration, thecomputational device 202 may be in direct wired or wireless communication with theimage source 216. In still other alternatives, theimage source 216 may be established within thememory 212 of thecomputational device 202. In a network configuration, thecomputational device 202 may be provided with an optional network orcommunications device 218 so as to enable a connection to theimage source 216 via anetwork 102. - As shown in
FIG. 3 , a flow diagram of anexemplary algorithm 300 by which animage reporting device 200 may conduct an image reporting session is provided. In aninitial step 302, one or more images of a sample structure to be interpreted may be captured and/or recorded. The images may include, for instance, one or more two-dimensional medical images, one or more three-dimensional medical images, or any combination thereof. The sample structure to be interpreted may be, for instance, a patient, a part of the anatomy of a patient, or the like. More specifically, in an image reporting session for medical applications, the images that are captured and/or recorded instep 302 may pertain to a mammography screening, a computer tomography (CT) scan, an ultrasound, an X-ray, a fluoroscopy, or the like. - In an
optional step 304, the captured or recorded images may be copied and retrievably stored at animage server 106, animage database 108, alocal host 110, or any othersuitable image source 216. Each of the copied and stored images may be associated with information linking the images to a sample subject or structure to be interpreted. For instance, medical images of a particular patient may be associated with the patient's identity, medical history, diagnostic information, or any other such relevant information. Such classification of images may allow a user to more easily select and retrieve certain images according to a desired area of interest, as inrelated step 306. For example, a physician requiring a mammographic image of a patient for the purposes of diagnosing breast cancer may retrieve the images by querying the patient's information via one of theinput devices 206. In a related example, a physician conducting a case study of particular areas of the breast may retrieve a plurality of mammographic images belonging to a plurality of patients by querying theimage server 106 and/ordatabase 108 for those particular areas. - Upon selecting a particular study in
step 306, one or more retrieved images may be displayed at theviewing device 208 of theimage reporting device 200 for viewing by the user as instep 308. In alternative embodiments, for example, wherein theimage source 216 orcapture device 104 is local to thehost 110,steps image database 108. -
Exemplary images 310 that may be presented at theviewing device 208 are provided inFIGS. 4A-4B . The views contained in each ofFIGS. 4A-4B may be simultaneously presented at a single display of aviewing device 208 to the reader so as to facilitate the reader's examination and comprehension of the underlying anatomical object. Alternatively, one or more components or views within each ofFIGS. 4A-4B may also be provided as individual views that are simultaneously and/or sequentially presentable at multiple displays of theviewing device 208. Theimages 310 may include one or more two-dimensional (2D) or three-dimensional (3D) views of an image representation of animage 312 to be interpreted. In the particular views ofFIGS. 4A-4B , two-dimensional medical image representations ormammographic images 310 of abreast 312 are provided. Moreover, the displays ofFIGS. 4A-4B may include the right mediolateral oblique (RMLO) view of thesample breast 312, as well as the right craniocaudal (RCC) view of thecorresponding sample breast 312. Alternatively, one or more three-dimensional views of asample breast structure 312 may be displayed at theviewing device 208 of the image interpretation andreporting device 200. - Additionally, the
images 310 may also provide views of an image representation of areference structure 314 for comparison. Thereference structure 314 may be any one of a prior view of thesample structure 312, a view of a generic structure related to thesample structure 312, a benchmark view of thesample structure 312, or the like. - The generic structure may be related to the sample structure by imaging modality. The generic structure may further be related to the generic structure by one or more attributes including size, dimensions, area, shape, volume, weight, density, location, anatomical organ, and orientation.
- The selected generic structure may have coordinate data defined therein. As will be explain in further detail below, the coordinate data may be used in describing the location of the region of interest in the anatomy. The system automatically selects a generic structure from among a plurality of generic structures based on the imaging modality. The system may further select the generic structure based on a comparison of content of the sample structure to the content of the generic structure.
- The
images 310 may even be provided using different imaging modalities such as computer tomography (CT) scan, an ultrasound, an X-ray, a fluoroscopy, or the like. These different imaging modalities may be linked using image registration techniques commonly known in the art. For the sake of clarity, the term registration as used herein refers to known techniques for correlating a point or a region of interest in a first image with the corresponding location or region in a second image. It should be appreciated that the term registration applies whether images are both from the same imaging modality or if the images were captured using different imaging modalities. Furthermore, thereference structure 314 may be automatically selected and supplied by theimage reporting device 200 in response to thesample structure 312 that is retrieved. Theimage reporting device 200 may prompt the user to confirm that theappropriate reference structure 314 was selected. Moreover, based on certain features of thesample structure 312 in question, theimage reporting device 200 may automatically retrieve acomparable reference structure 314 from a collection ofreference structures 314 stored at animage source 216,image database 108, or the like. Alternatively, a user may manually select and retrieve acomparable reference structure 314 for viewing. - Although some retrieved image representations of
sample structures 312 may already be in three-dimensional form, many retrieved image representations of asample structure 312 may only be retrievable in two-dimensional form. Accordingly, thestep 308 of displaying an image representation of asample structure 312 may further perform a mapping sequence so as to reconstruct and display a three-dimensional image representation of thesample structure 312 using any one of a variety of known mapping techniques. As shown inFIG. 5 , for example, a computer tomography (CT) image representation of asample structure 316 of a human head may be retrieved as a collection of two-dimensional images 318, wherein eachimage 318 may display one lateral cross-sectional view of thesample head structure 316. In such a case, the individualcross-sectional images 318 may be combined to reconstruct the three-dimensional head structure 316 shown. Such mapping techniques may be extended to reconstruct a three-dimensional representation of a complete human anatomy as onesample structure 312. Other known techniques for mapping, as demonstrated inFIG. 6 for example, may exist, wherein adeformable mesh 320 laid over a known data distribution may define the geometric transformation to a three-dimensional structure 322 of unknown data distribution after several iterations of local registrations. Additional mapping techniques may be used in which the deformation of a three-dimensional structure may be represented by a three-dimensional grid, for example, composed of tetraeders, or with three-dimensional radial basis functions. Depending on the resolution applied, the interior content of a three-dimensional image may be well-defined and segmented so as to be automatically discernable by software, for instance. For medical image interpretation practices, such voxel data and the resulting three-dimensional contents may be used to represent and distinguish between any underlying tissues, organs, bones, or the like, of a three-dimensional part of the human anatomy. Still further refinements for mapping may be applied according to, for instance, Hans Lamecker, Thomas Hermann Wenckebach, Hans-Christian Hege. Atlas-based 3D-shape reconstruction from x-ray images. Proc. Int. Conf. of Pattern Recognition (ICPR2006), volume I, p. 371-374, 2006, wherein commonly observed two-dimensional images may be processed and morphed according to a known three-dimensional model thereof so as to reconstruct a refined three-dimensional representation of the image initially observed. - In a similar manner, the
algorithm 300 may map ageneric structure 324, as shown inFIGS. 7A-7C , to thesample structure 312 ofFIGS. 4A-4B . Ageneric structure 324 may include any known or well-defined structure that is related to thesample structure 312 and/or comparable to thesample structure 312 in terms of size, dimensions, area, volume, weight, density, orientation, or other relevant attributes. Thus, the generic structure may be a prior image of the same structure thereby enabling longitudinal comparison of the region of interest. Thegeneric structure 324 may also be associated with known coordinate data. Coordinate data may include pixel data, bitmap data, three-dimensional data, voxel data, or any other data type or combinations of data suitable for mapping a known structure onto asample structure 312. For example, the embodiments ofFIGS. 7A-7C illustrate an image representation of ageneric breast structure 324 that is comparable in size and orientation to the correspondingsample breast structure 312, and further, includes coordinate data associated therewith. Moreover, in themammographic images 310 ofFIGS. 7A-7C , the coordinate data may be defined according to a coordinate system that is commonly shared by anysample breast structure 312 and sufficient for reconstructing a three-dimensional image, model or structure thereof. By mapping or overlaying the coordinate data of thegeneric structure 324 onto thesample structure 312, theimage reporting algorithm 300 may be enabled to spatially define commonly shared regions within thesample structure 312, and thus, facilitate any further interpretations and/or annotations thereof. By mapping, for instance, thegeneric breast structure 324 ofFIGS. 7A-7C to thesample structure 312 ofFIGS. 4A-4B , thealgorithm 300 may be able to distinguish, for example, the superior, inferior, posterior, middle, and anterior sections of thesample breast structure 312 as well as the respective clock positions. - As will be described below in further detail, different taxonomies are associated with each generic structure. Thus, the selection of a given generic structure restricts the universe of applicable taxonomies. Moreover, different taxonomies are associated with each imaging modality. The taxonomy used to describe a computer tomography (CT) scan of a sample structure is different from the taxonomy used to describe an ultrasound image of the same sample structure. Likewise, X-ray, a fluoroscopy, or the like each use their own unique taxonomy. The image reporting system of the present invention selects the appropriate taxonomy based on the imaging modality and the generic structure thereby facilitating ease of use and ensuring consistent usage of terminology in the reports.
- As with
reference structures 314, selection of a compatiblegeneric structure 324 may be automated by theimage reporting device 200 and/or thealgorithm 300 implemented therein. Specifically, animage database 108 may comprise a knowledgebase of previously mapped and storedsample structures 312 of various categories from which a best-fit structure may be designated as thegeneric structure 324 for a particular study. The knowledge representation may be stored within the knowledgebase. - As used in the present disclosure, the term “knowledge representation” includes identifying characteristics of biological structures and knowledge about visual representation of normal and abnormal tissue. The term “tissue” includes both bone and soft tissue, i.e., any biological structure. The term “knowledge representation” also includes genetic data, demographic data, effectiveness of treatments, behavioral data, nutritional data, i.e., any health-related data.
- Knowledge representations include identifying characteristics from annotated region of interests and the tracking of changes to the medical records related to the region of interest and the treatment outcomes. A preferred embodiment of the knowledge representation includes computer vision and machine learning frameworks such as the open-source software library TensorFlow, more specifically artificial convolutional neural networks to advance the knowledge representation with knowledge of identifying characteristics within image data. A convolutional neural network is trained with an initial data set as depicted in
FIG. 21 . The input data includes annotated image files typically in DICOM format, pathology results, health records, genetic data, behavioral data etc. The image data consists of positive space (pixel data within the region of interest) and associated finding and negative space (pixel data outside of the region of interest) and associated general findings. Non-image date such as pathology results, health records, genetic data, behavioral data etc. contain information on the accuracy of the diagnostic finding and effectiveness of the treatment recommendation. The training data may be fed into the learning process one at a time or as a batch. As such training is computationally demanding, a distributed approach depicted inFIG. 22 may be utilized. - Distributed Training of the Network as Part of the Knowledge Representation (
FIG. 22 ) - The knowledge of identifying characteristics within image data is used to automatically select regions of interest and automatically select a diagnostic finding for such region as part of the diagnostic process.
- The accuracy of the knowledge representation is continuously improved by means of online machine learning methods in which data becomes available in a sequential order and is used to update our best predictor for future data at each step, as opposed to batch learning techniques which generate the best predictor by learning on the entire training data set at once. This method of progressive incremental learning is independent of the number of class constraints and it can learn new classes while still retaining the knowledge of previous classes. Whenever a new class (non-native to the knowledge learned thus far) is encountered, the classifier gets remodeled automatically and the parameters are calculated in such a way that it retains the knowledge learned thus far.
- As the quality of images continuously improves and new imaging modalities emerge, the preferred embodiment ages older data by automatically assigning a lower weight to older training images whereas newer data is automatically assigned a higher weight.
- The preferred embodiment includes a sequence of machine learning models ML_t, where ML_{t+1} is trained later in time than ML_t, each model ML_t trained based on a set of training data D_t consisting of training samples s_{t,i} with respective training weights w_{t,i}. Each sample s_{t,i} that is similar to a sample s having a reduced weight w_{t,i}<w_{t−1, k}, and samples with updated outcome having an increased weight w_{t,i}>w_{t−1, k}.
- Diagnostic findings which are verified by non-image data such as pathology results is automatically assigned the highest weight. In one alternative, an approximated
generic structure 324 may be constructed based on an average of attributes of all previously mapped and storedsample structures 312 relating to the study in question. Accordingly, the ability of thealgorithm 300 to approximate a givensample structure 312 may improve with every successive iteration. Alternatively, a user may manually filter through animage source 216 and/or animage database 108 to retrieve a comparablegeneric structure 324 for viewing. - Referring back to the
algorithm 300 ofFIG. 3 , once the image representations of thesample structure 312 are mapped and displayed instep 308, thealgorithm 300 may enable selection of one or more points or regions of interest (ROIs) within the image representation of thesample structure 312 instep 328. As illustrated inFIGS. 4A-4B , a visual phenomena, abnormality, or region ofinterest 326 may be determined based on the contents of the image representation of thesample structure 312. For example, in themammographic images 310 ofFIGS. 4A-4B , a region ofinterest 326 may correspond to a plurality of calcifications disposed within thesample breast structure 312. Such a region ofinterest 326 may be determined manually by a user viewing thesample structure 312 from animage reporting device 200. One or more regions ofinterest 326 may also be automatically located by theimage reporting algorithm 300. For example, thealgorithm 300 may automatically and/or mathematically compare contents of the image representation of thesample structure 312 with the contents of image representation of thereference structure 314, as shown inFIGS. 4A-4B . In some embodiments, thealgorithm 300 may similarly enable recognition of contents within an image representation of ageneric structure 324. - During such comparisons, it may be beneficial to provide comparison views between a
sample structure 312 and areference structure 314, as demonstrated inFIGS. 4A-4B . However, not all image representations of thereference structure 314 may be retrieved in an orientation that is comparable to that of thesample structure 312, as shown inFIGS. 8A-8D . Accordingly, thealgorithm 300 may be configured to automatically warp the position, orientation and/or scale of the image representation of thereference structure 314 to substantially match that of thesample structure 312. In alternative embodiments, thealgorithm 300 may be configured to automatically warp the image representation of thesample structure 312 to that of thereference structure 314. - In an exemplary warping process, the
algorithm 300 may initially determine two ormore landmarks reference structures mammographic images 310 ofFIGS. 8A-8D , thefirst landmark 330 may be defined as the nipple of therespective breast structures second landmark 332 may be defined as the pectoralis major muscle line. Forming anorthogonal baseline 334 from thefirst landmark 330 to thesecond landmark 332 of eachstructure algorithm 300 may determine the spatial offset that needs to be adjusted. Based on the coordinate mapping performed earlier instep 308 and the detected differences between the respective landmark positions, thealgorithm 300 may automatically adjust, rotate, shift, scale or warp one or both of thesample structure 312 and thereference structure 314 to minimize the offset. For instance, in the example ofFIGS. 8A-8D , thealgorithm 300 may rotate the image representation of theprior reference structure 314 in the direction indicated byarrow 336 until the orientations of therespective landmark baselines 334 are substantially parallel. In an alternative embodiment, thegeneric structure 314 may be substituted for thereference structure 314, in which case similar warping processes may be employed to minimize any skewing of views. - Still referring to step 328 of
FIG. 3 , once at least one region ofinterest 326 has been determined, thealgorithm 300 may further link the region ofinterest 326 with the coordinate data that was mapped to thesample structure 312 duringstep 308. Such mapping may enable thealgorithm 300 to define the spatial location of the region ofinterest 326 with respect to thegeneric structure 314 and not only with respect to the view or image representation of thesample structure 312 shown. Moreover, thealgorithm 300 may be able to at least partially track the location of the region ofinterest 326 within thesample structure 312 regardless of the view, position, orientation or scale of thesample structure 312. In particular, if thealgorithm 300 is configured to provide multiple views of asample structure 312, as in the mammographic views ofFIGS. 4A-4B for example, step 340 of thealgorithm 300 may further provide a range or band ofinterest 338 in one or more related views corresponding to the region ofinterest 326 initially established. Based on manual input from a user or automated recognition techniques, step 342 of thealgorithm 300 may then determine the corresponding region ofinterest 326 from within the band ofinterest 338. - As in the warping techniques previously discussed, in order to perform the tracking steps 340 and 342 of
FIG. 3 , thealgorithm 300 may identify at least twolandmarks sample structure 312 in question. In the mammographic views ofFIGS. 9A-9B shown, for example, thefirst landmark 330 may be defined as the nipple, and thesecond landmark 332 may be defined as the pectoralis major muscle line. Thealgorithm 300 may then define abaseline 334 a as, for example, an orthogonal line extending from thenipple 330 to the pectoralismajor muscle line 332. As demonstrated inFIG. 9A , a user may select the region ofinterest 326 a on the right mediolateral oblique (RMLO) view of thesample breast structure 312. After the selection, thealgorithm 300 may project the region ofinterest 326 a onto thebaseline 334 a, from which thealgorithm 300 may then determine afirst distance 344 a and asecond distance 346. Thefirst distance 344 a may be determined by the depth from thefirst landmark 330 to the point of projection of the region ofinterest 326 a on thebaseline 334 a. Thesecond distance 346 may be defined as the projected distance from the region ofinterest 326 a to thebaseline 334 a, or a distance above or below the baseline in the mammogram example. Based on thefirst distance 344 a, thealgorithm 300 may determine a set of corresponding baseline 34 b andfirst distance 344 b in the right craniocaudal (RCC) view ofFIG. 9B . Using thebaseline 334 b andfirst distance 344 b determined in the second view ofFIG. 9b , thealgorithm 300 may further determine the corresponding band ofinterest 338 and display the band ofinterest 338 as shown. From within the band ofinterest 338 provided, thealgorithm 300 may then enable a second selection or determination of the corresponding region ofinterest 326 b in the second view. Using the region ofinterest 326 b determined in the second view, thealgorithm 300 may define athird distance 348 as the distance from the region ofinterest 326 b to thebaseline 334 b, or the lateral distance from thenipple 330. Based on the first, second and third distances 344 a-b, 346, 348, thealgorithm 300 may be configured to determine the quadrant or the spatial coordinates of the region ofinterest 326 a-b. Notably, while therespective baselines 334 a-b, and/or the first distances 344 a-b, of the first and second views ofFIGS. 9A and 9B may be comparable in size and configuration, such parameters may be substantially different in other examples. In such cases, warping, or any other suitable process, may be used to reconfigure the respective volumes shown, as well as the respective parameters defined between commonly shared landmarks, to be in a more comparable form between the different views provided. - In a related modification, the
algorithm 300 may be configured to superimpose a tracked region ofinterest 326 to a corresponding location on areference structure 314 which may be a reference structure, prior sample structure,generic structure 324, or the like. As in previous embodiments, thealgorithm 300 may initially determine control points (landmarks) that may be commonly shared by both thesample structure 312 and thereference structure 314. With respect tomammographic images 310, the control points may be defined as the nipple, the center of mass of the breast, the endpoints of the breast contour, or the like. Using such control points and a warping scheme, such as a thin-plate spline (TPS) modeling scheme, or the like, thealgorithm 300 may be able to warp or fit the representations of thereference structure 314 to those of thesample structure 312. Once a region ofinterest 326 is determined and mapped within thesample structure 312, the spatial coordinates of the region ofinterest 326 may be similarly overlaid or mapped to thewarped reference structure 314. Alternatively, a region ofinterest 326 that is determined within thereference structure 314 may be similarly mapped onto asample structure 312 that has been warped to fit thereference structure 314. - The aforementioned concept of superimposing a tracked region of
interest 326 to a corresponding location on aprior reference structure 314 may be extended to include multiple regions of interest. This enables one to readily determine the longitudinal progression in terms of growth or size and/or number of region(s) of interest over time. Initially there may be only one region of interest which may later grow or shrink. Mapping the initial image on the subsequent image enables accurate tracking of the growth or shrinkage of the region of interest. Additional regions of interest may develop over time and thealgorithm 300 enables the user to accurately compare the region of interest longitudinally, i.e., over time. Importantly, the registration process may be automated to facilitate tracking the region of interest over time. However, even with a fully automated registration process it is desirable to prompt the user to manually confirm the registration or mapping of the region of interest and/or identified lesions within the region of interest. Alternatively, the fully automated system may allow the user to select the region of interest. -
FIG. 16 shows asample structure 312 with the region of interest shown in hashed lines. The user is able to access historical information regarding a region of interest using a pointing device such as a mouse, touch sensitive screen or the like.FIG. 17 is a table which was accessed by the user showing changes in the region of interest. In the example illustrated inFIG. 17 the comparison is made between the current measurements and the previous measurements for the region of interest. - The manual input from the user may consist of simply selecting a region of interest using a pointing device or a touch sensitive screen. In response to the manual input the
algorithm 300 may display the outline or contours of a region of interest. In the event that thealgorithm 300 cannot detect the outline of the region of interest it may prompt the user to manually trace the outline using a pointing device or the like or thealgorithm 300 may simply place a circle or the like around the region of interest. Thealgorithm 300 uses the outline of the region of interest to automatically compute the size and/or volume of the region. - Alternatively, the
algorithm 300 may use automated recognition techniques to identify and display one or more items of interest. The user is then prompted to accept or reject each item of interest. Alternatively, instead of prompting to accept or reject each item of interest, the user may be allowed to select a new or different region of interest. - Regardless of how the item of interest (region of interest) is identified (manual or automated) the user is then prompted to describe the item of interest using the knowledge representation corresponding to the imaging modality and/or the generic structure. To aid the user the knowledge representation presented to the user is both organ and modality specific. Alternatively, the knowledge representation presented to the user may be specific to organ or the imaging modality. Thus, terms which do not pertain to the imaging modality of the sample are not presented, nor are terms which do not pertain to the organs encompassing the region of interest. More specifically, the system may automatically select at least one diagnostic finding or prompt the user to select at least one diagnostic finding from the focused set knowledge representations. The system may retrievably store the at least one diagnostic finding in an electronic record such as an electronic medical record. The system may monitor (track) the electronic record for changes to the at least one diagnostic finding and/or the addition of new diagnostic findings, and may use such changes or new diagnostic findings to update the knowledge representation.
- The selected generic structure may be related to the sample structure by imaging modality and one or more attributes such as size, dimensions, area, shape, volume, weight, density, location, anatomical organ, and orientation. The selected generic structure may have coordinate data defined therein.
- Certain phenomena only occur in certain parts of certain objects such as anatomical organs. The knowledge of where certain phenomena are most likely to occur allows the system to provide a focused set of knowledge representations as a user interface (graphic or audio or both) to an image analyst. This focused knowledge representation allows the analyst to report the findings more efficiently. Focusing the knowledge representation may also be guided by other analytics from other data such as patient history, demographic, geolocation, genetic data etc. In addition to the structured knowledge representation in form of ontologies, the image analyst might add additional findings in form of free text entry. The image reporting system utilizes natural language analytics in form of statistical semantic analysis of text which is entered as free text and advise on patterns found in the free text. These patterns are basis for the evolution of the ontology. Further extensions of such mapping, marking and tracking may provide more intuitive three-dimensional representations of a
sample structure 312, as shown for example inFIGS. 10A-10B . As a result of several iterations of mapping sets and subsets of known coordinate data to asample structure 312, thealgorithm 300 may be able to distinguish the different subcomponents of thesample structure 312 as separable segments, or subsets of data that are grouped according to like characteristics. For instance, in thesample structure 312 ofFIGS. 10A-10B , each mammary gland may be defined as onesegment 350.Such algorithms 300 may enable a user to navigate through three-dimensional layers of thesample structure 312 and select any point therein as a region ofinterest 326. In response, thealgorithm 300 may determine the subcomponent orsegment 350 located nearest to the region ofinterest 326 indicated by the user and highlight thatsegment 350 as a whole for further tracking, as shown for example inFIG. 10B . - Once at least one region of
interest 326 has been determined and mapped, thealgorithm 300 may further enable anannotation 352 of the region ofinterest 326 in an annotatingstep 354. For example, a physician viewing the two regions ofinterest 326 inFIGS. 9A-9B may want to annotate or identify the respective contents of regions ofinterest 326 as a cluster of microcalcifications and a spiculated nodule.Such annotations 352 may be received at theinput device 206 in verbal form by way of a microphone, in typographical form by way of a keyboard, or the like. More specifically, theannotations 352 may be provided in the respective views of thesample structure 312 as plain text, graphics, playback links to audio and/or video clips, or the like. Once entered, eachannotation 352 may be spatially associated and tracked with its respective region ofinterest 326 so as to be accessible and viewable in any related views depicting those regions ofinterest 326. Data associating eachannotation 352 with its respective region ofinterest 326 may further be retrievably stored with theimages 310 via animage server 106 and animage database 108 that is associated with, for example, a Picture Archiving and Communication System (PACS) in accordance with Digital Imaging and Communications in Medicine (DICOM). In an alternative embodiment, thealgorithm 300 may be configured to receive anannotation 352 at the first instance of identifying a region ofinterest 326 and before any tracking of the region ofinterest 326 is performed to related views. Once theannotation 352 has been associated with the first determination of a region ofinterest 326, any corresponding regions ofinterest 326 tracked in subsequent views may automatically be linked with the sameinitial annotation 352. Thealgorithm 300 may also allow a user to edit previously established associations or relationships betweenannotations 352 and their respective regions ofinterest 326. - Turning back to the
algorithm 300 ofFIG. 3 , step 356 of thealgorithm 300 may configure animage reporting device 200 to allow generation of a report based on the mapped regions ofinterest 326 and accompanyingannotations 352. As previously noted, the coordinate data of thegeneric structure 324 may conform to any common standard (taxonomy) for identifying spatial regions therein. For example, common standards for identifying regions of the breast may be illustrated by the coordinate maps of a generic breast structure inFIGS. 7A-7C . Once asample structure 312 is mapped with such coordinate data, thealgorithm 300 may be able to automatically identify the spatial location of any region ofinterest 326 orannotation 352 indicated within thesample structure 312. Thealgorithm 300 may then further expand upon such capabilities by automatically translating the spatial coordinates and/or corresponding volumetric data of the regions ofinterests 326 and theannotations 352 into character strings or phrases commonly used in a report. - With reference to
FIG. 11A , anexemplary report 358 may be automatically provided in response to the regions ofinterest 326 andannotations 352 ofFIGS. 9A-9B . As previously discussed, the mammographic representations ofFIGS. 9A-9B depict two regions ofinterest 326 including a cluster of microcalcifications and a spiculated nodule. According to the coordinate system ofFIGS. 7A-7C , the location of the cluster of microcalcifications may correspond to the superior aspect of the RLMO view at 11 o'clock, while the location of the spiculated nodule may correspond to the medial aspect of the LCC view at 10 o'clock. Thealgorithm 300 may use this information to automatically generate one or more natural language statements or other forms of descriptions indicating the findings to be included into therelevant fields 360 of thereport 358, as shown inFIG. 11A . More specifically, the descriptions may include a location statement describing the spatial coordinates of the region ofinterest 326, a location statement describing the underlying object within thesample structure 312 that corresponds to the spatial coordinates of the region ofinterest 326, a descriptive statement describing the abnormality discovered within the region ofinterest 326, or any modification or combination thereof. Thealgorithm 300 may also provide standard report templates havingadditional fields 362 that may be automatically filled by the algorithm 300 (which may be manually over-ridden by the user) or manually filled by a user. For example, thefields 362 may be filled with data associated and stored for or with the patient and/or images, such as the exam type, clinical information, and the like, as well as any additional analytical findings, impressions, recommendations, and the like, input by the user while analyzing theimages 310. - In further alternatives, the underlying object and/or abnormality may be automatically identified based on a preprogrammed or predetermined association between the spatial coordinates of the region of
interest 326 and known characteristics of thesample structure 312 in question. The known characteristics may define the spatial regions and subregions of thesample structure 312, common terms (taxonomy) for identifying or classifying the regions and subregions of thesample structure 312, common abnormalities normally associated with the regions and subregions of thesample structure 312, and the like. Such characteristic information may be retrievably stored in, for example, animage database 108 or an associatednetwork 102. Furthermore, subsequent or newfound characteristics may be stored within thedatabase 108 so as to extend the knowledge of thedatabase 108 and improve the accuracy of thealgorithm 300 in identifying the regions, subregions, abnormalities, and the like. Based on such a knowledgebase of information, thealgorithm 300 may be extended to automatically generate natural language statements or any other form of descriptions which preliminarily speculate on the type of abnormality that is believed to be in the vicinity of a marked region ofinterest 326. Thealgorithm 300 may further be extended to generate descriptions which respond to a user's identification of an abnormality so as to confirm or deny the identification based on the predetermined characteristics. For example, thealgorithm 300 may indicate a possible error to the user if, according to itsdatabase 108, the abnormality identified by the user is not plausible in the marked region ofinterest 326. Thealgorithm 300 may use risk factors contained in the medical record of the patient as part of its decision criteria in indicating possible error or omission or to highlight potential concerns correlated with the risk factors. The user may choose to over-ride the error flag and may optionally provide a reason for over-riding the flag. Alternatively, the user may amend the identification of the abnormality. Thus, if the abnormality identified by the user is not commonly associated with a particular organ or with the patient's risk factors then the potential error will be flagged which may lead the user to revise the patient's risk factors. Moreover, the patient's risk factors indicate a high correlation or predisposition for a particular abnormality which was not identified by the user then the potential error will be flagged which may lead the user to more closely examine the region of interest for any over-looked abnormalities. One of the aspects of the present invention which should not be overlooked or minimized is the image reporting device and method of the present invention provides an image-based medical record which allows for tracking of diagnosis, decision on treatment and outcomes on a region of interest by region of interest (i.e. lesion by lesion) basis. The system may retrievably store the at least one diagnostic finding (diagnosis) in an electronic record. The system may monitor (track) the electronic record for changes to the at least one diagnostic finding and/or the addition of new diagnostic findings, and may use such changes or new diagnostic findings to update the knowledge representation. The system may monitor (track) the electronic record for changes to the patient outcome, and may use such changes to update the knowledge representation. - There are a variety of ways to access the stored information including selecting an (already identified) region of interest by, for example, touching the displayed region with a finger (touch sensitive screen) or using a pointing device. The image reporting device assigns each region of interest a unique label or identifier, and such identifier may also be used to access information pertaining to the diagnosis, treatment, and outcome of treatment. Once the user has selected a given region of interest, he/she is able to select prior annotations, display prior diagnosis, prior decisions on treatment and outcomes of such decisions—all on a region of interest by region of interest basis.
- Access to prior annotations or the like may be made by, for example, a right mouse click or the like on the region of interest. Moreover, it should be noted that the image reporting system is intended to be used by both radiologists and oncologists. The radiologist uses the
image reporting device 200 to enter diagnostic information and the oncologist uses the image reporting device to enter treatment information as well as treatment outcomes. In this manner the image reporting device facilitates collaboration and efficient sharing of information. In other alternatives, thealgorithm 300 may automatically generate a web-basedreport 358, as shown inFIGS. 11B-11C for example, that may be transmitted to animage server 106 and/or animage database 108, and viewable via a web browser at ahost 110, or the like. As in thereport 358 ofFIG. 11A , the web-basedreport 358 may be comprised of initiallyempty fields 362 which may be automatically filled by theimage reporting system 200. The web-basedreport 358 may alternatively be printed and filled manually by a user. Thereport 358 may further provide an image representation of thesample structure 312 studied as apreview image 364. Thereport 358 may additionally offer other view types, as shown for example inFIG. 11C . - In contrast to the
report 358 ofFIG. 11B , thereport 358 ofFIG. 11C may provide alarger preview image 364 of thesample structure 312 and larger collapsible fields for easier viewing by a user. Providing such a web-based format of thereport 358 may enable anyone with authorization to retrieve, view and/or edit thereport 358 from anyhost 110 with access to theimage source 216, for example, animage server 106 and animage database 108 of a Picturing Archiving and Communication System (PACS). In still further modifications,FIG. 12 schematically illustrates animage reporting system 400 that may incorporate aspects of theimage reporting device 200, as well as thealgorithm 300 associated therewith, and may be provided with additional features including integration with internal and/or external knowledge representation systems. - As shown, the
image reporting system 400 may be implemented in, for example, themicroprocessor 210 and/or memories 212-214 of theimage reporting device 200. More specifically, theimage reporting system 400 may be implemented as a set of subroutines that is performed concurrently and/or sequentially relative to, for example, one or more steps of theimage reporting algorithm 300 ofFIG. 3 . An important aspect of theimage reporting device 200 is that annotation, procedure history and outcomes are tracked on a lesion by lesion level. Selection of any lesion (by for example, right-clicking on a pointing device such as a mouse or the like) will enable the user to choose to display the previous diagnosis, treatment decisions, and longitudinal progression. - In this manner the user is able to see if the treatment regimen has been effective, where the current treatment regimen falls within the internal and/or external knowledge representation systems (ontology). In this manner the user will readily discern whether the current treatment is working and if not will see the next course of action recommended by the knowledge representation systems.
- As shown in
FIG. 12 , once animage 401 of a sample structure that has been captured by animage capture device 104 is forwarded to theappropriate network 102 having animage server 106 and animage database 108, theimage 401 may further be forwarded to themicroprocessor 210 of theimage reporting device 200. In accordance with theimage reporting algorithm 300 ofFIG. 3 , a segmenting subroutine orsegmenter 402 of themicroprocessor 210 may process theimage 401 received into subsets of data orsegments 403 that are readily discernable by thealgorithm 300. Based on thesegmented image 403 of the sample structure and comparisons with adatabase 404 ofgeneric structures 405, a mapping subroutine ormapper 406 may reconstruct a two- or three-dimensional image representation of the sample structure for display at theviewing device 208. - In addition to the image representation, the
mapper 406 may also provide asemantic network 407 that may be used to aid in the general articulation of the sample structure, or the findings, diagnoses, natural language statements, annotations, or any other form of description associated therewith. For example, in association with an X-ray of a patient's breast or a mammogram, thesemantic network 407 may suggest commonly accepted nomenclature for the different regions of the breast, common findings or disorders in breasts, and the like. Themapper 406 may also be configured to access more detailed information on the case at hand such that thesemantic network 407 reflects knowledge representations that are more specific to the particular patient and the patient's medical history. For example, based on the patient's age, weight, lifestyle, medical history, and any other relevant attribute, thesemantic network 407 may be able to advise on the likelihood whether a lesion is benign or requires a recall. Moreover, thesemantic network 407 may display or suggest commonly used medical terminologies or knowledge representations that may relate to the particular patient and/or sample structure such that the user may characterize contents of the image representations in a more streamlined fashion. - Still referring to
FIG. 12 , themapper 406 may refer to a knowledge representation broker orbroker subroutine 408 which may suggest an appropriate set of terminologies (e.g. taxonomy), or knowledge representations (e.g. ontology), based on a structural triangulation or correlation of all of the data available. Thebroker subroutine 408 may access knowledge representations from external and/or internal knowledge representation databases and provide the right combination of knowledge representations with the right level of abstraction to the reader. More specifically, based on a specific selection, such as an anatomical object, made by the reader, thebroker 408 may be configured to determine the combination of knowledge representation databases that is best suited as a reference for themapper 406 and point themapper 406 to only those databases. For a selection within a mammography scan, for instance, thebroker subroutine 408 may selectively communicate with or refer themapper 406 to one or more externally maintained sources, such as a Systematized Nomenclature of Medicine-Clinical Terms (SNOMED-CT)database 410, a Breast Imaging-Reporting and Data System (BI-RADS)database 412, aRadLex database 414 of common radiological terms, or any other external database of medical terminologies that may be used for characterizing findings within a sample structure and generating a natural language statement or any other form of description corresponding thereto. Themapper 406 may then refer to those knowledge representation databases in characterizing the selection for the reader using refined knowledge representations. - The
broker 408 may also be configured to enable the reader to select one or more of the resulting knowledge representations to explore further refinements. Thebroker 408 may additionally be configured to determine an appropriate level of abstraction of the reader's selection based at least partially on certain contexts that may be relevant to the reader. The contexts may include data pertaining to the patient, the institution to which the reader belongs, the level of expertise of the reader, the anatomical objects in the immediate focus or view of the reader, and the like. The contexts may further include attributes pertaining to different interpretation styles and formats, such as iterative interactive reporting, collective reporting, and the like. Based on such contexts as well as the anatomical object selected by the reader, theimage reporting system 400 may be able to provide more refined knowledge representations of the selected object that additionally suit the level of understanding or abstraction of the particular reader. Thebroker subroutine 408 may similarly access knowledge representations from an internally maintained dynamic knowledge representation database 416. The dynamic knowledge representation database 416 may further provide thebroker 408 with the intelligence to provide the right combination of knowledge representations with the right level of abstraction. - Information generated by the
mapper 406 may be provided in graphical form and, at least in part, as atransparent layer 418 such that the mapped information may be viewed at theviewing device 208 without obstructing theoriginal image 401 upon which it may be overlaid. A user viewing the information displayed at theviewing device 208 may provide any additional information, such as regions of interest, annotations, statements of findings or diagnoses within the sample structure, and the like. Information input by the user, as well as any other data relevant to the patient, such as the patient's identification, demographic information, medical history, and the like, may be forwarded to a reporting subroutine orreport engine 420 for report generation. - The
report engine 420 may generate a report, for example, in accordance with thealgorithm 300 disclosed inFIG. 3 . Furthermore, thereport engine 420 may forward the generated report to amedical record database 422 for storage and subsequent use by other care providers attending to the patient. As an additional or optional feature, thereport engine 420 ofFIG. 12 may be configured to forward a copy of the generated report to a tracking subroutine orcase tracker 424. - Among other things, the
case tracker 424 may serve as a quality tracking mechanism which monitors the amendments or findings in subsequent reports for any significant inconsistencies, such as mischaracterizations, oversights, new findings or diagnoses, disease progression or the like, and responds accordingly by adjusting one or more probability models associated with the particular knowledge representation in question. Thecase tracker 424 may monitor or track changes the electronic record such as changes to the at least one diagnostic finding and/or the addition of new diagnostic findings and/or treatment outcomes. Thecase tracker 424 may adjust the knowledge representation to reflect the changes to the diagnostic findings and/or the new diagnostic findings. - Probability models may be managed by the dynamic knowledge representation database 416 of the
image reporting system 400 and configured to suggest knowledge representations that most suitably represents the anatomical object selected by the reader. Probability models may statistically derive the most appropriate knowledge representation based on prior correlations of data between selected elements or anatomical objects and their corresponding characterizations by physicians, doctors, and the like. Furthermore, the correlations of data and any analytics provided by the probability models may be dynamically updated, validated and invalidated according to any revisions as deemed necessary by thecase tracker 424. For example, upon receipt of an alteration of the medical record, which reflects the performance of a treatment, the probability model of the knowledge representation may be validated or altered based on the content of the amendments of the medical record. - Based on the tracked results, the
case tracker 424 may update the probability model within the dynamic knowledge representation database 416. For instance, a previous data entry of the dynamic knowledge representation database 416 which characterizes a structure with an incorrect statement or finding may be invalidated and replaced with a new data entry which correctly associates the structure with the new amendments or finding. Alternatively, the amendments or finding may be added to the existing statements as an additional finding for a particular combination of information. In such a manner, thecase tracker 424 may continuously update and appropriately correct or enrich the representations stored in the dynamic knowledge representation database 416. - The
case tracker 424 may use analytics to review free-form (natural language) text entered by the user such as diagnostic finding statements to study patterns of such analysis which may in turn be used to update the focused knowledge representation. - With such access to one or more of a plurality of
knowledge databases image reporting system 400 may be able to determine the best suited natural language statement or description for characterizing elements or findings within a sample structure. Moreover, theimage reporting system 400 including at least, for example, acase tracker 424, a dynamic knowledge representation database 416 and aknowledge representation broker 408, may provide a feedback loop through which theimage reporting algorithm 300 may generate reports with more streamlined terminologies, automatically expand upon its knowledge representations, as well as adjust for any inconsistencies between related reports and findings. - The
medical record 422 may include a variety of patient information. The following list of patient information is intended to be representative but not exhaustive. The medical record may include some or all of the following: data corresponding to physical activities of the patient, patient genetic predisposition including DNA, medical history including prior cancer diagnosis, prior surgery, prior and current drug regimen, blood analysis information including pharmacological (drug absorption data), nutrition and the results of pathology reports. The term risk factors as used herein is intended to refer to one or more items of information from the medical record which either increase or decrease a person's predisposition to certain diseases. Such factors may include age, weight, family history, and the like. The data corresponding to physical activities may be collected using a Nike Fuel Band, Apple iWatch or like data collection devices such as known in the art. - Based on the aforementioned characterizing elements or findings within the sample structure the
algorithm 300 and/or image reporting system may provide real time decision support by displaying recommendations based on guidelines for management of such findings. For example, in the context of the human lung, the Fleischner Society and the National Comprehensive Cancer Networks (NCCN) each provide guidelines for follow-up and management based on the size of the lesion and the presence of risk factors such as smoking, family history or the like. For each at least one region of interest, the system may automatically select follow-up care and/or prompt (allow) the user to select from a focused set of follow-up care options. The follow-up care is stored in the electronic record. - As will be explained below, the system monitors the electronic record for changes to the follow-up care and may use such changes to update the knowledge representation.
- As shown in
FIG. 14 thealgorithm 300 prompts the user to select from one of the guidelines or enter a user specified instruction for follow-up care. In the illustration depicted inFIG. 14 the NCCN and Fleischner reflect two different guidelines which the user may select or enter a free-form (natural language) instruction in the box provided. The real time decision support may utilize guidelines found in a local database 426 (FIG. 12 ) or may access a third-party database 428 over the network. The follow-up care (selected or entered) is stored in the electronic record (electronic medical record). - In addition to showing the follow-up guidelines recommended by one or more third-party institutions such as Fleischner or NCCN, the
algorithm 300 may provide a hyperlink to a knowledge base or the like providing additional insight into the guidelines. See, e.g.FIG. 15 . The additional insight may, for example, include showing where the current treatment falls within an overall decision tree. - In still further modifications, one or more contents within the
transparent layer 418 of the report may be configured to interact with a user through theuser interface 204, or the like. For example, thetransparent layer 418 may include an interactive knowledge representation displaying semantic relationships between key medical terminologies contained in statements of the report. Using a pointer device, or any othersuitable input device 206, a user may select different terms within the report so as to expand upon the selected terms and explore other medical terminologies associated therewith. As the reader interacts with the knowledge representation, the broker might provide a different level of abstraction and a different combination of knowledge representations to assist in hypothesis building and provide information about probability of a malignancy to the reader. - A user viewing the report may also make new structural selections from within the image representation of the sample structure displayed. Based on the mapped locations of the user input, such selections made within the
transparent layer 418 of the report may be communicated to theknowledge representation broker 408. More particularly, based on the new text selected by the user, thebroker subroutine 408 may generate a new semantic network to be displayed within thetransparent layer 418 of the report. Based on the new structure or substructure selected by the user, thebroker subroutine 408 may determine any new set of medical terminologies, statements, findings, and the like, to include into the report. - The
broker subroutine 408 may refer to any one or more of theknowledge representation databases FIG. 12 in determining the ontologies and medical terminologies. Any required updates or changes to the report, or at least thetransparent layer 418 thereof, may be communicated from thebroker subroutine 408 to thereport engine 420 such that a new and updated report is automatically generated for immediate viewing. Turning toFIGS. 13A-13B , another exemplary display or user interface that may be provided to the reader at theviewing device 208 is provided. More specifically, the display may follow a format that is similar to the display shown inFIGS. 4A-4B but with the additional feature of providing the reader with knowledge representations, for instance, in accordance with theimage reporting system 400 ofFIG. 12 . - As in previous embodiments, a reader may choose to provide an annotation for a selected region of
interest 326 by pointing to or indexing the region ofinterest 326 via theinput device 206. In response to the anatomical object underlying or corresponding to the indexed region ofinterest 326, theimage reporting system 400 ofFIG. 12 may advise a focused set of knowledge representations most commonly associated with the anatomical object (e.g. an ontology). As shown inFIG. 13A , the knowledge representations may be presented to the reader in the form of a hierarchical menu or diagram showing semantic relationships, or the like. One or more of the knowledge representations displayed may be hierarchically configured and expandable to further reveal specific or more refined knowledge representations. For example, in the embodiment ofFIG. 13A , the higher-level knowledge representation associated with the selected region ofinterest 326 may correspond to the lesion of a breast. Expanding upon this knowledge representation may then yield a plurality of common findings within the lesion of the breast. One or more of the resulting findings may also be expanded upon to reveal more refined subcategories, such as breast lumps, calcifications, nodules, sinuses, ulcerations, and the like. From the resulting subcategories, the reader may use theinput device 206 to select the most appropriate finding that applies to the patient at hand. Once a knowledge representation is selected, the knowledge representation may be displayed as the annotation associated with the selected region ofinterest 326, as shown for example inFIG. 13B . -
FIG. 18 is a flowchart of a method for integrating image analysis, longitudinal tracking of a region of interest and updating of a knowledge representation. - In
step 1802, an image representation of a sample structure is retrieved from an electronic storage medium which such as an image database. The image database may be a PACS database (Picture Archiving and Communication System). Instep 1804, the system automatically selects a generic structure from a database based on an imaging modality of the sample structure. At least one focused set of knowledge representations is stored in a second database. In some cases, the second database is the same database in which the generic structure is stored and, in some cases, the second database a different database. The knowledge representation is associated with or related to the selected generic structure by one or attributes such as imaging modality, size, dimensions, area, shape, volume, weight, density, location, anatomical organ, and orientation. - In
step 1806, the selected generic structure is mapped by the system to the sample structure, and instep 1808 the system automatically determines at least one region of interest within the sample structure and/or allows the user to select a region of interest. - In
step 1810 the system automatically selects at least one diagnostic finding and/or allows the user to select at least one diagnostic finding from the focused set knowledge representations. In other words, the system automatically selects at least one diagnostic finding. If the user disagrees with the automatically selected diagnostic finding, the user may select at least one diagnostic finding from a focused set of diagnostic findings. It should be understood that the diagnostic findings are focused to provide findings which are relevant in terms of imaging modality, anatomical organ or the like. If the user doesn't find the desired diagnostic finding in the focused set of findings then the user may enter a diagnostic finding using free-form text. Instep 1812, the system retrievably stores the at least one diagnostic finding (the automatically selected diagnostic finding(s) or the diagnostic finding(s) selected or entered by the user) in the electronic record. - In
step 1814, the system monitors or tracks the electronic record for changes to the at least one diagnostic finding and/or the addition of new diagnostic findings. The system uses the changes to the diagnostic findings and/or new diagnostic findings to update the knowledge representation in the second database. - The method may end at
step 1814 or may optionally continue to step 1816 in which the coordinate data associated with the generic structure is used to generate natural language statements describing a location of the region of interest in the anatomy. The system automatically generates a diagnostic report based on the at least one diagnostic finding. The diagnostic report includes the natural language statements describing the location in the anatomy of the region of interest. The system stores the diagnostic report in the electronic record. - It should be understood that unless expressly stated otherwise, each of the method steps disclosed herein are performed by the system. Thus, the system automatically selects the region of interest, and the system monitors for changes to the electronic record.
- In the aforementioned method of
FIG. 18 , the selected generic structure may be related to the sample structure by imaging modality and one or more attributes such as size, dimensions, area, shape, volume, weight, density, location, anatomical organ, and orientation. Moreover, the selected generic structure may have coordinate data defined therein. - In the method of
FIG. 18 , the knowledge representation may be specific to an anatomical organ in which the region of interest is located and/or the imaging modality used to capture the sample structure. - In the method of
FIG. 18 , the step of automatically selecting a generic structure (from among a plurality of generic structures) may be based on the imaging modality and/or a comparison of content of the sample structure to the content of the generic structure. - The method of
FIG. 18 may end atstep 1814 orstep 1816 or the method may continue from either or both of these steps to step 1818 in which the system (algorithm) automatically selects follow-up care or allows the user to select from a focused set of follow-up care options for each at least one region of interest. More particularly, the user can change the automatically selected follow-up care option(s) automatically selected by the system by selecting at least one follow-up care option from a focused-sect of options or by entering a new follow-up care using free-form text. The system stores the follow-up care option(s) in the electronic record. -
Step 1814 may also include checking for changes to the selected follow-up care and using such changes to update the knowledge representation in the second database. Additionally or alternatively,Step 1814 may include checking for changes to the previously stored treatment outcome and using such changes to update the knowledge representation in the second database. -
FIG. 19 is a flowchart of another method for integrating image analysis, longitudinal tracking of a region of interest and updating of a knowledge representation. Instep 1902 the system (algorithm) retrieve an image representation of sample structure depicting at least a portion of an anatomical organ from an image database. Instep 1904 the system automatically determine at least one region of interest within the sample structure. If the user disagrees with the region(s) of interest automatically determined, the user is allowed the user to select a region of interest. Thereafter instep 1906, the system automatically selects at least one diagnostic finding. If the user disagrees, the user is allowed to select at least one diagnostic finding from a focused set of knowledge representations stored in a database or enter a diagnostic finding using free-form text. The focused set of knowledge representations is specific to one or more attributes of the sample structure such as imaging modality, size, dimensions, area, shape, volume, weight, density, location, anatomical organ, and orientation. Instep 1908, the at least one diagnostic finding is stored in the electronic record. Then instep 1910, the system monitors or tracks the electronic record for changes and/or additions to the at least one diagnostic finding. If such changes are detected, the system uses the changes to update the knowledge representation. -
Step 1910 or may optionally include checking for changes to treatment outcome and using such changes to update the knowledge representation in the database. -
FIG. 20 is a flow diagram of a method for automatically improving a knowledge representation for an image reporting system. - In
step 2002 the system records at least one diagnostic finding for a given region of interest in an electronic record. Instep 2004, the system monitors or tracks the electronic record for changes to the at least one diagnostic finding for the region of interest. Instep 2006, if such a change is detected the system automatically updates a knowledge representation stored in a database to reflect the changes to the at least one diagnostic finding. The method may terminate atstep 2006 or may optionally include steps 2008-2014. - In
step 2008 the system retrieves an image representation of sample structure depicting at least a portion of an anatomical organ from an image database. Instep 2010 the system automatically determines at least one region of interest within the sample structure. Additionally or alternatively, the user is allowed to select a region of interest. Instep 2012, the system automatically selects at least one diagnostic finding. Additionally or alternatively, the user is allowed to select at least one diagnostic finding from a focused set of knowledge representations specific to at least one of the anatomical organ and an imaging modality used to capture the image representation. Further still, the user may enter a diagnostic finding using free-form text. Instep 2014, the system retrievably stores the at least one diagnostic finding in the electronic record. -
Step 2004 may optionally include checking for changes to treatment outcome and using such changes to update the knowledge representation in the database. - Based on the foregoing, it can be seen that the disclosed method and apparatus provide an improved system and method for generating and managing image reports. The disclosed image reporting device and algorithms serve to automate several of the intermediary steps involved with the processes of generating and recalling image reports today. More specifically, the disclosed method and apparatus serves to integrate automated computer aided image mapping, recognition and reconstruction techniques with automated image reporting techniques. Furthermore, the disclosed method and apparatus aids in streamlining the language commonly used in image reporting as well as providing a means to automatically track subsequent and related cases for inconsistencies.
Claims (16)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US16/363,032 US20190220978A1 (en) | 2010-07-21 | 2019-03-25 | Method for integrating image analysis, longitudinal tracking of a region of interest and updating of a knowledge representation |
Applications Claiming Priority (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US36649210P | 2010-07-21 | 2010-07-21 | |
US14/093,470 US10282840B2 (en) | 2010-07-21 | 2013-11-30 | Image reporting method |
US16/363,032 US20190220978A1 (en) | 2010-07-21 | 2019-03-25 | Method for integrating image analysis, longitudinal tracking of a region of interest and updating of a knowledge representation |
Related Parent Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US14/093,470 Continuation US10282840B2 (en) | 2010-07-21 | 2013-11-30 | Image reporting method |
Publications (1)
Publication Number | Publication Date |
---|---|
US20190220978A1 true US20190220978A1 (en) | 2019-07-18 |
Family
ID=51259242
Family Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US14/093,470 Active 2032-10-04 US10282840B2 (en) | 2010-07-21 | 2013-11-30 | Image reporting method |
US16/363,032 Pending US20190220978A1 (en) | 2010-07-21 | 2019-03-25 | Method for integrating image analysis, longitudinal tracking of a region of interest and updating of a knowledge representation |
Family Applications Before (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US14/093,470 Active 2032-10-04 US10282840B2 (en) | 2010-07-21 | 2013-11-30 | Image reporting method |
Country Status (1)
Country | Link |
---|---|
US (2) | US10282840B2 (en) |
Cited By (14)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20210045716A1 (en) * | 2019-08-13 | 2021-02-18 | GE Precision Healthcare LLC | Method and system for providing interaction with a visual artificial intelligence ultrasound image segmentation module |
WO2021030684A1 (en) * | 2019-08-15 | 2021-02-18 | Advanced Solutions Life Sciences, Llc | Systems and methods for automating biological structure identification utilizing machine learning |
JP6908806B1 (en) * | 2020-09-16 | 2021-07-28 | BonBon株式会社 | Programs, information processing equipment, methods |
EP3916734A1 (en) * | 2020-05-27 | 2021-12-01 | GE Precision Healthcare LLC | Methods and systems for a medical image annotation tool |
US20220028058A1 (en) * | 2020-07-21 | 2022-01-27 | International Business Machines Corporation | Deep learning modeling using health screening images |
US11308619B2 (en) | 2020-07-17 | 2022-04-19 | International Business Machines Corporation | Evaluating a mammogram using a plurality of prior mammograms and deep learning algorithms |
US20220121330A1 (en) * | 2019-01-11 | 2022-04-21 | Google Llc | System, User Interface and Method For Interactive Negative Explanation of Machine learning Localization Models In Health Care Applications |
US20220336071A1 (en) * | 2019-08-12 | 2022-10-20 | Smart Reporting Gmbh | System and method for reporting on medical images |
US11514334B2 (en) | 2020-02-07 | 2022-11-29 | International Business Machines Corporation | Maintaining a knowledge database based on user interactions with a user interface |
US11527319B1 (en) * | 2019-09-13 | 2022-12-13 | PathAI, Inc. | Systems and methods for frame-based validation |
US11538567B2 (en) * | 2018-05-15 | 2022-12-27 | Intex Holdings Pty Ltd | Expert report editor |
US11830183B2 (en) | 2020-09-03 | 2023-11-28 | Merative Us L.P. | Treatment planning based on multimodal case similarity |
US11928221B2 (en) | 2021-11-29 | 2024-03-12 | Bank Of America Corporation | Source code clustering for automatically identifying false positives generated through static application security testing |
US11941115B2 (en) | 2021-11-29 | 2024-03-26 | Bank Of America Corporation | Automatic vulnerability detection based on clustering of applications with similar structures and data flows |
Families Citing this family (24)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP5620414B2 (en) * | 2012-01-18 | 2014-11-05 | 株式会社スクウェア・エニックス | Game device |
WO2015175031A1 (en) * | 2014-02-17 | 2015-11-19 | General Electric Company | Method and system for processing scanned images |
JP6490985B2 (en) * | 2014-03-11 | 2019-03-27 | キヤノンメディカルシステムズ株式会社 | Medical image processing apparatus and medical image processing system |
US11630874B2 (en) * | 2015-02-25 | 2023-04-18 | Koninklijke Philips N.V. | Method and system for context-sensitive assessment of clinical findings |
US20170083665A1 (en) * | 2015-09-23 | 2017-03-23 | Siemens Healthcare Gmbh | Method and System for Radiology Structured Report Creation Based on Patient-Specific Image-Derived Information |
JP7008017B2 (en) * | 2015-10-14 | 2022-01-25 | コーニンクレッカ フィリップス エヌ ヴェ | Systems and methods to generate accurate radiology recommendations |
EP3369020B1 (en) * | 2015-10-30 | 2021-06-30 | Koninklijke Philips N.V. | Image context aware medical recommendation engine |
US10402967B2 (en) * | 2015-12-21 | 2019-09-03 | Koninklijke Philips N.V. | Device, system and method for quality assessment of medical images |
CN107239722B (en) * | 2016-03-25 | 2021-11-12 | 佳能株式会社 | Method and device for extracting diagnosis object from medical document |
WO2018002265A1 (en) * | 2016-06-30 | 2018-01-04 | Koninklijke Philips N.V. | Generation and personalization of a statistical breast model |
EP3488367B1 (en) * | 2016-07-21 | 2022-03-30 | Koninklijke Philips N.V. | Annotating medical images |
US10140421B1 (en) * | 2017-05-25 | 2018-11-27 | Enlitic, Inc. | Medical scan annotator system |
EP3457353B1 (en) * | 2017-09-18 | 2020-11-25 | Siemens Healthcare GmbH | Method and system for obtaining a true shape of objects in a medical image |
CN109583440B (en) * | 2017-09-28 | 2021-12-17 | 北京西格码列顿信息技术有限公司 | Medical image auxiliary diagnosis method and system combining image recognition and report editing |
EP3692542A1 (en) * | 2017-10-05 | 2020-08-12 | Koninklijke Philips N.V. | System and method to automatically prepare an attention list for improving radiology workflow |
US10796430B2 (en) * | 2018-04-24 | 2020-10-06 | General Electric Company | Multimodality 2D to 3D imaging navigation |
KR20210064210A (en) * | 2018-09-24 | 2021-06-02 | 홀로직, 인크. | Breast Mapping and Abnormal Positioning |
US11457871B2 (en) | 2018-11-21 | 2022-10-04 | Enlitic, Inc. | Medical scan artifact detection system and methods for use therewith |
US11145059B2 (en) | 2018-11-21 | 2021-10-12 | Enlitic, Inc. | Medical scan viewing system with enhanced training and methods for use therewith |
US11282198B2 (en) | 2018-11-21 | 2022-03-22 | Enlitic, Inc. | Heat map generating system and methods for use therewith |
US10818386B2 (en) | 2018-11-21 | 2020-10-27 | Enlitic, Inc. | Multi-label heat map generating system |
EP3751575A1 (en) * | 2019-06-11 | 2020-12-16 | Esaote S.p.A. | A method for generating diagnostic reports and an imaging system carrying out the said method |
US11462315B2 (en) | 2019-11-26 | 2022-10-04 | Enlitic, Inc. | Medical scan co-registration and methods for use therewith |
US11669678B2 (en) | 2021-02-11 | 2023-06-06 | Enlitic, Inc. | System with report analysis and methods for use therewith |
Family Cites Families (20)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
BR0013268A (en) | 1999-08-09 | 2002-07-02 | Univ Wake Forest | Process implemented by computer to create a database that belongs to the analysis of an image and system to create a database that belongs to the analysis of an image |
US7756727B1 (en) | 2000-09-21 | 2010-07-13 | Carestream Health, Inc. | Method and apparatus for case building and presentation of healthcare procedures |
US6599130B2 (en) | 2001-02-02 | 2003-07-29 | Illinois Institute Of Technology | Iterative video teaching aid with recordable commentary and indexing |
US20050096530A1 (en) * | 2003-10-29 | 2005-05-05 | Confirma, Inc. | Apparatus and method for customized report viewer |
US7394946B2 (en) * | 2004-05-18 | 2008-07-01 | Agfa Healthcare | Method for automatically mapping of geometric objects in digital medical images |
US7756317B2 (en) | 2005-04-28 | 2010-07-13 | Carestream Health, Inc. | Methods and systems for automated detection and analysis of lesion on magnetic resonance images |
KR20080021723A (en) * | 2005-06-02 | 2008-03-07 | 더 메디패턴 코포레이션 | System and method of computer-aided detection |
US7979383B2 (en) | 2005-06-06 | 2011-07-12 | Atlas Reporting, Llc | Atlas reporting |
DE102006021036B4 (en) | 2006-04-28 | 2010-04-08 | Image Diagnost International Gmbh | Apparatus and method for computer aided analysis of mammograms |
US7945083B2 (en) * | 2006-05-25 | 2011-05-17 | Carestream Health, Inc. | Method for supporting diagnostic workflow from a medical imaging apparatus |
US7773791B2 (en) | 2006-12-07 | 2010-08-10 | Carestream Health, Inc. | Analyzing lesions in a medical digital image |
US7844087B2 (en) | 2006-12-19 | 2010-11-30 | Carestream Health, Inc. | Method for segmentation of lesions |
DE102007057015A1 (en) | 2007-11-23 | 2009-05-28 | Image Diagnost International Gmbh | Digital mammograms representing method for mammography interpretation of e.g. breast, involves displaying strip in mediolateral oblique mammogram for spot in craniocaudal mammogram, where strip indicates region in which spot is located |
US8150113B2 (en) | 2008-01-23 | 2012-04-03 | Carestream Health, Inc. | Method for lung lesion location identification |
US8150121B2 (en) | 2008-05-06 | 2012-04-03 | Carestream Health, Inc. | Information collection for segmentation of an anatomical object of interest |
DE102008035566B4 (en) | 2008-07-30 | 2018-09-06 | Image Diagnost International Gmbh | A method of processing a finding entered in a mammogram |
US8189886B2 (en) | 2008-08-13 | 2012-05-29 | Carestream Health, Inc. | Method for detecting anatomical structures |
US8953856B2 (en) * | 2008-11-25 | 2015-02-10 | Algotec Systems Ltd. | Method and system for registering a medical image |
US8290227B2 (en) | 2009-03-27 | 2012-10-16 | Carestream Health, Inc. | Method and system for diagnostics support |
US8311301B2 (en) | 2010-12-10 | 2012-11-13 | Carestream Health, Inc. | Segmenting an organ in a medical digital image |
-
2013
- 2013-11-30 US US14/093,470 patent/US10282840B2/en active Active
-
2019
- 2019-03-25 US US16/363,032 patent/US20190220978A1/en active Pending
Cited By (18)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11538567B2 (en) * | 2018-05-15 | 2022-12-27 | Intex Holdings Pty Ltd | Expert report editor |
US11934634B2 (en) * | 2019-01-11 | 2024-03-19 | Google Llc | System, user interface and method for interactive negative explanation of machine learning localization models in health care applications |
US20220121330A1 (en) * | 2019-01-11 | 2022-04-21 | Google Llc | System, User Interface and Method For Interactive Negative Explanation of Machine learning Localization Models In Health Care Applications |
US20220336071A1 (en) * | 2019-08-12 | 2022-10-20 | Smart Reporting Gmbh | System and method for reporting on medical images |
US20210045716A1 (en) * | 2019-08-13 | 2021-02-18 | GE Precision Healthcare LLC | Method and system for providing interaction with a visual artificial intelligence ultrasound image segmentation module |
WO2021030684A1 (en) * | 2019-08-15 | 2021-02-18 | Advanced Solutions Life Sciences, Llc | Systems and methods for automating biological structure identification utilizing machine learning |
US11915823B1 (en) * | 2019-09-13 | 2024-02-27 | PathAI, Inc. | Systems and methods for frame-based validation |
US11527319B1 (en) * | 2019-09-13 | 2022-12-13 | PathAI, Inc. | Systems and methods for frame-based validation |
US11514334B2 (en) | 2020-02-07 | 2022-11-29 | International Business Machines Corporation | Maintaining a knowledge database based on user interactions with a user interface |
EP3916734A1 (en) * | 2020-05-27 | 2021-12-01 | GE Precision Healthcare LLC | Methods and systems for a medical image annotation tool |
US11308619B2 (en) | 2020-07-17 | 2022-04-19 | International Business Machines Corporation | Evaluating a mammogram using a plurality of prior mammograms and deep learning algorithms |
US20220028058A1 (en) * | 2020-07-21 | 2022-01-27 | International Business Machines Corporation | Deep learning modeling using health screening images |
US11734819B2 (en) * | 2020-07-21 | 2023-08-22 | Merative Us L.P. | Deep learning modeling using health screening images |
US11830183B2 (en) | 2020-09-03 | 2023-11-28 | Merative Us L.P. | Treatment planning based on multimodal case similarity |
JP2022049586A (en) * | 2020-09-16 | 2022-03-29 | BonBon株式会社 | Program, information processing device, and method |
JP6908806B1 (en) * | 2020-09-16 | 2021-07-28 | BonBon株式会社 | Programs, information processing equipment, methods |
US11928221B2 (en) | 2021-11-29 | 2024-03-12 | Bank Of America Corporation | Source code clustering for automatically identifying false positives generated through static application security testing |
US11941115B2 (en) | 2021-11-29 | 2024-03-26 | Bank Of America Corporation | Automatic vulnerability detection based on clustering of applications with similar structures and data flows |
Also Published As
Publication number | Publication date |
---|---|
US10282840B2 (en) | 2019-05-07 |
US20140219500A1 (en) | 2014-08-07 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US20190220978A1 (en) | Method for integrating image analysis, longitudinal tracking of a region of interest and updating of a knowledge representation | |
US9014485B2 (en) | Image reporting method | |
US11164045B2 (en) | Complex image data analysis using artificial intelligence and machine learning algorithms | |
Wu et al. | Comparison of chest radiograph interpretations by artificial intelligence algorithm vs radiology residents | |
US7529394B2 (en) | CAD (computer-aided decision) support for medical imaging using machine learning to adapt CAD process with knowledge collected during routine use of CAD system | |
US10733727B2 (en) | Application of deep learning for medical imaging evaluation | |
US11462315B2 (en) | Medical scan co-registration and methods for use therewith | |
US20230108955A1 (en) | Deep-learning-based medical image interpretation system for animals | |
CN117501375A (en) | System and method for artificial intelligence assisted image analysis | |
JP2024009342A (en) | Document preparation supporting device, method, and program | |
US10438351B2 (en) | Generating simulated photographic anatomical slices | |
WO2021107099A1 (en) | Document creation assistance device, document creation assistance method, and program | |
US20220399107A1 (en) | Automated protocoling in medical imaging systems | |
US20230334663A1 (en) | Development of medical imaging ai analysis algorithms leveraging image segmentation | |
JP7007469B2 (en) | Medical document creation support devices, methods and programs, trained models, and learning devices, methods and programs | |
WO2020099941A1 (en) | Application of deep learning for medical imaging evaluation | |
US20240127917A1 (en) | Method and system for providing a document model structure for producing a medical findings report | |
EP4339961A1 (en) | Methods and systems for providing a template data structure for a medical report | |
JP7368592B2 (en) | Document creation support device, method and program | |
US11205520B1 (en) | Physician-guided machine learning system for assessing medical images to facilitate locating of a historical twin | |
EP4328855A1 (en) | Methods and systems for identifying a candidate medical finding in a medical image and providing the candidate medical finding | |
WO2022196106A1 (en) | Document creation device, method, and program | |
US20240078089A1 (en) | System and method with medical data computing | |
Taylor | Computer assisted decision making for image understanding in medicine |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: PRE-INTERVIEW COMMUNICATION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NON FINAL ACTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NON FINAL ACTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: FINAL REJECTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: ADVISORY ACTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NON FINAL ACTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: FINAL REJECTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: RESPONSE AFTER FINAL ACTION FORWARDED TO EXAMINER |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NON FINAL ACTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER |
|
STCV | Information on status: appeal procedure |
Free format text: APPEAL BRIEF (OR SUPPLEMENTAL BRIEF) ENTERED AND FORWARDED TO EXAMINER |
|
STCV | Information on status: appeal procedure |
Free format text: EXAMINER'S ANSWER TO APPEAL BRIEF MAILED |
|
STCV | Information on status: appeal procedure |
Free format text: ON APPEAL -- AWAITING DECISION BY THE BOARD OF APPEALS |