EP4684399A1 - Method(s) and/or system(s) for generating a medical imaging report - Google Patents

Abstract

A computer-implemented method is configured to link each anatomical tissue in a set of anatomical tissues to a corresponding set of possible findings and a different anatomical identifier, map voxels of image data to the different anatomical identifiers, receive a first user input selecting a first anatomical identifier, displaying a first list of possible findings for the first anatomical identifier, receiving a second user input selecting a first finding from the possible findings for the first anatomical identifier, and adding the first finding to a first entry in a list of selected findings.

Description

METHOD(S) AND/OR SYSTEM(S) FOR GENERATING A MEDICAL IMAGING REPORT

TECHNICAL FIELD

The following generally relates to a medical imaging report and more particularly to generating a structured medical imaging report, and is amenable to creating other structured reports.

BACKGROUND

Example workflow for a radiology examination includes a referring clinician prescribing an imaging examination of a subject via an imaging order, a radiology department/ center performing the imaging examination of the subject in accordance with the imaging order, the radiology department/center creating a report based on findings at least from a radiologist’s interpretation of an image from the imaging examination, and the radiology department/center providing the referring clinician with access to the report. Such reports can be unstructured or structured. An example of an unstructured report is a report generated by a radiologist dictating free text into a recording device and a transcriptionist transcribing the recording to generate the report. In another example, transcription (speech-to-text) software is utilized to transcribe the free text of the radiologist into the report. An example of a structured report is a digital form with editable report fields that are to be filled in with entries from predetermined lists of entries for the fields.

With unstructured reports, different clinicians describe what is imaged using different language in different ways to describe the same thing. Structured reports in radiology can lead to a more standardized and thus quality-controlled reporting from the radiologist to the referring clinician, which in turn will lead to better informed treatment decisions and thus better patient outcomes. However, it might be challenging for radiologists to keep track of which report fields they have filled themselves, which report fields have been automatically pre-filled by a model (e.g., artificial intelligence, etc.), which report field they have / have not checked, etc. In addition, software solutions typically enforce a certain workflow, e.g., for ensuring completeness of the structured reports, instead of supporting well-established diagnostic routines of the radiologists. As such, there is an unresolved need for an improved approach(s) for generating a structured report.

SUMMARY

Aspects described herein address the above-referenced problems and/or others. In one aspect, a computer-implemented method is configured to link each anatomical tissue in a set of anatomical tissues to a corresponding set of possible findings and a different anatomical identifier, map voxels of image data to the different anatomical identifiers, receive a first user input selecting a first anatomical identifier, displaying a first list of possible findings for the first anatomical identifier, receiving a second user input selecting a first finding from the possible findings for the first anatomical identifier, and adding the first finding to a first entry in a list of selected findings.

In another aspect, a computer-implemented method is configured to link each anatomical tissue in a set of anatomical tissues to a corresponding set of possible findings and a different anatomical identifier, display a user interactive decision tree, wherein each section of the decision tree is linked to an anatomical identifier of the anatomical identifiers, receive a first user-input selecting a section of the decision tree, display a first field of a structured report corresponding to the selected section and a first list of possible findings for a first anatomical tissue corresponding to the first field based on a first corresponding anatomical identifier, and display a sub-portion of the image data relevant to assessing the first anatomical tissue for selecting a first finding from the first list of possible findings for the first field.

In another aspect, a system includes a processor that is configured to execute computer-executable instructions, which causes the processor to display a user interface for generating a structured report, including displaying fields of the structured report and displaying image data or the image data and a user-interactive decision tree, receive a user input at the image data or the user-interactive decision tree; and automatically open a first section of the user-interactive decision tree corresponding to selected image data based on the user input at the image data or automatically render a region of the image data corresponding to a selected section of the user-interactive decision tree

Those skilled in the art will recognize still other aspects of the present application upon reading and understanding the attached description.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention may take form in various components and arrangements of components, and in various steps and arrangements of steps. The drawings are only for purposes of illustrating the embodiments and are not to be construed as limiting the invention.

FIG. 1 diagrammatically illustrates an example system, in accordance with an embodiment(s) herein. FIG. 2 diagrammatically illustrates example content of a user interface for generating a structured report, in accordance with an embodiment(s) herein.

FIG. 3 illustrates an example of a portion of a decision tree displayed in the user interface, in accordance with an embodiment s) herein.

FIG. 4 illustrates an example of a portion of a structured report displayed in the user interface, in accordance with an embodiment(s) herein.

FIG. 5 illustrates an example of a portion of image data displayed in the user interface, in accordance with an embodiment s) herein.

FIG. 6 illustrates an example of the user interface in connection with a display monitor, in accordance with an embodiment(s) herein.

FIG. 7 illustrates another example of the user interface in connection with a display monitor, in accordance with an embodiment(s) herein.

FIG. 8 illustrates the example of the portion of the structured report with a list of possible options for an editable field of the structured report, in accordance with an embodiment(s) herein.

FIG. 9 illustrates an example of the portion of the image data with a slice navigation bar, in accordance with an embodiment(s) herein.

FIG. 10 illustrates an example of the portion of the image data with a next slice control, in accordance with an embodiment s) herein.

FIG. 11 illustrates a part of the example of the portion of the decision tree with nodes visually highlighted to indicate a status of the nodes, in accordance with an embodiment s) herein.

FIG. 12 illustrates another part of the example of the portion of the decision tree with nodes visually highlighted to indicate a status of the nodes, in accordance with an embodiment s) herein.

FIG. 13 illustrates the example of the portion of the structured report where an unpopulated field is populated with an option from the list of possible options, in accordance with an embodiment(s) herein.

FIG. 14 illustrates the part of the example of the portion of the decision tree with nodes visually highlighted to indicate a status of the nodes, in accordance with an embodiment s) herein.

FIG. 15 illustrates an example method, in accordance with an embodiment(s) herein. FIG. 16 illustrates another example method, in accordance with an embodiment(s) herein.

FIG. 17 illustrates a correlation matrix, in accordance with an embodiment(s) herein.

DESCRIPTION OF EMBODIMENTS

FIG. 1 diagrammatically illustrates an example system 102. The system 102 includes a computing system 104, such as a computer, a workstation, etc. The computing system 104 includes a processor 106 and computer readable storage medium 108. Non-limiting examples of suitable processors include a central processing unit (CPU), a microprocessor ( P), graphics processing unit (GPU), and/or other processor. The computer readable storage medium 108 includes non-transitory storage medium such as physical memory, a memory device, etc., and excludes transitory medium. The computing system 104 further includes input/output (I/O) 110. The computing system 104 can be part of a picture archiving and communication system (PACS), an advanced visualization system for radiologists such as Philips® Intellispace® Portal, a Cardiology Information System (CVIS) such as Philips® Intellispace® Cardiovascular or Cardiology PACS (C-PACS), a computer workstation, a server, and/or other specialized apparatus for radiology or cardiology workflow or reading images.

An input device 112 is in electrical communication with the computing system 104 via the VO 110. A non-limiting example of the input device 112 includes a keyboard, a mouse, a microphone, etc. The input device 112 includes one or more input devices. An output device 114 is also in electrical communication with the computing system 104 via the I/O 110. A non-limiting example of the output device 114 includes a display monitor, a speaker, etc. The output device 114 includes one or more output devices. In one instance, the input device 112 and the output device 114 are separate devices (e.g., a keyboard and a display monitor). In another instance, the input device 112 and the output device 114 are the same device (e.g., a touch-screen monitor).

In the illustrated embodiment, a remote resource 116 is also in communication with the computing system 104 via the VO 110. The remote resource 116 includes one or more remote resource(s). Non-limiting examples of the remote resource 116 include an imaging system, a computing and/or archival system, and/or other resources. Non-limiting examples of the imaging system include a magnetic resonance imaging (MRI), a computed tomography (CT), an X-ray, etc. system. Non-limiting examples of the computing and/or archival system includes cloud processing resources, a server, a workstation, a Radiology Information System (RIS), Hospital Information System (HIS), an electronic medical record (EMR), a PACS, and/or other computing and/or archival system.

The processor 106 is configured to execute a computer readable instruction encoded or embedded in the computer readable storage medium 108. At least one computer readable instruction, when executed by the processor 106, causes the processor 106 to present a user interface (UI) for generating a structured report. As described in greater detail below, in one instance(s), the UI includes a user-interactive decision tree mapped at least to fields of the structured report and the image data where a user uses the decision tree, the structured report and/or the image data to drive generation of the structured report, and, in another instance, the user-interactive decision tree is omitted, closed or not used, the image data is mapped to the fields of the structured report and the user uses the structured report and/or the image data to drive generation of the structured report.

In one instance(s), using the decision tree allows for visually tracking the reporting progress and enables a quick navigation to the most relevant slices for not-yet-assessed nodes in the decision tree, which can overcome issues with keeping track of which report fields have been filled by radiologists, which report fields have been automatically pre-filled by a model, and which report fields have / have not been checked, etc., and using the image data allows embedding the selection of the required findings for the structured report into the (individually) well-established diagnostic workflow, which can overcome solutions that typically enforce a certain workflow, e.g., for ensuring completeness of the structured reports, instead of supporting well-established diagnostic routines of the radiologists.

FIG. 2 shows a non-limiting example of a UI 202 for generating a structured report. The UI 202 includes multiple windows, including a window 204, a window 206 and a window 208. At least a sub-portion of a decision tree (DT) 210 is displayed in one of the windows 204, 206 and 208, at least a sub-portion of a structured report (SR) 212 is displayed in another one of the windows 204, 206 and 208, and at least a sub-portion of image data (ID) 214 is displayed in the remaining window of the windows 204, 206 and 208. In the illustrated instance, the window 204 includes the decision tree 210, the window 206 includes the structured report 212, and the window 208 includes the image data 214. In another instance, the decision tree 210, the structured report 212 and the image data 214 are in different windows.

It is to be appreciated that the numbering of the windows (204, 206 and 208) is by order on introduction for explanatory purposes and does not impart any location of any window in the UI 202. For example, the window 204 does not have to be the leftmost window of the illustrated horizontal arrangement of the windows 204, 206 and 208. In addition, the illustrated horizontal arrangement of the windows 204, 206 and 208 is not limiting. For example, the windows 204, 206 and 208 can alternatively be arranged vertically (top to bottom) or some combination thereof (e.g., one horizontal and a vertical arrangement of two). At least two of the windows 204, 206 and 208 may also be arranged partially or fully overlapping. In general, the user can place the windows 204, 206 and 208 at any desired location with respect to each other.

Briefly turning to FIG. 3, a sub-portion 302 of a non-limiting example of the decision tree 210 is illustrated. Other style decision trees are also contemplated herein. The subportion 302 includes a set of nodes 304i, . . ., 304_n (collectively referred herein as nodes 304), where n is a positive integer greater than or equal to one, corresponding to different anatomy, including Anatomyi, . . ., Anatomy.

For each of the nodes 304, the sub-portion 302 includes a set of branches connected to a set of nodes corresponding to sub-anatomy, where a particular anatomy of a node includes sub-anatomy. For example, the illustrated node 304i includes branches 306i, . . ., 306k (collectively referred herein as branches 306), where k is a positive integer greater than or equal to one, connected respectively to a second set of nodes 308i, . . ., 308k (collectively referred herein as nodes 308), corresponding to Sub-Anatomyi, ..., Sub-Anatomyk.

In some instances, one or more of the sub-anatomy can be further delineated into different sub-anatomy (e.g., Sub- Anatomy i_a, Sub-Anatomyib, etc.). By way of non-limiting example, the knee (e.g., Anatomy;) includes sub-anatomy such as menisci (Sub -Anatomy;) and ligaments (Sub-Anatomyj), where the sub-anatomy menisci can be further delineated into medial meniscus (Sub- Anatomy ;_a) and lateral meniscus (Sub- Anatomy ib) and the sub-anatomy ligaments can be further delineated into cruciate ligaments (Sub-Anatomyj_a), medial collateral ligament (Sub-Anatomy^) and lateral collateral ligament (Sub-Anatomyj_c). However, for sake of brevity, the illustrated sub-portion 302 does show other sub levels.

For each of the nodes 308, the sub-portion 302 includes a set of branches connected to a set of nodes corresponding to a binary decision of whether a finding has been detected for the sub-anatomy of the node (e.g., likely “normal” / no finding and “abnormal” / finding). For example, the node 308i includes branches 310i and 3102 (collectively referred herein as branches 310), connected respectively to a set of nodes 312i and 3122 (collectively referred herein as nodes 312), corresponding respectively to Abnormal and Normal.

For the node 312i corresponding to Abnormal, the sub-portion 302 includes a set of branches 314i, . . . , 314_m (collectively referred herein as branches 314), where m is a positive integer greater than or equal to one, connected respectively to a set of nodes 316i, . . . , 316_m (collectively referred herein as nodes 316), corresponding to a set of different pathologies for the Sub-Anatomyi, including Pathologyi, Pathology_m.

For each of the nodes 316 corresponding to the set of different pathologies, the sub-portion 302 includes a set of branches 318i, ..., 318_P (collectively referred herein as branches 318), where p is a positive integer greater than or equal to one, connected respectively to a set of nodes 320i, . . ., 320_p (collectively referred herein as nodes 320), corresponding to Grade i, ..., Grade_p.

An example of a knee grading scheme includes the grading scheme for osteoarthritis severity grades (cartilage) by the International Cartilage Repair Society (ICRS). With this grading, “Grade 0” refers to normal cartilage (i.e. “Intact”), “Grade 1” refers to nearnormal cartilage with superficial lesions, “Grade 2” refers to cartilage with lesions extending to less than 50% of the depth of the cartilage, “Grade 3” refers to cartilage with defects that extend to more than 50% of the depth of the cartilage, and “Grade 4” refers to severely abnormal cartilage that the cartilage defects reaches to subchondral bone. Other grading schemes are also contemplated herein.

In one instance, the displayed sub-portion 302 of the decision tree 210 includes the entire decision tree 210. In another instance, the displayed sub-portion 302 includes less than the entire decision tree 210. For example, the displayed sub-portion 302 may include the delineation as shown in Fig. 3, more nodes or less nodes. In one instance, the nodes in the displayed sub-portion 302 of the decision tree 210 are based on predetermined display criteria, as described in greater detail below.

It is to be appreciated that the sub-portion 302 of the decision tree 210 is not limited to a horizontal (nodes 304, . . ., 320), left to right flow, with sub-nodes within any particular node being arranged vertically (nodes 304i, . . ., 304_n). In another instance, the subportion 302 of the decision tree 210 is arranged vertically from top to bottom flow. In this instance, the sub-nodes 304 of at least one node can be arranged horizontally, vertically and/or otherwise. In another instance, the nodes 304, . . ., 320 may be arranged with a combination of horizontal and vertical flow.

Briefly turning to FIG. 4, a non-limiting example of a sub-portion 402 of the structured report 212 is illustrated. Similar to the displayed sub-portion 302 of the decision tree 210, the sub-portion 402 may include the entire structured report or less than the entire structured report. For explanatory purposes and sake of brevity, the illustrated sub-portion 402 includes fields corresponding to a path of the sub-portion 402 of the structured report 212 shown in FIG. 3, namely, from 304i through 308i and 312i to 316 and 320. For example, the illustrated sub-portion 402 includes editable report fields 404, 406, 408, 410 and 412, respectively corresponding to the nodes 304, 308, 312, 316 and 320 of FIG. 3, which are identified respectively at 414, 416, 418, 420 and 422. In the illustrated subportion 402, entries have been selected for the editable report fields 404, 406 and 408, but not yet for the editable report fields 410 and 412. As discussed in greater detail below, the entries can be selected based on user input selecting an entry from a list of possible entries and/or automatically based on results from a machine learning or other algorithm.

Briefly turning to FIG. 5, a non-limiting example of a sub-portion 502 of the image data 214 is illustrated. Similar to the displayed sub-portion 302 of the decision tree 210 and the displayed sub-portion 402 of the structured report 212, the sub-portion 502 may include the entire image data (e.g., a three-dimensional (3D) rendering and/or one or more visible two- dimensional (2D) images/slices thereof) or less than the entire image data (e.g., a 3D rendering and/or one or more visible 2D images/slices corresponding to the sub-portion of the decision tree 210 and/or the structured report 212). The illustrated sub-portion 502 includes a slice relevant to editable report field 410 of FIG. 4.

In one instance, the windows 204, 206 and 208 are displayed on a single display monitor. In another instance, the windows 204, 206 and 208 are respectively displayed on a different display monitor. In another instance, two of the windows 204, 206 and 208 are displayed on one display monitor and the remaining window of the windows 204, 206 and 208 is displayed on a different display monitor. One or more of the windows 204, 206 and 208 may be configured to be “minimized” and/or “closed” or never open. For example, in one instance, a user may, via an input (e.g., mouse, keyboard, voice activation, gesture, etc.), “minimize” a window in that the content of the window is no longer made visible and a graphic such as an icon, etc. representing the window is displayed on the screen in place of the window.

By way of non-limiting example, FIG. 6 shows a single monitor 602 with windows 206 and 208 open, but not maximized in that they do not cover the entire display area 604, and a graphic 606 representing the minimized window 204. In FIG. 7, the window 204 is either “closed” or never opened. For example, if a user decides not to use the decision tree 210, the user can close the window displaying the decision tree 210 or never open the window displaying the decision tree 210 in the first place. In one instance, the user can open a “closed” or never opened window. Alternatively, FIG. 7 shows an instance in which the decision tree 210 is not part of the structured report generating software and is not displayed and cannot be used.

With further reference to FIGS. 1-7, as briefly discussed above, the displayed subportions 302, 402 and/or 502 can include less than the entire decision tree 210, the entire structured report 212 and/or the entire image data 214. In one instance, the displayed information is based on a default, a user preference, a healthcare entity preference, and/or otherwise. For example, the full decision tree 210 might be quite detailed and a user may prefer only a partial view. In this instance, downstream nodes may not be shown until required, e.g., in connection with FIG. 3, the gradings may not be displayed until a pathology is selected, and then only the gradings for the selected pathology are displayed.

In another instance, interaction with one of the displayed sub-portions 302, 402 and/or 502 of the decision tree 210, the structured report 212 and/or the image data 214 controls what is displayed in the other sub-portions 302, 402 and/or 502 of the decision tree 210, the structured report 212 and/or the image data 214. For this, an anatomical model is used to link the decision tree 210, the structured report 212 and/or the image data 214 together. In one instance, the anatomical model(s) provides a multi-label output, e.g., to be capable of segmenting a multitude of anatomical structures. In another instance, the anatomical model(s) provides a combination of multiple segmentation models providing single-label outputs or a combination thereof. In another instance, the anatomical model(s) provides both or a combination thereof.

A suitable anatomical model provides a map from an image voxel to a set of anatomical identifiers (IDs), e.g., voxekyz includes tissue of anatomy; or anatomiesi-_n, where the anatomical identifiers map to the same anatomy; or anatomiesi-_n in the decision tree 210 and/or the structured report 212. Examples of suitable models include U-NET or F-NET-based convolutional neural networks (CNNs), and the like. Other suitable examples are discussed in EP3493154, filed 12/1/2017 and entitled “Segmentation system for segmenting an object in an image,” US9824457B2, filed 8/21/2015 and entitled “Model-based segmentation of an anatomical structure,” EP 3120323, filed 2/27/2015 and entitled “Image processing apparatus and method for segmenting a region of interest,” all of which are incorporated in their entireties herein by reference.

By way of non-limiting example, in one instance, user interaction with the structured report 212 determines the displayed sub-portion 302 of the decision tree 210. For example, in one instance, when a user selects (e.g., “clicks” on, hovers over, points to, verbally identifies, etc.) a section of the structured report 212, the displayed sub-portion 302 of the decision tree 210 corresponds to the anatomical tissue covered in the section of the structured report 212. Additionally, or alternatively, the displayed sub-portion 502 of the image data 214 corresponds to the anatomical tissue covered in the section of the structured report 212.

In another instance, user interaction with the image data 214 determines the displayed sub-portion 302 of the decision tree 210. For example, in one instance, when a user selects (e.g., “clicks” on, hovers over, points to, verbally identifies, etc.) a section (e.g., an image, a set of images, a region of a 3D rendering, etc.) of the image data 214, the displayed subportion 302 of the decision tree 210 corresponds to the anatomical tissue covered in the section of the image data 214. Additionally, or alternatively, the displayed sub-portion 502 of the structured report 212 corresponds to the anatomical tissue in the section of the image data 214.

In another instance, user interaction with the displayed sub-portion 302 of the decision tree 210 determines the displayed sub-portion 302 of the structured report 212 and/or the displayed sub-portion 502 of the image data 214. For example, in one instance, the displayed sub-portion 502 of the image data 214 includes a slice(s) intersecting a center-of-mass of a mesh, voxels classified as “structure X” in response to a user “clicking” on a decision tree node of the displayed sub-portion 302 of the decision tree 210 and/or the displayed sub-portion 302 of the structured report 212 related to the “structure X,” etc.

The following describes examples of user interaction with the UI 202 using the image data 214 to drive generation of the structured report, with or without an availability of the decision tree 210.

In one instance, a user input selects a portion of the displayed image data 214. In response thereto, the processor 106 displays a section of the structured report 212 covering the anatomical tissue in the selected portion of the displayed image data 214 along with a list of possible findings for the anatomical tissue. As discussed herein, a set of possible entries, including findings, can be presented in a drop-down, a pop-up menu, a list box and/or otherwise. FIG. 8 shows an example of a list 800 presented in response to selecting a portion of the displayed image data 214. The list includes options 802, 804 and 806. This same list 800 and/or other list can be invoked and presented in response to user interaction with the decision tree 210.

In one instance, the displayed list contains only findings that are linked to anatomical IDs contained in the user-selected image portion (e.g., a cruciate ligament tear is only available for selection when the cruciate ligament is contained in the image portion). In one instance, additionally, findings linked to anatomical IDs within a predetermined vicinity (e.g., within x centimeter (cm)) of the user-selected image portion is suggested. In one instance, this may account for segmentation errors / imprecise user interaction, e.g., the tibial and femoral cartilage defects can be displayed when the user selects a voxel which, according to the anatomical model, belongs to the neighboring meniscus.

In one instance, the user-selected image portion of the image data is selected by “clicking” on voxels or slices, free hand drawing of a bounding box, placing and sizing a predetermined bounding box, etc. Additionally, or alternatively, the user-selected image portion of the image data is selected by hovering a pointing device such as a mouse, etc. over a voxel, an area and /or a slice for example for a predetermined period of time. In one instance, voxels that are connected to anatomical tissue already covered in the report are marked as such (e.g., with a color, an icon, etc.) so the clinician knows the anatomical tissue is already included in the structured report 212.

In one instance, in a separate viewing window, a list of all anatomical tissues is displayed, and the appearance of the anatomical tissues depends on the classification (e.g., “not viewed” = font characteristic 1, “normal” = font characteristic 2, “abnormal” = font characteristic 3). In one instance, the appearance further depends on whether the finding is a suggestion by a classification model, confirmed, changed by the clinician, e.g., font characteristic A: model suggestion, font characteristic B: selected by the radiologist).

In another instance, a 3D rendering of the anatomical tissues is displayed and similarly, the rendering of the anatomical tissues (e.g. transparency, color) depends on the classification status. In one instance, a clinician can click on an anatomical tissue in the list / rendering and the displayed image data 214 automatically includes a most relevant sequence(s) and/or slice(s) for the corresponding anatomical tissue (e.g., based on the anatomical model, e.g., a slice containing the center-of-mass or the largest diameter etc. of the anatomical tissue, or based on a saliency map for the suggested finding) is displayed. In one instance, an anatomical tissue is classified as “not viewed” if the clinician has not viewed all relevant slices for assessing that structure, which can be pre-defined, e.g., it may be defined that the meniscus should at least be viewed in coronal view on at least a proton density (PD) sequence).

In one instance, the list of possible findings displayed by the processor 106 corresponding to the selected portion of the image data 214 includes all of the findings which are linked to the anatomical IDs contained in the user-selected image portion with no further filtering / sorting. In another instance, the entries in the list are sorted according to a classification model. For this, the model can be a statistical model. In one instance, the model output is a statistical prevalence (e.g., based on historical data) of the findings.

In one instance, in response to a clinician not electing any voxel of the displayed image data 214, anatomical tissue in the displayed image data 214 is automatically classified as “normal / no finding” in the structured report 212. In another instance, when the clinician has already decided on a finding in a particular anatomical region, the subject finding and/or findings are excluded because of the selected finding is removed from further displayed list for user interaction. For example, if the clinician input selects an entry identifying a ligament as torn, an entry for a strained ligament is no longer a possible entry and is automatically removed from the displayed list.

In one instance, the decision to automatically remove a possible entry is rulebased or based on a rule. For instance, continuing with the above example, a rule could state that where torn is selected as the entry for ligament, non-torn ligament states such as strained, bruised, etc. are automatically removed from the displayed list. The rule can be invoked in response to the clinician input selecting the entry identifying the ligament as torn and/or otherwise. In one instance, the clinician can undo the selection. In this instance, the automatically removed entry is automatically added back to the list of possible entries and can be selected by the clinician.

In one instance in which a classification model is used for sorting the list of possible findings, various user interactions are contemplated. For example, in one instance, a most likely finding is automatically selected when the clinician does not interact any further with the list of possible findings and moves (e.g., scrolls, etc.) to a next slice. In one instance, the clinician is warned when a manual selection does not match the output of the classification model, e.g., the classification model has identified an abnormality (e.g., a certain probability threshold for an abnormality is exceeded) in an anatomical structure but the radiologist did not click on any voxel of the anatomical structure or the other way around. This warning may be a pop-up warning, or the classification model proposed finding may be displayed with a different color.

In one instance, the window (and/or a separate window) in which the displayed sub-portion 502 of the image data 214 is displayed includes a slice navigation bar with graphical indicia (e.g., color highlight, icons, etc.) that identify a slice(s) of the image data 214 displaying anatomical tissues for which the classification model has identified abnormal findings such that the radiologist can quickly navigate to slices with possible abnormal findings. In another instance, in addition to a sorted list, an indication of which area / region of interest (ROI) led to the abnormal classification is shown to the user, e.g. by a colored overlay in the image or a bounding box.

FIG. 9 shows an example with a slice navigation bar 900 to a side of the displayed sub-portion 502 of the image data 214. A current location marker 902 indicates a location of the displayed slice 502 within the image data. A relevant slice location markers 904 indicates a slice in the image data 214 with anatomical tissues for which the classification model has identified abnormal findings. When the user clicks on the marker 904, the portion of the 502 of the image data 214 displays the slice. FIG. 10 shows an alternative instance in which a “next’ control 1002 is displayed in the UI, and the user invokes the “next’ control to display the slice in the image data 214. Another instance includes both the slice navigation bar 900 and the “next’ control 1002.

The following describes examples of user interaction with the UI 202 using the decision tree 210 to drive generation of the structured report.

In one instance, the displayed sub-portion 302 of the decision tree 210 is visually interactive. For example, in one instance, a characteristic of a font of one or more nodes of the decision tree 210 visually imparts information about the one or more nodes. Examples of suitable characteristics include, but are not limited to, a color (e.g., a hue, a tint, a tone, a shade, etc.), a style (e.g., regular, italic, bold, bold italic, etc.), an effect (e.g., all capital letters, small capital letters, etc.), a size, a background, a transparency/opacity, etc.

By way of non-limiting example, the font characteristic of one or more nodes of the decision tree 210 can identify an availability of finding. For example, in one instance, a characteristic of a font for an available finding can include one characteristic and a characteristic of a font for an unavailable finding can include a different characteristic. For example, a color of a font for an available finding can be a color 1 and a color of a font for an unavailable finding can be a color 2, where the color 1 and the color 2 are different colors.

For example, FIG. 11, which shows a sub-portion of FIG. 3, indicates the Finding field 418 in the displayed portion 402 of the structured report 212 shown in FIG. 4 is populated, but the Pathology field 410 is empty, using a lighter shade of grey for the text at 312i in the decision tree 210 and a darker shade of gray for the text in fields 316i, . . . , 316_m in the decision tree 210. In FIG. 4, the editable field 408 for the Finding field 418 is populated with Abnormal and the editable field 410 for the Pathology field 420 is not populated / empty.

In one instance, the font characteristic changes in response to a clinician entering a finding into an unpopulated field. FIG. 12 shows the text in field 316i changes to the same lighter shade of grey for the text at 312i in the decision tree 210 in response to the user selecting a pathology for the editable field 410 for the Pathology field 420, and FIG. 13 shows the structured report of FIG. 4 updated to reflect a pathology (i.e. Pathologyi) has been selected for the editable field 410 for the Pathology field 420.

In one instance, the characteristic of a font changes based on a state of the finding. For example, in one instance, a font characteristic changes in response to a clinician confirming or rejecting an automatically populated finding. In one instance, the automatically populated finding is a finding that was determined based on artificial intelligence (Al) such as machine learning (ML). For example, in one instance the finding is determined using a neural network, such as convolutional neural network, that was trained on a set of images with findings where the output is a set of probabilities for each finding. Where the probability of an Al-determined finding satisfies a predetermined threshold, the Al-determined finding it automatically populated, and where the probability of an Al-determined finding does not satisfy the predetermined threshold, the Al-determined finding is not automatically populated.

In an instance where the editable fields 408 and 410 in FIG. 13 are automatically populated as such, FIG. 14 shows the text at 312i in the decision tree 210 italic, indicating the user accepted the automatically populated entry, and the text at 316i in the decision tree 210 underlined, indicating the user rejected the automatically populated entry. In general, the change of the characteristic of the font is rule-based or based on a rule. For instance, in the above example, a rule would state that if an automatically populated Al-determined finding is accepted by a clinician, then the text of the entry will be displayed in italic. The same rule or another rule would state that if the automatically populated Al-determined finding is rejected by the clinician, then the text of the entry will be underlined.

Additionally, or alternatively, a characteristic of the font depends on a certainty of automatically populated entry, e.g., based on a cross-entropy of classification results, e.g., where one characteristic indicates a high certainty while a different characteristic indicates a low certainty. Additionally, or alternatively, different characteristic of the font indicate different information, e.g., a status of an assessment of a decision tree nodes (e.g. bold = checked by the radiologist, and normal = not yet checked) while a color indicates a finding availability and/or certainty.

In one instance, in response to a user selecting a node of the decision tree 210, the processor 106 displays a set of possible entries for the node. For example, where the selected node is the pathology node 316i in FIG. 14, which has been populated with Pathologyi as shown in FIG. 13, a set of possible entries for the node are displayed with the structured report 212 for the editable field 412 for the Grade field 422 (similar to FIG. 8). The window can include a drop-down menu, a pop-up menu, a list box and/or otherwise. A user selects an entry from the list to populate the field.

Where the possible entries are findings, the entries can be sorted, e.g., based on statistical prevalence determined based on historical data, according to an output of a classification model. In another instance, a most likely finding is automatically selected for a node when the clinician does not interact with the possible entries associated with the node and instead selects another node of the decision tree210. Alternatively, the finding is automatically set to normal, and a warning is displayed to apprise the user when the most likely finding was not “normal.”

In one instance, an unpopulated node of the decision tree 210 upstream from another unpopulated node of the decision tree 210 is automatically populated in response to the downstream node of the decision tree 210 being populated. For example, if a clinician selects a “grade 3 meniscal tear of the anterior horn of the lateral meniscus” in the downstream part of the decision tree 210, then the upstream decisions “meniscus abnormal / normal?,” “Which part of the meniscus?” and “Above grade 1?” are implicitly made by the clinician and are automatically populated and set to “checked and selected by the clinician.”

In one instance, after the fields of a node have been completed, the user selects a next node in the decision tree 210 to complete. Alternatively, the processor 106 recommends a next node for the user to complete. In one instance, the processor 106 is triggered to provide a recommendation based on a user input, e.g., a user activating a “done” control button on the screen with a mouse, a stylus (.e.g., a pen, a finger, etc.), gesture, a verbal command, etc., or, similarly, a “next” control button. In one instance, the recommended node is highlighted, e.g. visually via a flashing light, a change of color, etc., audibly via a speaker under execution of speech software, etc. In another instance, the recommendation causes the node to be automatically selected and opened for interactive assessment.

In one instance, the processor 106 recommends a next node for reporting based on a maximum correlation with findings in the current node. By way of non-limiting example, after the anterior horn of the lateral meniscus has been completed, the processor 106 recommends the pars intermedia of the lateral meniscus since findings in these two areas are highly correlated. In one instance, such correlations can be computed by analyzing a sample collection of findings, e.g., from a population a subjects with a same or similar finding. In another example, the processor 106 recommends a next node for reporting based on an annotation history of the clinician, e.g., recommending a node that the clinician has most often chosen.

By way of non-limiting example, with respect to the maximum correlation, in one instance available medical records from a database, a server, a RIS, a HIS, an EMR, a PACS, etc. are analyzed to identify a population of subjects with a same or similar finding. With respect to the above example, the same or similar finding would be a finding for the anterior horn of the lateral meniscus. The identified population is analyzed to identify other related findings. In the above example, the other related findings would include findings related at least to the meniscus such as a part of the meniscus neighboring the anterior horn, e.g., the pars intermedia of the lateral meniscus. A statistical analysis is performed to determine a probability, a likelihood, etc. with which each of the other related findings occurs in the medical records of the identified population. The maximum correlation corresponds to findings with a highest probability, likelihood, etc. The processor 106 recommends the node corresponding to the maximum correlation. With respect to the above example, the maximum correlation corresponds to findings for the pars intermedia of the lateral meniscus, and the node corresponding thereto is recommended as the next node. Other examples of correlated findings include bone marrow edema and cartilage damage, anterior cruciate ligament tear and a bone bruise, etc.

Such correlations between findings can be predetermined and stored in a look-uptable (LUT), a matrix, a bar graph, and/or otherwise. Briefly turning to FIG. 17, an example of a matrix 1702 of correlations between findings for a set of findings is illustrated. It to be understood that the set of findings in the matrix 1702 is non-limiting; other correlation matrices may include more or less findings, additional and/or different findings, etc. The matrix 1702 of correlations shows the set of findings (on a first axis 1704) versus the same set of findings (on a second axis 1706).

Each intersection of two findings in the matrix 1702 of correlations indicates a correlation between the two findings, and a value of a correlation between two findings is represented as a gray level of a gray scale 1708 that is mapped to correlation values. By way of example, a finding 1710 of “meniscus tear” intersects with a finding 1712 of “bone trauma” at 1714 and intersects with a finding of 1716 “cartilage any defect” at 1718. Each of the intersections 1714 and 1718 is represented by a gray level of the gray scale 1708. The finding 1710 of “meniscus tear” also intersects with other findings, which are not discussed in detail.

In this example, from the gray scale 1708, the finding 1710 of “meniscus tear” is more positively correlated with the finding of 1716 “cartilage any defect” relative to the finding 1712 of “bone trauma” as the gray level at 1718 for the finding 1716 “cartilage_any_defect” maps to a higher gray level (around 0.27) in the gray scale 1708 and the gray level at 1714 for the finding 1712 of “bone trauma” maps to a lower gray level (around 0.00) in the gray scale 1708. The finding 1710 of “meniscus_tear” intersects with other findings in the matrix 1702 with correlation values between the correlation values for “bone trauma” and “cartilage_any_defect.”

With the matrix 1702, after the clinician enters the finding 1710 of “meniscus tear” in the entry for the current node, the processor 106 determines the finding 1716 “cartilage any defect” maps to a highest positive correlation for the findings in the matrix 1702 and recommends, as the next node, the node corresponding to the finding 1716 “cartilage_any_defect.”

In another instance, an Al-based method is used to identify a typical disease pattern. The disease patterns can be defined as a collection of findings that go along, for instance, with a typical kind of injury (e.g. knee sprain, twisted knee). The next node for reporting can be derived from the typical collection of findings. The Al-based method can be trained by having a database of a specific type of imaging examinations (e.g. MR knee) and by assigning typical patterns to the examinations in the database. When the Al algorithm is applied to an examination of this specific type, the Al algorithm can classify the examination with respect to the patterns, e.g., twisted knee examinations commonly include a finding of a knee sprain. As such, the processor 106 recommends the node that corresponds to knee sprains as the next node.

In another instance, collaborative filtering can be used to identify individuals from a data set that have a similar structure of pathologies. By way of non-limiting example, where a clinician confirms a meniscus tear, the processor 106 can look up other subjects in the data set that have a meniscus tear and identify other pathologies these subjects have in common with the subject under evaluation. The probability for these findings can be determined, and the node corresponding to the finding with a highest occurrence can be recommended as the next node.

It is to be appreciated that one or more (including all) of the aforementioned features can be automated, e.g., performed by the system without user input. However, to an extent that a user desires more control, one or more (including all) of the aforementioned features can be semi-automated, requiring user input, e.g., a user input accepting/rejecting an automated action.

FIG. 15 discloses a computer-implemented method. It is to be appreciated that the ordering of the acts in one or more of the method is not limiting. As such, other orderings are contemplated herein. In addition, one or more acts may be omitted, and/or one or more additional acts may be included. A linking step 1502 links each anatomical tissue in a set of anatomical tissues to a corresponding set of possible findings and a different anatomical identifier. A mapping step 1504 maps voxels of image data to the different anatomical identifiers. A receiving step 1506 receives a first user input selecting a region of the image data. A displaying step 1508 displays a first field of a structured report with a first list of possible findings for first anatomical tissue in the selected region based on a first corresponding anatomical identifier. A receiving step 1510 receives a second user input selecting a first finding from the first list of possible findings for the first anatomical tissue. A populating step 1512 populates the first field of a structured report corresponding to the first anatomical tissue with the selected first finding.

FIG. 16 discloses another computer-implemented method. It is to be appreciated that the ordering of the acts in one or more of the method is not limiting. As such, other orderings are contemplated herein. In addition, one or more acts may be omitted, and/or one or more additional acts may be included. A linking step 1602 links each anatomical tissue in a set of anatomical tissues to a corresponding set of possible findings and a different anatomical identifier. A displaying step 1604 displays a user interactive decision tree, wherein each section of the decision tree is linked to an anatomical identifier of the anatomical identifiers. A receiving step 1606 receives a first user-input selecting a section of the decision tree. A displaying step 1608 displays a first field of a structured report corresponding to the selected section and a first list of possible findings for a first anatomical tissue corresponding to the first field based on a first corresponding anatomical identifier. A displaying step 1610 displays a subportion of the image data relevant to assessing the first anatomical tissue for selecting a first finding from the first list of possible findings for the first field.

The above methods can be implemented by way of computer readable instructions, encoded, or embedded on the computer readable storage medium, which, when executed by a computer processor, cause the processor to carry out the described acts or functions. Additionally, or alternatively, at least one of the computer readable instructions is carried out by a signal, carrier wave or other transitory medium, which is not computer readable storage medium.

While some examples herein have been provided in the field of radiology, the systems, methods and/or operations described herein may be implemented in other areas where structured reporting is used such as cardiology, etc. Furthermore, while some examples herein are described in connection with a knee imaging examination, the systems, methods and/or operations described herein may be implemented for other anatomy and/or organs such the heart, e.g., in the case of a CVIS or cardiology workflow.

While the invention has been illustrated and described in detail in the drawings and foregoing description, such illustration and description are to be considered illustrative or exemplary and not restrictive; the invention is not limited to the disclosed embodiments. Other variations to the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed invention, from a study of the drawings, the disclosure, and the appended claims. The word “comprising” does not exclude other elements or steps, and the indefinite article “a” or “an” does not exclude a plurality. A single processor or other unit may fulfill the functions of several items recited in the claims. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

A computer program may be stored/distributed on a suitable medium, such as an optical storage medium or a solid-state medium supplied together with or as part of other hardware, but may also be distributed in other forms, such as via the Internet or other wired or wireless telecommunication systems. Any reference signs in the claims should not be construed as limiting the scope.

Claims

Claim 1. A computer-implemented method, comprising: linking each anatomical tissue in a set of anatomical tissues to a corresponding set of possible findings and a different anatomical identifier; mapping voxels of image data to the different anatomical identifiers; receiving a first user input selecting a first anatomical identifier; displaying possible findings for the first anatomical identifier; receiving a second user input selecting a first finding from the possible findings for the first anatomical identifier; and adding the first finding to a first entry in a list of selected findings.

Claim 2. The computer-implemented method of claim 1, wherein selecting the first anatomical identifier comprises receiving a user input selecting a region of the image data.

Claim 3. The computer-implemented method of claim 1, wherein selecting the first anatomical identifier comprises: displaying a user interactive decision tree, wherein each section of the decision tree is linked to a corresponding anatomical identifier of the different anatomical identifiers; receiving a user-input selecting a section of the decision tree; and displaying a sub-portion of the image data relevant to assessing the anatomical tissue corresponding to the selected anatomical identifier.

Claim 4. The computer-implemented method of any of claims 1 to 3, further comprising: populating a first field of a structured report corresponding to the first anatomical identifier with the first finding in the first entry in the list of selected findings.

Claim 5. The computer-implemented method of claim 4, wherein the first entry in the list is the first field of the structured report corresponding to the anatomical identifier

Claim 6. The computer-implemented method of any of claims 1 to 5, further comprising: displaying an image slice navigation bar that indicates images slices of the image data that include anatomical tissue identified by a classification model as abnormal.

Claim 7. The computer-implemented method of any of claims 1 to 6, further comprising: marking voxels of the image data that are linked to the populated first field.

Claim 8. The computer-implemented method of any of claims 4 to 7, further comprising: automatically populating a second field of the structured report that is upstream of the first field in response to populating the first field.

Claim 9. The computer-implemented method of any of claim 8, further comprising: automatically populating a third field of the structured report corresponding to anatomical tissue absent from the selected region to identify the anatomical tissue as normal.

Claim 10. The computer-implemented method of any of claims 1 to 9, further comprising: displaying a list of all anatomical tissues, each anatomical tissue rendered with a visual characteristic that identifies whether the anatomical tissue has been viewed or not viewed, including a visual characteristic that identifies whether a viewed anatomical tissue is classified as normal or abnormal.

Claim 11. A computer-implemented method, comprising: linking each anatomical tissue in a set of anatomical tissues to a corresponding set of possible findings and a different anatomical identifier; displaying a user interactive decision tree, wherein each section of the decision tree is linked to an anatomical identifier of the anatomical identifiers; receiving a first user-input selecting a section of the decision tree; displaying a first field corresponding to the selected section and a first list of possible findings for a first anatomical tissue corresponding to the first field based on a first corresponding anatomical identifier; and displaying a sub-portion of the image data relevant to assessing the first anatomical tissue for selecting a first finding from the first list of possible findings for the first field.

Claim 12. The computer-implemented method of claim 11, further comprising: mapping voxels of image data to the different anatomical identifiers; receiving a second user input selecting a region of the image data; displaying a second field with a second list of possible findings for second anatomical tissue in the selected region based on a second corresponding anatomical identifier; receiving a third user input selecting a second finding from the second list of possible findings for the second anatomical tissue; and adding the second finding to a first entry in a list of selected findings.

Claim 13. The computer-implemented method of any of claims 11 to 12, further comprising: identifying the section includes a field with an automatically populated finding with a first visual highlight; and changing the first visual highlight to a second visual highlight in response to a user accepting or rejecting the automatically populated finding.

Claim 14. The computer-implemented method of any of claims 11 to 13, further comprising: identifying a certainty of the automatically populated finding with a third visual highlight.

Claim 15. The computer-implemented method of any of claims 11 to 14, further comprising: identifying the section includes an unpopulated field with a fourth visual highlight; and changing the fourth visual highlight to a fifth visual highlight in response to a user adding a finding to the unpopulated field.

Claim 16. The computer-implemented method of any of claims 11 to 14, further comprising: automatically displaying a next field based on a predetermined correlation with findings in the selected section.

Claim 17. The computer-implemented method of any of claims 11 to 14, further comprising: automatically displaying a next field based on an annotation history of the user.

Claim 18. A system (102), comprising: a memory (108) with computer-executable instructions; and a processor (106) configured to execute the computer-executable instructions, which causes the processor to: display a user interface for generating a structured report, including displaying fields of the structured report and displaying image data or the image data and a user- interactive decision tree, receive a user input at the image data or the user-interactive decision tree; and automatically open a first section of the user-interactive decision tree corresponding to selected image data based on the user input at the image data or automatically render a region of the image data corresponding a selected second section of the user-interactive decision tree.

Claim 19. The system of claim 18, wherein the computer-executable instructions further cause the processor to: link each anatomical tissue in a set of anatomical tissues to a corresponding set of possible findings and a different anatomical identifier; display a user interactive decision tree, wherein each section of the decision tree is linked to an anatomical identifier of the anatomical identifiers; receive a first user-input selecting a section of the decision tree; display a first field of a structured report corresponding to the selected section and a first list of possible findings for a first anatomical tissue corresponding to the first field based on a first corresponding anatomical identifier; and display a sub-portion of the image data relevant to assessing the first anatomical tissue for selecting a first finding from the first list of possible findings for the first field.

Claim 20. The system of claim 18, wherein the computer-executable instructions further cause the processor to: link each anatomical tissue in a set of anatomical tissues to a corresponding set of possible findings and a different anatomical identifier; map voxels of image data to the different anatomical identifiers; receive a first user input selecting a region of the image data; display a first field of a structured report with a first list of possible findings for first anatomical tissue in the selected region based on a first corresponding anatomical identifier; receive a second user input selecting a first finding from the first list of possible findings for the first anatomical tissue; and populate the first field of a structured report corresponding to the first anatomical tissue with the selected first finding