EP4690233A1

EP4690233A1 - Artificial intelligence (ai) to assist data completeness check and improve radiology workflow efficiency

Info

Publication number: EP4690233A1
Application number: EP24714855.4A
Authority: EP
Inventors: Xin Wang; Sihao CHEN; Radhika DESHPANDE; Yuechen Qian; Sandeep Madhukar Dalal; Kyle HENNEBERRY
Original assignee: Koninklijke Philips NV
Current assignee: Koninklijke Philips NV
Priority date: 2023-03-24
Filing date: 2024-03-21
Publication date: 2026-02-11
Also published as: WO2024200174A1

Abstract

A radiology information technology (IT) system includes a computer storing radiology examination datasets produced by corresponding radiology examinations, the radiology examination datasets including radiology images; apply a rule to the radiology examination datasets to select (i) a first suspect set of radiology examination datasets for completeness review and (ii) a remaining set of radiology examination datasets not selected for completeness review by the rule; apply a model to the remaining set of radiology examination datasets to select a second suspect set of radiology examination datasets for completeness review; and annotate the radiology examination datasets of the first suspect set and the second suspect set with a completeness review request flag.

Description

2023PF00071

ARTIFICIAL INTELLIGENCE (Al) TO ASSIST DATA COMPLETENESS CHECK AND IMPROVE RADIOLOGY WORKFLOW EFFICIENCY

FIELD

[0001] The following relates generally to the radiology examination arts, picture archiving and communication system arts, artificial intelligence arts, quality control arts, and related arts.

BACKGROUND

[0002] A radiologist typically has a fixed amount of time allotted to complete the review of a radiology examination (i.e., imaging study or, more concisely herein, study) for a diagnosis and report. In some radiology departments, studies are classified by factors such as imaging modality or medical procedure code (e.g., CPT code), and studies are assigned Relative Value Unit (RVU) points with corresponding allotted time for reading the study. A radiologist is expected to perform readings with a certain number of total RVU points per shift. The heavy workload of radiologists can lead to fatigue and burn-out. If a radiology study has missing information, this requires the radiologist to interrupt the reading process to obtain the missing information, or the radiologist may need to put the study aside until the missing information is provided. The missing information could include missing items such as missing patient medical history information, missing information about relevant past imaging examinations (for example, if the current study is part of a series that is tracking progression of a medical condition), missing image views or slices (in which case additional imaging data acquisition might be required), a missing study order, missing information about image acquisition settings used in the study, or so forth. When the radiologist returns to the study upon receipt of the missing information, the radiologist will need to recall where the reading was left off before continuing. This can result in longer study reading time thus reducing efficiency of the radiologist, and has the potential to introduce errors into the final radiology report.

[0003] A radiology examination (i.e., study) is usually performed at the request of an ordering physician who prepares an examination request that (among other content) specifies the reason for examination. More detailed information may be provided such as requests for specific views, imaging sequences, or the like. A radiology technologist performs the radiology examination to acquire images in accord with these instructions. The resulting images are annotated and stored in a picture archiving and communication system (PACS) database, usually in a standard DICOM format. At some time thereafter, the radiology examination data is retrieved from the PACS and reviewed by a radiologist who prepares a radiology report identifying the radiologist’s findings. In preparing this report, the radiologist should review all relevant information, which may include for example reviewing past radiology examinations of the same patient (e.g., to determine whether a cancerous tumor or growth has increased or decreased in size), past patient examinations of other types, information on the patient’s treatment regimen (e.g., when and what type of chemotherapy, radiation therapy, or other cancer treatment regimen events have occurred), reviewing the patient’s medical history to identify any relevant chronic conditions, and/or so forth.

[0004] A radiologist operating in a teleradiology setting may face further difficulty as access to the missing information may be more limited. In a teleradiology workflow, a hospital typically will contract with a teleradiology service provider, and will send a radiology examination dataset to the teleradiology service provider. The radiology examination dataset are normally stored at the hospital PACS (e.g., the radiology images and radiology order). The radiology examination dataset provided to a teleradiology service provider may include other information such as the patient’s treatment regimen, past radiology examination information, relevant patient medical history information, and so forth. A radiologist at the teleradiology service provider then reviews the radiology examination dataset and prepares a radiology report which is transmitted back to the hospital. While a contracted teleradiology service provider was just described, other teleradiology scenarios are possible, such as a central teleradiology service handling the reading of all radiology examinations generated by a network of hospitals, or a governmental teleradiology service handling cases generated by military or veteran’s hospitals, as some further examples.

[0005] When a radiologist working at a hospital or at a teleradiology service opens an imaging examination dataset in a PACS database, the radiologist may find data are incomplete for them to interpret the exam and complete the report, such as missing certain views, slices, and/or series, missing prior images or reports, missing or incomplete contrast information, missing patient treatment regimen information, missing an ultrasound technologist worksheet, etc. In this case, the radiologist then has to obtain the information directly, or contacts a radiology workflow support team to have it resolved. In either case, this disrupts radiologist’s reading workflow and reduces efficiency. To avoid this situation, all radiology examination datasets can be routed to the radiology workflow support team first to check data completeness, and then be assigned to the radiologists once any data incompleteness issues are resolved. However, checking all the cases is time-consuming and labor intensive.

[0006] The following discloses certain improvements to overcome these problems and others.

SUMMARY

[0007] In one aspect, a radiology system includes a computer programmed to store radiology examination datasets produced by corresponding radiology examinations, the radiology examination datasets including radiology images; apply a rule to the radiology examination datasets to select (i) a first suspect set of radiology examination datasets for completeness review and (ii) a remaining set of radiology examination datasets not selected for completeness review by the rule; apply a model to the remaining set of radiology examination datasets to select a second suspect set of radiology examination datasets for completeness review; and annotate the radiology examination datasets of the first suspect set and the second suspect set with a completeness review request flag.

[0008] In another aspect, a method of processing radiology examination datasets includes: using an electronic processor, classifying each radiology examination dataset as requiring completeness review or as ready for reading; providing a quality control user interface for performing completeness review of the radiology examination datasets classified as requiring completeness review; for each reviewed radiology examination data set generating a request for additional information to complete the radiology examination dataset or re-classifying the radiology examination dataset as ready for reading; and providing a radiology reading user interface for reading and preparing radiology reports on the radiology examination datasets ready for reading.

[0009] In another aspect, a non-transitory storage medium stores instructions readable and executable by an electronic processor to perform a method of assessing radiology examination datasets produced by corresponding radiology examinations and including radiology images, the method including: applying a rule to the radiology examination datasets to select (i) a first suspect set of radiology examination datasets for completeness review and (ii) a remaining set of radiology examination datasets not selected for completeness review by the rule; applying a model to the remaining set of radiology examination datasets to select a second suspect set of radiology examination datasets for completeness review; and annotating the radiology examination datasets of the first suspect set and the second suspect set with a completeness review request flag.

[0010] One advantage resides in providing targeted, automatic routing of radiology examinations to either a radiology workflow support team or directly to a radiologist.

[0011] Another advantage resides in providing such routing using a combination of rules and artificial intelligence (Al) to determine radiology exams that have missing data.

[0012] Another advantage resides in providing such routing which is tunable as to the ratio of cases sent to the support team versus directly to radiologists to account for current staffing or other workflow efficiency considerations.

[0013] Another advantage resides in reducing interruptions in a radiologist’s reading workflow while increasing efficiency.

[0014] Another advantage resides in reducing an amount of exams to be reviewed by a workflow support team.

[0015] Another advantage resides in updating a PACS with a flag of exams having (or likely to have) missing data.

[0016] A given embodiment may provide none, one, two, more, or all of the foregoing advantages, and/or may provide other advantages as will become apparent to one of ordinary skill in the art upon reading and understanding the present disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The disclosure may take form in various components and arrangements of components, and in various steps and arrangements of steps. The drawings are only for purposes of illustrating the preferred embodiments and are not to be construed as limiting the disclosure.

[0018] FIGURE 1 diagrammatically illustrates an illustrative radiology information technology system in accordance with the present disclosure.

[0019] FIGURE 2 shows a sequence chart of exemplary flow chart operations of the system of FIGURE 1.

[0020] FIGURE 3 is a flowchart of a method depicting an example of a method performed by the system of FIGURE 1.

[0021] FIGURE 4 is a sequence diagram illustrating an example of the method of FIGURE 3 performed by the system of FIGURE 1. [0022] FIGURE 5 diagrammatically illustrates a radiology information technology system in accordance with another nonlimiting illustrative embodiment the present disclosure.

[0023] FIGURE 6 diagrammatically illustrates a radiology information technology system in accordance with another nonlimiting illustrative embodiment the present disclosure.

DETAILED DESCRIPTION

[0024] With reference to FIGURE 1, an illustrative radiology information technology (IT) system 10 is diagrammatically shown. The illustrative system 10 is in a teleradiology service context (e.g., where radiology services are conducted remotely for a customer like a hospital, imaging center, or radiology lab under a contract for example as described above); however, it will be appreciated that the disclosed approaches for providing targeted, automatic routing of radiology examinations to either a radiology workflow support team to obtain missing information, or directly to a radiologist for reading, can be usefully employed in other settings such as a hospital radiology department or similar. The illustrative system 10 is configured to aid in maintaining and updating one or more picture archiving and communication system (PACS) 12 (hereinafter referred to as a PACS 12). Generally, PACS 12 may be a standalone PACS or a server/client system; in a server/client PACS system for example, images can be stored on a server and read by a radiologist at the hospital or radiology department, or resources are otherwise distributed or shared. However, in the teleradiology context as described herein, PACS 12 stores imaging examinations and provides the imaging examinations to remotely located PACS clients where they are read by a radiologist as part of a teleradiology service under contract to a customer hospital or radiology lab. Two PACS 12 are shown in FIGURE 1 by way of illustrative example - a first hospital PACS 12 storing a first set of imaging examinations 13 generated from images by a first set of medical imaging devices 14. By way of some nonlimiting illustrative examples, the medical imaging device(s) may be a magnetic resonance imaging (MRI) scanner, a computed tomography (CT) scanner, a positron emission tomography (PET) scanner, a gamma camera for performing single photon emission computed tomography (SPECT), an interventional radiology (IR) device, or so forth. A second radiology lab PACS 12 stores a second set imaging examinations 13 generated from images by a second set of medical imaging devices 14. Although two PACS 12 is shown in FIGURE 1, the IT system 10 can maintain and update any suitable number of PACS 12 (indicated in FIGURE 1 with ellipses). While two PACS systems 12 are shown as illustration, more generally the teleradiology service described below may serve dozens or more hospitals, radiology laboratories, and/or the like, each having a PACS 12.

[0025] The system 10 also includes an electronic processor such as an illustrative computer 16 which may in some embodiments be a server computer, e.g., in a teleradiology context the computer 16 may host a server-based portion of a Teleradiology PACS, which provides at least a portion of the IT infrastructure for a teleradiology service that serves various hospitals, radiology departments, or the like represented by the two illustrative PACS 12. In some implementations, the Teleradiology PACS includes client-side components running on workstations or other suitable electronic processing devices 18, in which case, all or some functionality of computer 16 described in the following paragraphs may also be applied at the reading workstation 18 (e.g., the processing of functions can distributed over computer 16 and device 18 in any number of permutations), or alternatively, a Teleradiology PACS may also be a standalone PACS (not shown) that is not implemented in a server/client architecture, in which case all functionality of computer 16 may be applied on such a standalone PACS where the PACS is not implemented in a server/client architecture. In other words, the functionality of computer 16 is only illustrated by way of example and may be applied by or on any form of PACS implementation with the goal of completeness review of a radiology examination dataset before the radiologist takes up review of the radiology examination dataset. The computer 16 includes typical components such as an electronic processor and a non-transitory computer readable medium storing instructions executable by the electronic processor. At least one radiology reading workstation or other suitable electronic processing device 18 (three of which are shown in FIGURE 1) includes an electronic processor and a non-transitory computer readable medium storing instructions executable by the electronic processor, and is configured to provide a user interface 20 for performing radiology examination readings. Each radiology electronic processing device 18 can be staffed by a radiologist and is programmed to display radiology images of a radiology examination dataset undergoing reading and to receive a radiology report 17 on the radiology examination dataset undergoing reading by the radiologist. At least one quality control (QC) review electronic processing device 22 (three of which are shown in FIGURE 1) is staffed by a QC reviewer and is configured to provide a user interface 24 for performing completeness reviews of radiology examination readings. Each QC review electronic processing device 22 is programmed to display a summary of the contents of a radiology examination dataset undergoing completeness review by a QC support member and receive a request for additional information to complete the radiology examination dataset undergoing completeness review.

[0026] The PACS(s) 12, the computer 16, the radiology electronic processing device(s) 18, and the QC review electronic processing device(s) 22 are electronically connected with one another via an electronic network 28 (e.g., a wired connection, a local area network (LAN), the Internet, wireless Wi-Fi or 4G/5G interface or the like for connection to the Internet and/or an intranet, and so forth). In the illustrative teleradiology setting, radiology examination datasets 13 are uploaded from the hospital (or radiology lab) PACS 12 to the computer 16 which hosts the teleradiology PACS (or a server-side portion thereof) for reading by radiologists employed by the teleradiology service. In another approach, the radiology examination datasets 13 may be transferred directly from the medical imaging device 14 to the computer 16 which hosts the teleradiology PACS, as diagrammatically indicated in FIGURE 1 by a transfer path arrow DP. The radiology reports 17 prepared by radiologists at the radiology workstations 18 are sent to the customer (e.g., hospital, imaging center, or radiology lab) via the electronic network 28. Note that while for simplicity of illustration FIGURE 1 shows the prepared radiology reports 17 transferred from the radiology workstations 18 to the electronic network 28, more typically the prepared radiology reports 17 will be first stored at the teleradiology PACS hosted by the computer 16, or in another suitable database (e.g., a RIS), and then will be sent from the computer 16 to the hospital or radiology lab PACS 12 of the customer via the electronic network 28.

[0027] The foregoing provides some nonlimiting illustrative examples of PACS systems. In general, a PACS is a specialized apparatus and technology for medical imaging that provides support for a radiologist (or cardiologist in the case of cardiovascular imaging, but radiologist use cases will be exemplified more readily throughout this disclosure) in reading (e.g., accessing, processing, and/or analyzing) images created by different imaging modalities such as, but not limited to, Digital Radiography (DR), Computed Tomography (CT), Magnetic Resonance (MR), Ultrasound (US), and Nuclear Medicine (NM). Moreover, while FIGURE 1 illustrates a teleradiology context, the disclosed approaches for providing targeted, automatic routing of radiology examinations to either a radiology workflow support team or directly to a radiologist can be usefully employed in other contexts (e.g., see FIGURE 5 described later herein).

[0028] In some illustrative embodiments, the computer 16 hosts a PACS, and the targeted, automatic routing of radiology examinations is implemented in the PACS, or as a PACS add-on component (e.g., a plug-in). A PACS can be designed in many different ways. An exemplary PACS technology infrastructure may include imaging device interfaces, storage devices, host computers, communication networks, and display systems often integrated by a flexible software package for supporting a radiologist in reading a patient case (or otherwise referred to as reading an image study) through a diagnostic workflow. Common specialized hardware components may include for example, patient data servers, data/modality interfaces, PACS controllers with database and archive, and display workstations connected by communication networks for handling and managing efficient data/image flow. A PACS also contains various hardware and software for enabling a user (e.g., a radiologist) to examine image slices at very high fidelity across multiple UIs. A PACS is a synergy of specialized hardware and flexible software. The flexible software also includes various functions such as, for example, but not limited to, measurement, segmentation, tumor or lesion identification, landmark detection, visualization, and reporting on clinical findings.

[0029] Medical images or image studies such as the illustrative radiology examination datasets 13 are typically transmitted to a PACS electronically/digitally e.g., via a communication channel and/or via the Digital Imaging and Communications in Medicine (DICOM) protocol, which includes a file format definition and a network communications protocol, and uses for example, Transmission Control Protocol (TCP) / Internet Protocol (IP) TCP/IP to communicate between systems. Images to be read at the PACS and other non-image patient information are often referred to collectively as an image study (or, as alternatively used herein, a radiology examination dataset 13). Image studies are typically assigned to a radiologist(s) and displayed in a worklist for a radiologist of the group who will read the images in the image studies, for example at one of the illustrative radiology reading stations 18. The radiologist loads and reads the image study (including the images) at the radiology reading station 18, reviews regions of interest in the image(s), identifies clinical findings in the image(s), and creates a report with a diagnosis, which is saved (e.g., in the PACS hosted by the computer 16 or to some other information system such as a RIS, HIS, EMR, et cetera). As a nonlimiting illustrative example, in an oncological imaging study, a primary region of interest may be a malignant tumor and surrounding tissue, and findings may include by way of nonlimiting illustrative example, physical dimensions of the tumor, metrics of tumor density, a finding of whether the cancer has metastasized (and if so metrics of the extent thereof), and/or so forth. The radiologist prepares the report 17 (for example, called a radiology report or the like) summarizing these findings, the report may be transmitted to another information system via HL-7 and the images (or certain key image(s) identified by the radiology reading) may be transmitted via DICOM. Non-image data, such as a scanned document, may be incorporated, e.g., using formats such as Portable Document Format (PDF).

[0030] The PACS may also provide a reporting environment sometimes employing a report template that is filled in by the radiologist, an image viewer (integrated with the reporting environment and/or standalone), and so forth. To assist the radiologist in reading an image study, the PACS may provide various manual, semi-automated, or automated image analysis tools, such as: anatomy labeling; contouring (i.e., identifying the boundary of) a tumor or other region of interest; measurement of clinically significant metrics such as tumor dimensions and/or tumor density; comparison of a current image with a corresponding image of a previous study of the patient (e.g., to assess tumor growth or shrinkage). In the case of cardiology PACS, cardiovascular metrics may be assessed such as artery wall thickening; and/or so forth. Such image analysis tools reduce the time and effort by the radiologist (or cardiologist as the case may be) in reading an imaging study, and thereby greatly increase the throughput, efficiency, and clinical accuracy of study readings.

[0031] Furthermore, the PACS may include various artificial intelligence (Al) algorithms (not shown) to analyze medical images to automatically detect various clinical findings, such as detecting a meniscus tear, a bone fracture, potentially malignant lesions, and so forth. In most jurisdictions, the radiology reading must be performed by a suitably trained and credentialed radiologist. Radiologists must often complete a review of an image study within a certain period of time, which is sometimes predetermined. Radiologists can become fatigued or “burned out” from having to read many image studies under time pressure (e.g., within a certain time period). Al finding detectors may serve in a supporting role, detecting proposed clinical findings that are then presented to the radiologist during the radiology reading for acceptance or rejection by the radiologist.

[0032] With continuing reference to FIGURE 1, the teleradiology PACS hosted at the computer 16 is programmed to receive the radiology examination datasets 13 that are outsourced for reading by the teleradiology service. Ideally, each radiology examination dataset 13 would be a complete dataset with all information needed for the radiologist at the teleradiology service to prepare a complete radiology report on his or her clinical findings. However, in practice, some of the uploaded radiology examination datasets may be incomplete, e.g., missing the radiology order specifying the reason for examination, missing prior radiology examination information, missing patient treatment information, missing relevant patient data, or so forth, or may be missing imaging views, sequences, or the like that are needed to perform the reading. In such cases, QC personnel staffing the QC workstations 22 should identify these incomplete radiology examination datasets and coordinate with the hospital, radiology laboratory, or other source to complete the dataset. However, if the teleradiology service is handling a large workload of radiology examinations outsourced to the teleradiology service for reading, then it may be difficult or impossible for the QC staff to review every incoming radiology examination dataset for completeness. On the other hand, if the QC staff does not review a certain fraction of incoming radiology examination datasets, then some incomplete radiology examination datasets are likely to reach the radiologists, who will put some time attempting to read the examination before realizing there is missing information that prevents a clinically acceptable radiology reading.

[0033] To address this problem, the computer 16 is further programmed to apply a rule 30 to the radiology examination datasets to select (i) a first suspect set 32 of radiology examination datasets for completeness review and (ii) a remaining set 34 of radiology examination datasets not selected for completeness review by the rule 30. The rule 30 in some illustrative embodiments selects a radiology examination dataset for completeness review based on one or more of originating hospital or radiology department; reference to a prior radiology examination in the radiology examination dataset; imaging modality of the images of the radiology examination dataset; information on an intravascularly administered contrast agent in the radiology examination dataset; and/or a timestamp of the radiology examination dataset. The rule 30 comprises one or more rules derived from a plurality of criteria including, for example, patterns in cases with missing information, a prevalence of cases with missing information from certain facilities and modalities, a distribution of types of missing information including prior images or reports, contrast agent details, ultrasound worksheets, and incomplete current images.

[0034] Retrospective data analysis brings forth observations, and patterns in cases with missing information, such as prevalence of cases with missing information from certain facilities and modalities, or distribution of types of missing information like prior images/report, contrast details, ultrasound worksheets or incomplete current images. Based on these insights and inferences, the case selection of data for a completeness check using the rule 30 can be constructed. Rule based case selection, such as only selecting the cases from facilities and modalities that have higher missing rate or excluding from the quality check workflow certain hours in the day that are observed to have the least number of cases with missing information helps in improving the efficiency of the radiologists and/or the QC support member.

[0035] The rule 30 is typically manually constructed based on prior experience such as which hospitals or radiology laboratories tend to send incomplete radiology examination datasets for readings, which examination time windows tend to have a high fraction of incomplete examinations, or so forth. In some cases, the rule 30 may be generated automatically or semiautomatically, e.g., by statistical analysis of past received radiology examination datasets. However, this rule-based approach alone does not cope well with changing capacities and trends. To keep up with the dynamic workflow, changing radiologist capacity, and onboarding of new facilities and/or modalities, retrospective data analysis may be done periodically to gather inferences and patterns for deriving up-to-date pre-defined rules, i.e., to update the rule 30. Moreover, the rules may be unable to account for less frequent causes of incomplete examination datasets. For example, a given hospital may usually provide complete examination datasets, and so the rule 30 usually does not place examination datasets from that hospital in the first suspect set 32 of radiology examination datasets. However, if that hospital is understaffed for a short period of time, it may then produce some incomplete cases. If the understaffing is very short term, it may not be feasible for the rule 30 to be updated to catch this short-term understaffing.

[0036] In a first example, for all facilities and based on history data, “facility 1”, “facility 2”, “facility3”, and “facility4” are determined to be the top four facilities that have the highest fraction of incomplete cases. A corresponding generated rule 30 could be to send all radiology examination datasets 13 from these four facilities to the QC review electronic processing device(s) 22, but skipping studies from all other facilities. In a second example, for all study types (e.g., modality, body part being imaged, and so forth) and based on history data, “study type 1”, “study type 2”, “study type 3”, and “study type 4” are the top four study types that having a highest fraction of incomplete cases. A corresponding generated rule 30 could be to send all studies in those four study types to the QC review electronic processing device(s) 22, but skipping studies from all other study types.

[0037] To provide flexibility to cope with changing capacities and trends, the computer 16 is further programmed to apply a model 36 to the remaining set 34 of radiology examination datasets (i.e., those radiology examination datasets which are not in the first suspect set 32 of radiology examination datasets) to select a second suspect set 38 of radiology examination datasets for completeness review. In some embodiments, the model 36 comprises an artificial intelligence (Al) model (e.g., an artificial neural network such as convolutional neural network or recurrent neural network, a decision tree, an ensemble model, or any other suitable model) configured to receive, as inputs, data related to at least a modality of the medical imaging device(s) 12 used to acquire the images of patients included in the radiology examination datasets 13, the images themselves, facilities where the medical imaging devices 12 are located, a time of an imaging examination, a body art of the patient that is imaged, and so forth. The Al model 36 then outputs a probability that one of more of the radiology examination datasets is missing information or is not missing information. In some embodiments, the computer 16 is programmed to control the applying of the model 36 to the remaining set 34 of radiology examination datasets to select the second suspect set 38 of radiology examination datasets for completeness review so that a total number of radiology examination datasets in the combined first and second suspect sets 32, 38 does not exceed a maximum number for completeness review. This ensures that the QC support members are not overwhelmed with an excessive number of datasets to review. The maximum number for completeness review is selected (i.e., adaptively adjusted) based on a current capacity to perform completeness reviews of radiology examination datasets. The current capacity may be determined, for example, based on the number of QC personnel currently on-staff, optionally adjusted by their seniority or another metric of QE personnel efficiency. This approach adapts well to the dynamic workflow, changing radiologist capacity and onboarding of new facilities and/or modalities, without having the need to do frequent retrospective data analysis.

[0038] In response to input of a radiology examination dataset 13 to the Al model 36, the output of the Al model 36 is a likelihood or probability that the radiology examination dataset 13 is complete (or, alternatively, the Al model 36 could output a likelihood or probability that the radiology examination dataset 13 is incomplete). Then an operating point (e.g., a threshold confidence score) for binary classification of the output of the Al model 36 is dynamically selected based on how many cases the radiology practice would like to select for manual check based on the Al output. For example, if the Al model 36 outputs a probability in a range of 0-1 that the dataset is complete (so that the limit 0 means the dataset is certainly not complete while 1 means the dataset is certainly complete), then examination datasets whose score is below the threshold should be sent for QC review, and hence lowering the threshold will result in fewer cases being sent for QC review while raising the threshold will result in more cases being sent for QC review. Conversely, if the Al model 36 outputs a probability the dataset is incomplete (i.e., not complete), then examination datasets whose score is above the threshold should be sent for QC review, and hence raising the threshold will result in fewer cases being sent for QC review while lowering the threshold will result in more cases being sent for QC review.

[0039] This threshold for selection of cases is suitably dynamically determined by metrics of the current workflow, including capacity of the radiology workflow support team, total study volume, radiologist capacity, the time-to-report (TR) status of the exams, etc. This can be performed in a variety of manners. As a nonlimiting illustrative example, if the rules select 50 cases for day 1 and 70 cases for day 2, then for day 1 the selection threshold applied to the output of the Al model 36 should be chosen to select 50 cases (to provide 50 rules-selected cases and 50 Al-selected cases for a total of 100 cases); but the selection threshold should be changed for day 2 so that the changed threshold applied to the output of the Al model 36 should select only 30 cases for day 2 (to again provide a total of 100 cases, here 70 rules-selected cases and 30 AI- selected cases). To implement this, the operating point (threshold) used in day 2 should be higher than day 1 (in the case where the output of the Al model 36 is a probability of dataset incompleteness). In addition, the threshold could be selected not only based on the total number of cases to be selected by the Al model 36, but also on the total volume received and/or the QC capacity. For example, if the QC team has a member who is on vacation then the QC team may only be able to handle 80 cases a day, and the operating point (threshold) is adjusted accordingly. [0040] Thus, in another nonlimiting illustrative example, the capacities of the QC team and the radiologists may be considered. For example, if for some day, the radiologist capacity is high but QC capacity is low, the planned number of cases to be selected for QC should be lowered so that cases are not stuck in the QC queue. In this situation, the Al threshold should be increased to reduce the number of cases selected by the Al model 36 for handling by the (lower capacity) QC team.

[0041] The Al model 36 can be developed (i.e., trained) to predict the probability of a study having missing data based on the ordering information, such as sending facility, modality, body parts, etc. A multi-layer perception (MLP) model can be used to take the following ordering information as input, “number of images”, “facility name”, “study day of week”, “hour of the day”, “contrast status (with or without)”, “modality”, “body parts”, “STAT or Routine”. The output is the probability of this case having missing data or not missing data. One-hot encoding can be applied on categorical input features. One of the implementations could be utilizing a fully- connected neural network comprising of multiple hidden layers, Rectified linear units (ReLU) non-linearity, softmax activation, binary cross-entropy loss function, and Adam optimizer. Alternatively, decision tree based machine learning algorithms (random forest, XGBoost, etc.) could also be used as the backbone model. The training data can be compiled, for example, as historical radiology examination datasets that passed the selection using the rules 30 and are labeled as to whether they are positive examples (i.e., were returned by the radiologist as incomplete and hence properly belonging to the second suspect set 38) or negative examples (i.e., were read by the radiologist and hence should not belong to the second suspect set 38). The Al model 36 is trained using such labeled training data to maximize its accuracy in classifying radiology examination datasets as to whether or not they should be assigned to the second suspect set 38.

[0042] The computer 16 is further programmed to annotate the radiology examination datasets of the first suspect set 32 and the second suspect set 38 with one or more completeness review request flags 40. In some embodiments, the radiology electronic processing device(s) 18 is configured to display, on the GUI 20, a list of at least a portion of the stored radiology examination datasets including displaying the completeness review request flags 40 annotated to the radiology examination datasets of the first suspect set 34 and the second suspect set 38. In this way, the radiologist is alerted that a radiology examination dataset is suspect and likely to be incomplete, and the radiologist can therefore skip that examination until it passes QC review. In another approach, the radiology electronic processing device(s) 18 is configured to receive only those radiology examination datasets 13 not annotated with the completeness review request flags 40. In this approach the radiologists do not see the suspect examinations at all until after they undergo QC review. In another alternative embodiment, computer 16 is configured to only send radiology examination datasets 13 not annotated with completeness review any request flags 40. In a further alternative embodiment and in light of modern PACs systems having systems for workload assignment through individual radiologist worklists (a list of radiology examinations to be read by a radiologist), radiology electronic processing device(s) 18 receives, but does not add or does not show flagged radiology examination datasets on a radiologist worklist so that a radiologist does not take up a radiology examination which may have missing information. In a further embodiment, if radiologist worklists are created or assigned at or by computer 16, flagged radiology examination datasets may be withheld or not added to a radiologist worklist at or by computer 16. Similar processing can apply at the QC review electronic processing devices 22. In one embodiment, the QC review electronic processing device(s) 22 is configured to display, on the GUI 24, a list of at least a portion of the stored radiology examination datasets (i.e., a queue of radiology examinations for reading) including displaying the completeness review request flags 40 linked to the radiology examination datasets of the first suspect set 34 and the second suspect set 38. For example, the QC review electronic processing device(s) 22 is configured to receive only those radiology examination datasets annotated with the completeness review request flags 40

[0043] The computer 16 is programmed to perform a method 100 of processing radiology examination datasets. With continuing reference to FIGURE 1 and further reference to FIGURE 2, an illustrative embodiment of the method 100 is diagrammatically shown as a sequence chart showing operations of the method 100 as a flowchart. In some examples, the method 100 may be performed at least in part by cloud processing.

[0044] To begin the method 100, the images are acquired of a patient using the medical imaging devices 14 and the images along with other information and/or metadata are stored in the PACSs 12 as radiology examination datasets 13, usually in DICOM format. At an operation 102, radiology examination datasets 13, comprising the images 13 and other information are transmitted to, and received by, the computer 16. The operation 102 may include adding additional information to the radiology examination datasets 13 that is not stored in the local PACS 12, such as patient data (e.g., age, gender, ethnicity, chronic medical conditions, etc.), past radiology examinations (possibly including the entire DICOM data for the past imaging examination, or possibly only the radiology report), and/or other relevant information for the radiology reading such as the patient’s current treatment regimen. Such information may not be available at the teleradiology service, and hence is added to the radiology examination dataset 13 before it is transmitted to the teleradiology PACS hosted at the computer 16.

[0045] At an operation 104, the computer 16 is configured to classify each radiology examination dataset as requiring completeness review or as ready for reading. To do so, the computer 16 is programmed to apply the rule 30 to the radiology examination datasets to select (i) the first suspect set 32 of radiology examination datasets for completeness review and (ii) the remaining set 34 of radiology examination datasets not selected for completeness review by the rule 30. The computer 16 then applies the model 36 to the remaining set 34 of radiology examination datasets to select the second suspect set 38 of radiology examination datasets for completeness review. A radiology examination dataset 13 is classified as requiring completeness review if it belongs to the first suspect set 32 or the second suspect set 38, and is classified as ready for reading if it does not belong to the first suspect set 32 and does not belong to the second suspect set 38 (i.e., it belongs in the remaining set 34 of radiology examination datasets not selected for completeness review).

[0046] At an operation 106, the computer 16 is configured to provide, on the QC review electronic processing device(s) 22, a QC GUI 24 for performing completeness review of the radiology examination datasets classified as requiring completeness review (i.e., those radiology examination datasets belonging to the first suspect set 32 or second suspect set 38). To do so, the computer 16 is configured to provide the QC GUI 24 with those radiology examination datasets classified as requiring completeness review, and not provide the QC GUI 24 with those radiology examination datasets classified as ready for reading. On the other hand, those radiology reports that are classified as ready for reading are sent directly to the radiologists for reading at the radiology workstations 18. Furthermore, the suspect radiology examination datasets that pass QC (either as-is or after being sent back to the customer to provide the completing or missing information) are also sent to the radiologists for reading at the radiology workstations 18.

[0047] FIGURE 3 shows another embodiment of the method 100, which could be implemented with the system of FIGURE 1 or with another suitable system. At an operation 202, one or more cases (i.e., the radiology examination datasets 13) are retrieved. At an operation 204, the rules 30 (or rule-based criteria) are determined (e.g., for certain facilities or modalities) to apply to the accessed cases. At an operation 206, cases are selected according to the determined rule 30 for the completeness QC review. At an operation 208, the Al model 36 is applied to the unselected cases based on a remaining QC capacity. FIGURE 3 also shows an operation 207 in which the Al model 36 is trained on one or more inputs, including, in one embodiment, data related to at least a modality of the medical imaging device(s) 12 used to acquire the images of patients included in the radiology examination datasets 13, the images themselves, facilities where the medical imaging devices 12 are located, a time of an imaging examination, a body art of the patient that is imaged, and so forth. At an operation 210, the QC is performed on the cases selected by the Al model 36.

[0048] FIGURE 4 shows a sequence chart of operations of the method 100, representing data flow between various modules. These modules include an imaging examination datasets storage module 300, which could for example be implemented by a PACS, a rules engine module 302 (e.g., implementing the rule 30 based selection of FIGURE 1), an Al engine module 304 (for example, implementing the Al model 36), a QC storage module 306 (which could, for example, be implemented by annotating the selected imaging examination datasets for QC review), and a QC review module 308 (for example, implemented as the QC review electronic processing device(s) 22 of FIGURE 1). The rules engine module 302 applies the rules 30, and is configured to receive radiological examination data (i.e., the radiology examination datasets 13). The rules engine module 302 can filter the received radiological examination data 13 to generate (i) radiological examination data 310 for the QC processing (corresponding to the radiological examination datasets 13 selected by the rules 30 of FIGURE 1) and (ii) radiological examination data 13 for input to the QC capacity Al processing module 304 for scoring by the Al model 36. To this end, the Al engine module 304 is configured to process the radiological examination data 13 and generate report features ingested by the Al model 36 to output the Al-selected radiological examination data 312 for QC processing. The combined radiological examination data 310 and 312 for QC review is stored in the QC store 306. QC personnel then utilize the QC review module 308 (e.g. the QC review electronic processing device(s) 22) to perform review of the selected examination data.

[0049] The illustrative embodiments of FIGURES 1-4 are in the context of a teleradiology service. However, it will be appreciated that the disclosed approaches for providing targeted, automatic routing of radiology examinations to either a radiology workflow support team or directly to a radiologist can be usefully employed in other settings. For example, a hospital may have a hospital radiology department with a staff of radiologists to handle all radiology examination readings in-house (that is, without sending the examinations to a teleradiology service). In such a case, the targeted, automatic routing of incoming radiology examinations can be used to route each examination uploaded to the hospital PACS to either the hospital’s radiology department workflow support team or directly to a radiologist on staff in the hospital’s radiology department. In this case, a complete radiology examination dataset may include hyperlinks to relevant information in the patient medical record, oncology department IT system, and/or so forth, and the quality control may include verifying these hyperlinks are present and valid.

[0050] With reference to FIGURE 5, another radiology information technology system is diagrammatically illustrated in accordance with another nonlimiting illustrative embodiment of the present disclosure. In this example, there is no teleradiology service; rather, the radiology reading is performed at the hospital, radiology lab, imaging center, or other imaging service provider that has at least one medical imaging device 14. The radiology examination datasets 13 are acquired by the medical imaging device 14 and are sent to an electronic processor such as an illustrative computer 116, which may for example be a server hosting a PACS system (or a serverside portion of a PACS system) of the hospital or radiology lab. In some implementations, the hospital or radiology lab PACS includes client-side components running on workstations or other suitable electronic processing devices 118 which are analogous to the workstations or other suitable electronic processing devices 18 of the embodiment of FIGURE 1, in which case, all or some functionality of computer 116 described in the following paragraphs may also be applied at the reading workstation 118 (e.g., the processing of functions can distributed over computer 116 and device 118 in any number of permutations), or alternatively, a hospital or radiology lab PACS may also be a standalone PACS (not shown) that is not implemented in a server/client architecture, in which case all functionality of computer 116 may be applied on such a standalone PACS where the PACS is not implemented in a server/client architecture. In other words, the functionality of computer 116 is only illustrated by way of example and may be applied by or on any form of PACS implementation at which the radiologist or other suitably trained medical professional is intended to perform the radiology examination reading with the goal of completeness review of a radiology examination dataset before the radiologist takes up review of the radiology examination dataset. The computer 116 is programmed to provide targeted, automatic routing of radiology examinations to either radiology workflow support (e.g., at review electronic processing device 122, which is analogous to the review electronic processing device 22 of FIGURE 1) or to a radiologist for reading (e.g., at the radiology workstation 118 for reading). The routing is performed as previously described with reference to FIGURE 1, e.g., by the computer 116 (e.g., a server, the workstation or other radiology electronic processing device 118, or a combination thereof, or a standalone PACS) applying the rule 30 to the radiology examination datasets 13 to select (i) a first suspect set 32 of radiology examination datasets for completeness review and (ii) a remaining set 34 of radiology examination datasets not selected for completeness review by the rule; applying the model 36 to the remaining set of radiology examination datasets to select a second suspect set 38 of radiology examination datasets for completeness review; and annotating the radiology examination datasets of the first suspect set and the second suspect set with a completeness review request flag 40. Radiology examination datasets 13 that are flagged with the completeness review request flag 40 are sent to the radiology workflow support (e.g., at review electronic processing device 122 in the hospital similar to device 22 exemplified in FIGURE 1); while, radiology examination datasets 13 that are not flagged with the completeness review request flag 40 are sent to a radiologist for reading, e.g., at the radiology workstation 118 for reading. The hospital or radiology lab PACS components 116 or 118, or the PACS in general as a standalone system, can be configured to withhold radiology examination datasets 13 that have been flagged for completeness review as described above in connection with teleradiology PACS (e.g., withholding or blocking the radiology examination datasets 13 from a radiologist worklist of assigned cases, withholding or blocking the radiology examination datasets 13 from being sent or received by a component of a PACS system, e.g., not sending radiology examination datasets 13 from computer 116 to radiology workstation 118, not receiving the radiology examination datasets by radiology workstation 118, or so forth).

[0051] With reference to FIGURE 6, yet another radiology information technology system is diagrammatically illustrated in accordance with yet another nonlimiting illustrative embodiment the present disclosure. This example is identical with that of FIGURE 5, except that the embodiment of FIGURE 6 does not include the review electronic processing device 122. The radiology examination datasets 13 are acquired by the medical imaging device 14 and are sent to the electronic processor, e.g., illustrative computer 116, which is programmed to provide targeted, automatic review of completeness of radiology examinations. This is done as previously described, by the computer 116 (e.g., a server, the workstation or other radiology electronic processing device 118, a combination thereof, or a standalone PACS) applying the rule 30 to the radiology examination datasets 13 to select (i) a first suspect set 32 of radiology examination datasets for completeness review and (ii) a remaining set 34 of radiology examination datasets not selected for completeness review by the rule; applying the model 36 to the remaining set of radiology examination datasets to select a second suspect set 38 of radiology examination datasets for completeness review; and annotating the radiology examination datasets of the first suspect set and the second suspect set with a completeness review request flag 40. In the embodiment of FIGURE 6, radiology examination datasets 13 that are flagged with the completeness review request flag 40 are not sent to the radiology workflow support (which is not included in this embodiment). However, when the radiologist retrieves the radiology examination for reading, the completeness review request flag 40 alerts the radiologist that the examination may be incomplete. This can improve efficiency of the radiologist by alerting as to likely incompleteness of an examination so that the radiologist can avoid the examination or first review the examination to see if anything is missing, and take appropriate action to remedy any incomplete information before commencing with reading of the radiology examination. In a variant (and not necessarily mutually exclusive) embodiment, the worklist which lists at least a portion of the queued radiology examination datasets for reading may include suitable flag markers indicating the completeness review request flag 40 next to (or otherwise associated with) each listed radiology examination dataset that is flagged with the completeness review request flag 40. This again increases radiologist efficiency by providing early notification of possible incompleteness of the subject radiology examination datasets that are so flagged in the worklist. In this embodiment, the computer 116 may for example be a server hosting a PACS system (or a server-side portion of a PACS system) of the hospital or radiology lab. In some implementations, the hospital or radiology lab PACS includes client-side components running on workstations or other suitable electronic processing devices 118, in which case, all or some functionality of computer 116 described in the following paragraphs may also be applied at the reading workstation 118 (e.g., the processing of functions can distributed over computer 116 and device 118 in any number of permutations), or alternatively, a hospital or radiology lab PACS may also be a standalone PACS (not shown) that is not implemented in a server/client architecture, in which case all functionality of computer 116 may be applied on such a standalone PACS where the PACS is not implemented in a server/client architecture. In other words, the functionality of computer 16 is only illustrated by way of example and may be applied by or on any form of PACS implementation at which the radiologist or other suitably trained medical professional is intended to perform the radiology examination reading with the goal of completeness review of a radiology examination dataset before the radiologist takes up review of the radiology examination dataset.

[0052] A non-transitory storage medium includes any medium for storing or transmitting information in a form readable by a machine (e.g., a computer). For instance, a machine-readable medium includes read only memory ("ROM"), solid state drive (SSD), flash memory, or other electronic storage medium; a hard disk drive, RAID array, or other magnetic disk storage media; an optical disk or other optical storage media; or so forth.

[0053] The methods illustrated throughout the specification, including the functions or functionality executed by for example computer 16, reading workstation 18, review electronic processing device 22, computer 116, reading workstation 118, review electronic processing device 122, or so forth, may be implemented as instructions stored on a non-transitory storage medium and read and executed by a computer or other electronic processor. The illustrated methods (e.g., method 100 and all related steps) may apply to all contexts and embodiments described herein including the teleradiology embodiment illustrated in FIGURE 1 and the hospital or radiology lab PACS embodiment(s) illustrated in FIGURES 5-6.

[0054] The disclosure has been described with reference to the preferred embodiments. Modifications and alterations may occur to others upon reading and understanding the preceding detailed description. It is intended that the exemplary embodiment be construed as including all such modifications and alterations insofar as they come within the scope of the appended claims or the equivalents thereof.

Claims

CLAIMS:

1. A radiology system comprising: a computer (16, 116) programmed to: store radiology examination datasets produced by corresponding radiology examinations, the radiology examination datasets including radiology images (13); apply a rule (30) to the radiology examination datasets to select (i) a first suspect set (32) of radiology examination datasets for completeness review and (ii) a remaining set (34) of radiology examination datasets not selected for completeness review by the rule; apply a model (36) to the remaining set of radiology examination datasets to select a second suspect set (38) of radiology examination datasets for completeness review; and annotate the radiology examination datasets of the first suspect set and the second suspect set with a completeness review request flag (40).

2. The system of claim 1, further comprising: at least one radiology electronic processing device (18, 118) configured to provide a user interface (20) for performing radiology examination readings, each radiology electronic processing device programmed to display radiology images of a radiology examination dataset undergoing reading and receive a radiology report on the radiology examination dataset undergoing reading; wherein the at least one radiology electronic processing device is further configured to display a list of at least a portion of the radiology examination datasets including displaying the completeness review request flags (40) annotated to the radiology examination datasets of the first suspect set and the second suspect set.

3. The system of claim 1, further comprising: at least one radiology electronic processing device (18, 118) configured to provide a user interface (20) for performing radiology examination readings, each radiology electronic processing device programmed to display radiology images of a radiology examination dataset undergoing reading and receive a radiology report on the radiology examination dataset undergoing reading; wherein the at least one radiology electronic processing device receives only those radiology examination datasets not annotated with the completeness review request flags (40).

4. The system of either one of claims 1 and 2, further comprising: at least one quality control (QC) review electronic processing device (22, 122) configured to provide a user interface (24) for performing completeness reviews of radiology examination datasets, each QC review electronic processing device programmed to display a summary of the contents of a radiology examination dataset undergoing completeness review and receive a request for additional information to complete the radiology examination dataset undergoing completeness review; wherein the at least one QC review electronic processing device is further configured to display a list of at least a portion of the radiology examination datasets including displaying the completeness review request flags annotated to the radiology examination datasets of the first suspect set (32) and the second suspect set (38).

5. The system of either one of claims 1 and 2, further comprising: at least one quality control (QC) review electronic processing device (22, 122) configured to provide a user interface (24) for performing completeness reviews of radiology examination readings, each QC review electronic processing device programmed to display a summary of the contents of a radiology examination dataset undergoing completeness review and receive a request for additional information to complete the radiology examination dataset undergoing completeness review; wherein the at least one QC review electronic processing device receives only those radiology examination datasets annotated with the completeness review request flags (40).

6. The system of any one of claims 1-5, wherein the rule (30) selects a radiology examination dataset for completeness review based on one or more of: originating hospital or radiology department; reference to a prior radiology examination in the radiology examination dataset, imaging modality of the images of the radiology examination dataset; information on an intravascularly administered contrast agent in the radiology examination dataset; and/or a timestamp of the radiology examination dataset.

7. The system of any one of claims 1-5, wherein the rule (30) comprises one or more rules derived from a plurality of criteria including patterns in cases with missing information, a prevalence of cases with missing information from certain facilities and modalities, a distribution of types of missing information including prior images or reports, contrast agent details, ultrasound worksheets, and incomplete current images.

8. The system of any one of claims 1-7, wherein the model (36) comprises an artificial intelligence (Al) model configured to: receive, as inputs, data related to number of data related to at least a modality of a medical imaging device used to acquire images of patients who are the subjects of the radiology examination datasets, images of the patients, facilities where the medical imaging devices are located, a time of an imaging examination, and a body art of the patient that is imaged; and output a probability that one of more of the radiology examination datasets is missing information or not missing information.

9. The system of any one of claims 1-8, wherein the server computer (16, 116) is further programmed to: controlling the applying of the model (36) to the remaining set of radiology examination datasets to select the second suspect set (38) of radiology examination datasets for completeness review so that a total number of radiology examination datasets in the combined first and second suspect sets does not exceed a maximum number for completeness review.

10. The system of claim 9, wherein the server computer (16, 116) is further programmed to: selecting the maximum number for completeness review based on a current capacity to perform completeness reviews of radiology examination datasets.

11. A method (100) of processing radiology examination datasets, the method comprising: using an electronic processor (16, 116), classifying each radiology examination dataset as requiring completeness review or as ready for reading; providing a quality control user interface (24) for performing completeness review of the radiology examination datasets classified as requiring completeness review; for each reviewed radiology examination data set generating a request for additional information to complete the radiology examination dataset or re-classifying the radiology examination dataset as ready for reading; and providing a radiology reading user interface (20) for reading and preparing radiology reports on the radiology examination datasets ready for reading.

12. The method (100) of claim 11, wherein the classifying includes: applying a rule (30) to the radiology examination datasets to select (i) a first suspect set (32) of radiology examination datasets for completeness review and (ii) a remaining set (34) of radiology examination datasets not selected for completeness review by the rule; and applying a model (36) to the remaining set of radiology examination datasets to select a second suspect set (38) of radiology examination datasets for completeness review; wherein a radiology examination dataset is classified as requiring completeness review if it belongs to the first suspect set or the second suspect set, and is classified as ready for reading if it does not belong to the first suspect set and does not belong to the second suspect set.

13. The method (100) of claim 12, wherein the model (36) comprises an artificial intelligence (Al) model.

14. The method (100) of claim 13, wherein the Al model (36) comprises an artificial neural network or a decision tree or an ensemble model.

15. The method (100) of any one of claims 11-14, further comprising: providing the quality control user interface (24) with those radiology examination datasets classified as requiring completeness review; and not providing the quality control user interface with those radiology examination datasets classified as ready for reading.

16. A non-transitory storage medium storing instructions readable and executable by an electronic processor (16, 116) to perform a method of assessing radiology examination datasets produced by corresponding radiology examinations and including radiology images (13), the method including: applying a rule (30) to the radiology examination datasets to select (i) a first suspect set (32) of radiology examination datasets for completeness review and (ii) a remaining set (34) of radiology examination datasets not selected for completeness review by the rule; applying a model (36) to the remaining set of radiology examination datasets to select a second suspect set (38) of radiology examination datasets for completeness review; and annotating the radiology examination datasets of the first suspect set and the second suspect set with a completeness review request flag (40).

17. The non-transitory storage medium of claim 16, wherein the rule (30) selects a radiology examination dataset for completeness review based on one or more of: originating hospital or radiology department; reference to a prior radiology examination in the radiology examination dataset, imaging modality of the images of the radiology examination dataset; information on an intravascularly administered contrast agent in the radiology examination dataset; and/or a timestamp of the radiology examination dataset.

18. The non-transitory storage medium of claim 16, wherein the method further includes: deriving the rule (30) from a plurality of criteria including patterns in cases with missing information, a prevalence of cases with missing information from certain facilities and modalities, a distribution of types of missing information including prior images or reports, contrast agent details, ultrasound worksheets, and incomplete current images.

19. The non-transitory storage medium of any one of claims 16-18, wherein the model (36) comprises an artificial intelligence (Al) model configured to: receive, as inputs, data related to number of data related to at least a modality of a medical imaging device used to acquire images of patients who are the subjects of the radiology examination datasets, images of the patients, facilities where the medical imaging devices are located, a time of an imaging examination, and a body art of the patient that is imaged; and output a probability that one of more of the radiology examination datasets is missing information or not missing information.

20. The non-transitory storage medium of any one of claims 16-19, wherein the method further includes: controlling the applying of the model (36) to the remaining set of radiology examination datasets to select the second suspect set (38) of radiology examination datasets for completeness review so that a total number of radiology examination datasets in the combined first and second suspect sets does not exceed a maximum number for completeness review.