US20200365262A1 - Self-correcting method for annotation of data pool using feedback mechanism - Google Patents
Self-correcting method for annotation of data pool using feedback mechanism Download PDFInfo
- Publication number
- US20200365262A1 US20200365262A1 US16/986,360 US202016986360A US2020365262A1 US 20200365262 A1 US20200365262 A1 US 20200365262A1 US 202016986360 A US202016986360 A US 202016986360A US 2020365262 A1 US2020365262 A1 US 2020365262A1
- Authority
- US
- United States
- Prior art keywords
- flowchart
- medical imaging
- imaging device
- representation
- image
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 77
- 230000008713 feedback mechanism Effects 0.000 title 1
- 238000002059 diagnostic imaging Methods 0.000 claims description 107
- 238000012423 maintenance Methods 0.000 claims description 90
- 238000012806 monitoring device Methods 0.000 claims description 23
- 238000012544 monitoring process Methods 0.000 claims description 19
- 230000006870 function Effects 0.000 claims description 17
- 238000012015 optical character recognition Methods 0.000 claims description 14
- 230000004044 response Effects 0.000 claims description 10
- 238000004422 calculation algorithm Methods 0.000 claims description 5
- 230000036962 time dependent Effects 0.000 claims 1
- 230000008569 process Effects 0.000 description 38
- 238000002600 positron emission tomography Methods 0.000 description 19
- 238000002591 computed tomography Methods 0.000 description 17
- 238000013459 approach Methods 0.000 description 13
- 238000002955 isolation Methods 0.000 description 12
- 230000008901 benefit Effects 0.000 description 11
- 230000000875 corresponding effect Effects 0.000 description 10
- 238000001514 detection method Methods 0.000 description 10
- 238000003384 imaging method Methods 0.000 description 10
- 239000010432 diamond Substances 0.000 description 8
- 238000013507 mapping Methods 0.000 description 7
- 101100080098 Caenorhabditis elegans nlp-30 gene Proteins 0.000 description 6
- 230000009471 action Effects 0.000 description 6
- 238000012545 processing Methods 0.000 description 6
- 238000012360 testing method Methods 0.000 description 6
- 238000004458 analytical method Methods 0.000 description 5
- 229910003460 diamond Inorganic materials 0.000 description 5
- 238000003058 natural language processing Methods 0.000 description 5
- 238000006243 chemical reaction Methods 0.000 description 4
- 230000001419 dependent effect Effects 0.000 description 4
- 238000000605 extraction Methods 0.000 description 4
- 230000003287 optical effect Effects 0.000 description 4
- 230000002123 temporal effect Effects 0.000 description 4
- 230000001960 triggered effect Effects 0.000 description 4
- 238000002372 labelling Methods 0.000 description 3
- 230000000670 limiting effect Effects 0.000 description 3
- 230000006978 adaptation Effects 0.000 description 2
- 230000004075 alteration Effects 0.000 description 2
- 230000015556 catabolic process Effects 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 230000003750 conditioning effect Effects 0.000 description 2
- 238000012790 confirmation Methods 0.000 description 2
- 238000012937 correction Methods 0.000 description 2
- 230000003247 decreasing effect Effects 0.000 description 2
- 230000006872 improvement Effects 0.000 description 2
- 230000007246 mechanism Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000002829 reductive effect Effects 0.000 description 2
- 238000005067 remediation Methods 0.000 description 2
- 238000012552 review Methods 0.000 description 2
- 238000013515 script Methods 0.000 description 2
- 238000002603 single-photon emission computed tomography Methods 0.000 description 2
- 238000002560 therapeutic procedure Methods 0.000 description 2
- 238000012879 PET imaging Methods 0.000 description 1
- 230000003044 adaptive effect Effects 0.000 description 1
- 230000002776 aggregation Effects 0.000 description 1
- 238000004220 aggregation Methods 0.000 description 1
- 230000006399 behavior Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 230000000295 complement effect Effects 0.000 description 1
- 239000002131 composite material Substances 0.000 description 1
- 238000013170 computed tomography imaging Methods 0.000 description 1
- 230000001276 controlling effect Effects 0.000 description 1
- 230000002596 correlated effect Effects 0.000 description 1
- 230000001934 delay Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000003745 diagnosis Methods 0.000 description 1
- 238000002405 diagnostic procedure Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000008030 elimination Effects 0.000 description 1
- 238000003379 elimination reaction Methods 0.000 description 1
- 238000001802 infusion Methods 0.000 description 1
- 238000007689 inspection Methods 0.000 description 1
- 238000009434 installation Methods 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 230000003137 locomotive effect Effects 0.000 description 1
- 230000007774 longterm Effects 0.000 description 1
- 238000002595 magnetic resonance imaging Methods 0.000 description 1
- 239000003550 marker Substances 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 230000008520 organization Effects 0.000 description 1
- 238000007781 pre-processing Methods 0.000 description 1
- 238000003908 quality control method Methods 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 238000011895 specific detection Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 208000024891 symptom Diseases 0.000 description 1
- 230000007704 transition Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G16H40/00—ICT specially adapted for the management or administration of healthcare resources or facilities; ICT specially adapted for the management or operation of medical equipment or devices
- G16H40/40—ICT specially adapted for the management or administration of healthcare resources or facilities; ICT specially adapted for the management or operation of medical equipment or devices for the management of medical equipment or devices, e.g. scheduling maintenance or upgrades
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/20—Natural language analysis
- G06F40/205—Parsing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/20—Natural language analysis
- G06F40/205—Parsing
- G06F40/211—Syntactic parsing, e.g. based on context-free grammar [CFG] or unification grammars
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N5/00—Computing arrangements using knowledge-based models
- G06N5/02—Knowledge representation; Symbolic representation
- G06N5/022—Knowledge engineering; Knowledge acquisition
- G06N5/025—Extracting rules from data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N5/00—Computing arrangements using knowledge-based models
- G06N5/04—Inference or reasoning models
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G16H30/00—ICT specially adapted for the handling or processing of medical images
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B6/00—Apparatus or devices for radiation diagnosis; Apparatus or devices for radiation diagnosis combined with radiation therapy equipment
- A61B6/02—Arrangements for diagnosis sequentially in different planes; Stereoscopic radiation diagnosis
- A61B6/03—Computed tomography [CT]
- A61B6/032—Transmission computed tomography [CT]
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B6/00—Apparatus or devices for radiation diagnosis; Apparatus or devices for radiation diagnosis combined with radiation therapy equipment
- A61B6/02—Arrangements for diagnosis sequentially in different planes; Stereoscopic radiation diagnosis
- A61B6/03—Computed tomography [CT]
- A61B6/037—Emission tomography
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B6/00—Apparatus or devices for radiation diagnosis; Apparatus or devices for radiation diagnosis combined with radiation therapy equipment
- A61B6/10—Safety means specially adapted therefor
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B6/00—Apparatus or devices for radiation diagnosis; Apparatus or devices for radiation diagnosis combined with radiation therapy equipment
- A61B6/52—Devices using data or image processing specially adapted for radiation diagnosis
- A61B6/5211—Devices using data or image processing specially adapted for radiation diagnosis involving processing of medical diagnostic data
- A61B6/5229—Devices using data or image processing specially adapted for radiation diagnosis involving processing of medical diagnostic data combining image data of a patient, e.g. combining a functional image with an anatomical image
- A61B6/5235—Devices using data or image processing specially adapted for radiation diagnosis involving processing of medical diagnostic data combining image data of a patient, e.g. combining a functional image with an anatomical image combining images from the same or different ionising radiation imaging techniques, e.g. PET and CT
-
- H—ELECTRICITY
- H05—ELECTRIC TECHNIQUES NOT OTHERWISE PROVIDED FOR
- H05G—X-RAY TECHNIQUE
- H05G1/00—X-ray apparatus involving X-ray tubes; Circuits therefor
- H05G1/08—Electrical details
- H05G1/26—Measuring, controlling or protecting
- H05G1/54—Protecting or lifetime prediction
Definitions
- the following relates generally to the medical imaging device maintenance arts, device monitoring arts, predictive maintenance arts, process flow automation arts, and related arts.
- Medical devices used in the healthcare industry such as magnetic resonance imaging (MRI) scanners, computed tomography (CT) scanners, positron emission tomography (PET) scanners, gamma cameras used in single photon emission computed tomography (SPECT), image-guided therapy (iGT) systems, and other medical imaging devices, or electrocardiograph (ECG) or patient monitor devices, and so forth, should be in good working condition to ensure doctors and patients receive correct information for medical diagnoses and patient monitoring and so forth.
- Medical imaging devices are expensive to replace and play a crucial role in diagnosis. Any downtime of these devices results in a loss of revenue to the medical institution, loss of quality treatment for patients, and introduces delays into patient treatment. Emphasis is thus placed on minimizing the downtime of medical imaging devices and ensuring uninterrupted operational status while maintaining quality of performance.
- Predictive maintenance is an important part of minimizing downtime of medical imaging devices. This approach entails predicting and proactively repairing or otherwise remediating possible failures of medical imaging device components in advance based on the machine logs and usage history. In this way, maintenance can be proactively performed to minimize or eliminate downtime and impact on patient care.
- a further problem is that the resulting predictive maintenance system is static. This is a problem because medical imaging device manufacturers are continually adding new products and improving existing product lines. These changes will not be reflected in the knowledge engine used for predictive maintenance. Consequently, manual updating of the knowledge engine must be performed on a frequent basis, which again requires extensive input from subject matter experts.
- FSEs field service engineers
- FSEs field service engineers
- FSEs can refer to printed manuals or e-books, which have fault isolation process flows.
- FSEs need to carry these manuals, then search for the appropriate content to follow the process flows in the form of a flow chart.
- FSEs follow the steps of the flow chart to perform fault isolation of the imaging system.
- the imaging system printed manuals can be in text form or PDF forms. These process flow diagrams are present as images within PDF documents. While the image format may allow for the FSE to zoom in on specific portions of the flow chart, difficulties can still arise, for example if the portion of the flow chart being traversed crosses onto another image/page.
- a non-transitory storage medium stores instructions readable and executable by an electronic processor to perform a monitoring method for generating maintenance alerts.
- the monitoring method includes: extracting component identifications (component IDs) identifying medical imaging device components and operating parameters of the identified medical imaging device components and associated operating parameter ranges from electronic medical imaging device manuals; generating a knowledge engine by operations including formulating the operating parameter ranges into a set of decision rules for classifying medical imaging device log data; and applying the knowledge engine to log data generated by a monitored medical imaging device to detect out-of-range log data generated by the monitored medical imaging device and to generate maintenance alerts in response to the detected out-of-range log data wherein the maintenance alerts are associated with component IDs contained in the detected out-of-range log data.
- a monitoring device for generating maintenance alerts.
- the monitoring device comprises a display, a non-transitory storage medium as set forth in the immediately preceding paragraph, and an electronic processor operatively connected with the display and with the non-transitory storage medium to perform the monitoring method further including displaying the generated maintenance alerts on the display.
- a monitoring method performed by an electronic processor for generating maintenance alerts.
- Component identifications are extracted which identify medical imaging device components in electronic medical imaging device manuals.
- Operating parameters of the identified medical imaging device components and associated operating parameter ranges are extracted from the electronic medical imaging device manuals based on numeric values, parameter terms identifying operating parameters, and linking terms or symbols indicative of equality or inequality that connect the numeric values and parameter terms.
- the operating parameter ranges are formulated into decision rules for classifying medical imaging device log data based on whether a value of the associated operating parameter is outside of the operating parameter range.
- the decision rules are applied to log data generated by a monitored medical imaging device to detect out-of-range log data generated by the monitored medical imaging device, and maintenance alerts are displayed on a display in response to the detected out-of-range log data.
- the maintenance alerts are generated from out-of-range log data and are associated with component IDs contained in the out-of-range log data.
- a monitoring device for generating maintenance alerts.
- the monitoring device includes a display, an electronic processor operatively connected with the display, and a non-transitory storage medium storing instructions readable and executable by the electronic processor to perform a monitoring method for generating maintenance alerts.
- decision rules are generated for classifying medical imaging device log data based on whether parameter values of operating parameters of medical imaging device components identified by component identifications (component IDs) in the log data are within operating parameter ranges for the respective operating parameters.
- the operating parameter ranges for the respective operating parameters are extracted from electronic medical imaging device manuals.
- the decision rules are applied to log data generated by a monitored medical imaging device to detect out-of-range log data generated by the monitored medical imaging device.
- the display is controlled to present maintenance alerts in response to the detected out-of-range log data wherein the maintenance alerts are associated with component IDs contained in the detected out-of-range log data.
- a non-transitory computer readable medium stores instructions executable by at least one electronic processor to perform a method of converting a flowchart to a structured electronic representation of the flowchart.
- the method includes: identifying, in an image of the flowchart, a plurality of shapes corresponding to flowchart blocks of the flowchart in the image; identifying arrows defining flow paths between the flowchart blocks in the image including identifying flowchart blocks connected by the arrows and identifying directionality of the defined flow paths based on arrowheads of the arrows; identifying text labels and their locations in the image and performing optical character recognition (OCR) on the image of the flowchart to determine text content of the identified text labels; associating the text labels with flowchart blocks or defined flow paths based on locations of the text labels, flowchart blocks, and arrows in the image; and generating the structured electronic representation of the flowchart based on the flowchart blocks, the flow paths, and the text labels.
- a non-transitory computer readable medium stores instructions executable by at least one electronic processor to perform a method of converting flowcharts to structured electronic representations.
- the method includes: identifying, in an image of a flowchart, a plurality of shapes corresponding to flowchart blocks of the flowcharts in the image; identifying a location and directionality of a plurality of arrows defining flow paths between the flowchart blocks in the image; generating a directed graph of the flowchart blocks; and converting the directed graph to a structured dialog flow representation.
- One advantage resides in providing a monitoring device for generating maintenance alerts for one or more monitored medical imaging devices, in which the knowledge engine of the monitoring device is generated automatically.
- Another advantage resides in providing such a monitoring device in which the knowledge base is continuously and efficiently updated based on service logs of the monitored medical imaging devices.
- Another advantage resides in providing such a monitoring device with computationally efficient generation and updating of the knowledge base.
- Another advantage resides in providing such a monitoring device in which the knowledge base is developed without input from subject matter experts.
- Another advantage resides in digitizing paper manuals for imaging systems.
- Another advantage resides in reducing errors or loss of process steps when following a process flow chart for servicing an imaging system.
- Another advantage resides in providing an interactive electronic flow chart in which a chatbot guides the user through the steps of the flow chart based on feedback provided to the chatbot by the user.
- Another advantage resides in providing digitized instructions for an FSE to provide service to an imaging system.
- a given embodiment may provide none, one, two, more, or all of the foregoing advantages, and/or may provide other advantages as will become apparent to one of ordinary skill in the art upon reading and understanding the present disclosure.
- the invention may take form in various components and arrangements of components, and in various steps and arrangements of steps.
- the drawings are only for purposes of illustrating the preferred embodiments and are not to be construed as limiting the invention.
- FIG. 1 diagrammatically illustrates a monitoring device for performing predictive maintenance including generating maintenance alerts for a monitored medical imaging device based on log data generated by the monitored medical imaging device, in which decision rules for generating the maintenance alerts are automatically generating.
- FIG. 2 diagrammatically illustrates a portion of a medical imaging device manual.
- FIG. 3 diagrammatically illustrates a table of components and key metrics derived from the manual portion shown in FIG. 2 .
- FIGS. 4, 5, and 6 diagrammatically illustrate aspects of generating the decision rules from medical imaging device manual content as described herein.
- FIGS. 7 and 8 diagrammatically illustrate aspects of mapping of medical imaging device log data to decision rules of the knowledge engine as described herein.
- FIG. 9 diagrammatically illustrates a method suitably performed by the monitoring device of FIG. 1 to construct decision rules of the knowledge engine.
- FIGS. 10 and 11 diagrammatically illustrate aspects of the reader module component of the method of FIG. 9 .
- FIG. 12 illustrates a system for converting a digital image of a flowchart to a structured electronic representation of the flowchart.
- FIG. 13 illustrates an exemplary digital image of a flowchart for conversion by the system of FIG. 12 .
- FIG. 14 illustrates a directed graph produced from the flowchart image of FIG. 13 .
- FIG. 15 illustrates an exemplary flow chart implemented by the system of FIG. 12 .
- electronic medical imaging device manuals are leveraged to extract component identifications (component IDs) identifying medical imaging device components and operating parameters of the identified medical imaging device components and associated operating parameter ranges.
- the electronic medical imaging device manuals can take substantially any machine-readable form, and may be online manuals, printed manuals that are scanned to pdf (or another electronic format) and processed by optical character recognition (OCR) to generate machine-readable text, and/or so forth.
- the electronic medical imaging device manuals may, for example, include one or more of a service manual, manufacturer specification(s), a user manual, an operating reference manual, and/or so forth.
- a knowledge engine is then generated by operations including formulating the operating parameter ranges into a set of decision rules for classifying medical imaging device log data.
- a decision rule may classify input log data as bad (i.e. out-of-range) log data if the input log data indicates a value for an operating parameter that is outside of the operating parameter range associated with that operating parameter.
- the knowledge engine is then applied to log data generated by a monitored medical imaging device to detect out-of-range log data generated by the monitored medical imaging device and to generate maintenance alerts in response to the detected out-of-range log data.
- the maintenance alerts are suitably associated with component IDs contained in the detected out-of-range log data, and may in some embodiments be formulated as maintenance recommendations.
- the alert may be formulated as “ ⁇ Component ID> operating temperature may be above its recommended operating temperature. Recommend to check temperature of ⁇ Component ID>.”
- accuracy of the generated maintenance alerts may be determined based on feedback extracted from a service log of the monitored medical imaging device, and the knowledge engine may be updated by adjusting the decision rules based on the determined accuracy of the generated maintenance alerts. For example, accuracy of a generated maintenance alert may be determined based on feedback extracted from the service log over a time interval extending from a time of generation of the maintenance alert to an end time which is dependent upon (and may be equal to) the mean time to failure (MTTF) of the medical imaging device component identified by the component ID associated with the maintenance alert.
- MTTF mean time to failure
- a field service engineer works through a fault isolation flowchart to identify a root cause and/or solution to the problem underlying the maintenance alert.
- the FSE conventionally consults a service manual when servicing diagnostic equipment.
- fault isolation flowcharts which provide step-by-step guidance for performing tests and inspections in order to determine a root cause or solution of a problem.
- Such manuals can cover existing and legacy diagnostic equipment. These are presented in paper form, or in non-machine readable image (e.g. JPEG) or PDF format, e.g., as images embedded in a PDF, ePub, or other electronic book (ebook) or digital document format.
- an approach to convert such paper flowcharts to a structured electronic form is provided.
- an image e.g., JPEG, BMP, PNG, or so forth
- Shapes with corresponding location anchors in the image are identified, which correspond to flowchart blocks.
- Optical character recognition (OCR) is applied to extract text with corresponding location anchors in the image.
- Connecting arrows are identified and the directionality of the arrows (e.g., based on the direction the arrowhead is pointing) is used to define flow paths between the blocks, and a directed graph representation of the blocks is generated.
- a function of each block may be determined.
- OCR'd text anchored inside the block or near a flow arrow can be processed by keyword detection, natural language processing (NLP), intent detection, or the like in order to extract the function information.
- NLP natural language processing
- This enables semantic labeling of the blocks and transitions of the directed graph.
- the directed graph may be displayed at this point for review and editing (if appropriate) by a domain expert.
- JSON JavaScript Object Notation
- XML extensible markup language
- the foregoing operations are typically performed offline via a server computer or the like to create the directed graph and JSON representations. These (or at least the JSON representation) are then made accessible to FSEs for use during service calls. This can be done by downloading an application program (“app”) to a tablet computer, cellphone, notebook computer, or other mobile device used by the FSE, in which the app includes a database of converted fault isolation flowcharts.
- the FSE can access the JSON (and optionally the directed graph) representation over the Internet via a website hosted at a remote server using a web browser or dedicated app.
- the app may provide the FSE with the option to download only selected flowcharts.
- the JSON representation is used to drive a chatbot that guides the FSE through the diagnostic process.
- the chatbot will present the text of each box, either on-screen (similar to an instant messaging chatbot), and/or using text to voice, and/or using a graphical user interface (GUI) chatbot avatar.
- GUI graphical user interface
- the FSE inputs any information needed to proceed (e.g., answer as to whether a performed test was positive or negative), and the chatbot proceeds accordingly through the JSON representation of the directed graph representing the fault isolation flowchart.
- a monitoring device for generating maintenance alerts for a medical imaging device 2 .
- the illustrative medical imaging device 2 is a PET/CT scanner including a positron emission tomography (PET) gantry 4 and a transmission computed tomography (CT) gantry 6 positioned in-line such that a patient can be loaded into either gantry 4 , 6 via a common robotic patient support or couch 8 .
- PET/CT scanner 2 is commonly used in numerous diagnostic and clinical tasks (e.g. oncology, cardiology, neurology, and so forth) as the CT imaging performed by the CT gantry 6 provides anatomical information that is complementary to functional information provided by PET imaging performed by the EPT gantry 4 .
- a CT image can be used to generate an attenuation map that is used to perform attenuation correction during PET image reconstruction.
- the PET/CT scanner 2 is a highly complex medical imaging device with thousands of components, e.g. as a very few non-limiting examples including dozens or hundreds of PET detector modules, an X-ray tube and dozens or hundreds of X-ray detector modules (components of the CT gantry 6 ), mechanical components for revolving the X-ray tube and X-ray detector array around the patient, numerous robotic mechanisms of the patient support or couch 8 , electrical power components such as a distribution transformer for delivering electrical power to the PET/CT 2 , and so forth (note, detailed medical imaging device components are not illustrated in FIG.
- Each component typically has one, two, three, or more operating parameters that are monitored by appropriate sensors.
- each PET detector module may include a temperature sensor that outputs module operating temperature, electrical sensors that monitor dark current and/or other electrical operating parameters, and/or so forth.
- a robotic component of the couch 8 may include position encoders that monitor positioning of a robotic actuator.
- the CT gantry may include sensors monitoring rotation speed and encoding rotational angular position.
- the distribution transformer may include sensors monitoring operating power factor, input and output voltage, and so forth. Again, these are merely a few non-limiting illustrative examples.
- the illustrative monitored medical imaging device 2 is a PET/CT scanner
- the monitored medical imaging device may more generally be any type of medical imaging device with sufficient complexity to justify continuous monitoring and generation of timely maintenance alerts as disclosed herein.
- the monitored medical imaging device may be a PET/CT scanner, a SPECT/CT scanner, a standalone gamma camera, standalone PET scanner, standalone CT scanner, an magnetic resonance (MR) imaging scanner, a PET/MR scanner, an image-guided therapy (iGT) device, or so forth.
- MR magnetic resonance
- iGT image-guided therapy
- the disclosed predictive maintenance monitoring devices and methods may be applied to monitor many such medical imaging devices, possibly of different types, in parallel.
- the illustrative monitoring device includes a server computer 10 or other electronic processor (e.g. desktop computer with sufficient computing capacity) operatively connected with a non-transitory storage medium 12 that stores instructions that are readable and executable by the electronic processor 10 to perform a monitoring method as disclosed herein, and a monitor interface device 14 which, in the illustrative embodiment, is embodied by a computer 16 that serves as a medical imaging device controller 14 for controlling the monitored medical imaging device 2 (e.g., to operate the robotic support 8 to load a patient into the appropriate gantry, to select and execute an imaging sequence or protocol, to perform reconstruction of the acquired imaging data to generate an image and to display the image on a display 18 of the computer, and/or so forth).
- a server computer 10 or other electronic processor e.g. desktop computer with sufficient computing capacity
- a monitor interface device 14 which, in the illustrative embodiment, is embodied by a computer 16 that serves as a medical imaging device controller 14 for controlling the monitored medical imaging device 2 (e.g
- the monitor interface device may be a separate computer, or may be integral with the computer 10 that performs the monitoring method.
- the monitor interface device includes the display 18 for presenting maintenance alerts or other information, and may optionally include one or more user input devices (e.g. illustrative keyboard 19 ) for receiving user inputs.
- the non-transitory storage medium 12 may be variously embodied, e.g. as a hard disk drive, RAID array, or other magnetic storage medium, a solid state drive (SSD) or other electronic storage medium, an optical disk or other optical storage medium, various combinations thereof, and/or so forth.
- the illustrative electronic processors 10 , 16 may be otherwise variously embodied and/or combined, and/or the various non-transitory storage media may be variously embodied, e.g. by way of linkages via electronic data networks or the like.
- the electronic processor 10 may be implemented as a cloud computing resource comprising an ad hoc combination of a number of server computers.
- the electronic processor 10 reads and executes instructions stored on the non-transitory storage medium 12 in order to perform a monitoring method including implementing a knowledge engine builder 20 that builds, and optionally subsequently adaptively maintains or updates, a knowledge engine 22 comprising a set of decision rules for generating maintenance alerts 24 .
- the knowledge engine builder 20 leverages inputs from available electronic medical imaging device operating data.
- a commonly available form of electronic medical imaging device servicing and operating data are electronic manuals, such as an illustrative service manual 25 , an illustrative set of manufacturer's specifications 26 , an illustrative user manual 27 , and an illustrative operating reference manual 28 .
- the manual content may be variously distributed amongst one or more such manuals, e.g. in some implementations a single manual may cover the combined content of the illustrative user manual 27 and operating reference manual 28 , and/or the service manual may include the manufacturer's specifications as one or more appendices of the service manual, rather than as a separate manufacturer's specifications document, and/or so forth.
- the electronic medical imaging device servicing and operating data may also include service notes compiled into electronic form (e.g. electronic service logs maintained by service engineers), updates to the manuals (e.g.
- the various electronic medical imaging device servicing and operating data 25 , 26 , 27 , 28 are preferably specific to the make and model of the medical imaging device 2 to be monitored, although such a requirement may be relaxed in instances in which different makes and/or models of a particular type of medical imaging device share certain systems or sub-systems.
- the knowledge engine builder 20 processes the content of the electronic medical imaging device servicing and operating data 25 , 26 , 27 , 28 to extract component identifications (component IDs) identifying medical imaging device components, and to extract operating parameters of the identified components, and to extract operating parameter ranges associated with the respective operating parameters.
- the manuals are assumed to be in electronic form with machine readable text. (If this is not initially the case, then optical scanning, photocopying, photography, or the like can be employed to generate digital images of the manual pages, followed by OCR, to covert a paper manual into electronic format with machine-readable text).
- Natural language processing (NLP) 30 is performed on text of the electronic medical imaging device servicing and operating data 25 , 26 , 27 , 28 to tokenize the text into individual tokens (e.g.
- the NLP 30 may include parsing of text and tables of the electronic medical imaging device manuals to delineate semantic units including sentences, paragraphs, list items, table rows, or so forth.
- Component entity recognition (CER) 32 is applied to identify component IDs in the NLP-processed text that identify medical imaging device components.
- the CER 32 may employ any type of named entity recognition (NER) algorithm or combination of NER algorithms, e.g. leveraging a domain-specific vocabulary list to identify component IDs, identifying component IDs based on factors such as part-of-speech (if the NLP 30 includes grammatical parsing), and/or so forth.
- NER named entity recognition
- NER named entity recognition
- the CER 32 also performs operating parameters extraction to extract operating parameters of the medical imaging device components identified by component IDs, along with associated operating parameter ranges.
- operating parameters are identified by identifying numeric values and parameter terms which are connected by linking terms or symbols indicative of equality or inequality, and associating parameters identified by the parameter terms with medical imaging device components whose component IDs occur in the same semantic units as the parameter terms.
- the sentence: “The PET detector modules should be kept at a temperature below 100° C.” can be processed by identifying the numeric value (100° C.) and a parameter term (temperature) which are connected by a linking term or symbol indicative of equality or inequality (below), and the operating parameter “temperature” is associated with the component ID (PET detector) occurring in the same semantic unit (same sentence) as the parameter term (temperature).
- Parameter terms can be identified using NER to identify “named entities” representing operating parameters, e.g. using a domain-specific vocabulary list, part-of-speech (if grammatical parsing if available), and/or so forth.
- machine logs 40 which are commonly generated by medical imaging devices.
- the machine logs are generated by the medical imaging device controller 14 based on readings of the various sensors of the components, sub-systems, and systems of the monitored medical imaging device 2 .
- the machine logs 40 may be stored on the non-transitory storage medium 12 using a logging format that typically employs a standardized syntax and formatting for the machine type and manufacturer of the medical imaging device 2 . Consequently, extract-transform-load (ETL) 42 of the machine log data 40 is straightforward based on the known standard log notation.
- ETL extract-transform-load
- Component IDs and operating parameters and their logged values are identified in the ETL-processed log data, and the knowledge engine 22 applies the decision rules 36 to this information to identify any operating parameters whose values are out of the corresponding operating parameter range. Identified out-of-range operating parameters trigger the generation of maintenance alerts 24 .
- the knowledge engine 22 is applied to the log data generated by the monitored medical imaging device 2 to detect bad log data generated by the monitored medical imaging device 2 and to generate maintenance alerts 24 in response to the detected bad log data.
- the maintenance alerts are associated with component IDs contained in the detected bad log data, e.g. the log data will report a temperature reading being associated with the component (identified by its component ID) for which the reading was taken.
- a maintenance alert may occur only after the out-of-range parameter value is detected for a certain time interval (possibly sensor-dependent) to avoid overaggressive triggering of maintenance alerts by occasional inaccurate sensor readings. It is also contemplated for maintenance alerts 24 to be coupled with out-of-range log data (or more specifically, out-of-range parameter values of the log data) on other than a 1:1 basis.
- a maintenance alert 24 may be generated in response to a detected combination of two or more parameters that are outside their respective operating parameter ranges in the out-of-range log data. That is, a decision rule may be applied to detect a combination of two or more parameters that are outside their respective operating parameter ranges in the out-of-range log data.
- a maintenance alert could be generated in response to a (possibly weighted) sum of different out-of-range parameters, e.g. if two parameters might be out-of-range by amounts that individually would not trigger a maintenance alert, but the combination of out-of-range parameters may trigger an alert.
- the alert may be triggered at lower tolerances, e.g. considering parameters identified generically as parameters A and B, a maintenance alert may be issued if: A>5%; B>10% or A>3% AND B>8%.
- the tolerance of parameter A is weighted higher than that of parameter B (40% v. 20% in the combination tolerance change).
- the maintenance alerts 24 are presented to the user, e.g. by being displayed in an “alerts” window shown on the display 18 of the monitor interface device 14 .
- a maintenance alert may be presented as a maintenance recommendation, e.g. “Recommend to check PET detector temperature, recent readings have exceeded the recommended upper limit of 100° C.”
- the maintenance alerts may be graded as to severity or urgency, e.g. based on how far out-of-range the parameter value is (e.g., a PET detector reading of 102° C. may trigger a lowest-level alert while a reading of 115° C. may trigger a higher-level alert), and/or based on the criticality of the component (e.g.
- an out-of-range parameter reading for the PET detectors may generate a higher-level alert than an out-of-parameter reading for a bearing of the patient support 8 ).
- the processing 30 , 32 of the manuals may include detection of urgency notations contained in the text of the manuals themselves. For example, it is not uncommon for a manual to highlight critical parameter ranges by keywords such as “IMPORTANT”—when such a keyword is detected in conjunction with an extracted decision rule then this decision rule may be tagged as being of high importance, and any maintenance alert triggered by such a rule will be assigned a high alert level.
- the maintenance alerts 24 may optionally also indicate severity or urgency in other terms, such as by the potential impact (e.g.
- the maintenance alerts 24 may optionally also be forwarded to the medical imaging device vendor or other maintenance service provider to potentially align service calls and/or parts distribution.
- feedback from the logs is used to adaptively adjust or tune the decision rules 36 applied by the knowledge engine 22 .
- This leverages service logs 44 which are typically maintained for the medical imaging device 2 , e.g. based on service records generated manually by service personnel and/or automatically by system software that, for example, detects and timestamps replacement of various components.
- the service logs 40 may be stored on the non-transitory storage medium 12 using a logging format that typically employs a standardized syntax and formatting for the machine type and manufacturer, and/or a standardized syntax and formatting used by the servicing organization. Consequently, ETL 46 of the service log data 44 is straightforward based on the known standard log notation.
- accuracy of generated maintenance alerts 24 is determined in a feedback analysis 48 based on feedback extracted from the service log 44 , and the knowledge engine 22 is updated by adjusting the decision rules 36 based on the determined accuracy of the generated maintenance alerts.
- accuracy of a generated maintenance alert may be determined based on feedback extracted from the service log 44 over a time interval extending from a time of generation of the maintenance alert to an end time that is dependent upon a mean time to failure (MTTF) of the medical imaging device component identified by the component ID associated with the maintenance alert.
- MTTF mean time to failure
- the end time may be a scaled value, e.g. 1.5 ⁇ MTTF.
- the maintenance alert may be deemed to be accurate. If no servicing is detected in this time interval and the subject component does not fail, then it may be assumed that the maintenance alert was not accurate, in that service personnel did not believe it appropriate to act upon the maintenance alert and indeed the maintenance alert did not accurately predict an impending failure of the subject component.
- the adjustment or remediation of the inaccurate decision rule may entail removal of the inaccurate decision rule, or adjustment of the operating parameter range of the inaccurate decision rule. Such adjustment may employ a percentage change, for example if the inaccurate decision rule triggers a maintenance alert if a maximum parameter value threshold is exceeded, then this threshold may be increased by 5% (or by 10%, or by some other chosen adjustment increment).
- the inaccurate decision rule triggers a maintenance alert if a minimum parameter value threshold is exceeded, then this threshold may be decreased by 5% (or by 10%, or by some other chosen adjustment increment).
- the inaccurate decision rule may be presented to a user for review, perhaps along with instances in which it has triggered alerts in the past, and the user may elect to make a manual adjustment to the inaccurate decision rule, or may elect to remove the rule altogether, or may elect to maintain the rule without adjustment.
- FIG. 1 Having provided an overview of a monitoring device for monitoring a medical imaging device 2 with reference to FIG. 1 , some additional embodiments and examples are presented in the following.
- the user manual 27 can inform about features of the medical imaging device and how to use it, its use case bounding conditions, and this helps to determine correct practice and use by the user.
- the service manual 25 can give further detailed information at the component level, its life span based on use, absolute life, symptoms and behaviors, action needs to be taken for each problem and other critical parameters and their bounds.
- manufacturer specifications 26 are an important element of cost and quality control for testing, calibration, and other measurement processes. Integrating the information from the above manuals provides comprehensive information for providing maintenance alerts in furtherance of predictive maintenance for the medical imaging device and its components.
- CER Component Entity Recognition
- Co-reference is carried out to identify the specific sentence where the association is not clear.
- Dependency parsing is performed to identify the associated patterns to the specific components that needs to be mapped. Further, each of the relevant sentences associated with the components are extracted and tabulated.
- FIG. 2 shows an example of manual content for a dedicated distribution transformer from a CT installation guide.
- FIG. 3 shows a component ID extracted from the content of FIG. 2 , namely the component ID “Dedicated Distribution Transformer”, and key metrics extracted for that component. All other extracted component IDs are similarly processed to arrive at a structured table ( FIG. 3 being a portion of this table) containing all the components (or, more specifically, the component IDs) and the associated key metrics that needs to be monitored for the smooth running of the medical imaging device.
- the creation of the decision rules 36 of the knowledge engine 22 entails combining data extracted from the different manuals in a cogent way for every monitored component of the medical imaging device. For each component, from the vast inputs that were cleaned in the operations 30 , 32 , data from service manuals, manufacturer specifications, and user manuals are consolidated in the form of tables. This process is performed by: (1) Taking the system details from the user manuals such as system make, model etc.
- a unique ID can be associated with each system; (2) For the system details from step (1), the corresponding data from the manufacturer specifications are extracted and tagged with a unique key associated with the user manual, and (i) Once the system details are in place, using the service manual, the service actions pertaining to the specific system configurations are consolidated and tagged with the same unique ID. Once the above three steps are done, the data is tagged per system model. A mapping procedure is then performed, in which each of the faults that the system is known to suffer are extracted and mapped to the errors that a system logs and correlations are drawn from these to predict the nature of the data. This process is diagrammatically shown in FIG. 4 .
- the components and the operating parameters extracted for each component using the CER 32 with their possible values can be represented using an unstructured data table, as shown in FIG. 5 .
- P 1 , P 2 , P 3 . . . , P N be the N operating parameters extracted from the fixed number of components and let Vk 1 , Vk 2 , Vk 3 . . . , Vk N represents possible parameter values for the respective components and parameters.
- Vk N represents possible parameter values for the respective components and parameters.
- These are the sets of parameters extracted from all the manuals using the NLP 30 and CER 32 .
- the objective is to find meaningful combinations (Ps 1 , Ps 2 , Ps 3 , . . . , Ps M , M ⁇ N) of these parameters and their respective optimal limits aiming to produce model a M c to produce good pool or bad pool based on each component. This can be explained notationally as given below:
- M 2 ⁇ YP i is the set of rules based on only one parameter
- M 2 Y(P i 1 , P i 2 ) is the set of rules based on each combination of tuple;
- M k Y(P i 1 , P i 2 , . . . , P i k ) is the set of rules based on k-tuple at a time;
- the output of the resulting knowledge engine 22 will contain the set of decision rules 36 for each of the components as depicted in the sample table of FIG. 6 , which aids in classifying the data into good and bad pool. Comparing FIG. 6 with FIG. 3 illustrates how the decision rules may be formulated as maintenance recommendations as shown in the rightmost column of FIG. 6 .
- the process of mapping of the knowledge engine 22 with the machine log data 40 involves parsing of machine log data into system readable format and storing in database system, this process is the ETL 42 indicated in FIG. 1 .
- the log data that is stored in database will be typically distributed in multiple tables of the database.
- the first step towards the goal of mapping of the knowledge engine 22 with the machine log 40 is to aggregate related log data from database.
- the log data can be related in two ways: (1) Temporal dependency of log information; and (2) shares a common component or sub-system.
- log data from databases is extracted and aligned in the temporal domain (increasing/decreasing) and temporal pattern-matching is used to extract dependent log information, e.g. as illustrated in FIG. 7 .
- the detection and aggregation of log information that belong to same type of component/sub-system can be performed using various approaches to detect correlating log data.
- One suitable approach is lexical analysis of content of log information and correlating lexical analysis output of two or more instances of log.
- a threshold on correlated value is used to categorize whether a given set of log information are related or not.
- Another illustrative approach is to count appearance or non-appearance of two or more error log data in given timeframe (typically data generated over a single day) and categorize as belong to same or different groups.
- Yet another illustrative approach is to compute mutual exclusivity of given set of log information with one another, two or more log information data having least mutual exclusive value will be considered as belong to same group of errors.
- each group 52 of segregated log information is sent to a splitter and lexical analyzer (SLA) unit 54 .
- SLA splitter and lexical analyzer
- the SLA unit 54 first splits log information into meaningful tokens (typically words) and then each token is cross-correlated with a set of vocabulary words 56 derived from source that generated knowledge engine (e.g. electronic medical imaging device servicing and operating data such as the illustrative user manual 27 , specification data 26 , service manual 25 ).
- source that generated knowledge engine e.g. electronic medical imaging device servicing and operating data such as the illustrative user manual 27 , specification data 26 , service manual 25 .
- Each element in the vocabulary 56 is assigned weight based on its importance.
- the weightage is decided by using domain expertise and/or by using statistical techniques like counting of words appearing in knowledge engine, number of connectivity of a given word with unique different works, etc.
- the vocabulary 56 is suitably derived from CER 32 .
- the mapper 50 uses weighted and tokenized data from the SLA unit 54 to map log information with entries (e.g. decision rules) of the knowledge engine 22 as at this stage both type of data (log data and knowledge engine rules) will be of same nature as well as employing common language owing to fact that log data is processed using the stored vocabulary 56 .
- the mapping 50 can be achieved by techniques such as word similarity matching or correlating the words.
- a method suitably performed by the monitoring device of FIG. 1 to construct the decision rules 36 of the knowledge engine 22 is illustrated.
- the various electronic medical imaging device servicing and operating data 25 , 26 , 27 , 28 are input to the knowledge engine builder 20 .
- a reader module e.g. illustrative NLP 30 and CER 32
- the knowledge engine 22 is built by operations including formulating the operating parameter ranges into the set of decision rules 36 for classifying medical imaging device log data as good or bad log data.
- the log files 40 from the medical imaging device 2 are pre-processed by the ETL 42 .
- the CER 32 is applied to the ETL-processed log data to extract component IDs and operating parameters and values thereof. This is suitably done by applying the same CER 32 used in processing the manuals to the ETL-processed log data.
- the decision rules 36 of the knowledge engine 22 are mapped on the log file output by the ETL/CER processing chain. The operation 66 may be performed as already described with reference to FIG. 8 .
- the output of the mapping 66 is used to segregate log data into good data (in which the operating parameters are in range as defined by the decision rules 36 ) or bad data (in which at least one operating parameter is out of range as defined by the decision rules 36 ).
- maintenance alerts 24 are issued based on detected bad data.
- the bad data may be used to update the decision rules 36 based on whether the maintenances alerts are determined to be accurate (e.g., based on information from the service log 44 on whether the subject component was serviced or failed during the MTTF time interval after issuance of the maintenance alert).
- the textual content of the raw manual data 80 typically includes tables and text (e.g. paragraphs, sentences, bullet points, ordered lists, or so forth).
- a table parser 82 parses table content into semantic units such as rows and/or cells, while a text parser 84 parses textual content into semantic units such as sentences or paragraphs.
- the NLP 30 and CER 32 are then applied as previously described to extract component IDs and operating parameters and associated operating parameter ranges or values. In the example of FIG.
- this latter processing is shown separately as a value engine component 86 , where parameter value association is performed on a sentence-wise, cell-wise, or other semantic unit basis, to generate component-wise output 88 .
- the component-wise outputs 88 from the various electronic medical imaging device servicing and operating data 25 , 26 , 27 , 28 are combined by an assimilator 90 to generate the combined unstructured data table 92 , as already described with reference to FIG. 5 .
- This data table 92 then serves as input to the knowledge engine building operation 64 of FIG. 9 .
- FIG. 1 shows a flowchart converter 100 implemented on the server computer 10 .
- the knowledge engine builder 20 further implements flowchart image extraction. For example, an image element of a document 25 , 26 , 27 , or 28 may be identified as depicting a flowchart by the knowledge engine builder 20 by detecting a caption of the image as a text block directly above or below the image in the document, and further detecting text content of that caption indicating a flow chart.
- This detection could entail detecting the keyword “flow chart” or “flowchart”, and/or performing NLP to detect the indication of a flowchart.
- the image may be identified as a flowchart by analyzing the image content, e.g. detection of rectangles, diamonds, or other geometric figures containing text and connected by connecting lines is a strong indication that the image depicts a flowchart.
- the flowchart converter 100 then processes the image of the flowchart to generate a structured electronic representation of the flowchart. This is done for each flowchart identified in the documents 25 , 26 , 27 , 28 to generate a database 101 of structured electronic representations of the respective flowcharts contained in the documents.
- a chatbot 102 can be implemented on a computer 150 that includes at least one user input device 152 (e.g., a keyboard and a mouse) and a display device 154 for displaying a graphical use interface (GUI) 156 .
- GUI graphical use interface
- a chatbot 102 can be displayed on the GUI 156 .
- the chatbot 102 provides a convenient mechanism for user navigation of the structured electronic representations of the flowcharts generated by the flowchart converter 100 and stored in the database 101 .
- FIG. 12 shows an example of the flowchart converter 100 .
- the flowchart converter 100 includes a process flow digitization module 104 , and a digitized process flow to chatbot conversion module 106 to generate the structured electronic representation of the flowchart for implementation by the chatbot 102 .
- the process flow digitization module 104 is configured to receive a set of scanned process flow images 108 as an input, e.g. received from the knowledge engine builder 20 in the illustrative embodiment.
- the images 108 depict the process flow or flowcharts, and may for example be JPEG images, BMP images, PNG images, or the like.
- the process flow digitization module 104 is configured to convert the images 108 into a process tree or graph, such as a directed graph (DG).
- DG directed graph
- FIG. 13 shows an example of an image 108 showing a flowchart 110 .
- the flowchart 110 includes shapes or blocks 112 typically used in flowcharts.
- the flowchart 110 includes start/stop blocks (e.g., ovals), process or action blocks (e.g., rectangles), decision blocks (e.g., diamonds), and arrows 114 connecting the blocks 112 .
- the flowchart may also include text in the blocks 112 and/or adjacent the arrows 114 .
- the illustrative flowchart 108 is fully captured in a single image, but this may not always be the case.
- a flowchart is on multiple pages of a service manual, this is indicated in the caption and is also indicated by connector blocks in the flowchart.
- a circle with a text label such as “A” or “To FIG. 7 b ” or “From FIG. 7 a ” or so forth graphically depicts the flow connections between the pages.
- FIG. 14 depicts a DG 116 corresponding to the flowchart 108 of FIG. 13 .
- the process flow digitization module 104 performs the following process on the image(s) 108 .
- a pre-processing operation is performed to remove noise from the images.
- a skew correction operation s performed to align the orientation of the images 108 .
- a shape detection operation is performed to detect one or more of the blocks or shapes 112 (e.g., oval, rectangle, diamond, arrows 114 , and so forth) and connecting arrows in the flow process.
- One approach for shape detection is to apply an edge detector (e.g.
- Sobel difference of Gaussians, Canny algorithm, et cetera
- shape-specific detection algorithms e.g., circle detector, rectangle detector, and so forth, each operating on the basis of pixel connectivity and a priori knowledge of the geometry of the shape.
- Detection of arrow connectors is also performed after the edge detector is applied, and is based on connectivity of lines between the identified shapes.
- a text detection process e.g., OCR
- OCR is performed to detect text in the detected shapes 112 , adjacent the detected shapes, or on top of the detected shapes (e.g. text on top of decision arrows 114 ).
- Each of the detected shapes and each block of detected text (which could be a word, one-line phrase, or contiguous multi-line block of text) are assigned a location anchor in the image 108 . These location anchors are used to associate text and shapes or text and arrow connectors. Once the shapes 112 and text are detected along with associated location anchors and connecting arrows, this information is analyzed across the image 108 is performed to generate the DG 116 .
- the detected shapes 112 and text are parsed from top to bottom, and a node 118 is created for each shape.
- Each node 118 includes a parent link pointing to a parent node 120 , child link(s) 122 connected to child nodes 124 , a node type (e.g., a process or decision step in the flowchart), and node text.
- the shapes 112 are again parsed from top to bottom and detected directed arrows 114 are used to connect the nodes 118 based on their proximity and direction to shapes 112 as present in the image 108 .
- the digitized process flow to chatbot conversion module 106 is configured to convert the DG 116 to a structured dialog flow representation suitable for presentation by the chatbot 102 .
- the digitized process flow to chatbot conversion module 106 converts the content of the DG 116 into a structured dialog flow representation that can be presented by the chat bot as a natural language conversation flow.
- the chatbot conversation flow is represented in a structured dialog flow representation, such as JavaScript Object Notation (JSON) representation 130 .
- JSON JavaScript Object Notation
- JSON representation 130 An example of a JSON representation 130 is provided below:
- the nodes 118 in the process flow each contains a link to a previous node or the parent node 120 .
- a node 118 is also differentiated as bot or user node to identify bot and user dialogues in a conversation flow.
- a user dialog node may, for example, receive the result of a test performed by the user in response to a preceding bot dialog node instructing to perform that test. Based on the received test result, flow may pass to one of several possible child nodes each of which is a bot dialog node presenting the next step of the flow chart.
- the parent node 120 in the process flow is delivered as the first bot dialogue when the chat bot conversation is initiated.
- the user response is matched against the user nodes 118 in the flow, which are linked to the previous delivered bot dialogue.
- Various string similarity-matching techniques like cosine similarity, shortest distance, etc. can be used to find the highest match.
- the matched node is then used to find the bot node linked to it, is delivered as the next bot dialogue in the conversation.
- the conversation continues using the same approach.
- the server computer 10 is configured as described above to perform method or process 200 of converting the flowchart 110 to a structured electronic representation, such as the DG 116 , and to convert the DG 116 to a structured dialog flow representation.
- the non-transitory storage medium 12 stores instructions which are readable and executable by the electronic processors 10 , 16 to perform disclosed operations including performing the method or process 200 .
- the method 100 may be performed at least in part by cloud processing.
- an illustrative embodiment of an instance of the method 200 is diagrammatically shown as a flowchart.
- a plurality of shapes 112 corresponding to flowchart blocks of the flowchart 110 are identified in one or more images 108 of the flow chart.
- one or more arrows 114 defining flow paths between the flowchart blocks in the image 108 are identified.
- arrows 114 can be identified by flowchart blocks connected by the arrows and identifying directionality of the defined flow paths based on arrowheads of the arrows.
- text labels and their locations in the image 108 , are identified.
- an OCR operation can be performed on the image 108 to determine text content of the identified text labels.
- text labels (and variants thereof) refers to a portion of the image 108 containing text
- text content refers to content that has undergone an OCR process. That is, the text content comprises a string of ASCII characters (or other suitable characters, such as Chinese or Indian characters).
- the OCR process is not limited to images 108 of the flowchart 110 generated by optical scanning.
- the operation 206 includes extracting text corresponding to location anchors in the image 108 .
- the locations anchors serve as a location marker for the identified text.
- the location anchor might be a single point (e.g., center of a block) or the location anchor might be an entire footprint of the shape 112 (or arrow 114 or text label) in the image 108 .
- the “footprint” of a block or shape 112 can be its outline or perimeter, while the “footprint” of an arrow 14 can be a line following the arrow.
- the “footprint” of a text label is a polygon enclosing and closely fitted to the area of the text in the image 108 .
- the text labels (from the operation 206 ) are associated with flowchart blocks, or defined flow paths in the flowchart, based on the locations of the text labels, the flowchart blocks or shapes 112 , and arrow 114 in the image 108 .
- the associating operation 208 includes determining the distances from the footprint of the text label to the footprints of the various blocks or shapes 112 and arrows 114 , and associating the text label to the block or arrow of shortest distance.
- a label “Yes” on an arrow 114 coming from a decision diamond shape 112 might be as close to the diamond as to the arrow, so a secondary consideration might be the text content of the text label. For example, text content of “Yes” or “No” is likely to be labeling an arrow 114 .
- text content inside a block or shape 112 is almost certainly properly associated to that block (so the distance is zero).
- a structured electronic representation such as the DG 116 is generated flowchart based on the flowchart blocks or shapes 112 , the flow paths, and the text labels.
- a structure of the structured electronic representation 116 is determined at least based at least on the flow paths between the flowchart blocks or shapes 112 of the image 108 .
- shape-indicated functions of the flowchart blocks 112 can be determined by comparing the identified shapes with standard flow chart shapes representing corresponding functions. For example, a flowchart typically uses oval shapes for start and stop operations, rectangles to represent process blocks, and diamonds to represent decision blocks. The “standard” blocks are compared with the shapes 112 identified in the operation 202 .
- functions are assigned to the identified flowchart blocks or shapes 112 based on the text labels associated to the flowchart blocks. Additionally or alternatively, functions are assigned to the identified flowchart blocks or shapes 112 based on the text labels associated to the defined flow paths. If the image 108 incorrectly shows a square representing a decision block (e.g., a decision block should be a diamond according to the standard flow chart shapes), it may be possible to recognize the function is a decision block because it will have two (or more) outgoing arrows (e.g., one labeled “Yes” followed if the decision is Yes, the other labeled “No” for if the decision is No).
- any time a block has two or more outgoing arrows 114 (regardless of the labels) it must be a decision block since the block must “decide” which outgoing arrow to follow.
- the text labels associated to the flow paths can likely be useful in determining which paths are which.
- the identifying operation 202 of the shapes 112 , the identifying operation 204 of the arrows 114 , and the generating of the structured representation 116 can be repeated for each image in a set of images 108 for the flow chart.
- a first page of a flowchart 110 can end with an arrow to a circle block labeled “A”, and then the next page begins with the same circle block labeled “A” and moves on to the next block.
- a connecting block or shape 112 in different pages can be identified, and the structured representation 116 can be generated based on a connection indicated by the connecting blocks.
- the structured representation 116 can be transferred to the computer 150 (see FIG. 1 ) operable by a user.
- the structured representation 116 can be displayed on the display device 154 of the computer 150 .
- the user can then enter, via the at least one user input device 152 , one or more inputs indicative of editing a portion of the structured electronic representation 116 .
- the structured representation 116 can then updated based on the user inputs.
- the structured electronic representation 116 can comprise the DG 116 .
- the method 200 can include an operation 212 that includes converting the directed graph 116 to a structured dialog flow representation (which can be considered another form of structured electronic representation of the flowchart), such as the JSON representation 130 .
- the JSON representation 130 can be presented on the GUI 156 on the computer 150 .
- the GUI 156 can comprise, in some examples, the chatbot 102 .
- the chatbot 102 is configured to guide the user through the JSON representation 130 . To do so, a current flowchart block of the JSON representation 130 is presented on the chatbot 102 .
- An input is received from the user by the chatbot 102 (e.g., a mouse click, or text input as a question).
- the current flowchart block can be updated based on the structure of the JSON representation 130 and the received input. These processes are repeated to guide the user through the flowchart 110 .
- the flowchart that is converted to a structured electronic representation is a fault isolation flowchart used in conjunction with diagnosing a problem with a medical imaging device.
- a structured electronic representation e.g. DG or structured dialog flow representation
- the disclosed approach can be employed to generate a structured electronic representation, and optionally a structured dialog flow representation presented by a chatbot, for fault isolation flowcharts employed in conjunction with other types of servicing tasks (e.g. other medical devices such as infusion pumps, mechanical ventilators, or so forth, or even more generally servicing tasks for other complex systems/devices such as aircraft, locomotives, HVAC systems, and/or so forth.
- other medical devices such as infusion pumps, mechanical ventilators, or so forth, or even more generally servicing tasks for other complex systems/devices such as aircraft, locomotives, HVAC systems, and/or so forth.
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Health & Medical Sciences (AREA)
- General Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- General Health & Medical Sciences (AREA)
- Computational Linguistics (AREA)
- Artificial Intelligence (AREA)
- Business, Economics & Management (AREA)
- Biomedical Technology (AREA)
- General Business, Economics & Management (AREA)
- Medical Informatics (AREA)
- Public Health (AREA)
- Primary Health Care (AREA)
- Epidemiology (AREA)
- Computing Systems (AREA)
- Software Systems (AREA)
- Audiology, Speech & Language Pathology (AREA)
- Mathematical Physics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Radiology & Medical Imaging (AREA)
- Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
- Quality & Reliability (AREA)
- Measuring And Recording Apparatus For Diagnosis (AREA)
- Medical Treatment And Welfare Office Work (AREA)
Abstract
Description
- This application is a continuation-in-part of Application Serial No. PCT/EP2019/055526, filed Mar. 6, 2019, which claims priority to Application Ser. No. U.S. 62/646,993, filed Mar. 23, 2018, both of which are incorporated herein by reference in their entireties.
- The following relates generally to the medical imaging device maintenance arts, device monitoring arts, predictive maintenance arts, process flow automation arts, and related arts.
- Medical devices used in the healthcare industry, such as magnetic resonance imaging (MRI) scanners, computed tomography (CT) scanners, positron emission tomography (PET) scanners, gamma cameras used in single photon emission computed tomography (SPECT), image-guided therapy (iGT) systems, and other medical imaging devices, or electrocardiograph (ECG) or patient monitor devices, and so forth, should be in good working condition to ensure doctors and patients receive correct information for medical diagnoses and patient monitoring and so forth. Medical imaging devices are expensive to replace and play a crucial role in diagnosis. Any downtime of these devices results in a loss of revenue to the medical institution, loss of quality treatment for patients, and introduces delays into patient treatment. Emphasis is thus placed on minimizing the downtime of medical imaging devices and ensuring uninterrupted operational status while maintaining quality of performance.
- Predictive maintenance is an important part of minimizing downtime of medical imaging devices. This approach entails predicting and proactively repairing or otherwise remediating possible failures of medical imaging device components in advance based on the machine logs and usage history. In this way, maintenance can be proactively performed to minimize or eliminate downtime and impact on patient care.
- However, predictive maintenance is difficult to implement in practice. Modern medical imaging devices produce huge volumes of log data, on the order of gigabytes or terabytes or higher. Furthermore, proactively identifying log data that statistically indicates a likely component failure is difficult due to the time lag between the log data and the subsequent component failure. Still further, determining which log data is useful in predicting a component failure is difficult. Conventionally, prior information has been used to identify these types of bad log data indicative of a likely component failure. The prior information is typically provided by subject matter experts with specialized technical knowledge of the medical imaging device components, their performance envelopes, and the possible problems. Generation of the knowledge engine for performing predictive maintenance is typically a laborious process requiring input of subject matter experts knowledgeable in the medical imaging devices and their components to develop decision rules that can be applied to log data of a monitored medical imaging device.
- A further problem is that the resulting predictive maintenance system is static. This is a problem because medical imaging device manufacturers are continually adding new products and improving existing product lines. These changes will not be reflected in the knowledge engine used for predictive maintenance. Consequently, manual updating of the knowledge engine must be performed on a frequent basis, which again requires extensive input from subject matter experts.
- Moreover, field service engineers (FSEs) frequently have to provide service to their customers by repairing their imaging systems. During the process of inspecting faulty imaging systems, FSEs can refer to printed manuals or e-books, which have fault isolation process flows. FSEs need to carry these manuals, then search for the appropriate content to follow the process flows in the form of a flow chart. FSEs follow the steps of the flow chart to perform fault isolation of the imaging system. Some of these flow charts can be complex, long and run across multiple pages with a lot of steps. It can be easy can lose track of the steps in the flow chart.
- The imaging system printed manuals can be in text form or PDF forms. These process flow diagrams are present as images within PDF documents. While the image format may allow for the FSE to zoom in on specific portions of the flow chart, difficulties can still arise, for example if the portion of the flow chart being traversed crosses onto another image/page.
- The following discloses certain improvements.
- In some embodiments disclosed herein, a non-transitory storage medium stores instructions readable and executable by an electronic processor to perform a monitoring method for generating maintenance alerts. The monitoring method includes: extracting component identifications (component IDs) identifying medical imaging device components and operating parameters of the identified medical imaging device components and associated operating parameter ranges from electronic medical imaging device manuals; generating a knowledge engine by operations including formulating the operating parameter ranges into a set of decision rules for classifying medical imaging device log data; and applying the knowledge engine to log data generated by a monitored medical imaging device to detect out-of-range log data generated by the monitored medical imaging device and to generate maintenance alerts in response to the detected out-of-range log data wherein the maintenance alerts are associated with component IDs contained in the detected out-of-range log data.
- In some embodiments disclosed herein, a monitoring device is disclosed for generating maintenance alerts. The monitoring device comprises a display, a non-transitory storage medium as set forth in the immediately preceding paragraph, and an electronic processor operatively connected with the display and with the non-transitory storage medium to perform the monitoring method further including displaying the generated maintenance alerts on the display.
- In some embodiments disclosed herein, a monitoring method performed by an electronic processor is disclosed for generating maintenance alerts. Component identifications (component IDs) are extracted which identify medical imaging device components in electronic medical imaging device manuals. Operating parameters of the identified medical imaging device components and associated operating parameter ranges are extracted from the electronic medical imaging device manuals based on numeric values, parameter terms identifying operating parameters, and linking terms or symbols indicative of equality or inequality that connect the numeric values and parameter terms. The operating parameter ranges are formulated into decision rules for classifying medical imaging device log data based on whether a value of the associated operating parameter is outside of the operating parameter range. The decision rules are applied to log data generated by a monitored medical imaging device to detect out-of-range log data generated by the monitored medical imaging device, and maintenance alerts are displayed on a display in response to the detected out-of-range log data. The maintenance alerts are generated from out-of-range log data and are associated with component IDs contained in the out-of-range log data.
- In some embodiments disclosed herein, a monitoring device is disclosed for generating maintenance alerts. The monitoring device includes a display, an electronic processor operatively connected with the display, and a non-transitory storage medium storing instructions readable and executable by the electronic processor to perform a monitoring method for generating maintenance alerts. In the monitoring method, decision rules are generated for classifying medical imaging device log data based on whether parameter values of operating parameters of medical imaging device components identified by component identifications (component IDs) in the log data are within operating parameter ranges for the respective operating parameters. The operating parameter ranges for the respective operating parameters are extracted from electronic medical imaging device manuals. The decision rules are applied to log data generated by a monitored medical imaging device to detect out-of-range log data generated by the monitored medical imaging device. The display is controlled to present maintenance alerts in response to the detected out-of-range log data wherein the maintenance alerts are associated with component IDs contained in the detected out-of-range log data.
- In some embodiments disclosed herein, a non-transitory computer readable medium stores instructions executable by at least one electronic processor to perform a method of converting a flowchart to a structured electronic representation of the flowchart. The method includes: identifying, in an image of the flowchart, a plurality of shapes corresponding to flowchart blocks of the flowchart in the image; identifying arrows defining flow paths between the flowchart blocks in the image including identifying flowchart blocks connected by the arrows and identifying directionality of the defined flow paths based on arrowheads of the arrows; identifying text labels and their locations in the image and performing optical character recognition (OCR) on the image of the flowchart to determine text content of the identified text labels; associating the text labels with flowchart blocks or defined flow paths based on locations of the text labels, flowchart blocks, and arrows in the image; and generating the structured electronic representation of the flowchart based on the flowchart blocks, the flow paths, and the text labels. A structure of the structured electronic representation is determined at least based at least on the flow paths between the flowchart blocks of the image.
- In some embodiments disclosed herein, a non-transitory computer readable medium stores instructions executable by at least one electronic processor to perform a method of converting flowcharts to structured electronic representations. The method includes: identifying, in an image of a flowchart, a plurality of shapes corresponding to flowchart blocks of the flowcharts in the image; identifying a location and directionality of a plurality of arrows defining flow paths between the flowchart blocks in the image; generating a directed graph of the flowchart blocks; and converting the directed graph to a structured dialog flow representation.
- One advantage resides in providing a monitoring device for generating maintenance alerts for one or more monitored medical imaging devices, in which the knowledge engine of the monitoring device is generated automatically.
- Another advantage resides in providing such a monitoring device in which the knowledge base is continuously and efficiently updated based on service logs of the monitored medical imaging devices.
- Another advantage resides in providing such a monitoring device with computationally efficient generation and updating of the knowledge base.
- Another advantage resides in providing such a monitoring device in which the knowledge base is developed without input from subject matter experts.
- Another advantage resides in digitizing paper manuals for imaging systems.
- Another advantage resides in reducing errors or loss of process steps when following a process flow chart for servicing an imaging system.
- Another advantage resides in providing an interactive electronic flow chart in which a chatbot guides the user through the steps of the flow chart based on feedback provided to the chatbot by the user.
- Another advantage resides in providing digitized instructions for an FSE to provide service to an imaging system.
- Other advantages include reduced total time to build predictive models, elimination of subjectivity in labelling hence reducing manual error, and auto feedback for re-learning and self-correcting decision rules used for generating maintenance alerts.
- A given embodiment may provide none, one, two, more, or all of the foregoing advantages, and/or may provide other advantages as will become apparent to one of ordinary skill in the art upon reading and understanding the present disclosure.
- The invention may take form in various components and arrangements of components, and in various steps and arrangements of steps. The drawings are only for purposes of illustrating the preferred embodiments and are not to be construed as limiting the invention.
-
FIG. 1 diagrammatically illustrates a monitoring device for performing predictive maintenance including generating maintenance alerts for a monitored medical imaging device based on log data generated by the monitored medical imaging device, in which decision rules for generating the maintenance alerts are automatically generating. -
FIG. 2 diagrammatically illustrates a portion of a medical imaging device manual. -
FIG. 3 diagrammatically illustrates a table of components and key metrics derived from the manual portion shown inFIG. 2 . -
FIGS. 4, 5, and 6 diagrammatically illustrate aspects of generating the decision rules from medical imaging device manual content as described herein. -
FIGS. 7 and 8 diagrammatically illustrate aspects of mapping of medical imaging device log data to decision rules of the knowledge engine as described herein. -
FIG. 9 diagrammatically illustrates a method suitably performed by the monitoring device ofFIG. 1 to construct decision rules of the knowledge engine. -
FIGS. 10 and 11 diagrammatically illustrate aspects of the reader module component of the method ofFIG. 9 . -
FIG. 12 illustrates a system for converting a digital image of a flowchart to a structured electronic representation of the flowchart. -
FIG. 13 illustrates an exemplary digital image of a flowchart for conversion by the system ofFIG. 12 . -
FIG. 14 illustrates a directed graph produced from the flowchart image ofFIG. 13 . -
FIG. 15 illustrates an exemplary flow chart implemented by the system ofFIG. 12 . - In embodiments disclosed herein, electronic medical imaging device manuals are leveraged to extract component identifications (component IDs) identifying medical imaging device components and operating parameters of the identified medical imaging device components and associated operating parameter ranges. The electronic medical imaging device manuals can take substantially any machine-readable form, and may be online manuals, printed manuals that are scanned to pdf (or another electronic format) and processed by optical character recognition (OCR) to generate machine-readable text, and/or so forth. The electronic medical imaging device manuals may, for example, include one or more of a service manual, manufacturer specification(s), a user manual, an operating reference manual, and/or so forth. A knowledge engine is then generated by operations including formulating the operating parameter ranges into a set of decision rules for classifying medical imaging device log data. For example, a decision rule may classify input log data as bad (i.e. out-of-range) log data if the input log data indicates a value for an operating parameter that is outside of the operating parameter range associated with that operating parameter. The knowledge engine is then applied to log data generated by a monitored medical imaging device to detect out-of-range log data generated by the monitored medical imaging device and to generate maintenance alerts in response to the detected out-of-range log data. The maintenance alerts are suitably associated with component IDs contained in the detected out-of-range log data, and may in some embodiments be formulated as maintenance recommendations. For example, if the temperature of a component is indicated as being outside of the operating temperature range for that component, then the alert may be formulated as “<Component ID> operating temperature may be above its recommended operating temperature. Recommend to check temperature of <Component ID>.”
- A further aspect is that accuracy of the generated maintenance alerts may be determined based on feedback extracted from a service log of the monitored medical imaging device, and the knowledge engine may be updated by adjusting the decision rules based on the determined accuracy of the generated maintenance alerts. For example, accuracy of a generated maintenance alert may be determined based on feedback extracted from the service log over a time interval extending from a time of generation of the maintenance alert to an end time which is dependent upon (and may be equal to) the mean time to failure (MTTF) of the medical imaging device component identified by the component ID associated with the maintenance alert.
- By the disclosed approaches, the need for manual development and curation of a set of decision rules for performing predictive maintenance is reduced or eliminated.
- In addition, improvements are disclosed for the diagnostic phase, in which a field service engineer (FSE) works through a fault isolation flowchart to identify a root cause and/or solution to the problem underlying the maintenance alert. The FSE conventionally consults a service manual when servicing diagnostic equipment. Of particular value are fault isolation flowcharts which provide step-by-step guidance for performing tests and inspections in order to determine a root cause or solution of a problem. Such manuals can cover existing and legacy diagnostic equipment. These are presented in paper form, or in non-machine readable image (e.g. JPEG) or PDF format, e.g., as images embedded in a PDF, ePub, or other electronic book (ebook) or digital document format.
- In some embodiments disclosed herein, an approach to convert such paper flowcharts to a structured electronic form is provided. To this end, an image (e.g., JPEG, BMP, PNG, or so forth) of the flowchart serves as input. Shapes with corresponding location anchors in the image are identified, which correspond to flowchart blocks. Optical character recognition (OCR) is applied to extract text with corresponding location anchors in the image. Connecting arrows are identified and the directionality of the arrows (e.g., based on the direction the arrowhead is pointing) is used to define flow paths between the blocks, and a directed graph representation of the blocks is generated.
- Based on comparison with standard flowchart symbols (e.g., box for a general operation, oval for “begin” or “end”, diamond for decisions, et cetera), a function of each block may be determined. Additionally or alternatively, OCR'd text anchored inside the block or near a flow arrow can be processed by keyword detection, natural language processing (NLP), intent detection, or the like in order to extract the function information. This enables semantic labeling of the blocks and transitions of the directed graph. Optionally, the directed graph may be displayed at this point for review and editing (if appropriate) by a domain expert.
- Next, the directed graph is converted to a structured dialog flow representation. In some examples, JavaScript Object Notation (JSON) can be employed for the dialog flow representation, although other structured formats such as extensible markup language (XML) might be suitable.
- The foregoing operations are typically performed offline via a server computer or the like to create the directed graph and JSON representations. These (or at least the JSON representation) are then made accessible to FSEs for use during service calls. This can be done by downloading an application program (“app”) to a tablet computer, cellphone, notebook computer, or other mobile device used by the FSE, in which the app includes a database of converted fault isolation flowcharts. Alternatively, the FSE can access the JSON (and optionally the directed graph) representation over the Internet via a website hosted at a remote server using a web browser or dedicated app. To reduce storage requirements at the mobile device, the app may provide the FSE with the option to download only selected flowcharts.
- In a preferred embodiment, the JSON representation is used to drive a chatbot that guides the FSE through the diagnostic process. Hence, the chatbot will present the text of each box, either on-screen (similar to an instant messaging chatbot), and/or using text to voice, and/or using a graphical user interface (GUI) chatbot avatar. After each step, the FSE inputs any information needed to proceed (e.g., answer as to whether a performed test was positive or negative), and the chatbot proceeds accordingly through the JSON representation of the directed graph representing the fault isolation flowchart.
- With reference to
FIG. 1 , a monitoring device is disclosed for generating maintenance alerts for amedical imaging device 2. The illustrativemedical imaging device 2 is a PET/CT scanner including a positron emission tomography (PET)gantry 4 and a transmission computed tomography (CT)gantry 6 positioned in-line such that a patient can be loaded into eithergantry couch 8. Such a PET/CT scanner 2 is commonly used in numerous diagnostic and clinical tasks (e.g. oncology, cardiology, neurology, and so forth) as the CT imaging performed by theCT gantry 6 provides anatomical information that is complementary to functional information provided by PET imaging performed by theEPT gantry 4. As just one illustration of this synergy, a CT image can be used to generate an attenuation map that is used to perform attenuation correction during PET image reconstruction. The PET/CT scanner 2 is a highly complex medical imaging device with thousands of components, e.g. as a very few non-limiting examples including dozens or hundreds of PET detector modules, an X-ray tube and dozens or hundreds of X-ray detector modules (components of the CT gantry 6), mechanical components for revolving the X-ray tube and X-ray detector array around the patient, numerous robotic mechanisms of the patient support orcouch 8, electrical power components such as a distribution transformer for delivering electrical power to the PET/CT 2, and so forth (note, detailed medical imaging device components are not illustrated inFIG. 1 ). Each component typically has one, two, three, or more operating parameters that are monitored by appropriate sensors. For example, each PET detector module may include a temperature sensor that outputs module operating temperature, electrical sensors that monitor dark current and/or other electrical operating parameters, and/or so forth. A robotic component of thecouch 8 may include position encoders that monitor positioning of a robotic actuator. The CT gantry may include sensors monitoring rotation speed and encoding rotational angular position. The distribution transformer may include sensors monitoring operating power factor, input and output voltage, and so forth. Again, these are merely a few non-limiting illustrative examples. While the illustrative monitoredmedical imaging device 2 is a PET/CT scanner, it will be appreciated that the monitored medical imaging device may more generally be any type of medical imaging device with sufficient complexity to justify continuous monitoring and generation of timely maintenance alerts as disclosed herein. For example, the monitored medical imaging device may be a PET/CT scanner, a SPECT/CT scanner, a standalone gamma camera, standalone PET scanner, standalone CT scanner, an magnetic resonance (MR) imaging scanner, a PET/MR scanner, an image-guided therapy (iGT) device, or so forth. Moreover, it will be appreciated that the disclosed predictive maintenance monitoring devices and methods may be applied to monitor many such medical imaging devices, possibly of different types, in parallel. - With continuing reference to
FIG. 1 , the illustrative monitoring device includes aserver computer 10 or other electronic processor (e.g. desktop computer with sufficient computing capacity) operatively connected with anon-transitory storage medium 12 that stores instructions that are readable and executable by theelectronic processor 10 to perform a monitoring method as disclosed herein, and amonitor interface device 14 which, in the illustrative embodiment, is embodied by acomputer 16 that serves as a medicalimaging device controller 14 for controlling the monitored medical imaging device 2 (e.g., to operate therobotic support 8 to load a patient into the appropriate gantry, to select and execute an imaging sequence or protocol, to perform reconstruction of the acquired imaging data to generate an image and to display the image on adisplay 18 of the computer, and/or so forth). In other embodiments, the monitor interface device may be a separate computer, or may be integral with thecomputer 10 that performs the monitoring method. The monitor interface device includes thedisplay 18 for presenting maintenance alerts or other information, and may optionally include one or more user input devices (e.g. illustrative keyboard 19) for receiving user inputs. - The
non-transitory storage medium 12 may be variously embodied, e.g. as a hard disk drive, RAID array, or other magnetic storage medium, a solid state drive (SSD) or other electronic storage medium, an optical disk or other optical storage medium, various combinations thereof, and/or so forth. Further, it will be appreciated that the illustrativeelectronic processors electronic processor 10 may be implemented as a cloud computing resource comprising an ad hoc combination of a number of server computers. - The
electronic processor 10 reads and executes instructions stored on thenon-transitory storage medium 12 in order to perform a monitoring method including implementing aknowledge engine builder 20 that builds, and optionally subsequently adaptively maintains or updates, aknowledge engine 22 comprising a set of decision rules for generating maintenance alerts 24. To this end, theknowledge engine builder 20 leverages inputs from available electronic medical imaging device operating data. A commonly available form of electronic medical imaging device servicing and operating data are electronic manuals, such as an illustrative service manual 25, an illustrative set of manufacturer'sspecifications 26, anillustrative user manual 27, and an illustrativeoperating reference manual 28. These are merely illustrative titles, and it will be appreciated that various medical imaging device manufacturers and users (e.g. hospitals) may employ different titling for the electronic medical imaging device manuals. Likewise, the manual content may be variously distributed amongst one or more such manuals, e.g. in some implementations a single manual may cover the combined content of theillustrative user manual 27 andoperating reference manual 28, and/or the service manual may include the manufacturer's specifications as one or more appendices of the service manual, rather than as a separate manufacturer's specifications document, and/or so forth. The electronic medical imaging device servicing and operating data may also include service notes compiled into electronic form (e.g. electronic service logs maintained by service engineers), updates to the manuals (e.g. base system parameter updates, or upgraded operating parameters due to system/software upgrades, and/or so forth), external data such as online servicing and operating data available at the medical imaging device vendor's website, and/or so forth. The various electronic medical imaging device servicing and operatingdata medical imaging device 2 to be monitored, although such a requirement may be relaxed in instances in which different makes and/or models of a particular type of medical imaging device share certain systems or sub-systems. - The
knowledge engine builder 20 processes the content of the electronic medical imaging device servicing and operatingdata data NLP 30 may include parsing of text and tables of the electronic medical imaging device manuals to delineate semantic units including sentences, paragraphs, list items, table rows, or so forth. - Component entity recognition (CER) 32 is applied to identify component IDs in the NLP-processed text that identify medical imaging device components. The
CER 32 may employ any type of named entity recognition (NER) algorithm or combination of NER algorithms, e.g. leveraging a domain-specific vocabulary list to identify component IDs, identifying component IDs based on factors such as part-of-speech (if theNLP 30 includes grammatical parsing), and/or so forth. Although theCER 32 is preferably a fully automated process, in alternative embodiments it is contemplated to be semi-supervised, e.g. with uncertain component IDs presented to a user for confirmation or rejection. TheCER 32 also performs operating parameters extraction to extract operating parameters of the medical imaging device components identified by component IDs, along with associated operating parameter ranges. In one approach, operating parameters are identified by identifying numeric values and parameter terms which are connected by linking terms or symbols indicative of equality or inequality, and associating parameters identified by the parameter terms with medical imaging device components whose component IDs occur in the same semantic units as the parameter terms. For example, the sentence: “The PET detector modules should be kept at a temperature below 100° C.” can be processed by identifying the numeric value (100° C.) and a parameter term (temperature) which are connected by a linking term or symbol indicative of equality or inequality (below), and the operating parameter “temperature” is associated with the component ID (PET detector) occurring in the same semantic unit (same sentence) as the parameter term (temperature). Parameter terms can be identified using NER to identify “named entities” representing operating parameters, e.g. using a domain-specific vocabulary list, part-of-speech (if grammatical parsing if available), and/or so forth. Again, this is preferably a fully automated approach but in some embodiments semi-supervised parameter term extraction is contemplated, e.g. by presenting uncertain parameter names to a user for confirmation or rejection. Operating parameter ranges associated with operating parameters are determined based on the numeric values and the equalities or inequalities indicated by the linking terms or symbols. In the last example, the extracted operating range for the operating parameter “temperature” of the component ID “PET detector” can be expressed as “temperature<100° C”. Theknowledge engine 22 is then generated by operations including formulating the operating parameter ranges into a set of decision rules 36 for classifying medical imaging device log data as good data (e.g. within operating range) or bad log data (e.g. outside operating range). Advantageously, the extraction of component IDs and operating parameters and associated operating parameter ranges, and the generating of theknowledge engine 22, does not require receiving input from a subject matter expert. - Monitoring of the monitored
medical imaging device 2 leverages machine logs 40 which are commonly generated by medical imaging devices. In illustrativeFIG. 1 , the machine logs are generated by the medicalimaging device controller 14 based on readings of the various sensors of the components, sub-systems, and systems of the monitoredmedical imaging device 2. The machine logs 40 may be stored on thenon-transitory storage medium 12 using a logging format that typically employs a standardized syntax and formatting for the machine type and manufacturer of themedical imaging device 2. Consequently, extract-transform-load (ETL) 42 of themachine log data 40 is straightforward based on the known standard log notation. Component IDs and operating parameters and their logged values are identified in the ETL-processed log data, and theknowledge engine 22 applies the decision rules 36 to this information to identify any operating parameters whose values are out of the corresponding operating parameter range. Identified out-of-range operating parameters trigger the generation of maintenance alerts 24. Said another way, theknowledge engine 22 is applied to the log data generated by the monitoredmedical imaging device 2 to detect bad log data generated by the monitoredmedical imaging device 2 and to generatemaintenance alerts 24 in response to the detected bad log data. The maintenance alerts are associated with component IDs contained in the detected bad log data, e.g. the log data will report a temperature reading being associated with the component (identified by its component ID) for which the reading was taken. Optionally, the generation of a maintenance alert may occur only after the out-of-range parameter value is detected for a certain time interval (possibly sensor-dependent) to avoid overaggressive triggering of maintenance alerts by occasional inaccurate sensor readings. It is also contemplated formaintenance alerts 24 to be coupled with out-of-range log data (or more specifically, out-of-range parameter values of the log data) on other than a 1:1 basis. Amaintenance alert 24 may be generated in response to a detected combination of two or more parameters that are outside their respective operating parameter ranges in the out-of-range log data. That is, a decision rule may be applied to detect a combination of two or more parameters that are outside their respective operating parameter ranges in the out-of-range log data. For example, a maintenance alert could be generated in response to a (possibly weighted) sum of different out-of-range parameters, e.g. if two parameters might be out-of-range by amounts that individually would not trigger a maintenance alert, but the combination of out-of-range parameters may trigger an alert. The alert may be triggered at lower tolerances, e.g. considering parameters identified generically as parameters A and B, a maintenance alert may be issued if: A>5%; B>10% or A>3% AND B>8%. In this example, the tolerance of parameter A is weighted higher than that of parameter B (40% v. 20% in the combination tolerance change). - The maintenance alerts 24 are presented to the user, e.g. by being displayed in an “alerts” window shown on the
display 18 of themonitor interface device 14. For example, a maintenance alert may be presented as a maintenance recommendation, e.g. “Recommend to check PET detector temperature, recent readings have exceeded the recommended upper limit of 100° C.” In some embodiments, the maintenance alerts may be graded as to severity or urgency, e.g. based on how far out-of-range the parameter value is (e.g., a PET detector reading of 102° C. may trigger a lowest-level alert while a reading of 115° C. may trigger a higher-level alert), and/or based on the criticality of the component (e.g. an out-of-range parameter reading for the PET detectors may generate a higher-level alert than an out-of-parameter reading for a bearing of the patient support 8). In some embodiments, theprocessing - In some embodiments, feedback from the logs is used to adaptively adjust or tune the decision rules 36 applied by the
knowledge engine 22. This leverages service logs 44 which are typically maintained for themedical imaging device 2, e.g. based on service records generated manually by service personnel and/or automatically by system software that, for example, detects and timestamps replacement of various components. The service logs 40 may be stored on thenon-transitory storage medium 12 using a logging format that typically employs a standardized syntax and formatting for the machine type and manufacturer, and/or a standardized syntax and formatting used by the servicing organization. Consequently,ETL 46 of theservice log data 44 is straightforward based on the known standard log notation. In general, accuracy of generatedmaintenance alerts 24 is determined in afeedback analysis 48 based on feedback extracted from theservice log 44, and theknowledge engine 22 is updated by adjusting the decision rules 36 based on the determined accuracy of the generated maintenance alerts. For example, accuracy of a generated maintenance alert may be determined based on feedback extracted from theservice log 44 over a time interval extending from a time of generation of the maintenance alert to an end time that is dependent upon a mean time to failure (MTTF) of the medical imaging device component identified by the component ID associated with the maintenance alert. To account for statistical variability, the end time may be a scaled value, e.g. 1.5×MTTF. If servicing of the subject component is recorded in theservice log 44 during this time interval, or if a failure of the subject component is recorded in theservice log 44, then the maintenance alert may be deemed to be accurate. On the other hand, if no servicing is detected in this time interval and the subject component does not fail, then it may be assumed that the maintenance alert was not accurate, in that service personnel did not believe it appropriate to act upon the maintenance alert and indeed the maintenance alert did not accurately predict an impending failure of the subject component. - If the
feedback analysis 48 determines that the maintenance alert was accurate, then no adaptation or remediation is performed. On the other hand, if thefeedback analysis 48 determines that the maintenance alert was not accurate, then this information is used to adjust or remediate the decision rule that triggered the maintenance alert. In preferred fully automated adaptation embodiments, the adjustment or remediation of the inaccurate decision rule may entail removal of the inaccurate decision rule, or adjustment of the operating parameter range of the inaccurate decision rule. Such adjustment may employ a percentage change, for example if the inaccurate decision rule triggers a maintenance alert if a maximum parameter value threshold is exceeded, then this threshold may be increased by 5% (or by 10%, or by some other chosen adjustment increment). Likewise, if the inaccurate decision rule triggers a maintenance alert if a minimum parameter value threshold is exceeded, then this threshold may be decreased by 5% (or by 10%, or by some other chosen adjustment increment). In a variant semi-supervised embodiment, the inaccurate decision rule may be presented to a user for review, perhaps along with instances in which it has triggered alerts in the past, and the user may elect to make a manual adjustment to the inaccurate decision rule, or may elect to remove the rule altogether, or may elect to maintain the rule without adjustment. - Having provided an overview of a monitoring device for monitoring a
medical imaging device 2 with reference toFIG. 1 , some additional embodiments and examples are presented in the following. - In regards to the
processing operations medical imaging device 2 that provide a lot of information on the device. Theuser manual 27 can inform about features of the medical imaging device and how to use it, its use case bounding conditions, and this helps to determine correct practice and use by the user. Theservice manual 25 can give further detailed information at the component level, its life span based on use, absolute life, symptoms and behaviors, action needs to be taken for each problem and other critical parameters and their bounds. Similarly,manufacturer specifications 26 are an important element of cost and quality control for testing, calibration, and other measurement processes. Integrating the information from the above manuals provides comprehensive information for providing maintenance alerts in furtherance of predictive maintenance for the medical imaging device and its components. Data ingested from theuser manual 27,service manual 25, and themanufacturer specifications 26 are parsed to extract the context relevant information from the documents. The Component Entity Recognition (CER) 32 is performed to identify names of components (i.e. component IDs) and names of operating parameters so as to identify relevant operating parameters for these components. Co-reference is carried out to identify the specific sentence where the association is not clear. Dependency parsing is performed to identify the associated patterns to the specific components that needs to be mapped. Further, each of the relevant sentences associated with the components are extracted and tabulated. - With reference to
FIGS. 2 and 3 , an example is shown.FIG. 2 shows an example of manual content for a dedicated distribution transformer from a CT installation guide.FIG. 3 shows a component ID extracted from the content ofFIG. 2 , namely the component ID “Dedicated Distribution Transformer”, and key metrics extracted for that component. All other extracted component IDs are similarly processed to arrive at a structured table (FIG. 3 being a portion of this table) containing all the components (or, more specifically, the component IDs) and the associated key metrics that needs to be monitored for the smooth running of the medical imaging device. - With reference now to
FIG. 4 , once the data has been extracted and the components table has been created, a composite is made that allows for the next step of data classification. The creation of the decision rules 36 of theknowledge engine 22 entails combining data extracted from the different manuals in a cogent way for every monitored component of the medical imaging device. For each component, from the vast inputs that were cleaned in theoperations FIG. 4 . - With reference to
FIG. 5 , the components and the operating parameters extracted for each component using theCER 32 with their possible values can be represented using an unstructured data table, as shown inFIG. 5 . Let P1, P2, P3 . . . , PN be the N operating parameters extracted from the fixed number of components and let Vk1, Vk2, Vk3 . . . , VkN represents possible parameter values for the respective components and parameters. These are the sets of parameters extracted from all the manuals using theNLP 30 andCER 32. The objective is to find meaningful combinations (Ps1, Ps2, Ps3, . . . , PsM, M≤N) of these parameters and their respective optimal limits aiming to produce model a Mc to produce good pool or bad pool based on each component. This can be explained notationally as given below: - M2→YPi is the set of rules based on only one parameter;
- M2=Y(Pi
1 , Pi2 ) is the set of rules based on each combination of tuple; - Mk=Y(Pi
1 , Pi2 , . . . , Pik ) is the set of rules based on k-tuple at a time; - Then
-
- be the models considering rules with optimal combination of all Mk.
- With reference to
FIG. 6 , the output of the resultingknowledge engine 22 will contain the set of decision rules 36 for each of the components as depicted in the sample table ofFIG. 6 , which aids in classifying the data into good and bad pool. ComparingFIG. 6 withFIG. 3 illustrates how the decision rules may be formulated as maintenance recommendations as shown in the rightmost column ofFIG. 6 . - In the following, some illustrative implementations of the
ETL 42 and application of theknowledge engine 22 shown inFIG. 1 are described. - The process of mapping of the
knowledge engine 22 with themachine log data 40 involves parsing of machine log data into system readable format and storing in database system, this process is theETL 42 indicated inFIG. 1 . The log data that is stored in database will be typically distributed in multiple tables of the database. The first step towards the goal of mapping of theknowledge engine 22 with themachine log 40 is to aggregate related log data from database. The log data can be related in two ways: (1) Temporal dependency of log information; and (2) shares a common component or sub-system. - With reference to
FIG. 7 , in the case of temporal dependency of log information, log data from databases is extracted and aligned in the temporal domain (increasing/decreasing) and temporal pattern-matching is used to extract dependent log information, e.g. as illustrated inFIG. 7 . - The detection and aggregation of log information that belong to same type of component/sub-system can be performed using various approaches to detect correlating log data. One suitable approach is lexical analysis of content of log information and correlating lexical analysis output of two or more instances of log. A threshold on correlated value is used to categorize whether a given set of log information are related or not. Another illustrative approach is to count appearance or non-appearance of two or more error log data in given timeframe (typically data generated over a single day) and categorize as belong to same or different groups. Yet another illustrative approach is to compute mutual exclusivity of given set of log information with one another, two or more log information data having least mutual exclusive value will be considered as belong to same group of errors.
- With reference to
FIG. 8 , once the log data is segregated into groups, the matching ofsystem log data 40 with theknowledge engine 22 is performed. To performmapping 50 of the log data, eachgroup 52 of segregated log information is sent to a splitter and lexical analyzer (SLA)unit 54. TheSLA unit 54 first splits log information into meaningful tokens (typically words) and then each token is cross-correlated with a set ofvocabulary words 56 derived from source that generated knowledge engine (e.g. electronic medical imaging device servicing and operating data such as theillustrative user manual 27,specification data 26, service manual 25). Each element in thevocabulary 56 is assigned weight based on its importance. The weightage is decided by using domain expertise and/or by using statistical techniques like counting of words appearing in knowledge engine, number of connectivity of a given word with unique different works, etc. Thevocabulary 56 is suitably derived fromCER 32. Themapper 50 uses weighted and tokenized data from theSLA unit 54 to map log information with entries (e.g. decision rules) of theknowledge engine 22 as at this stage both type of data (log data and knowledge engine rules) will be of same nature as well as employing common language owing to fact that log data is processed using the storedvocabulary 56. Themapping 50 can be achieved by techniques such as word similarity matching or correlating the words. - With reference to
FIG. 9 , a method suitably performed by the monitoring device ofFIG. 1 to construct the decision rules 36 of theknowledge engine 22 is illustrated. In an operation 60, the various electronic medical imaging device servicing and operatingdata knowledge engine builder 20. In anoperation 62, a reader module (e.g.illustrative NLP 30 and CER 32) is applied to extract component IDs and operating parameters from the manuals, as well as associated operating parameter ranges. In an operation 64, theknowledge engine 22 is built by operations including formulating the operating parameter ranges into the set of decision rules 36 for classifying medical imaging device log data as good or bad log data. As shown on the right-side ofFIG. 9 , during deployment of theknowledge engine 22, the log files 40 from themedical imaging device 2 are pre-processed by theETL 42. TheCER 32 is applied to the ETL-processed log data to extract component IDs and operating parameters and values thereof. This is suitably done by applying thesame CER 32 used in processing the manuals to the ETL-processed log data. In anoperation 66, the decision rules 36 of theknowledge engine 22 are mapped on the log file output by the ETL/CER processing chain. Theoperation 66 may be performed as already described with reference toFIG. 8 . In anoperation 68, the output of themapping 66 is used to segregate log data into good data (in which the operating parameters are in range as defined by the decision rules 36) or bad data (in which at least one operating parameter is out of range as defined by the decision rules 36). In anoperation 70, maintenance alerts 24 are issued based on detected bad data. In an optionaladaptive updating operation 72, the bad data may be used to update the decision rules 36 based on whether the maintenances alerts are determined to be accurate (e.g., based on information from theservice log 44 on whether the subject component was serviced or failed during the MTTF time interval after issuance of the maintenance alert). - With reference to
FIGS. 10 and 11 , an illustrative more detailed implementation of the reading operation 62 (e.g.illustrative NLP 30 and CER 32) is described. ReferencingFIG. 10 , the textual content of the rawmanual data 80 typically includes tables and text (e.g. paragraphs, sentences, bullet points, ordered lists, or so forth). Atable parser 82 parses table content into semantic units such as rows and/or cells, while atext parser 84 parses textual content into semantic units such as sentences or paragraphs. TheNLP 30 andCER 32 are then applied as previously described to extract component IDs and operating parameters and associated operating parameter ranges or values. In the example ofFIG. 10 , this latter processing is shown separately as avalue engine component 86, where parameter value association is performed on a sentence-wise, cell-wise, or other semantic unit basis, to generatecomponent-wise output 88. Turning toFIG. 11 , thecomponent-wise outputs 88 from the various electronic medical imaging device servicing and operatingdata assimilator 90 to generate the combined unstructured data table 92, as already described with reference toFIG. 5 . This data table 92 then serves as input to the knowledge engine building operation 64 ofFIG. 9 . - With returning reference to
FIG. 1 , during a field servicing task the user may want to perform fault isolation to determine the root cause and/or solution to a maintenance alert using a fault isolation flowchart. To this end,FIG. 1 shows aflowchart converter 100 implemented on theserver computer 10. To support the flowchart converter, theknowledge engine builder 20 further implements flowchart image extraction. For example, an image element of adocument knowledge engine builder 20 by detecting a caption of the image as a text block directly above or below the image in the document, and further detecting text content of that caption indicating a flow chart. This detection could entail detecting the keyword “flow chart” or “flowchart”, and/or performing NLP to detect the indication of a flowchart. Additionally or alternatively, the image may be identified as a flowchart by analyzing the image content, e.g. detection of rectangles, diamonds, or other geometric figures containing text and connected by connecting lines is a strong indication that the image depicts a flowchart. Theflowchart converter 100 then processes the image of the flowchart to generate a structured electronic representation of the flowchart. This is done for each flowchart identified in thedocuments database 101 of structured electronic representations of the respective flowcharts contained in the documents. Achatbot 102 can be implemented on acomputer 150 that includes at least one user input device 152 (e.g., a keyboard and a mouse) and adisplay device 154 for displaying a graphical use interface (GUI) 156. Achatbot 102 can be displayed on theGUI 156. Thechatbot 102 provides a convenient mechanism for user navigation of the structured electronic representations of the flowcharts generated by theflowchart converter 100 and stored in thedatabase 101. -
FIG. 12 shows an example of theflowchart converter 100. Theflowchart converter 100 includes a processflow digitization module 104, and a digitized process flow tochatbot conversion module 106 to generate the structured electronic representation of the flowchart for implementation by thechatbot 102. The processflow digitization module 104 is configured to receive a set of scannedprocess flow images 108 as an input, e.g. received from theknowledge engine builder 20 in the illustrative embodiment. Theimages 108 depict the process flow or flowcharts, and may for example be JPEG images, BMP images, PNG images, or the like. The processflow digitization module 104 is configured to convert theimages 108 into a process tree or graph, such as a directed graph (DG). -
FIG. 13 shows an example of animage 108 showing aflowchart 110. Theflowchart 110 includes shapes orblocks 112 typically used in flowcharts. For example, theflowchart 110 includes start/stop blocks (e.g., ovals), process or action blocks (e.g., rectangles), decision blocks (e.g., diamonds), andarrows 114 connecting theblocks 112. The flowchart may also include text in theblocks 112 and/or adjacent thearrows 114. Theillustrative flowchart 108 is fully captured in a single image, but this may not always be the case. Typically, if a flowchart is on multiple pages of a service manual, this is indicated in the caption and is also indicated by connector blocks in the flowchart. For example, in a conventional approach, a circle with a text label such as “A” or “ToFIG. 7b ” or “FromFIG. 7a ” or so forth graphically depicts the flow connections between the pages. -
FIG. 14 depicts aDG 116 corresponding to theflowchart 108 ofFIG. 13 . To generate theDG 116, the processflow digitization module 104 performs the following process on the image(s) 108. A pre-processing operation is performed to remove noise from the images. A skew correction operation s performed to align the orientation of theimages 108. A shape detection operation is performed to detect one or more of the blocks or shapes 112 (e.g., oval, rectangle, diamond,arrows 114, and so forth) and connecting arrows in the flow process. One approach for shape detection is to apply an edge detector (e.g. Sobel, difference of Gaussians, Canny algorithm, et cetera) to highlight/isolate edges, then apply shape-specific detection algorithms (e.g., circle detector, rectangle detector, and so forth, each operating on the basis of pixel connectivity and a priori knowledge of the geometry of the shape. Detection of arrow connectors is also performed after the edge detector is applied, and is based on connectivity of lines between the identified shapes. A text detection process (e.g., OCR) is performed to detect text in the detectedshapes 112, adjacent the detected shapes, or on top of the detected shapes (e.g. text on top of decision arrows 114). Each of the detected shapes and each block of detected text (which could be a word, one-line phrase, or contiguous multi-line block of text) are assigned a location anchor in theimage 108. These location anchors are used to associate text and shapes or text and arrow connectors. Once theshapes 112 and text are detected along with associated location anchors and connecting arrows, this information is analyzed across theimage 108 is performed to generate theDG 116. - To create the
DG 116, the detectedshapes 112 and text are parsed from top to bottom, and anode 118 is created for each shape. Eachnode 118 includes a parent link pointing to aparent node 120, child link(s) 122 connected tochild nodes 124, a node type (e.g., a process or decision step in the flowchart), and node text. In a second pass, theshapes 112 are again parsed from top to bottom and detected directedarrows 114 are used to connect thenodes 118 based on their proximity and direction toshapes 112 as present in theimage 108. - Once the
DG 116 is created, the digitized process flow tochatbot conversion module 106 is configured to convert theDG 116 to a structured dialog flow representation suitable for presentation by thechatbot 102. The digitized process flow tochatbot conversion module 106 converts the content of theDG 116 into a structured dialog flow representation that can be presented by the chat bot as a natural language conversation flow. The chatbot conversation flow is represented in a structured dialog flow representation, such as JavaScript Object Notation (JSON) representation 130. - An example of a JSON representation 130 is provided below:
-
{ ″publishTime″: 0, ″sessions″: [{ ″publishTime″: 0, ″topics″: [{ ″dialogues″: [{ ″actionType″: ″″, ″sequence″: ″″, ″previous″: ″″, ″actionValue″: ″″, ″deliveryMode″: ″″, ″Break″: false, ″alternatives″: [ ], ″id″: ″5ee1febe149b3f0001dcc105″, ″text″: ″Error Displayed on screen 208 with 351 (Note this Error is observed with FA system only)″, ″disableOpenAnswer″: false, ″type″: ″text″, ″characterId″: ″bot″ }, { ″previous″: ″5ee1febe149b3f0001dcc105″, ″actionValue″: ″″, ″Break″: false, ″label″: ″″, ″type″: ″text″, ″actionType″: ″″, ″sequence″: ″″, ″deliveryMode″: ″″, ″alternatives″: [ ], ″id″ : ″5ee1fed4149b3f0001dcc106″, ″text″: ″Do tube conditioning procedure″, ″disableOpenAnswer″: false, ″characterId″: ″bot″ },{ ″previous″ : ″5ee1fed4149b3f0001dcc106″, ″actionValue″: ″″, ″Break″: false, ″label″: ″″, ″type″: ″text″, ″actionType″: ″″, ″sequence″: ″″, ″deliveryMode″: ″″, ″alternatives″: [ ], ″id″ : ″5ee1feed149b3f0001dcc107″, ″text″: ″Does tube conditioning pass without error?″, ″disableOpenAnswer″: false, ″characterId″: ″bot″ },{ ″actionType″: ″″, ″sequence″: ″″, ″previous″ : ″5ee1feed149b3f0001dcc107″, ″actionValue″: ″″, ″deliveryMode″: ″″, ″Break″: false, ″alternatives″: [ ], ″id″ : ″5ee1fefc149b3f0001dcc108″, ″text″: ″Yes″, ″label″: ″″, ″disableOpenAnswer″: false, ″type″: ″text″ },{ ″actionType″: ″″, ″sequence″: ″″, ″previous″: ″5ee1feed149b3f0001dcc107″, ″actionValue″: ″″, ″deliveryMode″: ″″, ″Break″: false, ″alternatives″: [ ], ″id″: ″5ee1ff05149b3f0001dcc109″, ″text″: ″No″, ″label″: ″″, ″disableOpenAnswer″: false, ″type″: ″text″ },{ ″previous″: ″5ee1fefc149b3f0001dcc108″, ″actionValue″: ″″, ″Break″: false, ″label″: ″″, ″type″: ″action″, ″actionType″: ″Exit″, ″sequence″: ″″, ″deliveryMode″: ″″, ″alternatives″: [ ], ″id″: ″5ee1ff12149b3f0001dcc10a″, ″text″: ″″, ″disableOpenAnswer″: false, ″characterId″: ″bot″ },{ ″previous″: ″5ee1ff05149b3f0001dcc109″, ″actionValue″: ″″, ″Break″: false, ″label″: ″″, ″type″: ″text″, ″actionType″: ″″, ″sequence″: ″″, ″deliveryMode″: ″″, ″alternatives″: [ ], ″id″: ″5ee1ff24149b3f0001dcc10b″, ″text″: ″Replace monobloc″, ″disableOpenAnswer″: false, ″characterId″: ″bot″ },{ ″previous″: ″5ee1ff24149b3f0001dcc10b″, ″actionValue″: ″″, ″Break″: false, ″label″: ″″, ″type″: ″text″, ″actionType″: ″″, ″sequence″: ″″, ″deliveryMode″: ″″, ″alternatives″: [ ], ″id″: ″5ee1ff32149b3f0001dcc10c″, ″text″: ″Is error resolved?″, ″disableOpenAnswer″: false, ″characterId″: ″bot″ },{ ″actionType″: ″″, ″sequence″: ″″, ″previous″: ″5ee1ff32149b3f0001dcc10c″, ″actionValue″: ″″, ″deliveryMode″: ″″, ″Break″: false, ″alternatives″: [ ], ″id″: ″5ee1ff3f149b3f0001dcc10d″, ″text″: ″Yes″, ″label″: ″″, ″disableOpenAnswer″: false, ″type″: ″text″ },{ ″actionType″: ″″, ″sequence″: ″″, ″previous″: ″5ee1ff32149b3f0001dcc10c″, ″actionValue″: ″″, ″deliveryMode″: ″″, ″Break″: false, ″alternatives″: [ ], ″id″: ″5ee1ff49149b3f0001dcc10e″, ″text″: ″No″, ″label″: ″″, ″disableOpenAnswer″: false, ″type″: ″text″ },{ ″previous″: ″5ee1ff3f149b3f0001dcc10d″, ″actionValue″: ″″, ″Break″: false, ″label″: ″″, ″type″: ″action″, ″actionType″: ″Exit″, ″sequence″: ″″, ″deliveryMode″: ″″, ″alternatives″: [ ], ″id″: ″5ee1ff56149b3f0001dcc10f″, ″text″: ″″, ″disableOpenAnswer″: false, ″characterId″: ″bot″ }, { ″previous″: ″5ee1ff49149b3f0001dcc10e″, ″actionValue″: ″″, ″Break″: false, ″label″: ″″, ″type″: ″text″, ″actionType″: ″″, ″sequence″: ″″, ″deliveryMode″: ″″, ″alternatives″: [ ], ″id″: ″5ee1ff67149b3f0001dcc110″, ″text″: ″Replace SCPU″, ″disableOpenAnswer″: false, ″characterId″: ″bot″ },{ ″previous″: ″5ee1ff67149b3f0001dcc110″, ″actionValue″: ″″, ″Break″: false, ″label″: ″″, ″type″: ″action″, ″actionType″: ″Exit″, ″sequence″: ″″, ″deliveryMode″: ″″, ″alternatives″: [ ], ″id″: ″5ee1ff75149b3f0001dcc111″, ″text″: ″″, ″disableOpenAnswer″: false, ″characterId″: ″bot″ }], ″publishTime″: 0, ″sequence″: ″″, ″firstTopicInSession″: true, ″name″: ″Topic 1″, ″description″: ″First topic″, ″id″: ″5ee1fe4f149b3f0001dcc104″ }], ″name″: ″Session 1″, ″description″: ″First session″, ″id″: ″5ee1fe4f149b3f0001dcc103″ }], ″defaultLanguage″: ″English″, ″name″: ″IGT Fault Isolation Scripts″, ″description″: ″Program for IGT fault isolation scripts″, ″id″: ″5ee1fe4f149b3f0001dcc102″, ″contentType″: ″Dialogue flow″ } - The
nodes 118 in the process flow (excepting the first, e.g. “start” node) each contains a link to a previous node or theparent node 120. Anode 118 is also differentiated as bot or user node to identify bot and user dialogues in a conversation flow. A user dialog node may, for example, receive the result of a test performed by the user in response to a preceding bot dialog node instructing to perform that test. Based on the received test result, flow may pass to one of several possible child nodes each of which is a bot dialog node presenting the next step of the flow chart. Theparent node 120 in the process flow is delivered as the first bot dialogue when the chat bot conversation is initiated. Thereafter, the user response is matched against theuser nodes 118 in the flow, which are linked to the previous delivered bot dialogue. Various string similarity-matching techniques like cosine similarity, shortest distance, etc. can be used to find the highest match. The matched node is then used to find the bot node linked to it, is delivered as the next bot dialogue in the conversation. The conversation continues using the same approach. - With continuing reference to
FIGS. 1 and 12-15 , theserver computer 10 is configured as described above to perform method orprocess 200 of converting theflowchart 110 to a structured electronic representation, such as theDG 116, and to convert theDG 116 to a structured dialog flow representation. Thenon-transitory storage medium 12 stores instructions which are readable and executable by theelectronic processors process 200. In some examples, themethod 100 may be performed at least in part by cloud processing. - With reference to
FIG. 16 , an illustrative embodiment of an instance of themethod 200 is diagrammatically shown as a flowchart. At anoperation 202, a plurality ofshapes 112 corresponding to flowchart blocks of theflowchart 110 are identified in one ormore images 108 of the flow chart. At anoperation 204, one ormore arrows 114 defining flow paths between the flowchart blocks in theimage 108 are identified. In some examples,arrows 114 can be identified by flowchart blocks connected by the arrows and identifying directionality of the defined flow paths based on arrowheads of the arrows. - At an
operation 206, text labels, and their locations in theimage 108, are identified. In addition, an OCR operation can be performed on theimage 108 to determine text content of the identified text labels. As used herein, the term “text labels” (and variants thereof) refers to a portion of theimage 108 containing text, while the term “text content” refers to content that has undergone an OCR process. That is, the text content comprises a string of ASCII characters (or other suitable characters, such as Chinese or Indian characters). Moreover, the OCR process is not limited toimages 108 of theflowchart 110 generated by optical scanning. In some embodiments, theoperation 206 includes extracting text corresponding to location anchors in theimage 108. The locations anchors serve as a location marker for the identified text. For example, the location anchor might be a single point (e.g., center of a block) or the location anchor might be an entire footprint of the shape 112 (orarrow 114 or text label) in theimage 108. The “footprint” of a block orshape 112 can be its outline or perimeter, while the “footprint” of anarrow 14 can be a line following the arrow. The “footprint” of a text label is a polygon enclosing and closely fitted to the area of the text in theimage 108. - At an
operation 208, the text labels (from the operation 206) are associated with flowchart blocks, or defined flow paths in the flowchart, based on the locations of the text labels, the flowchart blocks or shapes 112, andarrow 114 in theimage 108. In some examples, the associatingoperation 208 includes determining the distances from the footprint of the text label to the footprints of the various blocks or shapes 112 andarrows 114, and associating the text label to the block or arrow of shortest distance. In some examples, a label “Yes” on anarrow 114 coming from adecision diamond shape 112 might be as close to the diamond as to the arrow, so a secondary consideration might be the text content of the text label. For example, text content of “Yes” or “No” is likely to be labeling anarrow 114. Furthermore, text content inside a block orshape 112 is almost certainly properly associated to that block (so the distance is zero). - At an
operation 210, a structured electronic representation, such as theDG 116, is generated flowchart based on the flowchart blocks or shapes 112, the flow paths, and the text labels. A structure of the structuredelectronic representation 116 is determined at least based at least on the flow paths between the flowchart blocks or shapes 112 of theimage 108. To do so, in some embodiments, shape-indicated functions of the flowchart blocks 112 can be determined by comparing the identified shapes with standard flow chart shapes representing corresponding functions. For example, a flowchart typically uses oval shapes for start and stop operations, rectangles to represent process blocks, and diamonds to represent decision blocks. The “standard” blocks are compared with theshapes 112 identified in theoperation 202. - In other embodiments, functions are assigned to the identified flowchart blocks or shapes 112 based on the text labels associated to the flowchart blocks. Additionally or alternatively, functions are assigned to the identified flowchart blocks or shapes 112 based on the text labels associated to the defined flow paths. If the
image 108 incorrectly shows a square representing a decision block (e.g., a decision block should be a diamond according to the standard flow chart shapes), it may be possible to recognize the function is a decision block because it will have two (or more) outgoing arrows (e.g., one labeled “Yes” followed if the decision is Yes, the other labeled “No” for if the decision is No). In fact, any time a block has two or more outgoing arrows 114 (regardless of the labels) it must be a decision block since the block must “decide” which outgoing arrow to follow. The text labels associated to the flow paths can likely be useful in determining which paths are which. - In some embodiments, the identifying
operation 202 of theshapes 112, the identifyingoperation 204 of thearrows 114, and the generating of the structuredrepresentation 116 can be repeated for each image in a set ofimages 108 for the flow chart. For example, a first page of aflowchart 110 can end with an arrow to a circle block labeled “A”, and then the next page begins with the same circle block labeled “A” and moves on to the next block. In this example, a connecting block orshape 112 in different pages can be identified, and the structuredrepresentation 116 can be generated based on a connection indicated by the connecting blocks. - In other embodiments, the structured
representation 116 can be transferred to the computer 150 (seeFIG. 1 ) operable by a user. The structuredrepresentation 116 can be displayed on thedisplay device 154 of thecomputer 150. The user can then enter, via the at least oneuser input device 152, one or more inputs indicative of editing a portion of the structuredelectronic representation 116. The structuredrepresentation 116 can then updated based on the user inputs. - In some embodiments, as noted, the structured
electronic representation 116 can comprise theDG 116. In these embodiments, themethod 200 can include anoperation 212 that includes converting the directedgraph 116 to a structured dialog flow representation (which can be considered another form of structured electronic representation of the flowchart), such as the JSON representation 130. In these embodiments, the JSON representation 130 can be presented on theGUI 156 on thecomputer 150. TheGUI 156 can comprise, in some examples, thechatbot 102. Thechatbot 102 is configured to guide the user through the JSON representation 130. To do so, a current flowchart block of the JSON representation 130 is presented on thechatbot 102. An input is received from the user by the chatbot 102 (e.g., a mouse click, or text input as a question). The current flowchart block can be updated based on the structure of the JSON representation 130 and the received input. These processes are repeated to guide the user through theflowchart 110. - In the illustrative embodiments, the flowchart that is converted to a structured electronic representation (e.g. DG or structured dialog flow representation) is a fault isolation flowchart used in conjunction with diagnosing a problem with a medical imaging device. More generally, the disclosed approach can be employed to generate a structured electronic representation, and optionally a structured dialog flow representation presented by a chatbot, for fault isolation flowcharts employed in conjunction with other types of servicing tasks (e.g. other medical devices such as infusion pumps, mechanical ventilators, or so forth, or even more generally servicing tasks for other complex systems/devices such as aircraft, locomotives, HVAC systems, and/or so forth.
- The invention has been described with reference to the preferred embodiments. Modifications and alterations may occur to others upon reading and understanding the preceding detailed description. It is intended that the exemplary embodiment be construed as including all such modifications and alterations insofar as they come within the scope of the appended claims or the equivalents thereof.
Claims (20)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US16/986,360 US20200365262A1 (en) | 2018-03-23 | 2020-08-06 | Self-correcting method for annotation of data pool using feedback mechanism |
Applications Claiming Priority (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US201862646993P | 2018-03-23 | 2018-03-23 | |
PCT/EP2019/055526 WO2019179765A1 (en) | 2018-03-23 | 2019-03-06 | Self-correcting method for annotation of data pool using feedback mechanism |
US16/986,360 US20200365262A1 (en) | 2018-03-23 | 2020-08-06 | Self-correcting method for annotation of data pool using feedback mechanism |
Related Parent Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/EP2019/055526 Continuation-In-Part WO2019179765A1 (en) | 2018-03-23 | 2019-03-06 | Self-correcting method for annotation of data pool using feedback mechanism |
Publications (1)
Publication Number | Publication Date |
---|---|
US20200365262A1 true US20200365262A1 (en) | 2020-11-19 |
Family
ID=65686880
Family Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US16/981,331 Active 2041-07-09 US11967421B2 (en) | 2018-03-23 | 2019-03-06 | Self-correcting method for annotation of data pool using feedback mechanism |
US16/986,360 Pending US20200365262A1 (en) | 2018-03-23 | 2020-08-06 | Self-correcting method for annotation of data pool using feedback mechanism |
Family Applications Before (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US16/981,331 Active 2041-07-09 US11967421B2 (en) | 2018-03-23 | 2019-03-06 | Self-correcting method for annotation of data pool using feedback mechanism |
Country Status (4)
Country | Link |
---|---|
US (2) | US11967421B2 (en) |
EP (1) | EP3769318B1 (en) |
CN (1) | CN111886653A (en) |
WO (1) | WO2019179765A1 (en) |
Cited By (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20200285810A1 (en) * | 2019-03-05 | 2020-09-10 | App2Check S.r.l. | System and method for extracting information from unstructured or semi-structured textual sources |
US11263555B2 (en) * | 2019-05-06 | 2022-03-01 | Sap Se | Generating explanations of machine learning predictions in matching problems |
CN114898391A (en) * | 2022-07-12 | 2022-08-12 | 苏州阿尔脉生物科技有限公司 | Method and device for determining chemical reaction route and electronic equipment |
US11456912B2 (en) * | 2019-03-25 | 2022-09-27 | International Business Machines Corporation | Automatic suppression of alerts during maintenance |
Families Citing this family (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
EP3955707A1 (en) * | 2020-08-12 | 2022-02-16 | Danlex EOOD | Method and system for predicting a failure probability of a component of an x-ray system |
WO2023083647A1 (en) * | 2021-11-09 | 2023-05-19 | Koninklijke Philips N.V. | Systems and methods for extracting diagnostic and resolution procedures from heterogenous information sources |
CN116167365B (en) * | 2023-04-18 | 2023-06-30 | 安徽思高智能科技有限公司 | Flow chart generation method based on form template |
Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2012037429A2 (en) * | 2010-09-16 | 2012-03-22 | Siemens Corporation | Failure prediction and maintenance |
US20130110424A1 (en) * | 2011-10-28 | 2013-05-02 | General Electric Company | Apparatus and method to detect power |
US20140046697A1 (en) * | 2010-09-01 | 2014-02-13 | Robert Deward Rogers | Medical information navigation engine (mine) system |
US20180336081A1 (en) * | 2013-04-29 | 2018-11-22 | Moogsoft Inc. | System and methods for decomposing events from managed infrastructures that includes a feedback signalizer functor |
Family Cites Families (19)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US4853946A (en) * | 1986-11-14 | 1989-08-01 | Picker International, Inc. | Diagonostic service system for CT scanners |
US5463768A (en) | 1994-03-17 | 1995-10-31 | General Electric Company | Method and system for analyzing error logs for diagnostics |
US6442542B1 (en) | 1999-10-08 | 2002-08-27 | General Electric Company | Diagnostic system with learning capabilities |
US6650949B1 (en) | 1999-12-30 | 2003-11-18 | General Electric Company | Method and system for sorting incident log data from a plurality of machines |
US6598179B1 (en) | 2000-03-31 | 2003-07-22 | International Business Machines Corporation | Table-based error log analysis |
US7260249B2 (en) * | 2002-09-27 | 2007-08-21 | Confirma Incorporated | Rules-based approach for processing medical images |
US7853546B2 (en) * | 2007-03-09 | 2010-12-14 | General Electric Company | Enhanced rule execution in expert systems |
US20090089237A1 (en) * | 2007-09-28 | 2009-04-02 | General Electric Company | Method and system for remotely updating detection knowledge of systems |
CN103536287B (en) * | 2012-07-11 | 2015-11-25 | 上海联影医疗科技有限公司 | The remote monitoring maintenance system of medical imaging devices and method |
US10579928B2 (en) * | 2012-09-17 | 2020-03-03 | Siemens Aktiengesellschaft | Log-based predictive maintenance using multiple-instance learning |
US20140266713A1 (en) * | 2013-03-14 | 2014-09-18 | Carefusion 303, Inc. | Predictive Maintenance For Medical Devices |
US9996670B2 (en) * | 2013-05-14 | 2018-06-12 | Zynx Health Incorporated | Clinical content analytics engine |
JP6482536B2 (en) * | 2013-06-03 | 2019-03-13 | コーニンクレッカ フィリップス エヌ ヴェKoninklijke Philips N.V. | Medical device alert signal processing |
CN104239158A (en) | 2013-06-07 | 2014-12-24 | Sap欧洲公司 | Analysis engine for automatic analysis and error log linking |
JP6199486B2 (en) * | 2013-06-18 | 2017-09-20 | コーニンクレッカ フィリップス エヌ ヴェKoninklijke Philips N.V. | Processing medical device status information |
CN104461842B (en) | 2013-09-23 | 2018-02-16 | 伊姆西公司 | Based on daily record similitude come the method and apparatus of handling failure |
US10019305B2 (en) * | 2013-12-11 | 2018-07-10 | Robert Bosch Gmbh | Internet-based diagnostic assistant for device analysis |
CN104734901B (en) * | 2013-12-23 | 2019-06-28 | 武汉联影医疗科技有限公司 | A kind of management system and its method of medical imaging device availability |
US9552249B1 (en) | 2014-10-20 | 2017-01-24 | Veritas Technologies | Systems and methods for troubleshooting errors within computing tasks using models of log files |
-
2019
- 2019-03-06 EP EP19709470.9A patent/EP3769318B1/en active Active
- 2019-03-06 CN CN201980021383.1A patent/CN111886653A/en active Pending
- 2019-03-06 WO PCT/EP2019/055526 patent/WO2019179765A1/en active Application Filing
- 2019-03-06 US US16/981,331 patent/US11967421B2/en active Active
-
2020
- 2020-08-06 US US16/986,360 patent/US20200365262A1/en active Pending
Patent Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20140046697A1 (en) * | 2010-09-01 | 2014-02-13 | Robert Deward Rogers | Medical information navigation engine (mine) system |
WO2012037429A2 (en) * | 2010-09-16 | 2012-03-22 | Siemens Corporation | Failure prediction and maintenance |
US20130110424A1 (en) * | 2011-10-28 | 2013-05-02 | General Electric Company | Apparatus and method to detect power |
US20180336081A1 (en) * | 2013-04-29 | 2018-11-22 | Moogsoft Inc. | System and methods for decomposing events from managed infrastructures that includes a feedback signalizer functor |
Cited By (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20200285810A1 (en) * | 2019-03-05 | 2020-09-10 | App2Check S.r.l. | System and method for extracting information from unstructured or semi-structured textual sources |
US11456912B2 (en) * | 2019-03-25 | 2022-09-27 | International Business Machines Corporation | Automatic suppression of alerts during maintenance |
US11263555B2 (en) * | 2019-05-06 | 2022-03-01 | Sap Se | Generating explanations of machine learning predictions in matching problems |
CN114898391A (en) * | 2022-07-12 | 2022-08-12 | 苏州阿尔脉生物科技有限公司 | Method and device for determining chemical reaction route and electronic equipment |
Also Published As
Publication number | Publication date |
---|---|
WO2019179765A1 (en) | 2019-09-26 |
CN111886653A (en) | 2020-11-03 |
EP3769318A1 (en) | 2021-01-27 |
US20210012891A1 (en) | 2021-01-14 |
US11967421B2 (en) | 2024-04-23 |
EP3769318B1 (en) | 2024-05-08 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US20200365262A1 (en) | Self-correcting method for annotation of data pool using feedback mechanism | |
AU2019383320B2 (en) | Visualizing comment sentiment | |
US20180107801A1 (en) | Automatic disease detection from unstructured textual reports | |
US10667794B2 (en) | Automatic detection of disease from analysis of echocardiographer findings in echocardiogram videos | |
CN110720124A (en) | Monitoring patient speech usage to identify potential speech and associated neurological disorders | |
US20220028507A1 (en) | Workflow for automatic measurement of doppler pipeline | |
CN104598367A (en) | System and method for automatically managing fault events of data center | |
US10628749B2 (en) | Automatically assessing question answering system performance across possible confidence values | |
US11645556B2 (en) | Generating software services for a software solution determined from analysis of unstructured text input using machine learning | |
US10617396B2 (en) | Detection of valve disease from analysis of doppler waveforms exploiting the echocardiography annotations | |
CN106776728A (en) | Code check method, system and development platform based on log file analysis | |
US20230418721A1 (en) | System and method for automated or semi-automated identification of malfunction area(s) for maintenance cases | |
US20230119186A1 (en) | Methods, apparatuses and computer program products for predicting measurement device performance | |
US11822578B2 (en) | Matching machine generated data entries to pattern clusters | |
CN111309585A (en) | Log data testing method, device and system, electronic equipment and storage medium | |
CN110738054B (en) | Method, system, electronic equipment and storage medium for identifying hotel information in mail | |
Dixit et al. | Extracting semantics from maintenance records | |
Dobrowolski et al. | Software Failure Log Analysis for Engineers | |
EP4177905A1 (en) | Systems and methods for extracting diagnostic and resolution procedures from heterogenous information sources | |
JP7393034B2 (en) | Information provision system, information provision method and program | |
CA3088692C (en) | Visualizing comment sentiment | |
Thulasy et al. | Document mining on aircraft maintenance human errors and skillset mismatch issues | |
WO2023083647A1 (en) | Systems and methods for extracting diagnostic and resolution procedures from heterogenous information sources | |
WO2023084518A1 (en) | Method, system and computer program product for management and review of product consumer medicine information | |
CN116362257A (en) | Bug grading method and grading device based on semantic analysis technology |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: KONINKLIJKE PHILIPS N.V., NETHERLANDS Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:SREENIVASAN, RITHESH;MOORKAN, THASNEEM;PATIL, RAVINDRA BALASAHEB;AND OTHERS;REEL/FRAME:053533/0113 Effective date: 20200819 |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NON FINAL ACTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NON FINAL ACTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: FINAL REJECTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: RESPONSE AFTER FINAL ACTION FORWARDED TO EXAMINER |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |