Classifications

• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F18/00—Pattern recognition
  • G06F18/20—Analysing
  • G06F18/25—Fusion techniques
  • G06F18/254—Fusion techniques of classification results, e.g. of results related to same input data
  • G06F18/256—Fusion techniques of classification results, e.g. of results related to same input data of results relating to different input data, e.g. multimodal recognition
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V10/00—Arrangements for image or video recognition or understanding
  • G06V10/40—Extraction of image or video features
  • G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
  • G06V10/443—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components by matching or filtering
  • G06V10/449—Biologically inspired filters, e.g. difference of Gaussians [DoG] or Gabor filters
  • G06V10/451—Biologically inspired filters, e.g. difference of Gaussians [DoG] or Gabor filters with interaction between the filter responses, e.g. cortical complex cells
  • G06V10/454—Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V2201/00—Indexing scheme relating to image or video recognition or understanding
  • G06V2201/03—Recognition of patterns in medical or anatomical images

Definitions

• the present invention in some embodiments thereof, relates to medical anatomical images and, more specifically, but not exclusively, to systems and methods for pre-processing images for feeding into a classification neural network.
• a system for prioritizing patients for treatment for an acute medical condition requiring early and rapid treatment thereof based on a created a triage list of anatomical images likely depicting a visual finding type indicative of the acute medical condition comprises: at least one hardware processor executing a code for: feeding each one of a plurality of anatomical images into a visual filter neural network for outputting a classification category indicative of a target body region depicted at a target sensor orientation and a rotation relative to a baseline, rejecting a sub-set of the plurality of anatomical images classified into another classification category, rotating to the baseline a remaining sub- set of the plurality of anatomical images classified as rotated relative to the baseline, identifying pixels for each respective image of the plurality of anatomical images having outlier pixel intensity values denoting an injection of content, adjusting the outlier pixel intensity values of the identified pixels to values computed as a function of non-outlier pixel intensity values, feeding each one of the remaining sub- set of the plurality of anatomic
• a system for training a visual filter neural network for selection of anatomical images for inputting into a classification neural network for detecting a visual finding type indicative of an acute medical condition for early and rapid treatment thereof comprises: receiving a target body region and a target sensor orientation of a target anatomical image defined by the classification neural network, creating a training dataset by labeling each one of a plurality of anatomical images stored by a medical imaging storage server with a respective label indicative of a target body region captured at a target sensor orientation defined by the classification neural network and a rotation relative to a baseline, or with a respective label indicative of at least one of a non-target body region and a non-target sensor orientation, and training the visual filter neural network based on the training dataset, for classifying a target anatomical image into a classification category indicative of the target body region depicted at the target sensor angle and the rotation relative to the baseline, or into another classification category indicative of at least one of a non-target body region and a non-target sensor orientation, wherein
• a system for increasing accuracy of a classification neural network in detecting a visual finding type indicative of an acute medical condition for early and rapid treatment thereof comprising at least one hardware processor executing a code for: receiving a plurality of anatomical images from a medical imaging storage server, feeding each one of the plurality of anatomical images into a visual filter neural network for outputting a classification category indicative of a target body region depicted at a target sensor orientation and a rotation relative to a baseline defined by the classification neural network, or another classification category indicative of at least one of a non-target body region and a non-target sensor orientation, rejecting a sub-set of the plurality of anatomical images classified into the another classification category, to obtain a remaining sub- set of the plurality of anatomical images, rotating to the baseline the remaining sub-set of the plurality of anatomical images classified as rotated relative to the baseline, creating a training dataset from the remaining sub-set of the plurality of anatomical images, and training a classification neural network according to the
• a system for increasing accuracy of a classification neural network in detecting a visual finding type indicative of an acute medical condition for early and rapid treatment thereof comprises: at least one hardware processor executing a code for: receiving a plurality of anatomical images from a medical imaging storage server, feeding each one of the plurality of anatomical images into a visual filter neural network for outputting a classification category indicative of a target body region depicted at a target sensor orientation and a rotation relative to a baseline defined by the classification neural network, or another classification category indicative of at least one of a non-target body region and a non-target sensor orientation, rejecting a sub-set of the plurality of anatomical images classified into the another classification category, to obtain a remaining sub- set of the plurality of anatomical images, rotating to the baseline the remaining sub-set of the plurality of anatomical images classified as rotated relative to the baseline, and feeding each one of the remaining sub- set of the plurality of anatomical images into the classification neural network for detecting the visual finding type
• accuracy of the classification neural network in detecting the visual finding type indicative of the acute medical condition is increased for the remaining sub- set of the plurality of anatomical images in comparison to detecting the visual finding type for the plurality of anatomical images by the classification neural network without rejection of any anatomical images by the visual filter neural network.
• accuracy of the classification neural network in detecting the visual finding type indicative of the acute medical condition is increased for the remaining sub-set of the plurality of anatomical images with adjusted outlier pixel intensity values and rotation to baseline, in comparison to detecting the visual finding type for the plurality of anatomical images by the classification neural network without rejection of any anatomical images by the visual filter neural network, without adjustment of outlier pixel intensity values, and without rotation to baseline.
• the system further comprises code for and/or the method further comprises at least one of: diagnosing the acute medical condition and treating the patient for the acute medical condition.
• the visual filter neural network selects chest x-rays depicting at least one of AP and PA orientation, and rejects at least one of non-chest x-rays and lateral orientation.
• the visual filter neural network is installed client-side, on a client terminal in communication with the medical imaging storage server over a network, wherein the client terminal hosts the classification neural network.
• a single classification category is indicative of the depicted body region, the target sensor orientation and the rotation relative to the baseline.
• the classification neural network is trained according to a training dataset of training anatomical medical images that were not rejected by the visual filter neural network, had outlier pixel intensity values denoting injected content adjusted, and rotated to the baseline.
• the visual filter neural network outputs the classification category further indicative of a target imaging modality type or the another classification category further indicative of a non-target imaging modality type, wherein the rejected sub-set of the plurality of anatomical images include anatomical images classified into the another classification category.
• the plurality of anatomical images are stored by a medical imaging server according to a medical imaging storage format, and wherein the visual filter neural network rejects the sub-set of the plurality of anatomical images independently of metadata defined by the medical imaging storage format and associated with the respective anatomical image.
• the medical imaging server comprise a PACS server
• the medical imaging storage format is DICOM®
• the metadata of DICOM® stores an indication of the target body region and the target sensor orientation.
• the adjusting is performed for the respective image having outlier pixel intensity values stored with a pixel depth, that is different than a pixel depth of the respective image when presented on a display.
• the system further comprises code for and/or the method further comprises computing, for each respective image, a histogram of pixel intensity values, wherein the outlier pixel intensity values are selected based on one or two extreme bins of the histogram that are spaced apart from another bin by an empty bin that does not include any pixels.
• the outlier pixel intensity values are adjusted to a value computed as a function of the another bin and all pixels in the respective image.
• the function is computed one of: (i) a minimum of the pixel intensity values in the another bin, less a constant multiplied by the median pixel intensity values of all pixels in the respective image, and (ii) a maximum of the pixel intensity values in the another bin, added to a constant multiplied by the median pixel intensity values of all pixels in the respective image, and (iii) a smallest interest such that two raised to the smallest integer minus one is greater than the maximum of the pixel intensity values in the another bin.
• the classification neural network comprises a single-label neural network computed by at least one of fine-tuning and retraining a trained multi-label neural network according to a single-label training dataset of a plurality of anatomical images labeled with an indication of the visual finding type, wherein the multi-label neural network is trained to compute likelihood of each of a plurality of visual finding types based on a multi-label training dataset storing a plurality of anatomical images labeled with the plurality of visual finding types.
• the system further comprises code for and/or the method further comprises providing a plurality of classification neural networks, each designed for processing anatomical images of a certain combination of a plurality of combinations of target body region and target sensor orientation, providing a plurality of visual filter neural networks, each designed for classification of anatomical images into a classification category indicative of the certain combination of the plurality of combinations, wherein each certain visual filter neural network corresponds to a certain classification neural network, and feeding the plurality of anatomical images into each one of the plurality of classification neural networks to obtain a respective sub- set of the plurality of anatomical images, and feeding each respective sub-set of the plurality of anatomical images into the corresponding classification neural network.
• FIG. 1 is a flowchart of a process for adjusting pixel intensity values of injected content of anatomical images and/or using a visual filter neural network to exclude irrelevant anatomical images from being fed into a classification neural network that outputs an indication of likelihood of a visual finding type being depicted in the received anatomical image, in accordance with some embodiments of the present invention
• FIG. 2 is a diagram of components of a system for adjusting pixel intensity values of injected content of anatomical images and/or using a visual filter neural network to exclude irrelevant anatomical images from being fed into a classification neural network and/or for training the visual filter neural network and/or for creating the visual filter neural network, in accordance with some embodiments of the present invention
• FIG. 3 is a flowchart of a process for creating the visual filter neural network, in accordance with some embodiments of the present invention.
• FIG. 4 is a schematic of a histogram computed for the original pixel intensity values, and an adjusted histogram that corrects for injected content, in accordance with some embodiments of the present invention
• FIG. 5 includes an AP/PA chest x-ray before adjustment of pixel intensity values denoting injected content, and an x-ray depicting the AP/PA chest x-ray after adjustment of pixel intensity values denoting injected content, in accordance with some embodiments of the present invention.
• FIG. 6 is a graph of ROC curves computed for the experiments for computationally evaluating the visual filter neural network in increasing classification accuracy of a classification neural network, in accordance with some embodiments of the present invention. DESCRIPTION OF SPECIFIC EMBODIMENTS OF THE INVENTION
• the present invention in some embodiments thereof, relates to medical anatomical images and, more specifically, but not exclusively, to systems and methods for pre-processing images for feeding into a classification neural network.
• the term sensor orientation refers to the orientation of the patient relative to the imaging modality sensor and/or receiver (e.g., Anterior-posterior (AP), PA, lateral), and may include the anatomical orientation of the body of the patient during capture of the image (e.g., left lateral decubitus, supine).
• AP Anterior-posterior
• PA PA
• lateral the orientation of the body of the patient during capture of the image
• the anatomical orientation of the body of the patient during capture of the image e.g., left lateral decubitus, supine.
• An aspect of some embodiments of the present invention relates to systems, methods, an apparatus, and/or code instructions (i.e., stored on a data storage device and executable by one or more hardware processor(s)) for treatment of a patient suffering from an acute medical condition requiring early and rapid treatment thereof (e.g., pneumothorax, fracture, acute appendicitis, pneumoperitoneum, pneumomediastinum) based on a created a triage list of anatomical images likely depicting a visual finding type indicative of the acute medical condition.
• Anatomical images are obtained, optionally from a medical imaging server, such as a PACS server storing images according to the DICOM® format.
• Each one of the anatomical images is fed into a visual filter neural network for outputting a classification category indicative of a target body region depicted at a target sensor orientation and a rotation relative to a baseline, for example, AP/PA chest x-ray, rotated by 90 degrees clockwise.
• the target body region depicted at target sensor orientation is defined by the classification neural network architecture, for example, designed to detect pneumothorax in AP/PA chest x-rays.
• a sub-set of the anatomical images classified into another classification category i.e., not the classification category indicative of the target body region and sensor orientation
• Anatomical images classified as rotated from baseline are rotated back to baseline. Pixels of images having outlier pixel intensity values denoting an injection of content are identified.
• Injected content may include, for example, text and/or metadata injected into the image, such as patient ID, image data (e.g., imaging modality, type of image), and letters indicating side of the patient.
• the outlier pixel intensity values of the identified pixels are adjusted to values computed as a function of non-outlier pixel intensity values. It is noted that the pixel adjustment may be performed before and/or after processing by the visual filter neural network, and before and/or after rotation.
• the remaining sub-set of the anatomical images (which have passed through the visual filter neural network i.e., non-rejected, have been rotated, and with adjusted outlier pixel intensity values) are into the classification neural network for detecting the visual finding type.
• Instructions are generated for creating a triage list for which the classification neural network detected the indication, optionally ranked based on likelihood (e.g., probability) of the indication being depicted in the respective image,.
• Patients likely suffering from the acute medical condition denoted by the indication are selected for early and rapid treatment thereof based on the triage list, for example, based on a radiologist (or other healthcare worker) reviewing the images according to the triage list and/or based on a physician examining the patients according to the triage list.
• An aspect of some embodiments of the present invention relates to systems, methods, an apparatus, and/or code instructions (i.e., stored on a data storage device and executable by one or more hardware processor(s)) for increasing accuracy of a classification neural network in detecting a visual finding by a visual filter neural network used to filter irrelevant anatomical images prior to feeding into the classification neural network and/or the visual filter neural network used to exclude irrelevant anatomical images from a training dataset for training the classification neural network.
• Multiple anatomical images are obtained, for example, from a storage server such as a PACS server.
• Each of the anatomical images is fed into a visual filter neural network.
• the visual filter neural network may output a classification category indicating that the anatomical image is relevant for being fed into the classification neural network.
• the classification category denotes that the respective anatomical image depicts a target body region (e.g., chest) and/or a target sensor orientation (e.g., AP/PA), and/or a rotation relative to a baseline rotation (e.g., 90 degree clockwise, 180 degrees, 270 degree clockwise).
• the visual filter neural network may output another classification category indicative that the respective anatomical image depicts a non-target body region (e.g., non-chest) and/or a non-target sensor orientation (e.g., non- AP/PA).
• the images classified into the other classification category are rejected, leaving a remaining sub-set of anatomical images. Images of the remaining-subset classified as being rotated are re-rotated back to baseline.
• the remaining sub-set of anatomical images (which include the images rotated back to baseline) are fed into the classification neural network, and/or are used to create a training dataset for training the classification neural network.
• images for including in the training dataset for training the classification neural network are rotated to baseline and/or processed to adjust outlier pixel intensity values denoting injected content.
• the classification neural network may detect an indication of likelihood of a visual finding being depicted in the received anatomical image.
• the visual finding may denote an acute medical condition for early and rapid treatment thereof, for example, pneumothorax, pneumoperitoneum, pneumomediastinum, and fracture.
• An aspect of some embodiments of the present invention relates to systems, methods, an apparatus, and/or code instructions (i.e., stored on a data storage device and executable by one or more hardware processor(s)) for training a visual filter neural network for selection of anatomical images for inputting into a classification neural network and/or for selection of anatomical images for creating a training dataset for training the classification neural network.
• a target body region and/or a target sensor orientation are defined according to the classification neural network.
• the classification neural network has been trained to process anatomical images having the target body region and/or target sensor orientation for detecting likelihood of the visual finding.
• a training dataset is created from anatomical images (e.g., stored in a storage server, such as a PACS server) labeled with an indication of the target body region and target sensor orientation and rotation relative to baseline, or with a label indicative of non-target body region and/or non-target sensor orientation.
• the visual filter neural network is trained based on the training dataset, for classifying a target anatomical image into a classification category indicative of the target body region depicted at the target sensor angle and a rotation relative to the baseline, or into another classification category indicative of a non-target body region and/or a non-target sensor orientation.
• the visual filter neural network is installed client-side, on the client terminal in communication with the medical imaging storage server (e.g., PACS server).
• the client terminal hosts the classification neural network.
• the client terminal (or another client terminal) presents the results of the analysis by the classification neural network that is fed the filtered anatomical images.
• pixels of the anatomical images (before and/or after being passed through the visual filter neural network, before and/or after rotation) having outlier pixel intensity values are identified.
• the outlier pixel intensity values are indicative of injected content, for example, patient name, patient ID, and left or right side.
• the pixel intensity values of the identified outlier pixels are adjusted to values computed as a function of non-outlier pixel intensity values. By adjusting the pixel intensity values of the most extreme values, the dynamic range is improved, improve the ability of the classification neural network to detect fine features.
• Inventors discovered that using visual filter neural network to exclude irrelevant images (i.e., images that do not conform to the target body region and/or target sensor orientation and/or target anatomical imaging modality set for the classification neural network) increases the accuracy of the classification neural network in detecting likelihood of the respective anatomical image depicting a visual finding type.
• the accuracy is increased when the visual filter neural network is used to exclude images from a training dataset used to train the classification neural network, and/or exclude images from being fed into the classification neural network.
• the accuracy is increased in comparison to using the classification neural network without passing the anatomical images through the visual filter neural network and/or in comparison to training the classification neural network on a training dataset created from the images without passing the anatomical images through the visual filter neural network.
• adjusting the values of identified outlier pixel intensity values denoting injected content results in an increased accuracy of the classification neural network in detecting likelihood of a visual finding type being depicted in the respective anatomical image, and/or results in the increased accuracy of detecting the visual finding type when the classification neural network is trained using a training dataset of images for which identified outlier pixel intensity values have been adjusted.
• the adjustment of the extreme pixel intensity values improves the dynamic range, and/or improves accuracy of the classification neural network in detecting fine features that would otherwise be difficult to detect when the original pixel intensity values are maintained.
• the pixel intensities of an x-ray image usually lie in a smooth and continuous range (also termed herein as the normal range).
• artificial pixels that were injected synthetically e.g., text labels, such as patient ID, patient name, indication of patient side (left or right)
• have gray levels that lie far above or below the above mentioned range and as a result could skew the computation of the network.
• These outlier pixel intensities are adjusted as described herein, to an intensity level that is closer to the normal range.
• the highest accuracy may be achieved with a combination of using the visual filter NN and/or rotating the images to baseline and/or adjusting the outlier pixel intensity values denoting injected for creating the training dataset for training the classification neural network, and/or for processing images being fed into the trained classification neural network.
• the visual finding may be an acute finding, which is not normally present, and representing a medical problem.
• the acute finding may progress or remain stable, but in either case it may be indicative of a situation that in which the clinical state of the patient is worsening.
• the acute finding may be indicative of the need for urgent medical treatment. Delay in treatment of the acute finding leads to increases in complications for the patient.
• the visual finding may be a fine feature, which may be easily missed by a radiologist.
• Examples of such acute, fine, easily missed visual findings include: pneumothorax in a chest x-ray, pneumomediastinum in a chest x-ray, and pneumoperitoneum in an abdominal x-ray, and fracture in a limb x-ray.
• the improvement provided by at least some of the systems, methods, apparatus, and/or code instructions described herein may include a reduction in the amount of time for alerting a user (e.g., treating physician) to the presence of a visual finding type in an anatomical image for rapid diagnosis and/or treatment thereof.
• At least some of the systems, methods, apparatus, and/or code instructions described herein improve the technical field of automated analysis of anatomical images by neural networks to identify likelihood of the presence of a visual finding in a medial image, optionally a fine visual finding, optionally representing an acute medical condition requiring urgent diagnosis and treatment, which may easily be missed by a radiologist.
• the improvement is to the field of neural networks and/or image processing.
• To identify such visual findings in anatomical images requires a classifier with high accuracy, which is not provided by any standard classifier.
• Such standard classifiers use an off the shelf classifier (e.g., neural network), and a training dataset of labeled anatomical images.
• the visual filter neural network which excludes irrelevant images increases the accuracy of the classification neural network, directly by excluding the irrelevant images from being fed into the classification neural network, and/or indirectly by excluding the irrelevant images from the training dataset used to train the classification neural network which increases the accuracy of the trained classification neural network.
• the process of adjusting pixel intensity values of outlier pixels denoting injected content increases the accuracy of the classification neural network, directly by adapting the pixel intensity values of images being fed into the classification neural network, and/or indirectly by adapting the pixel intensity values of images of the training dataset used to train the classification neural network which increases the accuracy of the trained classification neural network.
• the increase in accuracy may be at least due to the formation of a‘pure’ image by removal of‘noise’ in the form of the injected content.
• the injected content for example, objects placed next to the patient for calibration, patient name and/or patient ID, and letter indicating left and/or right side, represent content irrelevant to detection of a visual finding type in the anatomical images.
• Presence of the injected content results in extra‘noise’ for the classifier neural network to process, which reduces accuracy without benefit. Removal of the noisy injected content increases the accuracy of the classifier neural network. It is noted that the injected content may be removed for creating the training dataset. In which case, the classification neural network trained on the training dataset with removed injected content, has an increase in accuracy when processing target anatomical images that have been processed to remove injected content.
• the accuracy of the classification neural network in detecting likelihood of a visual finding type being depicted in a target anatomical image is further improved by the visual filter neural network which removes dependence on DICOM® metadata (and/or other corresponding metadata of other medical data storage formats) that would otherwise be required.
• Classification neural networks operating without the visual filter neural network are based on access to the DICOM® metadata in order to determine the body portion depicted in the target anatomical image, and/or the sensor orientation at which the target anatomical image is captured and/or the imaging modality type used to capture the image.
• DICOM® metadata may be erroneous and/or missing, lowering the accuracy of providing the relevant image for processing by the relevant classification neural network, which leads to an overall lower accuracy of the classification neural network in correctly classifying the target anatomical image.
• the visual filter neural network described herein does not use DICOM® metadata (and/or other corresponding metadata of other storage formats).
• the visual filter neural network excludes irrelevant images and/or includes relevant images from the actual (e.g., raw) image alone without relying on DICOM® metadata.
• the disconnect from the DICOM® metadata indirectly increases the accuracy of the classification neural network by removing erroneous and/or missing DICOM® metadata as a source of error.
• the visual filter neural network may be installed client-side, optionally on the client terminal executing the classification neural network corresponding to the visual filter neural network.
• the client-side installation is in contrast to server-side installations of standard applications that work with the imaging server (e.g., PACS). For example, applications that integrated with the PACS server and/or PACS viewer.
• the visual filter neural network is independent of the PACS viewer used by a client terminal to view images stored on the PACS server.
• the client-side installation architecture described herein enables generation of multiple different types of visual filter neural networks, each for a corresponding classification neural network, all of which are fed from the same set of anatomical images stored on the storage server.
• the same x-rays are fed to one filter neural network for excluding non-chest and non- AP/PA x-ray (e.g., lateral view) which is associated with a classification neural network to detect pneumothorax, and to another filter neural network for excluding non-supine view abdominal x- rays for detecting pneumoperitoneum.
• the client-side architecture disconnects the visual filter neural network from reliance on the imaging server and/or image viewer, for example, not relying on DICOM® metadata.
• At least some of the systems, methods, apparatus, and/or code instructions described herein improve the medical process of diagnosis and/or treatment of acute medical conditions in a patient, for example, within an emergency room setting.
• At least some of the systems, methods, apparatus, and/or code instructions described herein provide a triage system that identifies likelihood of anatomical images (e.g., chest x-rays) including a visual finding indicating an acute medical condition requiring urgent treatment, for example, pneumothorax.
• the medical images having identified visual findings are triaged for priority viewing by a healthcare professional (e.g., radiologist, emergency room physician), for example, by ranking according to a priority score, for example, probability of the respective image having the visual finding.
• a healthcare professional e.g., radiologist, emergency room physician
• images likely having pneumothorax visual findings are prioritized, optionally according to computed probability of having the pneumothorax visual finding.
• the triage system enables rapid diagnosis of pneumothorax, which leads to rapid treatment of the pneumothorax, saving the patient from complication of delayed treatment of pneumothorax and/or missing the pneumothorax entirely.
• the triage system is enabled, at least due to the visual filter neural network that excludes irrelevant images from being fed into the classification neural network, and/or by the visual filter neural network that excludes irrelevant images from being included in the training dataset for training the classification neural network, and/or by the process of adapting outlier pixel intensity values of injected content of anatomical images being fed into the classification neural network and/or included in the training dataset.
• the present invention may be a system, a method, and/or a computer program product.
• the computer program product may include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present invention.
• the computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device.
• the computer readable storage medium may be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing.
• a non-exhaustive list of more specific examples of the computer readable storage medium includes the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, and any suitable combination of the foregoing.
• RAM random access memory
• ROM read-only memory
• EPROM or Flash memory erasable programmable read-only memory
• SRAM static random access memory
• CD-ROM compact disc read-only memory
• DVD digital versatile disk
• memory stick a floppy disk, and any suitable combination of the foregoing.
• a computer readable storage medium is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiber-optic cable), or electrical signals transmitted through a wire.
• Computer readable program instructions described herein can be downloaded to respective computing/processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and/or a wireless network.
• the network may comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and/or edge servers.
• a network adapter card or network interface in each computing/processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing/processing device.
• Computer readable program instructions for carrying out operations of the present invention may be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages.
• the computer readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server.
• the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider).
• electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGA), or programmable logic arrays (PLA) may execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present invention.
• These computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.
• These computer readable program instructions may also be stored in a computer readable storage medium that can direct a computer, a programmable data processing apparatus, and/or other devices to function in a particular manner, such that the computer readable storage medium having instructions stored therein comprises an article of manufacture including instructions which implement aspects of the function/act specified in the flowchart and/or block diagram block or blocks.
• the computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process, such that the instructions which execute on the computer, other programmable apparatus, or other device implement the functions/acts specified in the flowchart and/or block diagram block or blocks.
• each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s).
• the functions noted in the block may occur out of the order noted in the figures.
• two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved.
• FIG. 1 is a flowchart of a process for adjusting pixel intensity values of injected content of anatomical images and/or using a visual filter neural network to exclude irrelevant anatomical images from being fed into a classification neural network that outputs an indication of likelihood of a visual finding type being depicted in the received anatomical image, in accordance with some embodiments of the present invention.
• FIG. 1 also depicts a process for using the visual filter neural network to exclude irrelevant anatomical images from being included in the training dataset used to train the classification neural network.
• the visual filter neural network increases accuracy of the classification neural network.
• System 200 also includes components for using the visual filter neural network to exclude irrelevant anatomical images from being included in the training dataset used to train the classification neural network.
• FIG. 3 is a flowchart of a process for creating the visual filter neural network, in accordance with some embodiments of the present invention.
• System 200 may implement the acts of the method described with reference to FIG. 1 and/or FIG. 3, optionally by a hardware processor(s) 202 of a computing device 204 executing code instructions stored in a memory 206.
• classification neural network 222A is a single-label neural network computed from a multi-label neural network using a respective multi label training dataset and a single-label training dataset, as described with reference to co-filed Application having Attorney Docket No. 76282.
• each anatomical image e.g., chest x-ray
• visual filter neural network 206A for exclusion of irrelevant images (e.g., non-chest x-rays, and/or non-x-ray images and/or non AP-PA images).
• the chest x-ray images (before or after filtering) may be further processed for removal of outlier pixel intensity values and/or adjusting pixel intensity values by executing pixel adjustment code 206E.
• the system provides a triage of the anatomical images, by generating a priority worklist 222B.
• the worklist 222B is generated by ranking the chest x-rays according to a priority score computed based on the likelihood. The higher the probability that a certain chest x-ray has a visual finding indicating pneumothorax, the higher the ranking on the worklist.
• a healthcare practitioner e.g., radiologist, ER physician
• the healthcare practitioner is directed to the most urgent chest x-rays most likely to have a visual finding indicative of pneumothorax, reducing the time to diagnose and treat the patient for the pneumothorax in comparison to standard systems that do not provide the triage feature. Patients determined to have pneumothorax may be treated by a physician to remove the excess air.
• Computing device 204 may be implemented as, for example, a client terminal, a server, a virtual server, a radiology workstation, a virtual machine, a computing cloud, a mobile device, a desktop computer, a thin client, a Smartphone, a Tablet computer, a laptop computer, a wearable computer, glasses computer, and a watch computer.
• Computing 204 may include an advanced visualization workstation that sometimes is add-on to a radiology workstation and/or other devices for presenting indications of the visual finding type to the radiologist.
• Computing device 204 may include locally stored software that performs one or more of the acts described with reference to FIG. 1 and/or FIG. 3, and/or may act as one or more servers (e.g., network server, web server, a computing cloud, virtual server) that provides services (e.g., one or more of the acts described with reference to FIG. 1 and/or FIG.
• servers e.g., network server, web server, a computing cloud, virtual server
• services e.g., one or more of the acts described with reference to FIG. 1 and/or FIG.
• client terminals 208 e.g., client terminal used by a user for viewing anatomical images, remotely located radiology workstations, remote picture archiving and communication system (PACS) server, remote electronic medical record (EMR) server
• client terminals 208 e.g., client terminal used by a user for viewing anatomical images, remotely located radiology workstations, remote picture archiving and communication system (PACS) server, remote electronic medical record (EMR) server
• PPS remote picture archiving and communication system
• EMR remote electronic medical record
• Client terminal(s) 208 may be implemented as, for example, a radiology workstation, a desktop computer (e.g., running a PACS viewer application), a mobile device (e.g., laptop, smartphone, glasses, wearable device), and nurse station server.
• a radiology workstation e.g., running a PACS viewer application
• a mobile device e.g., laptop, smartphone, glasses, wearable device
• nurse station server e.g., a radiology workstation
• desktop computer e.g., running a PACS viewer application
• mobile device e.g., laptop, smartphone, glasses, wearable device
• the training of the visual filter neural network, and the application of the visual filter neural network to exclude irrelevant anatomical images may be implemented by the same computing device 204, and/or by different computing devices 204, for example, one computing device 204 trains the visual filter neural network and, and transmits the trained visual filter neural network to a server device 204.
• Computing device 204 receives 2D images, and/or 2D slices (optionally extracted from 3D imaging data) captured by an anatomical imaging device(s) 212, for example, an x-ray machine, a magnetic resonance imaging (MRI) device, a computer tomography (CT) machine, and/or an ultrasound machine.
• Anatomical images captured by imaging machine 212 may be stored in an image repository 214, for example, a storage server (e.g., PACS server), a computing cloud, virtual memory, and a hard disk.
• the anatomical images stored by image repository 214 may include images of patients optionally associated with text based radiology reports. Training images 216 are created based on the captured anatomical images and text based radiology reports, as described herein.
• Training images 216 may be used to train the visual filter neural network, as described herein.
• training images and training dataset may be interchanged. It is noted that training images 216 may be stored by a server 218, accessibly by computing device 204 over network 210, for example, a publicly available training dataset, and/or a customized training dataset created for training the visual filter neural network, as described herein.
• Anatomical images captured by imaging machine(s) 212 depict internal anatomical features and/or anatomical structures within the body of the target patient.
• Exemplary anatomical images include 2D x-ray images captured by an x-ray machine.
• Exemplary x-ray anatomical images include: AP and PA views of the chest, abdominal x-rays, and x-rays of limbs. Selected views of the x-ray images may be defined as the best view for detecting the visual finding type.
• Computing device 204 may receive the anatomical images for filtering, and/or receive training images 216 , from imaging device 212 and/or image repository 214 using one or more imaging interfaces 220, for example, a wire connection (e.g., physical port), a wireless connection (e.g., antenna), a local bus, a port for connection of a data storage device, a network interface card, other physical interface implementations, and/or virtual interfaces (e.g., software interface, virtual private network (VPN) connection, application programming interface (API), software development kit (SDK)).
• a wire connection e.g., physical port
• a wireless connection e.g., antenna
• local bus e.g., a port for connection of a data storage device
• a network interface card e.g., other physical interface implementations
• virtual interfaces e.g., software interface, virtual private network (VPN) connection, application programming interface (API), software development kit (SDK)
• Hardware processor(s) 202 may be implemented, for example, as a central processing unit(s) (CPET), a graphics processing unit(s) (GPET), field programmable gate array(s) (FPGA), digital signal processor(s) (DSP), and application specific integrated circuit(s) (ASIC).
• processors 202 may include one or more processors (homogenous or heterogeneous), which may be arranged for parallel processing, as clusters and/or as one or more multi core processing units.
• Memory 206 stores code instruction for execution by hardware processor(s) 202, for example, a random access memory (RAM), read-only memory (ROM), and/or a storage device, for example, non-volatile memory, magnetic media, semiconductor memory devices, hard drive, removable storage, and optical media (e.g., DVD, CD-ROM).
• RAM random access memory
• ROM read-only memory
• storage device for example, non-volatile memory, magnetic media, semiconductor memory devices, hard drive, removable storage, and optical media (e.g., DVD, CD-ROM).
• memory 206 may store image processing code 206A that implement one or more acts and/or features of the method described with reference to FIG. 1, and/or training code 206B that execute one or more acts of the method described with reference to FIG.
• trained classification neural network 222 A and/or code of visual filter neural network code 206C for filtering the anatomical images prior to processing by the trained single-label neural network and/or prior to being used for training the single-label and/or multi-label neural network and/or mini-AUC code 206D for selecting single label neural networks according to a target sensitivity and/or specificity and/or pixel adjustment code 206E for adjusting pixel intensity values for removal of outliers, as described herein.
• client terminal(s) may locally store and/or execute image processing code 206A, visual filter neural network 206C, and/or code instructions of trained classification neural network 222A and/or priority list 222B and/or mini-AUC code 206D and/or pixel adjustment code 206E.
• Computing device 204 may include a data storage device 222 for storing data, for example, code instructions of trained classification neural network 222 A, priority list 222B (generated as described herein), visual filter neural network 206C, mini-AUC code 206D, and/or training images 216.
• Data storage device 222 may be implemented as, for example, a memory, a local hard-drive, a removable storage device, an optical disk, a storage device, and/or as a remote server and/or computing cloud (e.g., accessed over network 210).
• code instructions of trained classification neural network 222A, visual filter neural network 206C, training images 216, priority list 222B, and/or mini-AUC code 206D, and/or pixel adjustment code 206E may be stored in data storage device 222, with executing portions loaded into memory 206 for execution by processor(s) 202.
• priority list 222B is provided to image server 214, for example, for instructing the priority presentation of images stored by image server 214.
• computing device 204 provides instructions for image server 214 to generate priority list 222B.
• Computing device 204 may include data interface 224, optionally a network interface, for connecting to network 210, for example, one or more of, a network interface card, a wireless interface to connect to a wireless network, a physical interface for connecting to a cable for network connectivity, a virtual interface implemented in software, network communication software providing higher layers of network connectivity, and/or other implementations.
• Computing device 204 may access one or more remote servers 218 using network 210, for example, to download updated training images 216 and/or to download an updated version of image processing code, training code, visual filter neural network, and/or the trained classification neural network.
• imaging interface 220 and data interface 224 may be implemented as a single interface (e.g., network interface, single software interface), and/or as two independent interfaces such as software interfaces (e.g., as application programming interfaces (API), network ports) and/or hardware interfaces (e.g., two network interfaces), and/or combination (e.g., single network interface, and two software interfaces, two virtual interfaces on a common physical interface, virtual networks on a common network port).
• API application programming interfaces
• the term/component imaging interface 220 may sometimes be interchanged with the term data interface 224.
• Computing device 204 may communicate using network 210 (or another communication channel, such as through a direct link (e.g., cable, wireless) and/or indirect link (e.g., via an intermediary computing device such as a server, and/or via a storage device) with one or more of:
• network 210 or another communication channel, such as through a direct link (e.g., cable, wireless) and/or indirect link (e.g., via an intermediary computing device such as a server, and/or via a storage device) with one or more of:
• Client terminal(s) 208 for example, when computing device 204 acts as a server that computes likelihood of the visual finding in anatomical images, provides the image storage server with the computed likelihood for determining a priority score of the respective anatomical image for creating the priority list, and where the highest ranked anatomical images are viewed on a display of the client terminal 208.
• server 218 is implemented as image server 214, for example, a PACS server.
• Server 218 may store new anatomical images as they are captured, and/or may store the training dataset.
• Server 214 may store and/or generate priority list 222B.
• server 218 is in communication with image server 214 and computing device 204.
• Server 218 may coordinate between image server 214 and computing device 204, for example, transmitting newly received anatomical images from server 218 to computing device 204 for filtering by visual filter neural network 206C and for insertion into classification neural network 222A for classification thereof (e.g., computation of likelihood of having a visual finding), and transmitting an indication of the computed likelihood from computing device 204 to server 218.
• Server 218 may compute priority scores and/or rank the anatomical images according to the computed likelihood for computing the priority list.
• Server 218 may send a list of priority ranked anatomical images and/or the priority list to image server 214, optionally for presentation to a healthcare provider on the display of the client terminal.
• Client terminal 208 may access the anatomical images of the priority list via server 218, which obtains the images from image server 214.
• server 218, may access the anatomical images of the priority list via server 218, which obtains the images from image server 214.
• one or more of the described functions of server 218 are performed by computing device 204 and/or imager server 214.
• Anatomical image repository 214 that stores anatomical images and/or imaging device 212 that outputs the anatomical images.
• Computing device 204 includes or is in communication with a user interface 226 that includes a mechanism designed for a user to enter data (e.g., patient data) and/or view the indications of identified visual findings.
• exemplary user interfaces 226 include, for example, one or more of, a touchscreen, a display, a keyboard, a mouse, and voice activated software using speakers and microphone.
• the visual filter neural network(s) is trained and/or provided. An exemplary process of training the visual filter neural network(s) is described with reference to FIG. 3.
• the visual filter neural network is designed to exclude anatomical images that do not depict a target body region (e.g., chest, abdomen, limb) at a target sensor orientation (e.g., AP/PA, lateral, supine, left lateral decubitus), and optionally of a target imaging modality type (e.g., x-ray, 2D CT slices, ultrasound images).
• a target body region e.g., chest, abdomen, limb
• a target sensor orientation e.g., AP/PA, lateral, supine, left lateral decubitus
• a target imaging modality type e.g., x-ray, 2D CT slices, ultrasound images.
• multiple visual filter neural network(s) are provided and/or trained.
• Each visual filter neural network is provided and/or trained to filter images according to a corresponding target classification neural network.
• one visual filter is designed to filter images being fed into a classification neural network that requires AP/PA chest x-rays
• another visual filter is designed to filter images being fed into a classification neural network that requires supine abdominal x-rays
• yet another visual filter is designed to filter images being fed into a classification neural network that requires lateral chest x-rays.
• the visual filter neural network is installed client-side, optionally on the client terminal that hosts the classification neural network.
• the client terminal is in communication with the medical imaging storage server over a network.
• the client-side installation is in contrast to server- side installation, for example, integrated with the PACS server.
• the client-side installation is independent of the PACS server, and independent of the DICOM® (or other protocol) defining storage of the anatomical images and/or providing metadata for the anatomical images (which may be erroneous).
• the client-side installation allows installation of multiple visual filter neural networks, each at respective client terminals that host different classification neural networks.
• the visual filter neural network is installed server- side, for example, when the classification neural network(s) are executed by the back-end server.
• one or more anatomical images are received, for example, from a medical imaging storage server (.e.g., a PACS server), an EMR server, from the anatomical imaging device, and/or from a storage device (e.g., portable storage medium, storage server).
• the anatomical images are stored by the medical imaging server according to a medical imaging storage format.
• the medical imaging server is a PACS server, and the medical imaging storage format is DICOM®.
• the metadata of DICOM® stores an indication of the target body region and the target sensor orientation. However, such metadata is commonly erroneous.
• the visual filter neural network described herein rejects anatomical images that do not depict the target body region at the target sensor orientation, independently of the DICOM® metadata.
• the DICOM® metadata is not necessarily accessed and/or used.
• the anatomical images may be, for example, 2D images (e.g., x-ray, ultrasound) and/or 2D slices of 3D images (e.g., of CT and/or MRI scans).
• 2D images e.g., x-ray, ultrasound
• 2D slices of 3D images e.g., of CT and/or MRI scans.
• the images may be obtained one at a time, for example, as the anatomical images are captured and stored, and/or may be obtained as a batch, for example, all images captured in the last 15 minutes.
• the images may be captured from different anatomical imaging modality machines, and/or captured at different sensor orientations.
• Exemplary anatomical imaging device includes an x-ray machine that captures a two dimensional anatomical image.
• Anatomical images may be stored as single images, a series of multiple independent images, and/or set of slices (e.g., 2D slices of a 3D volume image).
• each one of the anatomical images is fed into the neural network(s), as described herein.
• the anatomical images are fed into the visual filter neural network.
• the same anatomical images are fed into multiple different visual filter neural networks.
• the visual filter neural network outputs a classification category indicating that the respective anatomical image depicts the target body region at the target sensor orientation.
• the classification category may further include an indication of whether the respective anatomical image is at baseline or rotated relative to the baseline.
• the classification category may further include an indication of whether the respective anatomical image depicts a target anatomical imaging modality (e.g., x-ray, ultrasound, CT slice).
• a single classification category is indicative of the target sensor orientation and the rotation relative to the baseline, and optionally the depicted body region and optionally the target imaging modality.
• There may be multiple values for the classification category each denoting a respective combination of target body region and/or target sensor orientation and/or rotation and/or target imaging modality.
• the visual filter neural network outputs another classification category indicating that the respective anatomical image fails to depict one or more of: target body region, target sensor orientation, and target imaging modality.
• a rotation is not necessarily outputted in this case, since rotation may be corrected, while non-target body region, non-target sensor angle, and non-target imaging modality cannot be corrected.
• a single value of the classification category is outputted, for example, REJECT.
• multiple values of the classification category are outputted, indicative of the body region and/or sensor orientation and/or imaging modality depicted in the image. Such classification results may be used to forward the image to the correct classification neural network.
• a sub-set of the anatomical images classified into the classification category indicative of non-target body region and/or non-target sensor orientation and/or non-target imaging modality are rejected.
• the rejected images denote irrelevant images, i.e., irrelevant with respect to the classification neural network.
• the remaining sub-set of anatomical images denote relevant anatomical images i.e., relevant with respect to the classification neural network.
• a sub- set of the anatomical images classified into the classification category indicative of target body region and target sensor orientation and optionally target imaging modality are selected, indicative of relevant images.
• the remaining images may be rejected and/or ignored.
• the visual filter neural network selects chest images depicting AP and/or PA orientation and optionally x-rays, and/or rejects non-chest x-rays and/or lateral orientation and optionally non-x-ray images. For example, when the visual filter neural network is used with the single-label neural network described with reference to co-filed Application 76282.
• the visual filter neural network rejects and/or selects the anatomical images independently of metadata defined by the medical imaging storage format (e.g., DICOM®) associated with the respective anatomical image. For example, the DICOM® metadata is ignored.
• DICOM® medical imaging storage format
• images classified by the visual filter neural network as being rotated relative to baseline are re-rotated to return to the baseline.
• the rotation may be performed by rotation code.
• the amount of rotation may be based on the classification results of the visual filter neural network that classifies the amount of rotation of the respective image, for example, when the visual filter neural network computes a 90 degree clockwise rotation from baseline, the image may be rotated 90 degrees counter clockwise to return to baseline (or 270 degrees clockwise).
• act 110 may be implemented prior to act 108.
• outlier pixel intensity values representing injected content are adjusted.
• the injected content includes additional data that is not part of the anatomical image itself and/or not a part of the patient’s body, for example, a reference object placed beside the patient when taking the image for scaling the image and/or marking the side of the patient, characters (e.g., text, symbols) added to the image.
• the characters may be, for example, the name of the patient, ID of the patient, date of image.
• image for which the adjustment of pixel intensity value is performed has a pixel depth that is different than a pixel depth of the respective image when presented on a display.
• the pixel intensity value is stored in for example the range 10-16 bits and presented as 8 bits. The adjustment is performed for the stored image with the higher pixel depth.
• the outlier pixel intensity values representing injected content are identified based on an analysis of a created histogram of pixel intensity values.
• the histogram partition may be selected, for example, based on a predefined number of bins, and/or by code according to an analysis of the pixel intensity values of the image. For example, when the pixel intensity value is stored as a 16 bit integer, a 75 bin histogram may be used.
• the outlier pixel intensity values are selected based on one or two extreme bins of the histogram that are spaced apart from another bin by an empty bin that does not include any pixels. When the two extreme bins are adjacent (i.e., not spaced apart by an empty bin), no fix is necessary.
• the outlier pixel intensity values of the identified pixels may be adjusted to values computed as a function of non-outlier pixel intensity values.
• the values are computed as a function of the other bin and all pixels in the respective image.
• the function may be computed, for example, as: (i) a minimum of the pixel intensity values in the other bin, less a constant multiplied by the median pixel intensity values of all pixels in the respective image, and (ii) a maximum of the pixel intensity values in the other bin, added to a constant multiplied by the median pixel intensity values of all pixels in the respective image, and (iii) a smallest interest such that two raised to the smallest integer minus one is greater than the maximum of the pixel intensity values in the other bin.
• the following exemplary function may be used:
• the fixed value is a computed intensity value for replacing the current pixel intensity values of the left most bin
• B denotes the bin closest to the most extreme bin, optionally to the left most bin (i.e., the second most extreme bin, the first non-empty bin after the empty next to the most extreme left bin.)
• the minimal gray level (i.e., pixel intensity value) in bin B is determined to be 123.
• the following exemplary function may be used:
• C denotes the bin closest to the most extreme bin, optionally to the right most bin (i.e., the second most extreme bin, the first non-empty bin after the empty next to the most extreme right bin.)
• the fixed value for the outlier pixels is clipped by 2 m - 1, where m denotes the smallest integer such that 2 m > max(C).
• FIG. 4 is a schematic of a histogram 402 computed for the original pixel intensity values, and an adjusted histogram 404 that corrects for injected content, in accordance with some embodiments of the present invention.
• Histogram 402 is computed for the original pixel intensity values, as described herein. It is noted that since the two left most non-empty bins are denoted 404 and 406 are not separated by at least one empty bin, no fix is performed for bins 404 and 406. Since right most bins 408 410 are separated by at least one empty bin 412 (in fact as shown, many empty bins 412), a fix is performed for bin 410 by adjusting the pixel intensity values of pixels falling in bin 410 according to the equation
• act 112 may be implemented before acts 106, and/or 110.
• Implement act 112 after 106 and 110 reduces processing resource requirements, since anatomical images that are excluded by the filter not are processed for adaptation of their outlier pixel intensity values.
• the set of images may be used to create a training dataset for training a classification neural network (acts 114-116) and/or fed into the trained classification neural network (act 118-124). It is noted that the classification neural network trained using the training dataset that includes the filtered images may be fed filtered anatomical images. The same classification neural network trained in act 116 may be fed filtered images in act 118.
• a training dataset of filtered anatomical medical images i.e., not rejected and/or selected by the visual filter neural network
• optionally images rotated to the baseline is created. Each image the training dataset is labeled with a label indicative of the desired classification performed by the classification neural network.
• the classification neural network is trained according to the created training dataset.
• the post-filtering anatomical images i.e., the remaining non-rejected and/or selected sub-set of the anatomical images
• the classification neural network is fed into the classification neural network.
• images passed through respective visual filter neural networks are fed into the corresponding classification neural network.
• the classification neural network is obtained according to act 116, i.e., trained using a training dataset created from anatomical images filtered using the visual filter neural network (act 114), optionally the same visual filter neural network as in act 106.
• the classification neural network may output an indication of likelihood of a visual finding type being depicted in the respective anatomical image.
• the visual finding type may be indicative of an acute medical condition, which may require early and rapid treatment to prevent or reduce complications and/or morbidity arising from delayed treatment. For example, detecting an indication of pneumothorax is AP and/or PA chest x-rays.
• the classification neural network is implemented as a single-label neural network computed by fine-tuning and/or retraining a trained multi-label neural network.
• the multi-label neural network is trained according to a single-label training dataset of anatomical images labeled with an indication of the visual finding.
• the multi-label neural network is trained to compute likelihood of each of multiple visual findings being depicted in the fed image, based on a multi-label training dataset storing anatomical images labeled with an indication of respective depicted visual finding types.
• the images included in the single-label training dataset and/or the multi-label training dataset are optionally images filtered by the visual filter neural network. Additional details of the single-label neural network and the multiple-label neural network are described with reference to co-filed Application Attorney Docket No. 76282.
• instructions for creating a triage list may be generated, and/or the triage list may be generated.
• the triage list includes anatomical images determined as likely to depict the visual finding type(s) likely to be depicted as outputted by the classification neural network(s).
• the triage list may be for directing a priority of manual review by a human reviewer (e.g., radiologist).
• the triage list is ranked by decreasing likelihood of the indication of the visual finding type(s) based on a confidence score computed by the classification neural network(s). For example, images having a higher computed probability score of depicting the visual finding type(s) are ranked higher in the list than other images having lower probability scores.
• an aggregated priority score may be computed.
• the aggregated score may be weighed, for example, according to clinical urgency of the medical condition associated with the respective visual finding.
• the ranking of images may be according to the aggregated priority score.
• priority list and triage list are interchangeable.
• the triage list is for manual review by a human user (e.g., radiologist, emergency room physician, surgeon) of respective target anatomical images computed as likely depicting the indication of the visual finding type.
• a human user e.g., radiologist, emergency room physician, surgeon
• the priority list may be created by the computing device, and provided to image server, and/or the client terminal.
• the computing device provides instructions for creating the priority list to the image server, and the image server crates the priority list.
• the list may be viewed by the user (e.g., within the PACS viewer) optionally for manual selection of images for viewing, and/or may define automatic sequential loading of images for viewing by the user (e.g., within the PACS viewer).
• the user may manually view images in the priority list, optionally according to the ranking.
• the acute medical condition may be diagnosed and/or treated.
• the visual finding type is a sign of the acute medical condition.
• the patient may be treated for the acute medical condition.
• the patient may be diagnosed and/or treated, for example, by insertion of a needle or chest tube to remove the excess air.
• a target body region e.g., chest, abdomen, limb, head
• a target sensor orientation e.g., AP, PA, lateral, left lateral decubitus
• the target body region and the target sensor orientation are defined by the target classification neural network into which the target anatomical images are fed. When multiple classification neural networks are implemented, multiple combinations of the target body region and target sensor orientation are received.
• a target anatomical imaging modality is defined according to the target classification neural network.
• the anatomical images may be obtained, for example, from a data storage server, a publicly available image dataset, and/or other sources.
• images may be preprocessed to increase the number of training images and/or variety of the training image, for example, with the following additional augmentations: random horizontal flip, random crop, random rotation, and random zoom.
• a training dataset for training the visual filter neural network is created.
• each one of the anatomical images is associated with a respective label indicative of a target body region and a target sensor orientation.
• each anatomical image may be labeled with an indication of the actual depicted body region and depicted sensor orientation.
• the label may be indicative of whether the body region and sensor orientation are according to the defined target, or non-target.
• each of the anatomical images is labeled with an indication of a rotation relative to a baseline.
• each of the anatomical images is further labeled with an indication of anatomical imaging modality type, for example, the value of the actual anatomical modality type, and/or an indication of whether the anatomical imaging modality type is target or non-target.
• an indication of anatomical imaging modality type for example, the value of the actual anatomical modality type, and/or an indication of whether the anatomical imaging modality type is target or non-target.
• the labeling may be performed, for example, manually by a user, and/or automatically based on DICOM® metadata with a review by the user to check for errors in the DICOM® metadata.
• the visual filter neural network is trained based on the training dataset.
• Multiple visual filter neural networks may be trained on the same training dataset, according to their respective requirements for body region and/or sensor orientation and/or imaging modality type.
• the visual filter neural network is trained for classifying a target anatomical image into a classification category indicative of the target body region depicted at the target sensor angle and optionally at the target imaging modality type, and optionally the rotation relative to the baseline, or into another classification category indicative of non-target body region and/or non-target sensor orientation and/or non-target anatomical imaging modality.
• the target anatomical image is rejected when classified into the classification category indicative of non-target body region and/or a non-target sensor orientation and/or non-target anatomical imaging modality type.
• additional code is provided in association with the visual filter neural network for rotating target anatomical images determined to be rotated back to the baseline.
• the anatomical images that are not rejected and/or selected by the visual filter neural network are inputted into the classification neural network for determining likelihood of the visual finding type being depicted therein, as described herein.
• the trained visual filter neural network(s) are provided, for example, locally stored and/or transmitted to the computing device.
• FIG. 5 includes an AP/PA chest x-ray before adjustment of pixel intensity values denoting injected content 502, and an x-ray 504 depicting the AP/PA chest x-ray after adjustment of pixel intensity values denoting injected content, as described herein, in accordance with some embodiments of the present invention.
• x-ray 504 depicts an improvement in dynamic range. The accuracy of detection of minor and/or fine visual features, such as pneumothorax, by the classification neural network is improved when the classification neural network is fed x-ray 504 in comparison to the classification neural network being fed x-ray 502.
• Inventors performed a computational evaluation according to the systems and/or methods and/or apparatus and/or code instructions described herein, based on the features and/or system components discussed with reference to FIGs. 1-3.
• FIG. 6 is a graph of ROC curves computed for the experiments for computationally evaluating the visual filter neural network in increasing classification accuracy of a classification neural network, in accordance with some embodiments of the present invention.
• Curated internal validation set (internal validation 2801, 323 chest serieses including Clalit health service organization, rapid rad and intermountain data sources).
• Curated internal validation set (internal validation 2801, 323 chest serieses including Clalit health service organization, rapid rad and intermountain data sources). • All images in the baseline dataset were inverted (e.g. were intentionally assigned monochrome 1 photometric interpretation not supported by the pnx classification neural network)
• pnx_net i.e., classification neural network
• Inventors performed additional computational evaluations of the visual filter created according to the systems and/or methods and/or apparatus and/or code instructions described herein, based on the features and/or system components discussed with reference to FIGs. 1-3.
• the visual filter neural network was designed to receive an anatomical image (i.e., 2D image) as input, and to output one of the following classification categories (also referred to herein as labels): NOTjCHEST, PA-0 , PA-1, PA-2, PA-3, AP-0, AP-1, AP-2, AP-3, LA TERAL_LEFT-0, LA TERAE_LEFT - 1 , LA TERAE_LEFT -2 , EATERAE_LEFT-3, EATERAE_RIGHT-0,
• LA TERAEJUGHT - 1 EA TERAEJUGHT -2 , and LA TERAEJUGHT -3.
• PA, AP, LATERAL _LEFT ' , LATERAL_RIGHT indicate the sensor orientation.
• the non-zero numbers (1, 2, 3) denote an amount of rotation of the image, clockwise, in multiples of 90 degrees i.e. PA-3 means the image is a PA view of a chest x-ray that was rotated by 270 degrees.
• the input image into the visual filter neural network has size 192x192.
• the model architecture used as a basis for the visual filter neural network was InceptionV3.
• the training set had 20,000 images.
• AP, PA, LATERAL, and NON-CHEST image labels were obtained by processing the DICOM® tag.
• the LATERAL images were assigned EATERAE_RIGHT and EATERAE_EEFT labels by manual inspection. Rotated images (with associated rotation labels) were generated artificially and on-the-fly during training.
• the model was validated against a held-out validation set and achieved an accuracy of over
• Inventors performed a computational evaluation of the images for which pixel intensity values were adjusted according to the systems and/or methods and/or apparatus and/or code instructions described herein, based on the features and/or system components discussed with reference to FIGs. 1-3.
• a set of 177,000 anatomical images was created, wherein the DICOM® bit depth is different than the actual bit depth, for example 10 vs. 12.
• a subset of 50 images was selected as a compact validation dataset, termed OPIF.
• the process for adjusting outlier pixel intensity values was run on the OPIF validation dataset.
• the following plots were generated for every image: original image converted to 8-bit (using min-max), histogram of original full bit depth image, fixed image converted to 8-bit (using min-max), and histogram of fixed full bit depth image.
• the results were manually analyzed. First, it verified that the histogram was fixed correctly and that the image does not contain outlier pixel intensity values. In addition, the conversion of the fixed image to 8- bit was verified.
• composition or method may include additional ingredients and/or steps, but only if the additional ingredients and/or steps do not materially alter the basic and novel characteristics of the claimed composition or method.
• a compound or “at least one compound” may include a plurality of compounds, including mixtures thereof.
• range format is merely for convenience and brevity and should not be construed as an inflexible limitation on the scope of the invention. Accordingly, the description of a range should be considered to have specifically disclosed all the possible subranges as well as individual numerical values within that range. For example, description of a range such as from 1 to 6 should be considered to have specifically disclosed subranges such as from 1 to 3, from 1 to 4, from 1 to 5, from 2 to 4, from 2 to 6, from 3 to 6 etc., as well as individual numbers within that range, for example, 1, 2, 3, 4, 5, and 6. This applies regardless of the breadth of the range.

Landscapes

• Engineering & Computer Science (AREA)
• Data Mining & Analysis (AREA)
• Theoretical Computer Science (AREA)
• Bioinformatics & Cheminformatics (AREA)
• General Engineering & Computer Science (AREA)
• Bioinformatics & Computational Biology (AREA)
• Computer Vision & Pattern Recognition (AREA)
• Evolutionary Biology (AREA)
• Evolutionary Computation (AREA)
• Physics & Mathematics (AREA)
• Artificial Intelligence (AREA)
• General Physics & Mathematics (AREA)
• Life Sciences & Earth Sciences (AREA)
• Image Analysis (AREA)
• Apparatus For Radiation Diagnosis (AREA)
• Magnetic Resonance Imaging Apparatus (AREA)
• Image Processing (AREA)

Applications Claiming Priority (4)

Publications (2)

Family

ID=66448425

Family Applications (3)

Family Applications Before (2)

Country Status (4)

Families Citing this family (17)

Family Cites Families (7)

• 2019
  • 2019-05-07 EP EP19173136.3A patent/EP3567525A1/de active Pending
  • 2019-05-07 WO PCT/IB2019/053726 patent/WO2019215606A1/en unknown
  • 2019-05-07 WO PCT/IB2019/053724 patent/WO2019215604A1/en unknown
  • 2019-05-07 EP EP19800738.7A patent/EP3791325A4/de not_active Withdrawn
  • 2019-05-07 WO PCT/IB2019/053725 patent/WO2019215605A1/en active Application Filing
  • 2019-05-07 EP EP19800865.8A patent/EP3791310A4/de not_active Withdrawn
  • 2019-05-07 DE DE202019005911.3U patent/DE202019005911U1/de active Active
  • 2019-05-07 JP JP2019087284A patent/JP2019195627A/ja active Pending

Also Published As

Similar Documents

Legal Events