Classifications

• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T7/00—Image analysis
  • G06T7/0002—Inspection of images, e.g. flaw detection
  • G06T7/0012—Biomedical image inspection
• • A—HUMAN NECESSITIES
  • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
  • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
  • A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
  • A61B1/00002—Operational features of endoscopes
  • A61B1/00004—Operational features of endoscopes characterised by electronic signal processing
  • A61B1/00009—Operational features of endoscopes characterised by electronic signal processing of image signals during a use of endoscope
  • A61B1/000094—Operational features of endoscopes characterised by electronic signal processing of image signals during a use of endoscope extracting biological structures
• • A—HUMAN NECESSITIES
  • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
  • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
  • A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
  • A61B1/00002—Operational features of endoscopes
  • A61B1/00004—Operational features of endoscopes characterised by electronic signal processing
  • A61B1/00009—Operational features of endoscopes characterised by electronic signal processing of image signals during a use of endoscope
  • A61B1/000096—Operational features of endoscopes characterised by electronic signal processing of image signals during a use of endoscope using artificial intelligence
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T7/00—Image analysis
  • G06T7/60—Analysis of geometric attributes
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T7/00—Image analysis
  • G06T7/60—Analysis of geometric attributes
  • G06T7/62—Analysis of geometric attributes of area, perimeter, diameter or volume
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T7/00—Image analysis
  • G06T7/70—Determining position or orientation of objects or cameras
  • G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V10/00—Arrangements for image or video recognition or understanding
  • G06V10/20—Image preprocessing
  • G06V10/255—Detecting or recognising potential candidate objects based on visual cues, e.g. shapes
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V10/00—Arrangements for image or video recognition or understanding
  • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
  • G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V10/00—Arrangements for image or video recognition or understanding
  • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
  • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V20/00—Scenes; Scene-specific elements
  • G06V20/40—Scenes; Scene-specific elements in video content
  • G06V20/46—Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V20/00—Scenes; Scene-specific elements
  • G06V20/50—Context or environment of the image
• • G—PHYSICS
  • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
  • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
  • G16H70/00—ICT specially adapted for the handling or processing of medical references
  • G16H70/20—ICT specially adapted for the handling or processing of medical references relating to practices or guidelines
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/10—Image acquisition modality
  • G06T2207/10016—Video; Image sequence
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/10—Image acquisition modality
  • G06T2207/10068—Endoscopic image
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/20—Special algorithmic details
  • G06T2207/20081—Training; Learning
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/20—Special algorithmic details
  • G06T2207/20084—Artificial neural networks [ANN]
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/30—Subject of image; Context of image processing
  • G06T2207/30004—Biomedical image processing
  • G06T2207/30028—Colon; Small intestine
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/30—Subject of image; Context of image processing
  • G06T2207/30004—Biomedical image processing
  • G06T2207/30028—Colon; Small intestine
  • G06T2207/30032—Colon polyp
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/30—Subject of image; Context of image processing
  • G06T2207/30004—Biomedical image processing
  • G06T2207/30096—Tumor; Lesion
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/30—Subject of image; Context of image processing
  • G06T2207/30168—Image quality inspection
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V2201/00—Indexing scheme relating to image or video recognition or understanding
  • G06V2201/03—Recognition of patterns in medical or anatomical images
  • G06V2201/032—Recognition of patterns in medical or anatomical images of protuberances, polyps nodules, etc.

Definitions

• the present disclosure relates generally to the field of imaging systems and computer-implemented systems and methods for processing real-time video. More specifically, and without limitation, this disclosure relates to systems, methods, and computer-readable media for processing frames of real-time video and performing object detection and characterization.
• the systems and methods disclosed herein may be used in various applications, such as medical image analysis for polyp detection and characterization, including determining the classification, size, and location of polyps.
• the systems and methods disclosed herein may also be implemented to provide real time image processing capabilities, such as identifying, based on one or more of object characteristics, a medical guideline and presenting, in real-time on a display device, information for the medical guideline.
• the at least one processor may be further configured to generate a confidence value associated with the identified medical guideline.
• the trained characterization network may include: a trained classification network configured to determine a classification associated with the object of interest and to generate a classification confidence value associated with the determined classification, a trained location network configured to determine a location associated with the object of interest and to generate a location confidence value associated with the determined location, and a trained size network configured to determine a size associated with the object of interest and to generate a size confidence value associated with the determined size.
• the at least one processor may be further configured to present, on the display device, information associated with at least one of the classification, the location, or the size.
• FIG. 4A illustrates an example system for processing real-time video, consistent with embodiments of the present disclosure.
• computing device 160 may also be configured to relay the original, non-augmented video from image device 140 directly to display device 180.
• computing device 160 may perform a direct relay under predetermined conditions, such as when there is no overlay or other augmentation to be generated.
• computing device 160 may perform a direct relay if operator 120 transmits a command as part of a control signal to computing device 160 to do so.
• the commands from operator 120 may be generated by operation of button(s) and/or key(s) included on an operator device and/or an input device (not shown), such as a mouse click, a cursor hover, a mouseover, a button press, a keyboard input, a voice command, an interaction performed in virtual or augmented reality, or any other input.
• Processor(s) 230 and/or memory 240 may also include machine-readable media for storing software or sets of instructions.
• “Software” as used herein refers broadly to any type of instructions, whether referred to as software, firmware, middleware, microcode, hardware description language, or otherwise. Instructions may include code (e.g., in source code format, binary code format, executable code format, or any other suitable format of code). The instructions, when executed by one or more processors 230, may cause the processor(s) to perform the various operations and functions described in further detail herein.
• computing device 200 may include one or more machine-learning models used to implement the neural networks described herein, and may retrieve or receive weights or parameters of machine-learning models, training information or training feedback, medical guidelines and/or guideline rules, and/or any other data and information described herein.
• a medical image device may be any device capable of producing videos or one or more images of a human body or a portion thereof, such as an endoscopy device, an X-ray machine, a CT machine, or an MRI machine, as described above.
• a medical procedure may be any action performed with the intention of determining, detecting, measuring, or diagnosing a patient condition, such as an endoscopy, a gastroscopy, a colonoscopy, or an enteroscopy.
• the medical procedure may be used to identify objects of interest (e.g., lesions or polyps) in a location in the human body. Locations in the human body may be the rectum, sigmoid colon, descending colon, transverse colon, ascending colon, or cecum. It is to be understood, however, that the disclosed systems and methods may be employed in other contexts and applications.
• a polyp may also be characterized based on its size.
• a polyp size may be expressed as a numeric value or a size classification.
• the size of a polyp may be, for example, expressed using any suitable metric value such as millimeters (mm) although any other metric may be used (e.g., inches).
• a polyp may thus have a size of 1 mm, 5 mm, 10 mm, and so forth.
• the at least one processor of computing device 160 may be configured to apply one or more neural networks to determine a confidence value.
• the confidence value associated with an output may be implicitly defined in one hot encoding formulation, where for each possible class a score is output by the network.
• neural network calibration methods may be used such as mixup and label smoothing to control the range and distribution of confidence values or scores.
• a dedicated output node is added to each neural network that provides a confidence estimation and an abstention term can be included in the loss function to train the neural network accordingly. With the extra term in the loss function, the neural network can predict low confidence values when the estimation error is high due to, for example, low quality or cluttered images.
• the neural network is trained to predict both the output and confidence score or label.
• a first set of training frames (or portions of frames) containing or not containing an object of interest may be labeled as “adenoma”
• a second set of training frames (or portions of frames) containing or not containing an object of interest may be labeled as “non-adenoma” or another classification (e.g., “serrated”).
• Other labeling conventions could be used both in binary (e.g. “hyperplastic” vs “non-hyperplastic”) and in multiple classes (e.g. “adenoma” vs “sessile serrated” vs “hyperplastic”).
• Weights or other parameters of the location network may be adjusted based on its output with respect to a third, non- labeled set of training frames (or portions of frames) until a convergence or other metric is achieved, and the process may be repeated with additional training frames (or portions thereof) or with live data as described herein.
• the at least one processor of computing device 160 may be configured to apply one or more neural networks that implement a trained size network configured to determine a size associated with the object of interest, and in some embodiments the trained classification network may also be configured to generate a size confidence value associated with the determined size.
• the size may be the same or similar as those described above (e.g., a numeric value or a size classification).
• the location network may be trained using a plurality of training frames or portions thereof labeled based on size.
• the at least one processor of computing device 160 may be configured to identify, based on one or more of the plurality of features and/or the confidence values, a medical guideline.
• a “medical guideline,” as used herein, may refer to any information provided with the aim of aiding in the determination, diagnosis, or treatment of a patient condition.
• the identified medical guideline may include an instruction to leave or resect the object of interest.
• the medical guideline may also include an identification or description of a specific type of resection.
• Similar techniques may be used to make the displayed information more quickly discernable or distinguishable to the clinician or operator such as using unique colors for the presented information depending on, e.g., the type of classification and/or urgency of a recommended medical guideline (e.g., green for “non-adenoma” and/or “leave” but red for “adenoma” and/or “resect”).
• a recommended medical guideline e.g., green for “non-adenoma” and/or “leave” but red for “adenoma” and/or “resect”.
• characterization network 430 may determine and display device 470 may display classification information (e.g., “adenoma” or “non adenoma”, or “hyperplastic” or “non-hyperplastic”), location information (e.g., “rectum” or “caecum”), size information (e.g., “diminutive” or “non-diminutive” or “small” or “large”), and/or a medical guideline (e.g., “leave” or “resect” or “biopsy”) for the detected object of interest.
• classification information e.g., “adenoma” or “non adenoma”, or “hyperplastic” or “non-hyperplastic”
• location information e.g., “rectum” or “caecum”
• size information e.g., “diminutive” or “non-diminutive” or “small” or “large”
• a medical guideline e.g., “lea
• real-time processing can be provided with one of three options: (i) there is no output for the first N-1 frames or (ii) the output for the first N-1 frames only depends on the current frame (i.e., there is no buffering in the initial phase) or (iii) the output for the first N-1 frames only depends on the last frame and all the previous ones available . Additionally, there could be other intervals during which one or more of the networks 440-460 are not providing an output. During intervals where there is no output, real-time processing system 400 may communicate the status of the system by causing appropriate messages to be displayed (e.g., status messages such as "processing,” "buffering,” or "analyzing") via display device 470.
• appropriate messages e.g., status messages such as "processing,” "buffering,” or "analyzing
• Real-time processing system 400 may also determine the number of instances of neural networks 440-460 to run in parallel based on other real-time processing requirements or factors such as processing time delay(s) or restriction(s) due to available system resources (e.g., available hardware and software resources) and accuracy requirements in detecting objects and features of interest of detected objects.
• Real-time processing system 400 may also achieve real time processing requirements by adjusting the sampling rate to select a subset of frames from image device 410. Additionally, or alternatively, real-time processing system 400 may sample frames and/or persistent objects detected in the received frames to meet real-time processing requirements.
• Encoder network 485 and latent representation 486 may be implemented with one or more neural networks that are trained using a combination of unsupervised reconstruction loss and a supervised loss based on classification, location, and size tasks. Additionally, or alternatively, encoder network 485 can be trained with a loss from the contrastive loss family such as triplet loss or quadruplet loss which enforces a structured organization of the latent space. In this way, the latent space can assign a similar representation to image frames belonging to the same object and a more robust distance metric can be defined between latent representations. As discussed above, encoder network 485 may embed the inherrent structure of the detected object(s) by projecting into a latent space, for example, latent representation 486.
• the embedded representation in latent space i.e. , the output of latent representation 486) may be fed in parallel to the three characterization networks (i.e., classification network 440, location network 405, and size network 460) to determine the characteristics or features for the object.
• the trained neural networks 440-460 will be small (i.e., just a few fully connected layers) since the encoding part is shared and performed within the encoder network 485. This reduces the overall computational cost and efficiency of the characterization network 430. Consequently, real-time processing system 480 benefits from a reduction in time needed to process and characterize objects of interest and provide output to display device 470.
• the trained frame quality network may comprise one or more suitable machine learning networks or algorithms for determining a quality value associated with one or more frames in the real-time video, including one or more neural networks (e.g., a deep neural network, a convolutional neural network, a recursive neural network, etc.), a random forest, a support vector machine, or any other suitable model as described above trained to determine a frame quality.
• the frame quality network may be trained using a plurality of training frames or portions thereof labeled based on one or more quality values or classifications.
• a first set of training frames may be labeled as “sufficient quality,” and a second set of training frames (or portions of frames) may be labeled as “not sufficient quality.”
• Weights or other parameters of the frame quality network may be adjusted based on its output with respect to a third, non-labeled set of training frames (or portions of frames) until a convergence or other metric is achieved, and the process may be repeated with additional training frames (or portions thereof) or with live data as described herein.
• the at least one processor may apply one or more neural networks that implement a trained size network configured to determine a size associated with the object of interest, as discussed above.
• the at least one processor may identify, based on one or more of the classification, the location, and the size, a medical guideline, as discussed above.
• the at least one processor may present, in real-time on a display device during the medical procedure, information for the identified medical guideline, as discussed above.
• systems and methods consistent with the present disclosure include the following implementations and aspsects.
• the medical procedure may include at least one of an endoscopy, a gastroscopy, a colonoscopy, or an enteroscopy.
• the object of interest may include at least one of a formation on or of human tissue, a change in human tissue from one type of cell to another type of cell, an absence of human tissue from a location where the human tissue is expected, or a lesion.

Landscapes

• Engineering & Computer Science (AREA)
• Physics & Mathematics (AREA)
• Theoretical Computer Science (AREA)
• General Physics & Mathematics (AREA)
• Health & Medical Sciences (AREA)
• Computer Vision & Pattern Recognition (AREA)
• Medical Informatics (AREA)
• General Health & Medical Sciences (AREA)
• Multimedia (AREA)
• Life Sciences & Earth Sciences (AREA)
• Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
• Radiology & Medical Imaging (AREA)
• Evolutionary Computation (AREA)
• Surgery (AREA)
• Artificial Intelligence (AREA)
• Quality & Reliability (AREA)
• Geometry (AREA)
• Public Health (AREA)
• Software Systems (AREA)
• Databases & Information Systems (AREA)
• Computing Systems (AREA)
• Biophysics (AREA)
• Veterinary Medicine (AREA)
• Animal Behavior & Ethology (AREA)
• Molecular Biology (AREA)
• Heart & Thoracic Surgery (AREA)
• Biomedical Technology (AREA)
• Pathology (AREA)
• Optics & Photonics (AREA)
• Signal Processing (AREA)
• Epidemiology (AREA)
• Primary Health Care (AREA)
• Bioethics (AREA)
• Measuring And Recording Apparatus For Diagnosis (AREA)
• Apparatus For Radiation Diagnosis (AREA)
• Image Analysis (AREA)
• Endoscopes (AREA)
• Geophysics And Detection Of Objects (AREA)
• Burglar Alarm Systems (AREA)

Priority Applications (9)

Applications Claiming Priority (4)

Publications (1)

Family

ID=82786643

Family Applications (1)

Country Status (10)

Families Citing this family (4)

Family Cites Families (5)

• 2022
  • 2022-07-11 TW TW111125956A patent/TW202309926A/zh unknown
  • 2022-07-12 IL IL310111A patent/IL310111A/en unknown
  • 2022-07-12 EP EP22750805.8A patent/EP4371061A1/en active Pending
  • 2022-07-12 AU AU2022312668A patent/AU2022312668A1/en active Pending
  • 2022-07-12 JP JP2024501862A patent/JP2024526751A/ja active Pending
  • 2022-07-12 KR KR1020247004829A patent/KR20240033023A/ko active Pending
  • 2022-07-12 WO PCT/EP2022/069364 patent/WO2023285407A1/en not_active Ceased
  • 2022-07-12 CA CA3222272A patent/CA3222272A1/en active Pending
  • 2022-07-12 MX MX2024000658A patent/MX2024000658A/es unknown
  • 2022-07-12 US US18/578,337 patent/US20240296560A1/en active Pending

Non-Patent Citations (2)

Also Published As

Similar Documents

Legal Events