WO2024012080A1 - 内镜辅助检查系统、方法、装置及存储介质 - Google Patents

内镜辅助检查系统、方法、装置及存储介质 Download PDF

Info

Publication number
WO2024012080A1
WO2024012080A1 PCT/CN2023/097402 CN2023097402W WO2024012080A1 WO 2024012080 A1 WO2024012080 A1 WO 2024012080A1 CN 2023097402 W CN2023097402 W CN 2023097402W WO 2024012080 A1 WO2024012080 A1 WO 2024012080A1
Authority
WO
WIPO (PCT)
Prior art keywords
site
observation
endoscopic
observation point
current
Prior art date
Application number
PCT/CN2023/097402
Other languages
English (en)
French (fr)
Inventor
王玉峰
Original Assignee
天津御锦人工智能医疗科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 天津御锦人工智能医疗科技有限公司 filed Critical 天津御锦人工智能医疗科技有限公司
Publication of WO2024012080A1 publication Critical patent/WO2024012080A1/zh

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G06T7/0012—Biomedical image inspection
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/084—Backpropagation, e.g. using gradient descent
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10016—Video; Image sequence
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10068—Endoscopic image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30004—Biomedical image processing
    • G06T2207/30092—Stomach; Gastric

Definitions

  • the present application relates to the field of medical technology, and in particular to an endoscopic auxiliary examination system, method, device and storage medium.
  • Gastroscopy is an important means of examining abnormalities of the upper gastrointestinal tract and is the gold standard for the diagnosis of upper gastrointestinal diseases such as the esophagus, stomach, and duodenum.
  • it is necessary to determine the part of the stomach where the video image was taken based on the video image captured by the endoscope.
  • a complete gastroscopy report needs to include the oropharynx, esophagus, cardia, and fundus of the stomach.
  • Gastrointestinal endoscopists are required to take real-time pictures during gastroscopy and conduct further examinations in a timely manner when suspicious parts are found.
  • Gastrointestinal endoscopists are basically overloaded. High workload will reduce the quality of endoscopists' endoscopic examinations, and may easily lead to problems such as incomplete coverage of examination sites, incomplete detection of lesions, and incomplete image collection.
  • this application provides an endoscopic auxiliary inspection system, method, device and storage medium.
  • this application provides an endoscopic auxiliary inspection system, which includes:
  • the anatomical part recognition module is used to identify the collected endoscopic images frame by frame, determine each observation point corresponding to each frame of endoscopic image, and map each observation point to each preset site;
  • An attention recording module configured to determine the site site observation sequence based on the recorded observation time of each observation point determined by the anatomical site identification module and the observation completeness of each site;
  • a guidance module configured to determine the target to be transferred based on the current site site corresponding to the current frame endoscopic image determined by the anatomical site identification module, the site site observation sequence, and the preset physical depth of the current site site. Site; guide endoscopy according to the current site location and the target site.
  • the present application provides an endoscopy-assisted inspection method.
  • Checking methods include:
  • Endoscopy is guided based on the current site location and the target site.
  • an endoscopic auxiliary inspection device which includes: a memory, a processor, and a computer program stored on the memory and executable on the processor;
  • the present application provides a computer-readable storage medium.
  • the computer-readable storage medium stores an endoscopic auxiliary examination program.
  • the endoscopic auxiliary examination program is executed by a processor, the above-mentioned methods are implemented. Steps of endoscopically assisted examination methods.
  • Each embodiment of the present application identifies the parts of the upper gastrointestinal mucosa under endoscopic observation, explores the correlation between various parts of the digestive tract, and prompts the operating path of the endoscopic observation, guiding the endoscopist to follow the most efficient
  • the scope is transported along the route to avoid repeated inspections, save time, improve the fluency of gastroscopy, and ensure the integrity of gastroscopy observation without omissions and dead ends.
  • Figure 1 is a block diagram of an endoscope-assisted inspection system provided by various embodiments of the present application
  • Figure 2 is a schematic diagram of a convolution layer provided by various embodiments of the present application.
  • Figure 3 is a flow chart of the endoscopic-assisted inspection method provided by various embodiments of the present application.
  • An embodiment of the present invention provides an endoscopic auxiliary inspection system, as shown in Figure 1.
  • the endoscopic auxiliary inspection system includes:
  • the S10 video acquisition module uses a video capture card to transmit digital/analog video signals such as HDMI, DVI, SDI, and S-Video to the host of the endoscopic auxiliary inspection system.
  • the video signal is read through Opencv and transcoded into RGB frame by frame. Image format endoscopic image;
  • the S20 anatomical part recognition module is used to identify the collected endoscopic images frame by frame, determine each observation point corresponding to each frame of endoscopic image, and assign each observation point to Map to each preset site location respectively;
  • S30 attention recording module used to determine the site site observation sequence based on the recorded observation time of each observation point determined by the anatomical site identification module and the observation completeness of each site;
  • S40 is a guidance module configured to determine the location to be transferred based on the current site site corresponding to the current frame endoscopic image determined by the anatomical site identification module, the site site observation sequence, and the preset physical depth of the current site site.
  • Target site guide endoscopy according to the current site location and the target site.
  • the embodiments of the present invention are particularly suitable for gastroscopy.
  • the embodiments of the present invention identify the mucosal parts of the upper gastrointestinal tract under endoscopic observation, explore the correlation between various parts of the digestive tract, and prompt and guide the operating path of the endoscopic observation.
  • the endoscopist performs the endoscopy according to the most efficient route to avoid repeated examinations, save time, improve the fluency of gastroscopy, and ensure the integrity of gastroscopy observation without omissions or dead ends.
  • the following uses a specific implementation mode to describe in detail the endoscopic auxiliary examination system of the embodiment of the present invention, including S10 video acquisition module, S20 anatomical part identification module, S30 attention recording module, S40 guidance module and S50 monitoring module; wherein,
  • video capture card uses the video capture card to transmit digital/analog video signals such as HDMI, DVI, SDI, S-Video, etc. to the host of the endoscopic auxiliary inspection system.
  • the video signals are read through Opencv and transcoded into RGB image format frame by frame.
  • the S201 model training unit is used to train the artificial intelligence neural network multi-label recognition model; the model can be implemented by neural networks with classification functions such as ResNet, VGG, DenseNet, etc.
  • classification functions such as ResNet, VGG, DenseNet, etc.
  • the training process includes:
  • Step 2011, build the network structure: the network is mainly composed of 4 groups of convolutional layers containing spatial grouping structures, 2 groups of self-attention network layers, 1 group of fully connected layers and 1 group of prediction layers connected in series;
  • the convolution layer containing the spatial grouping structure is shown in Figure 2.
  • the internal group number (Group) is set to 64 and the bottleneck width is set to 32 (bottleneck-width), which is used to perform dimension transformation and Feature extraction, final output feature vector F 0 ;
  • the vector F 0 and the feature vector Q 0 are jointly weighted with self-attention to obtain the feature vector
  • the one-dimensional feature vector F is used to derive the derivative for each preset category to obtain the probability (p 0 , p 1 ,..., p n ).
  • the specific formula is:
  • p n refers to the probability of solving category n
  • b n is a trainable bias hyperparameter for category n
  • Step 2012 Label the pre-collected sample pictures (upper gastrointestinal tract observation pictures), and the labeling results are 1-n of the preset subdivided anatomical parts (i.e., observation point parts);
  • Step 2013 use the above endoscopic image as input information of the neural network model, and Based on the pre-marked multi-label results, the loss function is used for calculation.
  • the loss function formula is as follows;
  • L(x) refers to the loss value of a given input image x during the training process
  • N refers to the number of all categories
  • n refers to the current category
  • p n refers to the predicted probability of n categories
  • ⁇ refers to the bias value , usually takes 0 or 1, this embodiment takes 1
  • y n refers to the true value of whether the given input image x has label n.
  • Step 2014 after obtaining the loss value L(x), use back propagation to update the model parameters; specifically, 500 rounds of training are preset, and the above steps 2011 and 2012 are repeated for all annotated data in each round of training until the model reaches the predetermined level.
  • the number of rounds or the loss value is less than 1e-4 and stop to obtain the constructed neural network model.
  • S202 recognition unit used to identify the collected endoscopic examination images frame by frame through the neural network multi-label recognition model trained by the model training unit, and determine each observation point corresponding to each frame of endoscopic examination image; that is to say , use the above-mentioned trained model to identify the acquired frame-by-frame images, and finally through the prediction of the prediction layer, the subdivided anatomical part identification result is obtained, that is, the currently observed observation point part T ⁇ 1,2...,j ⁇ ;
  • the S203 anatomical part mapping unit is used to map each observation point part to each preset station part according to the preset mapping relationship between the observation point part and the station part.
  • the subdivided anatomical part identification results obtained above are mapped to the corresponding site parts.
  • the mapping relationship is as follows: Corresponding physical depths are set respectively. In this embodiment, the physical depth is level 9.
  • four sets of convolutional layers are used to abstract features of the 2D endoscopic image to obtain the output feature vector
  • the first attention layer is used to weight the inside of the one-dimensional feature vector to obtain the weighted feature Q 0
  • Features F 0 and Q 0 are jointly weighted through the second attention layer to obtain features Q 0 to
  • the inference step can establish the correlation between F 0 global and F 0 key areas, thereby bringing better performance to multi-label recognition.
  • the S301 time recording unit is used to record the frame-by-frame observation time of the observation point and the observation completeness of the site determined by the anatomical part identification module, and determine the site based on the observation time of each observation point and the observation completeness of each site. Part observation sequence and observation point part observation sequence.
  • b represents the optimal observation time of the observation point
  • m n represents the observation time of each target observation point corresponding to the site
  • n represents the number of target observation points corresponding to the site.
  • the heat map display unit is used to display the observation sequence of the observation point location on the digestive tract UI interface in the form of a heat map. That is, the observation sequence M n of the observation point part of the time recording unit is displayed on the GUI interface in the form of a heat map. After the endoscopy is completed, you can view the heat map to review the length of time the doctor paid attention to each part during the entire examination. Based on the length of time, the entire examination process can be analyzed afterwards, which parts lacked enough attention, and which parts received the doctor's focus. . Specifically, this embodiment uses a heat map module to reflect attention time.
  • S401 guides the probability prediction unit to determine the site location transition probability matrix according to the current site location, the site location observation sequence, and the preset physical depth of the current site location; The largest value is determined as the target site.
  • the site location transition probability matrix S k [s 1 , s 2 ,...,s k ] (s k represents the probability of moving from the current site to site k), and calculates the maximum value in the matrix to obtain the new target site T i′ or target observation point T n .
  • the first objective function is:
  • s i′ represents the probability of transferring to target site i′
  • M k (i′) represents the observation completeness of target site i′
  • D i′ represents the physical depth of target site i′
  • D i represents the depth of site part i Physical depth
  • ⁇ is an arbitrary constant ( ⁇ >0), and the value in this embodiment is 1.
  • a guidance prompt unit is configured to draw a site guidance line between the current site and the target site on the preset digestive tract UI interface when it is determined that the target site is different from the current site; When it is determined that the target site is the same as the current site, the standard movement of the endoscope is prompted according to the preset observation direction of the target observation point to be transferred determined by the guidance probability prediction unit.
  • an animated guidance prompt will be displayed on the preset upper gastrointestinal tract UI interface based on the new target site.
  • the Bézier curve will be used to draw site guidance lines inside the UI model to prompt the user to proceed.
  • the target observation point and the position of the corresponding preset target observation point prompt the user to turn left, turn right, move forward, backward or continue to observe, and draw a rotation animation on the node.
  • the guidance prompt unit is used to perform animated guidance prompts on the preset upper gastrointestinal tract UI interface according to the new target node to be transferred, and the Bézier curve is used internally in the UI model.
  • the second objective function is:
  • s n represents the probability of moving to the target observation point part n
  • M n (n) represents the observation time of the target observation point part n
  • b(n) is the optimal observation time of the target observation point part n (optimal observation time Take 10 seconds)
  • O(i) represents the set of target observation point parts corresponding to site part i.
  • This embodiment applies two objective functions to deduce the AI recognition results to obtain the transfer location.
  • an action prompt is given that is consistent with the current state. Compared with the fixed prompts in the prior art, It can effectively improve the detection effect.
  • the S501 mirror operation recognition unit is used to identify the current mirror movement. Specifically, for S30 The observation matrix is modeled to obtain a pre-trained mirror operation recognition model, which can recognize the doctor's actions such as turning the mirror, entering the mirror, and withdrawing the mirror, and obtains the current mirror movement. If it is detected that the doctor has performed an incorrect operation, a prompt will be given through the system so that doctors of different levels can perform inspections according to standardized actions.
  • a sequential queue matrix is first used to obtain the site location identification results and observation point location identification results in S20 every 0.5 seconds, and store them in the sequential queue matrix; a total of 10 seconds, a total of 20 sets of recognition results are stored; when the 11th After the second result is stored, the original result of the 1st second will be discarded, and the 11th second will be inserted at the end of the sequential queue matrix.
  • a camera action recognition model is constructed. First, collect the contents of the above sequential queue matrix, and manually mark the actions of each sequential queue matrix, including "observation in place”, “lens zoomed in”, “lens away”, “left-handed mirror”, “right-rotated mirror”, Seven regular actions of "entering the camera” and “exiting the camera” construct a data set for training the model.
  • the model uses a dynamic Bayesian network for modeling, taking each sequential queue matrix in the annotated data set as input, and using the EM algorithm to perform parameter training on the dynamic Bayesian network to obtain a preset camera action recognition model.
  • the sequential queue matrix is continuously collected according to the rules, and the mirror movement is recognized through the preset mirror movement recognition model.
  • a deviation warning unit is used to provide a prompt on the digestive tract UI interface when it is determined that the current mirror movement is inconsistent with the standard mirror movement. That is, text prompts appear on the GUI interface based on the guidance action prompted by the S40 module and the current recognized camera movement. If the camera movement is inconsistent with the transfer node, a prominent mark will appear on the GUI interface and animation guidance prompts will be given. That is to say, obtain the guidance action prompted in S40, and determine whether the guidance action prompted in S40 is consistent with the camera movement performed by the operator; if they are inconsistent, the GUI interface will provide an early warning, for example, a red exclamation mark will be displayed in the upper right corner of the animation area, giving User prompts.
  • An artificial intelligence neural network multi-label classification model adopted in various embodiments of the present invention, Multi-label recognition on gastroscopy images.
  • multiple observation points usually appear in the same field of view captured by the lens, such as the anterior wall of the gastric antrum and the lesser curvature of the gastric antrum, the lesser curvature of the gastric fundus and the posterior wall of the gastric fundus, etc.
  • Each embodiment of the present invention is closer to Actual clinical scenarios, statistics are more accurate.
  • Various embodiments of the present invention adopt a structure of 4 groups of convolutional layers + 2 groups of attention layers + 1 group of multi-label prediction layers to form the neural network model of S2.
  • the 2D image is characterized by abstraction through the 4 groups of convolutional layers, and we obtain Feature F 0 , and the first attention layer performs attention weighting on the interior of feature F to obtain feature Q 0 , and then uses the second attention layer to jointly weight features F 0 and Q 0 to obtain the feature Q 0 to
  • the inference step is intended to establish the correlation between F 0 global and F 0 key areas, thereby bringing better performance to multi-label recognition;
  • the guidance method for gastroscopy disclosed in various embodiments of the present invention can provide doctors with refined full-coverage operation prompts. When a site shift occurs, it prompts the endoscope to enter or exit, and when the target observation point shifts, it prompts left or right rotation. .
  • the method of identifying the gastroscopy physician's operating actions proposed by each embodiment of the present invention uses multi-label observation sequences and machine learning methods to deduce the doctor's current endoscopic movement, and combines the S4 module to give early warning information.
  • the presentation method and interface of doctor attention proposed by various embodiments of the present invention can record multi-label recognition results, and form a heat map representation of the recording results
  • Various embodiments of the present invention can prompt the doctor in real time about the gastroscopy route and observation angle, respectively corresponding to the doctor's advance and retreat of the mirror and lens rotation.
  • the prompts are simple and easy to understand, and reduce duplication and omission of inspection parts.
  • Various embodiments of the present invention can provide real-time feedback on the consistency between the actual operation route and the preset route, helping doctors to correct deviations as soon as possible; compared with the previous feedback that could only provide feedback on covered parts and traveled routes, the intelligent tracking and prediction of this navigation function The effect is more prominent.
  • the heat map in various embodiments of the present invention vividly records the entire process of the doctor's part observation, which is an objective feedback of the doctor's subjective attention. It is equivalent to flattening the colonoscopy process and is a reflection of the inspection. major innovations in feedback methods.
  • Various embodiments of the present invention display the doctor's examination progress by identifying the current part of the gastroscope and analyzing the relationship between the parts, providing a suitable examination path, providing the operator with more scientific endoscopy guidance, improving the completeness and fluency of the detection, and avoiding the risk of Repeated checks and omissions reduce the patient's pain.
  • the invention can enable endoscopists of different levels to improve to a level that meets basic standards.
  • An embodiment of the present invention provides an endoscopic auxiliary inspection method, as shown in Figure 3.
  • the endoscopic auxiliary inspection method includes:
  • S101 identify the collected endoscopic examination images frame by frame, determine each observation point corresponding to each frame of endoscopic examination image, and map each observation point to each preset site;
  • S104 Guide endoscopy according to the current site location and the target site.
  • the embodiment of the present invention displays the progress of the doctor's examination by identifying the current part of the gastroscope and analyzing the relationship between the parts, provides a suitable examination path, provides the operator with more scientific guidance on how to operate the endoscope, improves the completeness and fluency of detection, and avoids duplication. Check and omit to reduce the patient’s pain.
  • the invention can enable endoscopists of different levels to improve to a level that meets basic standards.
  • S101 includes: recording frame-by-frame observation completeness of site locations and observation time of observation point locations, based on the observation time of each observation point location and each station The observation time of the point location determines the observation sequence of the site location;
  • S102 includes: determining a site location transition probability matrix based on the current site location, the site location observation sequence, and the preset physical depth of the current site location; determining the maximum value in the site location transition probability matrix as the Describe the target site;
  • a site guide line between the current site and the target site is drawn on the preset digestive tract UI interface; when it is determined that the target site is different from the current site, When the current site locations are the same, the standard movement of the endoscope is prompted according to the preset observation direction of the target observation point location to be transferred determined by the guidance probability prediction unit.
  • the site location transition probability is determined through a first objective function, and the target observation point location is determined based on the observation time of the current site location and the observation location set of the current site through a second objective function;
  • the first objective function is:
  • s i′ represents the probability of transferring to target site i′
  • M k (i′) represents the observation completeness of target site i′
  • D i′ represents the physical depth of target site i′
  • D i represents the depth of site part i Physical depth
  • ⁇ is an arbitrary constant
  • the second objective function is:
  • s n represents the probability of being transferred to the target observation point part n
  • M n (n) represents the observation time of the target observation point part n
  • b(n) is the optimal observation time of the target observation point part n
  • O(i ) represents the set of target observation point parts corresponding to site part i.
  • S102 also includes: recording frame-by-frame sites determined by the anatomical part identification module.
  • the observation completeness of the parts and the observation time of the observation point parts Based on the observation time of each observation point part and the observation completeness of each station part, the observation sequence of the observation point parts is determined;
  • the observation sequence of observation points is displayed on the digestive tract UI interface in the form of a heat map.
  • the endoscopic auxiliary examination method further includes: identifying the current endoscopic movement; and when it is determined that the current endoscopic movement is inconsistent with the standard endoscopic movement, providing a prompt on the digestive tract UI interface.
  • An embodiment of the present invention provides an endoscopic auxiliary inspection device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor;
  • An embodiment of the present invention provides a computer-readable storage medium.
  • the computer-readable storage medium stores an endoscopic auxiliary inspection program.
  • the endoscopic auxiliary inspection program is executed by a processor, any one of the methods in Embodiment 2 is implemented.
  • Embodiment 2 to Embodiment 4 For the specific implementation of Embodiment 2 to Embodiment 4, please refer to Embodiment 1, which has corresponding technical effects.
  • the methods of the above embodiments can be implemented by means of software plus the necessary general hardware platform. Of course, it can also be implemented by hardware, but in many cases the former is better. implementation.
  • the technical solution of the present invention can be embodied in the form of a software product in essence or the part that contributes to the existing technology.
  • the computer software product is stored in a storage medium (such as ROM/RAM, disk, CD), including several instructions to cause a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in various embodiments of the present invention.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • General Health & Medical Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Computational Linguistics (AREA)
  • Software Systems (AREA)
  • Mathematical Physics (AREA)
  • General Engineering & Computer Science (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Artificial Intelligence (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • Computing Systems (AREA)
  • Medical Informatics (AREA)
  • Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
  • Radiology & Medical Imaging (AREA)
  • Quality & Reliability (AREA)
  • Endoscopes (AREA)

Abstract

本申请涉及一种内镜辅助检查系统、方法、装置及存储介质。所述系统包括：解剖部位识别模块，用于对采集的内镜检查图像进行逐帧识别，确定每帧内镜检查图像对应的每个观察点部位，并将每个观察点部位分别映射到各预设站点部位；关注度记录模块，用于根据记录的所述解剖部位识别模块确定的各个观察点部位的观察时间和各个站点部位的观察完整度，确定站点部位观测序列；引导模块，用于根据所述解剖部位识别模块确定的当前帧内镜检查图像对应的当前站点部位、所述站点部位观测序列以及所述当前站点部位的预设物理深度，确定待转移到的目标站点；根据所述当前站点部位和所述目标站点，引导内镜检查。本申请引导内镜检查医师按照最有效率的路线进行运镜。

Description

内镜辅助检查系统、方法、装置及存储介质
交叉引用说明
本申请要求于2022年7月15日提交中国专利局、申请号为202210828744.7，发明名称为“内镜辅助检查系统、方法、装置及存储介质”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及医疗技术领域，尤其涉及一种内镜辅助检查系统、方法、装置及存储介质。
背景技术
胃镜检查是检查上消化道异常的重要手段，是食管、胃、十二指肠等上消化道疾病诊断的金标准。在现有胃镜检查中，需要根据内窥镜拍摄到的视频图像，判断该视频图像拍摄时所在的胃部部位，通常一份完整的胃镜检查报告需要包含口咽部、食管、贲门、胃底、胃体、胃角、胃窦、幽门、十二指肠球部及降部10个部位的至少31张图片，若发现存在病灶或者可疑的部位还需靠近进行更细节的拍摄。要求消化内镜医师在胃镜检查过程中实时拍摄图片，发现可疑部位时及时进行进一步的检查。
目前，消化内镜医师需要长时间经验的积累才能顺利流畅地完成一次胃镜检查。对于经验不太丰富的消化内镜医师来说，遗漏检查部位或者未能发现可疑区域经常存在。遗漏检查部位就需要患者再重新经历一次痛苦的检查，既消磨患了者的时间与金钱，也浪费了医院的检测资源；未能发现可疑区域更是会把患者的生命置于危险的境地。另外， 消化内镜医师基本处于超负荷状态。高负荷工作会降低内镜医师内镜检查的质量，易出现检查部位覆盖不全、病灶检出不全、图像采集不全面等问题。
因此，在上消化道内镜检查过程中，对于保证检查部位覆盖齐全、病灶检出全面等问题，以及保证消化内镜医师以正确的操作手法对上消化道每一个部位黏膜进行完整仔细的观察，以提高临床胃镜质量控制等问题，亟需一种上消化道内镜辅助检查技术，从而辅助消化内镜医师顺利流畅的完成上消化道内镜检查，确保每一例上消化道内镜检查操作的正确性和黏膜观察的完整性。
发明内容
为了解决上述技术问题或者至少部分地解决上述技术问题，本申请提供了一种内镜辅助检查系统、方法、装置及存储介质。
第一方面，本申请提供了一种内镜辅助检查系统，所述内镜辅助检查系统包括：
解剖部位识别模块，用于对采集的内镜检查图像进行逐帧识别，确定每帧内镜检查图像对应的每个观察点部位，并将每个观察点部位分别映射到各预设站点部位；
关注度记录模块，用于根据记录的所述解剖部位识别模块确定的各个观察点部位的观察时间和各个站点部位的观察完整度，确定站点部位观测序列；
引导模块，用于根据所述解剖部位识别模块确定的当前帧内镜检查图像对应的当前站点部位、所述站点部位观测序列以及所述当前站点部位的预设物理深度，确定待转移到的目标站点；根据所述当前站点部位和所述目标站点，引导内镜检查。
第二方面，本申请提供了一种内镜辅助检查方法，所述内镜辅助检 查方法包括：
对采集的内镜检查图像进行逐帧识别，确定每帧内镜检查图像对应的每个观察点部位，并将每个观察点部位分别映射到各预设站点部位；
根据记录的各个观察点部位的观察时间和各个站点部位的观察完整度，确定站点部位观测序列；
根据当前帧内镜检查图像对应的当前站点部位、所述站点部位观测序列以及所述当前站点部位的预设物理深度，确定待转移到的目标站点；
根据所述当前站点部位和所述目标站点，引导内镜检查。
第三方面，本申请提供了一种内镜辅助检查装置，所述内镜辅助检查装置包括：存储器、处理器及存储在所述存储器上并可在所述处理器上运行的计算机程序；
所述计算机程序被所述处理器执行时，实现如上所述的内镜辅助检查方法的步骤。
第四方面，本申请提供了一种计算机可读存储介质，所述计算机可读存储介质上存储有内镜辅助检查程序，所述内镜辅助检查程序被处理器执行时，实现如上所述的内镜辅助检查方法的步骤。
本申请实施例提供的上述技术方案与现有技术相比具有如下优点：
本申请各实施例通过对内镜观察上消化道黏膜部位的进行识别，挖掘消化道各部位之间的关联关系，并对运镜观察的操作路径进行提示，引导内镜检查医师按照最有效率的路线进行运镜，避免重复检查，节省时间，提高胃镜检查的流畅度，并且保证胃镜观察的完整性，不产生遗漏和死角。
附图说明
此处的附图被并入说明书中并构成本说明书的一部分，示出了符合本发明的实施例，并与说明书一起用于解释本发明的原理。
为了更清楚地说明本发明实施例或现有技术中的技术方案，下面将对实施例或现有技术描述中所需要使用的附图作简单地介绍，显而易见地，对于本领域普通技术人员而言，在不付出创造性劳动性的前提下，还可以根据这些附图获得其他的附图。
图1为本申请各个实施例提供的内镜辅助检查系统的框图；
图2为本申请各个实施例提供的卷积层的示意图；
图3为本申请各个实施例提供的内镜辅助检查方法的流程图。
具体实施方式
应当理解，此处所描述的具体实施例仅仅用以解释本发明，并不用于限定本发明。
在后续的描述中，使用用于表示元件的诸如“模块”、“部件”或“单元”的后缀仅为了有利于本发明的说明，其本身没有特定的意义。因此，“模块”、“部件”或“单元”可以混合地使用。
实施例一
本发明实施例提供一种内镜辅助检查系统，如图1所示，所述内镜辅助检查系统包括：
S10视频获取模块，利用视频采集卡，将HDMI、DVI、SDI、S-Video等数字/模拟视频信号传到内镜辅助检查系统的主机内，通过Opencv读取视频信号，逐帧转码成RGB图像格式内镜检查图像；
S20解剖部位识别模块，用于对采集的内镜检查图像进行逐帧识别，确定每帧内镜检查图像对应的每个观察点部位，并将每个观察点部位 分别映射到各预设站点部位；
S30关注度记录模块，用于根据记录的所述解剖部位识别模块确定的各个观察点部位的观察时间和各个站点部位的观察完整度，确定站点部位观测序列；
S40引导模块，用于根据所述解剖部位识别模块确定的当前帧内镜检查图像对应的当前站点部位、所述站点部位观测序列以及所述当前站点部位的预设物理深度，确定待转移到的目标站点；根据所述当前站点部位和所述目标站点，引导内镜检查。
本发明实施例特别适用于胃镜检查，本发明实施例对内镜观察上消化道黏膜部位的进行识别，挖掘消化道各部位之间的关联关系，并对运镜观察的操作路径进行提示，引导内镜检查医师按照最有效率的路线进行运镜，避免重复检查，节省时间，提高胃镜检查的流畅度，并且保证胃镜观察的完整性，不产生遗漏和死角。
以下用一具体实施方式，详细描述本发明实施例的内镜辅助检查系统，包括S10视频获取模块、S20解剖部位识别模块、S30关注度记录模块、S40引导模块和S50监测模块；其中，
S10视频获取模块
利用视频采集卡，将HDMI、DVI、SDI、S-Video等数字/模拟视频信号传到内镜辅助检查系统的主机内，通过Opencv读取视频信号，逐帧转码成RGB图像格式。
S20解剖部位识别模块
S201模型训练单元，用于训练人工智能神经网络多标签识别模型；其中，该模型可以由ResNet、VGG、DenseNet等具有分类功能的神经网络实现，本实施方式中为了提高分类效果，可选地选择空间分组结构的卷积层、自注意力网络层、全连接层以及预测层构建网络结构。也就 是说，可选地用于根据空间分组结构的卷积层、自注意力网络层、全连接层以及预测层构建网络结构，将预先标注的样本图片作为所述网络结构的输入信息进行训练，获得神经网络多标签识别模型；例如，卷积层、自注意力网络层、全连接层和预测层的组数分别为4、2、1和1。训练过程包括：
步骤2011，构建网络结构：网络主要由4组包含空间分组结构的卷积层、2组自注意力网络层、1组全连接层以及1组预测层串联得到；
具体地，所述包含空间分组结构的卷积层如图2所示，内部组数(Group)设置为64，瓶颈宽度设置为32(bottleneck-width)，用于对输入的图像进行维度变换及特征提取，最终输出特征向量F0；
对于2组自注意力网络层，首先利用第一组自注意力网络层，对特征向量F0进行自注意力加权，得到特征向量Q0；然后利用第二组自注意力网络层，对特征向量F0与特征向量Q0联合进行自注意力加权，得到特征向量
对于全连接层，用于对特征向量及特征向量F0进行线性投影，得到一维特征向量F；
对于预测层，利用所述一维特征向量F，对预设的每个类别分别求导，得到概率(p0,p1,...,pn)，具体公式为：
其中，pn指的是对类别n求解的概率，为针对类别n的可训练超参数，bn为针对类别n的可训练的偏置超参数
步骤2012，对预先收集的样本图片(上消化道部位观察图片)进行标注，其标注结果为预设的细分解剖部位(即观察点部位)中的1-n个；
步骤2013，将上述内窥镜图像作为神经网络模型的输入信息，并 根据预先标注好的多标签结果，利用损失函数进行计算，所述损失函数公式如下；
其中L(x)指的是给定输入图像x在训练过程中的损失值；N指所有类别数；n指当前类别；pn指的是n类别的预测概率；γ指的是偏置值，通常取0或1，本实施例取1；yn指的是给定输入图像x是否具有标签n的真值。
步骤2014，得到所述损失值L(x)后，利用反向传播更新模型参数；具体地，训练预先设置500轮，每轮训练中对所有标注数据重复上述步骤2011、2012，直至模型达到预设轮数或损失值小于1e-4时停止，得到构建好的神经网络模型。
S202识别单元，用于通过所述模型训练单元训练的神经网络多标签识别模型对采集的内镜检查图像进行逐帧识别，确定每帧内镜检查图像对应的每个观察点部位；也就是说，利用上述训练好的模型，对获取到的逐帧图像进行识别，最终经过预测层的预测，得到细分解剖部位识别结果，即当前观测的观察点部位T{1,2…,j}；
S203解剖部位映射单元，用于根据预先设置的观察点部位和站点部位的映射关系，将每个观察点部位分别映射到各预设站点部位。换言之，对上述得到的细分解剖部位识别结果进行所属站点部位映射，映射关系如下表，得到站点部位识别结果，即当前观测的站点部位Ti；其中可以根据内镜检查顺序，对各站点部位分别设置对应的物理深度，本实施方式中物理深度为9级。

本实施例中通过4组卷积层对2D内窥镜图像进行特征抽象，得到输出特征向量，并通过第一注意力层对一维特征向量的内部进行注意力加权得到加权特征Q0，再通过第二注意力层对特征F0及Q0进行联合加权得到特征Q0到的推理步骤可以建立F0全局与F0重点区域的关联性，从而为多标签识别带来更好的性能。
S30关注度记录模块
S301时间记录单元，用于记录解剖部位识别模块确定的逐帧的观察点部位的观察时间及站点部位的观察完整度，根据各个观察点部位的观察时间和各个站点部位的观察完整度，确定站点部位观测序列和观察点部位观测序列。
首先利用时间记录单元记录站点部位观察完整度及观察点部位观察时间。具体地，可以将所得到的观察点部位识别结果的帧数记录下来，并实时将累计记录量转化为时间进行存储，得到观察点部位观测序列Mn＝[m1,m2,…,mn]；
其次，根据观察点部位观测时间及部位映射关系，统计每个站点部位的完整度(完整度根据下述公式得到)，得到站点部位集合观测序列Mk＝[o1,o2,…,ok]
其中，b表示观察点部位最优观测时间，mn代表站点部位对应的每个目标观察点部位观测时间，n表示站点部位对应目标观察点部位数量
S302热力图显示单元，用于通过热力图形式在所述消化道UI界面显示所述观察点部位观测序列。即，将时间记录单元的观察点部位观测序列Mn通过热力图形式显示到GUI界面上。在内镜检查完成后，可查看热力图来回顾检查全程中，医生对于各个部位关注的时间长短，根据时间长短可以事后分析整个检查过程，哪些部位缺少足够的关注，哪些部位得到的医生重点关注。具体地，本实施例利用一个热力图模块体现关注度时间。热力图由WebCanvas绘制，其分块分区根据细分解剖部位而定，继而将记录到的观察点部位观测序列Mn＝[m1,m2,…,mn]映射成饱和度，给对应的色块着色。
S40引导模块
用于在胃镜检查中的路线引导规划，根据计算结果，在显示界面上提示最合理的检查路径，引导医生进行最有效率的运镜操作，包括：
S401引导概率预测单元，用于根据所述当前站点部位、所述站点部位观测序列以及所述当前站点部位的预设物理深度，确定站点部位转移概率矩阵；将所述站点部位转移概率矩阵内极大值确定为所述目标站点。具体地，利用第一目标函数，根据S20中当前观测站点部位Ti、S30中站点部位观测序列Mk及预设物理深度D，计算得到站点部位转移概率矩阵Sk＝[s1,s2,…,sk](sk表示从当前站点部位转移到站点部位k的概率)，并计算出矩阵内极大值，得到新的目标站点Ti′或目标观察点Tn。
所述第一目标函数为：
其中，si′表示转移到目标站点i′的概率；Mk(i′)表示目标站点i′的观察完整度；Di′表示目标站点i′的物理深度；Di表示站点部位i的物理深度；λ为任意常数(λ>0)，本实施例取值为1。
S402引导提示单元，用于当判定所述目标站点与所述当前站点部位不同时，在预设的消化道UI界面上，绘制所述当前站点和所述目标站点之间的站点引导线；当判定所述目标站点与所述当前站点部位相同时，根据所述引导概率预测单元确定的待转移到的目标观察点部位的预设观察方位，提示内镜检查的标准运镜动作。
详细地，若出现新的目标站点，则根据新的目标站点，在预设好的上消化道UI界面上，进行动画引导提示，其UI模型内部利用Bézier曲线绘制站点引导线，提示使用者进镜或退镜到目标站点Ti′；
若持续在当前站点部位进行观察，则根据第二目标函数得到新的 目标观察点以及对应预设的目标观察点部位方位，提示使用者左旋、右旋、前进、后退或持续观察，并绘制在节点的旋转动画。
详细地，若Ti′≠Ti，利用引导提示单元，根据新的待转移到的目标节点，在预设好的上消化道UI界面上，进行动画引导提示，其UI模型内部利用Bézier曲线绘制站点引导线，并绘制从Ti到Tk的绿色动态闪烁连接线进行提示；同时，根据预设的物理深度，做出“进镜”或“退镜”的文字提示。
若Ti′＝Ti，表示应持续在当前站点部位进行观测；使用第二目标函数，计算出观察点部位转移概率矩阵Sn＝[s1,s2,…,sn](sn表示从当前观察点部位转移到目标观察点部位n的概率)，并计算出矩阵内极大值，得到新的目标观察点部位Tn；
所述第二目标函数为：
其中，sn表示转移到目标观察点部位n的概率；Mn(n)表示目标观察点部位n的观察时间；b(n)为目标观察点部位n的最优观察时间(最优观察时间取10秒)；O(i)表示站点部位i所对应的目标观察点部位集合。
利用引导提示单元，根据预设的观察点部位方位，提示左旋、右旋、前进、后退、或持续观察的文字，并绘制在Ti节点的旋转操作动画。
本实施例应用了两个目标函数对AI识别结果进行推导得到转移部位，同时根据预设部位在现实世界的方位、深度，给予符合当前状态的动作提示，相对于现有技术中的固定提示，可以有效提高检测效果。
S50监测模块
S501运镜操作识别单元，用于识别当前运镜动作。具体地，对S30 的观测矩阵进行建模，得到预先训练好的运镜操作识别模型，实现对医生的转镜、进镜、退镜等行为动作进行识别，得到当前运镜动作。如果检测到医生执行了错误的操作动作，则通过系统进行提示，使得不同水平的医生都能按照规范动作进行检查。
详细地，首先采用一种顺序队列矩阵，每0.5秒获取一次S20中站点部位识别结果及观察点部位识别结果，存储到顺序队列矩阵中；共存储10秒，共计20组识别结果；当第11秒的结果存储进来后，会将原先第1秒的结果舍弃，将第11秒插入到顺序队列矩阵的末尾。
其次构建一个运镜动作识别模型。首先采集上述顺序队列矩阵的内容，并由人工标注每个顺序队列矩阵的动作，包括“原地观察”、“镜头拉近”、“镜头远离”、“左旋镜”、“右旋镜”、“进镜”、“退镜”七个常规动作，构建出用于训练模型的数据集。
模型采用动态贝叶斯网络进行建模，将标注好的数据集中每个顺序队列矩阵作为输入，利用EM算法对动态贝叶斯网络进行参数训练，得到预设的运镜动作识别模型。
最后在检查过程中按照规则不间断采集顺序队列矩阵，通过预设的运镜动作识别模型对运镜动作进行识别。
S502偏离预警单元，用于当判定当前运镜动作与所述标准运镜动作不一致时，在所述消化道UI界面进行提示。即，根据S40模块提示的引导动作与识别到的当前运镜动作，并在GUI界面出现文字提示。若其运镜动作与转移节点出现相悖时，在GUI界面出现显著标识，并给予动画引导提示。也就是说，获取S40中提示的引导动作，判别S40中提示的引导动作与操作者执行的运镜动作是否一致；若不一致GUI界面进行预警提示，例如，在动画区域右上角显示红色叹号，给予使用者提示。
本发明各个实施例采用的一种人工智能神经网络多标签分类模型， 对胃镜图像进行多标签识别。可以在胃镜检查中，镜头拍摄的同一视野中通常会出现多个观察点部位，例如胃窦前壁与胃窦小弯、胃底小弯与胃底后壁等，本发明各个实施例更贴近临床实际场景，统计更精准。
本发明各个实施例采用的一种4组卷积层+2组注意力层+1组多标签预测层的结构组成S2的神经网络模型，通过4组卷积层对2D图像进行特征抽象，得到特征F0，并通过第一注意力层对特征F的内部进行注意力加权得到特征Q0，再通过第二注意力层对特征F0及Q0进行联合加权得到特征Q0到的推理步骤意在建立F0全局与F0重点区域的关联性，从而为多标签识别带来更好的性能；
本发明各个实施例公开的胃镜检查的引导方式，可以为医生提供精细化的全覆盖操作提示，当出现站点转移时提示进镜或退镜、当出现目标观察点部位转移时提示左旋、右旋。
本发明各实施例提出的对胃镜检查医师操作动作的识别方式，通过多标签观测序列和机器学习方法，来推理出医生的当前的运镜动作，并联合S4模块给出预警信息。
本发明各个实施例提出的医生关注度的呈现方式、界面；能对多标签识别结果进行记录，并将记录结果形成热力图表示
本发明各个实施例可以实时提示医生胃镜检查的行进路线与观察角度，分别对应医生的进退镜与镜头旋转，提示简单易懂，减少检查部位的重复及遗漏。
本发明各个实施例可以实时反馈实际操作路线与预设路线之间的一致性，帮助医生第一时间纠正偏差；相比之前仅能反馈已覆盖部位和已行进路线，本导航功能的智能跟踪预测效果更加突出。
本发明各个实施例中热力图生动记录医生部位观察全程，是医生主观关注度的一个客观反馈，相当于把肠镜检查流程平面化，是检查反 馈手段的重大创新。
本发明各个实施例通过对胃镜当前部位的识别和部位关联的分析，显示医生检查进度，提供合适的检查路径，为操作者提供更科学的运镜引导，提高检测的完整度和流畅性，避免重复检查和遗漏，降低患者的痛苦感受。本发明可以让不同水平的内镜医生，都能提高到符合基本规范的水平。
实施例二
本发明实施例提供一种内镜辅助检查方法，如图3所示，所述内镜辅助检查方法包括：
S101，对采集的内镜检查图像进行逐帧识别，确定每帧内镜检查图像对应的每个观察点部位，并将每个观察点部位分别映射到各预设站点部位；
S102，根据记录的各个观察点部位的观察时间和各个站点部位的观察完整度，确定站点部位观测序列；
S103，根据当前帧内镜检查图像对应的当前站点部位、所述站点部位观测序列以及所述当前站点部位的预设物理深度，确定待转移到的目标站点；
S104，根据所述当前站点部位和所述目标站点，引导内镜检查。
本发明实施例通过对胃镜当前部位的识别和部位关联的分析，显示医生检查进度，提供合适的检查路径，为操作者提供更科学的运镜引导，提高检测的完整度和流畅性，避免重复检查和遗漏，降低患者的痛苦感受。本发明可以让不同水平的内镜医生，都能提高到符合基本规范的水平。
在一些实施例中，S101包括：记录逐帧的站点部位的观察完整度及观察点部位的观察时间，根据各个观察点部位的观察时间和各个站 点部位的观察时间，确定站点部位观测序列；
S102包括：根据所述当前站点部位、所述站点部位观测序列以及所述当前站点部位的预设物理深度，确定站点部位转移概率矩阵；将所述站点部位转移概率矩阵内极大值确定为所述目标站点；
当判定所述目标站点与所述当前站点部位不同时，在预设的消化道UI界面上，绘制所述当前站点和所述目标站点之间的站点引导线；当判定所述目标站点与所述当前站点部位相同时，根据所述引导概率预测单元确定的待转移到的目标观察点部位的预设观察方位，提示内镜检查的标准运镜动作。
可选地，通过第一目标函数确定站点部位转移概率，通过第二目标函数确定根据所述当前站点部位的观察时间和所述当前站点的观察部位集合，确定所述目标观察点部位；
所述第一目标函数为：
其中，si′表示转移到目标站点i′的概率；Mk(i′)表示目标站点i′的观察完整度；Di′表示目标站点i′的物理深度；Di表示站点部位i的物理深度；λ为任意常数；
所述第二目标函数为：
其中，sn表示待转移到目标观察点部位n的概率；Mn(n)表示目标观察点部位n的观察时间；b(n)为目标观察点部位n的最优观察时间；O(i)表示站点部位i所对应的目标观察点部位集合。
可选地，S102还包括：记录解剖部位识别模块确定的逐帧的站点 部位的观察完整度及观察点部位的观察时间，根据各个观察点部位的观察时间和各个站点部位的观察完整度，确定观察点部位观测序列；
通过热力图形式在所述消化道UI界面显示所述观察点部位观测序列。
在一些实施例中，内镜辅助检查方法还包括：识别当前运镜动作；当判定当前运镜动作与所述标准运镜动作不一致时，在所述消化道UI界面进行提示。
实施例三
本发明实施例提供一种内镜辅助检查装置，所述内镜辅助检查装置包括：存储器、处理器及存储在所述存储器上并可在所述处理器上运行的计算机程序；
所述计算机程序被所述处理器执行时，实现如实施例二中任意一项所述的内镜辅助检查方法的步骤。
实施例四
本发明实施例提供一种计算机可读存储介质，所述计算机可读存储介质上存储有内镜辅助检查程序，所述内镜辅助检查程序被处理器执行时，实现如实施例二中任意一项所述的内镜辅助检查方法的步骤。
实施例二至实施例四的具体实现可以参阅实施例一，具有相应的技术效果。
需要说明的是，在本文中，术语“包括”、“包含”或者其任何其他变体意在涵盖非排他性的包含，从而使得包括一系列要素的过程、方法、物品或者装置不仅包括那些要素，而且还包括没有明确列出的其他要素，或者是还包括为这种过程、方法、物品或者装置所固有的要素。在没有更多限制的情况下，由语句“包括一个……”限定的要素，并不排除在包括该要素的过程、方法、物品或者装置中还存在另外的相同要素。
上述本发明实施例序号仅仅为了描述，不代表实施例的优劣。
通过以上的实施方式的描述，本领域的技术人员可以清楚地了解到上述实施例方法可借助软件加必需的通用硬件平台的方式来实现，当然也可以通过硬件，但很多情况下前者是更佳的实施方式。基于这样的理解，本发明的技术方案本质上或者说对现有技术做出贡献的部分可以以软件产品的形式体现出来，该计算机软件产品存储在一个存储介质(如ROM/RAM、磁碟、光盘)中，包括若干指令用以使得一台终端(可以是手机，计算机，服务器，空调器，或者网络设备等)执行本发明各个实施例所述的方法。
上面结合附图对本发明的实施例进行了描述，但是本发明并不局限于上述的具体实施方式，上述的具体实施方式仅仅是示意性的，而不是限制性的，本领域的普通技术人员在本发明的启示下，在不脱离本发明宗旨和权利要求所保护的范围情况下，还可做出很多形式，这些均属于本发明的保护之内。

Claims (10)

  1. 一种内镜辅助检查系统，其特征在于，所述内镜辅助检查系统包括：
    解剖部位识别模块，用于对采集的内镜检查图像进行逐帧识别，确定每帧内镜检查图像对应的每个观察点部位，并将每个观察点部位分别映射到各预设站点部位；
    关注度记录模块，用于根据记录的所述解剖部位识别模块确定的各个观察点部位的观察时间和各个站点部位的观察完整度，确定站点部位观测序列；
    引导模块，用于根据所述解剖部位识别模块确定的当前帧内镜检查图像对应的当前站点部位、所述站点部位观测序列以及所述当前站点部位的预设物理深度，确定待转移到的目标站点；根据所述当前站点部位和所述目标站点，引导内镜检查。
  2. 根据权利要求1所述的内镜辅助检查系统，其特征在于，所述关注度记录模块包括时间记录单元；所述引导模块包括引导概率预测单元和引导提示单元；
    所述时间记录单元，用于记录解剖部位识别模块确定的逐帧的站点部位的观察完整度及观察点部位的观察时间，根据各个观察点部位的观察时间和各个站点部位的观察完整度，确定站点部位观测序列；
    所述引导概率预测单元，用于根据所述当前站点部位、所述站点部位观测序列以及所述当前站点部位的预设物理深度，确定站点部位转移概率矩阵；将所述站点部位转移概率矩阵内极大值确定为所述目标站点；
    所述引导提示单元，用于当判定所述目标站点与所述当前站点部位不同时，在预设的消化道UI界面上，绘制所述当前站点和所述目标 站点之间的站点引导线；当判定所述目标站点与所述当前站点部位相同时，根据所述引导概率预测单元确定的待转移到的目标观察点部位的预设观察方位，提示内镜检查的标准运镜动作。
  3. 根据权利要求2所述的内镜辅助检查系统，其特征在于，所述引导概率预测单元，还用于通过第一目标函数确定站点部位转移概率，通过第二目标函数确定根据所述当前站点部位的观察时间和所述当前站点的观察部位集合，确定所述目标观察点部位；
    所述第一目标函数为：
    其中，si′表示转移到目标站点i′的概率；Mk(i′)表示目标站点i′的观察完整度；Di′表示目标站点i′的物理深度；Di表示站点部位i的物理深度；λ为任意常数；
    所述第二目标函数为：
    其中，sn表示待转移到目标观察点部位n的概率；Mn(n)表示目标观察点部位n的观察时间；b(n)为目标观察点部位n的最优观察时间；O(i)表示站点部位i所对应的目标观察点部位集合。
  4. 根据权利要求2所述的内镜辅助检查系统，其特征在于，所述关注度记录模块还包括热力图显示单元；
    所述时间记录单元，用于记录解剖部位识别模块确定的逐帧的站点部位的观察完整度及观察点部位的观察时间，根据各个观察点部位的观察时间和各个站点部位的观察完整度，确定观察点部位观测序列；
    所述热力图显示单元，用于通过热力图形式在所述消化道UI界面 显示所述观察点部位观测序列。
  5. 根据权利要求2所述的内镜辅助检查系统，其特征在于，所述内镜辅助检查系统还包括监测模块；所述监测模块包括运镜操作识别单元和偏离预警单元；
    所述运镜操作识别单元，用于识别当前运镜动作；
    所述偏离预警单元，用于当判定当前运镜动作与所述标准运镜动作不一致时，在所述消化道UI界面进行提示。
  6. 根据权利要求1-4中任意一项所述的内镜辅助检查系统，其特征在于，所述解剖部位识别模块包括模型训练单元、识别单元和解剖部位映射单元；
    所述模型训练单元，用于根据空间分组结构的卷积层、自注意力网络层、全连接层以及预测层构建网络结构，将预先标注的样本图片作为所述网络结构的输入信息进行训练，获得神经网络多标签识别模型；
    所述识别单元，用于通过所述模型训练单元训练的神经网络多标签识别模型对采集的内镜检查图像进行逐帧识别，确定每帧内镜检查图像对应的每个观察点部位；
    所述解剖部位映射单元，用于根据预先设置的观察点部位和站点部位的映射关系，将每个观察点部位分别映射到各预设站点部位。
  7. 根据权利要求6所述的内镜辅助检查系统，其特征在于，所述模型训练单元，具体用于通过所述空间分组结构的卷积层对输入的样本图片进行维度变换及特征提取，输出特征向量F0；
    通过第一组自注意力网络层，对特征向量F0进行自注意力加权，得到特征向量Q0；通过第二组自注意力网络层，对特征向量F0与特征向量Q0联合进行自注意力加权，得到特征向量
    通过所述全连接层对特征向量及特征向量F0进行线性投影，得到一维特征向量F；
    根据所述一维特征向量F，通过所述预测层，利用对预设的每个类别分别求导，得到多标签识别概率；
    并根据预先标注好的多标签结果，利用损失函数进行计算，得到损失值后，利用反向传播更新模型参数；在达到预设轮数或损失值小于预设值停止，得到神经网络多标签识别模型。
  8. 一种内镜辅助检查方法，其特征在于，所述内镜辅助检查方法包括：
    对采集的内镜检查图像进行逐帧识别，确定每帧内镜检查图像对应的每个观察点部位，并将每个观察点部位分别映射到各预设站点部位；
    根据记录的各个观察点部位的观察时间和各个站点部位的观察完整度，确定站点部位观测序列；
    根据当前帧内镜检查图像对应的当前站点部位、所述站点部位观测序列以及所述当前站点部位的预设物理深度，确定待转移到的目标站点；
    根据所述当前站点部位和所述目标站点，引导内镜检查。
  9. 一种内镜辅助检查装置，其特征在于，所述内镜辅助检查装置包括：存储器、处理器及存储在所述存储器上并可在所述处理器上运行的计算机程序；
    所述计算机程序被所述处理器执行时，实现如权利要求8中所述的内镜辅助检查方法的步骤。
  10. 一种计算机可读存储介质，其特征在于，所述计算机可读存储 介质上存储有内镜辅助检查程序，所述内镜辅助检查程序被处理器执行时，实现如权利要求8中所述的内镜辅助检查方法的步骤。
PCT/CN2023/097402 2022-07-15 2023-05-31 内镜辅助检查系统、方法、装置及存储介质 WO2024012080A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202210828744.7 2022-07-15
CN202210828744.7A CN114913173B (zh) 2022-07-15 2022-07-15 内镜辅助检查系统、方法、装置及存储介质

Publications (1)

Publication Number Publication Date
WO2024012080A1 true WO2024012080A1 (zh) 2024-01-18

Family

ID=82772331

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2023/097402 WO2024012080A1 (zh) 2022-07-15 2023-05-31 内镜辅助检查系统、方法、装置及存储介质

Country Status (2)

Country Link
CN (1) CN114913173B (zh)
WO (1) WO2024012080A1 (zh)

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114913173B (zh) * 2022-07-15 2022-10-04 天津御锦人工智能医疗科技有限公司 内镜辅助检查系统、方法、装置及存储介质
CN116563216B (zh) * 2023-03-31 2024-02-20 河北大学 基于标准站点智能识别的内镜超声扫查控制优化系统及方法

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20180260971A1 (en) * 2017-03-08 2018-09-13 Esaote S.P.A. Method For Controlling Image Appearance Features In MRI Systems, Image Appearance Feature Control User Interface Operating According To The Said Method And MRI System Comprising Said User Interface
CN109146884A (zh) * 2018-11-16 2019-01-04 青岛美迪康数字工程有限公司 内窥镜检查监控方法及装置
CN109846444A (zh) * 2019-02-26 2019-06-07 重庆金山医疗器械有限公司 一种胶囊自动导航系统及导航方法
CN111415564A (zh) * 2020-03-02 2020-07-14 武汉大学 基于人工智能的胰腺超声内镜检查导航方法及系统
CN114913173A (zh) * 2022-07-15 2022-08-16 天津御锦人工智能医疗科技有限公司 内镜辅助检查系统、方法、装置及存储介质

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108615037A (zh) * 2018-05-31 2018-10-02 武汉大学人民医院（湖北省人民医院） 基于深度学习的可控胶囊内镜操作实时辅助系统及操作方法
CN113164010A (zh) * 2018-11-21 2021-07-23 株式会社Ai医疗服务 利用消化器官的内窥镜影像的疾病的诊断支持方法、诊断支持系统、诊断支持程序及存储了该诊断支持程序的计算机可读记录介质
CN113344926B (zh) * 2021-08-05 2021-11-02 武汉楚精灵医疗科技有限公司 胆胰超声图像识别方法、装置、服务器及存储介质
CN114224448B (zh) * 2021-12-21 2023-11-10 武汉大学 穿刺路径规划装置、设备和计算机可读存储介质

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20180260971A1 (en) * 2017-03-08 2018-09-13 Esaote S.P.A. Method For Controlling Image Appearance Features In MRI Systems, Image Appearance Feature Control User Interface Operating According To The Said Method And MRI System Comprising Said User Interface
CN109146884A (zh) * 2018-11-16 2019-01-04 青岛美迪康数字工程有限公司 内窥镜检查监控方法及装置
CN109846444A (zh) * 2019-02-26 2019-06-07 重庆金山医疗器械有限公司 一种胶囊自动导航系统及导航方法
CN111415564A (zh) * 2020-03-02 2020-07-14 武汉大学 基于人工智能的胰腺超声内镜检查导航方法及系统
CN114913173A (zh) * 2022-07-15 2022-08-16 天津御锦人工智能医疗科技有限公司 内镜辅助检查系统、方法、装置及存储介质

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
XU, MING ET AL.: "Research on Assisted Quality Control System of Digestive Endoscopy based on Deep Learning", CHINESE JOURNAL OF DIGESTIVE ENDOSCOPY, vol. 38, no. 2, 28 February 2021 (2021-02-28), pages 107 - 110, XP009552639, ISSN: 1007-5232 *

Also Published As

Publication number Publication date
CN114913173A (zh) 2022-08-16
CN114913173B (zh) 2022-10-04

Similar Documents

Publication Publication Date Title
US12059125B2 (en) Systems and methods for generating and displaying a study of a stream of in-vivo images
WO2024012080A1 (zh) 内镜辅助检查系统、方法、装置及存储介质
CN109146884B (zh) 内窥镜检查监控方法及装置
US10860930B2 (en) Learning method, image recognition device, and computer-readable storage medium
JP6215236B2 (ja) 生体内画像ストリーム中の運動性事象を表示するためのシステムおよび方法
US20220296081A1 (en) Method for real-time detection of objects, structures or patterns in a video, an associated system and an associated computer readable medium
WO2021147429A1 (zh) 内窥镜图像展示方法、装置、计算机设备及存储介质
US20110032347A1 (en) Endoscopy system with motion sensors
EP1847940A2 (en) System for assessing a patient condition and method for processing related data
JP2019520916A (ja) 対象の消化管における粘膜疾患の評価及び監視のためのシステム及び方法
CN101273916B (zh) 评估患者状况的系统和方法
JP7503592B2 (ja) 内視鏡検査処置の視覚化のためのユーザインターフェース
WO2021139672A1 (zh) 医疗辅助操作方法、装置、设备和计算机存储介质
US20220369920A1 (en) Phase identification of endoscopy procedures
US20230206435A1 (en) Artificial intelligence-based gastroscopy diagnosis supporting system and method for improving gastrointestinal disease detection rate
CN111144271A (zh) 一种内镜下自动识别活检部位及活检数量的方法及系统
CN116563216B (zh) 基于标准站点智能识别的内镜超声扫查控制优化系统及方法
JP2007105458A (ja) 画像データベースにおける画像の認識のためのシステム及び方法
US20220361739A1 (en) Image processing apparatus, image processing method, and endoscope apparatus
CN113763360A (zh) 消化内镜模拟器检查质量评估方法及系统
CN110742690A (zh) 一种用于配置内窥镜的方法及终端设备
WO2023095208A1 (ja) 内視鏡挿入ガイド装置、内視鏡挿入ガイド方法、内視鏡情報取得方法、ガイドサーバ装置、および画像推論モデル学習方法
CN118076315A (zh) 用于分析内窥镜检查过程的检查质量的计算机实现的系统和方法
Figueiredo et al. Dissimilarity measure of consecutive frames in wireless capsule endoscopy videos: a way of searching for abnormalities
KR102714219B1 (ko) 위장 질환 및 위암 검출률 향상을 위한 인공 지능 기반 위 내시경 영상 진단 보조 시스템 및 방법

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 23838574

Country of ref document: EP

Kind code of ref document: A1