CN116542952A - Endoscopic coverage rate evaluation method and system based on three-dimensional reconstruction - Google Patents

Endoscopic coverage rate evaluation method and system based on three-dimensional reconstruction Download PDF

Info

Publication number
CN116542952A
CN116542952A CN202310554551.1A CN202310554551A CN116542952A CN 116542952 A CN116542952 A CN 116542952A CN 202310554551 A CN202310554551 A CN 202310554551A CN 116542952 A CN116542952 A CN 116542952A
Authority
CN
China
Prior art keywords
frame
coverage rate
image
pose
depth
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202310554551.1A
Other languages
Chinese (zh)
Inventor
余德平
刘也琪
刘苓
文雅娜
杨胜源
张斌
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Sichuan University
Original Assignee
Sichuan University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Sichuan University filed Critical Sichuan University
Priority to CN202310554551.1A priority Critical patent/CN116542952A/en
Publication of CN116542952A publication Critical patent/CN116542952A/en
Pending legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G06T7/0012—Biomedical image inspection
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0464—Convolutional networks [CNN, ConvNet]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/70—Denoising; Smoothing
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10004—Still image; Photographic image
    • G06T2207/10012—Stereo images
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30004—Biomedical image processing
    • G06T2207/30096—Tumor; Lesion
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
    • Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
    • Y02P90/30—Computing systems specially adapted for manufacturing

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • General Health & Medical Sciences (AREA)
  • Software Systems (AREA)
  • Health & Medical Sciences (AREA)
  • General Engineering & Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Computing Systems (AREA)
  • Data Mining & Analysis (AREA)
  • Molecular Biology (AREA)
  • Evolutionary Computation (AREA)
  • Mathematical Physics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Artificial Intelligence (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Quality & Reliability (AREA)
  • Geometry (AREA)
  • Computer Graphics (AREA)
  • Radiology & Medical Imaging (AREA)
  • Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
  • Medical Informatics (AREA)
  • Image Analysis (AREA)

Abstract

The invention relates to the technical field of image processing, in particular to an endoscopic coverage rate assessment method, an endoscopic coverage rate assessment system, a storage medium and computer equipment based on three-dimensional reconstruction, which comprise the following steps: s01, screening information frame images: inputting the inspection image into a trained classified neural network, screening information frames, and if the screening result meets the requirement, entering S02, otherwise, early warning; s02, colonoscope depth prediction: inputting the information frame screened in the step S01 into a trained depth prediction neural network to obtain a depth map D; s03, estimating the pose of the colonoscope: inputting the RGB-D data output in the step S02 into a trained pose estimation network, and outputting to obtain the pose P of the endoscope; s04, reconstructing a colon three-dimensional model; s05, calculating the coverage rate, reconstructing a three-dimensional model of the intestinal tract according to the endoscopic image in the inspection process, calculating the coverage rate, and carrying out prompt and early warning when the coverage rate is less than 90%, so that the occurrence of missed diagnosis is avoided, and the inspection efficiency is improved.

Description

Endoscopic coverage rate evaluation method and system based on three-dimensional reconstruction
Technical Field
The invention relates to the technical field of image processing, in particular to an endoscopic coverage rate assessment method, an endoscopic coverage rate assessment system, a storage medium and computer equipment based on three-dimensional reconstruction.
Background
Colorectal cancer is one of the major cancers that endanger life health as a fifth cancer of national morbidity and mortality. Colonoscopy is the most common examination method for screening colorectal cancer, usually using colonoscopy to examine various parts of colorectal, and during the examination, an endoscopist controls an electronic endoscope to observe whether various parts have lesions such as polyps and tumors, and if abnormality is found, biopsy can be performed by the electronic endoscope operation. However, in the actual operation process, due to the complex colon structure, the operation level of the endoscopist is uneven, and the examination effect is easily reduced due to missed diagnosis. The reasons for missed diagnosis are mainly two aspects, namely polyps appear in the field of view and the physician cannot detect them, which is mainly caused by fatigue or inexperience of the physician. On the other hand polyps are not present in the field of view, which is complicated by the fact that the main digestive tract is, or the lack of experience of the endoscopist results in a blind spot during the examination.
It is generally considered that in a section of qualified colonoscopy process, the check coverage rate of the colon should reach more than 90% -95%, but no effective measurement mechanism can intuitively and accurately reflect the index at present. The domestic and foreign endoscopy society indicates that polyp detection rate and time for mirror withdrawal are adopted as standards for ensuring the detection quality, because the improvement of the two indexes depends on non-blind area detection in the detection. However, the above two indexes only provide subjective feedback on the physician's operation during the examination. To essentially solve the problem that polyps may not appear in the field of view, it is necessary to quantify the inspection coverage and feed back in real time.
Disclosure of Invention
The invention aims to provide an endoscopic coverage rate assessment method, an endoscopic coverage rate assessment system, a storage medium and computer equipment based on three-dimensional reconstruction, wherein a three-dimensional model of intestinal tracts is reconstructed according to endoscopic images in an inspection process, the coverage rate is calculated, prompt and early warning are carried out when the coverage rate is less than 90%, the occurrence of missed diagnosis is avoided, and the inspection efficiency is improved.
In order to achieve the above purpose, the present invention provides the following technical solutions:
an endoscopic coverage rate evaluation method based on three-dimensional reconstruction is characterized by comprising the following steps:
s01, inputting the inspection image into a trained classified neural network, screening information frames, and entering S02 if the screening result meets the requirement, otherwise, early warning;
s02, inputting the information frames screened in the step S01 into a trained depth prediction neural network to obtain a depth map D;
s03, inputting the RGB-D data output in the step S02 into a trained pose estimation network, and outputting to obtain the pose P of the endoscope;
s04, inputting the depth map D, the pose P and the image information obtained in the step S02 and the step S03 into a three-dimensional reconstruction algorithm to reconstruct the surface of the inspected part;
s05, calculating coverage rate according to the three-dimensional model reconstructed in the step S04.
Further, the judgment algorithm of the information frame in step S01 is as follows: the endoscopically acquired RGB image is converted to HSV color space and noise is removed using a gaussian filter.
Further, the training method of the deep prediction neural network in step S02 combines supervised training and unsupervised training, and specifically includes the steps of:
s021, performing unsupervised training by using a real endoscope picture, and ensuring that the network can learn the characteristics of the picture of the checked part;
s022, freezing the shallow layer of the network, and performing supervised training by using a phantom data set with real depth to ensure the accuracy of network depth prediction.
Further, the specific method of the unsupervised training in step S021 is as follows: inputting an image of a detected partInputting the three pictures into a depth prediction neural network and a pose estimation network to obtain three corresponding depth maps ∈ ->Obtaining a pose transformation matrix in a pose estimation network>Wherein>And->Two-frame image->Is the original frame,/->Is a converted frame obtained by converting the original frame by depth and pose, < >>Is the target frame, i.e. the frame next to the original frame, also +.>Target frame desired to be converted, +.>Is->Frame at next moment, transition frame->And original frame->The mapping relation of the corresponding points is as follows:
，
wherein the method comprises the steps ofFor points in the original frame +.>For a point in the converted frame after conversion of the original frame,/for the original frame>For the conversion pose matrix of the original frame to the target frame,/for the conversion pose matrix of the original frame to the target frame,>is an internal reference of the camera; conversion frame->And (2) target frame->The minimum error is the unsupervised loss function as follows:
，
wherein, the liquid crystal display device comprises a liquid crystal display device,for average absolute error loss, +.>Is a loss of structural similarity; />Is a transition frame->The value of the pixel p involved, +.>Is the target frame->The value of the pixel P comprised, P being the point comprised in the picture, +.>The total number of points in the picture; />Respectively is image +.>Mean value of->Is->Covariance of image,>respectively is image +.>Variance of the image; />Usually take the value +.>L is the gray dynamic range of the image, and if the image is in unit8 format, L takes 255.
Further, the specific steps of step S04 are as follows:
s041, inputting a current frame RGB image, converting the current frame RGB image into a three-dimensional space according to a depth map D and a pose P, and creating the current frame RGB image as a face element;
s042, each time a new frame is input, projecting the new frame to the frame according to the depth map D and the pose P of the frameThree-dimensional space, based on its depth z, calculates a threshold interval，/>Taking a value of 0.5 as a threshold parameter, if the existing adjacent pixel points in the space are in a threshold range, the current bin is regarded as a supporting bin, the projection point is indicated to exist and is not required to be processed, if the existing adjacent pixel points are smaller than the minimum value of the interval, the projection point is regarded as a conflict bin, the projection point is indicated to be in conflict with the established bin and needs to be removed, if the existing adjacent pixel points are larger than the maximum value of the interval, the projection point is regarded as an enlarged bin, and the projection point is indicated to be outside the established bin and needs to be newly established;
s043, traversing the surface elements to perform triangular gridding after all the surface elements are established, and finally reconstructing an effective three-dimensional model surface;
and S044, the reconstructed three-dimensional model is fused with the RGB image to render color, and a relatively real colon three-dimensional model is presented.
Further, the coverage rate calculation method in step S05 is as follows: the three-dimensional model is a lumen structure, a central axis is calculated in the lumen structure, an average radius is calculated according to the distance between the central axis and the surface of the colon, a cylinder is fitted according to the average radius, and the complete surface area of the colon is estimated according to the surface area of the cylinder; the reconstructed three-dimensional model of the colon calculates the reconstructed area and the ratio of the reconstructed area to the estimated complete surface area is the coverage rate.
Further, in step S01, the acquisition frequency of the inspection image is acquired every 10 seconds, and the screening result is determined by: judging whether the number of information frames in the period of time is more than 90%, if so, entering a step S02, otherwise, sending out early warning to remind an operator to check the part again, and if the coverage rate calculated in the step S05 is less than 90%, sending out early warning reminding.
In another aspect, the present invention provides an endoscopic coverage assessment system based on three-dimensional reconstruction, comprising: the data screening module is used for screening the information frames of the inspection images;
the image processing module is used for processing the information frame to obtain a depth map D and a pose P;
the three-dimensional model reconstruction module is used for carrying out three-dimensional reconstruction on the depth map D and the pose image to obtain the surface of the inspected part;
and the data calculation module is used for calculating the reconstructed three-dimensional model to obtain coverage rate.
Another aspect of the present invention provides a storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the endoscopic coverage assessment method based on three-dimensional reconstruction described above.
In another aspect, the present invention provides a computer device, including a storage medium, a processor, and a computer program stored in the storage medium and executable by the processor, where the computer program when executed by the processor implements the steps of the method for evaluating coverage of endoscopy based on three-dimensional reconstruction.
Compared with the prior art, the technical scheme of the invention has the following beneficial effects:
(1) The non-information frames of the acquired images are removed by screening the non-information frames, so that the detection quality and efficiency of operators on the detected parts are improved, and the three-dimensional reconstruction of the detected parts is effectively ensured;
(2) The depth prediction neural network combines supervised training and unsupervised training, which is favorable for obtaining a depth map with high accuracy, compared with the prior art: the supervised training mode needs the corresponding label, but the real endoscope cannot obtain the depth label, the body model data with the real depth is obtained, the characteristics of the body model cannot completely simulate the real detected part, and if the real detected part is adopted for the unsupervised training, the accuracy is difficult to ensure, so that the method has the advantage.
(3) And reconstructing a three-dimensional model of the detected part by screening and processing the acquired images, calculating coverage rate, and judging whether the detection omission phenomenon exists according to a set threshold value, thereby improving the quality and efficiency of detection.
Drawings
FIG. 1 is a schematic diagram of a flow frame according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of a judgment algorithm of an information frame according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of an unsupervised real colon depth prediction framework in accordance with an embodiment of the present invention;
FIG. 4 is a plot of colon volume model data with true depth information according to an embodiment of the present invention;
FIG. 5 is a schematic diagram of a supervised body modular data depth prediction framework in accordance with an embodiment of the present invention;
FIG. 6 is a schematic diagram of a colonoscope pose estimation algorithm based on RGB-D according to an embodiment of the present invention;
FIG. 7 is a schematic representation of a three-dimensional surface reconstruction of a colon based on a bin in accordance with an embodiment of the present invention;
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
The drawings are for illustrative purposes only and are not to be construed as limiting the present patent;
the terminology used in the embodiments of the application is for the purpose of describing particular embodiments only and is not intended to be limiting of the embodiments of the application. As used in the examples of this application and the appended claims, the singular forms "a," "an," "the," and "the" include
"the" is also intended to include the majority form unless the context clearly indicates otherwise. It should also be understood that the term "and/or" as used herein refers to and encompasses any or all possible combinations of one or more of the associated listed items.
When the following description refers to the accompanying drawings, the same numbers in different drawings refer to the same or similar elements, unless otherwise indicated. The implementations described in the following exemplary examples are not representative of all implementations consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with some aspects of the present application as detailed in the accompanying claims.
In the description of this application, it should be understood that the terms "first," "second," "third," and the like are used merely to distinguish between similar objects and are not necessarily used to describe a particular order or sequence, nor should they be construed to indicate or imply relative importance. The specific meaning of the terms in this application will be understood by those of ordinary skill in the art as the case may be.
Furthermore, in the description of the present application, unless otherwise indicated, "a plurality" means two or more. "and/or",
the association relationship describing the association object may represent that there are three relationships, e.g., a and/or B, may represent: a exists alone, A and B exist together, and B exists alone. The character "/" generally indicates that the context-dependent object is an "or" relationship. The invention is further illustrated in the following figures and examples.
In order to solve the limitations of the prior art, the present embodiment provides a technical solution, and the technical solution of the present invention is further described below with reference to the drawings and the embodiments.
Referring to fig. 1, an endoscopic coverage evaluation method based on three-dimensional reconstruction, which mainly takes an inspection image of an olympus electronic endoscope widely used in the prior art as input, and the inspection site is a colon, comprises the following steps:
s01, screening information frame images: the acquired inspection images are input into a trained classified neural network for information frame screening, and S02 is carried out if screening results meet requirements, otherwise, early warning is carried out, the classified neural network can be any classified neural network such as VGG16 and ResNet101, the VGG16 model is well suitable for classification and positioning tasks, and the names of the VGG16 model are abbreviations from oxford university geometry group (Visual Geometry Group). The number of convolutional layers is a number of 6 configurations for VGG, A, A-LRN, B, C, D, E, respectively, depending on the size of the convolutional kernel, with D and E being the most commonly used VGG16 and VGG19.ResNet was proposed in 2015 because it is "simple and practical" and many methods have been developed to accomplish on the basis of ResNet50 or ResNet101, and have found wide application in detection, segmentation, identification, etc.
The training mode of the classified neural network adopts the training mode of other classified networks in the field in the prior art, the acquisition frequency of the inspection image is that the image of the Olympus endoscope is acquired once every 10 seconds, and the judging method of the screening result is as follows: judging whether the number of information frames in the period of time is more than 90%, if so, entering step S02, otherwise, sending out early warning to remind an operator to check the part again;
s02, colonoscope depth prediction: inputting the information frame screened in the step S01 into a trained depth prediction neural network to obtain a depth map D;
s03, estimating the pose of the colonoscope: inputting the RGB-D data output in the step S02 into a trained pose estimation network, and outputting to obtain the pose P of the colonoscope, wherein the RGB-D data are paired RGB images and depth images D, and the RGB images are checked images of the colon, wherein the pose estimation network can be any pose estimation network based on the RGB-D data or a traditional pose estimation algorithm such as PnP/ICP and the like, pnP is a motion estimation method for solving 3D to 2D points, ICP is a pose estimation problem aiming at a plurality of 3D to 3D points, and the calculation mode adopts a common method in the field in the prior art;
s04, reconstructing a colon three-dimensional model: inputting the depth map D, the pose P and the image information obtained in the step S02 and the step S03 into a three-dimensional reconstruction algorithm to reconstruct the surface of the inspected part;
s05, calculating coverage rate: and (3) calculating the coverage rate according to the three-dimensional model reconstructed in the step (S04), and if the calculated coverage rate is smaller than 90%, sending out early warning reminding.
Referring to fig. 2, since the operation of the digestive tract endoscopic process is complex and the steps of flushing and inflating are accompanied, an image incapable of judging the surface characteristics of the colon is often caused, such as an image in the case of extremely close distance of the inner wall of the digestive tract or polluted lens, the image is blurred, the surface condition of the colon cannot be judged, the image has no detection meaning, the detection quality and efficiency of the digestive tract inspection by a clinician can be reduced, and the subsequent three-dimensional reconstruction of the digestive tract is prevented, so that non-information frames need to be eliminated, and the judgment algorithm of the information frames is as follows: the RGB image collected by the electronic endoscope is converted into HSV color space, a Gaussian filter is used for removing noise, a slightly blurred frame can be classified into a blurred category to smooth image features, wherein the blurred frame and an information frame are marked by a professional doctor, the blurred frame is usually an image of which the image can hardly recognize the characteristics of the inner wall of the intestinal canal due to the operation that a lens is attached to the inner wall of the intestinal canal or water is sprayed, the slightly blurred frame is defined as an image which is usually collected under the conditions of slight virtual focus or body fluid adhesion to the lens, the image can roughly recognize the outline of the inner wall of the intestinal canal, but the image is not useful for extracting the characteristics of the intestinal canal, and the image can be removed by adding filtering. The images are then input into a trained classification neural network, which may be any other image recognition network such as VGG16, res net50, alexNet, etc., where VGG16/res net50/AlexNet is a classical convolutional neural network that is widely used in the industry for image classification. The training sets that train the network are divided into two categories, information frames and non-information frames.
Referring to fig. 3-5, in step S02 colonoscope depth prediction, the pictures are information frames containing useful information, and the neural network is generally classified into supervised and unsupervised training. The supervised training mode requires a label corresponding to the model, however, a real colonoscope cannot obtain a depth label, body model data with real depth is provided, and the characteristics of the body model cannot completely simulate a real colon. If the real colon is adopted for unsupervised training, the precision is difficult to ensure; if the phantom data with the real depth are adopted, the model data with the real depth cannot have a good prediction effect on the real colon, and the training method of the depth prediction neural network adopts the combination of supervised training and unsupervised training, and comprises the following specific steps:
s021, performing unsupervised training by using a real endoscope picture, and ensuring that a network can learn the characteristics of the picture of the checked part, wherein the structure of the network is divided into a depth prediction network and a pose estimation network, the two networks are trained together, the depth prediction network and the pose estimation network can be any network used for depth prediction or pose estimation, such as RNN-DP, poseCNN and the like, wherein the RNN-DP is one of common depth prediction networks, the PoseCNN is one of pose estimation networks widely used in the industry, and the networks are widely used in the industry and have public source codes;
s022, freezing the shallow layer of the network, performing supervised training by using a phantom data set with real depth, guaranteeing the accuracy of network depth prediction, and performing supervised training after freezing the shallow layer by adopting the depth prediction network in S021.
Referring to fig. 3, the specific method of the unsupervised training in step S021 is as follows: inputting colon imagesInputting the three pictures into a depth prediction neural network and a pose estimation network to obtain three corresponding depth mapsObtaining a pose transformation matrix in a pose estimation network>Since the colonoscope is checked during the process of moving back, the image of the next frame often contains the image of the previous frame, so that the point on the image of the current frame can be obtained by the corresponding point in the previous frame according to pose transformation and depth map conversion. Wherein>And->Two-frame image->Is the original frame,/->Is a converted frame obtained by converting the original frame by depth and pose, < >>Is the target frame, i.e. the frame next to the original frame, also +.>Target frame desired to be converted, +.>Is->Frame at next moment, transition frame->And original frame->The mapping relation of the corresponding points is as follows:
，
wherein the method comprises the steps ofFor points in the original frame +.>For a point in the converted frame after conversion of the original frame,/for the original frame>For the conversion pose matrix of the original frame to the target frame,/for the conversion pose matrix of the original frame to the target frame,>is an internal reference of the camera; according to the above relation, original frame +.>Outputting depth via depth prediction network>Original frame->And target frame->Input pose network output pose +.>Then obtaining a conversion frame +.>Switch frame->And (2) target frame->The minimum error is the unsupervised loss function as follows:
，
wherein, the liquid crystal display device comprises a liquid crystal display device,for average absolute error loss, +.>Is a loss of structural similarity; />Is a transition frame->The value of the pixel p involved, +.>Is the target frame->The value of the pixel P comprised, P being the point comprised in the picture, +.>The total number of points in the picture; />Respectively is image +.>Mean value of->Is->Covariance of image,>respectively is image +.>Variance of the image; />Usually take the value +.>L is the gray dynamic range of the image, and if the image is in unit8 format, i.e. an unsigned 8bit integer, is the digital type of the stored image, L takes 255.
Referring to fig. 4 and 5, S022 performs supervised training. The data set selects phantom data with real depth information, the network structure with supervision training is consistent with the depth prediction network in the step S021, parameters of the frozen shallow layer are unchanged, and only deep network parameters are trained. The loss function is calculated between the predicted depth map and the real depth map, and can be used by cross entropy loss function or the previous stageAnd->。
Referring to fig. 6, the paired real colon pictures and depth maps obtained in the previous stage are input into a pose estimation network to obtain the endoscope pose of the corresponding frame, and the endoscope pose is represented by a rotation matrix R and a translation matrix T. The pose estimation network in this step may be any pose estimation network based on RGB-D data, or may use conventional pose calculation algorithms, such as triangulation, pnP or ICP methods.
Referring to fig. 7, step S04 reconstructs a three-dimensional model of the colon: and inputting the image information data with the depth information D and the pose P output in the previous stage into a surface reconstruction algorithm based on the surface elements to obtain the three-dimensional model of the colon. The principal steps of the surface reconstruction algorithm based on the surface elements are as follows:
s041, inputting a current frame RGB image, converting the current frame RGB image into a three-dimensional space according to a depth map D and a pose P, and creating the current frame RGB image as a face element;
s042, each time a new frame is input, projecting the new frame into a three-dimensional space according to the depth map D and the pose P of the frame, and calculating a threshold interval according to the depth z of the new frame，/>The method comprises the steps that a threshold value is set as a threshold value parameter, namely a threshold value for controlling the creation of a bin, the value is 0.5, if an existing adjacent pixel point in a space is within a threshold range, the current bin is regarded as a supporting bin, the fact that the projection point is existing and is not required to be processed is indicated, if the minimum value of the projection point is smaller than a section, the projection point is regarded as a conflict bin, the projection point is required to be truncated with the created bin, if the projection point is larger than the maximum value of the section, the projection point is regarded as an enlarged bin, and the projection point is beyond the created bin, and a new bin is required to be created;
s043, traversing the surface elements to perform triangular gridding after all the surface elements are established, and finally reconstructing an effective three-dimensional model surface;
and S044, the reconstructed three-dimensional model is fused with the RGB image to render color, and a relatively real colon three-dimensional model is presented.
The coverage rate calculation method in step S05 is as follows: the three-dimensional model is a lumen structure, a central axis is calculated in the lumen structure, an average radius is calculated according to the distance between the central axis and the surface of the colon, a cylinder is fitted according to the average radius, and the complete surface area of the colon is estimated according to the surface area of the cylinder; the reconstructed colon three-dimensional model calculates the reconstructed area, and the ratio of the reconstructed area to the estimated complete surface area is coverage rate, and the application in the examination process is that: and in the colonoscopy process, a doctor acquires colonoscopy pictures, inputs the colonoscopy pictures into the algorithm, reconstructs the three-dimensional surface of the colon according to the algorithm, outputs the current coverage rate, displays the three-dimensional surface on a second screen, and carries out early warning prompt when the information frame is too small or the current coverage rate is too low.
The colonoscopy coverage rate evaluation algorithm provided by the invention comprises the following steps: (1) The non-information frames of the acquired images are removed by screening the non-information frames, so that the detection quality and efficiency of operators on the detected parts are improved, and the three-dimensional reconstruction of the detected parts is effectively ensured;
(2) The depth prediction neural network combines supervised training and unsupervised training, which is favorable for obtaining a depth map with high accuracy, compared with the prior art: the supervised training mode needs the corresponding label, but the real endoscope cannot obtain the depth label, the body model data with the real depth is obtained, the characteristics of the body model cannot completely simulate the real detected part, and if the real detected part is adopted for the unsupervised training, the accuracy is difficult to ensure, so that the method has the advantage.
(3) And reconstructing a three-dimensional model of the detected part by screening and processing the acquired images, calculating coverage rate, and judging whether the detection omission phenomenon exists according to a set threshold value, thereby improving the quality and efficiency of detection.
The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
The foregoing description of the preferred embodiments of the invention is not intended to be limiting, but rather is intended to cover all modifications, equivalents, and alternatives falling within the spirit and principles of the invention.

Claims (10)

1. An endoscopic coverage rate evaluation method based on three-dimensional reconstruction is characterized by comprising the following steps:
s01, screening information frame images: inputting the inspection image into a trained classified neural network, screening information frames, and if the screening result meets the requirement, entering S02, otherwise, early warning;
s02, depth prediction: inputting the information frame screened in the step S01 into a trained depth prediction neural network to obtain a depth map D;
s03, pose estimation: inputting the RGB-D data output in the step S02 into a trained pose estimation network, and outputting to obtain the pose P of the endoscope;
s04, reconstructing a three-dimensional model: inputting the depth map D, the pose P and the image information obtained in the step S02 and the step S03 into a three-dimensional reconstruction algorithm to reconstruct the surface of the inspected part;
s05, calculating coverage rate: and (3) calculating coverage rate according to the three-dimensional model reconstructed in the step S04.
2. The method for evaluating the coverage rate of an endoscopic examination based on three-dimensional reconstruction according to claim 1, wherein the judgment algorithm of the information frame in step S01 is as follows: the endoscopically acquired RGB image is converted to HSV color space and noise is removed using a gaussian filter.
3. The method for evaluating the coverage rate of an endoscopic examination based on three-dimensional reconstruction according to claim 1 or 2, wherein the training method of the depth prediction neural network in step S02 adopts a combination of supervised training and unsupervised training, and specifically comprises the following steps:
s021, performing unsupervised training by using a real endoscope picture, and ensuring that the network can learn the characteristics of the picture of the checked part;
s022, freezing the shallow layer of the network, and performing supervised training by using a phantom data set with real depth to ensure the accuracy of network depth prediction.
4. The method for evaluating coverage rate of an endoscopic examination based on three-dimensional reconstruction according to claim 3, wherein the specific method for unsupervised training in step S021 is as follows: inputting an image of a detected partInputting the three pictures into a depth prediction neural network and a pose estimation network to obtain three corresponding depth maps ∈ ->Obtaining a pose transformation matrix in a pose estimation network>Wherein>And->Two-frame image->Is the original frame,/->Is a converted frame obtained by converting the original frame by depth and pose, < >>Is the target frame, i.e. the frame next to the original frame, also +.>Desired to be converted intoTarget frame of arrival->Is->Frame at next moment, transition frame->And original frame->The mapping relation of the corresponding points is as follows:
，
wherein the method comprises the steps ofFor points in the original frame +.>For a point in the converted frame after conversion of the original frame,/for the original frame>For the conversion pose matrix of the original frame to the target frame,/for the conversion pose matrix of the original frame to the target frame,>is an internal reference of the camera; conversion frame->And (2) target frame->The minimum error is the unsupervised loss function as follows:
，
wherein, the liquid crystal display device comprises a liquid crystal display device,for average absolute error loss, +.>Is a loss of structural similarity; />Is a transition frame->The value of the pixel p involved, +.>Is the target frame->The value of the pixel P comprised, P being the point comprised in the picture, +.>The total number of points in the picture; />Respectively is image +.>Mean value of->Is->Covariance of image,>respectively, imagesVariance of the image; />Usually take the value +.>L is the gray dynamic range of the image.
5. The method for evaluating the coverage rate of an endoscopic examination based on three-dimensional reconstruction according to claim 4, wherein the specific steps of step S04 are as follows:
s041, inputting a current frame RGB image, converting the current frame RGB image into a three-dimensional space according to a depth map D and a pose P, and creating the current frame RGB image as a face element;
s042, each time a new frame is input, projecting the new frame into a three-dimensional space according to the depth map D and the pose P of the frame, and calculating a threshold interval according to the depth z of the new frame，/>Taking a value of 0.5 as a threshold parameter, if the existing adjacent pixel points in the space are in a threshold range, the current bin is regarded as a supporting bin, the projection point is indicated to exist and is not required to be processed, if the existing adjacent pixel points are smaller than the minimum value of the interval, the projection point is regarded as a conflict bin, the projection point is indicated to be in conflict with the established bin and needs to be removed, if the existing adjacent pixel points are larger than the maximum value of the interval, the projection point is regarded as an enlarged bin, and the projection point is indicated to be outside the established bin and needs to be newly established;
s043, traversing the surface elements to perform triangular gridding after all the surface elements are established, and finally reconstructing an effective three-dimensional model surface;
and S044, rendering colors by fusing the reconstructed three-dimensional model with the RGB image, and displaying a relatively real three-dimensional model of the detected part.
6. The method for evaluating coverage rate of an endoscopic examination based on three-dimensional reconstruction according to claim 5, wherein the method for calculating coverage rate in step S05 comprises: the three-dimensional model is a lumen structure, a central axis is calculated in the lumen structure, an average radius is calculated according to the distance between the central axis and the surface of the colon, a cylinder is fitted according to the average radius, and the complete surface area of the colon is estimated according to the surface area of the cylinder; the reconstructed three-dimensional model of the colon calculates the reconstructed area and the ratio of the reconstructed area to the estimated complete surface area is the coverage rate.
7. The method for evaluating the coverage rate of an endoscopic examination based on three-dimensional reconstruction according to claim 6, wherein the acquisition frequency of the examination image in step S01 is acquired every 10 seconds, and the screening result is determined by: judging whether the number of information frames in the period of time is more than 90%, if so, entering a step S02, otherwise, sending out early warning to remind an operator to check the part again, and if the coverage rate calculated in the step S05 is less than 90%, sending out early warning reminding.
8. An endoscopic coverage assessment system based on three-dimensional reconstruction, comprising:
the data screening module is used for screening the information frames of the inspection images;
the image processing module is used for processing the information frame to obtain a depth map D and a pose P;
the three-dimensional model reconstruction module is used for carrying out three-dimensional reconstruction on the depth map D, the pose P and the image to obtain the surface of the inspected part;
and the data calculation module is used for calculating the reconstructed three-dimensional model to obtain coverage rate.
9. A storage medium having a computer program stored thereon, characterized by: the computer program, when executed by a processor, implements the steps of the three-dimensional reconstruction-based endoscopic coverage assessment method as defined in any one of claims 1 to 7.
10. A computer device, characterized by: comprising a storage medium, a processor, a computer program stored in the storage medium and executable by the processor, which computer program, when executed by the processor, realizes the steps of the three-dimensional reconstruction-based endoscopic coverage assessment method according to any one of claims 1 to 7.
CN202310554551.1A 2023-05-17 2023-05-17 Endoscopic coverage rate evaluation method and system based on three-dimensional reconstruction Pending CN116542952A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202310554551.1A CN116542952A (en) 2023-05-17 2023-05-17 Endoscopic coverage rate evaluation method and system based on three-dimensional reconstruction

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202310554551.1A CN116542952A (en) 2023-05-17 2023-05-17 Endoscopic coverage rate evaluation method and system based on three-dimensional reconstruction

Publications (1)

Publication Number Publication Date
CN116542952A true CN116542952A (en) 2023-08-04

Family

ID=87443288

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202310554551.1A Pending CN116542952A (en) 2023-05-17 2023-05-17 Endoscopic coverage rate evaluation method and system based on three-dimensional reconstruction

Country Status (1)

Country Link
CN (1) CN116542952A (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117796745A (en) * 2024-02-29 2024-04-02 四川大学 Method for estimating advancing and retreating distance of digestive endoscope lens

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117796745A (en) * 2024-02-29 2024-04-02 四川大学 Method for estimating advancing and retreating distance of digestive endoscope lens
CN117796745B (en) * 2024-02-29 2024-05-03 四川大学 Method for estimating advancing and retreating distance of digestive endoscope lens

Similar Documents

Publication Publication Date Title
JP5113841B2 (en) Computer-aided analysis using video from an endoscope
US10957043B2 (en) AI systems for detecting and sizing lesions
CN108292366B (en) System and method for detecting suspicious tissue regions during endoscopic surgery
JP6150583B2 (en) Image processing apparatus, endoscope apparatus, program, and operation method of image processing apparatus
JP5576782B2 (en) Image processing apparatus, image processing method, and image processing program
JP5851160B2 (en) Image processing apparatus, operation method of image processing apparatus, and image processing program
WO2008044466A1 (en) Image processing device, image processing method, and image processing program
CN112001895B (en) Thyroid calcification detection device
CN113017702B (en) Method and system for identifying extension length of small probe of ultrasonic endoscope and storage medium
CN116542952A (en) Endoscopic coverage rate evaluation method and system based on three-dimensional reconstruction
CN117012344B (en) Image analysis method for 4CMOS camera acquisition
CN111784686A (en) Dynamic intelligent detection method, system and readable storage medium for endoscope bleeding area
WO2022251112A1 (en) Phase identification of endoscopy procedures
CN111563876A (en) Medical image acquisition method and display method
Izadi et al. Whitenner-blind image denoising via noise whiteness priors
US20230110263A1 (en) Computer-implemented systems and methods for analyzing examination quality for an endoscopic procedure
Georgieva et al. Ultrasound image processing for improving diagnose of renal diseases
Liedlgruber et al. Endoscopic image processing-an overview
Ohura et al. Computer-aided diagnosis method for detecting early esophageal cancer from endoscopic image by using dyadic wavelet transform and fractal dimension
Singh et al. Dark channel processing for medical image enhancement
Martínez et al. Estimating the size of polyps during actual endoscopy procedures using a spatio-temporal characterization
CN112734749A (en) Vocal leukoplakia auxiliary diagnosis system based on convolutional neural network model
JP2019213929A (en) Image processing apparatus and image processing method
Armin Automated visibility map from colonoscopy video to support clinical diagnosis and improve the quality of colonoscopy
Robinson et al. Segmentation of the biliary tree in mrcp data

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination