WO2023124876A1

WO2023124876A1 - Endoscope image detection auxiliary system and method, medium and electronic device

Info

Publication number: WO2023124876A1
Application number: PCT/CN2022/137565
Authority: WO
Inventors: 边成; 李剑; 赵秋阳; 赵家英; 石小周; 杨志雄; 薛云鹤; 李帅; 刘威
Original assignee: 小荷医疗器械(海南)有限公司
Priority date: 2021-12-29
Filing date: 2022-12-08
Publication date: 2023-07-06
Also published as: CN114332019B; CN114332019A

Abstract

The present invention relates to an endoscope image detection auxiliary system and method, a medium and an electronic device. The system comprises an image processing module which is used for processing in real time an endoscope image collected by an endoscope so as to obtain a tissue image; a cavity positioning module which is used for determining a target point of a tissue cavity corresponding to the tissue image, wherein, when there is a tissue cavity in the tissue image, the target point is a center point of the tissue cavity corresponding to the endoscope image, and when there is no tissue cavity in the tissue image, the target point is a direction point of the tissue cavity corresponding to the endoscope image; a polyp recognition module which is used for carrying out real-time polyp recognition on the tissue image obtained by the endoscope in a withdrawal mode, thus obtaining a polyp recognition result; and a display module which is used for displaying the target point and the polyp recognition result.

Description

Endoscope image detection auxiliary system, method, medium and electronic device

This disclosure claims the priority of the Chinese patent application number "202111643635.X" filed on December 29, 2021, with the title of "Endoscopic Image Detection Auxiliary System, Method, Medium, and Electronic Equipment". The entire content of the patent application is incorporated by reference in this disclosure.

technical field

The present disclosure relates to the field of image processing, and in particular, to an endoscope image detection auxiliary system, method, medium and electronic equipment.

Background technique

As a commonly used inspection method, endoscopy can enable doctors to observe the real state of the internal environment of the human body more intuitively. It is widely used in the medical field for the detection of polyp lesions and cancer, so that patients can receive effective intervention in the early stages of the disease. and treatment.

Endoscopic examination is usually divided into the process of entering and withdrawing the mirror. The process of entering the mirror is usually controlled by the physician. 1. The quality of patient’s intestinal preparation is uneven, and the endoscope and intestinal tract are always in a state of relative motion. There may be bubble occlusion, overexposure, motion blur, etc. in the endoscopic image, and blind areas of vision are prone to appear, which makes doctors You need to rely on your own experience to control the endoscope, which may lead to slow or even failure of the endoscope, and may even cause harm to the examinee. higher.

Contents of the invention

This Summary is provided to introduce a simplified form of concepts that are described in detail later in the Detailed Description. This summary of the invention is not intended to identify key features or essential features of the claimed technical solution, nor is it intended to be used to limit the scope of the claimed technical solution.

In a first aspect, the present disclosure provides an endoscopic image detection assistance system, the system comprising:

The image processing module is used to process the endoscope image collected by the endoscope in real time to obtain tissue images;

The cavity positioning module is used to determine the target point of the tissue cavity corresponding to the tissue image, and the target point is used to indicate the next target movement point of the endoscope at its current position; wherein, When there is a tissue cavity in the tissue image, the target point is the center point of the tissue cavity corresponding to the endoscopic image, and when there is no tissue cavity in the tissue image, the target point is The direction point of the tissue cavity corresponding to the endoscopic image;

The polyp identification module is used to perform real-time polyp identification on the tissue image obtained by the endoscope in the retracting mirror mode, and obtain a polyp identification result;

A display module, configured to display the target point and the polyp recognition result.

In a second aspect, the present disclosure provides a method for assisting endoscopic image detection, the method comprising:

Real-time processing of endoscopic images collected by the endoscope to obtain tissue images;

In response to determining that the endoscope is in the mirror-in mode, determine a target point of the tissue cavity corresponding to the tissue image, and the target point is used to indicate the next time the endoscope is in the mirror-in mode at its current position. Target moving point; wherein, when there is a tissue cavity in the tissue image, the target point is the center point of the tissue cavity corresponding to the endoscopic image, and when there is no tissue cavity in the tissue image , the target point is a direction point of the tissue cavity corresponding to the endoscopic image;

In response to determining that the endoscope is in the retracting mirror mode, perform real-time polyp identification on the tissue image obtained by the endoscope in the retracting mirror mode, and obtain a polyp identification result;

In the image display area in the display interface, the target point and the polyp recognition result are displayed.

In a third aspect, the present disclosure provides a computer-readable medium on which a computer program is stored, and when the program is executed by a processing device, the steps of the method described in the second aspect are implemented.

In a fourth aspect, the present disclosure provides an electronic device, including:

a storage device on which a computer program is stored;

A processing device configured to execute the computer program in the storage device to implement the steps of the method of the second aspect.

Thus, through the above technical solution, the target point in the tissue image can be determined through real-time detection and recognition of the cavity in the tissue image during the endoscope's lensing process, thereby providing reliable support for the lensing process. Accurate automatic navigation improves the efficiency and accuracy of endoscope entry, reduces the high requirements for physician experience in the use of endoscopes, avoids harm to the examinee, and improves user experience.

Other features and advantages of the present disclosure will be described in detail in the detailed description that follows.

Description of drawings

The above and other features, advantages and aspects of the various embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic and that elements and elements are not necessarily drawn to scale. In the attached picture:

Fig. 1 is a block diagram of an endoscopic image detection auxiliary system provided according to an embodiment of the present disclosure;

2A and 2B are schematic diagrams showing target points according to an embodiment of the present disclosure;

Fig. 3 is a schematic diagram of a display interface provided according to an embodiment of the present disclosure;

Figure 4 is a schematic diagram of a three-dimensional reconstruction of the intestinal tract;

Fig. 5 is a flowchart of an endoscopic image detection assistance method provided according to an embodiment of the present disclosure;

FIG. 6 is a schematic structural diagram showing an electronic device suitable for implementing an embodiment of the present disclosure.

Detailed ways

Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the drawings, it should be understood that the disclosure may be embodied in various forms and should not be construed as limited to the embodiments set forth herein; A more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for exemplary purposes only, and are not intended to limit the protection scope of the present disclosure.

It should be understood that the various steps described in the method implementations of the present disclosure may be executed in different orders, and/or executed in parallel. Additionally, method embodiments may include additional steps and/or omit performing illustrated steps. The scope of the present disclosure is not limited in this regard.

As used herein, the term "comprise" and its variations are open-ended, ie "including but not limited to". The term "based on" is "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one further embodiment"; the term "some embodiments" means "at least some embodiments." Relevant definitions of other terms will be given in the description below.

It should be noted that concepts such as "first" and "second" mentioned in this disclosure are only used to distinguish different devices, modules or units, and are not used to limit the sequence of functions performed by these devices, modules or units or interdependence.

It should be noted that the modifications of "one" and "multiple" mentioned in the present disclosure are illustrative and not restrictive, and those skilled in the art should understand that unless the context clearly indicates otherwise, it should be understood as "one or more" multiple".

The names of messages or information exchanged between multiple devices in the embodiments of the present disclosure are used for illustrative purposes only, and are not used to limit the scope of these messages or information.

As shown in FIG. 1, it is a block diagram of an endoscopic image detection auxiliary system provided according to an embodiment of the present disclosure. As shown in FIG. 1, the endoscopic image detection auxiliary system 10 may include:

The image processing module 100 is configured to perform real-time processing on endoscopic images collected by the endoscope to obtain tissue images.

Among them, in medical endoscope image recognition, the endoscope performs real-time shooting inside the living body such as the human body to obtain a video stream, so that frames can be extracted from the video stream according to a preset acquisition period to acquire endoscopic images. As an example, the collected endoscopic image can be processed, such as cropping, normalizing, resampling, etc. to process the endoscopic image to a preset size, so as to obtain the tissue image, that is, in the current detection process, the corresponding The image of the tissue in order to facilitate the unified processing of the tissue image. Exemplarily, the currently detected tissue may be intestinal tract, chest cavity, abdominal cavity, etc.

The original signal of the image collected by the endoscope usually contains the equipment information of the endoscope and the personal information of the examinee. In the embodiment of the present disclosure, the image processing module can process the tissue based on the Yolo V4 algorithm The image is detected from the endoscope image, and then the device information and personal information are cut and deleted, so that the tissue image only contains the in-vivo image corresponding to the endoscope, which protects privacy information and facilitates subsequent image processing. In addition, during the endoscopic inspection process, many invalid images may be collected during the moving process of the endoscope due to instability in the process of entering the mirror or improper position of the endoscope. Low quality image. These invalid images can interfere with the endoscopic inspection results. Therefore, after the endoscopic image is obtained, it can be judged first whether the endoscopic image is valid, and if the endoscopic image is an invalid image, the endoscopic image can be directly discarded. If the endoscopic image is a valid image, then the corresponding tissue image is determined based on the endoscopic image, so as to reduce unnecessary data processing and improve processing speed. For example, a pre-trained recognition model can be used to recognize the tissue image to determine whether the tissue image is valid. The recognition model can be obtained by training based on a convolutional neural network, which is not specifically limited in the present disclosure.

The cavity positioning module 200 is configured to determine the target point of the tissue cavity corresponding to the tissue image, and the target point is used to indicate the next target movement point of the endoscope at its current position; wherein , when there is a tissue cavity in the tissue image, the target point is the center point of the tissue cavity corresponding to the endoscopic image, and when there is no tissue cavity in the tissue image, the target point is the direction point of the tissue cavity corresponding to the endoscopic image.

As an example, if there is a tissue cavity in the tissue image, the target point is the center point of the tissue cavity corresponding to the endoscopic image, such as point A in Figure 2A; if there is no tissue cavity in the tissue image, the target point The point is the direction point of the tissue cavity corresponding to the endoscopic image, as shown in Figure 2B, if there is no tissue cavity in the tissue image, then the direction point of the navigation can be determined, that is, point B in Figure 2B Show.

For example, the physician user can select the current mode during the use of the endoscope, that is, the mirror-in mode or the mirror-out mode. In the mirror-in mode, it is used to control the endoscope to reach the ileocele. Human internal tissue examination. Among them, the endoscope needs to move along the center of the tissue cavity as far as possible during the endoscope entry process, so that it can effectively avoid touching the tissue surface mucosa of the examinee and avoid causing damage to the examinee. Therefore, in this embodiment, when it is determined that the endoscope is in the mirror-in mode, it may be further determined whether there is a tissue cavity in the tissue image.

For example, a detection model may be pre-trained to identify and detect whether there is a tissue cavity in the tissue image. For example, the detection model can be trained according to pre-collected training samples, which can include training images and labels corresponding to the training images, and the labels are used to indicate whether there are tissue cavities in the training images, so that The training image is used as the input of the model, and the label corresponding to the training image is used as the target output of the model to update and train the model to obtain the detection model. For example, the detection model may be CNN (Convolutional Neural Networks, convolutional neural network) or LSTM (Long Short-Term Memory, long-term short-term memory network), Encoder in Transformer, etc., which is not specifically limited in this disclosure.

As an example, the tissue cavity can be intestinal cavity, gastric cavity, etc. Taking the intestinal cavity as an example, if there is a tissue cavity in the collected tissue image of the intestinal cavity after the endoscope enters the intestinal cavity, then further The central point of the intestinal lumen is determined, that is, the center of the space section surrounded by the intestinal lumen wall. When the endoscope moves forward along the central point of the intestinal lumen, the automatic navigation of the endoscope is realized.

Exemplarily, the central point of the tissue cavity may be identified through a key point identification model. In this embodiment, for example, the tissue image may be marked by a professional physician based on experience. For the convenience of labeling, the position of the center point of the tissue cavity in the training image can be circled, and the center of the marked circle is the position of the center point, that is, the label corresponding to the training image, and a training sample including the training image and label can be obtained . Moreover, in order to improve the generalization of the model, unlabeled training images can also be included in the training samples. Wherein, the key point identification model may include a student sub-network, a teacher sub-network and a discriminant sub-network, the student sub-network and the teacher sub-network have the same network structure, and the teacher sub-network is used to determine the student sub-network The predicted labeling features corresponding to the training images in , during the training process of the key point recognition model, the weight of the prediction loss corresponding to the student sub-network is based on the predicted labeling features of the teacher sub-network and the discriminant sub-network OK out.

In order to improve the accuracy of model recognition, different preprocessing methods can be used to preprocess the training images based on the training samples to obtain different processed images corresponding to the training images. For example, the preprocessing method can be data enhancement, for example, it can be Color, brightness, chroma, saturation transformation and other non-affine transformation methods to ensure that the position is not deformed. Thus, different processed images can be used as the input images of the teacher sub-network and the student sub-network respectively to train the key point recognition model.

If there is no tissue cavity in the tissue image of the intestinal cavity collected after the endoscope enters the intestinal cavity, the direction point of the intestinal cavity can be further determined, and the direction point is the center point of the predicted tissue cavity relative to The point relative to the tissue image indicates that the endoscope should be deflected in the direction of the direction point, so as to provide direction guidance for the advancement of the endoscope.

In this embodiment, when it is determined that there is no tissue cavity in the tissue image, the latest N tissue images including the tissue image can be formed into an image sequence, based on the image sequence and the direction point recognition model Make direction point predictions. Wherein, N may be a positive integer, representing the number of tissue images included in the image sequence, which may be set according to actual application scenarios. Exemplarily, the direction point recognition model includes a convolutional subnetwork, a time cyclic subnetwork and a decoding subnetwork, the convolutional subnetwork can be used to obtain the spatial features of the image sequence, and the time cyclic subnetwork can be used to obtain the image sequence time features, the decoding subnetwork can be used to decode based on the spatial features and the time features to obtain the direction points and ensure the accuracy of direction point recognition.

For example, the number of training images contained in the training image sequence of the direction point recognition model can be limited according to the actual use scenario, for example, N can be set to 5, that is, each training image sequence can contain 5 training images, that is, Predict the direction point of the tissue cavity in the current state based on the last 5 training images. Wherein, the label image corresponding to the training image sequence is used to indicate the position of the direction point of the tissue cavity in the last image predicted based on the multiple images, so that the direction point recognition model can be performed based on the above training image sequence train.

The polyp identification module 300 is configured to perform real-time polyp identification on the tissue image obtained by the endoscope in the withdrawal mode, and obtain a polyp identification result.

The display module 400 is configured to display the target point and the polyp recognition result.

Physicians can detect the human body during the withdrawal process of the endoscope, for example, polyps can be detected. In this process, in order to facilitate the physician's observation, real-time polyp identification can be performed on the tissue image obtained during the retraction process, so as to provide data reference for the physician during the real-time detection process.

In a possible embodiment, the polyp identification module includes:

The polyp detection sub-module is configured to detect the tissue image obtained by the endoscope in the retracting mirror mode based on the detection model, and determine the position information of the polyp when it is detected that there is a polyp in the tissue image.

As an example, the detection model implemented by GFLv2 (Generalized Focal Loss V2) can be used in the polyp detection sub-module for detection. The detection model can judge the prediction reliability of the predicted position according to the distribution while outputting the predicted position. The training method is There are technologies, so I won't repeat them here. For example, the predicted position whose predicted reliability is greater than a threshold can be displayed in the image display area in the display interface, and the predicted reliability can also be displayed at the same time, as shown in Figure 3, the predicted position can be detected by a solid line It is displayed in the form of a box to provide an auxiliary reminder to the doctor, so as to improve the accuracy of polyp detection and the detection level of polyps with high specificity.

In some embodiments, before the tissue image is input into the detection model in the polyp detection sub-module for detection, the tissue image can be preprocessed by the preprocessing module. For example, the tissue image can be uniformly standardized to a preset size, for example The preset size may be 512*512, so that the image processed by the preprocessing module is input into the detection model for detection, and the position information of the polyp detected in the tissue image is obtained.

A polyp identification submodule, configured to extract a detected image corresponding to the position information from the tissue image; determine the classification of the polyp according to the detected image and the identification model;

The display module is further configured to display the tissue image, and display the identification corresponding to the position information of the polyp and the classification in the tissue image.

As an example, after the polyp detection submodule determines the position information of the polyp in the tissue image, the detection image corresponding to the position information may be extracted from the tissue image based on the position information. For example, when extracting a detection image, the region corresponding to the location information may be enlarged and then extracted, so as to ensure the integrity of the extracted detection image. For example, as shown in Figure 3, it is the detection frame corresponding to the position information of the detected polyp, then when the detection image is extracted based on this position, the detection frame can be enlarged by 1.5 times, that is, the corresponding frame in the dotted line frame in Figure 3 can be extracted The image of is used as the detection image. Afterwards, the detection image is input into the polyp recognition sub-module for polyp classification and recognition.

Among them, the polyp identification sub-module can include a classification model based on resnet18. This classification model can obtain corresponding training samples by pre-labeling and classifying a large number of polyp images, so that the polyp image is used as the input of the model, and the labeled classification is used as the target. The output is trained to obtain the classification model. Exemplarily, the classification category may cover various classifications such as adenoma, hyperplastic polyp, carcinoma, inflammatory polyp, and submucosal tumor. Similarly, before the detection image is input into the classification model in the polyp recognition sub-module for classification, the detection image can be preprocessed by the preprocessing module. For example, the detection image can be uniformly standardized to a preset size, such as a preset size It can be 256*256, so that the image processed by the preprocessing module is input into the classification model for classification, and the polyp recognition result is obtained, so that the polyp can be displayed in the image display area of the display interface and in the tissue image The identification corresponding to the location information and the classification, the identification corresponding to the location information may be represented by detecting a square box or a circular box.

Therefore, through the above technical solution, the polyp position in the tissue image can be detected first, the accuracy of polyp detection in the tissue image can be improved, and the missed detection rate of polyps can be reduced. Furthermore, the detection image for polyp identification can be extracted based on the detected position information, thereby reducing the amount of data processing for polyp identification, while avoiding the influence of other parts of the tissue image on polyp identification, and further improving polyp identification. accuracy.

As shown above, a mode selection control can be set in the control area of the display interface, and then the current use mode of the endoscope can be determined in response to the user's operation on the mode selection control in the control area. In another possible embodiment, the system further includes:

a pattern recognition module, configured to recognize the tissue image of the endoscope according to the image recognition model, and determine that the endoscope The image mode of the endoscope is the mirror-in mode, and when the parameter corresponding to the ileocecal valve image output by the image recognition model is greater than the second parameter threshold, it is determined that the image mode of the endoscope is the mirror-out mode.

In this embodiment, the image recognition model can be a network model based on ViT (Vision Transformer), and the training images can be pre-labeled in three categories: in vitro images, in vivo images and ileocecal valve images, so that the training images can be trained The image is used as input, and the labeled category is used as the target output for training to obtain the image recognition model. Correspondingly, in the detection process of the endoscope, the tissue image can be classified and recognized based on the image recognition model, wherein the parameters output by the image recognition model correspond to the in-vivo image, that is, the output category of the image recognition model is the in-vivo image Similarly, the parameter corresponding to the ileocecal valve image output by the image recognition model is the corresponding probability value when the output category of the image recognition model is the ileocecal valve image.

Wherein, the first parameter threshold and the second parameter threshold may be set according to actual application scenarios, and may be the same or different, and are not specifically limited in the present disclosure. For example, the threshold value of the first parameter is 0.85, and the threshold value of the second parameter is 0.9. After the tissue image is input into the image recognition model, the output result is an in-vivo image, and the corresponding probability value is 0.9. At this time, the image mode of the endoscope is determined Should be in mirror mode. In the subsequent detection process, for another tissue image, after inputting the tissue image into the image recognition model, the output result is an ileocecal valve image, and the corresponding probability value is 0.92. At this time, it is determined that the image mode of the endoscope should be Mirror mode.

A mode switching module, used to switch the image mode of the endoscope when the image mode of the endoscope is an in vitro mode and the mode recognition module determines that the image mode of the endoscope is an in-scope mode To the mirror-in mode, when the image mode of the endoscope is the mirror-in mode and the pattern recognition module determines that the image mode of the endoscope is the mirror-back mode, the image of the endoscope The mode is switched to the mirror-back mode, and second prompt information is output, and the second prompt information is used to remind to enter the mirror-back mode.

Among them, during the use of the endoscope, the corresponding mode sequence should be in vitro mode, mirror-in mode, and mirror-out mode. Therefore, after the pattern recognition module determines the image mode of the endoscope, it can be combined with the current endoscope's image mode. The image mode and the recognized image mode determine whether switching is required. For example, the current image mode of the endoscope is the in vitro mode, and the image mode determined based on the tissue images obtained in real time is the endoscope mode, which means that the endoscope enters the body at this time, and then it can automatically switch to the endoscope mode, If the current image mode of the endoscope is the same as the image mode determined by the pattern recognition module, there is no need to switch. Afterwards, when the current image mode of the endoscope is the mirror-in mode, and the image mode determined based on the tissue images obtained in real time is the mirror-out mode, it means that the endoscope has reached the ileocele at this time, and the next step is to withdraw the mirror During the inspection process, it can automatically switch to the mirror withdrawal mode, and prompt the user to prompt the current endoscope to enter the mirror withdrawal mode, that is, to enter the inspection stage next.

Thus, through the above technical solution, automatic recognition and switching of the image mode of the endoscope can be realized based on the images collected by the endoscope during the use of the endoscope, without manual operation by the user, and the use mode of the endoscope can Matching with its actual use status, it provides auxiliary reference and convenience for users to use the endoscope, and can identify the use process of the endoscope, providing reliable data support for subsequent processing of different modes.

The internal tissues of the human body for endoscopic examination are usually soft tissues. During the operation of the doctor's mirror, for example, the intestinal tract will peristalsis, and the doctor will perform operations such as flushing and undoing loops during the endoscopic examination, which will cause the doctor to It is difficult to clearly understand the extent of its inspection during endoscopy. Based on this, the present disclosure also provides the following embodiments.

In a possible embodiment, the system also includes:

The blind area ratio detection module is used to determine the display ratio and the blind area ratio according to the tissue image obtained by the endoscope in the mirror withdrawal mode, wherein the sum of the display ratio and the blind area ratio is 1, and the blind area can be Due to factors such as mucous membranes, missed inspection areas or inspection blind areas, etc., the blind area ratio can be understood as the blind area (that is, the part that cannot be observed in the field of view of the endoscope) in the process of endoscopic examination to the total internal surface area of the tissue (that is, the endoscopic The proportion of all) that should be observed in the inspection.

For example, when it is determined that the endoscope is in the mirror retracting mode, the blind spot ratio detection can be automatically turned on, or the blind spot ratio detection can be turned on in response to the user's selection operation of the blind spot ratio detection control in the control area of the display interface, To start performing the above functions, determine the blind area ratio and display ratio.

As an example, according to the tissue image obtained by the endoscope in the retracting mirror mode, the determination of the display ratio and the blind area ratio may be determined according to the clarity of the tissue image. In an actual application scenario, when the definition of the tissue image is low, there will be relatively more blind spots in the tissue image, and the proportion of blind spots in the field of view can be predicted based on the clarity of the tissue image. Therefore, experienced physicians can annotate the proportion of the blind area on the historically collected tissue images, then use the historically collected tissue images as the input of the model, and use the marked blind area ratio as the target output of the model to train the neural network model to obtain the blind area. A scale prediction model for predicting the corresponding blind zone scale based on tissue images. Wherein, the training method of the blind area proportion prediction model can adopt the training method commonly used in this field to carry out training, will not go into details here. Thus, the tissue image obtained by the endoscope in the retracting mirror mode can be input into the blind area ratio prediction model, so as to obtain the corresponding blind area ratio, and further determine the display ratio based on the blind area ratio.

As another example, the blind area ratio detection module is further configured to: perform three-dimensional reconstruction according to the tissue image obtained by the endoscope in the mirror-back mode to obtain a three-dimensional tissue image; and obtain a three-dimensional tissue image according to the three-dimensional tissue image and the three-dimensional The target corresponding to the tissue image is fitted to the tissue image, and the display ratio and blind area ratio are determined.

As shown in FIG. 4 , taking the intestinal tract as an example, it is a schematic diagram of a three-dimensional tissue image obtained through three-dimensional reconstruction based on an endoscopic image, that is, the intestinal mucosa. Exemplarily, the three-dimensional reconstruction technology in the field can be used for reconstruction, such as the SurfelMeshing fusion algorithm. For example, two adjacent frames of tissue images can be input into the depth network Depth Network and the pose network Pose Network respectively, then the corresponding depth map and pose information can be obtained, and the pose information can represent the movement of the endoscope in the tissue Processes, for example, can include rotation matrices and translation vectors. Afterwards, the tissue image, depth map and posture information are reconstructed based on the 3D reconstruction model to obtain a 3D tissue image. Among them, both the depth network and the attitude network can be trained based on the ResNet50 network, which will not be repeated here.

Among them, the intestinal tract can be similar to a tubular structure. Due to the limitation of the field of view of the endoscope, when the intestinal tract is reconstructed based on the endoscopic image, it may appear as shown at W1, W2, W3, and W4 in Figure 4. The cavity position, that is, the cavity position does not appear in the tissue image, that is, the invisible part in endoscopic examination, that is, the blind area described in the present disclosure. Physicians cannot observe this part of the area during endoscopic examination, and if there are too many unseen areas, it is easy to miss detection. In this embodiment, the ratio of the invisible part of the mucosa appearing in the inspection image to the total mucosal area of the tissue can be represented by the proportion of the blind area, and the proportion of the blind area can be used to indicate the invisible part of the current tissue, so as to facilitate the endoscope The comprehensiveness of the inspection is characterized.

Exemplarily, the blind spot ratio detection module is also used for:

The ratio of the number of point clouds in the three-dimensional tissue image to the number of point clouds of the target fitting tissue image is determined as the display ratio, and the blind area ratio is determined according to the display ratio.

Wherein, the three-dimensional tissue image is obtained after reconstruction based on the two-dimensional tissue image, and then the three-dimensional tissue image can be projected to its corresponding target fitting tissue image, and according to the projection of the three-dimensional tissue image and the target fitting tissue image The overlapping area determines the aspect ratio. The target fitting tissue image corresponding to the three-dimensional tissue image is the complete cavity structure predicted based on the structure corresponding to the tissue in the tissue image. Taking the intestinal tract in Figure 4 as an example, the corresponding target fitting tissue image is corresponding to The image of the tubular structure can be fitted into a cylindrical structure. Wherein, corresponding standard structural features can be set for different tissues, and after the three-dimensional tissue image is determined, fitting can be performed based on the standard structural features to obtain the target fitted tissue image. For example, the Monte Carlo method can be used to evenly distribute K test points (K≥100) on the target fitting tissue image, and then count the number Λ of test points in the visible area and the number Ω of test points in the blind area , then the display ratio φ=Λ/(Λ+Ω), then the blind area ratio is 1-φ.

The display module is also used to display the display ratio and/or the blind area ratio, and when the blind area ratio is greater than the blind area threshold, display first prompt information, the first prompt information is used to indicate that there is a leak check risk.

Wherein, the display ratio is used to represent the proportion of the area viewed by the physician in the process of endoscopic image detection to the overall tissue, and the blind area ratio is used to represent the proportion of the area not viewed by the physician in the process of endoscopic image detection. In the embodiment, the display ratio can be displayed, the blind area ratio can also be displayed, or the display ratio and the blind area ratio can be displayed at the same time, so that it is convenient for the doctor to know the accuracy of the detection process in time. At the same time, when the proportion of the blind area is large, it will be prompted. For example, the prompt message is displayed on the display interface. Mirror", you can directly display the prompt message, you can also prompt by voice, or you can prompt through a pop-up window, so as to remind the doctor, so that the doctor can know in time that the mucosa coverage of the inspection area is insufficient during the process of withdrawing the mirror, and it is prone to missed inspections phenomenon, so that the doctor can adjust the direction of the endoscope according to the prompt information, or execute the endoscope retraction process, or re-execute the mirror retraction process, so as to reduce the risk of missed detection of endoscopy to a certain extent, and lay a solid foundation for subsequent polyp detection. Identification and inspection provides reliable and comprehensive data support while being user-friendly.

In some embodiments, the system also includes:

The three-dimensional positioning module is used to perform three-dimensional reconstruction according to the tissue image obtained by the endoscope in the endoscopic mode to obtain a three-dimensional tissue image; A point is determined as a three-dimensional target point of the target point in the three-dimensional tissue image.

Wherein, the manner of performing 3D reconstruction has been described in detail above, and will not be repeated here. As an example, the tissue cavity is usually fitted to a tubular structure, therefore, the centerline of the three-dimensional tissue image can be the centerline of the fitted tubular structure, to ensure the distance from the surrounding tissue mucosa during the movement of the endoscope, Avoid damage to the mucous membrane of the tissue. Therefore, after determining the target point in the tissue image, in order to ensure the accuracy of the endoscope, the target point can be mapped to the corresponding position on the center line, that is, the point closest to the target point , to ensure the accuracy and rationality of the three-dimensional target point.

The attitude determination module is configured to determine the attitude information of the endoscope according to the tissue image corresponding to the current location and the tissue image in the historical period.

Exemplarily, the images during the process of endoscope mirroring can be monitored and displayed. Illustratively, a dataset can be formed in advance based on manual annotations of real endoscopic images and their bounding boxes and adding pose information to real images. Then the pose estimation model can be implemented based on the ResNet50 network. For example, endoscopic image sequences can be used as input, and the labeled pose information can be used as the target output of the model to adjust the network and perform regression to achieve model training. Thus, the tissue image corresponding to the current location and the tissue images in the historical period can be formed into an image sequence, so as to determine the posture information of the endoscope based on the above-mentioned posture estimation model.

A trajectory determination module, configured to generate a navigation trajectory according to the current location, the attitude information and the three-dimensional target point. Wherein, the attitude information is used to characterize the attitude of the current endoscope, and the three-dimensional target point is used to represent the end point of the movement of the endoscope, so that the movement to the three-dimensional target point can be determined based on the current position and attitude information. The trajectory information of , that is, the navigation trajectory. Wherein, the manner of trajectory prediction may adopt a prediction manner commonly used in the art, which is not limited in the present disclosure.

The display module is also used to display the posture information of the endoscope and the three-dimensional tissue image, and display the three-dimensional target point and the navigation track in the three-dimensional tissue image.

As an example, when the endoscope is controlled to enter the endoscope, the three-dimensional positioning and the endoscope navigation can be automatically started. As another example, a 3D camera navigation control can be displayed in the control area of the display interface, then when the user needs to navigate, the control can be clicked, and then in response to the user performing the 3D navigation control in the control area of the display interface The selection operation of the mirror navigation control can be positioned through the three-dimensional positioning module. Afterwards, the attitude information of the endoscope can be further determined to determine the navigation trajectory. As an example, the three-dimensional tissue image can be displayed on the display interface, and the three-dimensional target point and the navigation track can be displayed in the three-dimensional tissue image, that is, the position point of the next movement is displayed, and the position point of moving to the position point is displayed. The proposed path realizes the automatic navigation of the endoscope, and at the same time, it is also convenient for the doctor to know the current movement path and status of the endoscope in the body, and it is convenient to perform manual intervention to ensure the accuracy of the endoscope process. , to improve user experience.

In the actual application scenario, the colonoscope and the gastroscope share a set of endoscope hosts. If the examinee is inspected by both the gastroscope and the colonoscope, the video received by the system often alternates between the gastroscope and the colonoscope. Most of the detections are different, and the detection algorithm for colonoscopy is not suitable for gastroscopy. Based on this, in a possible embodiment, the control of the corresponding detection mode can be displayed in the control area of the display interface, and in response to the selection operation of the control, the selected mode can be used as the detection mode of the endoscope. detection mode.

Therefore, the present disclosure also provides the following embodiments to be applicable to the endoscopic image detection process in multiple scenarios. In a possible embodiment, the system also includes:

The image classification module is configured to classify the tissue images of the endoscope in the endoscope mode, and obtain target classifications corresponding to the tissue images, wherein the target classification includes abnormal classification and multiple endoscope classifications.

For example, in the present disclosure, an image classification model can be pre-trained to classify tissue images. The image classification model can be obtained by training based on PreResNet18. The categories of image classification can include abnormal classification and endoscope classification, wherein the abnormal classification includes none Signal images, in vitro images, etc. Endoscope classification can include colonoscopy images and gastroscopic images.

In this embodiment, endoscopic images can be pre-classified and marked by experienced physicians. No-signal images and in-vitro images can be marked as abnormal classification, colonoscopic images can be marked as colonoscopic classification, and gastroscopic images can be marked as Classification of gastroscopy to obtain training samples. After that, the endoscope image can be used as the input of the model, and the label corresponding to the endoscope image can be used as the target output of the model to train the model to obtain the image classification model, so as to classify the tissue images collected by the endoscope.

The detection pattern recognition module is used to update the counter of each endoscope classification according to the target classification corresponding to the tissue image, and when the value of the counter corresponding to any endoscope classification reaches a counting threshold, stop each of the The counting operation of the counter, and determine the detection mode of the endoscope according to the endoscope classification whose value of the counter reaches the counting threshold, wherein, each of the endoscope classifications has its corresponding counter, and each of the counters The value is initially zero.

In this embodiment, the detection mode currently adopted by the endoscope can be determined according to the classification corresponding to the tissue images collected during the movement of the endoscope. As an example, classification may be performed on images of tissues during movement of the endoscope. The detection pattern recognition module updates the counter of each endoscope classification according to the target classification corresponding to the tissue image in the following manner:

When the target classification is an endoscope classification, add one to the counter corresponding to the target classification, and subtract one from the counter corresponding to the endoscope classification whose counter value is not zero except for the target classification Operation: when the target category is an abnormal category, the counter corresponding to each endoscope category whose counter value is not zero is decremented by one, so as to update the counter of each endoscope category.

Exemplarily, the endoscope classification includes colonoscopy classification and gastroscope classification, the counter corresponding to the colonoscopy classification is count1, and the counter corresponding to the gastroscopy classification is count2, which are initialized to 0 respectively. If the determined classification corresponding to the tissue image is an abnormal classification, then the counters corresponding to the colonoscopy classification and the gastroscopy classification are both 0. The classification corresponding to the determined tissue image is colonoscopy classification, at this time, an operation may be added to the counter count1 corresponding to the colonoscopy classification, and count1 is 1 at this time. Then repeat the above process for other tissue images, if the determined count1 is 49, count2 is 3, and the count threshold is 50. When the classification corresponding to the next determined tissue image is colonoscopy classification, at this time, add one operation to count1, that is, count1 is 50, and perform a subtraction operation on count2, that is, count2 is 2. At this time, the corresponding colonoscopy classification When the value of the counter reaches the counting threshold, it can be determined that the detection mode is the colonoscopy mode, and the counting operation of each of the counters is stopped. In this way, the detection mode of the endoscope can be automatically determined by classifying and counting the part of the tissue images during the moving process of the endoscope, which saves user operations and assists the user in using the endoscope.

The polyp identification module is further configured to: perform real-time polyp identification on the tissue image obtained by the endoscope in the retracting mode according to the polyp identification model corresponding to the detection mode determined by the detection mode identification module.

Wherein, the classification and features of image recognition for colonoscopy classification and gastroscopy classification may be different. Therefore, in this embodiment, for each detection mode, the polyps used for polyp identification in this detection mode can be pre-determined. The identification model, wherein, the manner of determining the polyp identification model has been described in detail above, and will not be repeated here.

Thus, through the above example, the detection mode of the endoscope can be automatically determined by classifying the tissue images collected during the moving process of the endoscope, and the corresponding polyp recognition model under the detection model can be used for image recognition, thereby It can improve the automation level of endoscope use, improve the degree of applicability to actual application scenarios, and improve the accuracy of polyp identification to a certain extent, providing reliable data support for physicians to analyze endoscopic results.

In a possible embodiment, the system also includes:

A speed determination module, configured to acquire a plurality of historical tissue images corresponding to the current tissue image when the mode of the endoscopic image is the mirror-back mode, and calculate the similarity between each of the historical tissue images and the current tissue image degree, and determine the mirror moving speed according to the plurality of similarities.

The display module is also used to display the mirror withdrawal speed corresponding to the endoscope.

Wherein, during the moving process of the endoscope, if the moving speed of the endoscope is slow, the adjacent tissue images should be similar. Therefore, in this embodiment, the back-off can be performed based on the similarity between adjacent tissue images. Mirror speed is evaluated. The method for calculating the similarity between images may be calculated by using a common calculation method in the art, which is not limited in the present disclosure.

For example, a mapping relationship between the similarity interval and the mirror-off speed can be preset, wherein the higher the similarity, the slower the mirror-out speed. For example, the determined average value of the plurality of similarities can be used to determine the corresponding average similarity during the moving process of the plurality of tissue images, and based on the similarity interval to which the average similarity belongs, the speed corresponding to the similarity interval can be used as the The corresponding mirror retraction speed during the process. And further, the mirror retraction speed can be displayed in the image display area of the display interface during the mirror retraction process. Thus, the mirror retraction speed can be determined by detecting the similarity between adjacent tissue images in the mirror retraction process, and the mirror retraction speed is displayed, so that the user can be prompted, to a certain extent On the one hand, it can avoid the missed detection of polyps caused by the excessive speed of withdrawing the mirror, and improve the convenience of users.

During the use of the endoscope, the cleanliness of the tissue cavity is one of the important indicators to measure the quality of the inspection, and the Boston scoring method is usually used at present. However, because physicians mainly focus on the endoscopic examination process and lesion discovery process, the attention allocated to cleanliness evaluation is limited. And there is subjectivity among different physicians in assessing cleanliness.

Based on this, the present disclosure also provides the following embodiments. In some embodiments, the system also includes:

The cleanliness determination module is used to classify the tissue images of the endoscope in the mirror withdrawal mode, and determine the number of tissue images under each cleanliness category; according to each of the cleanliness categories The number of tissue images determines the cleanliness of the tissue corresponding to the tissue images.

As an example, the cleanliness detection can be automatically started after it is determined that the endoscope is in the mirror withdrawal mode, or it can be started in response to the user's selection operation of the cleanliness detection control in the control area of the display interface. In practical application scenarios, there is a cleaning operation in the endoscope's lens entry stage, and it is not suitable to include the cleanliness in the tissue cavity before the cleaning operation into the overall evaluation. Therefore, in this embodiment, by classifying the cleanliness of the tissue images in the mirror-back mode, the cleanliness classification of a single tissue image can be classified based on the cleanliness classification model implemented by the Vision Transformer network. The endoscopic images are marked in advance, and can be marked as 0, 1, 2, 3 according to the Boston scoring method. Afterwards, the endoscopic image can be used as the input of the model, and the label corresponding to the endoscopic image can be used as the target output of the model for training. In this way, it can be ensured that the determined cleanliness of the tissue fits the actual application scenario, and the accuracy of the determined cleanliness can be improved.

In a possible embodiment, the cleanliness determination module is also used for:

Acquire the number of tissue images under a cleanliness category according to the order of the scores corresponding to the cleanliness category from small to large;

Determine the relationship between the ratio of the number of tissue images under the current cleanliness classification to the target total number and the threshold corresponding to the current cleanliness classification, wherein the target total number is the sum of the numbers of tissue images under each cleanliness category;

If the ratio of the number of tissue images under the current cleanliness classification to the total target quantity is greater than or equal to the threshold corresponding to the cleanliness classification, then the score corresponding to the cleanliness classification is used as the cleanliness of the organization;

If the ratio of the number of tissue images under the current cleanliness classification to the total number of targets is less than the threshold corresponding to the cleanliness classification, then if the next cleanliness classification is not the cleanliness classification with the largest score, the next cleanliness classification will be used as new current cleanliness classification, and re-execute the step of determining the ratio of the number of tissue images under the current cleanliness classification to the target total quantity and the threshold value corresponding to the current cleanliness classification; In the case of the largest cleanliness category, the score of the next cleanliness category is determined as the cleanliness of the tissue.

As in the example above, the scores corresponding to the cleanliness categories are 0, 1, 2, and 3 in descending order, and the number of tissue images under each cleanliness category can be acquired in this order. For example, first obtain the number of tissue images whose cleanliness classification score is 0, for example, the number is S0, then further determine the ratio of the number of tissue images under the current cleanliness category to the total target number and the current cleanliness The size relationship of the threshold corresponding to the classification. Among them, the total number of targets is Sum, and the threshold corresponding to each cleanliness category can be set according to the actual application scenario. For example, if the score of cleanliness classification is 0, which corresponds to the threshold N0, then when S0/Sum is greater than or equal to N0, it is determined that the cleanliness of the tissue is 0. Otherwise, further obtain the number of tissue images whose cleanliness classification score is 1, for example, the number is S1, and the cleanliness classification score is 1 corresponding to the threshold N1, then when S1/Sum is greater than or equal to N1, determine the tissue cleanliness is 1. Otherwise, further obtain the number of tissue images whose cleanliness classification score is 2, for example, the number is S2, and the cleanliness classification score is 2 corresponding to the threshold N2, then when S2/Sum is greater than or equal to N2, determine The cleanliness of the tissue is 2. Otherwise, the next cleanliness classification is obtained, that is, the cleanliness classification with a score of 3, which is the cleanliness classification with the largest score, and the score of the next cleanliness classification can be directly determined as the cleanliness of the organization, namely Determine the cleanliness of the tissue as 3.

Therefore, through the above technical solution, only the tissue image in the process of retracting the mirror can be detected for the classification of cleanliness, thereby reducing the amount of data processing. At the same time, based on the data of tissue images under each category, the overall evaluation of the cleanliness in the mirror removal process can be performed, the accuracy of the determined cleanliness can be improved, and it is more in line with the Boston scoring standard, which is convenient for users to use.

In a possible embodiment, when the inspection is completed after the endoscope retraction process, the polyp identification result, cleanliness, and mirror retraction speed obtained in the above process can be formed as Reports are output for unified viewing and management.

Based on the same inventive concept, the present disclosure also provides an auxiliary method for endoscopic image detection, as shown in FIG. 5 , the method includes:

In step 11, the endoscope image collected by the endoscope is processed in real time to obtain a tissue image;

In step 12, in response to determining that the endoscope is in the endoscope mode, determine the target point of the tissue cavity corresponding to the tissue image, and the target point is used to indicate that the endoscope is at its current position The next target moving point of the endoscope; wherein, when there is a tissue cavity in the tissue image, the target point is the center point of the tissue cavity corresponding to the endoscopic image, and there is no tissue cavity in the tissue image When there is a tissue cavity, the target point is a direction point of the tissue cavity corresponding to the endoscopic image;

In step 13, in response to determining that the endoscope is in the retracting mirror mode, real-time polyp identification is performed on the tissue image obtained by the endoscope in the retracting mirror mode, and a polyp identification result is obtained;

In step 14, the target point and the polyp recognition result are displayed in the image display area of the display interface.

In some embodiments, the method also includes:

According to the tissue image obtained by the endoscope in the mirror withdrawal mode, a display ratio and a blind area ratio are determined, and the sum of the display ratio and the blind area ratio is 1;

Displaying the display ratio and/or the blind area ratio, and displaying first prompt information when the blind area ratio is greater than a blind area threshold, where the first prompt information is used to indicate that there is a risk of missed detection.

In some embodiments, the determination of the display ratio and the blind area ratio according to the tissue image obtained by the endoscope in the mirror withdrawal mode includes:

Performing three-dimensional reconstruction according to the tissue image obtained by the endoscope in the retracting mirror mode to obtain a three-dimensional tissue image;

A display ratio and a blind area ratio are determined according to the three-dimensional tissue image and the target fitting tissue image corresponding to the three-dimensional tissue image.

In some embodiments, the determining the display ratio and blind area ratio according to the three-dimensional tissue image and the target fitting tissue image corresponding to the three-dimensional tissue image includes:

In some embodiments, the method also includes:

Performing three-dimensional reconstruction according to the tissue image obtained by the endoscope in the endoscope mode to obtain a three-dimensional tissue image;

determining a point on the centerline of the three-dimensional tissue image that is closest to the target point as the three-dimensional target point of the target point on the three-dimensional tissue image;

Determine the posture information of the endoscope according to the tissue image corresponding to the current location and the tissue image in the historical period, and generate a navigation track according to the current location, the posture information and the three-dimensional target point;

The three-dimensional tissue image is displayed, and the three-dimensional target point and the navigation track are displayed in the three-dimensional tissue image.

In some embodiments, the method also includes:

In response to the user's selection operation of the detection mode in the control area of the display interface, the detection mode selected by the user is determined as the detection mode of the endoscope;

The real-time polyp identification of the tissue image obtained by the endoscope in the mirror withdrawal mode includes:

According to the polyp identification model corresponding to the detection mode, real-time polyp identification is performed on the tissue image obtained by the endoscope in the retracting mirror mode.

In some embodiments, the method also includes:

Classifying the tissue image of the endoscope in the endoscope mode to obtain a target classification corresponding to the tissue image, wherein the target classification includes an abnormal classification and a plurality of endoscope classifications;

According to the target classification corresponding to the tissue image, the counter of each endoscope classification is updated, and when the value of the counter corresponding to any endoscope classification reaches the counting threshold, the counting operation of each of the counters is stopped, and according to The endoscope classification whose value of the counter reaches the counting threshold determines the detection mode of the endoscope, wherein each of the endoscope classifications has its corresponding counter, and the value of each of the counters is initially zero;

In some embodiments, updating the counter of each endoscope classification according to the target classification corresponding to the tissue image includes:

In some embodiments, the method also includes:

Recognizing the tissue image of the endoscope according to the image recognition model;

If the parameter corresponding to the in-vivo image output by the image recognition model is greater than the first parameter threshold, it is determined that the image mode of the endoscope is the endoscope mode; if the parameter corresponding to the ileocecal valve image output by the image recognition model Greater than the second parameter threshold, it is determined that the image mode of the endoscope is the mirror-back mode;

When the image mode of the endoscope is an in vitro mode and the image mode of the endoscope determined based on the image recognition model is an endoscope mode, switch the image mode of the endoscope to an endoscope mode mode, when the image mode of the endoscope is the mirror-in mode, and the image mode of the endoscope determined based on the image recognition model is the mirror-back mode, the image of the endoscope The mode is switched to the mirror-back mode, and second prompt information is output, and the second prompt information is used to remind to enter the mirror-back mode.

In some embodiments, the method also includes:

When the mode of the endoscopic image is the mirror-back mode, obtain a plurality of historical tissue images corresponding to the current tissue image, and calculate the similarity between each of the historical tissue images and the current tissue image, according to the plurality of historical tissue images The above similarity determines the mirror withdrawal speed;

The mirror withdrawal speed corresponding to the endoscope is displayed in the image display area.

In some embodiments, the method also includes:

Carrying out cleanliness classification on the tissue images of the endoscope in the withdrawal mode, and determining the number of tissue images under each cleanliness category;

According to the quantity of tissue images under each cleanliness category, the cleanliness of the tissue corresponding to the tissue image is determined.

In some embodiments, the determining the cleanliness of the tissue corresponding to the tissue image according to the number of tissue images under each cleanliness category includes:

In some embodiments, the real-time identification of polyps on the tissue images obtained by the endoscope in the retracting mirror mode includes:

Based on the detection model, the tissue image obtained by the endoscope in the withdrawal mode is detected, and when polyps are detected in the tissue image, the position information of the polyps is determined;

extracting a detected image corresponding to the location information from the tissue image, and determining the classification of the polyp according to the detected image and the recognition model;

The tissue image is displayed in the image display area, and the identification corresponding to the position information of the polyp and the classification are displayed in the tissue image.

Wherein, the specific implementation manners of the above steps have been described in detail above, and will not be repeated here.

Referring now to FIG. 6 , it shows a schematic structural diagram of an electronic device 600 suitable for implementing an embodiment of the present disclosure. The terminal equipment in the embodiment of the present disclosure may include but not limited to such as mobile phone, notebook computer, digital broadcast receiver, PDA (personal digital assistant), PAD (tablet computer), PMP (portable multimedia player), vehicle terminal (such as mobile terminals such as car navigation terminals) and fixed terminals such as digital TVs, desktop computers and the like. The electronic device shown in FIG. 6 is only an example, and should not limit the functions and application scope of the embodiments of the present disclosure.

As shown in FIG. 6, an electronic device 600 may include a processing device (such as a central processing unit, a graphics processing unit, etc.) 601, which may be randomly accessed according to a program stored in a read-only memory (ROM) 602 or loaded from a storage device 608. Various appropriate actions and processes are executed by programs in the memory (RAM) 603 . In the RAM 603, various programs and data necessary for the operation of the electronic device 600 are also stored. The processing device 601, ROM 602, and RAM 603 are connected to each other through a bus 604. An input/output (I/O) interface 605 is also connected to the bus 604 .

Typically, the following devices can be connected to the I/O interface 605: input devices 606 including, for example, a touch screen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; including, for example, a liquid crystal display (LCD), speaker, vibration an output device 607 such as a computer; a storage device 608 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 609. The communication means 609 may allow the electronic device 600 to communicate with other devices wirelessly or by wire to exchange data. While FIG. 6 shows electronic device 600 having various means, it should be understood that implementing or having all of the means shown is not a requirement. More or fewer means may alternatively be implemented or provided.

In particular, according to an embodiment of the present disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product, which includes a computer program carried on a non-transitory computer readable medium, where the computer program includes program code for executing the method shown in the flowchart. In such an embodiment, the computer program may be downloaded and installed from a network via communication means 609, or from storage means 608, or from ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the methods of the embodiments of the present disclosure are performed.

It should be noted that the computer-readable medium mentioned above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, electrical connections with one or more wires, portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable Programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In the present disclosure, however, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave carrying computer-readable program code therein. Such propagated data signals may take many forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which can transmit, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device . Program code embodied on a computer readable medium may be transmitted by any appropriate medium, including but not limited to wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

In some embodiments, the client and the server can communicate using any currently known or future network protocols such as HTTP (HyperText Transfer Protocol, Hypertext Transfer Protocol), and can communicate with digital data in any form or medium The communication (eg, communication network) interconnections. Examples of communication networks include local area networks ("LANs"), wide area networks ("WANs"), internetworks (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future developed network of.

The above-mentioned computer-readable medium may be included in the above-mentioned electronic device, or may exist independently without being incorporated into the electronic device.

The computer-readable medium carries one or more programs, and when the one or more programs are executed by the electronic device, the electronic device: performs real-time processing on endoscopic images collected by the endoscope to obtain tissue images; In response to determining that the endoscope is in the mirror-in mode, determine a target point of the tissue cavity corresponding to the tissue image, and the target point is used to indicate the next time the endoscope is in the mirror-in mode at its current position. Target moving point; wherein, when there is a tissue cavity in the tissue image, the target point is the center point of the tissue cavity corresponding to the endoscopic image, and when there is no tissue cavity in the tissue image , the target point is the direction point of the tissue cavity corresponding to the endoscope image; in response to determining that the endoscope is in the retracting mirror mode, the tissue image obtained by the endoscope in the retracting mirror mode is performed Real-time polyp recognition, obtaining polyp recognition results; displaying the target points and the polyp recognition results in the image display area in the display interface.

Computer program code for carrying out operations of the present disclosure may be written in one or more programming languages, or combinations thereof, including but not limited to object-oriented programming languages—such as Java, Smalltalk, C++, and Includes conventional procedural programming languages - such as "C" or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (for example, using an Internet service provider to connected via the Internet).

The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, program segment, or portion of code that contains one or more logical functions for implementing specified executable instructions. It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or they may sometimes be executed in the reverse order, depending upon the functionality involved. It should also be noted that each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, can be implemented by a dedicated hardware-based system that performs the specified functions or operations , or may be implemented by a combination of dedicated hardware and computer instructions.

The modules involved in the embodiments described in the present disclosure may be implemented by software or by hardware. Among them, the name of the module does not constitute a limitation of the module itself under certain circumstances. For example, the image processing module can also be described as "a module for real-time processing of endoscopic images collected by the endoscope to obtain tissue images" .

The functions described herein above may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: Field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), Application Specific Standard Products (ASSPs), System on Chips (SOCs), Complex Programmable Logical device (CPLD) and so on.

In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media would include one or more wire-based electrical connections, portable computer discs, hard drives, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), optical fiber, compact disk read only memory (CD-ROM), optical storage, magnetic storage, or any suitable combination of the foregoing.

According to one or more embodiments of the present disclosure, Example 1 provides an endoscopic image detection assistance system, the system comprising:

According to one or more embodiments of the present disclosure, Example 2 provides the system of Example 1, wherein the system further includes:

A blind area ratio detection module, configured to determine a display ratio and a blind area ratio based on the tissue image obtained by the endoscope in the mirror withdrawal mode, and the sum of the display ratio and the blind area ratio is 1;

According to one or more embodiments of the present disclosure, Example 3 provides the system of Example 2, wherein the blind spot ratio detection module is further used for: performing three-dimensional reconstruction according to the tissue image obtained by the endoscope in the retracting mode , obtaining a three-dimensional tissue image; and determining a display ratio and a blind area ratio according to the three-dimensional tissue image and the target fitting tissue image corresponding to the three-dimensional tissue image.

According to one or more embodiments of the present disclosure, Example 4 provides the system of Example 3, wherein the dead zone ratio detection module is further used for:

According to one or more embodiments of the present disclosure, Example 5 provides the system of Example 1, wherein the system further includes:

The three-dimensional positioning module is used to perform three-dimensional reconstruction according to the tissue image obtained by the endoscope in the endoscopic mode to obtain a three-dimensional tissue image; A point is determined as a three-dimensional target point of the target point in the three-dimensional tissue image;

A posture determination module, configured to determine the posture information of the endoscope according to the tissue image corresponding to the current location and the tissue image in the historical period;

A trajectory determination module, configured to generate a navigation trajectory according to the current location, the attitude information and the three-dimensional target point;

According to one or more embodiments of the present disclosure, Example 6 provides the system of Example 1, wherein the system further includes:

An image classification module, configured to classify the tissue image of the endoscope in the endoscope mode, and obtain the target classification corresponding to the tissue image, wherein the target classification includes abnormal classification and multiple endoscope classifications;

The detection pattern recognition module is used to update the counter of each endoscope classification according to the target classification corresponding to the tissue image, and when the value of the counter corresponding to any endoscope classification reaches a counting threshold, stop each of the The counting operation of the counter, and determine the detection mode of the endoscope according to the endoscope classification whose value of the counter reaches the counting threshold, wherein, each of the endoscope classifications has its corresponding counter, and each of the counters The value is initially zero;

According to one or more embodiments of the present disclosure, Example 7 provides the system of Example 6, wherein the detection pattern recognition module classifies the counter of each endoscope according to the target classification corresponding to the tissue image in the following manner Make an update:

According to one or more embodiments of the present disclosure, Example 8 provides the system of Example 1, wherein the system further includes:

a pattern recognition module, configured to recognize the tissue image of the endoscope according to the image recognition model, and determine that the endoscope The image mode of the mirror is the mirror-in mode, and when the parameter corresponding to the ileocecal valve image output by the image recognition model is greater than the second parameter threshold, it is determined that the image mode of the endoscope is the mirror-out mode;

According to one or more embodiments of the present disclosure, Example 9 provides the system of Example 1, wherein the system further includes:

A speed determination module, configured to acquire a plurality of historical tissue images corresponding to the current tissue image when the mode of the endoscopic image is the mirror-back mode, and calculate the similarity between each of the historical tissue images and the current tissue image Degree, according to a plurality of described similarities, determine the mirror-removing speed;

According to one or more embodiments of the present disclosure, Example 10 provides the system of Example 9, wherein the system further includes:

According to one or more embodiments of the present disclosure, Example 11 provides the system of Example 1, wherein the polyp identification module includes:

The polyp detection sub-module is used to detect the tissue image obtained by the endoscope in the withdrawal mode based on the detection model, and determine the position information of the polyp when it is detected that there is a polyp in the tissue image;

According to one or more embodiments of the present disclosure, Example 12 provides an endoscopic image detection assistance method, the method comprising:

According to one or more embodiments of the present disclosure, Example 13 provides the method of Example 12, wherein the method further includes:

According to one or more embodiments of the present disclosure, Example 14 provides the method of Example 13, wherein the determining the display ratio and the blind area ratio according to the tissue image obtained by the endoscope in the retracting mirror mode includes:

According to one or more embodiments of the present disclosure, Example 15 provides the method of Example 14, wherein, according to the three-dimensional tissue image and the target fitting tissue image corresponding to the three-dimensional tissue image, the display ratio and the blind area ratio are determined ,include:

According to one or more embodiments of the present disclosure, Example 16 provides the method of Example 12, wherein the method further includes:

According to one or more embodiments of the present disclosure, Example 17 provides the method of Example 12, wherein the method further includes:

According to one or more embodiments of the present disclosure, Example 18 provides the method of Example 12, wherein the method further includes:

According to the target classification corresponding to the tissue image, the counter of each endoscope classification is updated, and when the value of the counter corresponding to any endoscope classification reaches the counting threshold, the counting operation of each of the counters is stopped, and according to The endoscope category whose value of the counter reaches the counting threshold determines the detection mode of the endoscope, wherein each of the endoscope categories has its corresponding counter, and the value of each of the counters is initially zero;

According to one or more embodiments of the present disclosure, Example 19 provides the method of Example 18, wherein the updating the counter of each endoscope classification according to the object classification corresponding to the tissue image includes:

According to one or more embodiments of the present disclosure, Example 20 provides the method of Example 12, wherein the method further includes:

According to one or more embodiments of the present disclosure, Example 21 provides the method of Example 12, wherein the method further includes:

According to one or more embodiments of the present disclosure, Example 22 provides the method of Example 21, wherein the method further includes:

According to one or more embodiments of the present disclosure, Example 23 provides the method of Example 12, wherein the real-time identification of polyps on the tissue image obtained by the endoscope in the retracting mirror mode includes:

According to one or more embodiments of the present disclosure, Example 24 provides a computer-readable medium on which a computer program is stored, and when the program is executed by a processing device, the steps of the method described in any one of Examples 12-23 are implemented .

According to one or more embodiments of the present disclosure, Example 25 provides an electronic device, comprising:

a storage device on which a computer program is stored;

A processing device configured to execute the computer program in the storage device, so as to implement the steps of the method in any one of Examples 12-23.

The above description is only a preferred embodiment of the present disclosure and an illustration of the applied technical principles. Those skilled in the art should understand that the disclosure scope involved in this disclosure is not limited to the technical solution formed by the specific combination of the above-mentioned technical features, but also covers the technical solutions formed by the above-mentioned technical features or Other technical solutions formed by any combination of equivalent features. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in this disclosure.

In addition, while operations are depicted in a particular order, this should not be understood as requiring that the operations be performed in the particular order shown or performed in sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Likewise, while the above discussion contains several specific implementation details, these should not be construed as limitations on the scope of the disclosure. Certain features that are described in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination.

Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are merely example forms of implementing the claims. Regarding the apparatus in the foregoing embodiments, the specific manner in which each module executes operations has been described in detail in the embodiments related to the method, and will not be described in detail here.

Claims

An endoscopic image detection auxiliary system, wherein the system includes:

The image processing module is used to process the endoscope image collected by the endoscope in real time to obtain tissue images;

The cavity positioning module is used to determine the target point of the tissue cavity corresponding to the tissue image, and the target point is used to indicate the next target movement point of the endoscope at its current position; wherein, When there is a tissue cavity in the tissue image, the target point is the center point of the tissue cavity corresponding to the endoscopic image, and when there is no tissue cavity in the tissue image, the target point is The direction point of the tissue cavity corresponding to the endoscopic image;

The polyp identification module is used to perform real-time polyp identification on the tissue image obtained by the endoscope in the retracting mirror mode, and obtain a polyp identification result;

A display module, configured to display the target point and the polyp recognition result.
The system according to claim 1, wherein said system further comprises:

A blind area ratio detection module, configured to determine a display ratio and a blind area ratio based on the tissue image obtained by the endoscope in the mirror withdrawal mode, and the sum of the display ratio and the blind area ratio is 1;

The display module is also used to display the display ratio and/or the blind area ratio, and when the blind area ratio is greater than the blind area threshold, display first prompt information, the first prompt information is used to indicate that there is a leak check risk.
The system according to claim 2, wherein the blind area ratio detection module is further used for: performing three-dimensional reconstruction according to the tissue image obtained by the endoscope in the retracting mirror mode to obtain a three-dimensional tissue image; and according to the three-dimensional The tissue image and the target corresponding to the three-dimensional tissue image are fitted to the tissue image, and a display ratio and a blind area ratio are determined.
The system according to claim 3, wherein the blind spot ratio detection module is also used for:

The ratio of the number of point clouds in the three-dimensional tissue image to the number of point clouds of the target fitting tissue image is determined as the display ratio, and the blind area ratio is determined according to the display ratio.
The system according to claim 1, wherein said system further comprises:

The three-dimensional positioning module is used to perform three-dimensional reconstruction according to the tissue image obtained by the endoscope in the endoscopic mode to obtain a three-dimensional tissue image; A point is determined as a three-dimensional target point of the target point in the three-dimensional tissue image;

A posture determination module, configured to determine the posture information of the endoscope according to the tissue image corresponding to the current location and the tissue image in the historical period;

A trajectory determination module, configured to generate a navigation trajectory according to the current position, the attitude information and the three-dimensional target point;

The display module is also used to display the posture information of the endoscope and the three-dimensional tissue image, and display the three-dimensional target point and the navigation track in the three-dimensional tissue image.
The system according to claim 1, wherein said system further comprises:

An image classification module, configured to classify the tissue image of the endoscope in the endoscope mode, and obtain the target classification corresponding to the tissue image, wherein the target classification includes abnormal classification and multiple endoscope classifications;

The detection pattern recognition module is used to update the counter of each endoscope classification according to the target classification corresponding to the tissue image, and when the value of the counter corresponding to any endoscope classification reaches a counting threshold, stop each of the The counting operation of the counter, and determine the detection mode of the endoscope according to the endoscope classification whose value of the counter reaches the counting threshold, wherein, each of the endoscope classifications has its corresponding counter, and each of the counters The value is initially zero;

The polyp identification module is further configured to: perform real-time polyp identification on the tissue image obtained by the endoscope in the retracting mode according to the polyp identification model corresponding to the detection mode determined by the detection mode identification module.
The system according to claim 6, wherein the detection pattern recognition module updates the counter of each endoscope classification according to the target classification corresponding to the tissue image in the following manner:

When the target classification is an endoscope classification, add one to the counter corresponding to the target classification, and subtract one from the counter corresponding to the endoscope classification whose counter value is not zero except for the target classification Operation: when the target category is an abnormal category, the counter corresponding to each endoscope category whose counter value is not zero is decremented by one, so as to update the counter of each endoscope category.
The system according to claim 1, wherein said system further comprises:

a pattern recognition module, configured to recognize the tissue image of the endoscope according to the image recognition model, and determine that the endoscope The image mode of the mirror is the mirror-in mode, and when the parameter corresponding to the ileocecal valve image output by the image recognition model is greater than the second parameter threshold, it is determined that the image mode of the endoscope is the mirror-out mode;

A mode switching module, used to switch the image mode of the endoscope when the image mode of the endoscope is an in vitro mode and the mode recognition module determines that the image mode of the endoscope is an in-scope mode To the mirror-in mode, when the image mode of the endoscope is the mirror-in mode and the pattern recognition module determines that the image mode of the endoscope is the mirror-back mode, the image of the endoscope The mode is switched to the mirror-back mode, and second prompt information is output, and the second prompt information is used to remind to enter the mirror-back mode.
The system according to claim 1, wherein said system further comprises:

A speed determination module, configured to acquire a plurality of historical tissue images corresponding to the current tissue image when the mode of the endoscopic image is the mirror-back mode, and calculate the similarity between each of the historical tissue images and the current tissue image Degree, according to a plurality of described similarities, determine the mirror-removing speed;

The display module is also used to display the mirror withdrawal speed corresponding to the endoscope.
The system according to claim 9, wherein said system further comprises:

The cleanliness determination module is used to classify the tissue images of the endoscope in the mirror withdrawal mode, and determine the number of tissue images under each cleanliness category; according to each of the cleanliness categories The number of tissue images determines the cleanliness of the tissue corresponding to the tissue images.
The system of claim 1, wherein the polyp identification module comprises:

The polyp detection sub-module is used to detect the tissue image obtained by the endoscope in the withdrawal mode based on the detection model, and determine the position information of the polyp when it is detected that there is a polyp in the tissue image;

A polyp identification submodule, configured to extract a detected image corresponding to the position information from the tissue image; determine the classification of the polyp according to the detected image and the identification model;

The display module is further configured to display the tissue image, and display the identification corresponding to the position information of the polyp and the classification in the tissue image.
An auxiliary method for endoscopic image detection, wherein the method includes:

Real-time processing of endoscopic images collected by the endoscope to obtain tissue images;

In response to determining that the endoscope is in the mirror-in mode, determine a target point of the tissue cavity corresponding to the tissue image, and the target point is used to indicate the next time the endoscope is in the mirror-in mode at its current position. Target moving point; wherein, when there is a tissue cavity in the tissue image, the target point is the center point of the tissue cavity corresponding to the endoscopic image, and when there is no tissue cavity in the tissue image , the target point is a direction point of the tissue cavity corresponding to the endoscopic image;

In response to determining that the endoscope is in the retracting mirror mode, perform real-time polyp identification on the tissue image obtained by the endoscope in the retracting mirror mode, and obtain a polyp identification result;

In the image display area in the display interface, the target point and the polyp recognition result are displayed.
The method according to claim 12, wherein said method further comprises:

According to the tissue image obtained by the endoscope in the mirror withdrawal mode, a display ratio and a blind area ratio are determined, and the sum of the display ratio and the blind area ratio is 1;

Displaying the display ratio and/or the blind area ratio, and displaying first prompt information when the blind area ratio is greater than a blind area threshold, where the first prompt information is used to indicate that there is a risk of missed detection.
The method according to claim 13, wherein said determining the display ratio and the blind area ratio according to the tissue image obtained by the endoscope in the mirror withdrawal mode includes:

Performing three-dimensional reconstruction according to the tissue image obtained by the endoscope in the retracting mirror mode to obtain a three-dimensional tissue image;

A display ratio and a blind area ratio are determined according to the three-dimensional tissue image and the target fitting tissue image corresponding to the three-dimensional tissue image.
The method according to claim 14, wherein said determining a display ratio and a blind area ratio according to the three-dimensional tissue image and the target fitting tissue image corresponding to the three-dimensional tissue image comprises:

The ratio of the number of point clouds in the three-dimensional tissue image to the number of point clouds of the target fitting tissue image is determined as the display ratio, and the blind area ratio is determined according to the display ratio.
The method according to claim 12, wherein said method further comprises:

Performing three-dimensional reconstruction according to the tissue image obtained by the endoscope in the endoscope mode to obtain a three-dimensional tissue image;

determining a point on the centerline of the three-dimensional tissue image that is closest to the target point as the three-dimensional target point of the target point on the three-dimensional tissue image;

Determine the posture information of the endoscope according to the tissue image corresponding to the current location and the tissue image in the historical period, and generate a navigation track according to the current location, the posture information and the three-dimensional target point;

The three-dimensional tissue image is displayed, and the three-dimensional target point and the navigation track are displayed in the three-dimensional tissue image.
The method according to claim 12, wherein said method further comprises:

In response to the user's selection operation of the detection mode in the control area of the display interface, the detection mode selected by the user is determined as the detection mode of the endoscope;

The real-time polyp identification of the tissue image obtained by the endoscope in the mirror withdrawal mode includes:

According to the polyp identification model corresponding to the detection mode, real-time polyp identification is performed on the tissue image obtained by the endoscope in the retracting mirror mode.
The method according to claim 12, wherein said method further comprises:

Classifying the tissue image of the endoscope in the endoscope mode to obtain a target classification corresponding to the tissue image, wherein the target classification includes an abnormal classification and a plurality of endoscope classifications;

According to the target classification corresponding to the tissue image, the counter of each endoscope classification is updated, and when the value of the counter corresponding to any endoscope classification reaches the counting threshold, the counting operation of each of the counters is stopped, and according to The endoscope classification whose value of the counter reaches the counting threshold determines the detection mode of the endoscope, wherein each of the endoscope classifications has its corresponding counter, and the value of each of the counters is initially zero;

The real-time polyp identification of the tissue image obtained by the endoscope in the mirror withdrawal mode includes:

According to the polyp identification model corresponding to the detection mode, real-time polyp identification is performed on the tissue image obtained by the endoscope in the retracting mirror mode.
The method according to claim 18, wherein said updating the counter of each endoscope classification according to the target classification corresponding to the tissue image comprises:

When the target classification is an endoscope classification, add one to the counter corresponding to the target classification, and subtract one from the counter corresponding to the endoscope classification whose counter value is not zero except for the target classification Operation: when the target category is an abnormal category, the counter corresponding to each endoscope category whose counter value is not zero is decremented by one, so as to update the counter of each endoscope category.
The method according to claim 12, wherein said method further comprises:

Recognizing the tissue image of the endoscope according to the image recognition model;

If the parameter corresponding to the in-vivo image output by the image recognition model is greater than the first parameter threshold, it is determined that the image mode of the endoscope is the endoscope mode; if the parameter corresponding to the ileocecal valve image output by the image recognition model Greater than the second parameter threshold, it is determined that the image mode of the endoscope is the mirror-back mode;

When the image mode of the endoscope is an in vitro mode and the image mode of the endoscope determined based on the image recognition model is an endoscope mode, switch the image mode of the endoscope to an endoscope mode mode, when the image mode of the endoscope is the mirror-in mode, and the image mode of the endoscope determined based on the image recognition model is the mirror-back mode, the image of the endoscope The mode is switched to the mirror-back mode, and second prompt information is output, and the second prompt information is used to remind to enter the mirror-back mode.
The method according to claim 12, wherein said method further comprises:

When the mode of the endoscopic image is the mirror-back mode, obtain a plurality of historical tissue images corresponding to the current tissue image, and calculate the similarity between each of the historical tissue images and the current tissue image, according to the plurality of historical tissue images The above similarity determines the mirror withdrawal speed;

The mirror withdrawal speed corresponding to the endoscope is displayed in the image display area.
The method according to claim 21, wherein said method further comprises:

Carrying out cleanliness classification on the tissue images of the endoscope in the withdrawal mode, and determining the number of tissue images under each cleanliness category;

According to the quantity of tissue images under each of the cleanliness categories, the cleanliness of the tissue corresponding to the tissue images is determined.
The method according to claim 12, wherein said performing real-time polyp identification on the tissue image obtained by the endoscope in the retracting mirror mode comprises:

Based on the detection model, the tissue image obtained by the endoscope in the withdrawal mode is detected, and when polyps are detected in the tissue image, the position information of the polyps is determined;

extracting a detection image corresponding to the position information from the tissue image, and determining the classification of the polyp according to the detection image and the recognition model;

The tissue image is displayed in the image display area, and the identification corresponding to the position information of the polyp and the classification are displayed in the tissue image.
A computer-readable medium, on which a computer program is stored, wherein the program implements the steps of any one of claims 12-23 when executed by a processing device.
An electronic device comprising:

a storage device on which a computer program is stored;

A processing device configured to execute the computer program in the storage device to implement the steps of the method according to any one of claims 12-23.