WO2023138619A1

WO2023138619A1 - Endoscope image processing method and apparatus, readable medium, and electronic device

Info

Publication number: WO2023138619A1
Application number: PCT/CN2023/072905
Authority: WO
Inventors: 边成; 李永会; 杨延展; 杨志雄
Original assignee: 小荷医疗器械(海南)有限公司
Priority date: 2022-01-21
Filing date: 2023-01-18
Publication date: 2023-07-27
Also published as: CN114429458A

Abstract

The present invention relates to the technical field of image processing, and relates to an endoscope image processing method and apparatus, a readable medium, and an electronic device. The method comprises: obtaining a tissue image set collected by an endoscope in a tissue to be tested; determining, according to the tissue image set, a depth image and an attitude parameter corresponding to each tissue image; determining a motion trajectory of the endoscope according to the attitude parameter corresponding to each tissue image; determining, according to the depth image corresponding to each tissue image, a contour of the tissue to be tested; and determining a blind area proportion in an endoscope examination process according to the motion trajectory and the contour of the tissue to be tested. According to the present invention, the motion trajectory of the endoscope and the contour of the tissue to be tested are determined according to the depth image and the attitude parameter corresponding to each tissue image, and on this basis, the blind area proportion in the examination process is determined, so that the monitoring of an examination range can be achieved, and missed examinations can be effectively avoided, thereby ensuring the effectiveness of an endoscope examination.

Description

Endoscopic image processing method, device, readable medium and electronic equipment

Cross References to Related Applications

This application claims the priority of the Chinese patent application with application number 202210074391.6 and titled "Processing Method, Device, Readable Medium, and Electronic Equipment for Endoscopic Image" filed on January 21, 2022. The entire content of this application is incorporated by reference in this application.

technical field

The present disclosure relates to the technical field of image processing, and in particular, to a processing method, device, readable medium, and electronic device for an endoscope image.

Background technique

As a commonly used and effective inspection method, endoscopy has been widely used in the medical field because it can visually observe the internal tissues of the human body. When the endoscope enters the internal tissues of the human body for inspection, there may be blind spots in the field of vision. If the blind spots are too large, it may lead to missed inspections and further lead to invalid inspections. An optional solution is to perform 3D modeling based on the images collected by the endoscope to determine the proportion of the blind area. Whether the 3D modeling is accurate will directly affect the accuracy of the proportion of the blind area. Since human tissues (such as intestines, stomach, etc.) are soft tissues, the endoscope will inevitably touch the tissue walls of the soft tissues during the process of entering the soft tissues, causing large displacements of the soft tissues, resulting in errors in the results of 3D modeling. At the same time, if a polyp is encountered during the inspection, the inspector will flush water, remove the polyp, etc., which will also reduce the accuracy of the 3D modeling.

Contents of the invention

This Summary is provided to introduce a simplified form of concepts that are described in detail later in the Detailed Description. This summary of the invention is not intended to identify key features or essential features of the claimed technical solution, nor is it intended to be used to limit the scope of the claimed technical solution.

In a first aspect, the present disclosure provides a method for processing endoscopic images, the method comprising:

Obtaining a set of tissue images collected by the endoscope in the tissue to be tested, the set of tissue images including a plurality of tissue images arranged according to the acquisition time;

Determining a depth image and a pose parameter corresponding to each of the tissue images according to the tissue image set;

determining the motion trajectory of the endoscope according to the posture parameters corresponding to each of the tissue images;

determining the contour of the tissue to be measured according to the depth image corresponding to each of the tissue images;

According to the motion trajectory and the outline of the tissue to be measured, the proportion of the blind area during the endoscopic examination is determined.

In a second aspect, the present disclosure provides an endoscopic image processing device, the device comprising:

An acquisition module, configured to acquire a set of tissue images collected by the endoscope in the tissue to be measured, the set of tissue images including a plurality of tissue images arranged according to the acquisition time;

A positioning module, configured to determine a depth image and a pose parameter corresponding to each of the tissue images according to the tissue image set;

A trajectory determination module, configured to determine the movement trajectory of the endoscope according to the posture parameters corresponding to each of the tissue images;

A contour determination module, configured to determine the contour of the tissue to be measured according to the depth image corresponding to each of the tissue images;

A processing module, configured to determine a blind area ratio during the endoscopic inspection process according to the motion track and the contour of the tissue to be measured.

In a third aspect, the present disclosure provides a computer-readable medium on which a computer program is stored, and when the program is executed by a processing device, the steps of the method described in the first aspect of the present disclosure are implemented.

In a fourth aspect, the present disclosure provides an electronic device, including:

a storage device on which a computer program is stored;

A processing device configured to execute the computer program in the storage device to implement the steps of the method described in the first aspect of the present disclosure.

Through the above technical solution, the present disclosure first acquires the tissue images collected by the endoscope in the tissue to be measured according to multiple collection moments. Then, according to the tissue image set, the depth image and pose parameters corresponding to each tissue image are determined. Then, according to the posture parameters corresponding to each tissue image, the motion trajectory of the endoscope is determined, and according to the depth image corresponding to each tissue image, the contour of the tissue to be measured is determined. Finally, according to the motion trajectory and the outline of the tissue to be measured, the proportion of the blind area during the endoscopic examination is determined. The present disclosure uses the depth image corresponding to the tissue image to And posture parameters, determine the trajectory of the endoscope and the outline of the tissue to be tested, and determine the proportion of the blind area in the inspection process, which can realize the monitoring of the inspection range, effectively avoid missed inspections, and ensure the effectiveness of endoscopic inspection.

Other features and advantages of the present disclosure will be described in detail in the detailed description that follows.

Description of drawings

The above and other features, advantages and aspects of the various embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic and that elements and elements are not necessarily drawn to scale. In the attached picture:

Fig. 1 is a flow chart of a method for processing endoscopic images according to an exemplary embodiment;

Fig. 2 is a flow chart of another endoscopic image processing method shown according to an exemplary embodiment;

Fig. 3 is a flow chart of another endoscopic image processing method shown according to an exemplary embodiment;

Fig. 4 is a schematic diagram showing the outline of the tissue to be measured according to an exemplary embodiment;

Fig. 5 is a schematic diagram of a positioning model according to an exemplary embodiment;

Fig. 6 is a flow chart of another endoscopic image processing method shown according to an exemplary embodiment;

Fig. 7 is a schematic diagram of a depth sub-model and an attitude sub-model according to an exemplary embodiment;

Fig. 8 is a flowchart showing a training positioning model according to an exemplary embodiment;

Fig. 9 is a schematic diagram of another attitude sub-model according to an exemplary embodiment;

Fig. 10 is a flow chart showing another training positioning model according to an exemplary embodiment;

Fig. 11 is a flow chart of another endoscopic image processing method shown according to an exemplary embodiment;

Fig. 12 is a block diagram of an endoscopic image processing device according to an exemplary embodiment;

Fig. 13 is a block diagram of another endoscopic image processing device according to an exemplary embodiment;

Fig. 14 is a block diagram of another endoscopic image processing device according to an exemplary embodiment;

Fig. 15 is a block diagram of another endoscopic image processing device according to an exemplary embodiment;

Fig. 16 is a block diagram of an electronic device according to an exemplary embodiment.

Detailed ways

Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the drawings, it should be understood that the disclosure may be embodied in various forms and should not be construed as limited to the embodiments set forth herein, but are provided so that the disclosure will be more thorough and complete. It should be understood that the drawings and embodiments of the present disclosure are for exemplary purposes only, and are not intended to limit the protection scope of the present disclosure.

It should be understood that the various steps described in the method implementations of the present disclosure may be executed in different orders, and/or executed in parallel. Additionally, method embodiments may include additional steps and/or omit performing illustrated steps. The scope of the present disclosure is not limited in this respect.

As used herein, the term "comprise" and its variations are open-ended, ie "including but not limited to". The term "based on" is "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one further embodiment"; the term "some embodiments" means "at least some embodiments." Relevant definitions of other terms will be given in the description below.

It should be noted that concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the sequence or interdependence of the functions performed by these devices, modules or units.

It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative and not restrictive, and those skilled in the art should understand that unless the context clearly indicates otherwise, it should be understood as "one or more".

The names of messages or information exchanged between multiple devices in the embodiments of the present disclosure are used for illustrative purposes only, and are not used to limit the scope of these messages or information.

Fig. 1 is a flowchart of a method for processing endoscopic images according to an exemplary embodiment. As shown in Fig. 1, the method may include the following steps:

Step 101, acquire a tissue image set collected by an endoscope in a tissue to be measured, the tissue image set includes a plurality of tissue images arranged according to the acquisition time.

For example, during an endoscopic examination, the endoscope will continuously collect histograms in the tissue to be tested according to the preset collection cycle image to get an organized image set. The tissue image set may include multiple tissue images arranged according to the acquisition time, and the interval between the acquisition time corresponding to any two adjacent tissue images is the acquisition period. Specifically, multiple tissue images collected within a preset time period (for example: 30s) can be used as a tissue image set, or a preset number of tissue images (for example: 100) collected continuously can be used as a tissue image set, which is not specifically limited in the present disclosure. It should be noted that the endoscope described in the embodiments of the present disclosure may be, for example, a colonoscope, a gastroscope, etc. If the endoscope is a colonoscope, then the tissue to be measured is the intestinal tract, and the tissue image is the intestinal tract image. If the endoscope is a gastroscope, the tissue to be measured may be the esophagus, stomach, or duodenum, and the image of the above tissue may be an image of the esophagus, stomach, or duodenum. The endoscope can also be used to acquire images of other tissues, which is not specifically limited in the present disclosure.

During the endoscopic inspection process, many invalid images may be collected due to unstable approach or improper position of the endoscope, such as images blocked by obstacles, overexposure, and low definition. These invalid images can interfere with the endoscopic examination results. Therefore, after obtaining the tissue image set, it may first be judged whether the plurality of tissue images contained therein are valid, so as to filter out invalid tissue images. If a tissue image is an invalid image, the tissue image can be discarded directly. If the tissue image is a valid image, the tissue image can be retained to obtain a filtered tissue image set, which can reduce unnecessary data processing and improve processing speed. For example, a pre-trained recognition model can be used to recognize each tissue image in the tissue image set to determine whether the tissue image is valid. The recognition model can be, for example, CNN (English: Convolutional Neural Networks, Chinese: Convolutional Neural Network) or LSTM (English: Long Short-Term Memory, Chinese: Long Short-Term Memory Network), or an Encoder in a Transformer (such as a Vision Transformer), which is not specifically limited in this disclosure. Furthermore, each tissue image in the tissue image set may also be preprocessed, which may be understood as performing enhancement processing on the data included in each tissue image. In order to ensure the quality of the tissue image, the preprocessing will not modify the blur or color of the tissue image. Therefore, the preprocessing can include: multi-crop processing, flip processing (including: left-right flip, up-down flip, rotation, etc.), random affine transformation, size transformation (English: Resize) and other processing. The final preprocessed tissue image can be an image of a specified size (for example, it can be 384*384).

Step 102, according to the tissue image set, determine the depth image and pose parameters corresponding to each tissue image.

For example, for each tissue image in the tissue image set, the depth image and pose parameters corresponding to each tissue image may be sequentially determined. The depth image corresponding to each tissue image includes the depth (also can be understood as distance) of each pixel in the tissue image, so the corresponding depth image can reflect the geometry of the visible surface in the tissue image without being affected by the texture, color, etc. in the tissue image, that is, the corresponding depth image can represent the structural information of the tissue to be measured corresponding to the tissue image. The attitude parameters corresponding to each tissue image can be understood as the attitude parameters of the endoscope when acquiring the tissue image, and the attitude parameters corresponding to multiple continuous tissue images can represent the movement process of the endoscope in the tissue to be measured. The attitude parameters can include, for example, a rotation matrix and a translation vector.

Step 103, according to the posture parameters corresponding to each tissue image, determine the movement trajectory of the endoscope.

Step 104: Determine the contour of the tissue to be measured according to the depth image corresponding to each tissue image.

For example, the posture parameters corresponding to each tissue image in the tissue image set can represent the movement process of the endoscope in the tissue to be measured, so the motion trajectory of the endoscope in the tissue to be measured can be obtained according to the posture parameters corresponding to each tissue image, and the motion trajectory can include the position of the endoscope when acquiring each tissue image, and can also include the angle of the endoscope when acquiring each tissue image.

At the same time, the corresponding depth image can represent the structural information of the tissue corresponding to the tissue image. Therefore, the contour of the tissue to be measured can be obtained according to the depth image corresponding to each tissue image. The contour of the tissue to be measured can reflect the overall shape of the tissue to be measured, and can also be understood as a template of the tissue to be measured. If the endoscope is a colonoscope as an example, then the tissue to be tested is the intestinal tract, and the outline of the tissue to be tested can be a distorted cylinder. Specifically, the centerline of the tissue to be measured can be determined according to multiple depth images corresponding to the tissue image set, and then the contour of the tissue to be measured can be obtained according to a preset radius. Modeling may also be performed according to multiple depth images corresponding to the tissue image set to obtain the outline of the tissue to be measured, which is not specifically limited in the present disclosure. It should be noted that the execution order of step 103 and step 104 shown in FIG. 1 is an exemplary implementation manner, and step 104 may be executed first, and then step 103 may be executed, or step 103 and step 104 may be executed simultaneously, which is not specifically limited in the present disclosure.

Step 105, according to the motion trajectory and the outline of the tissue to be measured, determine the proportion of the blind area during the endoscopic examination.

For example, after obtaining the motion trajectory and the outline of the tissue to be measured, the field of view of the endoscope when capturing each tissue image can be determined according to the position and angle of the endoscope when capturing each tissue image. The field of view can be understood as the area of the tissue to be measured that the endoscope can observe when capturing the tissue image. Then the visual field areas corresponding to each tissue image can be spliced to obtain the area that can be observed during the endoscopic examination, so as to obtain the blind area ratio during the endoscopic examination. Specifically, the ratio of the observed area can be determined according to the ratio of the area that can be observed during the endoscopic examination to the outline of the tissue to be measured, and then the blind area ratio can be determined, that is, the blind area ratio Example=1-Observation Area Ratio. The blind area ratio can be understood as the ratio of the blind area (that is, the part that cannot be observed in the field of view of the endoscope) to the outline of the tissue to be measured during the endoscopic examination. Determining the proportion of the blind area on the basis of the depth image and the motion track can reflect the inspection range in the inspection process in time, thereby avoiding missed inspections and ensuring the effectiveness of endoscopic inspections.

To sum up, in the present disclosure, firstly, the tissue images collected by the endoscope in the tissue to be measured are acquired according to multiple acquisition moments. Then, according to the tissue image set, the depth image and pose parameters corresponding to each tissue image are determined. Then, according to the posture parameters corresponding to each tissue image, the motion trajectory of the endoscope is determined, and according to the depth image corresponding to each tissue image, the contour of the tissue to be measured is determined. Finally, according to the motion trajectory and the outline of the tissue to be measured, the proportion of the blind area during the endoscopic examination is determined. According to the depth image and attitude parameters corresponding to the tissue image, the present disclosure determines the movement trajectory of the endoscope and the outline of the tissue to be measured, and determines the blind area ratio in the inspection process, so as to realize the monitoring of the inspection range and effectively avoid missed inspections, thereby ensuring the effectiveness of endoscopic inspection.

In an implementation manner, the implementation manner of step 102 may be:

According to each tissue image and the historical tissue image corresponding to the tissue image in turn, the depth image and attitude parameters corresponding to the tissue image are determined through the pre-trained positioning model, and the collection time of the historical tissue image is before the collection time of the tissue image.

For example, each tissue image and the corresponding historical tissue image may be sequentially input into the pre-trained positioning model, so that the positioning model determines the corresponding depth image and pose parameters of the tissue image according to the tissue image and the corresponding historical tissue image. Wherein, the collection time of the corresponding historical tissue image is before the collection time of the tissue image, that is, the tissue image set, and the corresponding historical tissue image is located before the tissue image, which may be the tissue image set, the previous tissue image before the tissue image. For example, the tissue image collected by the endoscope at time t can be denoted as It, then the historical tissue image corresponding to the tissue image can be denoted as It-1, that is, the image collected by the endoscope at time t-1.

The positioning model can be understood as a SLAM (English: Simultaneous Localization and Mapping, Chinese: Simultaneous Localization and Mapping) model, which can simultaneously determine the corresponding depth image and attitude parameters according to each tissue image and the historical tissue image corresponding to the tissue image. The positioning model can determine the depth image corresponding to each tissue image, and there is no need to add a depth sensor when the endoscope is inspected, which is convenient for operation and saves costs. At the same time, the positioning model can determine the attitude parameters to accurately obtain the motion trajectory of the endoscope.

In another implementation manner, the pose parameters may include a rotation matrix and a translation vector, and the motion trajectory may include the position and angle of the endoscope when capturing each tissue image. Correspondingly, the implementation manner of step 103 is:

According to the rotation matrix and translation vector corresponding to each tissue image, and the position and angle of the endoscope when acquiring the historical tissue image corresponding to the tissue image, the position and angle of the endoscope when acquiring the tissue image are determined.

For example, the position and angle of the endoscope when acquiring the tissue image can be determined according to the posture parameters corresponding to each tissue image, and then the position and angle of the endoscope when acquiring all the tissue images can be arranged according to the sequence indicated by the acquisition time, so as to obtain the movement trajectory of the endoscope. Specifically, the position of the endoscope when acquiring the tissue image can be determined according to the position of the historical tissue image corresponding to the tissue image and the translation vector corresponding to the tissue image in sequence, and the angle of the endoscope when acquiring the tissue image can be determined according to the angle when the endoscope acquires the historical tissue image corresponding to the tissue image and the rotation matrix corresponding to the tissue image.

For example, the position and angle of the first tissue image in the tissue image set can be set as a preset initial position and initial angle, and then the position and angle of the second tissue image can be determined according to the position and angle of the first tissue image, and the corresponding rotation matrix and translation vector of the second tissue image. Then determine the position and angle of the third tissue image according to the position and angle of the second tissue image, and the corresponding rotation matrix and translation vector of the third tissue image, and so on, to obtain the movement track of the endoscope in the tissue to be measured.

Fig. 2 is a flowchart of another endoscopic image processing method shown according to an exemplary embodiment, as shown in Fig. 2, step 104 may include:

Step 1041: Determine the centerline of the tissue to be measured according to the depth image corresponding to each tissue image.

Step 1042, determine the outline of the tissue to be measured according to the centerline of the tissue to be measured.

For example, according to the depth image corresponding to each tissue image, the midpoint of the tissue to be measured in the tissue image can be determined, and then the midpoints of the tissue to be measured in each tissue image can be connected to obtain the centerline of the tissue to be measured. Then, according to the preset radius and centerline, the outline of the tissue to be measured is determined. Taking the tissue image as an intestinal image and the tissue to be tested as an example of the intestinal tract, the contour of the intestinal tract is a cylinder established according to the preset radius and centerline. Specifically, the manner of determining the midpoint of the tissue to be measured in each tissue image may first determine the distance of boundaries in the tissue image (for example, may include: left boundary, right boundary, upper boundary, lower boundary, etc.), and then determine a point that is equally distant from each boundary in the depth image as the midpoint of the tissue to be measured in the tissue image.

Fig. 3 is a flow chart of another endoscopic image processing method shown according to an exemplary embodiment. As shown in Fig. 3, the implementation of step 105 may include:

Step 1051, according to the position and angle of the endoscope when collecting each tissue image, and the viewing angle of the endoscope, determine the field of view corresponding to the tissue image.

Step 1052: Determine the total visual field area according to the visual field area corresponding to each tissue image.

Step 1053, according to the total field of view and the outline of the tissue to be measured, determine the proportion of the blind area.

For example, after obtaining the motion trajectory and the outline of the tissue to be measured, the field of view of the endoscope when acquiring each tissue image can be determined according to the position and angle of the endoscope when acquiring each tissue image, and the viewing angle of the endoscope itself. The viewing angle of the endoscope is determined by the optical lens of the endoscope, and the viewing angle may be, for example, 100 degrees or 120 degrees. The field of view area can be understood as the area of the tissue to be measured covered by the tissue image. Take the outline of the tissue to be measured as shown in Figure 4 as an example, wherein the thick solid line represents the outline of the tissue to be measured (for the convenience of presentation, a two-dimensional section is used here to represent the outline of the tissue to be measured. In actual situations, the outline of the tissue to be measured is three-dimensional, such as a cylinder), where k(0) represents the position of the endoscope at time t0. Correspondingly, the angle of the endoscope at time t0 can be expressed as α(0) (it should be noted that α(0) is not shown in FIG. It can be obtained that the field of view corresponding to the tissue image collected at time t0 is point A to point B on the contour. Specifically, a Monte Carlo method (English: Monte Carlo method) can be used to evenly distribute X test points (X≥100) on the contour of the tissue to be tested, and then determine the area of the visual field according to the number of test points included in the visual field.

Afterwards, the visual field area corresponding to each tissue image can be spliced to obtain the total visual field area. Specifically, the visual field areas corresponding to each tissue image can be summed, and the summation result can be used as the total visual field area, or the test points covered in the visual field areas corresponding to each tissue image can be summed to obtain the total number of test points included in the total visual field area, which can be used as the total visual field area. Finally, the ratio of the total visual field area to the total area of the outline of the tissue to be measured can be used as the ratio of the observation area, and then (1-the ratio of the observation area) can be used as the ratio of the blind area. For example, the total number of test points included in the total field of view is Y, that is, Y test points are covered in the area that can be observed during the endoscopic examination, and there are X test points distributed on the outline of the tissue to be tested, so the ratio of the observation area can be determined to be Y/X first, and then the ratio of the blind area can be further determined to be 1-Y/X.

In an implementation manner, step 1051 may be implemented through the following steps:

Step 1) Convert the position of the endoscope when acquiring the tissue image into a center position corresponding to the center line of the tissue to be measured according to the posture parameters corresponding to each tissue image.

Step 2) Determine the central viewing angle corresponding to the central position according to the posture parameters corresponding to the tissue image, the viewing angle of the endoscope, and the angle of the endoscope when collecting the tissue image.

Step 3) Determine the maximum viewing area corresponding to the center position.

Step 4) Determine the field of view corresponding to the tissue image according to the central viewing angle and the maximum field of view.

For example, in order to quickly determine the field of view corresponding to each tissue image and improve the efficiency of image processing, the position and viewing angle of the endoscope may be converted to the central position and central viewing angle on the centerline of the tissue to be measured. It can be understood that, when the endoscope is at the position when the tissue image is collected, the field of view that can be observed by the endoscope according to the angle when the tissue image is collected is the same as the field of view that the endoscope can observe by the central perspective according to the angle when the tissue image is collected at the central position. Also shown in Fig. 4, wherein, d(0) indicates that the position of the endoscope at time t0 is converted to the center position on the center line, the viewing angle of the endoscope is φ, and the corresponding central viewing angle can be δ, so that the field of view observed by the endoscope at the central viewing angle on d(0) according to the angle when the tissue image is collected is also from point A to point B. Specifically, the central position and the corresponding central viewing angle can be determined in the following ways:

A vertical line can be drawn from the position of the endoscope when the tissue image is collected to the outline of the tissue to be measured, and the position where the vertical line intersects the center line is the center position, ie d(0). Then, a geometric transformation can be performed according to the viewing angle φ of the endoscope and the angle of the endoscope when collecting the tissue image, so as to obtain the central viewing angle δ.

Afterwards, according to the central position, the maximum viewing area corresponding to the central position can be determined. The maximum viewing area corresponding to the center position can be understood as the maximum range that the endoscope can observe at the center position, that is, the maximum range that can be observed by rotating the optical lens of the endoscope by 360 degrees. Then, the visual field area corresponding to the tissue image may be determined according to the central viewing angle and the maximum visual field area. Specifically, the visual field area corresponding to the tissue image may be determined according to the product of the ratio of the central viewing angle to 360 degrees and the maximum visual field area. For example, if the central viewing angle is 120 degrees and the number of test points included in the largest field of view is 210, then the number of test points included in the field of view corresponding to the tissue image is 210*(120/360)=70.

In an implementation manner, the structure of the positioning model may be as shown in FIG. 5 , which includes: a depth sub-model and an attitude sub-model. Among them, the input of the depth sub-model and the input of the attitude sub-model are used as the input of the positioning model, and the output of the depth sub-model and the output of the attitude sub-model are used as the output of the positioning model.

Fig. 6 is a flow chart showing another endoscopic image processing method according to an exemplary embodiment. As shown in Fig. 6 , the positioning model includes: a depth sub-model and an attitude sub-model. Step 102 may include:

Step 1021: Input the tissue image into the depth sub-model to obtain a depth image corresponding to the tissue image output by the depth sub-model.

For example, the tissue image can be used as an input of the depth sub-model, and the depth sub-model can output a depth image corresponding to the tissue image. The structure of the depth sub-model can be shown in (a) in Figure 7, which can be a UNet structure, which includes multiple stride convolution layers (English: stride conv) to downsample the tissue image, for example, it can downsample to 1/8 of the resolution of the tissue image, and then use multiple transpose convolution layers (English: transpose conv) to upsample to the resolution of the tissue image to obtain the depth image corresponding to the tissue image.

Step 1022, input the tissue image and the corresponding historical tissue image into the pose sub-model, so as to obtain the pose parameters corresponding to the tissue image output by the pose sub-model.

For example, the tissue image and the corresponding historical tissue image can be used as input of the attitude sub-model, and the attitude sub-model can output the rotation matrix and translation vector corresponding to the tissue image. Specifically, the tissue image and the corresponding historical tissue image may be concatenated (English: Concat), so as to input the concatenated result into the attitude sub-model. The structure of the pose sub-model can be shown in (b) in Figure 7, which can be a ResNet structure (for example, ResNet34). The stitching result of the tissue image and the corresponding historical tissue image is input into the initial convolution pooling layer, through multiple residual blocks (English: Residual block) in the middle, and finally the rotation matrix and translation vector corresponding to the tissue image are output by the fully connected layer.

Fig. 8 is a flow chart showing a training positioning model according to an exemplary embodiment. As shown in Fig. 8, the positioning model is trained through the following steps:

Step A, input the sample tissue image into the depth sub-model to obtain the sample depth image corresponding to the sample tissue image, and input the historical sample tissue image into the depth sub-model to obtain the historical sample depth image corresponding to the historical sample tissue image, the historical sample tissue image is an image collected before the sample tissue image.

For example, a sample tissue image (denoted as I _a ) is used as an input of the depth sub-model, and the depth sub-model can output a sample depth image (denoted as D _a ) corresponding to the sample tissue image. Similarly, the historical sample tissue image (denoted as I _b ) is used as the input of the depth sub-model, and the depth sub-model can output the historical sample depth image (denoted as D _b ) corresponding to the historical sample tissue image. Wherein, the sample tissue image may be obtained by extracting frames from an endoscopic video, and the endoscopic video may be a video recorded during a previous endoscopic examination, and may be obtained by selecting different endoscopic examinations for different users. Further, when frame-picking the endoscopic video, invalid images (such as images blocked by obstacles, overexposed, and low in definition) can be filtered out. Correspondingly, the historical sample tissue image is the tissue image of the previous frame of the sample tissue image.

Step B, input the sample tissue image and the historical sample tissue image into the attitude sub-model to obtain the output of the attitude sub-model, the sample attitude parameters corresponding to the sample tissue image and the internal parameters of the endoscope for collecting the sample tissue image. The endoscopic internal parameters include focal length and translation size.

For example, the sample tissue image and the historical sample tissue image can be used as the input of the attitude sub-model, and the attitude sub-model can output the sample attitude parameters corresponding to the sample tissue image and the internal parameters of the endoscope (denoted as K) for collecting the sample tissue image. Wherein, the internal parameters of the endoscope may include focal length and translation size, and the sample attitude parameters include a sample rotation matrix (represented as R) and a sample translation vector (represented as t). Specifically, the sample tissue image and the historical sample tissue image may be spliced, so as to input the spliced result into the pose sub-model.

In the training phase, the attitude sub-model can also add a linear layer (represented as an intrinsic layer) on the basis of the convolutional pooling layer, multiple residual blocks, and fully connected layers, as shown in Figure 9. The fully connected layer (denoted as pose layer) outputs the sample pose parameters, and the linear layer is able to output endoscopic intrinsic parameters. The form of the internal parameter K of the endoscope can be:

Among them, f _x and f _y respectively represent the focal length of the endoscope in the X and Y directions (in pixels), and c _x and _cy represent the origin at X, Y, respectively. The translation size in the Y direction in pixels. The attitude sub-model can obtain the internal parameters of the endoscope while obtaining the attitude parameters of the sample. It is not necessary to calibrate the endoscope in advance, which is easy to operate, and can be adapted to various endoscopes, which improves the scope of application of the depth sub-model.

In step C, the target loss is determined according to the internal parameters of the endoscope, the sample depth image, the historical sample depth image and the sample pose parameters.

Step D, with the goal of reducing the target loss, use the backpropagation algorithm to train the localization model.

For example, according to the internal parameters of the endoscope, the sample depth image, the historical sample depth image and the sample pose parameters, and aiming at reducing the target loss, the localization model can be trained by using the backpropagation algorithm. When training the positioning model, the sample tissue images and historical sample tissue images used to train the positioning model can be quickly obtained without pre-labeling. That is to say, the positioning model adopts an unsupervised learning training method.

Further, the initial learning rate for training the positioning model can be set to: 1e-2, the Batch size can be set to: 16*4, the optimizer can be set to: SGD, the Epoch can be set to: 500, and the size of the sample tissue image can be set to: 384×384.

Fig. 10 is a flow chart showing another training positioning model according to an exemplary embodiment. As shown in Fig. 10 , the implementation of step C may include:

Step C1, according to the sample depth image, the sample attitude parameters and the internal parameters of the endoscope, the historical sample tissue images are interpolated to obtain the interpolated tissue images.

Step C2, determining the photometric loss according to the sample tissue image and the interpolated tissue image.

For example, differentiable bilinear interpolation can be performed on historical sample tissue images by using the sample depth image, sample attitude parameters, and endoscope internal parameters to obtain an interpolated tissue image. Luminosity loss is thereby determined from the sample tissue image and the interpolated tissue image. The interpolated tissue image can be understood as an image obtained by observing content in the sample tissue image from the perspective of collecting historical sample tissue images. According to the principle of beam adjustment method, the pixel gray level of the same spatial point should be fixed in each image. Therefore, when the images collected from different viewing angles are converted to another viewing angle, the pixels at the same position in the two images under the same viewing angle should be the same. Therefore, the photometric loss can be understood as the difference between the sample tissue image and the interpolated tissue image. For example, the photometric loss can be determined by formula 1:

Among them, L _p represents the photometric loss, p represents the pixel point, N represents the effective pixel point in the sample tissue image, and |N| represents the number of effective pixel points. I _a (p) represents the pixel value of p in the sample tissue image _{, and} I' _a (p) represents the pixel value of p in the interpolated tissue image. || || ₁ means the L1 norm, which is more robust to discrete points.

Step C3, determining a smoothing loss according to the gradient of the sample depth image and the gradient of the sample tissue image.

For example, in the low-texture region of the sample tissue image (or interpolated tissue image), due to less image feature information, the performance of the photometric loss is weak, so smooth loss can be added as a regular term to constrain the generated sample depth image. The smoothing loss can be determined according to the gradient of the sample depth image and the gradient of the sample tissue image. The smoothing loss can ensure that the sample depth image is generated under the guidance of the sample tissue image, so that the generated sample depth image can retain more gradient information at the edge, that is, the edge is more obvious and the detail information is richer. For example, the smoothing loss can be determined by formula 2:

where L _s represents the smoothing loss, Indicates the gradient of p in the sample tissue image, Denotes the gradient of p in the sample depth image.

Step C4, transforming the sample depth image into a first depth image according to the sample pose parameters and the internal parameters of the endoscope.

Step C5, transforming the historical sample depth image into a second depth image according to the sample pose parameters and the internal parameters of the endoscope.

Step C6, determining consistency loss according to the first depth image and the second depth image.

For example, since the sample tissue image and the historical sample tissue image face the same three-dimensional space, there is spatial consistency between the sample depth image and the historical sample depth image. The sample depth image can be transformed into the first depth image (expressed as ), and transform the historical sample depth image into a second depth image (denoted as D _b ') by using the sample pose parameters and the internal parameters of the endoscope. Wherein, the first depth image can be understood as converting the sample depth image into a depth image obtained by observing content in the sample tissue image from the perspective of collecting historical sample tissue images through attitude transformation. The second depth image can be understood as a depth image obtained by observing content in the sample tissue image from the perspective of collecting the historical sample tissue image by interpolating the historical sample depth image.

The consistency loss is then determined based on the first depth image and the second depth image. That is, the consistency loss can reflect the difference between the first depth image and the second depth image. Through training, consistency can be propagated to multiple sample depth images, which also ensures the scale consistency of multiple sample depth images, which is equivalent to smoothing multiple sample depth images to ensure spatial consistency. For example, the consistency loss can be determined by formula 3:

where L _G represents the consistency loss, represents the depth of p in the first depth image, and D' _b (p) represents the depth of p in the second depth image.

Step C7, determining the target loss according to the photometric loss, smoothing loss and consistency loss.

Exemplarily, the target loss can be determined from photometric loss, smoothness loss and consistency loss. For example, the weighted sum of photometric loss, smoothing loss and consistency loss can be obtained by formula 4 to obtain the target loss:

L=αL _p +βL _s +γL _G Formula 4

Wherein, α, β, and γ are weights corresponding to photometric loss, smoothing loss, and consistency loss, respectively, where α can be 0.7, β can be 0.7, and γ can be 0.3.

In yet another implementation, step C2 may include:

Luminosity loss is determined according to the sample tissue image, the interpolated tissue image, and the structural similarity between the sample tissue image and the interpolated tissue image.

For example, when the endoscope collects sample tissue images and historical sample tissue images, the lighting conditions may change. Therefore, SSIM (English: Structural Similarity, Chinese: Structural Similarity) can be introduced to determine the photometric loss, so as to avoid the interference of photometric loss due to changes in lighting conditions. SSIM can reflect the similarity of local structures. The improved photometric loss can be determined by Equation 5:

Among them, _λ1 and _λ2 represent the preset weights respectively, and SSIM(p) represents the pixel-by-pixel SSIM between the sample tissue image and the interpolated tissue image. Wherein, λ ₁ can be 0.7, and λ ₂ can be 0.3.

Further, the pixel-by-pixel SSIM between the sample tissue image and the interpolated tissue image can be determined by formula 6:

Among them, x represents the image block centered on p in the sample tissue image (the size can be 3*3), and y represents the image block centered on p in the interpolation tissue image is an image block of the same size in the center, τ _x represents the average value of pixel values in x, τ _y represents the average value of pixel values in y, σ _x represents the standard deviation of pixel values in x, and σ _y represents the standard deviation of pixel values in y. _ε1 and _ε2 represent preset constants, _ε1 may be, for example, 0.0001, _{and ε2} may be, for example, 0.0009.

Fig. 11 is a flowchart of another endoscopic image processing method shown according to an exemplary embodiment. As shown in Fig. 11, after step 105, the method may further include:

Step 106, outputting the ratio of the blind area, and sending a prompt message when the ratio of the blind area is greater than or equal to a preset ratio threshold, the prompt message is used to indicate that there is a risk of missed detection.

For example, after the blind area ratio is determined, the blind area ratio can be output, for example, the blind area ratio can be displayed in real time on a display interface for displaying tissue images, so as to display the inspection range during endoscopy in real time. Further, when the proportion of the blind spot is greater than or equal to a preset ratio threshold (for example, 20%), a prompt message can be sent to remind the doctor that there is a large blind spot in the current field of view of the endoscope, and there is a risk of missed detection. The presentation form of the prompt information may include: at least one of a text form, an image form, and a sound form. For example, the prompt information can be text or image prompts such as "the current risk of missed detection is high", "please re-examine", "please perform back-up", etc., and the prompt information can also be voice prompts, beeps with a specified frequency, or alarm sounds. In this way, the doctor can adjust the direction of the endoscope according to the prompt information, or execute the withdrawal of the endoscope, or re-examine. Thus, the proportion of blind spots can be monitored in real time during the endoscopic examination by the doctor, and a prompt can be given when the proportion of blind spots is large, thereby effectively avoiding missed detection and ensuring the effectiveness of endoscopic examination.

Fig. 12 is a block diagram of an endoscopic image processing device according to an exemplary embodiment. As shown in Fig. 12, the device 200 may include:

The acquisition module 201 is configured to acquire a tissue image set collected by the endoscope in the tissue to be measured, and the tissue image set includes a plurality of tissue images arranged according to the acquisition time.

The positioning module 202 is configured to determine the depth image and pose parameters corresponding to each tissue image according to the tissue image set.

The trajectory determination module 203 is configured to determine the movement trajectory of the endoscope according to the posture parameters corresponding to each tissue image, and the movement trajectory includes the position and angle of the endoscope when each tissue image is collected.

The contour determination module 204 is configured to determine the contour of the tissue to be measured according to the depth image corresponding to each tissue image.

The processing module 205 is configured to determine the proportion of the blind area during the endoscopic inspection process according to the motion track and the outline of the tissue to be measured.

In an application scenario, the positioning module 202 can be used for:

In another application scenario, the attitude parameters may include a rotation matrix and a translation vector, and the motion trajectory may include the position and angle of the endoscope when capturing each tissue image. Correspondingly, the trajectory determination module 203 can be used for:

Fig. 13 is a block diagram of another endoscopic image processing device shown according to an exemplary embodiment. As shown in Fig. 13 , the contour determination module 204 may include:

The centerline determination sub-module 2041 is configured to determine the centerline of the tissue to be measured according to the depth image corresponding to each tissue image.

The contour determination sub-module 2042 is configured to determine the contour of the tissue to be measured according to the centerline of the tissue to be measured.

Fig. 14 is a block diagram of another endoscopic image processing device according to an exemplary embodiment. As shown in Fig. 14, the processing module 205 may include:

The field of view determination sub-module 2051 is configured to determine the field of view corresponding to the tissue image according to the position and angle of the endoscope when collecting each tissue image, and the viewing angle of the endoscope.

The total visual field determination sub-module 2052 is configured to determine the total visual field area according to the visual field area corresponding to each tissue image.

The blind area determination sub-module 2053 is configured to determine the proportion of the blind area according to the total visual field area and the outline of the tissue to be measured.

In one implementation, the field of view determining submodule 2051 can be used to implement the following steps:

In an implementation manner, the positioning model includes: a depth sub-model and an attitude sub-model. The positioning module 202 can be used for:

The tissue image is input into the depth sub-model to obtain a depth image corresponding to the tissue image output by the depth sub-model. The tissue image and the corresponding historical tissue image are input into the attitude sub-model, so as to obtain the attitude parameters corresponding to the tissue image output by the attitude sub-model.

In another implementation, the localization model is trained by the following steps:

In yet another implementation, the implementation of step C may include:

In yet another implementation, step C2 may include:

Fig. 15 is a block diagram of another endoscopic image processing device according to an exemplary embodiment. As shown in Fig. 15, the device 200 may also include:

The prompting module 206 is configured to output the blind area ratio after determining the blind area ratio during the endoscopic examination according to the motion track and the outline of the tissue to be measured, and send a prompt message when the blind area ratio is greater than or equal to a preset ratio threshold, and the prompt information is used to indicate that there is a risk of missed detection.

Regarding the apparatus in the foregoing embodiments, the specific manner in which each module executes operations has been described in detail in the embodiments related to the method, and will not be described in detail here.

To sum up, in the present disclosure, firstly, the tissue images collected by the endoscope in the tissue to be measured are acquired according to multiple acquisition moments. Then, according to the tissue image set, the depth image and pose parameters corresponding to each tissue image are determined. Then, according to the posture parameters corresponding to each tissue image, the motion trajectory of the endoscope is determined, and according to the depth image corresponding to each tissue image, the contour of the tissue to be measured is determined. Finally, according to the motion trajectory and the outline of the tissue to be measured, the proportion of the blind area during the endoscopic examination is determined. The present disclosure is based on the depth image corresponding to the tissue image and the attitude parameter Number, determine the trajectory of the endoscope and the outline of the tissue to be tested, and determine the proportion of the blind area in the inspection process, which can monitor the inspection range, effectively avoid missed inspections, and ensure the effectiveness of endoscopic inspection.

Referring now to FIG. 16 , it shows a schematic structural diagram of an electronic device (for example, the executive body of the embodiment of the present disclosure, which may be a terminal device or a server) 300 suitable for implementing the embodiments of the present disclosure. The terminal devices in the embodiments of the present disclosure may include, but are not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Tablet Computers), PMPs (Portable Multimedia Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. The electronic device shown in FIG. 16 is only an example, and should not limit the functions and application scope of the embodiments of the present disclosure.

As shown in FIG. 16 , an electronic device 300 may include a processing device (such as a central processing unit, a graphics processing unit, etc.) 301, which may perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage device 308 into a random access memory (RAM) 303. In the RAM 303, various programs and data necessary for the operation of the electronic device 300 are also stored. The processing device 301, ROM 302, and RAM 303 are connected to each other through a bus 304. An input/output (I/O) interface 305 is also connected to the bus 304 .

Generally, the following devices may be connected to the I/O interface 305: an input device 306 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 307 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 308 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 309. The communication means 309 may allow the electronic device 300 to perform wireless or wired communication with other devices to exchange data. While FIG. 16 shows electronic device 300 having various means, it should be understood that implementing or having all of the means shown is not a requirement. More or fewer means may alternatively be implemented or provided.

In particular, according to an embodiment of the present disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product, which includes a computer program carried on a non-transitory computer readable medium, where the computer program includes program code for executing the method shown in the flowchart. In such an embodiment, the computer program may be downloaded and installed from a network via communication means 309, or from storage means 308, or from ROM 302. When the computer program is executed by the processing device 301, the above-mentioned functions defined in the methods of the embodiments of the present disclosure are performed.

It should be noted that the computer-readable medium mentioned above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any combination thereof. More specific examples of computer readable storage media may include, but are not limited to, electrical connections having one or more wires, portable computer diskettes, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), fiber optics, portable compact disk read only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing. In the present disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In the present disclosure, however, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave carrying computer-readable program code therein. Such propagated data signals may take many forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted by any appropriate medium, including but not limited to wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

In some embodiments, the terminal device and the server can communicate using any currently known or future-developed network protocols such as HTTP (HyperText Transfer Protocol, Hypertext Transfer Protocol), and can be interconnected with any form or medium of digital data communication (for example, a communication network). Examples of communication networks include local area networks ("LANs"), wide area networks ("WANs"), internetworks (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future developed networks.

The above-mentioned computer-readable medium may be included in the above-mentioned electronic device, or may exist independently without being incorporated into the electronic device.

The above-mentioned computer-readable medium carries one or more programs, and when the above-mentioned one or more programs are executed by the electronic device, the electronic device: acquires a set of tissue images collected by the endoscope in the tissue to be measured, and the set of tissue images includes a plurality of tissue images arranged according to the acquisition time; according to the set of tissue images, determine the depth image and attitude parameters corresponding to each of the tissue images; determine the motion trajectory of the endoscope according to the attitude parameters corresponding to each of the tissue images; Outline of the tissue to be measured to determine the blindness during the endoscopic examination area ratio.

Computer program code for carrying out operations of the present disclosure may be written in one or more programming languages, or combinations thereof, including but not limited to object-oriented programming languages—such as Java, Smalltalk, C++, and conventional procedural programming languages—such as the “C” language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In cases involving a remote computer, the remote computer can be connected to the user computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagram may represent a module, program segment, or portion of code that includes one or more executable instructions for implementing specified logical functions. It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or they may sometimes be executed in the reverse order, depending upon the functionality involved. It should also be noted that each block in the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, can be implemented by special purpose hardware-based systems that perform the specified functions or operations, or by combinations of special purpose hardware and computer instructions.

The modules involved in the embodiments described in the present disclosure may be implemented by software or by hardware. Wherein, the name of the module does not constitute a limitation of the module itself under certain circumstances, for example, the obtaining module may also be described as "a module for obtaining the tissue image set".

The functions described herein above may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: Field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), Application Specific Standard Products (ASSPs), System on Chips (SOCs), Complex Programmable Logic Devices (CPLDs), and the like.

In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include one or more wire-based electrical connections, a portable computer disk, a hard disk, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), optical fiber, compact disk read only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

According to one or more embodiments of the present disclosure, Example 1 provides a method for processing an endoscope image, comprising: acquiring a tissue image set collected by an endoscope in a tissue to be measured, the tissue image set including a plurality of tissue images arranged according to the acquisition time; according to the tissue image set, determining a depth image and an attitude parameter corresponding to each tissue image; determining a motion track of the endoscope according to the attitude parameter corresponding to each tissue image; A blind zone ratio during the endoscopy procedure is determined.

According to one or more embodiments of the present disclosure, Example 2 provides the method of Example 1. According to the tissue image set, determining the depth image and attitude parameters corresponding to each tissue image includes: sequentially according to each tissue image and the historical tissue image corresponding to the tissue image, and using a pre-trained positioning model to determine the depth image and attitude parameters corresponding to the tissue image. The acquisition time of the historical tissue image is before the acquisition time of the tissue image.

According to one or more embodiments of the present disclosure, Example 3 provides the method of Example 1, the attitude parameters include a rotation matrix and translation vector, and the motion trajectory includes the position and angle of the endoscope when acquiring each of the tissue images; determining the movement trajectory of the endoscope according to the attitude parameters corresponding to each of the tissue images includes: determining the position and angle of the endoscope when acquiring the tissue image according to the rotation matrix and translation vector corresponding to each of the tissue images, and the position and angle of the endoscope when acquiring the historical tissue image corresponding to the tissue image.

According to one or more embodiments of the present disclosure, Example 4 provides the method of Example 1. The determining the contour of the tissue to be measured according to the depth image corresponding to each of the tissue images includes: determining the centerline of the tissue to be measured according to the depth image corresponding to each of the tissue images; and determining the contour of the tissue to be measured according to the centerline of the tissue to be measured.

According to one or more embodiments of the present disclosure, Example 5 provides the method of Example 1. The determining the blind area ratio during the endoscopic examination according to the motion track and the outline of the tissue to be measured includes: determining the field of view area corresponding to the tissue image according to the position and angle of the endoscope when collecting each tissue image and the viewing angle of the endoscope; determining the total field of view area according to the field of view area corresponding to each tissue image; and determining the blind area ratio according to the total field of view area and the outline of the tissue to be tested.

According to one or more embodiments of the present disclosure, Example 6 provides the method of Example 5. According to the position and angle of the endoscope when acquiring each of the tissue images, and the viewing angle of the endoscope, determining the field of view corresponding to the tissue image includes: according to the attitude parameters corresponding to each of the tissue images, converting the position of the endoscope when acquiring the tissue images into a center position corresponding to the center line of the tissue to be measured; Determine the central viewing angle corresponding to the central position; determine the maximum visual field area corresponding to the central position; determine the visual field area corresponding to the tissue image according to the central viewing angle and the maximum visual field area.

According to one or more embodiments of the present disclosure, Example 7 provides the method of Example 2, wherein the positioning model includes: a depth sub-model and a pose sub-model; determining the depth image and pose parameters corresponding to the tissue image through the pre-trained positioning model according to each tissue image and the historical tissue image corresponding to the tissue image in turn, including: inputting the tissue image into the depth sub-model to obtain the depth image corresponding to the tissue image output by the depth sub-model; inputting the tissue image and the corresponding historical tissue image into the pose sub-model to obtain the pose corresponding to the tissue image output by the pose sub-model parameters.

According to one or more embodiments of the present disclosure, Example 8 provides the method of Example 7. The positioning model is obtained through the following steps of training: inputting the sample tissue image into the depth sub-model to obtain the sample depth image corresponding to the sample tissue image, and inputting the historical sample tissue image into the depth sub-model to obtain the historical sample depth image corresponding to the historical sample tissue image, the historical sample tissue image is an image collected before the sample tissue image; The sample posture parameters corresponding to the image and the internal parameters of the endoscope for collecting the sample tissue image, the internal parameters of the endoscope include focal length and translation size; according to the internal parameters of the endoscope, the sample depth image, the historical sample depth image and the sample posture parameters, determine the target loss; aiming at reducing the target loss, use the back propagation algorithm to train the positioning model.

According to one or more embodiments of the present disclosure, Example 9 provides the method of Example 8. The determining target loss according to the internal parameters of the endoscope, the sample depth image, the historical sample depth image, and the sample pose parameters includes: performing interpolation on the historical sample tissue image according to the sample depth image, the sample pose parameter, and the endoscope internal parameters to obtain an interpolated tissue image; determining a photometric loss according to the sample tissue image and the interpolated tissue image; transforming the sample depth image into a first depth image according to the sample pose parameter and the endoscope internal parameter; transforming the historical sample depth image into a second depth image according to the sample pose parameter and the endoscope internal parameter; determining a consistency loss according to the first depth image and the second depth image; determining the target loss according to the photometric loss, the smoothing loss, and the consistency loss.

According to one or more embodiments of the present disclosure, Example 10 provides the method of Example 9, the determining the photometric loss according to the sample tissue image and the interpolated tissue image includes: determining the photometric loss according to the sample tissue image, the interpolated tissue image, and the structural similarity between the sample tissue image and the interpolated tissue image.

According to one or more embodiments of the present disclosure, Example 11 provides the methods of Examples 1 to 10. After determining the blind area ratio during the endoscopic inspection according to the motion track and the contour of the tissue to be tested, the method further includes: outputting the blind area ratio, and sending a prompt message when the blind area ratio is greater than or equal to a preset ratio threshold, and the prompt information is used to indicate that there is a risk of missed detection.

According to one or more embodiments of the present disclosure, Example 12 provides an endoscope image processing device, including: an acquisition module, used to acquire a tissue image set collected by an endoscope in a tissue to be measured, the tissue image set including a plurality of tissue images arranged according to the acquisition time; a positioning module, used to determine the depth image and posture parameters corresponding to each of the tissue images according to the tissue image set; a trajectory determination module, used to determine the movement trajectory of the endoscope according to the posture parameters corresponding to each of the tissue images; The outline of the tissue to be measured; a processing module, configured to determine a blind area ratio during the endoscopic inspection process according to the motion track and the outline of the tissue to be measured.

According to one or more embodiments of the present disclosure, Example 13 provides a computer-readable medium on which a computer program is stored, and when the program is executed by a processing device, the steps of the methods described in Example 1 to Example 11 are implemented.

According to one or more embodiments of the present disclosure, Example 14 provides an electronic device, including: a storage device, on which a computer program is stored; a processing device, configured to execute the computer program in the storage device, so as to implement the steps of the methods described in Example 1 to Example 11.

The above description is only a preferred embodiment of the present disclosure and an illustration of the applied technical principles. Those skilled in the art should understand that the disclosure scope involved in this disclosure is not limited to the technical solutions formed by the specific combination of the above technical features, but also covers other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in this disclosure.

In addition, while operations are depicted in a particular order, this should not be understood as requiring that the operations be performed in the particular order shown or performed in sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Likewise, while the above discussion contains several specific implementation details, these should not be construed as limitations on the scope of the disclosure. Certain features that are described in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination.

Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are merely example forms of implementing the claims. Regarding the apparatus in the foregoing embodiments, the specific manner in which each module executes operations has been described in detail in the embodiments related to the method, and will not be described in detail here.

Claims

A method for processing endoscopic images, characterized in that the method comprises:

Obtaining a set of tissue images collected by the endoscope in the tissue to be tested, the set of tissue images including a plurality of tissue images arranged according to the acquisition time;

Determining a depth image and a pose parameter corresponding to each of the tissue images according to the tissue image set;

determining the motion trajectory of the endoscope according to the posture parameters corresponding to each of the tissue images;

determining the contour of the tissue to be measured according to the depth image corresponding to each of the tissue images;

According to the motion trajectory and the outline of the tissue to be measured, the proportion of the blind area during the endoscopic examination is determined.
The method according to claim 1, characterized in that, according to the set of tissue images, determining the depth image and attitude parameters corresponding to each of the tissue images comprises:

In turn, according to each tissue image and the historical tissue image corresponding to the tissue image, the depth image and attitude parameters corresponding to the tissue image are determined through the pre-trained positioning model, and the collection time of the historical tissue image is before the collection time of the tissue image.
The method according to claim 1, wherein the posture parameters include a rotation matrix and a translation vector, and the motion trajectory includes the position and angle of the endoscope when acquiring each of the tissue images; and determining the motion trajectory of the endoscope according to the posture parameters corresponding to each of the tissue images includes:

The position and angle of the endoscope when acquiring the tissue image are determined according to the rotation matrix and translation vector corresponding to each tissue image, and the position and angle of the endoscope when acquiring the historical tissue image corresponding to the tissue image.
The method according to claim 1, wherein the determining the contour of the tissue to be measured according to the depth image corresponding to each of the tissue images comprises:

determining the centerline of the tissue to be measured according to the depth image corresponding to each of the tissue images;

The contour of the tissue to be measured is determined according to the centerline of the tissue to be measured.
The method according to claim 1, characterized in that, according to the motion trajectory and the outline of the tissue to be measured, determining the proportion of the blind area during the endoscopic examination comprises:

According to the position and angle of the endoscope when collecting each of the tissue images, and the viewing angle of the endoscope, determine the field of view corresponding to the tissue image;

Determine the total field of view according to the field of view corresponding to each of the tissue images;

The blind area ratio is determined according to the total field of view area and the outline of the tissue to be measured.
The method according to claim 5, wherein, according to the position and angle of the endoscope when collecting each of the tissue images, and the viewing angle of the endoscope, determining the field of view corresponding to the tissue image includes:

Converting the position of the endoscope when acquiring the tissue image into a center position corresponding to the center line of the tissue to be measured according to the posture parameters corresponding to each of the tissue images;

determining a central viewing angle corresponding to the central position according to the posture parameter corresponding to the tissue image, the viewing angle of the endoscope, and the angle of the endoscope when collecting the tissue image;

determining the maximum field of view area corresponding to the central position;

Determine the visual field area corresponding to the tissue image according to the central viewing angle and the maximum visual field area.
The method according to claim 2, wherein the positioning model comprises: a depth sub-model and an attitude sub-model;

According to each tissue image and the historical tissue image corresponding to the tissue image in turn, the depth image and attitude parameters corresponding to the tissue image are determined through the pre-trained positioning model, including:

inputting the tissue image into the depth sub-model to obtain a depth image corresponding to the tissue image output by the depth sub-model;

The tissue image and the corresponding historical tissue image are input into the attitude sub-model, so as to obtain the attitude parameters corresponding to the tissue image output by the attitude sub-model.
The method according to claim 7, wherein the positioning model is obtained through the following steps of training:

inputting a sample tissue image into the depth sub-model to obtain a sample depth image corresponding to the sample tissue image, and inputting a historical sample tissue image into the depth sub-model to obtain a historical sample depth image corresponding to the historical sample tissue image, where the historical sample tissue image is an image collected before the sample tissue image;

inputting the sample tissue image and the historical sample tissue image into the pose sub-model to obtain the pose sub-model output Wherein, the sample posture parameter corresponding to the sample tissue image and the internal parameters of the endoscope used to acquire the sample tissue image, the internal endoscope parameters include focal length and translation size;

determining target loss based on the endoscope internal parameters, the sample depth image, the historical sample depth image, and the sample pose parameters;

Aiming at reducing the target loss, the positioning model is trained using a backpropagation algorithm.
The method according to claim 8, wherein the determining target loss according to the internal parameters of the endoscope, the sample depth image, the historical sample depth image and the sample pose parameters comprises:

Interpolating the historical sample tissue image according to the sample depth image, the sample posture parameter and the endoscope internal parameter to obtain an interpolated tissue image;

determining photometric loss based on the sample tissue image and the interpolated tissue image;

determining a smoothing loss based on the gradient of the sample depth image and the gradient of the sample tissue image;

transforming the sample depth image into a first depth image according to the sample pose parameter and the endoscope internal parameter;

transforming the historical sample depth image into a second depth image according to the sample pose parameter and the endoscope internal parameter;

determining a consistency loss from the first depth image and the second depth image;

The target loss is determined from the photometric loss, the smoothing loss and the consistency loss.
The method according to claim 9, wherein the determining the photometric loss according to the sample tissue image and the interpolated tissue image comprises:

The photometric loss is determined according to the sample tissue image, the interpolated tissue image, and the structural similarity between the sample tissue image and the interpolated tissue image.
The method according to any one of claims 1-10, characterized in that, after determining the blind area ratio in the endoscopic examination process according to the motion trajectory and the outline of the tissue to be measured, the method further comprises:

The blind area ratio is output, and when the blind area ratio is greater than or equal to a preset ratio threshold, a prompt message is sent, and the prompt information is used to indicate that there is a risk of missed detection.
A device for processing endoscopic images, characterized in that the device comprises:

An acquisition module, configured to acquire a tissue image set collected by the endoscope in the tissue to be measured, the tissue image set including a plurality of tissue images arranged according to the acquisition time;

A positioning module, configured to determine a depth image and a pose parameter corresponding to each of the tissue images according to the tissue image set;

A trajectory determination module, configured to determine the movement trajectory of the endoscope according to the posture parameters corresponding to each of the tissue images;

A contour determination module, configured to determine the contour of the tissue to be measured according to the depth image corresponding to each of the tissue images;

A processing module, configured to determine a blind area ratio during the endoscopic inspection process according to the motion track and the contour of the tissue to be measured.
A computer-readable medium, on which a computer program is stored, characterized in that, when the program is executed by a processing device, the steps of the method described in any one of claims 1-11 are implemented.
An electronic device, characterized in that it comprises:

a storage device on which a computer program is stored;

A processing device, configured to execute the computer program in the storage device, so as to realize the steps of the method according to any one of claims 1-11.