WO2023024701A1

WO2023024701A1 - Panoramic endoscope and image processing method thereof

Info

Publication number: WO2023024701A1
Application number: PCT/CN2022/102948
Authority: WO
Inventors: 杜武华; 徐健玮; 佴广金; 钱大宏; 黄显峰
Original assignee: 山东威高宏瑞医学科技有限公司
Priority date: 2021-08-23
Filing date: 2022-06-30
Publication date: 2023-03-02
Also published as: CN115708658A

Abstract

A panoramic endoscope (1), comprising a main camera (110) disposed directly in front of a distal end (10) of the endoscope; and a plurality of secondary cameras disposed on a side wall of the distal end (10), wherein the field of view of the main camera (110) and the field of view of the plurality of secondary cameras jointly cover a spherical field of view. The panoramic endoscope (1) uses a plurality of secondary cameras to cooperate with the main camera (110) to form a spherical field of view to observe positions directly in front of, beside and behind the lens simultaneously, thereby overcoming the blind area of vision caused by occlusion of the intestinal wall, and reducing missed diagnosis of polyps. The polyps appearing beside and behind the lens can be collected by the secondary cameras in time without the need for an operator to manipulate a snake bone to make the endoscopic lens bend backward and observe, thereby simplifying endoscope insertion and withdrawal operations.

Description

Panoramic endoscope and its image processing method

technical field

The invention belongs to the technical field of medical detection, and in particular relates to a panoramic endoscope and a medical image recognition monitoring system.

Background technique

Endoscopic surgery is the search and screening of the interior of the human body by means of a medical device (endoscope) within hollow organs or cavities of the body for medical purposes. Unlike most other medical imaging devices, endoscopes are inserted directly into organs and typically use optical cameras at visible or near-visible frequencies, such as infrared beams, to produce images and video frames from inside the organ. Typically, endoscopic surgery is used to examine the organ being tested for suspicious localized areas, such as polyps, tumors, or evidence of cancer cells.

Colorectal cancer is the second most lethal cancer in China and the third most lethal cancer in the world. Early detection and removal of colorectal polyps is crucial to reduce the mortality of colorectal cancer patients. A colonoscopy procedure is an endoscopic examination of the distal part of the large intestine (large intestine) and small intestine with an optical camera (usually with a CCD or CMOS camera) on a fiber optic or flexible tube passed through the anus. The gold standard for colorectal cancer screening, colonoscopy can reduce the risk of death from colorectal cancer through early detection of tumors and removal of precancerous lesions. It provides visual diagnosis (eg, ulcers, polyps) and provides an opportunity to biopsy or resect lesions suspected of colorectal cancer.

The main interest of the general public in colonoscopy is the removal of polyps smaller than 1 millimeter (mm) or smaller. Once the polyps are removed, they can be studied with the help of a microscope to determine if they are precancerous. It takes 15 years or less for polyps to turn into cancer. For every 1.0% increase in the adenoma detection rate (ADR), the risk of colorectal cancer will decrease by 3.0%; on the contrary, the missed diagnosis of adenoma may lead to further development of the tumor and delay the timing of treatment: the error rate of polyp detection is estimated to be 4% -12%, but recent clinical studies show that the false detection rate may be as high as 25%. False detection of colorectal cancer can lead to subsequent metastatic colon cancer, with a survival rate of less than 10%. The detection rate of adenoma largely depends on the quality of bowel preparation and the experience of doctors, so there is strong subjectivity. Two problems were faced during the endoscopy: (1) a polyp appeared in the field of view of the endoscope, but the doctor did not find it; (2) a polyp never appeared in the field of view of the endoscope.

Since the introduction of the concept of deep neural networks, many studies related to AI-assisted endoscopic analysis have been reported. However, these mainly focus on the auxiliary polyp detection task and are only relevant to problem (1). The current computer-aided polyp detection system can only help doctors find those polyps that appear in the endoscopic field of view, but for those blind spots caused by the doctor's operating experience and the complex intestinal environment, some polyps have never been detected. The current computer-aided detection system cannot do anything about the missed diagnosis caused by the appearance in the endoscopic field of view. Therefore, how to realize the quality control of the blind area of endoscopic examination through new hardware design and computer-aided system is a problem that we need to consider.

Contents of the invention

On the one hand, the purpose of the present application is to provide a panoramic endoscope to help doctors find polyps that are not in the field of vision due to intestinal wall folds.

The application discloses a panoramic endoscope, comprising:

The main camera is arranged directly in front of the tip of the endoscope;

A plurality of sub-cameras are arranged on the side wall of the tip;

Wherein, the field of view of the main camera and the fields of view of the plurality of secondary cameras together cover a spherical field of view.

In a preferred example, it also includes,

a main lighting lens, arranged directly in front of the tip, and connected to a light source through a light guide, for providing lighting for the main camera; and

A secondary lighting lens is arranged on the side wall of the tip and connected to the light source through a light guide for providing lighting for the secondary camera.

In a preferred example, the number of pixels of the primary camera is higher than that of the secondary camera.

In a preferred example, the viewing angle of the spherical field of view is not less than 330 degrees.

The present application also discloses a panoramic endoscope image processing method, including:

Collect images synchronously from the main camera and multiple sub-cameras of the panoramic endoscope as described above;

Carrying out polyp detection on images synchronously collected from the main camera and a plurality of sub-cameras;

If a polyp is detected, a polyp alarm is issued, and a prompt message of the camera to which the image of the detected polyp belongs is output.

In a preferred example, the step of performing polyp detection on the images synchronously collected from the main camera and multiple auxiliary cameras includes: center enhancement, denoising, and color rendering processing to improve image quality and recognition.

In a preferred example, when a polyp is detected, the alarm method includes at least one of the following: highlighting, framing, flashing or audio prompts on the screen of the external monitor.

In a preferred example, the images collected by the main camera and the auxiliary camera are shot in a white light mode.

In a preferred example, the captured images are input into a trained target detection model to detect whether polyps appear in real time;

The target detection model includes a first encoder and a detector;

The first encoder is configured to perform feature extraction on an image obtained in white light mode to obtain a feature map;

The detector is configured to perform regression on the input feature map to obtain the coordinates of the location of the polyp.

In a preferred example, polyp detection is performed through a convolutional neural network, and the extracted target features include at least one of the following: blood vessel color features, glandular tube features, and edge features.

The panoramic endoscope and medical image recognition monitoring system of the present application have at least the following technical effects:

1. Multiple auxiliary cameras cooperate with the main camera to form a spherical field of view, and observe the front, side and rear of the lens at the same time, which overcomes the blind area of vision caused by the obstruction of the intestinal wall and reduces the missed diagnosis of polyps;

2. For the polyps that appear on the side and rear of the lens, they can be captured by the auxiliary camera in time, without the need for the operator to manipulate the snake bone to make the endoscopic lens bend backward and observe, which simplifies the operation of entering or withdrawing the lens.

A large number of technical features are recorded in the description of the application, which are distributed in various technical solutions. If all possible combinations of technical features (ie technical solutions) of the application are to be listed, the description will be too lengthy. In order to avoid this problem, the technical features disclosed in the summary of the invention above, the technical features disclosed in the following embodiments and examples, and the technical features disclosed in the drawings can be freely combined with each other to form a Various new technical solutions (these technical solutions should be deemed to have been recorded in this specification), unless the combination of such technical features is technically infeasible. For example, feature A+B+C is disclosed in one example, and feature A+B+D+E is disclosed in another example, and features C and D are equivalent technical means that play the same role. It can be used as soon as it is used, and it is impossible to use it at the same time. Feature E can be combined with feature C technically. Then, the solution of A+B+C+D should not be regarded as recorded because it is technically infeasible, and A+B+ The C+E scheme should be considered as documented.

Description of drawings

Fig. 1 is a schematic diagram of a panoramic endoscope according to the present application;

Fig. 2 is a schematic diagram of a light source of a panoramic endoscope according to an embodiment of the present application;

Fig. 3 is a working schematic diagram of a panoramic endoscope according to an embodiment of the present application;

Fig. 4 is a schematic view of a panoramic endoscope according to an embodiment of the present application;

FIG. 5 is a schematic flow diagram of a medical image recognition monitoring system according to an embodiment of the present application;

Fig. 6 is a structural schematic diagram of a panoramic endoscope including a structured light path according to an embodiment of the present application;

7 is a schematic structural diagram of a target detection model according to the method for measuring the absolute size of a lesion under a panoramic endoscope according to the present application;

Fig. 8 is a schematic structural diagram of a U-NET network based on an encoder-decoder structure according to the absolute size measurement method of a panoramic endoscope lesion according to the present application;

Fig. 9 is a schematic diagram of a lesion size calculation model according to the method for measuring the absolute size of a lesion under a panoramic endoscope according to the present application.

Explanation of reference signs:

1- Panoramic endoscope

10 - Apex

110-main camera

111-Main lighting lens

112-dominant beam

1121-leading beam interface

120-The first camera

121-the first lighting lens

122-Slave light guide

1221-Slave light guide interface

130-Second camera

131-Second lighting lens

140-The third camera

141-The third lighting lens

2- light box

201-light source

300-structured light path

31 - Structured light projector

32-fiber

33-coupler

34-Structured light generator

Detailed ways

In the following description, many technical details are proposed in order to enable readers to better understand the application. However, those skilled in the art can understand that the technical solutions claimed in this application can be realized even without these technical details and various changes and modifications based on the following implementation modes.

the term

As described herein, "panoramic endoscope", "endoscope" and "endoscope" can be used interchangeably, and all refer to the panoramic endoscope of the present invention.

Spherical field of view: The spherical field of view mentioned in this application refers to a spherical or approximately spherical field of view formed by taking a point near the main camera and the sub-camera on the tip 10 of the panoramic endoscope as the center of the sphere. Since the spherical shape can be composed of countless spherical sectors, the spherical field of view in this application can also be composed of a number of specified spherical sector fields of view.

A first embodiment of the present invention relates to a panoramic endoscope. As shown in FIG. 1 , more specifically, a panoramic endoscope 1 . The panoramic endoscope 1 has a flexible, rigid tip 10, optionally formed from a snake bone. A plurality of cameras are set on the tip 10, specifically, a main camera 110 is included, and the shooting direction of the main camera 110 is towards the advancing direction of the panoramic endoscope 1; and evenly distributed, there are optionally 2-6 sub-cameras, preferably 3 (as shown in FIG. 1 ): a first sub-camera 120 , a second sub-camera 130 , and a third sub-camera 140 .

As shown in FIG. 4 , the field of view of the main camera and the fields of view of the plurality of secondary cameras together cover a spherical field of view. Referring to the left half of Figure 4, in one embodiment, when only the main camera exists, the field of view is a spherical sector with a vertex angle of 140 degrees (the main camera can be regarded as the center of the sphere, and the vertex is located at the center of the sphere) when When there are three sub-cameras, the field of view of each sub-camera (for example, the first sub-camera 120) covers a spherical sector with an apex angle of 120 degrees, so that the combined field of view of the main camera and the sub-camera can cover a field of view not less than 330 degrees. degrees of spherical field of view.

The main camera 110 is surrounded by a main lighting lens 111, and one or more secondary lighting lenses are provided around each secondary camera to provide side lighting, such as the first secondary lighting lens 121, the second secondary lighting lens 131, the first secondary lighting lens 131, and the like in Figure 2. Three pairs of lighting lenses 141 . Wherein the main lighting lens and the secondary lighting lens are respectively connected to the leading light beam interface and the secondary light guiding interface of the light box through the main light beam and the secondary light guiding light interface. Optionally, the light source in the light box can be one of xenon lamp, laser or LED, and can adjust the light intensity and switch the lighting spectrum. The leading light beam interface can provide stronger light intensity and NBI or multi-spectral switching, while the light intensity of the secondary light beam is weaker, only providing auxiliary lighting, and does not need the function of spectrum switching.

In one embodiment, the slave guide beam is generated by splitting the master beam. The light splitting device may be a light splitter. According to the corresponding proportion of optical power, the optical splitter distributes the light of the leading beam to the link of the secondary guiding beam.

The quality of the main camera and the secondary camera are different. The main camera 110 is a high-definition camera, optionally 2 million pixels, with a resolution of 1920*1080; the three secondary cameras are standard-definition cameras, smaller in size, optionally 10-2 million pixels, preferably 160,000 pixels, The resolution is 480*320. With the development of technology, the size of the camera will become smaller and lower, and the cost will be lower and lower, and the resolution of the main camera and the secondary camera can be further improved. However, the resolution of the main camera is usually higher than that of the secondary camera.

As shown in Figure 3, when in use, the apex 10 is inserted into the image acquisition device group and electrically connected to an external processor for real-time acquisition of multiple images with a spherical viewing angle not less than 330°, and transmitted to the external image processor .

Another embodiment of the present invention relates to a panoramic endoscope image processing method.

In this example, the panoramic endoscope of the present invention is used to acquire images during a colon exam and to identify the location of different types of suspicious tissue, in this case polyps. Polyps can be: large/medium polyps, small polyps and most importantly flat polyps to detect. In addition to polyps, the system can identify other suspicious tissue anywhere in or on the sides of colonic folds. Perform online detection of polyps or other suspicious tissue during actual colonoscopy surgery, sigmoidoscopy, etc. In one embodiment, multiple images are acquired from at least one of the following positions of the patient's body: abdominal cavity, esophagus, stomach, nasal cavity, trachea, bronchi, uterine cavity, and vagina.

Step 501 of the method: synchronously collect images using a main camera and multiple secondary cameras of the panoramic endoscope.

Step 502: Perform polyp detection on the images synchronously collected from the main camera and multiple auxiliary cameras. In some embodiments, the images captured synchronously by the main camera and multiple sub-cameras need to undergo one or more of the following preprocessing: histogram improvement algorithm; adaptive enhancement of contrast, brightness and image frame according to predefined standards Color normalization; super-resolution improvement of image frames; unbalanced scaling of luma and color channels in image frames to achieve color frequency dominance, where each color channel is individually equalized and controlled to remove noise enhancement; applied signal-to-noise measurements and reduce noise or filter frames accordingly; verify that the image is in focus and filter out-of-focus frames; or any combination thereof.

Step 503: If a polyp is detected, a polyp alarm is issued, and a prompt message of the camera to which the image of the detected polyp belongs is output. In one embodiment, if a polyp is detected in an image captured by one of the secondary cameras (for example, the first secondary camera 120), the image of the first secondary camera 120 is output to the medical operator through an output device (for example, a display, etc.). Prompt information. When a polyp is detected, the alarming method includes at least one of the following: highlighting, framing, flashing or audio prompting on the screen of the external monitor. In the subsequent interaction with the I/O device, the operator can control the area where the main camera collects the alarm prompts to obtain images with higher resolution for observation.

Optionally, in one embodiment, the convolutional neural network model includes three models trained according to the qualified picture library, part library and part feature library, which are respectively used for whether the endoscopic image is qualified, part judgment, and part feature recognition and determination of the extent of the cancer. The model is Resnet50, developed in Python language, packaged into a RESTful API (RESTful network interface) and called by other modules.

The panoramic endoscope shoots in white light mode, and the captured images are input into a trained target detection model to detect the presence of polyps in real time.

If a polyp is detected, the polyp area is automatically segmented in real time using a deep neural network based on an encoder-decoder structure.

In one embodiment, the panoramic endoscope also has a structured light channel 300 , that is, another channel independent of the common illumination light channel, which can be used to measure the size of the lesion under the endoscope. As shown in FIG. 6 , the structured light channel 300 includes a structured light generator 34 , a coupler 33 , an optical fiber 32 and a structured light projector 31 .

The structured light generator 34 generates visible light with a wavelength between 400-700nm or the passband width of the generated light is less than 5% of the central wavelength value. The actual selection will be determined through experiments to minimize reflection and obtain the clearest structured light image .

The coupler 33 is configured to realize coupling between the light source 201 and the optical fiber 32; the structured light generator 34 is configured to cut in and out between the coupler 33 and the light source 201; one end of the optical fiber 32 is connected to the coupler 33, and the other end is connected to To the structured light projector 31; the main illumination lens 111 and the structured light projector 31 are arranged at the end of the endoscope. After passing through the coupler 33 and the structured light fiber 32 , the structured light is projected onto the tissue surface by the structured light projector 31 . After the structured light is projected onto the surface of the object to be measured, it is modulated by the height of the object to be measured. The modulated structured light is collected by the camera system and sent to the computer for analysis and calculation to obtain the three-dimensional data of the object to be measured.

Therefore, the illumination of the panoramic endoscope has the following two modes: structured light mode and white light mode;

When switching to the structured light mode, the illumination path of the illumination lens is blocked, and the structured light generator 34 cuts in between the coupler 33 and the light source 201 to generate structured light;

When switching to the white light mode, both the structured light channel 300 and the channel of the illumination lens provide general illumination light.

Afterwards, the absolute size measurement system of the lesion under the endoscope was switched to the structured light mode for structured light imaging.

Afterwards, the image processing system was used to analyze the collected structured light images, and the polyp size was calculated in combination with the results of polyp segmentation.

Optionally, in one embodiment, the target detection model includes a first encoder and a detector; the first encoder is configured to perform feature extraction on an image obtained in white light mode to obtain a feature map; the extracted target features include the following At least one: blood vessel color feature, glandular tube feature, edge feature. The detector is configured to regress on the input feature map to obtain the coordinates of where the polyps are located.

Optionally, in one embodiment, the deep neural network based on encoder-decoder structure includes:

A second encoder for input images, including convolutional layers, activation layers, pooling layers;

Decoder for output results, including convolution layer, activation layer, concatenation layer, upsampling layer;

The feature maps whose size difference is less than a predetermined threshold are fused together by cross-layer feature fusion between the second encoder and decoder.

Optionally, in one embodiment, for the target detection model shown in FIG. 7 , the image x under white light is used as input, which is input into an encoder for feature extraction, and the feature map obtained after feature extraction is obtained by regression of a detector The y-coordinate of where the polyp is located. The encoder and detector are composed of deep neural networks, including convolutional layers, downsampling layers, upsampling layers, pooling layers, batch normalization layers, activation layers, etc. Through the above-mentioned target detection model, when the system detects the presence of polyps, the system will prompt the doctor with a red box on the monitor to indicate the rectangular area where the polyps are located, and an alarm will sound. The system then performs fine segmentation of the polyp area.

The fine segmentation of the polyp region is based on a deep neural network with an encoder-decoder structure. The left half is the encoder, including convolutional layers, activation layers, and pooling layers. The right half is the decoder, including convolutional layers, activation layers, concatenation layers, and upsampling layers. The feature maps with similar sizes are fused together directly between the encoder and the decoder through cross-layer feature fusion, which can better extract context information and global information. As shown in Figure 8 is a U-NET network based on encoder-decoder structure.

Once the lesion is segmented, the system automatically switches or manually switches to the structured light imaging mode, and at the same time, the rear end of the lighting channel is closed by the blocking lens to prevent the passage of the lighting source. The structured light is projected onto the tissue surface through the above-mentioned structured light channel, and the deformation of the structured light image will be collected by the endoscope camera and sent to the image processing center for processing. After the collected structured light image is sent to the image processing center, the image processing center combines the segmentation result and the deformation of the structured light image to calculate the three-dimensional contour of the lesion through the phase shift method and give the lesion size.

Optionally, in one embodiment, as shown in FIG. 9 , a lesion size calculation model. P is the optical center of the projector, and C is the optical center of the camera. O is the intersection of the optical axis of the camera and the optical axis of the projector. Let the horizontal plane passing through point O be the reference X axis in the calculation. L1 and L2 are the distances from the optical center of the camera and the optical center of the projector to the X-axis, respectively. d is the distance from the optical center of the camera to the optical center of the projector along the X-axis direction. The A-B-O plane is a hypothetical virtual plane, and the hypothetical virtual plane is parallel to the connecting line PC between the optical center of the projector and the optical center of the camera. For a certain point Q on the surface of the object, the height Z of point Q relative to the reference plane can be obtained from the triangular relationship:

Since the plane-to-plane projection is a linear mapping, the projector pattern mapping also has a fixed period on the virtual plane ABO, and AB and

into a linear relationship.

Then calculate the absolute phase of the Q point

When the sinusoidal fringe image is projected onto a three-dimensional diffuse surface, the image of a point Q(x,y) observed from the camera can be expressed as:

Where A(x, y) is the background light intensity, B(x, y)/A(x, y) represents the contrast of the grating stripes,

is the phase value. For the N-step phase shift method, the phase difference between each step of sinusoidal grating phase shift and the projection phase of the previous step of sinusoidal grating phase shift is 2π/N, that is, the following equations:

where i=1,2,...,N. The system of equations has N equations and three unknown quantities, and it can be solved when N>=3. Solutions have to:

Through phase unwrapping, the absolute phase value of point Q (x, y) can be obtained

Will

Putting it into the above formula can get the height Z between the point Q and the virtual plane ABO.

By calibrating the camera parameters, the absolute coordinates (X, Y, Z) of point Q in the real world can be calculated according to Z, and the absolute coordinates (X, Y, Z) of each point on the collected structured light image can be calculated by calculating ), combined with the results of the above polyp segmentation, the three-dimensional size of the polyp can be obtained.

All documents mentioned in this application are considered to be included in the disclosure content of this application in their entirety so that they can be used as a basis for amendments when necessary. In addition, it should be understood that after reading the above disclosure of the present application, those skilled in the art may make various changes or modifications to the present application, and these equivalent forms also fall within the scope of protection claimed in the present application.

Claims

A panoramic endoscope is characterized in that it comprises:

The main camera is arranged directly in front of the tip of the endoscope;

A plurality of sub-cameras are arranged on the side wall of the tip;

Wherein, the field of view of the main camera and the fields of view of the plurality of secondary cameras together cover a spherical field of view.
The panoramic endoscope according to claim 1, further comprising:

a main lighting lens, arranged directly in front of the tip, and connected to a light source through a light guide, for providing lighting for the main camera; and

A secondary lighting lens is arranged on the side wall of the tip and connected to the light source through a light guide for providing lighting for the secondary camera.
The panoramic endoscope according to claim 1, wherein the number of pixels of the main camera is higher than that of the secondary camera.
The panoramic endoscope according to claim 1, wherein the viewing angle of the spherical field of view is not less than 330 degrees.
A panoramic endoscope image processing method, characterized in that it comprises:

Collect images synchronously from the main camera and a plurality of secondary cameras of the panoramic endoscope according to any one of claims 1-4;

Carrying out polyp detection on images synchronously collected from the main camera and a plurality of sub-cameras;

If a polyp is detected, a polyp alarm is issued, and a prompt message of the camera to which the image of the detected polyp belongs is output.
The panoramic endoscope image processing method according to claim 5, wherein the step of detecting polyps on images synchronously collected from the main camera and a plurality of secondary cameras comprises: central enhancement, denoising, and color rendering , to improve image quality and recognition.
The panoramic endoscope image processing method according to claim 5, wherein when a polyp is detected, the alarm mode includes at least one of the following: highlighting on the screen of an external monitor, adding a frame, and flashing or audio prompts.
The panoramic endoscope image processing method according to claim 5, wherein the images collected by the main camera and the secondary camera are taken in a white light mode.
The panoramic endoscope image processing method as claimed in claim 5, characterized in that, the captured image is input into a trained target detection model to detect whether polyps occur in real time;

The target detection model includes a first encoder and a detector;

The first encoder is configured to perform feature extraction on an image obtained in white light mode to obtain a feature map;

The detector is configured to perform regression on the input feature map to obtain the coordinates of the location of the polyp.
The panoramic endoscope image processing method according to claim 5, wherein the polyp detection is performed through a convolutional neural network, and the extracted target features include at least one of the following: blood vessel color features, glandular tube features, and edge features.