CN115381389A

CN115381389A - System and method for measuring absolute size of focus under endoscope

Info

Publication number: CN115381389A
Application number: CN202110567035.3A
Authority: CN
Inventors: 杜武华; 钱大宏; 徐健玮; 黄显峰; 蒋志新
Original assignee: Shandong Weigao Hongrui Medical Technology Co Ltd
Current assignee: Shandong Weigao Hongrui Medical Technology Co Ltd
Priority date: 2021-05-24
Filing date: 2021-05-24
Publication date: 2022-11-25

Abstract

The application relates to the field of endoscopes and discloses a system and a method for measuring the absolute size of a focus under an endoscope, wherein the method comprises the following steps: a light box, an illumination path, and a structured light path; a light source is arranged in the light box; the illumination path comprises a shielding lens, a first coupler, a first optical fiber and an illumination lens; the shutter lens is configured to be cut-in and cut-out between the first coupler and the light source; one end of the first optical fiber is connected to the first coupler, and the other end of the first optical fiber is connected to the illumination lens; the structured light pathway includes a structured light generator, a second coupler, a second optical fiber, and a structured light projector; the structured light generator is configured to be cut-in and cut-out between the second coupler and the light source; one end of the second optical fiber is connected to the second coupler, and the other end is connected to the structured light projector.

Description

System and method for measuring absolute size of focus under endoscope

Technical Field

The application relates to the field of endoscopes, in particular to a system and a method technology for measuring the absolute size of a focus under an endoscope.

Background

The front end of the traditional endoscope in the prior art mainly comprises an air and water supply nozzle, an objective lens, an illuminating lens and an instrument working channel. Wherein the illumination lens is used for illumination and the objective lens is used for collecting images. Typically, gastroscopes have two illumination points and enteroscopes have three illumination points. The light source is generated by the lamp box, passes through the light box to the optical fiber beam coupler, and is transmitted to the front end illuminating lens of the endoscope by the light guide beam for illumination.

In the application of endoscopic detection of polyps, accurate measurement of polyp size is important for the screening and monitoring of Colorectal cancer (CRC). Adenoma size is an important long-term predictor of CRC, subjective polyp size measurement is important for CRC risk stratification and inter-follow-up interval time, with shorter inter-follow-up intervals for adenomas greater than or equal to 1cm. Moreover, the size of the tumor under the colonoscope is closely related to the determination of the nature of the tumor and the selection of the treatment regimen. For polyps larger than 3cm, the endoscopic resection risk is high, and intestinal section resection is required; but for colonic polyps less than 3cm, they can be completely resected by endoscopic submucosal resection (EMR) or Endoscopic Submucosal Dissection (ESD), with a low incidence of complications. Even for polyps smaller than 3cm, the endoscopic treatment modalities for polyps of different sizes are different. Therefore, many guidelines use polyp size as an important indication in the formulation of endoscopic treatment for colonic polyps. However, there is no standard for determining the size of polyps under the endoscope, and in clinical work, the size of polyps is often determined by a physician only by visual inspection based on his own experience, which is sometimes far from the actual polyp size. Eichenseer et al compared the endoscopically estimated size of a polyp of 10-25 mm with a directly measured size of a resected tissue specimen by 15 endoscopists and found that the average rate of variation was 73.6% (13-127%), the error rate was 62.6% (0-91%), and overestimation was more common, resulting in providing inappropriate colorectal tumor screening recommendations for some patients.

In a study published in the Gastroenterology journal of 2016, polyps were also included which had both direct endoscopic judgment and post-resection pathology measurements and were completely resected. The authors compared polyp measurement methods for endoscopists and pathologists and analyzed the relevant factors in the endoscope or polyp that caused the erroneous measurements at the same time. In this study, post-resection polyps were immediately immersed in formalin and sent to the pathologist for further processing. The size of the pathology measured by a ruler with a minimum scale of mm is the gold standard. Polyps are divided into two groups, greater than 1cm and less than 1cm respectively. Polyps are also classified into two groups, overestimated (high endoscopic estimate) and underestimated (low endoscopic estimate), based on patient gender, polyp morphology, histology, and polyp location. During the study, 50 endoscopists completed 9146 colonoscopes with 12 endoscopists and resected 6067 polyps for a total of 1528 lesions from 1422 patients meeting inclusion criteria. These endoscopists have an average of 15 years of endoscopic experience. Researchers found that polyp estimates under endoscopes were mostly centered at 1cm and 2cm, with most estimates being higher than pathological values. The above findings suggest that endoscopists tend to estimate polyp size more rounded to larger measurements, which are more common in large polyps, and that larger estimates will shorten the follow-up interval. About half (46%) of polyps with an endoscopic estimate of greater than 1cm diameter are under 1cm in actual size, and the follow-up time for these patients may be shortened from the more modest 5-10 years to 3 years. Conversely, a small fraction of patients with low estimates receive inappropriate extension of follow-up time. Researchers have indicated that if the estimated size of a polyp is the only criterion to assess the progression of an adenoma, about 10% of patients will receive unreasonable follow-up.

Some current endoscopic polyp size measurement methods mainly involve direct visual observation by the endoscopist or estimation using some instruments as a reference. Such as by taking the size of the opening of the bioptome as a reference or by measuring through a scale on some instrument connecting line. But this may result in less accurate measurements due to the different distances of the tool and the lesion from the lens, the different sizes and shapes of the lesions displayed at different distances and at different angles.

There is a need for an endoscopic system that can accurately measure polyp size.

Disclosure of Invention

An object of the present invention is to provide an endoscopic lesion absolute size measurement system capable of automatically and accurately measuring the size of polyps in an endoscope.

Embodiments of the present application provide an endoscopic lesion absolute size measurement system comprising: a light box, an illumination path and a structured light path;

a light source is arranged in the light box;

the illumination path comprises a shielding lens, a first coupler, a first optical fiber and an illumination lens; the first coupler is configured to enable coupling between the optical box and the first optical fiber; the shutter lens is configured to cut in and out between the first coupler and the light source; one end of the first optical fiber is connected to the first coupler, and the other end of the first optical fiber is connected to the illumination lens;

the structured light pathway comprises a structured light generator, a second coupler, a second optical fiber, and a structured light projector; the second coupler is configured to enable coupling between the optical box and the second optical fiber; the structured light generator is configured to be cut-in and cut-out between the second coupler and the light source; one end of the second optical fiber is connected to the second coupler, and the other end of the second optical fiber is connected to the structured light projector;

the illumination lens and the structured light projector are disposed at an end of an endoscope.

In a preferred embodiment, the structured light generator is cut in and out simultaneously with the masking lens.

In a preferred embodiment, when switched to the structured light mode, the blocking lens cuts between the first coupler and the light source, blocking the illumination path; the structured light generator cuts into the second coupler and the light source to generate structured light;

when the white light mode is switched, the shielding lens and the structural light generator are cut out simultaneously, and the first optical fiber and the second optical fiber conduct illumination light.

In a preferred embodiment, the structured light generator generates visible light having a wavelength between 400 and 700 nm.

In a preferred embodiment, the pass band width of the light generated by the structured light generator is less than 5% of the value of the central wavelength.

In a preferred embodiment, the end of the endoscope is further provided with a nozzle, an objective lens and an instrument outlet.

In a preferred embodiment, the light source is a xenon lamp, a laser or an LED.

The application also discloses a method for measuring the absolute size of a focus under an endoscope, which comprises the following steps:

as described above, the endoscope lesion absolute size measurement system performs shooting in a white light mode, and inputs a shot image into a trained target detection model to detect whether polyps appear in real time;

if polyps are detected, automatically segmenting polyp regions in real time by using a deep neural network based on a coder-decoder structure;

switching the focus absolute size measuring system under the endoscope to a structured light mode to perform structured light imaging;

the acquired structured light image is analyzed using an image processing system and the polyp size is calculated in combination with the polyp segmentation results.

In a preferred embodiment, the object detection model comprises a first encoder and a detector;

the first encoder is configured to perform feature extraction on an image obtained in a white light mode to obtain a feature map;

the detector is configured to regress the input feature map to obtain coordinates of a location where a polyp is located.

In a preferred embodiment, the encoder-decoder structure-based deep neural network includes:

a second encoder for inputting an image, including a convolutional layer, an active layer, and a pooling layer;

the decoder used for outputting the result comprises a convolution layer, an activation layer, a serial layer and an upper sampling layer;

and fusing the feature maps with the size difference smaller than a preset threshold value together through cross-layer feature fusion between the second encoder and the decoder.

The system and the method for measuring the absolute size of the focus under the endoscope have the following technical effects:

the system and the method for measuring the absolute size of the focus under the endoscope can accurately measure the size of polyp, so that a user can reasonably layer CRC risks of a patient, reasonably arrange follow-up interval time and provide a proper colorectal tumor screening suggestion.

The present specification describes a number of technical features distributed throughout the various technical aspects, and if all possible combinations of technical features (i.e. technical aspects) of the present specification are listed, the description is made excessively long. In order to avoid this problem, the respective technical features disclosed in the above summary of the invention of the present application, the respective technical features disclosed in the following embodiments and examples, and the respective technical features disclosed in the drawings may be freely combined with each other to constitute various new technical solutions (which should be regarded as having been described in the present specification) unless such a combination of the technical features is technically infeasible. For example, in one example, the feature a + B + C is disclosed, in another example, the feature a + B + D + E is disclosed, and the features C and D are equivalent technical means for the same purpose, and technically only one feature is used, but not simultaneously employed, and the feature E can be technically combined with the feature C, then the solution of a + B + C + D should not be considered as being described because the technology is not feasible, and the solution of a + B + C + E should be considered as being described.

Drawings

FIG. 1 is a schematic block diagram of an endoscopic lesion absolute size measurement system according to the present application;

FIG. 2 is a schematic flow chart of a method of measuring absolute lesion size under an endoscope according to the present application;

FIG. 3 is a schematic structural diagram of an object detection model of an endoscopic lesion absolute size measurement method according to the present application;

fig. 4 is a schematic structural diagram of a U-NET network based on an encoder-decoder structure according to the endoscopic lesion absolute size measurement method of the present application;

FIG. 5 is a schematic diagram of a lesion size calculation model for an endoscopic lesion absolute size measurement method according to the present application;

description of reference numerals:

1. a light source,

2. A structured light pathway,

3. An illumination path,

4. A shielding lens,

5. A first coupler,

6. A first optical fiber,

7. An illumination lens,

8. A structured light generator,

9. A second coupler,

10. A second optical fiber,

11. A structured light projector,

12. A nozzle,

13. An objective lens,

14. An appliance outlet,

15. A light box.

Detailed Description

In the following description, numerous technical details are set forth in order to provide a better understanding of the present application. However, it will be understood by those skilled in the art that the technical solutions claimed in the present application may be implemented without these technical details and with various changes and modifications based on the following embodiments.

To make the objects, technical solutions and advantages of the present application more clear, embodiments of the present application will be described in further detail below with reference to the accompanying drawings.

A first embodiment of the present application relates to an endoscopic lesion absolute size measurement system, as shown in fig. 1, including: a light box, an illumination path and a structured light path;

a light source is arranged in the light box;

the illumination path comprises a shielding lens, a first coupler, a first optical fiber and an illumination lens; the first coupler is configured to effect coupling between the optical box and the first optical fiber; the shutter lens is configured to be cut-in and cut-out between the first coupler and the light source; one end of the first optical fiber is connected to the first coupler, and the other end of the first optical fiber is connected to the illumination lens;

the structured light path comprises a structured light generator, a second coupler, a second optical fiber and a structured light projector; the second coupler is configured to enable coupling between the optical box and the second optical fiber; the structured light generator is configured to be switched in and out between the second coupler and the light source; one end of the second optical fiber is connected to the second coupler, and the other end of the second optical fiber is connected to the structured light projector;

the illumination lens and the structured light projector are disposed at an end of the endoscope.

Optionally, in an embodiment, the structured light is projected to the tissue surface by the structured light projector after passing through the coupler and the structured light fiber. The structured light is projected to the surface of the object to be measured and then is modulated by the height of the object to be measured, and the modulated structured light is collected by a camera system and is transmitted to a computer for analysis and calculation so as to obtain the three-dimensional data of the object to be measured.

The structured light generator and the shielding lens are cut in and out simultaneously.

When the structured light mode is switched, the shielding lens is cut into the space between the first coupler and the light source to shield the illumination channel; the structured light generator is switched between the second coupler and the light source to generate structured light;

when switching to the white light mode, the shutter glasses and the structured light generator are switched out simultaneously, and both the first optical fiber and the second optical fiber conduct illumination light.

Optionally, in an embodiment, the light generator may be switched by a key or system control. If the structured light generator is switched in, the rear end shielding lens of the illumination light guide beam is switched in at the same time, the illumination light source is prevented from passing through, and only the structured light projector projects the structured light. The collected structured light image is sent to a host computer for processing, and three-dimensional data contained in the structured light image is analyzed; if the structured light generator is cut out, the rear end shielding lens of the illumination light guide beam is cut out simultaneously, so that the illumination light source passes through to carry out white light illumination, and meanwhile, the structured light channel is used as a general illumination channel to carry out white light illumination.

The actual choice of the structured light generator to generate visible light at wavelengths between 400 and 700nm or to generate light with a pass band width below 5% of the value of the central wavelength is determined experimentally to minimize reflections to obtain the clearest structured light image.

Optionally, in one embodiment, the end of the endoscope is further configured with a nozzle, an objective lens and an instrument outlet.

Optionally, in one embodiment, the light source is a xenon lamp, a laser, or an LED.

A second embodiment of the present application relates to a method for measuring an absolute size of a lesion under an endoscope, comprising the steps of, as shown in fig. 2:

in step 202, the system for measuring the absolute size of a lesion under an endoscope as described in the first embodiment is used to perform a photographing in a white light mode, and the photographed image is input to a trained object detection model to detect whether a polyp is present or not in real time.

If a polyp is detected, step 204 is entered and the polyp region is automatically segmented in real time using a deep neural network based on the encoder-decoder structure.

Then, in step 206, the system for measuring the absolute size of the lesion under the endoscope is switched to the structured light mode for structured light imaging.

Thereafter, in step 208, the acquired structured light image is analyzed using the image processing system, and a polyp size is calculated in combination with the polyp segmentation results.

In a preferred embodiment, the object detection model comprises a first encoder and a detector; the first encoder is configured to perform feature extraction on an image obtained in the white light mode to obtain a feature map; the detector is configured to regress the input feature map, resulting in coordinates of where the polyp is located.

In a preferred embodiment, the deep neural network based on the encoder-decoder structure includes:

and fusing the feature maps with the size difference smaller than the preset threshold value together between the second encoder and the decoder through cross-layer feature fusion.

Optionally, in an embodiment, as shown in the target detection model shown in fig. 3, an image x under white light is used as an input, an encoder is used to perform feature extraction, and a feature map obtained after feature extraction is regressed by a detector to obtain a coordinate y of a position where a polyp is located. The encoder and the detector are composed of a deep neural network and comprise a convolutional layer, a downsampling layer, an upsampling layer, a pooling layer, a batch normalization layer, an activation layer and the like. With the above object detection model, when the system detects the presence of a polyp, the system will prompt the physician with a red frame on the monitor for the rectangular region where the polyp is located and sound an alarm. The system then performs a fine segmentation of the polyp region.

The fine segmentation of polyp regions is based on a deep neural network of encoder-decoder architecture. The left half part is an encoder which comprises a convolution layer, an activation layer and a pooling layer. The right half is a decoder, which comprises a convolutional layer, an active layer, a serial layer and an up-sampling layer. The characteristic graphs with similar sizes are fused together directly through cross-layer characteristic fusion between the encoder and the decoder, and context information and global information can be extracted better. Fig. 4 shows a U-NET network based on an encoder-decoder architecture.

Once the focus is segmented, the system automatically switches or manually switches to a structured light imaging mode, and meanwhile, the rear end of the illumination path is closed through a shielding lens to prevent the illumination light source from passing through. And the structured light is projected to the surface of the tissue through the structured light channel, and the deformation of the structured light image is collected by the endoscope camera and is transmitted to the image processing center for processing. After the acquired structured light image is transmitted to an image processing center, the image processing center combines the segmentation result and the deformation of the structured light image, calculates the three-dimensional contour of the focus by a phase shift method, and gives the size of the focus.

Alternatively, in one embodiment, a lesion size calculation model is provided, as shown in fig. 5. P is the projector optical center and C is the camera optical center. O is the intersection of the camera optical axis and the projector optical axis. And a horizontal plane passing through the point O is taken as a reference X axis in the calculation. L1 and L2 are the distances from the camera and projector optical centers, respectively, to the X-axis. d is the distance from the camera optical center to the projector optical center along the X-axis direction. The a-B-O plane is a hypothetical imaginary plane parallel to the connecting line PC between the projector optical center and the camera optical center. For a certain point Q of the surface of the object, the height Z of the point Q relative to the reference plane can be solved by the triangular relation:

since the plane-to-plane projection is a linear mapping, the projector pattern mapping also has a fixed period on the imaginary plane ABO, AB and

and has a linear relationship.

Then, the absolute phase of the Q point is calculated

When projecting a sinusoidal fringe image onto a three-dimensional diffuse reflective surface, the image of a certain point Q (x, y) observed from the camera can be represented as:

where A (x, y) is the background light intensity, B (x, y)/A (x, y) represents the contrast of the grating fringes,

is a phase value. For the N-step phase shift method, the phase difference between the phase shift of the sinusoidal grating in each step and the projection phase of the phase shift of the sinusoidal grating in the previous step is 2 pi/N, namely the following equation:

where i =1,2, …, N. The equation set has N equations, three unknowns, and can be solved when N > = 3. Obtaining by solution:

by phase unwrapping, the absolute phase value of the Q point (x, y) can be obtained

Will be provided with

And the height Z of the Q point from the virtual plane ABO can be obtained by substituting the formula.

By calibrating the camera parameters, the absolute coordinates (X, Y, Z) of the Q point in the real world can be calculated according to Z, and the three-dimensional size of the polyp can be obtained by calculating the absolute coordinates (X, Y, Z) of each point on the acquired structured light image and combining the polyp segmentation result.

It is noted that, in the present patent application, relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, the use of the verb "comprise a" to define an element does not exclude the presence of another, same element in a process, method, article, or apparatus that comprises the element. In the present patent application, if it is mentioned that a certain action is executed according to a certain element, it means that the action is executed according to at least the element, and two cases are included: performing the action based only on the element, and performing the action based on the element and other elements. The expression of a plurality of, a plurality of and the like includes 2, 2 and more than 2, more than 2 and more than 2.

This specification includes combinations of the various embodiments described herein. Separate references to "one embodiment" or a particular embodiment, etc., do not necessarily refer to the same embodiment; however, these embodiments are not mutually exclusive, unless indicated as mutually exclusive or as would be apparent to one of ordinary skill in the art. It should be noted that the term "or" is used in this specification in a non-exclusive sense unless the context clearly dictates otherwise.

All documents mentioned in this application are to be considered as being integrally included in the disclosure of this application so as to be subject to modification as necessary. Further, it is understood that various changes or modifications may be made to the present application by those skilled in the art after reading the above disclosure of the present application, and such equivalents are also within the scope of the present application as claimed.

Claims

1. An endoscopic lesion absolute size measurement system, comprising: a light box, an illumination path, and a structured light path;

a light source is arranged in the light box;

the structured light pathway includes a structured light generator, a second coupler, a second optical fiber, and a structured light projector; the second coupler is configured to enable coupling between the optical box and the second optical fiber; the structured light generator is configured to be cut-in and cut-out between the second coupler and the light source; one end of the second optical fiber is connected to the second coupler, and the other end of the second optical fiber is connected to the structured light projector;

2. The system of claim 1, wherein the structured light generator is cut into and out of the masking lens simultaneously.

3. The system of claim 2, wherein the system further comprises a processor,

when switched to the structured light mode, the blocking lens cuts between the first coupler and the light source, blocking the illumination path; the structured light generator cuts in between the second coupler and the light source to generate structured light;

4. The system of claim 1, wherein the structured light generator generates visible light having a wavelength between 400 nm and 700 nm.

5. The system of claim 1, wherein the structured light generator generates light having a pass band width that is less than 5% of a value of the central wavelength.

6. The system of claim 1, wherein the end of the endoscope is further configured with a nozzle, an objective lens, and an instrument outlet.

7. The system of claim 1, wherein the light source is a xenon lamp, a laser, or an LED.

8. An endoscopic lesion absolute size measuring method is characterized by comprising the following steps:

the system for measuring the absolute size of a lesion under an endoscope as set forth in any one of claims 1 to 7, which is photographed in a white light mode, and inputs the photographed image into a trained object detection model to detect whether polyps are present or not in real time;

9. The method of claim 8, wherein the target detection model comprises a first encoder and a detector;

10. The method of endoscopic lesion absolute size measurement according to claim 8, wherein said encoder-decoder structure-based deep neural network comprises: