CN113827171B

CN113827171B - Endoscopic imaging method, endoscopic imaging system and software program product

Info

Publication number: CN113827171B
Application number: CN202110690676.8A
Authority: CN
Inventors: T·于尔根
Original assignee: Olympus Winter and Ibe GmbH
Current assignee: Olympus Winter and Ibe GmbH
Priority date: 2020-06-23
Filing date: 2021-06-22
Publication date: 2024-05-03
Anticipated expiration: 2041-06-22
Also published as: US20210397865A1; DE102020116473A1; CN113827171A; EP3928680B1; JP7261835B2; US11857165B2; EP3928680A1; JP2022002701A

Abstract

The present invention relates to an endoscopic imaging method, an endoscopic imaging system and a software program product, the system comprising a video endoscope with at least one CMOS image sensor, a light source unit, an image evaluation unit, a control unit and a display device. White light images are captured under white light illumination by means of a video endoscope and evaluated in real time by an image evaluation unit to check for the presence of at least one structure having at least one predefined feature. When the image evaluation unit detects that at least one structure having at least one predefined feature is present in the white light image, special light illumination is generated by means of the light source unit using at least one special light, and an image of the video stream is captured under the special light illumination and image-processed in a special light processing mode, comprising identifying a sub-region of the white light image comprising structures having the predefined feature detected in the white light image and reading out only the sub-region of the image sensor, the image data read out of the sub-region being further processed into the special light image.

Description

Endoscopic imaging method, endoscopic imaging system and software program product

Technical Field

The invention relates to a method for endoscopic imaging, an endoscopic imaging system and a software program product.

Background

In addition to having a white light pattern that includes white light illumination of the entire RGB spectrum, modern endoscopic imaging systems also provide various special light imaging patterns that are critical to the visible spectrum (VIS). Such Special Light Imaging (SLI) modes include, for example, narrowband imaging (NBI), near infrared fluorescence imaging (NIRF), and Red Dichroism Imaging (RDI). RDI is a proprietary method of Olympus Technology, described in U.S. patent nos. 10,034,600 B2 and 9,775,497 B2. This list is not exhaustive.

Near infrared fluorescence imaging (NIRF) can be used to analyze and assess vascular perfusion, confirm anatomy of the hepatobiliary system, discover lymph nodes, or visualize ureters after using exogenous contrast agents such as ICG (indocyanine green), CY5.5, ZW800, or ZW-1. Red Dichroism Imaging (RDI) can be used to identify the source of arterial bleeding. Narrowband imaging (NBI) can help differentiate benign hyperplasia from cancerous tissue or a precursor of cancer, such as differentiating between NICE-1 and NICE-2 type intestinal polyps, to determine whether resections of polyps are needed.

Typical examinations and procedures are performed by doctors using white light illumination for Visual Imaging (VIS). If the physician finds an abnormality in the white light image that requires more scrutiny in the special light imaging mode, the physician will choose the appropriate special light imaging mode based on their training and experience.

Disclosure of Invention

Against this background, it is an object of the present invention to assist a physician in improved forms of endoscopic imaging during examination or surgery.

This object is achieved by a method of endoscopic imaging with an endoscopic imaging system comprising a video endoscope with at least one CMOS image sensor, a light source unit designed to generate white light and at least one special light, an image evaluation unit, a control unit and a display device, and further developed to capture white light images by means of the video endoscope under white light illumination and evaluate the white light images in real time by the image evaluation unit to check for the presence of at least one structure with at least one predefined feature, wherein when the image evaluation unit detects the presence of at least one structure with at least one predefined feature in the at least one white light image a special light imaging mode is set in which the light source unit uses the at least one special light to generate special light illumination and to capture one or more images of a video stream under the special light illumination and to image process one or more images of a video stream under a special light processing mode, the image processing comprising: identifying a sub-region of the white light image comprising the structure having the predefined feature detected in the white light image, and reading out only the sub-region of the image sensor, wherein the image data read out from the sub-region is further processed into a special light image.

The method according to the invention is based on the fact that: most endoscopic video systems (i.e., video endoscopes or cameras for endoscopes) are equipped with CMOS image sensors instead of CCD image sensors. Unlike CCD image sensors, many CMOS image sensors can be read out only in sub-regions, so-called "regions of interest" (ROIs), which significantly improves the read rate. The sub-region is typically a rectangular region, which in the context of the present invention is dimensioned such that it completely contains the structure, the boundaries of which, if applicable, occupy for example 2% to 20% of the extent of the long sides of the rectangle. For example, intestinal polyps typically occupy about 10% to 20% of the image area in an image. In many clinical situations, such ROIs extend over equivalent areas in the complete image. In this case, for example, a CMOS image sensor having HD resolution of 1980x1080 pixels can be read out exclusively and very quickly in the corresponding sub-region containing polyps (for example, in rectangular regions where coordinates of the diagonal points are 600/200 and 1061/662, for example). The resulting ROI containing intestinal polyps will have a size of 461x462 pixels (about 213,000 pixels) and therefore will be only about one tenth of the size of a full HD image.

By means of the method according to the invention, a high image refresh rate can thus be obtained even when the special light imaging mode has been activated, because the sub-areas of the CMOS image sensor are read out and processed faster than the entire image data of the CMOS image sensor are read out in the white light imaging mode. In case no sub-region is selected, the image refresh rate in case of alternating sequences of white light images and special light images will be halved, e.g. from 50Hz to 25Hz, wherein the white light image sequence will be at 25Hz and the special light image sequence will also be at 25Hz. In the case of rapid movements, this may lead to motion artefacts.

Many special light imaging modes are also weak. The required high sensitivity or gain factor can lead to significant image noise. Noise may be reduced by increasing the exposure time or combining consecutive frames. These measures are severely limited in normal time-sequential imaging without sub-region selection, whereas the higher image refresh rate possible under special light conditions according to the invention can be used to increase the effective exposure time of the special light image via a tighter continuity of the special light image.

The method according to the invention also makes it possible to assist the attending physician by image recognition of white light images, in which structures with predefined features are automatically detected. For example, such structures may be perfused blood vessels, lymph nodes, arterial bleeding or intestinal polyps, although this list is not exhaustive. The predefined features of these structures are well known and are well documented based on a large number of image materials and can be identified by means of automated image evaluation. According to the invention, in this case a suitable special light imaging is provided, by means of which the identified structure is shown in more detail and more precisely. The doctor does not have to make a decision to set the special light imaging mode, and they can see the special light imaging without doing anything.

The ongoing inspection is typically handled as a video stream. The method according to the invention can be applied to an ongoing video stream, but also to individual images separated from the video stream by means of a snapshot function for more careful examination and recording, if required by the physician.

The sub-regions are selected in the image evaluation unit and instructions as to which sub-region of the CMOS image sensor to read out are issued directly by the image evaluation unit or via the control unit. The image evaluation unit may also be integrated as software in the control unit as a functional unit, for example as a program or a subroutine of a program.

The relevant structure is not visible under normal conditions of various inspections. In these cases, it is advantageous to display the white light image by means of the display device if the image evaluation unit does not detect a structure having at least one predefined feature in the white light image.

In an embodiment, the special light image is displayed by means of a display device alone or as a composite image superimposed on the white light image. The separate display of the special light image has the advantage of an undisturbed display, which is easy to interpret for the eyes of a trained physician. Displaying the composite image, wherein the special light image is superimposed consistently on the white light image at the location of the structure, provides the advantage of embedding the special light image into the surrounding structure. Such fused or composite images may be referred to as Augmented Reality Images (ARI). For a composite image, the special light image may be superimposed on the white light image with a certain transparency (e.g. between 5% and 50%) such that the underlying structure of the white light image that is not visible in the special light image may still be seen, or the special light image may be superimposed on the white light image in an opaque manner (transparency 0% to 5%).

In an embodiment with a sequence of composite images displayed in a video stream, in case the image evaluation unit detects the presence of at least one structure with at least one predefined feature in the white light image, an alternating white light and special light illumination is generated in synchronization with the sequence of video images, wherein in each case a pair of images consisting of the white light image and the special light image is processed and combined into a composite image. A snapshot function may be used to capture a single frame from the composite image video stream and for more careful inspection and recording. Thus, within the scope of this patent application and the present invention, the term "special light imaging mode" also includes the case: white light images and special light images are captured, further processed and combined together in an alternating sequence.

In one embodiment, the video image sequence and the synchronized alternating white light and special light illumination occur at such a temporal rhythm: wherein the duration of the special light illumination and the special light image capture is shorter than the duration of the white light illumination and the white light image capture. This is possible because for special light image capturing only one sub-area of the CMOS image sensor is read out in each case and only the image data of said sub-area is further processed in the image evaluation unit, in particular because no structure recognition is applied to the special light image. Thus, it is no longer necessary to halve the frequency of image display (i.e. the continuity of a single frame), but an image sequence with a high frequency similar to a pure white video sequence can be realized, whereby motion artifacts are largely prevented. One possibility to achieve this is that a control unit, which may be integrated with the image evaluation unit, instructs the image sensor and the light source unit in the video endoscope to capture images or sub-images in a correspondingly synchronized time sequence and to provide white light illumination and special light illumination.

In an embodiment, the light source unit is designed to generate one or more of the following as special light: illumination for narrowband imaging (NBI), illumination for near infrared fluorescence imaging (NIRF) and illumination for red bi-color imaging (RDI), wherein in particular RDI illumination is generated in case of detected bleeding and/or NBI illumination is generated in case of detected intestinal polyps. The image evaluation unit may be designed to select one of a plurality of special light modes available based on the detected structure with predefined characteristics. Alternatively, the predefined features and the special light pattern may or may have been pre-selected and set. This may be useful, for example, in the case of searching for an examination of a particular structure (e.g., in colonoscopy) or surgery to predict, detect and stop bleeding.

In an embodiment, the structure having at least one predefined feature is detected by means of an artificial intelligence based image evaluation algorithm and/or the sub-region comprising the structure is selected by means of an artificial intelligence based image evaluation algorithm and/or the special light pattern is selected by means of an artificial intelligence based image evaluation algorithm. In these cases, the image evaluation algorithm may be based on one or more classified neural networks, in particular CNNs, which have been pre-trained using white light images comprising structures having at least one predefined feature.

In an embodiment, the image evaluation unit evaluates the special light image to check for the presence of the corresponding structure, in particular to confirm the presence of the structure or otherwise to prompt a departure from the special light imaging mode again. Special light imaging modes are particularly suitable for showing relevant structures in a manner that is very high in contrast and thus clearly identifiable, so that the evaluation of special light images with respect to these structures is of great importance for the presence of the corresponding structures. The corresponding image evaluation is also faster than in a white light image, since only a sub-area of the white light image is present in the special light image, so that a smaller amount of data needs to be evaluated.

The object of the invention is also achieved by an endoscopic imaging system comprising a video endoscope with at least one CMOS image sensor, a light source unit designed to generate white light and at least one special light, an image evaluation unit, a control unit and a display device, wherein the control unit is connected to the light source unit, the video endoscope and the image evaluation unit for signal transmission and the image evaluation unit is connected to the video endoscope for signal transmission, wherein the imaging system with the video endoscope, the light source unit, the image evaluation unit, the control unit and the display device is designed and configured to perform the above-mentioned method according to the invention.

The video endoscope of the imaging system according to the invention may comprise a CMOS sensor in the distal end region of the endoscope shaft or in the handle, or be designed as a combination of an endoscope and a camera head comprising a CMOS sensor as an image sensor.

Furthermore, the object of the invention is also achieved by means of a software program product comprising a program code medium for an endoscopic imaging system according to the invention, the software program product comprising an image evaluation program component for execution in the image evaluation unit of the imaging system and a control program component for execution in the control unit of the imaging system, wherein the image evaluation program component and the control program component are designed to perform the above-described method according to the invention if executed in the image evaluation unit and control unit.

The endoscopic imaging system according to the invention and the software program product according to the invention relate to the above-described method according to the invention and achieve their advantages, features and characteristics in the same way.

Further features of the invention will become apparent from the description of embodiments according to the invention in conjunction with the claims and the accompanying drawings. Individual features or combinations of features may be implemented according to embodiments of the invention.

Within the scope of the invention, features specified by "particularly" or "preferably" are understood to be optional features.

Drawings

The invention is described below based on exemplary embodiments with reference to the accompanying drawings without limiting the general idea of the invention, whereby reference is explicitly made to the accompanying drawings with respect to all details according to the invention which are not explained in more detail in the text. In the accompanying drawings:

figure 1 is a schematic view of an endoscopic imaging system according to the present invention,

FIG. 2 is a schematic diagram of an imaging concept, and

Fig. 3 is an exemplary flow chart of a method according to the present invention.

In the drawings, identical or similar elements and/or parts are provided in each case with the same reference numerals; so that duplicate description will be omitted throughout.

List of reference numerals

2. Body part

4. Body cavity

6. Structure of the

10. Endoscopic imaging system

20. Video endoscope

22. Endoscope shaft

24. Inlet lens

26 CMOS image sensor

28. Handle

30. Light source unit

40. Image evaluation unit

42. White light image

44. Sub-regions

46. Special light image

48. Composite image

50. Control unit

60. Display device

100. Capturing white light images

102. Analysis of white light images

104. Displaying white light images

106. Setting special light imaging mode

108. Capturing special light images

110. Generating and displaying composite images

Detailed Description

Fig. 1 is a schematic view of an endoscopic imaging system 10 according to the present invention in a surgical situation. An examination is performed in a body cavity 4 of a body 2 of a patient by means of a video endoscope 20, wherein structures 6 having predefined features, such as intestinal polyps, hemorrhages, perfused blood vessels, etc., are found.

The video endoscope 20 includes an endoscope shaft 22 on a handle 28, and on the distal end of the shaft 22, a CMOS image sensor 26 is disposed behind an entrance lens 24 and an optical system (not shown here). Alternatively, video endoscope 20 may also be designed as a combination of a conventional endoscope and a camera head.

The video endoscope 20 is connected to a light source unit 30, which light source unit 30 may also be part of the video endoscope 20 or, alternatively, part of the control device of the endoscopic imaging system 10. The light source unit 30 having various light sources and filter units is designed to alternately generate white light and special light according to one or more special light irradiation programs, which is guided into the body cavity 4 through the video endoscope 20 and irradiates the examination environment.

The imaging system 10 further includes an image evaluation unit 40 and a control unit 50, which are also connected to the light source unit 30 and the video endoscope 20, and a display device 60 connected to the image evaluation unit 40. The various components, in particular the image evaluation unit 40 and the control unit 50, are designed to perform the method according to the invention. For this purpose, they are provided with a program code medium which performs image evaluation and control.

Fig. 2 shows the imaging concept in schematic form. Imaging and image acquisition is based on the capturing of white light images 42, which white light images 42 are to be evaluated by means of an image evaluation unit 40. Such white light images 42 may, for example, have HD resolution and be generated from the light source unit 30 under white light illumination. If the white light image 42 comprises a structure 6 having predefined characteristics, such as characteristics of a perfused blood vessel, a hemorrhage, an intestinal polyp, etc., the image evaluation unit 40 identifies a sub-region 44 of the white light image 42 containing the structure 6.

Because the structure 6 is detected, special light illumination for special light imaging is set for the next image and the CMOS sensor 26 is read out only in the sub-area 44. This representation of the readout sub-area 44, in particular the pseudo-color representation, is superimposed as a special light image 46 at the correct position on the white light image 42, resulting in a composite image 48 which is displayed to the attending physician on a display device 60. The composite image 48 may be part of a video stream in a special light imaging mode in which pairs of white light images and special light images are captured in each case and combined into or used as a single image for more careful inspection and recording. Preferably, in the case of a video stream in a special light imaging mode, each white light image is analyzed and the sub-region 44 is redetermined, as the video endoscope 20 and the structure 6 can be moved relative to each other.

The selection of a particular special light imaging mode is either done a priori by the physician, who for example is performing a particular examination (e.g. colonoscopy), wherein only one special light imaging mode is relevant; or which particular light imaging mode is selected is decided by the image evaluation algorithm in the image evaluation unit 40 based on the found structure 6. Such selection algorithms may be, for example, based on a neural network that has been trained using previously captured white light images with corresponding structures in order to identify and classify the corresponding structures and corresponding best particular light imaging patterns.

Fig. 3 is an exemplary flow chart of a method according to the present invention. The method starts with capturing a white light image 42 (method step 100) which is then analyzed to check for the presence of a structure 6 having predefined features (method step 102). If no structure 6 is found, then the white light image 42 is displayed as such (method step 104) and a new white light image 42 is captured (method step 100). If the structure 6 is actually found in the white light image 42, the sub-region 44 of the white light image 42 in which the structure 6 is located is identified. Subsequently, a preset or suitable special light imaging mode with corresponding special light illumination is set (method step 106), an image is captured under the special light illumination that has been set and the image data of the CMOS image sensor 26 is read out only in the sub-areas 44 and further processed into a special light image 46 (method step 108). The special light image 46 is then superimposed on the white light image 42 and displayed as a composite image 48 (method step 110). This process is repeated from method step 100 until the inspection is complete or until the inspection mode according to the invention is terminated.

All mentioned features (including features taken alone from the figures and individual features disclosed in combination with other features) are considered essential for the invention, both individually and in combination. Embodiments according to the invention may be realized by means of the individual features or combinations of features.

Claims

1. A method of endoscopic imaging with an endoscopic imaging system (10), the endoscopic imaging system (10) comprising a video endoscope (20) with at least one CMOS image sensor (26), a light source unit (30) designed to generate white light and at least one special light, an image evaluation unit (40), a control unit (50) and a display device (60),

It is characterized in that the method comprises the steps of,

Capturing a white light image (42) by means of the video endoscope (20) under white light illumination, and evaluating the white light image (42) by the image evaluation unit (40) in real time to check for the presence of at least one structure (6) having at least one predefined feature,

Wherein when the image evaluation unit (40) detects the presence of at least one structure (6) having at least one predefined feature in at least one white light image (42), a special light imaging mode is set in which the light source unit (30) generates special light illumination using the at least one special light and captures one or more images of a video stream under the special light illumination and image processing of the one or more images of the video stream in a special light processing mode, the image processing comprising: -identifying a sub-region (44) of the white light image (42) containing the structure (6) having the predefined feature detected in the white light image, and-reading out only this sub-region (44) of the image sensor (26), wherein the image data read out from the sub-region (44) is further processed into a special light image (46).

2. Method according to claim 1, characterized in that the white light image (42) is displayed by means of the display device (60) if the image evaluation unit (40) does not detect a structure with at least one predefined feature in the white light image (42).

3. The method according to claim 1, characterized in that the special light image (46) is displayed alone or as a composite image (48) superimposed on a white light image (42) by means of the display device (60).

4. Method according to claim 1, characterized in that, in the event that the image evaluation unit (40) detects the presence of at least one structure (6) with at least one predefined feature in the white light image (42), an alternating white light illumination and special light illumination is generated in synchronization with the video image sequence, wherein in each case a pair of images consisting of white light image (42) and special light image (46) is processed and combined into a composite image (48).

5. The method of claim 4, wherein the sequence of video images and the synchronized alternating white light illumination and special light illumination occur at a temporal rhythm in which the duration of the special light illumination and special light image capture is shorter than the duration of the white light illumination and white light image capture.

6. The method according to claim 1, characterized in that the light source unit (30) is designed to generate one or more of the following as special light: illumination for narrowband imaging (NBI), illumination for near infrared fluorescence imaging (NIRF), and illumination for Red Dichroism Imaging (RDI).

7. A method according to claim 6, wherein RDI illumination is generated in case of detected bleeding and/or NBI illumination is generated in case of detected intestinal polyps.

8. Method according to claim 1, characterized in that the image evaluation unit (40) is designed to select one of a plurality of special light modes available based on the detected structure (6) with the predefined features.

9. The method according to claim 8, wherein the predefined features and the special light pattern are or have been pre-selected and set.

10. Method according to claim 8, characterized in that the structure (6) with at least one predefined feature is detected by means of an artificial intelligence based image evaluation algorithm and/or that the sub-area (44) containing the structure (6) is selected by means of an artificial intelligence based image evaluation algorithm and/or that the special light pattern is selected by means of an artificial intelligence based image evaluation algorithm.

11. The method according to claim 10, characterized in that the image evaluation algorithm is based on one or more classification neural networks that have been pre-trained using a white light image (42) comprising the structure (6) with at least one predefined feature.

12. The method of claim 11, wherein the one or more classification neural networks are CNNs.

13. The method according to claim 1, characterized in that the image evaluation unit (40) evaluates the special light image (46) to check for the presence of a corresponding structure (6).

14. An endoscopic imaging system (10), the endoscopic imaging system (10) comprising a video endoscope (20) with at least one CMOS image sensor (26), a light source unit (30) designed to generate white light and at least one special light, an image evaluation unit (40), a control unit (50) and a display device (60), wherein the control unit (50) is connected to the light source unit (30), the video endoscope (20) and the image evaluation unit (40) for signal transmission and the image evaluation unit (40) is connected to the video endoscope (20) for signal transmission, characterized in that the endoscopic imaging system (10) with the video endoscope (20), the light source unit (30), the image evaluation unit (40), the control unit (50) and the display device (60) is designed and configured to perform the method according to any one of claims 1 to 13.

15. A computer-readable storage medium comprising a program code medium for an endoscopic imaging system (10) according to claim 14, the computer-readable storage medium comprising an image evaluation program component for execution in an image evaluation unit (40) of the endoscopic imaging system (10), and a control program component for execution in a control unit (50) of the endoscopic imaging system (10), characterized in that the image evaluation program component and the control program component are designed to perform the method according to any one of claims 1 to 13 when the image evaluation program component and the control program component are executed in the image evaluation unit (40) and the control unit (50).