WO2017117710A1

WO2017117710A1 - Imaging system and method for endoscopy

Info

Publication number: WO2017117710A1
Application number: PCT/CN2016/070148
Authority: WO
Inventors: 库马阿兔; 黄士维; 刘楷哲; 汪彦佑
Original assignee: 秀传医疗社团法人秀传纪念医院
Priority date: 2016-01-05
Filing date: 2016-01-05
Publication date: 2017-07-13

Abstract

An imaging system (1) and method for endoscopy applicable for displaying an image of the inside of a body of a patient. The system comprises an endoscope camera module (102), an image processing module (108), a storage unit (110), an analysis module (116), and a display module (118). The endoscope camera module (102) comprises a light source (100), an infrared imaging device (104), and an image capture module (106). The infrared imaging device (104) and the image capture module (106) can be configured to photograph a target region and generate an infrared image signal and a visible light image signal. The image processing module (108) receives the infrared image signal and the visible light image signal, and generates an infrared image and a visible light image. The analysis module (116) analyzes the infrared image, generates at least one key region, and overlays the key region on the visible light image to generate an overlaid image displayed on the display module (118).

Description

Endoscope imaging system and method

Technical field

The present application relates to an endoscope imaging system and method. More specifically, the present application relates to an endoscope imaging system that analyzes a focused area of an infrared image and superimposes it on a visible light image to display a superimposed image. And methods.

Background technique

Endoscopic surgery is typically performed under visible light, where visible light allows the user to see the surface of the surgical anatomy during the procedure. In order to make the image below the surface of the organ visible, a special light source such as an infrared source is required. To this end, the patient will be injected with Indocyanine green (ICG) dye before surgery, so that the surgeon can see the infrared in the infrared image mode during the operation, such as lymph node resection and bile duct surgery. Light-labeled area containing lymph nodes or bile ducts. However, in the infrared image mode, the marked area cannot be seen, and it is difficult to perform surgery in the infrared light image mode.

Therefore, during the operation of the operation, the operator needs to continuously switch between the infrared image mode and the visible image mode, which not only causes inconvenience to the operator, but also prolongs the operation time and increases the chance of infection of the patient. .

At present, the solution to the above problem is to directly implant the infrared image into the visible light image. However, in such an image display mode, the displayed image will be indistinguishable to the operator, thus improving the difficulty of the operation. degree. For this reason, there is an urgent need for an endoscope imaging technology that can automatically superimpose an infrared image beyond a visible light image, and can selectively and accurately display a key area and automatically track a key area in a subsequent image.

Summary of the invention

In order to solve the above problems, an object of the present application is to provide an endoscope imaging system suitable for displaying images in a patient, which includes an endoscope photography module, an image processing module, a storage unit, an analysis module, and a display module. The endoscope photography module includes a light source, an infrared imager, and an image capture module. The light source is used to illuminate the patient. Infrared imaging The instrument is configured to capture a target area within the patient and produce an infrared image signal. The image capture module is configured to capture a target area and generate a visible light image signal. The image processing module is electrically connected to the endoscope photography module, and receives the infrared image signal and the visible light image signal to generate an infrared image and a visible light image, and associates the infrared image with the visible light image. The storage unit is electrically connected to the image processing module for storing infrared images and visible light images. The analysis module is electrically coupled to the image processing module and configured to analyze the infrared image to generate at least one focus area and superimpose the focus area onto the visible light image associated with the infrared image to produce a superimposed image. The display module is electrically connected to the analysis module and configured to display the superimposed image.

Preferably, the analysis module divides the infrared image into a plurality of key regions according to the intensity distribution of the infrared image, and each of the key regions respectively corresponds to different intensity ranges.

Preferably, the analysis module may superimpose the plurality of key regions in a color block manner to the visible light image associated with the infrared image to generate the superimposed image.

Preferably, the analysis module may further extract edges of the plurality of key regions, and superimpose the edges of the plurality of focus regions in a color line manner to the visible light image associated with the infrared image to generate the superimposed image.

Preferably, the analysis module may further calculate a matching point between two consecutive frames in the visible light image, and generate a transformation matrix according to the matching point, and apply the transformation matrix to the frame that has generated at least one key area and corresponds to the previous frame The infrared image is superimposed and the converted at least one focus area is superimposed on the infrared image corresponding to the subsequent picture frame to generate another superimposed image continuous with the superimposed image.

Preferably, the analysis module can calculate an error value of the matching point, and if the error value exceeds a predetermined range, the infrared imager can be configured to retake the target object and generate another infrared image signal.

According to another object of the present application, there is provided an endoscope imaging method suitable for the aforementioned endoscope imaging system, comprising the steps of: illuminating a target area in a patient with a light source; and using an infrared imager and an image capturing module Shoot the target area in the patient's body and divide The infrared image signal and the visible light image signal are generated; the image processing module is configured to receive the infrared image signal and the visible light image signal and generate the infrared image and the visible light image, and associate the infrared image with the visible light image; and store the infrared image and the visible light image in the storage unit; The analysis module is configured to analyze the infrared image to generate at least one key area, and superimpose the key area on the visible light image associated with the infrared image to generate the superimposed image; and display the superimposed image in the display module.

Preferably, in the step of configuring the analysis module to analyze the infrared image to generate at least one key area, the analysis module further divides the infrared image into a plurality of key areas according to the intensity distribution of the infrared image, and each of the key areas respectively correspond to different Strength range.

Preferably, the analysis module superimposes the plurality of key regions in a color block manner to the visible light image associated with the infrared image to generate the superimposed image.

Preferably, the analysis module further extracts edges of the plurality of key regions, and superimposes the edges of the plurality of key regions in a color line manner to the visible light image associated with the infrared image to generate the superimposed image.

Preferably, after the step of superimposing the focus area on the visible light image associated with the infrared image to generate the superimposed image, the analysis module further calculates a matching point between the consecutive two frames in the visible light image, and generates a conversion according to the matching point. a matrix, and applying a transformation matrix to the infrared image that has generated at least one focus area and corresponding to the previous frame, and superimposing the converted at least one key area on the infrared image corresponding to the subsequent frame to generate Another superimposed image that is continuous with the superimposed image.

Preferably, the analysis module calculates an error value of the matching point, and if the error value exceeds a predetermined range, the infrared imager is configured to retake the target object and generate another infrared image signal.

In summary, the endoscope imaging system and method of the present application can provide a clear field of view and accurately mark the organ region in the superimposed image provided to the operator, thereby eliminating the need to repeat the visible light image mode. And the time it takes to switch between infrared image modes, Further improve the safety, accuracy and speed of the operation. In addition, in the system, by detecting matching points and errors and calculating the conversion matrix, the time and system resources for retrieving the infrared image can be saved, and the key areas can be selectively and accurately displayed, and the subsequent images can be automatically tracked. Key areas can further improve image processing speed and save system resources.

DRAWINGS

The above and other features and advantages of the present application will become more apparent from the following detailed description of the exemplary embodiments thereof

1 is a block diagram showing an embodiment of an endoscope imaging system in accordance with the present application.

2 is an example of an infrared image of an embodiment of an endoscope imaging system in accordance with the present application.

3 is an illustration of a visible light image of an embodiment of an endoscope imaging system in accordance with the present application.

4 is an example of a processed infrared image of an embodiment of an endoscope imaging system in accordance with the present application.

5 is an example of a superimposed image produced by an embodiment of an endoscope imaging system in accordance with the present application.

6 is an illustration of a processed infrared image produced by another embodiment of an endoscope imaging system in accordance with the present application.

7 is an illustration of a processed infrared image produced in accordance with yet another embodiment of an endoscope imaging system of the present application.

8 is a schematic diagram of calculating matching points for an endoscope imaging system in accordance with the present application.

9 is an example of a superimposed image produced by another embodiment of endoscopic imaging in accordance with the present application.

10 is an example of a superimposed image produced by still another embodiment of endoscopic imaging in accordance with the present application.

11 is a flow chart showing an embodiment of an endoscope imaging method according to the present application.

FIG. 12 is a flow chart showing another embodiment of an endoscope imaging method according to the present application.

Description of the reference numerals

1: Endoscope imaging system

100: light source

102: Endoscope photography module

104: Infrared imager

106: Image capture module

108: Image Processing Module

110: storage unit

116: Analysis module

118: display module

120: power module

A, A', A", B, B', B", A1, A1', A2, A2': region

CP: Match point

detailed description

In order to understand the technical features, contents and advantages of the present application and the efficacies that can be achieved, the present application will be described in detail with reference to the accompanying drawings. The use of the instructions and the accompanying descriptions are not necessarily true proportions and precise configurations after the implementation of the present application. Therefore, the scope of the accompanying drawings and the configuration relationship should not be construed as limiting the scope of the application.

As used herein, the term "and/or" includes any and all combinations of one or more of the associated listed items. When the "at least one of" statements are prefixed to the list of components, the entire list component is modified rather than the individual components in the list.

Embodiments of the endoscope imaging system of the present application will be described in detail below with reference to the accompanying drawings. Please refer to FIG. 1 , which is a block diagram of an embodiment of an endoscope imaging system according to the present application. The endoscope imaging system 1 of the present application is suitable for displaying images in a patient, and includes an endoscope photography module 102, an image processing module 108, a storage unit 110, an analysis module 116, a display module 118, and a power module 120. The endoscope photography module 102 includes a light source 100, an infrared imager 104, and an image capture module 106. The light source 100 is configured to illuminate a patient, and the infrared imager 104 is configured to capture a target area in the patient's body, wherein the light source 100 can include a visible light source and an infrared light source, and the infrared imager 104 receives the infrared light illuminated by the target area reflected light source 100, And generate infrared image signals. Infrared image acquisition uses a photographic or photographic device to capture light waves in the infrared light field generated by the object being reflected by an infrared light source.

Please refer to FIG. 2, which is an example of an infrared image of an embodiment of an endoscope imaging system according to the present application. Before the operation, the patient will inject the Indocyanine green (ICG) dye several hours ago, so that the surgeon can see the image in the infrared image mode during the operation, such as lymph node resection and bile duct surgery. Dye-labeled areas containing lymph nodes or bile ducts. As shown in the figure, the area A in which the fluorescence is displayed is the effect of the fluorescent dye, but not the organ area, such as the area B.

In other words, the image capture module 106 is configured to capture a target area and generate a visible light image signal. The image processing module 108 is electrically connected to the endoscope camera module, and receives the infrared image signal and the visible light image signal, and generates the infrared image 112 and the visible light image 114, and associates the infrared image 112 with the visible light image 114. It should be noted that the visible light image 114 is usually an image with a continuous picture frame, where the association between the single picture frames is generated, that is, the infrared image 112 and the visible light image 114 of the first picture frame, and the method can be referred to. The use of a combined similarity measure (Applications of Soft Computing, 2009) by Juan Wachs et al., which is familiar to those skilled in the art, is not described herein. An example of a visible light image can be seen in FIG. 3, which is an example of a visible light image of an embodiment of an endoscope imaging system in accordance with the present application. In order to be able to see in clear images Under the operation, the operator usually uses visible light images, but there are some shortcomings in which the organs in the patient cannot be seen.

The storage unit 110 is electrically connected to the image processing module 108 for storing the infrared image 112 and the visible light image 114. The analysis module 116 is electrically coupled to the image processing module 108 and configured to analyze the infrared image 112 to generate at least one region of focus. For example, please refer to FIG. 4, which is an example of a processed infrared image of an embodiment of an endoscope imaging system in accordance with the present application. As shown in the figure, the infrared image 112 itself is a grayscale image, and FIG. 4 shows the infrared image 112 in blue by the same gray-scale, and the aforementioned region A corresponds to the converted high-luminance region. A', and area B corresponds to the low-luminance area B'. The grayscale image will be stored in the storage unit 110.

Please refer to FIG. 5, which is an example of superimposed images generated by an embodiment of an endoscope imaging system according to the present application. After obtaining the grayscale image, the analysis module 108 can analyze the infrared image 112. Specifically, the analysis module 108 can divide the infrared image 112 into multiple key regions according to the gray value distribution of the infrared image 112, and each key point The regions each correspond to a different range of gray values. As shown in the figure, the infrared image 112 is analyzed by the analysis module 108 and divided into two key regions of the gray value range. For the operator, the interested portion is only the organ region A displayed with high brightness. ', therefore, the key area can be divided into two parts, such as area A" and area B". The method for dividing the area of the infrared image 112 can be referred to the "Current methods in medical image segmentation 1, "Annual Review of Biomedical Engineering 2000" by DL Pham et al., which is a technique familiar to those skilled in the art. Therefore, it is not described here.

Here, the focus area is superimposed on the visible light image 114 associated with the infrared image 112 to generate a superimposed image, and the display module 118 is electrically coupled to the analysis module 116 to display the superimposed image. As shown in the figure, in each key area of the superimposed image, the area A" and the area B" are displayed in a uniform color block, providing the operator with a clear field of view and accurate marking of the organ area, eliminating the need for duplication. Switching between visible light image mode and infrared light image mode takes time to further improve the safety, accuracy and speed of the operation.

Please refer to FIG. 6, which is an example of a processed infrared image produced by another embodiment of an endoscope imaging system in accordance with the present application. As shown, the analysis module 116 can divide the infrared image 112 into three intensity ranges according to the intensity distribution of the infrared image 112, in other words, the number of key regions is three. Among them, the area of lower brightness is displayed in white, and the area of the organ can be divided into area A1 and area A2. Here, the user can set different numbers and different intensity ranges (ie, the gray value range of the grayscale image) according to requirements, so as to increase the flexibility of the system, and is not limited to the description in the specification of the present application.

Please refer to FIG. 7, which is an example of a processed infrared image produced by still another embodiment of the endoscope imaging system of the present application. In addition to the color area as shown in FIG. 5, the analysis module 116 can further extract the edges of the plurality of key areas, such as the edges of the areas A1 and A2 in FIG. 6, the color line area as shown in the figure. A1' and A2', and the edges of the plurality of focus areas are superimposed in color lines to the visible light image 114 associated with the infrared image 112 to generate a superimposed image. Here, the analysis module 116 may first generate an image containing only color lines and store it in the storage unit 110, or may directly superimpose the visible light image 112 to generate a superimposed image, which is not limited thereto. It is worth mentioning that the method of extracting the edge of the key area can be referred to the "Study and comparison of various image edge detection techniques" (Study and comparison of various image edge detection techniques, International Journal of Image Processing 2009). It is also a technique well known to those skilled in the art, and therefore will not be described here.

Please refer to FIG. 8 , which is a schematic diagram of calculating matching points according to the endoscope imaging system of the present application. As previously described, since the visible light image 114 is generally an image with a continuous frame, although the infrared image 112 and the visible light image 114 of the first frame have been processed, in order to display a continuous image, the second frame is required. The visible light image 114 is analyzed to produce another superimposed image. To this end, the analysis module 116 performs a matching point calculation on the visible light image between successive frames to generate a conversion matrix. The calculation of the matching point needs to first extract the feature points from the visible light image between successive frames, and then calculate according to the offset of the feature points to obtain the conversion matrix. Wherein, the matching point CP can pass any feature detection or feature comparison For an algorithm, refer to Li, Jing, and Nigel M. Allinson et al. for a comprehensive review of local local features for computer vision. "Neurocomputing 71.10 (2008): 1771- 1787), an example of matching point calculation is shown in Figure 8.

After obtaining the offset of the feature points, the transformation matrix can be calculated accordingly. Here, the transformation matrix can be a rigid body or non-rigid body transformation matrix between successive frames, which can be optimized by least squares (see Zitova, Barbara, and Jan). According to Flusser et al., "Image registration methods: a survey, Image and vision computing 21.11 (2003): 977-1000), the transformation matrix shown in the figure is calculated as follows:

Please refer to FIG. 9 and FIG. 10 together, which are examples of superimposed images generated by another embodiment of the endoscope imaging according to the present application and another embodiment. By applying this matrix to the aforementioned infrared image represented by a color patch or a color line, a superimposed image as shown in FIG. 9 or FIG. 10 can be produced. This superimposed image can replace the previously generated superimposed image to form a superimposed image with continuous images.

According to a preferred embodiment of the present application, the error value of the matching point is further calculated in the process of matching point calculation. When the analysis module 116 determines that the error value is too large, that is, the matching point offset is too large, or it is difficult to find the same matching point, the configuration system recaptures the infrared image in the patient.

The present application also provides an endoscope imaging method. Please refer to FIG. 11 , which is a flow chart illustrating an embodiment of an endoscope imaging method according to the present application. As shown, the endoscope imaging method is applicable to the above-described endoscope imaging system, which includes the following steps:

Step S101: illuminating a target area in the patient's body with a light source;

Step S102: taking an infrared imager and an image capturing module to capture a target area in the patient, and respectively generating an infrared image signal and a visible light image signal. Specifically, the infrared imager may first capture the fluorescent image and then switch to the image. Taking a module to capture a visible light image;

Step S103: After receiving the infrared image signal and the visible light image signal, the image processing module generates an infrared image and a visible light image, and associates the infrared image with the visible light image;

Step S104: storing the infrared image and the visible light image by using the storage unit;

Step S105: analyzing the infrared image by the analysis module to generate a key area, and superimposing the key area on the visible light image to generate a superimposed image;

Step S106: Display the superimposed image by the display module.

Here, the details of each step have been described above, and the repeated description is omitted.

The present application also provides an endoscope imaging method. Please refer to FIG. 12, which is a flow chart of another embodiment of an endoscope imaging method according to the present application. As shown, the endoscope imaging method is applicable to the above-described endoscope imaging system, which is continued from step S104 of the previous embodiment, and includes the following steps:

Step S201: After obtaining the infrared image, the analysis module divides the infrared image into a plurality of key regions according to the intensity distribution of the infrared image. Specifically, the analysis module 116 may count the infrared image 112 according to the intensity distribution of the infrared image 112. The intensity range is divided, and the image captured by the infrared image is mainly in the area of interest of the physician.

Step S202: superimposing a plurality of key regions on the visible light image to generate a superimposed image, wherein the key regions may be represented by color blocks in the superimposed image. In addition, optionally, after step S201, the method may first proceed to step S203, the analysis module further extracts edges of the plurality of key regions, and superimposes edges of the plurality of focus regions to the visible light image in a color line manner to generate Superimposed image.

Step S204: displaying the superimposed image by the display module. For this reason, only the visible light image of the first frame is currently processed. In order to display the continuous image, the visible image 114 of the second frame needs to be analyzed to generate another Add image.

Step S205: capturing a target area in the patient's body by using an image capturing module to generate a visible light image signal;

Step S206: After receiving the visible light image signal, the image processing module generates a visible light image, and the analysis module further calculates a matching point and an error value between the adjacent frames;

Step S207: The configuration analysis module determines whether the error value of the matching point is greater than a predetermined range. If yes, the process proceeds to step S208, the infrared imager is used to capture the target area of the patient to generate an infrared image signal, and the image processing module is configured to generate a new infrared image. (Step S209), that is, once an error has occurred in the use of the visible light image matching region, which is different from the original region of interest or the matching, it is necessary to perform the infrared image capturing and the infrared image segmentation again. And returning to step S201; if it is determined that the error value of the matching point is within the predetermined range, proceeding to step S210, configuring the analysis module to calculate a conversion matrix according to the matching point;

Step S211: The analysis module is configured to apply the transformation matrix to the plurality of key regions of the front frame. It should be noted that the endoscope imaging method of the present application can perform image matching only on the first image of the infrared image. To the visible light collection, the function of the subsequent image alignment is performed by the visible light image matching, so that the subsequent pictures can have the function of displaying the characteristic area, and then the process proceeds to step S203, or directly returns to step S202.

The details of each of the above steps have been described in the foregoing embodiments, and the repeated description thereof is omitted in order to avoid obscuring the present application.

In summary, the endoscope imaging system and method of the present application can provide a clear field of view and accurately mark the organ region in the superimposed image provided to the operator, thereby eliminating the need to repeat the visible light image mode. And the time it takes to switch between infrared image modes further enhances the safety, accuracy and speed of the operation. In addition, in the system, by detecting matching points and errors and calculating the conversion matrix, it is possible to omit the time of retrieving the infrared image. In addition to selectively and accurately displaying key areas, the inter- and system resources can automatically track key areas in subsequent images, further improving image processing speed and saving system resources.

Claims

An endoscope imaging system for displaying images in a patient, characterized by comprising:

Endoscopy camera module, including:

a light source for illuminating the patient;

An infrared imager configured to capture a target area within the patient and generate an infrared image signal;

An image capture module configured to capture the target area and generate a visible light image signal;

The image processing module is electrically connected to the endoscope camera module, and receives the infrared image signal and the visible light image signal to generate an infrared image and a visible light image, and associates the infrared image with the visible light image;

a storage unit electrically connected to the image processing module for storing the infrared image and the visible light image;

An analysis module electrically coupled to the image processing module, configured to analyze the infrared image to generate at least one focus area, and superimpose the focus area on the visible light image associated with the infrared image to generate an overlay Imagery;

The display module is electrically connected to the analysis module and configured to display the superimposed image.
The endoscope imaging system according to claim 1, wherein the analysis module divides the infrared image into a plurality of key regions according to an intensity distribution of the infrared image, and each of the focus regions respectively Corresponds to different intensity ranges.
The endoscope imaging system according to claim 2, wherein said analysis module superimposes said plurality of focus regions in a color patch on said visible light image associated with said infrared image, The superimposed image is generated.
The endoscope imaging system according to claim 2, wherein said analysis module further extracts edges of said plurality of focus regions and said plurality of focuses An edge of the region is superimposed in color lines onto the visible light image associated with the infrared image to produce the superimposed image.
The endoscope imaging system according to claim 1, wherein the analysis module further calculates a matching point between two consecutive frames in the visible light image, and generates a conversion matrix according to the matching point, And applying the conversion matrix to the infrared image that has generated the at least one focus area and corresponding to the previous frame, and superimposing the converted at least one key area on the corresponding The infrared image of the frame to create another superimposed image that is continuous with the superimposed image.
The endoscope imaging system according to claim 5, wherein said analysis module calculates an error value of said matching point, and if said error value exceeds a predetermined range, said infrared imager is configured to re The target object is photographed and another infrared image signal is generated.
An endoscope imaging method suitable for use in an endoscope imaging system according to claim 1, wherein the method comprises the following steps:

Irradiating the target area in the patient's body with the light source;

Taking the infrared imager and the image capturing module to capture the target area in the patient body, and respectively generating an infrared image signal and a visible light image signal;

Configuring the image processing module to receive the infrared image signal and the visible light image signal, and generate an infrared image and a visible light image, and associate the infrared image with the visible light image;

The infrared image and the visible light image are stored by the storage unit;

Configuring the analysis module to analyze the infrared image to generate at least one focus area, and superimposing the focus area on the visible light image associated with the infrared image to generate a superimposed image; and using the display module The superimposed image is displayed.
The endoscope imaging method according to claim 7, wherein in the step of configuring the analysis module to analyze the infrared image to generate the at least one focus area, the analysis module further performs the Intensity distribution of infrared images The infrared image is divided into a plurality of key regions, and each of the key regions respectively corresponds to a different intensity range.
The endoscope imaging method according to claim 8, wherein the analysis module superimposes the plurality of key regions in a color block manner on the visible light image associated with the infrared image, The superimposed image is generated.
The endoscope imaging method according to claim 8, wherein the analysis module further extracts edges of the plurality of focus regions, and stacks edges of the plurality of focus regions in a color line manner. Adding to the visible light image associated with the infrared image to generate the superimposed image.
The endoscope imaging method according to claim 7, wherein after the step of superimposing the focus area on the visible light image associated with the infrared image to generate a superimposed image, the analyzing The module further calculates a matching point between consecutive two frames in the visible light image, and generates a conversion matrix according to the matching point, and applies the conversion matrix to the at least one key area that has been generated and corresponds to the previous The infrared image of the frame, and the converted at least one key area is superimposed on the infrared image corresponding to the subsequent picture frame to generate another continuous with the superimposed image Add a picture.
The endoscope imaging method according to claim 11, wherein said analysis module calculates an error value of said matching point, and if said error value exceeds a predetermined range, said infrared imager is configured to re The target object is photographed and another infrared image signal is generated.