TWI581750B - Endoscope imaging system and method - Google Patents

Description

Endoscope imaging system and method

The present invention relates to an endoscope imaging system and method. More specifically, the present invention relates to an endoscope imaging system for displaying superimposed images by analyzing a key area of an infrared image and superimposing it on a visible light image. method.

Endoscopic surgery is often performed under visible light, where visible light allows the user to see the surface of the surgical anatomy during the procedure. In order to make the image below the surface of the organ visible, a special light source such as an infrared source is required. To this end, the patient will be injected with Indocyanine green (ICG) dye before surgery, so that the operator can see the infrared light in the infrared image mode during the operation, such as lymph node resection and bile duct surgery. Marked areas containing lymph nodes or bile ducts. However, in the infrared image mode, the marked area cannot be seen, and it is difficult to perform surgery in the infrared light image mode.

Therefore, during the operation of the operation, the operator needs to continuously switch between the infrared image mode and the visible image mode, which not only causes inconvenience to the operator, but also prolongs the operation time and increases the chance of the patient being infected.

At present, the solution to the above problem is to directly implant the infrared image into the visible light image. However, in such an image display mode, for the operator, the displayed image will be It is difficult to identify, thus improving the difficulty of surgery. For this reason, there is an urgent need for an endoscope imaging technology that can automatically superimpose infrared images onto visible light images, and can selectively and accurately display key areas, and automatically track key areas in subsequent images.

In order to solve the above problems, an object of the present invention is to provide an endoscope imaging system suitable for displaying images in a patient, including an endoscope photography module, an image processing module, a memory unit, an analysis module, and a display. Module. The endoscope photography module includes a light source, an infrared imager, and an image capturing module. The light source is used to illuminate the patient. The infrared imager is configured to capture a target area within the patient and generate an infrared image signal. The image capture module is configured to capture a target area and generate a visible light image signal. The image processing module is electrically connected to the endoscope photography module, which receives the infrared image signal and the visible light image signal and generates the infrared image and the visible light image, and associates the infrared image with the visible light image. The memory unit is electrically connected to the image processing module for storing infrared images and visible light images. The analysis module is electrically connected to the image processing module and configured to analyze the infrared image to generate at least one key area, and superimpose the key area on the visible light image associated with the infrared image to generate a superimposed image. The display module is electrically connected to the analysis module and configured to display the superimposed image.

Preferably, the analysis module divides the infrared image into a plurality of key regions according to the intensity distribution of the infrared image, and each of the key regions respectively corresponds to a different intensity range.

Preferably, the analysis module can superimpose the plurality of key regions in a color block to the visible light image associated with the infrared image to generate the superimposed image.

Preferably, the analysis module can further capture the edges of the plurality of key regions, and superimpose the edges of the plurality of key regions as color lines to the visible light image associated with the infrared image to generate the superimposed image.

Preferably, the analysis module can further calculate a matching point between two consecutive frames in the visible light image, and generate a conversion matrix according to the matching point, and apply the transformation matrix to the at least one key region and corresponding to the previous The infrared image of the frame is superimposed and the converted at least one focus area is superimposed on the infrared image corresponding to the subsequent frame to generate another superimposed image continuous with the superimposed image.

Preferably, the analysis module can calculate the error value of the matching point. If the error value exceeds the predetermined range, the infrared imager can be configured to re-shoot the target object and generate another infrared image signal.

According to another aspect of the present invention, an endoscope imaging method is provided, which is applicable to the aforementioned endoscope imaging system, comprising the steps of: illuminating a target area in a patient with a light source; and using an infrared imager and an image capturing mode The group captures the target area of the patient and generates infrared image signals and visible light image signals respectively; the image processing module receives the infrared image signals and the visible light image signals and generates infrared images and visible light images, and associates the infrared images with the visible light images; Storing the infrared image and the visible light image by using the memory unit; configuring the analysis module to analyze the infrared image to generate at least one key area, and superimposing the key area on the visible light image associated with the infrared image to generate the superimposed image; and displaying the superimposed image by the display module image.

Preferably, in the step of configuring the analysis module to analyze the infrared image to generate at least one key area, the analysis module further divides the infrared image into a plurality of key areas according to the intensity distribution of the infrared image, and each key area is separately Corresponds to different intensity ranges.

Preferably, the analysis module superimposes a plurality of key regions in a color block on the visible light image associated with the infrared image to generate a superimposed image.

Preferably, the analysis module further extracts edges of the plurality of key regions, and superimposes the edges of the plurality of key regions by color lines to the visible light images associated with the infrared images to generate the superimposed images.

Preferably, after the step of superimposing the focus area on the visible light image associated with the infrared image to generate a superimposed image, the analysis module further calculates a matching point between the two consecutive frames in the visible light image, and generates a matching point according to the matching point. Converting a matrix, and applying the transformation matrix to the infrared image that has generated at least one focus area and corresponding to the preceding frame, and superimposing the converted at least one key area on the infrared image corresponding to the subsequent frame Produces another superimposed image that is continuous with the superimposed image.

Preferably, the analysis module calculates the error value of the matching point. If the error value exceeds the predetermined range, the infrared imager is configured to re-shoot the target object and generate another infrared image signal.

In summary, with the endoscope imaging system and method of the present invention, in the superimposed image provided to the operator, the clear field of view and the accurate marking of the organ region can be omitted, and the repeated image in visible light mode and infrared can be omitted. Switching between optical image modes takes time to further improve the safety, accuracy and speed of the operation. In addition, in the system, by detecting the matching points and errors and calculating the conversion matrix, the time and system resources for retrieving the infrared image can be omitted, and in addition to selectively displaying the key areas, the images can be automatically displayed in subsequent images. Tracking key areas can further increase image processing speed and save system resources.

1‧‧‧Endoscope Imaging System

100‧‧‧Light source

102‧‧‧Endoscope photography module

104‧‧‧Infrared imager

106‧‧‧Image capture module

108‧‧‧Image Processing Module

110‧‧‧ memory unit

116‧‧‧Analysis module

118‧‧‧Display module

120‧‧‧Power Module

A, A', A", B, B', B", A1, A1', A2, A2'‧‧‧ areas

CP‧‧‧match point

The above and other features and advantages of the present invention will become more apparent from the detailed description of the exemplary embodiments illustrated in the accompanying drawings in which: FIG. 1 is an embodiment of an endoscope imaging system according to the present invention. The block diagram shown.

Figure 2 is an example of an infrared image of an embodiment of an endoscope imaging system in accordance with the present invention.

Figure 3 is an illustration of a visible light image of an embodiment of an endoscope imaging system in accordance with the present invention.

Figure 4 is an illustration of a processed infrared image of an embodiment of an endoscope imaging system in accordance with the present invention.

Figure 5 is an example of a superimposed image produced by an embodiment of an endoscope imaging system in accordance with the present invention.

Figure 6 is an illustration of a processed infrared image produced by another embodiment of an endoscope imaging system in accordance with the present invention.

Figure 7 is an illustration of a processed infrared image produced by a further embodiment of an endoscope imaging system in accordance with the present invention.

Figure 8 is a schematic diagram of the calculation of matching points for an endoscope imaging system in accordance with the present invention.

Figure 9 is an example of a superimposed image produced by another embodiment of endoscopic imaging in accordance with the present invention.

Fig. 10 is an example of superimposed images produced by still another embodiment of endoscopic imaging according to the present invention.

Figure 11 is a flow chart showing an embodiment of an endoscope imaging method according to the present invention.

Figure 12 is a flow chart showing another embodiment of an endoscope imaging method according to the present invention.

The technical features, contents, and advantages of the present invention, as well as the advantages thereof, can be understood by the present inventors, and the present invention will be described in detail with reference to the accompanying drawings. The subject matter is only for the purpose of illustration and description. It is not intended to be a true proportion and precise configuration after the implementation of the present invention. Therefore, the scope and configuration relationship of the attached drawings should not be interpreted or limited. First described.

As used herein, the term "and/or" includes any and all combinations of one or more of the associated listed items. When the phrase "at least one of" is preceded by a list of elements, the entire list of elements is modified instead of the individual elements in the list.

Embodiments of the endoscope imaging system of the present invention will be described in detail below with reference to the accompanying drawings. Please refer to FIG. 1, which is a block diagram of an embodiment of an endoscope imaging system in accordance with the present invention. The endoscope imaging system 1 of the present invention is suitable for displaying images in a patient, and includes an endoscope camera module 102, an image processing module 108, a memory unit 110, an analysis module 116, a display module 118, and a power source. Module 120. The endoscope photography module 102 includes a light source 100, an infrared imager 104, and an image capturing module 106. The light source 100 is configured to illuminate a patient's body, and the infrared imager 104 is configured to capture a target area in the patient's body. The light source 100 may include a visible light source and an infrared light source, and the infrared imager 104 receives the infrared light irradiated by the target area reflected light source 100. And generate infrared image signals. The infrared image capturing system uses a photographing or photographing device to extract light waves of an infrared light field generated by reflection of an object after being irradiated by an infrared light source.

Please refer to FIG. 2, which is an example of an infrared image of an embodiment of an endoscope imaging system in accordance with the present invention. Before the operation, the patient will inject the Indocyanine green (ICG) dye several hours ago, so that the surgeon can see the dye in the infrared image mode during the operation, such as lymph node resection and bile duct surgery. Marked areas containing lymph nodes or bile ducts. as the picture shows, The area A of the fluorescent display is the effect of the fluorescent dye illuminating, but not the organ area, for example, the area B is faint.

In other words, the image capture module 106 is configured to capture a target area and generate a visible light image signal. The image processing module 108 is electrically connected to the endoscope camera module, and receives the infrared image signal and the visible light image signal, and generates the infrared image 112 and the visible light image 114, and associates the infrared image 112 with the visible light image 114. It should be noted that the visible light image 114 is usually an image with a continuous frame, where the contact relationship refers to the relationship between the single frames, that is, the infrared image 112 and the visible light image 114 of the first frame. Refer to Juan Wachs et al., Multi-modal registration using a combined similarity measure (Applications of Soft Computing, 2009), which is a well-known technique in the field, and is therefore not in use. This statement. An example of a visible light image can be referred to FIG. 3, which is an example of a visible light image of an embodiment of an endoscope imaging system in accordance with the present invention. In order to perform surgery under a clear image, the operator usually uses visible light images, but there are some shortcomings in which the organs in the patient cannot be seen.

The memory unit 110 is electrically connected to the image processing module 108 for storing the infrared image 112 and the visible light image 114. The analysis module 116 is electrically coupled to the image processing module 108 and configured to analyze the infrared image 112 to generate at least one focus area. For example, please refer to FIG. 4, which is an example of a processed infrared image of an embodiment of an endoscope imaging system in accordance with the present invention. As shown in the figure, the infrared image 112 itself is a grayscale image, and the fourth image shows that the infrared image 112 is represented by blue by the same gray-scale, and the aforementioned region A corresponds to the converted image. The high brightness area A' and the area B correspond to the low brightness area B'. The grayscale image will be stored in the memory unit 110.

Please refer to FIG. 5, which is an example of superimposed images produced by an embodiment of an endoscope imaging system in accordance with the present invention. After obtaining the grayscale image, the analysis module 108 can divide the infrared image 112 In particular, the analysis module 108 can divide the infrared image 112 into a plurality of key regions according to the gray value distribution of the infrared image 112, and each of the key regions respectively corresponds to a different gray value range. As shown in the figure, the infrared image 112 is analyzed by the analysis module 108 and divided into two key areas of the gray value range. For the operator, the interested part is only the organ area A displayed with high brightness. ', therefore, the key area can be divided into two parts, such as area A" and area B". The method for dividing the area of the infrared image 112 can be referred to the "Current methods in medical image segmentation 1, "Annual Review of Biomedical Engineering 2000" by DL Pham et al. Knowing the technology, it is not described here.

The display module 118 is electrically connected to the analysis module 116 to display the superimposed image by superimposing the focus area on the visible light image 114 associated with the infrared image 112 to generate the superimposed image. As shown in the figure, in each key area of the superimposed image, the area A" and the area B" are displayed in a uniform color block, providing the operator with a clear field of view and accurate marking of the organ area, which eliminates the repetition of visible light. The time spent switching between image mode and infrared image mode further enhances the safety, accuracy and speed of the operation.

Please refer to Fig. 6, which is an example of a processed infrared image produced by another embodiment of an endoscope imaging system in accordance with the present invention. As shown, the analysis module 116 can divide the infrared image 112 into three intensity ranges according to the intensity distribution of the infrared image 112. In other words, the number of key regions is three. Among them, the area of lower brightness is displayed in white, and the area of the organ can be divided into area A1 and area A2. Here, the user can set different numbers and different intensity ranges (ie, gray value ranges of grayscale images) according to requirements to improve the flexibility of the system, and is not limited to the description in the specification of the present invention.

Please refer to Fig. 7, which is an example of a processed infrared image produced by a further embodiment of an endoscope imaging system in accordance with the present invention. Except that as shown in Figure 5, the key areas are represented by color blocks. In addition, the analysis module 116 can further capture the edges of the plurality of key regions, such as the edges of the regions A1 and A2 in FIG. 6, the color line regions A1' and A2' as shown in the figure, and the plurality of key regions. The edges are superimposed in color lines onto the visible light image 114 associated with the infrared image 112 to produce a superimposed image. Here, the analysis module 116 may first generate an image containing only color lines and store it to the memory unit 110, and may directly superimpose the visible light image 112 to generate a superimposed image, which is not limited thereto. It is worth mentioning that the method of extracting the edge of the key area can be referred to the "Study and comparison of various image edge detection techniques" (International Journal of Image Processing 2009). It is also known to those skilled in the art and is not described here.

Please refer to Fig. 8, which is a schematic diagram of calculating matching points for the endoscope imaging system according to the present invention. As described above, since the visible light image 114 is usually an image with a continuous frame, although the infrared image 112 and the visible light image 114 of the first frame have been processed, in order to display a continuous image, the second frame is required. The visible light image 114 is analyzed to produce another superimposed image. To this end, the analysis module 116 performs a matching point calculation on the visible light image between successive frames to generate a conversion matrix. The calculation of the matching point needs to first extract the feature points from the visible light image between the continuous frames, and then calculate according to the offset of the feature points to obtain the conversion matrix. The matching point CP can be performed by any feature detection or feature comparison algorithm, and can be referred to Li, Jing and Nigel M. Allinson et al. for a comprehensive review of the local features of computer images (A comprehensive review) "The current local features for computer vision." Neurocomputing 71.10 (2008): 1771-1787), the matching point calculation is shown in Fig. 8, for example.

After obtaining the offset of the feature points, the transformation matrix can be calculated accordingly. Here, the transformation matrix can be a rigid body or non-rigid body transformation matrix between successive frames, which can be optimized by least squares (see Zitova, Barbara, and According to Jan Flusser et al., "Image registration methods: a survey, Image and vision computing 21.11 (2003): 977-1000), the transformation matrix shown in the figure is calculated as follows :

Please refer to FIG. 9 and FIG. 10 together, which are respectively an example of superimposed images generated by another embodiment of the endoscope imaging according to the present invention and another embodiment. By applying this matrix to the aforementioned infrared image represented by color patches or color lines, a superimposed image as shown in Fig. 9 or Fig. 10 can be produced. This superimposed image can replace the previously generated superimposed image to form a superimposed image with continuous images.

According to a preferred embodiment of the present invention, the error value of the matching point is further calculated during the calculation of the matching point. When the analysis module 116 determines that the error value is too large, that is, the matching point offset is too large, or it is difficult to find the same matching point, the configuration system recaptures the infrared image in the patient.

The present invention also provides an endoscope imaging method. Please refer to FIG. 11, which is a flow chart of an embodiment of an endoscope imaging method according to the present invention. As shown in the figure, the endoscope imaging method is applicable to the above-mentioned endoscope imaging system, and includes the following steps: Step S101: illuminating a target area in a patient with a light source; Step S102: using an infrared imager and an image capturing module The target area of the patient is photographed, and the infrared image signal and the visible light image signal are separately generated. Specifically, the infrared image is captured by the infrared imager, and then switched to the image capturing module to capture the visible light image; step S103: After receiving the infrared image signal and the visible light image signal, the image processing module generates an infrared image and a visible light image, and associates the infrared image with the visible light image; Step S104: storing the infrared image and the visible light image by the memory unit; step S105: analyzing the infrared image by the analysis module to generate the key area, and superimposing the key area on the visible light image to generate the superimposed image; Step S106: displaying the superimposed image by the display module image.

Here, the details of each step have been described above, and the repeated description is omitted.

The present invention also provides an endoscope imaging method. Please refer to FIG. 12, which is a flow chart of another embodiment of an endoscope imaging method according to the present invention. As shown, the endoscope imaging method is applicable to the above-described endoscope imaging system, which is continued from step S104 of the previous embodiment, and includes the following steps: Step S201: After obtaining the infrared image, the analysis module divides the infrared image into a plurality of key regions according to the intensity distribution of the infrared image. Specifically, the analysis module 116 can display the infrared image 112 according to the intensity distribution of the infrared image 112. It is divided into several intensity ranges, and the image captured by the infrared image mainly lies in the area of interest of the physician.

Step S202: superimposing a plurality of key regions on the visible light image to generate a superimposed image, wherein the key regions may be represented by color blocks in the superimposed image. In addition, optionally, after step S201, the method may first proceed to step S203, the analysis module further extracts edges of the plurality of key regions, and superimposes edges of the plurality of key regions to the visible light image by color lines to generate Superimpose the image.

Step S204: displaying the superimposed image by the display module. For this reason, only the visible light image of the first frame is currently processed. In order to display the continuous image, the visible image 114 of the second frame needs to be analyzed to generate another superposition. image.

Step S205: capturing a target area in the patient's body by using an image capturing module to generate a visible light image signal; Step S206: After receiving the visible light image signal, the image processing module generates a visible light image, and the analysis module further calculates a matching point and an error value between the adjacent frames; Step S207: The configuration analysis module determines whether the error value of the matching point is greater than a predetermined range. If yes, the process proceeds to step S208, the infrared imager is used to capture the target area in the patient body to generate an infrared image signal, and the image processing module is configured to regenerate the new image. Infrared image (step S209), that is, when an error occurs in the use of the visible light image matching area that is different from the original region of interest or the matching, the infrared image capturing and the infrared image segmentation are required to be performed again. And returning to step S201; if it is determined that the error value of the matching point is within the predetermined range, proceeding to step S210, configuring the analysis module to calculate a conversion matrix according to the matching point; Step S211: The analysis module is configured to apply the transformation matrix to the plurality of key regions of the front frame. It should be noted that the image processing method of the present invention can only perform image matching on the first image of the infrared image. Switching to visible light acquisition, the function of subsequent image alignment is performed by matching the visible light image, so that the subsequent pictures can have the function of displaying the feature area, and then the process proceeds to step S203, or directly returns to step S202.

The details of each of the above steps have been described in the foregoing embodiments, and the repeated description thereof is omitted in order to avoid obscuring the present invention.

In summary, with the endoscope imaging system and method of the present invention, in the superimposed image provided to the operator, the clear field of view and the accurate marking of the organ region can be omitted, and the repeated image in visible light mode and infrared can be omitted. Switching between optical image modes takes time to further improve the safety, accuracy and speed of the operation. In addition, in the system, by detecting matching points and errors and calculating the conversion matrix, the time and system resources for retrieving the infrared image can be eliminated, except that it can be selectively and accurately Display key areas, and automatically track key areas in subsequent images, further improving image processing speed and saving system resources.

1‧‧‧Endoscope Imaging System

100‧‧‧Light source

102‧‧‧Endoscope photography module

104‧‧‧Infrared imager

106‧‧‧Image capture module

108‧‧‧Image Processing Module

110‧‧‧ memory unit

116‧‧‧Analysis module

118‧‧‧Display module

120‧‧‧Power Module

Claims (10)

An endoscope imaging system for displaying an image of a patient, comprising: an endoscope photography module comprising: a light source for illuminating the patient; an infrared imager configured to capture the image a target area of the patient and generating an infrared image signal; and an image capture module configured to capture the target area and generate a visible light image signal; an image processing module electrically connected to the internal view The photographic camera module receives the infrared image signal and the visible light image signal and generates an infrared image and a visible light image, and associates the infrared image with the visible light image; a memory unit is electrically connected to the image processing mode The group is configured to store the infrared image and the visible light image; an analysis module is electrically connected to the image processing module, configured to analyze the infrared image to generate at least one key area, and superimpose the key area The visible light image associated with the infrared image to generate a superimposed image; and a display module electrically connected The analysis module is configured to display the superimposed image, wherein the analysis module further calculates a matching point between two consecutive frames in the visible light image, and generates a conversion matrix according to the matching point, and applies the conversion matrix Forming the infrared image of the at least one focus area and corresponding to the preceding frame, and superimposing the converted at least one key area on Corresponding to the infrared image of the frame afterwards to generate another superimposed image continuous with the superimposed image. The endoscope imaging system of claim 1, wherein the analysis module divides the infrared image into a plurality of key regions according to an intensity distribution of the infrared image, and each of the key regions respectively corresponds to a different The range of strengths. The endoscope imaging system of claim 2, wherein the analysis module superimposes the plurality of key regions in a color block on the visible light image associated with the infrared image to generate the superimposed image. . The endoscope imaging system of claim 2, wherein the analysis module further extracts edges of the plurality of key regions, and superimposes edges of the plurality of key regions by color lines to The infrared image is associated with the visible light image to generate the superimposed image. The endoscope imaging system of claim 1, wherein the analysis module calculates an error value of the matching point, and if the error value exceeds a predetermined range, the infrared imager is configured to retake the target The object produces another infrared image signal. An endoscope imaging method is applicable to an endoscope imaging system according to claim 1, which comprises the steps of: illuminating a target area in a patient with the light source; The image capturing module captures the target area of the patient and generates an infrared image signal and a visible light image signal respectively. The image processing module is configured to receive the infrared image signal and the visible light image signal and generate an infrared image and a a visible light image, and correlating the infrared image with the visible light image; Storing the infrared image and the visible light image by the memory unit; configuring the analysis module to analyze the infrared image to generate at least one focus area, and superimposing the focus area on the visible light image associated with the infrared image to generate an overlay And displaying the superimposed image by the display module, wherein after the step of superimposing the focus area on the visible light image associated with the infrared image to generate a superimposed image, the analyzing module further calculates the visible image a matching point between two consecutive frames, and generating a conversion matrix according to the matching point, and applying the conversion matrix to the infrared image that has generated the at least one focus area and corresponding to the previous frame, and The converted at least one focus area is superimposed on the infrared image corresponding to the subsequent frame to generate another superimposed image continuous with the superimposed image. The method of imaging an endoscope according to claim 6, wherein in the step of configuring the analysis module to analyze the infrared image to generate the at least one focus area, the analysis module is further based on the infrared image The intensity distribution divides the infrared image into a plurality of key regions, and each of the key regions respectively corresponds to a different intensity range. The method for imaging an endoscope according to claim 7, wherein the analysis module superimposes the plurality of key regions as color patches on the visible light image associated with the infrared image to generate the superimposed image. . The method for imaging an endoscope according to claim 7, wherein the analysis module further extracts edges of the plurality of key regions, and superimposes edges of the plurality of key regions by color lines to The infrared image is associated with the visible light image to generate the superimposed image. An endoscope imaging method according to claim 9, wherein the analysis mode The group calculates an error value for the matching point, and if the error value exceeds a predetermined range, the infrared imager is configured to retake the target object and generate another infrared image signal.