WO2023168751A1 - Image fusion method and apparatus for double-fluorescence endoscope, and electronic device - Google Patents

Abstract

The present invention belongs to the field of image processing. Disclosed are an image fusion method and apparatus for a double-fluorescence endoscope, and an electronic device. The method comprises: acquiring an original white-light image and an original fluorescence image; adjusting the original fluorescence image so as to convert one of two types of pixels, in which there are more blue components than red components or there are less blue components than red components, in the original fluorescence image into green, convert the other type thereof into blue, and convert overexposed pixels in the original fluorescence image into cyan, so as to obtain a fluorescence-adjusted image; superimposing the original white-light image and the fluorescence-adjusted image; and outputting a fused image. By means of the method, instead of directly displaying the fluorescence color of a fluorescent dye, one of a region which is blue or purple-blue and a region which is red or fuchsia in the original fluorescence image is converted into blue, and the other region is converted into green, such that human eyes can more easily distinguish a boundary between the two fluorescence regions, and it is conducive for a doctor to clearly distinguish a boundary between a fluorescence-overexposed region and a fluorescence-free region.

Description

Dual fluorescence endoscope image fusion method, electronic equipment and device Technical field

The invention relates to a dual fluorescence endoscope image fusion method, electronic equipment and device, and belongs to the field of image processing.

Background technique

As minimally invasive technology continues to mature, more and more minimally invasive surgeries are replacing traditional surgical operations, and minimally invasive medical technology has become a new direction in medical development. Endoscopic minimally invasive medical surgery has the characteristics of small trauma, short operation, and fast postoperative recovery. It is favored by both doctors and patients. The endoscope market has also developed rapidly. Endoscope technology has been promoted to otolaryngology, general care, and other departments. Surgery, obstetrics and gynecology, thoracic surgery, urology, etc.

Traditional single-camera endoscopes only have white light images, which makes it difficult to identify the specific location of the lesion. It requires very high professionalism and clinical experience of doctors. Later, a new medical endoscope camera technology with dual cameras of white light and fluorescence appeared, and the imaging effect was better than that of a single-camera white light endoscope. White light is used to display basic images, and fluorescence is used to display the location of the lesion and the clear edge of the lesion, allowing doctors to handle the lesion clearly and intuitively, improving the doctor's surgical efficiency and success rate, and greatly reducing the difficulty of the doctor's surgery.

Fluorescence navigation endoscopy systems have been widely used in surgeries, providing effective guidance for intraoperative tumor marking and cholangiography in gynecological and hepatobiliary surgeries. In recent years, the dual-fluorescence navigation endoscope system has been introduced. With the fluorescent labeling of two dyes with different fluorescent colors, two target areas can be dyed and labeled with two dyes simultaneously, further enriching the application of fluorescence navigation. However, there are certain drawbacks to using two fluorescent dyes. For example, dual fluorescent endoscopes commonly use ICG (Indocyanine Green) and MB (Methylene Blue) as fluorescent dyes. The fluorescence color of ICG is purple-blue and the fluorescence color of MB is purple. Red, resulting in two problems: first, the distinction between purple-blue and purple-red is insufficient. After the fluorescence image and the white light image are fused, the distinction between the two fluorescent areas will be further reduced, affecting the boundary judgment; second Second, the purple-red color of the original MB fluorescence signal is not much different from the red color of the animal tissue itself. It is more difficult to distinguish after fusion with the white light image through ordinary fusion methods. Low boundary discrimination will greatly affect the accuracy of fluorescence surgery.

Contents of the invention

In order to overcome the shortcomings of the existing technology, the present invention provides a dual-fluorescence endoscope image fusion method, electronic equipment and device to clearly distinguish the boundaries of two fluorescent areas, the boundaries of fluorescent areas and non-fluorescent areas.

In the first aspect, this application provides a dual-fluorescence endoscope image fusion method, which includes the following steps:

Acquire white light original images and fluorescence original images;

The fluorescence original image is adjusted to convert one of the two categories of pixels in the fluorescence original image with more blue components than red components and less blue components than red components into green and the other into blue, and convert the overexposed pixels in the original fluorescence image to cyan to obtain a fluorescence adjusted image;

Superimpose the original white light image and the fluorescence adjusted image to obtain a fused image;

Output the fused image.

The dual fluorescence endoscope image fusion method provided by this application makes one of the blue, purple-blue areas and red, purple-red areas in the fluorescence image become blue, and the other becomes green, making the process The exposed area turns cyan, which is helpful to clearly distinguish the boundaries between the two fluorescent areas, the fluorescent area and the non-fluorescent area.

Optionally, the fluorescence original image is adjusted to convert one of the two types of pixels in the fluorescence original image, in which the blue component is more than the red component and the blue component is less than the red component, into green, and the other type. Convert the class to blue, and convert the overexposed pixels in the original fluorescence image to cyan. The steps to obtain the fluorescence adjusted image include:

Traverse the original fluorescence image to obtain the second red value, the second green value and the second blue value of each pixel position;

Calculate the hue value of the corresponding pixel position in the HSV color model according to the second red value, the second green value and the second blue value;

Calculate the color adjustment coefficient of the corresponding pixel position according to the ratio of the second red value to the second blue value and the hue value;

Calculate the fluorescence-adjusted green value and the fluorescence-adjusted blue value of the corresponding pixel position according to the color adjustment coefficient, so that the fluorescence-adjusted red value of the corresponding pixel position is 0;

The fluorescence-adjusted red value, fluorescence-adjusted green value and fluorescence-adjusted blue value of each pixel position are recombined to obtain the fluorescence-adjusted image.

Optionally, the fluorescence original image is adjusted to convert one of the two types of pixels in the fluorescence original image, in which the blue component is more than the red component and the blue component is less than the red component, into green, and the other type. Convert the class to blue, and convert the overexposed pixels in the original fluorescence image to cyan. The steps to obtain the fluorescence adjusted image include:

Traverse the original fluorescence image to obtain the second red value and the second blue value of each pixel position;

Calculate the corresponding pixel position color adjustment coefficient according to the percentage of the second red value to the sum of the second red value and the second blue value;

Calculate the fluorescence-adjusted green value and the fluorescence-adjusted blue value of the corresponding pixel position according to the color adjustment coefficient, so that the fluorescence-adjusted red value of the corresponding pixel position is 0;

The fluorescence-adjusted red value, fluorescence-adjusted green value and fluorescence-adjusted blue value of each pixel position are recombined to obtain the fluorescence-adjusted image.

The percentage of the second red value in the sum of the second red value and the second blue value can reflect whether there are more red components or more blue components in a certain pixel, and thus two types of pixels can be divided and combined. Convert colors.

Further, in the step of traversing the fluorescence original image to obtain the second red value and the second blue value of each pixel position, a second green value of each pixel position in the fluorescence original image is also obtained;

Before calculating the color adjustment coefficient of the corresponding pixel position according to the percentage of the second red value to the sum of the second red value and the second blue value, the step further includes:

Calculate the hue value of the corresponding pixel position in the HSV color model according to the second red value, the second green value and the second blue value;

The step of calculating the color adjustment coefficient of the corresponding pixel position based on the percentage of the second red value to the sum of the second red value and the second blue value includes:

The corresponding pixel position color adjustment coefficient is calculated according to the percentage of the second red value to the sum of the second red value and the second blue value and the hue value.

The overexposed area of the fluorescent original image is whitened, with as much blue component as red component, and will be converted into cyan. In fact, some purple pixels also have as much blue component as red component, so the purple area and the overexposed area need to be converted into cyan. To distinguish them, the hue value is taken into account when calculating the color adjustment coefficient. The hue values of whitening and purple are different, so they can be distinguished.

Optionally, the fluorescence original image is adjusted to convert one of the two types of pixels in the fluorescence original image, in which the blue component is more than the red component and the blue component is less than the red component, into green, and the other type. Convert the class to blue, and convert the overexposed pixels in the original fluorescence image to cyan. The steps to obtain the fluorescence adjusted image include:

Traverse the original fluorescence image to obtain the second red value and the second blue value of each pixel position;

Construct a sigmoid function with the dependent variable as the color adjustment coefficient, and the independent variable of the sigmoid function is the ratio of the second red value and the second blue value at the corresponding pixel position;

Calculate the fluorescence-adjusted green value and the fluorescence-adjusted blue value of the corresponding pixel position according to the color adjustment coefficient, so that the fluorescence-adjusted red value of the corresponding pixel position is 0;

The fluorescence-adjusted red value, fluorescence-adjusted green value and fluorescence-adjusted blue value of each pixel position are recombined to obtain the fluorescence-adjusted image.

Optionally, the color adjustment coefficient ranges from 0 to 1, and the step of calculating the fluorescence-adjusted green value and the fluorescence-adjusted blue value of the corresponding pixel position according to the color adjustment coefficient includes:

Let one of the fluorescence adjustment green value and the fluorescence adjustment blue value of the corresponding pixel position be adjusted to the color adjustment coefficient multiplied by the gray level, and the other value is adjusted to 1 minus the difference of the color adjustment coefficient multiplied by the gray level. degree level.

Optionally, there are steps before superimposing the white light original image and the fluorescence adjusted image to obtain the fused image:

Traverse the original fluorescence image to obtain the second red value, the second green value and the second blue value of each pixel position;

Calculate the fluorescence signal intensity of the corresponding pixel position according to the second red value, the second green value and the second blue value;

The step of superposing the white light original image and the fluorescence adjusted image to obtain a fused image includes:

The fluorescence adjusted image is adjusted according to the fluorescence signal intensity to obtain the fluorescence part of the fused image, the white light original image is adjusted according to the fluorescence signal intensity to obtain the white light part of the fused image, and the white light part of the fused image and the fused image are superimposed. The fluorescent part of the image is combined into a fused image.

Optionally, the fluorescence adjusted image is adjusted according to the fluorescence signal intensity to obtain the fluorescence part of the fused image, the white light original image is adjusted according to the fluorescence signal intensity to obtain the white light part of the fused image, and the white light part of the fused image is superimposed. The white light part and the fluorescence part of the fused image, the steps of obtaining the fused image include:

Traverse the white light original image to obtain the first red value, the first green value and the first blue value of each pixel position;

Subtract the fluorescence signal intensity from 1 to obtain the white light signal intensity at the corresponding pixel position;

The first red value, the first green value, and the first blue value are multiplied by the white light signal intensity respectively to obtain the red value of the white light part, the green value of the white light part, and the blue value of the white light part at the corresponding pixel position. value;

Obtain the fluorescence-adjusted green value and the fluorescence-adjusted blue value of each pixel position of the fluorescence-adjusted image;

The fluorescence-adjusted green value and the fluorescence-adjusted blue value are multiplied by the fluorescence signal intensity respectively to obtain the fluorescence part green value and the fluorescence part blue value of the corresponding pixel position;

Taking the red value of the white light part as the third red value of the corresponding pixel position of the fused image, superimposing the green value of the white light part and the green value of the fluorescent part to obtain the third green value of the corresponding pixel position of the fused image, superposition The blue value of the white light part and the blue value of the fluorescent part obtain the third blue value of the corresponding pixel position of the fused image;

The fused image is composed according to the third red value, the third green value and the third blue value at each pixel position.

In a second aspect, the present application provides an electronic device, including a processor and a memory. The memory stores computer-readable instructions. When the computer-readable instructions are executed by the processor, the operation is as in the first aspect. steps in the method.

In a third aspect, this application provides a dual-fluorescence endoscope image fusion device, including:

Acquisition module, used to acquire white light original images and fluorescence original images;

Adjustment module, used to adjust the fluorescence original image to convert one of the two types of pixels in the fluorescence original image with more blue components than red components and less blue components than red components into green and the other type. Convert to blue, and convert the overexposed pixels in the original fluorescence image to cyan to obtain a fluorescence adjusted image;

A superposition module, used to superimpose the white light original image and the fluorescence adjusted image to obtain a fused image;

An output module is used to output the fused image.

The beneficial effects of the present invention are: the dual fluorescence endoscope image fusion method of the present invention does not directly display the fluorescent color of the fluorescent dye, but integrates the blue and purple-blue areas and the red and purple-red areas in the original fluorescence image. One area turns blue and the other turns green. It is easier for the human eye to distinguish the boundaries of the two fluorescent areas, and the converted color is very easy to distinguish from the red and pink colors of human tissue. In addition, fluorescence The overexposed area in the image is converted from white to cyan, which can be easily distinguished from the white light image, which helps doctors clearly distinguish the boundary between the fluorescent overexposed area and the non-fluorescent area.

Additional features and advantages of the application will be set forth in the description which follows, and in part will be apparent from the description, or may be learned by practice of the application. The objectives and other advantages of the application may be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings.

Description of the drawings

Figure 1 is a flow chart of a dual-fluorescence endoscope image fusion method provided by an embodiment of the present application.

Figure 2 is a structural diagram of a dual-fluorescence endoscope image fusion device provided by an embodiment of the present application.

Figure 3 is a structural diagram of an electronic device provided by an embodiment of the present application.

Figure 4 is a function graph of equations 8 and 10.

Figure 5 is the location distribution diagram of dropping different concentrations of ICG and MB.

Label description: 201. Acquisition module; 202. Adjustment module; 203. Overlay module; 204. Output module; 301. Processor; 302. Memory; 303. Communication bus.

Detailed ways

Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements with the same or similar functions. The embodiments described below with reference to the drawings are exemplary and are only used to explain the present invention and are not to be construed as limitations of the present invention.

The following disclosure provides many different embodiments or examples of various structures for implementing the invention. In order to simplify the disclosure of the present invention, the components and arrangements of specific examples are described below. Of course, they are merely examples and are not intended to limit the invention. Furthermore, the present invention may repeat reference numbers and/or reference letters in different examples, such repetition being for purposes of simplicity and clarity and does not itself indicate a relationship between the various embodiments and/or arrangements discussed.

In the existing technology, when the fluorescence colors of the two fluorescent dyes used in dual fluorescence endoscopes are purple-blue and purple-red respectively, the boundaries of the two fluorescent areas are difficult to distinguish. When the fluorescence signal is weak, the purple-red fluorescence The red and pink colors of the area and the human tissue itself are difficult to distinguish. In addition, no matter what color the fluorescent area is (this problem exists for all fluorescent dyes), when the fluorescence signal is strong, the fluorescent area will be overexposed and whitened. After fusion with the white light image, it will be similar to the non-fluorescent area, and the fluorescence signal and marker concentration will be positive. Correlated, that is, areas with extremely high concentration of markers are easily misjudged as non-fluorescent areas. Based on these shortcomings of dual fluorescent endoscopes, this application provides a dual fluorescent endoscope image fusion method as follows.

Referring to Figure 1, the steps include:

S1: Acquire the original white light image and the original fluorescence image.

S2: Adjust the fluorescence original image to convert one of the two types of pixels in the fluorescence original image with more blue components than red components and less blue components than red components into green and the other into blue, and convert The overexposed pixels in the original fluorescence image are converted to cyan to obtain a fluorescence-adjusted image.

S3: Superimpose the original white light image and the fluorescence adjusted image to obtain a fused image.

S4: Output the fused image.

The area that originally appeared purple-blue in the original fluorescence image appears blue or green in the fused image, and the area that originally appeared purple-red in the original fluorescence image appears blue or green in the fused image. , human tissue appears red in the fused image, corresponding to the hue circle. These three colors are 120° apart from each other and are easy for the human eye to distinguish. The overexposed area appears cyan between blue and green, which can be clearly distinguished from the white light area. The distinction is helpful for doctors to clearly distinguish the boundaries between fluorescent overexposed areas and non-fluorescent areas.

The specific steps of step S2 include but are not limited to the following four calculation forms.

The first:

S211: Traverse the original fluorescence image to obtain the second red value r2, the second green value g2 and the second blue value b2 at each pixel position.

S212: Calculate the hue value H of the corresponding pixel position in the HSV color model based on the second red value r2, the second green value g2 and the second blue value b2. Converting the RGB color model to the HSV color model is a well-known technology. This embodiment only uses the H value in the HSV color model and does not involve the S value (saturation) and V value (brightness). The calculation method is as follows:

In the formula: R, G, and B respectively represent the R value, G value, and B value in the RGB value of a certain pixel, which correspond to r2, g2, and b2 during calculation. Cmax is the largest one among the R value, G value, and B value, and Cmin It is the smallest one among R value, G value and B value.

S213: Calculate the color adjustment coefficient β of the corresponding pixel position based on the ratio of the second red value r2 to the second blue value b2 and the hue value H.

The specific calculation steps can be:

Hc=H/360×255; Formula 2

The value range of the H value in the HSV model is 0 to 360. Hc represents the hue value after the range conversion. After the range conversion, the value range becomes 0 to 255.

Hl＝min[m,max(n,Hc)]; Formula 3

Hl represents the hue value after the range is limited. The hue value Hc after the conversion range is limited to the interval [m, n], where n is the artificial fixed value of Hc near the blue area on the hue circle, and m is the value of Hc on the hue circle. (After conversion by Equation 2, the range of the hue circle becomes 0 to 255) The artificial fixed value near the upper purple-red area is m>n. In this embodiment, n is 170 and m is 225.

Hs=(Hl-n)/(m-n); Formula 4

Hs represents the stretched hue value, and the value range after stretching becomes 0 to 1.

When b2 is 0, b2=1 is mandatory, and r2/b2 can also be transformed into (r2+1)/(b2+1).

S214: Calculate the fluorescence-adjusted green value and the fluorescence-adjusted blue value of the corresponding pixel position according to the color adjustment coefficient β, so that the fluorescence-adjusted red value of the corresponding pixel position is 0.

S215: Recombine the fluorescence-adjusted red value, fluorescence-adjusted green value and fluorescence-adjusted blue value of each pixel position to obtain a fluorescence-adjusted image. That is, a pixel includes three sub-pixels, and the color values of the three sub-pixels are the fluorescence-adjusted red value, the fluorescence-adjusted green value, and the fluorescence-adjusted blue value. Calculate the color values of the three sub-pixels of each pixel and calculate them by position. The fluorescence adjustment image is arrayed into a fluorescence adjustment image. The fluorescence adjustment image is not used for display, but is only an intermediate process of image processing.

The second type:

S221: Traverse the original fluorescence image to obtain the second red value r2 and the second blue value b2 at each pixel position.

S222: Calculate the corresponding pixel position color adjustment coefficient β according to the percentage of the second red value r2 to the sum of the second red value r2 and the second blue value b2.

That is β=r2/(r2+b2); Formula 6

S223: Calculate the fluorescence adjustment green value and fluorescence adjustment blue value of the corresponding pixel position according to the color adjustment coefficient, so that the fluorescence adjustment red value of the corresponding pixel position is 0;

S224: Recombine the fluorescence-adjusted red value, fluorescence-adjusted green value and fluorescence-adjusted blue value of each pixel position to obtain a fluorescence-adjusted image.

The third type is optimized based on the second type:

S231: When traversing the original fluorescence image to obtain the second red value r2 and the second blue value b2 at each pixel position, the second green value g2 at each pixel position in the original fluorescence image is also obtained. Equivalent to step S211.

S232: Calculate the hue value H of the corresponding pixel position in the HSV color model based on the second red value r2, the second green value g2, and the second blue value b2. The calculation method is the same as Equation 1.

S233: Calculate the color adjustment coefficient of the corresponding pixel position based on the percentage of the second red value r2 to the sum of the second red value r2 and the second blue value b2 and the hue value H. Calculate Hc and Hs first from Equation 2 to Equation 4, where n can be 178 and m can be 230, and then calculate β according to Equation 7.

Then steps S214 and S215 are performed.

The fourth type:

S211: Traverse the original fluorescence image to obtain the second red value r2 and the second blue value b2 at each pixel position.

S242: Construct a sigmoid function using the dependent variable as the color adjustment coefficient. The independent variable of the sigmoid function is the ratio of the second red value and the second blue value of the corresponding pixel position.

The sigmoid function is a commonly used formula in logic functions, and its prototype is shown in Equation 8.

x is the independent variable, σ(x) is the dependent variable, e is the natural constant, and Equation 8 can be changed into Equation 9.

d is used to translate the curve, and μ is used to adjust the stretching amplitude. Combining the application scenarios of this solution to construct a sigmoid function, Equation 9 can be transformed into Equation 10.

That is, μ=9, (-x+d) becomes r2/b2, and when b2 is 0, b2=1 is forced, or r2/b2 can be transformed into (r2+1)/(b2+1).

S214: Calculate the fluorescence adjustment green value and the fluorescence adjustment blue value of the corresponding pixel position according to the color adjustment coefficient, so that the fluorescence adjustment red value of the corresponding pixel position is 0.

S215: Recombine the fluorescence-adjusted red value, fluorescence-adjusted green value and fluorescence-adjusted blue value of each pixel position to obtain a fluorescence-adjusted image.

The color adjustment coefficient β calculated by the above four calculation methods can distinguish the purple-blue and purple-red areas in the original fluorescence image. In the blue to red area of the hue circle, the β value of the bluer part is less than 0.5, and The bluer the beta value, the smaller it is; the redder part's beta value is greater than 0.5, and the redder the beta value is, the greater it is; the overexposed area of the original fluorescence image is white, r2=b2=g2, β=0.5, use the color adjustment coefficient to transform the fluorescence The color of the original image can distinguish the purple-red area, purple-blue area and white (overexposed) area in the fluorescence original image.

Control the color adjustment coefficient in the range of 0 to 1, which can easily achieve color conversion:

Let one of the fluorescence adjustment green value and fluorescence adjustment blue value of the corresponding pixel position be adjusted to the color adjustment coefficient times the gray level, and the other value is adjusted to the difference of 1 minus the color adjustment coefficient times the gray level. Those skilled in the art should understand that the "gray scale" mentioned here refers to the highest level of brightness. Taking 16-bit color as an example, there are a total of 65536 gray levels. The brightness range is from 0 to 65535. 0 means no light at all. The brightest is 65535. It is impossible to take a value of 65536. The purple, blue and blue in the original fluorescence image Convert the area to blue, and convert the purple-red and red areas in the original fluorescence image to green, calculated as Equation 11.

Taking 8-bit color as an example, there are a total of 256 gray levels (0 to 255). The purple-blue and blue areas in the fluorescence original image are converted to green, and the purple-red and red areas in the fluorescence original image are converted to blue. The calculation is as shown in Equation 12.

In the first calculation form of step S2, if β is simply equal to r2/b2-0.5, the range of β exceeds 0 to 1, and the overexposed area and pure purple cannot be distinguished; therefore, in Equation 5, the stretched Hue value correction makes the β value range from 0 to 1 when the original fluorescence image color is within the purple-blue to purple-red area of the hue circle, and can distinguish between overexposed areas and pure purple areas.

In addition to introducing a hue value to prevent r2/b2-0.5 from exceeding 0 to 1, the sigmoid function can also be used to limit the β value, such as in the fourth calculation form of step S2. Referring to Figure 4, the image of the sigmoid function prototype is when the independent variable is equal to 0, the dependent variable is equal to 0.5; when the independent variable is less than 0, the dependent variable range is (0,0.5), and the smaller the independent variable, the closer the dependent variable is to 0 ; When the independent variable is greater than 0, the range of the dependent variable is (0.5, 1), and the smaller the independent variable, the closer the dependent variable is to 1. In the application scenario of this solution, when r2=b2, β=0.5, so the image of the sigmoid function prototype is translated to the right (relative to the plane coordinate system) by 0.5 units, that is, r2/b2-0.5+0.5 =r2/b2, at this time, when r2=b2, β=0.5; when r2>b2, β>0.5, and the redder, the closer β is to 1; when r2<b2, β<0.5, and the bluer , the closer β is to 0. In order to more clearly distinguish the purple-red and purple-blue colors of the original fluorescence image, μ in Equation 9 should be a positive number, so that when r2 and b2 are close, the β value can also deviate far from 0.5. When substituted into Equation 11 or Equation 12, it is reflected as Blue and green are more distinct. In Equation 9, a good stretching effect can be achieved when μ is greater than 5. If μ is too large, the difference in fluorescence intensity between the same fluorescent colors will not be obvious. Therefore, the value range of μ is preferably [5, 20], for example, Equation 10 Medium μ=9.

In the second calculation form of step S2, the range of β value is 0-1, but there is a problem that it cannot distinguish the overexposed area and the pure purple area. Therefore, it is further improved in the third calculation form of step S2, with the help of hue The value differentiates between overexposed areas and pure purple areas. Calculated according to Equation 4, the Hs of the overexposed area is 0, while the Hs of the pure purple area is not 0, so whether it is a purple area or an overexposed area can be reflected in β.

The fluorescence signal intensity α can also be calculated using the second red value r2, the second green value g2 and the second blue value b2, as shown in Equation 13.

The steps to superimpose the white light original image and the fluorescence adjusted image to obtain the fused image include:

The fluorescence adjusted image is adjusted according to the fluorescence signal intensity to obtain the fluorescence part of the fused image. The white light original image is adjusted according to the fluorescence signal intensity to obtain the white light part of the fused image. The white light part of the fused image and the fluorescence part of the fused image are superimposed to obtain the fused image.

The specific steps are to traverse the white light original image to obtain the first red value r1, the first green value g1 and the first blue value b1 at each pixel position.

Subtract the fluorescence signal intensity α from 1 to obtain the white light signal intensity at the corresponding pixel position.

The first red value, the first green value, and the first blue value are multiplied by the white light signal intensity respectively to obtain the red value of the white light part, the green value of the white light part, and the blue value of the white light part at the corresponding pixel position, that is, as shown in Equation 14.

Obtain the fluorescence-adjusted green value and fluorescence-adjusted blue value at each pixel position of the fluorescence-adjusted image. To facilitate subsequent calculations, the following uses 8-bit color as an example, and refers to Equation 12 to obtain the fluorescence-adjusted green value and fluorescence-adjusted blue value.

The fluorescence-adjusted green value and the fluorescence-adjusted blue value are multiplied by the fluorescence signal intensity respectively to obtain the green value of the fluorescence part and the blue value of the fluorescence part at the corresponding pixel position, as shown in Equation 15.

Use the red value of the white light part as the third red value r3 of the corresponding pixel position of the fused image, superimpose the green value of the white light part and the green value of the fluorescent part to obtain the third green value g3 of the corresponding pixel position of the fused image, superimpose the blue value of the white light part and the fluorescent part The blue value is obtained as the third blue value b3 at the corresponding pixel position of the fused image, and r3, g3, and b3 are all integers.

A fused image is formed according to the third red value, the third green value and the third blue value of each pixel position, such as Equation 16 or Equation 17.

Equation 16 is to convert the purple-blue and blue areas in the original fluorescence image to green, and convert the purple-red and red areas in the original fluorescence image to blue on the premise that there are 256 gray levels. Equation 17 is to convert the purple-blue and blue areas in the original fluorescence image to blue, and convert the purple-red and red areas in the original fluorescence image to green on the premise of 256 gray levels.

Effect verification:

On the white board 900, fluorescent dyes of different concentrations are added dropwise according to the positions in Figure 5, among which 901, 902, 903, 904, 905, and 906 are dropped with ICG solution, and the concentration gradually increases from 901 to 906; 907, 908, 909, MB solution is added dropwise at 910, 911, and 912, and the concentration gradually increases from 907 to 912. Illuminate the whiteboard with white light in a darkroom and capture the original white light image. Obtain the color of positions 901 to 912 in the white light image and one of the pixels on the whiteboard 900 as shown in Table 1. It should be noted that the color of the same drop solution position in the captured image is not necessarily highly uniform. Occasionally, the color of the pixels selected in the position with greater concentration will be lighter than that in the position with low concentration.

Table 1 Colors taken from 901 to 912 in the white light original image

Location	r1	g1		b1
901	103	98	68
902	106	99	73
903	108	104	75
904	107	103	74
905	103	102	72
906	101	101	77
907	122	125	118
908	121	123	118
909	122	122	120
910	117	119	116
911	109	124	121
912	78	111	118
900	147	147	147

Irradiate the whiteboard with ultraviolet light in the darkroom to excite the fluorescent dye to emit fluorescence. Photograph the original fluorescence image. Obtain the positions 901 to 912 in the original fluorescence image and the color of one pixel of the whiteboard 900. Process the data according to the first method in step S2 as follows Table 2.

Table 2 Colors taken from 901 to 912 in the original fluorescence image

Location	r2	g2	b2	H	hc	H1	Hs	β
901	17	6	twenty two	281	199	199	0.53	0.47
902	twenty one	12	39	260	184	184	0.25	0.19
903	61	47	124	251	178	178	0.15	0.10
904	100	80	179	252	179	179	0.16	0.12
905	203	193	246	251	178	178	0.15	0.20
906	252	252	252	0	0	170	0	0.5
907	11	3	26	261	185	185	0.27	0.17
908	10	2	twenty three	263	186	186	0.29	0.19
909	28	9	31	292	207	207	0.67	0.61
910	84	39	68	321	227	225	1	0.97
911	206	133	176	325	230	225	1	0.96
912	255	224	255	300	213	213	0.78	0.73
900	0	0	0	0	0	170	0	0.5

Then calculate the third red value, the third green value and the third blue value according to Equation 13 and Equation 16 as shown in Table 3 below.

Table 3 RGB values of fused images

Location	α	r3	g3		b3
901	0.06	97	100	71
902	0.09	96	109	71
903	0.30	75	142	60
904	0.47	57	158	55
905	0.83	17	188	54
906	0.99	1	127	127
907	0.05	116	129	114
908	0.05	115	127	115
909	0.09	111	120	123
910	0.25	88	91	149
911	0.67	36	46	206
912	0.96	3	69	185
900	0	147	147	147

As can be seen from Table 3, the fluorescence intensity of the pixels picked up from positions 901, 902, 907, 908, and 909 is low, and the color of the fused image is similar to the original white light image; positions 903, 904, and 905 are significantly enhanced after the fused image. Greenish cast; The original fluorescent image at position 906 is overexposed and whitened. After fusion, it appears to be cyan with equal amounts of green and blue light, which can be significantly different from the color of the white light original image; Positions 910 and 911 are significantly bluish after being enhanced by the fused image; positions The original fluorescent image of 912 is purple, and both the overexposed 906 and the overexposed 906 are r2=b2. After the fusion image is enhanced, it appears blue, which can be distinguished from the cyan in the overexposed area; no fluorescent dye is added at position 900, and the original fluorescent image Black appears in the fused image, and the color presented in the fused image is consistent with the color in the white light original image. It can be seen that the method of the present invention can enhance the resolution of fluorescence images of ICG and MB, making it easy for the human eye to distinguish the boundaries of the two fluorescent areas, and can enhance the resolution of the non-fluorescent area and the fluorescent overexposed area, making it easier for the human eye to distinguish It is easy to distinguish the boundaries between the fluorescent area and the non-fluorescent area. The color and brightness of the low-fluorescence area and the non-fluorescence area are roughly the same as the original white light image, which is helpful for doctors to see human tissues clearly for smooth operation.

Please refer to Figure 3. Figure 3 is a schematic structural diagram of an electronic device provided by an embodiment of the present application. The present application provides an electronic device, including: a processor 301 and a memory 302. The processor 301 and the memory 302 communicate through a communication bus 303 and /or other forms of connection mechanisms (not shown) to interconnect and communicate with each other. The memory 302 stores a computer program executable by the processor 301. When the computing device is running, the processor 301 executes the computer program to execute the above The method in any optional implementation of the embodiment to achieve the following functions: acquire the original white light image and the original fluorescence image; adjust the original fluorescence image so that the blue component in the fluorescence original image is more blue than the red component and the blue component is One of the two types of pixels with less components than red is converted to green, the other is converted to blue, and the overexposed pixels in the original fluorescence image are converted to cyan to obtain a fluorescence adjusted image; the white light original image and the fluorescence adjustment are superimposed image to obtain the fused image; output the fused image. This allows the doctor to clearly distinguish the boundary between the two fluorescent areas and the boundary between the non-fluorescent areas.

Please refer to Figure 2. Figure 2 is a dual-fluorescence endoscope image fusion device in some embodiments of the present application, including:

The acquisition module 201 is used to acquire the white light original image and the fluorescence original image;

The adjustment module 202 is used to adjust the fluorescence original image to convert one of the two types of pixels in the fluorescence original image, in which the blue component is more than the red component and the blue component is less than the red component, into green and the other into blue. color, and convert the overexposed pixels in the original fluorescence image to cyan to obtain the fluorescence adjusted image;

The superposition module 203 is used to superimpose the white light original image and the fluorescence adjusted image to obtain a fused image;

The output module 204 is used to output the fused image.

The application scenarios of the present invention are not limited to fluorescent dyes such as indocyanine green and methylene blue. The present invention can also be applied to other fluorescent dyes with fluorescent colors ranging from blue to purple and purple to red on the hue circle. Fluorescent dyes with fluorescent colors of orange, yellow, green, and cyan can also be mapped to the blue-violet area and purple-red area on the hue circle through hue value calculation, and the present invention can be used to solve the problem of overexposure and whitening.

In the description of this specification, reference to the description of the terms "one embodiment," "certain embodiments," "illustrative embodiments," "examples," "specific examples," or "some examples" or the like is meant to be in conjunction with the described embodiments or Examples describe specific features, structures, materials, or characteristics that are included in at least one embodiment or example of the invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

The above is the preferred embodiment of the present invention. It should be pointed out that for those of ordinary skill in the art, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications are also regarded as It is the protection scope of the present invention.

Claims (10)

1. A dual-fluorescence endoscope image fusion method, characterized by including the following steps:

   Acquire white light original images and fluorescence original images;

   The fluorescence original image is adjusted to convert one of the two categories of pixels in the fluorescence original image with more blue components than red components and less blue components than red components into green and the other into blue, and convert the overexposed pixels in the original fluorescence image to cyan to obtain a fluorescence adjusted image;

   Superimpose the original white light image and the fluorescence adjusted image to obtain a fused image;

   Output the fused image.
2. The dual fluorescence endoscope image fusion method according to claim 1, characterized in that the adjustment of the fluorescence original image is such that the blue component in the fluorescence original image is more than the red component and the blue component is more than the red component. One of the two types of pixels with small components is converted to green, the other is converted to blue, and the overexposed pixels in the original fluorescence image are converted to cyan. The steps of obtaining the fluorescence adjusted image include:

   Traverse the original fluorescence image to obtain the second red value, the second green value and the second blue value of each pixel position;

   Calculate the hue value of the corresponding pixel position in the HSV color model according to the second red value, the second green value and the second blue value;

   Calculate the color adjustment coefficient of the corresponding pixel position according to the ratio of the second red value to the second blue value and the hue value;

   Calculate the fluorescence-adjusted green value and the fluorescence-adjusted blue value of the corresponding pixel position according to the color adjustment coefficient, so that the fluorescence-adjusted red value of the corresponding pixel position is 0;

   The fluorescence-adjusted red value, fluorescence-adjusted green value and fluorescence-adjusted blue value of each pixel position are recombined to obtain the fluorescence-adjusted image.
3. The dual fluorescence endoscope image fusion method according to claim 1, characterized in that the adjustment of the fluorescence original image is such that the blue component in the fluorescence original image is more than the red component and the blue component is more than the red component. One of the two types of pixels with small components is converted to green, the other is converted to blue, and the overexposed pixels in the original fluorescence image are converted to cyan. The steps of obtaining the fluorescence adjusted image include:

   Traverse the original fluorescence image to obtain the second red value and the second blue value of each pixel position;

   Calculate the corresponding pixel position color adjustment coefficient according to the percentage of the second red value to the sum of the second red value and the second blue value;

   Calculate the fluorescence-adjusted green value and the fluorescence-adjusted blue value of the corresponding pixel position according to the color adjustment coefficient, so that the fluorescence-adjusted red value of the corresponding pixel position is 0;

   The fluorescence-adjusted red value, fluorescence-adjusted green value and fluorescence-adjusted blue value of each pixel position are recombined to obtain the fluorescence-adjusted image.
4. The dual fluorescence endoscope image fusion method according to claim 3, wherein the step of traversing the original fluorescence image to obtain the second red value and the second blue value of each pixel position also obtains the second red value and the second blue value of each pixel position. The second green value of each pixel position in the fluorescence original image;

   Before calculating the color adjustment coefficient of the corresponding pixel position according to the percentage of the second red value to the sum of the second red value and the second blue value, the step further includes:

   Calculate the hue value of the corresponding pixel position in the HSV color model according to the second red value, the second green value and the second blue value;

   The step of calculating the color adjustment coefficient of the corresponding pixel position based on the percentage of the second red value to the sum of the second red value and the second blue value includes:

   The corresponding pixel position color adjustment coefficient is calculated according to the percentage of the second red value to the sum of the second red value and the second blue value and the hue value.
5. The dual fluorescence endoscope image fusion method according to claim 1, characterized in that the adjustment of the fluorescence original image is such that the blue component in the fluorescence original image is more than the red component and the blue component is more than the red component. One of the two types of pixels with small components is converted to green, the other is converted to blue, and the overexposed pixels in the original fluorescence image are converted to cyan. The steps of obtaining the fluorescence adjusted image include:

   Traverse the original fluorescence image to obtain the second red value and the second blue value of each pixel position;

   Construct a sigmoid function with the dependent variable as the color adjustment coefficient, and the independent variable of the sigmoid function is the ratio of the second red value and the second blue value at the corresponding pixel position;

   Calculate the fluorescence-adjusted green value and the fluorescence-adjusted blue value of the corresponding pixel position according to the color adjustment coefficient, so that the fluorescence-adjusted red value of the corresponding pixel position is 0;

   The fluorescence-adjusted red value, fluorescence-adjusted green value and fluorescence-adjusted blue value of each pixel position are recombined to obtain the fluorescence-adjusted image.
6. The dual fluorescence endoscope image fusion method according to any one of claims 2 to 5, characterized in that the color adjustment coefficient ranges from 0 to 1, and the color adjustment coefficient of the corresponding pixel position is calculated according to the color adjustment coefficient. The steps for fluorescently adjusting the green value and fluorescently adjusting the blue value include:

   Let one of the fluorescence adjustment green value and the fluorescence adjustment blue value of the corresponding pixel position be adjusted to the color adjustment coefficient multiplied by the gray level, and the other value is adjusted to 1 minus the difference of the color adjustment coefficient multiplied by the gray level. degree level.
7. The dual fluorescence endoscope image fusion method according to claim 1, characterized in that there are steps before superposing the white light original image and the fluorescence adjusted image to obtain the fused image:

   Traverse the original fluorescence image to obtain the second red value, the second green value and the second blue value of each pixel position;

   Calculate the fluorescence signal intensity of the corresponding pixel position according to the second red value, the second green value and the second blue value;

   The step of superposing the white light original image and the fluorescence adjusted image to obtain a fused image includes:

   The fluorescence adjusted image is adjusted according to the fluorescence signal intensity to obtain the fluorescence part of the fused image, the white light original image is adjusted according to the fluorescence signal intensity to obtain the white light part of the fused image, and the white light part of the fused image and the fused image are superimposed. The fluorescent part of the image is combined into a fused image.
8. The dual fluorescence endoscope image fusion method according to claim 7, wherein the fluorescence adjustment image is adjusted according to the fluorescence signal intensity to obtain the fluorescence part of the fused image, and the fluorescence adjustment image is adjusted according to the fluorescence signal intensity. The white light original image is used to obtain the white light part of the fused image, and the white light part of the fused image and the fluorescent part of the fused image are superimposed. The steps of obtaining the fused image include:

   Traverse the white light original image to obtain the first red value, the first green value and the first blue value of each pixel position;

   Subtract the fluorescence signal intensity from 1 to obtain the white light signal intensity at the corresponding pixel position;

   The first red value, the first green value, and the first blue value are multiplied by the white light signal intensity respectively to obtain the red value of the white light part, the green value of the white light part, and the blue value of the white light part at the corresponding pixel position. value;

   Obtain the fluorescence-adjusted green value and the fluorescence-adjusted blue value of each pixel position of the fluorescence-adjusted image;

   The fluorescence-adjusted green value and the fluorescence-adjusted blue value are multiplied by the fluorescence signal intensity respectively to obtain the fluorescence part green value and the fluorescence part blue value of the corresponding pixel position;

   Taking the red value of the white light part as the third red value of the corresponding pixel position of the fused image, superimposing the green value of the white light part and the green value of the fluorescent part to obtain the third green value of the corresponding pixel position of the fused image, superposition The blue value of the white light part and the blue value of the fluorescent part obtain the third blue value of the corresponding pixel position of the fused image;

   The fused image is composed according to the third red value, the third green value and the third blue value at each pixel position.
9. An electronic device, characterized in that it includes a processor and a memory, and the memory stores computer-readable instructions. When the computer-readable instructions are executed by the processor, any of claims 1-8 is executed. A step in the method.
10. A dual-fluorescence endoscope image fusion device, characterized by including:

    Acquisition module, used to acquire white light original images and fluorescence original images;

    Adjustment module, used to adjust the fluorescence original image to convert one of the two types of pixels in the fluorescence original image with more blue components than red components and less blue components than red components into green and the other type. Convert to blue, and convert the overexposed pixels in the original fluorescence image to cyan to obtain a fluorescence adjusted image;

    A superposition module, used to superimpose the white light original image and the fluorescence adjusted image to obtain a fused image;

    An output module is used to output the fused image.

Images