WO2021033787A1

WO2021033787A1 - Visible and near-infrared image providing system and method which use single color camera and can simultaneously acquire visible and near-infrared images

Info

Publication number: WO2021033787A1
Application number: PCT/KR2019/010472
Authority: WO
Inventors: 배수진; 이대식; 김한석
Original assignee: 한국전기연구원
Priority date: 2019-08-19
Filing date: 2019-08-19
Publication date: 2021-02-25

Abstract

Disclosed is a visible and near-infrared image providing system which uses a single color camera and can simultaneously acquire visible and near-infrared images. The visible and near-infrared image providing system comprises: an emission light-source unit for emitting visible rays and near-infrared rays of preset wavelengths at a subject; and an image processing unit for processing image data detected by an image detection sensor comprising pixels, which are arranged in a preset pattern, detect a visible ray region of partial wavelengths of red, green, and blue colors, and have similar sensitivity in a near-infrared ray region, wherein the emission light-source unit simultaneously emit visible rays and near-infrared rays except in regions of partial wavelengths of red, green, and blue colors, and the image processing unit comprises a near-infrared ray detection value calculation unit, a visible ray detection value calculation unit, and an estimated visible ray detection value calculation unit.

Description

Visible and near-infrared image providing system and method capable of simultaneously acquiring visible and near-infrared images using a single color camera

The present invention relates to an imaging system, and more particularly, to a system and method for providing an optical image such as visible light or near-infrared fluorescence image for biological tissue examination.

Optical imaging using fluorescence has advantages such as sensitivity, selectivity, and convenient use. Since living tissues have less autofluorescence in the near-infrared region of 700 to 900 nm, various biological tissues that need to be visualized using near-infrared fluorescent materials and near-infrared fluorescence images that target diseased tissues including cancer are diverse in the medical field. Is being utilized.

As an example, ICG (Indocyanin green) binds to albumin, a water-soluble protein present in the cytoplasm and tissues of the body, and has a characteristic of fluorescently expressing at 835 nm at an infrared wavelength of 805 nm. The location and distribution of surrounding lymph nodes can be identified, and the flow of blood and lymph fluid can be observed.

On the other hand, in general, the visible and near-infrared image detection systems use individual image sensors for visible and near-infrared channels, or use a single sensor sequentially.

In the method of using a multi-channel detector, a fusion image is generated by simultaneously acquiring different wavelength images such as a visible light wavelength region and a near infrared wavelength region in order to observe the near-infrared fluorescence and surrounding tissues of a target object. However, since this method requires accurate optical alignment and a plurality of parts, there is a problem that the configuration is complicated and the price is high.

In addition, the method of using a single detector is a selective and sequential detection method for a specific wavelength range, and acquires visible and near-infrared wavelength images and generates a fused image through image processing. However, in this case, since the visible light image and the near-infrared fluorescence image are sequentially provided, there is a problem in that the object image cannot be detected simultaneously.

The present invention has been devised to solve the above-described conventional problems, and provides an imaging system and method capable of simultaneously acquiring and providing surrounding tissue images and near-infrared images by simultaneously detecting visible and near-infrared wavelengths while having a single camera. It aims to provide.

In order to achieve the above object, the system for providing visible and near-infrared rays according to the present invention includes a conventional CMOS in which an irradiation light source unit for irradiating visible and near-infrared rays of preset wavelengths to a subject and red, green, and blue filters are intersected in a predetermined pattern. A system for providing visible and near-infrared rays including a color camera including a CCD image sensor and an image processing unit that processes detected image data, wherein the irradiation light source unit receives visible and near-infrared rays excluding some wavelength regions of red, green, and blue. At the same time, the image processing unit includes a near-infrared ray detection value calculation unit, a visible ray detection value calculation unit, and an estimated visible ray detection value calculation unit.

The near-infrared detection value calculation unit calculates the near-infrared detection value of the surrounding pixels through the near-infrared detection value obtained from some pixels in which only near-infrared ray is detected among the image detection sensors, and the visible ray detection value calculation unit calculates the calculated near-infrared ray detection value and the obtained visible light. The corrected visible ray detection value is calculated through the ray detection value, and the estimated visible ray detection value calculator calculates estimated visible ray detection values of pixels in which only near-infrared rays are detected using the calculated visible ray detection values of other pixels.

According to this configuration, by using light of different wavelengths respectively detected from different pixels of a single image detection sensor, while equipped with a single camera, it simultaneously detects the visible and near-infrared wavelengths to simultaneously display surrounding tissue images and near-infrared images. Be able to acquire and provide.

In this case, the image system may further include an image detector including an image detection sensor, and the image detector may further include a filter to pass through a near-infrared region excluding a visible region and an excitation wavelength.

In addition, some wavelength regions excluded from the irradiation light source unit may include a red visible ray region, and the irradiation light source unit includes a visible light source unit for selectively irradiating visible light of a preset wavelength, and a preset excitation wavelength to excite fluorescence. It may include a near-infrared light source for irradiating near-infrared rays of.

In addition, an image output unit for outputting image data provided by the image processing unit may be further included, and in this case, the image output unit may convert a near-infrared fluorescence signal into a virtual color and output it.

In addition, the wavelength of visible light irradiated together with the near-infrared light of the excitation wavelength by the irradiation light source unit is blue, the pixel that detects the red and green wavelengths of visible light of the image detection sensor detects only the excited near-infrared fluorescence, and the image processing unit detects the near-infrared ray. Using the value, a near-infrared detection value in a blue visible ray detection pixel may be calculated, and an estimated blue visible ray detection value in a red and green visible ray detection pixel may be calculated.

In addition, the wavelength of visible light irradiated together with the near-infrared ray of the excitation wavelength by the irradiation light source unit is blue and green, the pixel detecting the red visible ray of the image detection sensor detects only the excited near-infrared fluorescence, and the image processing unit detects the near-infrared ray detection value. By using, near-infrared detection values in blue and green visible light detection pixels may be calculated, and estimated blue and green visible light detection values in red visible light detection pixels may be calculated.

In addition, a calibration unit for calibrating a detected value from the image detection sensor according to a preset reference may be further included.

In addition, the invention in which the system is implemented in the form of a method, and an image processing apparatus and an image processing method for executing the method are disclosed.

According to the present invention, irradiation light in the visible and near-infrared wavelength ranges set in consideration of the sensitivity characteristics of each color channel pixel in a single image detection sensor, and excitation light in the near-infrared wavelength range, and each detected in different pixels of the single image detection sensor Using light detection values of different wavelengths, it is possible to simultaneously acquire and provide surrounding tissue images and near-infrared fluorescence images by simultaneously detecting visible and near-infrared wavelengths while having a single camera.

1 is a schematic block diagram of a system for providing visible and near-infrared rays according to an embodiment of the present invention.

2 is a graph showing spectral characteristics of the image detection sensor of FIG. 1.

3 is a diagram illustrating a white light area irradiated from the visible light source unit of FIG. 1.

4 is a view showing a near-infrared region irradiated from the near-infrared light source of FIG. 1;

FIG. 5 is a diagram illustrating visible and near-infrared regions when visible and near-infrared rays are simultaneously irradiated from the irradiation light source of FIG. 1;

FIG. 6 is a diagram illustrating detection wavelength characteristics of an image detection unit filter unit of FIG. 1.

7 is a diagram illustrating an example of a Bayer pattern.

8 to 10 schematically illustrate the concept of a near-infrared fluorescence image (I _NIR ), a color background image (I _VIS ), and a fusion image (I) of a near-infrared fluorescence image (I _NIR ) and a color background image (I _{VIS), respectively.} Figure shown.

11 is a diagram illustrating an example of color value interpolation performed by the image processing unit of FIG. 1.

Hereinafter, preferred embodiments of the present invention will be described with reference to the accompanying drawings.

1 is a schematic block diagram of a system for providing visible and near-infrared rays according to an embodiment of the present invention. The visible and near-infrared image providing system 100 includes an irradiation light source unit 110, an image detection unit 120, an image processing unit 130, and an image output unit 140.

In addition, the irradiation light source unit 110 again includes a visible light source unit 112 and a near-infrared light source unit 114, the image detection unit 120 again includes an image detection sensor 122 and a filter unit 124, and an image processing unit 130 again includes a near infrared ray detection value calculation unit 132, a visible ray detection value calculation unit 134, an estimated visible ray detection value calculation unit 136, and a calibration unit 138.

The irradiation light source unit 110 may be implemented as a composite light source device including a visible light irradiation light source and a near-infrared fluorescence excitation light source, and at this time, the visible light irradiation light source may selectively adjust the wavelength of the irradiation light.

To this end, the visible light source unit 112 selectively irradiates visible light having a set wavelength in consideration of the spectral characteristics of the image detection sensor, and the near infrared light source unit 114 irradiates near infrared light having an excitation wavelength to excite the fluorescent material.

FIG. 2 is a graph showing spectral characteristics of the image detection sensor of FIG. 1, and FIGS. 3 and 4 are diagrams showing a white light region irradiated by a visible light source unit and a near infrared region irradiated by a near infrared light source unit, respectively.

In addition, FIG. 5 is a diagram illustrating visible and near-infrared regions when visible and near-infrared rays are simultaneously irradiated by the irradiation light source of FIG. 1. In particular, in FIG. 5, it can be seen that the visible light region is displayed only for some wavelengths selected by the visible light source unit 112.

The image detection unit 120 may be implemented as a single-chip color camera, and when the visible light irradiation light source and the near-infrared fluorescence excitation light source are simultaneously irradiated, the reflected light and the near-infrared fluorescence of the visible light irradiation light source can be simultaneously detected with a single-chip color camera. have.

The image detection sensor 122 includes pixels for detecting visible light and near-infrared light of some wavelengths of red, green, and blue, arranged according to a set pattern. In general, single-chip CMOS and CCD image detection sensors have a Bayer pattern with a color filter array (CFA), and R, G, and B filters have 50% G, depending on human visual characteristics. R and B are uniformly intersected so that each is 25%, and single color information is detected at each pixel location. 7 is a diagram illustrating an example of a Bayer pattern.

When outputting and displaying a color image, since each pixel of the image detection sensor is combined with only one of R, G, and B filters, one pixel detects only one color, so the missing color information for each pixel is the color value from the surrounding pixels. Color information is completed through demosaicing, which is an image processing process that is interpolated.

Meanwhile, in general, a black and white camera does not have an infrared cut filter, but a color camera incorporates a cut filter for color reproduction. However, recently, due to the development of image sensor and image processing technology, products without infrared cut filter have been released. In this case, in a single-chip RGB camera with no infrared cut-off filter and R, G, and B color filters disposed, the near-infrared region is detected with the same signal sensitivity in all color channels as shown below.

R Channel: R _VIS + R _NIR

G channel: G _VIS + G _NIR

B channel: B _VIS + B _NIR

This fact can be seen from FIG. 2 that the near-infrared region is detected with almost the same signal sensitivity in all color channels. Accordingly, R, G, and B pixels can also be used as near-infrared channels.

More specifically, I ^CFA When referring to a single color component of each pixel detected by the image detection device, the I ^CFA (i,j) of each pixel is i = 1, 2...M, j = 1, 2... When it is N, it can be expressed as follows.

R _ij , B _ij , and G _ij are the color components of CFA image I ^CFA , and if the color channel is 8 bits, the _{values of R ij} , B _ij and G _ij are 0-255. An image for outputting the missing color information for each pixel to the display through color value interpolation processing

) Can be obtained.

At this time, due to the color filter array spectral characteristics of the image detection sensor, ^{each pixel of the CFA image I CFA} includes not only the visible region of the color filter but also the near infrared region. Therefore, the near infrared signal is separated from each pixel.

and

Can be obtained by dividing as follows.

At this time,

,

Is the missing color information estimate.

For example, when observing near-infrared fluorescence, a wavelength less than _{λ 2} is the stopband limit wavelength of the color filter array red filter of the image detection sensor along with the near-infrared excitation light. Is irradiated with illumination light for observing surrounding tissues, the R channel of the single-chip RGB camera detects near-infrared fluorescence, and the G and B channels are light reflected from the surrounding tissues by irradiation with the irradiation light ( Reflection light) and near-infrared fluorescence can be detected.

R channel: R _NIR

G channel: G _VIS + G _NIR

B channel: B _VIS + B _NIR

The R channel has only near-infrared values and can be used to calculate estimated near-infrared values of the surrounding G and B channels.

,

Is the estimated near-infrared values of the B and G channels. By separating the near-infrared signal from each pixel, CFA image I ^CFA data having only visible light signals can be estimated and calculated.

CFA image having only visible light signal I A background image of an object to be observed may be obtained as follows through demosaicing, an image processing process of interpolating color values from surrounding pixels using ^CFA.

Accordingly, in the present invention, a near-infrared fluorescence image (I _NIR ) and a color background image (I _VIS ) can be simultaneously acquired, and these images can be displayed alone or by fusion (I).

That is, similar to a system having a single RGB camera and a system having a plurality of cameras or sensors, it is possible to simultaneously acquire a color background image in the visible light region and a fluorescence image in the near infrared region. Moreover, since the near-infrared fluorescence signal is separated, it is possible to produce an emphasis image that emphasizes only the intensity of the near-infrared fluorescence signal.

8 to 10 schematically illustrate the concept of a near-infrared fluorescence image (I _NIR ), a color background image (I _VIS ), and a fusion image (I) of a near-infrared fluorescence image (I _NIR ) and a color background image (I _{VIS), respectively.} It is a drawing shown.

The filter unit 124 passes through a near-infrared region excluding visible light and an excitation wavelength, and may be configured as a notch filter. 6 is a diagram illustrating a detection wavelength region after filtering by the filter of FIG. 1.

The near-infrared ray detection value calculating unit 132 calculates the near-infrared ray detection value of other pixels by using the near-infrared ray detection value obtained from some pixels in which only near-infrared ray is detected among the image detection sensors, and the visible ray detection value calculating unit 134 By using the calculated near-infrared detection value, the visible light detection value of other pixels is calculated, and the estimated visible ray detection value calculating unit 136 uses the calculated visible light detection value of the other pixels. The estimated visible light detection value is calculated.

The calibration unit 138 calibrates the detected value from the image detection sensor 122 according to a preset reference. As shown in FIG. 2, the R, G, and B sensitivities differ depending on the wavelength. For example, even in the case of R, the sensitivity is not 0 even in the 400 to 550 nm area of the shielding area, and there is a slight difference in the sensitivity of R, G, and B in the near infrared area. Therefore, it performs a function of calibrating it.

As an example, the visible light of the wavelength irradiated by the irradiation light source unit 110 is blue, the pixel detecting the visible light of the red and green wavelengths of the image detection sensor detects only near infrared rays, and the image processing unit 130 detects near infrared rays Using the value, a near-infrared detection value in a blue visible ray detection pixel may be calculated, and an estimated blue visible ray detection value in a red and green visible ray detection pixel may be calculated.

11 is a diagram for describing an example of color value interpolation performed by the image processing unit of FIG. 1. FIG. 11 shows an example of calculating a near-infrared ray value of a blue cell by using the near-infrared ray detection values of red and green cells around a blue cell.

More specifically, when the visible light source and the near-infrared fluorescence excitation light source are simultaneously irradiated, if the wavelength of the irradiated light is λ _EX <λ ₁ , the detected R and G channel signals are near-infrared fluorescence signals, and the detected B channel is near-infrared fluorescence. And reflected light of visible light irradiation.

Therefore, the near-infrared fluorescence image is obtained by estimating the near-infrared fluorescence signal of the B channel from the near-infrared fluorescence signal of the R channel and G channel, and the reflected light information of the R and G channels is estimated from the visible light signal of the B channel. The background image of the object may be acquired as follows.

When λ _EX <λ ₁ ,

R channel: R _NIR,

G channel: G _NIR

B channel: B _VIS + B _NIR

As another example, the visible light of the wavelength irradiated by the irradiation light source unit 110 is blue and green, the pixel detecting the red visible light of the image detection sensor 122 detects only near-infrared light, and the image processing unit 130 Using the near-infrared detection value, a near-infrared detection value in a blue and green visible ray detection pixel may be calculated, and an estimated blue and green visible ray detection value in a red visible ray detection pixel may be calculated.

More specifically, when the visible light source and the near-infrared fluorescence excitation light source are simultaneously irradiated, if λ ₁ <λ _EX <λ _{2, the} intensity of the near-infrared fluorescence signal of the G channel and the near-infrared fluorescence signal of the B channel from the R channel having only the near-infrared fluorescence signal And calculate the intensity of reflected light of the G and B channels. By estimating the reflected light information of the R channel from the intensity of the reflected light of the G channel and the B channel, a background image of the observation object composed of the reflected light may be obtained as follows.

When λ ₁ <λ _EX <λ ₂ ,

R channel: R _NIR

G channel: G _VIS + G _NIR

B channel: B _VIS + B _NIR

The image output unit 140 outputs image data provided by the image processing unit 130. In this case, the image output unit 140 may display reflected light of the visible light irradiation light source and near-infrared fluorescence at the same time or overlap each other, and at this time, the near-infrared fluorescence signal may be displayed in pseudo color.

Although the present invention has been described by some preferred embodiments, the scope of the present invention should not be limited thereto, and modifications or improvements of the above embodiments supported by the claims should also be reached.

Claims

An irradiation light source unit for irradiating visible light and near-infrared light having a preset wavelength on the subject; And

An image that is arranged according to a preset pattern and detects the visible light region of some wavelengths of red, green, and blue, and processes image data detected from an image detection sensor including pixels having sensitivity within a preset range in the near-infrared region. A system for providing visible and near-infrared rays including a processing unit,

The irradiation light source unit simultaneously irradiates visible light and near-infrared light excluding some of the red, green, and blue wavelength regions,

The image processing unit,

A near-infrared ray detection value calculator configured to calculate near-infrared ray detection values of other pixels by using near-infrared ray detection values obtained from some pixels in which only near-infrared rays are detected among the image detection sensors;

A visible light detection value calculator configured to calculate a visible light detection value of the other pixels by using the calculated near infrared ray detection value; And

And an estimated visible ray detection value calculator configured to calculate an estimated visible ray detection value of some of the pixels by using the visible ray detection value of the other pixels.
The method according to claim 1,

An imaging system, further comprising an image detection unit including the image detection sensor.
The method according to claim 2,

The image detection unit further comprises a filter unit for passing through a visible light region and a near infrared region excluding the excitation wavelength.
The method according to claim 1, wherein the irradiation light source unit

A visible light source unit for selectively irradiating visible light of the preset wavelength; And

Visible and near-infrared image providing system comprising a near-infrared light source unit for irradiating near-infrared rays of a preset excitation wavelength to excite fluorescence.
The method according to claim 1,

A visible and near-infrared image providing system, further comprising an image output unit that outputs image data provided by the image processing unit.
The method of claim 5,

The image output unit converts the near-infrared fluorescence signal into a virtual color and outputs the converted visible light and near-infrared image.
The method according to claim 1,

Visible and near-infrared image providing system, characterized in that the excluded some wavelength region includes a red visible light region.
The method of claim 7,

Visible light of a wavelength irradiated by the irradiation light source unit is blue,

In the pixel detecting visible light of red and green wavelengths of the image detection sensor, only near-infrared rays are detected,

The image processing unit calculates a near-infrared ray detection value in a blue visible ray detection pixel by using the near-infrared ray detection value, and calculates an estimated blue visible ray detection value in a red and green visible ray detection pixel. Video delivery system.
The method of claim 7,

The visible light of the wavelength irradiated by the irradiation light source is blue and green,

In the pixel detecting red visible light of the image detection sensor, only near-infrared light is detected,

The image processing unit calculates a near-infrared ray detection value in a blue and green visible ray detection pixel by using the near-infrared ray detection value, and calculates an estimated blue and green visible ray detection value in a red visible ray detection pixel. And near-infrared image providing system.
The method according to claim 1,

Visible and near-infrared image providing system, characterized in that it further comprises a calibration unit for calibrating the detected value from the image detection sensor according to a preset reference.
A light source irradiation step of irradiating visible light and near-infrared light having a preset wavelength on the subject; And

An image that is arranged according to a preset pattern and detects the visible light region of some wavelengths of red, green, and blue, and processes image data detected from an image detection sensor including pixels having sensitivity within a preset range in the near-infrared region. A method for providing an image of a simultaneous system of visible and near-infrared images including a processing step,

The light source irradiation step simultaneously irradiates visible light and near-infrared light except for some of the red, green, and blue wavelength regions,

The image processing step,

A near-infrared detection value calculating step of calculating near-infrared detection values of other pixels by using near-infrared detection values obtained from some pixels in which only near-infrared rays are detected among the image detection sensors;

A visible light detection value calculation step of calculating a visible light detection value of the other pixels by using the calculated near infrared ray detection value; And

And calculating an estimated visible ray detection value of some of the pixels by using the visible ray detection value of the other pixels.
The method of claim 11,

And an image detection step of detecting visible and near-infrared rays through the image detection sensor.
The method of claim 12,

The image detection step includes a filtering step of passing through a near-infrared region excluding visible light and the excitation wavelength.
The method of claim 11, wherein the light source irradiation step,

Selectively irradiating visible light of the preset wavelength,

A method for providing visible and near-infrared rays, comprising irradiating near infrared rays having a preset excitation wavelength together with the visible rays to excite fluorescence.
The method of claim 11,

And an image output step of outputting image data provided by the image processing step.
The method of claim 15,

In the image outputting step, the near-infrared fluorescence signal is converted into a virtual color and then output.
The method of claim 11,

The method for providing visible and near-infrared rays, characterized in that the excluded partial wavelength region includes a red visible ray region.
The method of claim 17,

Visible light of the wavelength irradiated by the irradiation light source step is blue,

In the pixel detecting visible light of red and green wavelengths of the image detection sensor, only near-infrared rays are detected,

The image processing step includes calculating a near-infrared detection value in a blue visible ray detection pixel using the near-infrared ray detection value, and calculating an estimated blue visible ray detection value in a red and green visible ray detection pixel. Method of providing near-infrared image.
The method of claim 17,

The visible light of the wavelength irradiated by the irradiation light source step is blue and green,

In the pixel detecting red visible light of the image detection sensor, only near-infrared light is detected,

The image processing step includes calculating a near-infrared detection value in a blue and green visible ray detection pixel using a near-infrared ray detection value, and calculating an estimated blue and green visible ray detection value in a red visible ray detection pixel. Method of providing light and near-infrared images.
The method of claim 11,

And a calibration step of calibrating the detected value by the image detection sensor according to a preset reference.