KR102190398B1 - System and method for providing visible ray image and near-infrared ray image, using a single color camera and capable of those images simultaneously - Google Patents

Abstract

Disclosed is a system for providing visible and near infrared images using a single color camera and capable of simultaneously obtaining visible and near infrared images. An irradiation light source unit that irradiates visible light and near-infrared rays of preset wavelengths to the subject, and pixels arranged according to a preset pattern and detecting visible light regions of some wavelengths of red, green, and blue, and having similar sensitivities to each other in the near-infrared region. A system for providing visible and near-infrared rays including an image processing unit that processes image data detected from an image detection sensor, wherein the irradiation light source unit simultaneously irradiates visible and near-infrared rays excluding some wavelength ranges of red, green, and blue, and the image processing unit Includes a near infrared ray detection value calculation unit, a visible ray detection value calculation unit, and an estimated visible ray detection value calculation unit.

Description

(SYSTEM AND METHOD FOR PROVIDING VISIBLE RAY IMAGE AND NEAR-INFRARED RAY IMAGE, USING A SINGLE COLOR CAMERA AND CAPABLE OF THOSE IMAGES) SIMULTANEOUSLY}

The present invention relates to an imaging system, and more particularly, to a system and method for providing an optical image such as visible light or near-infrared fluorescence image for biological tissue examination.

Optical imaging using fluorescence has advantages such as sensitivity, selectivity, and convenient use. Since biological tissues have little autofluorescence in the near-infrared region of 700 to 900 nm, various biological tissues that need to be visualized using near-infrared fluorescent materials and near-infrared fluorescence images targeting diseased tissues including cancer are diverse in the medical field. Is being utilized.

For example, ICG (Indocyanin green) binds to albumin, a water-soluble protein present in the cytoplasm and tissues of the body, and has a characteristic of expressing 835 nm fluorescence at an infrared wavelength of 805 nm, so it is difficult to identify blood vessels and lesions around the lesion. The location and distribution of lymph nodes can be identified, and the flow of blood and lymph fluid can be observed.

On the other hand, in general, the visible and near-infrared image detection systems use individual image sensors for visible and near-infrared channels, or use a single sensor sequentially.

In the method of using a multi-channel detector, a fusion image is generated by simultaneously acquiring different wavelength images such as a visible light wavelength region and a near infrared wavelength region in order to observe the near-infrared fluorescence and surrounding tissues of a target object. However, since this method requires accurate optical alignment and a plurality of parts, there is a problem that the configuration is complicated and the price is high.

In addition, the method of using a single detector is a selective and sequential detection method for a specific wavelength range, and acquires visible and near-infrared wavelength images and generates a fused image through image processing. However, in this case, since the visible light image and the near-infrared fluorescence image are sequentially provided, there is a problem in that the object image cannot be detected simultaneously.

US 9173554 B2

The present invention has been devised to solve the above-described conventional problems, and provides an imaging system and method capable of simultaneously acquiring and providing surrounding tissue images and near-infrared images by simultaneously detecting visible and near-infrared wavelengths while having a single camera. It aims to provide.

In order to achieve the above object, the system for providing visible and near-infrared rays according to the present invention includes a conventional CMOS in which an irradiation light source unit for irradiating visible and near-infrared rays of preset wavelengths to a subject and red, green, and blue filters are intersected in a predetermined pattern. A system for providing visible and near-infrared rays including a color camera including a CCD image sensor and an image processing unit that processes detected image data, wherein the irradiation light source unit receives visible and near-infrared rays excluding some wavelength regions of red, green, and blue. At the same time, the image processing unit includes a near-infrared ray detection value calculation unit, a visible ray detection value calculation unit, and an estimated visible ray detection value calculation unit.

The near-infrared detection value calculation unit calculates the near-infrared detection value of the surrounding pixels through the near-infrared detection value obtained from some pixels in which only near-infrared ray is detected among the image detection sensors, and the visible ray detection value calculation unit calculates the calculated near-infrared ray detection value and the obtained visible light. The corrected visible ray detection value is calculated through the ray detection value, and the estimated visible ray detection value calculator calculates estimated visible ray detection values of pixels in which only near-infrared rays are detected using the calculated visible ray detection values of other pixels.

According to this configuration, by using light of different wavelengths respectively detected from different pixels of a single image detection sensor, while equipped with a single camera, it simultaneously detects the visible and near-infrared wavelengths to simultaneously display surrounding tissue images and near-infrared images. Be able to acquire and provide.

In this case, the image system may further include an image detector including an image detection sensor, and the image detector may further include a filter to pass through a near-infrared region excluding a visible region and an excitation wavelength.

In addition, some wavelength regions excluded from the irradiation light source unit may include a red visible ray region, and the irradiation light source unit includes a visible light source unit for selectively irradiating visible light of a preset wavelength, and a preset excitation wavelength to excite fluorescence. It may include a near-infrared light source for irradiating near-infrared rays of.

In addition, an image output unit for outputting image data provided by the image processing unit may be further included, and in this case, the image output unit may convert a near-infrared fluorescence signal into a virtual color and output it.

In addition, the wavelength of visible light irradiated together with the near-infrared light of the excitation wavelength by the irradiation light source unit is blue, the pixel that detects the red and green wavelengths of visible light of the image detection sensor detects only the excited near-infrared fluorescence, and the image processing unit detects the near-infrared ray. Using the value, a near-infrared detection value in a blue visible ray detection pixel may be calculated, and an estimated blue visible ray detection value in a red and green visible ray detection pixel may be calculated.

In addition, the wavelength of visible light irradiated together with the near-infrared ray of the excitation wavelength by the irradiation light source unit is blue and green, the pixel detecting the red visible ray of the image detection sensor detects only the excited near-infrared fluorescence, and the image processing unit detects the near-infrared ray detection value. By using, near-infrared detection values in blue and green visible light detection pixels may be calculated, and estimated blue and green visible light detection values in red visible light detection pixels may be calculated.

In addition, a calibration unit for calibrating a detected value from the image detection sensor according to a preset reference may be further included.

In addition, the invention in which the system is implemented in the form of a method, and an image processing apparatus and an image processing method for executing the method are disclosed.

According to the present invention, irradiation light in the visible and near-infrared wavelength ranges set in consideration of the sensitivity characteristics of each color channel pixel in a single image detection sensor, and excitation light in the near-infrared wavelength range, respectively, detected in different pixels of the single image detection sensor. Using light detection values of different wavelengths, it is possible to simultaneously acquire and provide surrounding tissue images and near-infrared fluorescence images by simultaneously detecting visible and near-infrared wavelengths while having a single camera.

1 is a schematic block diagram of a system for providing visible and near-infrared rays according to an embodiment of the present invention.
2A is a graph showing spectral characteristics of the image detection sensor of FIG. 1.
2B is a diagram illustrating a white light area irradiated from the visible light source of FIG. 1.
2C is a view showing a near-infrared region irradiated from the near-infrared light source of FIG. 1;
FIG. 2D is a view showing visible and near infrared regions when visible and near infrared rays are simultaneously irradiated from the irradiation light source of FIG. 1;
FIG. 2E is a diagram illustrating detection wavelength characteristics of an image detection unit filter unit of FIG. 1.
3 is a diagram illustrating an example of a Bayer pattern.
4 to 6 schematically illustrate the concept of a near-infrared fluorescence image (I _NIR ), a color background image (I _VIS ), and a fusion image (I) of a near-infrared fluorescence image (I _NIR ) and a color background image (I _VIS ), respectively. Figure shown.
7 is a diagram illustrating an example of color value interpolation performed by the image processing unit of FIG. 1.

Hereinafter, preferred embodiments of the present invention will be described with reference to the accompanying drawings.

1 is a schematic block diagram of a system for providing a visible light near-infrared image according to an embodiment of the present invention. The visible and near-infrared image providing system 100 includes an irradiation light source unit 110, an image detection unit 120, an image processing unit 130, and an image output unit 140.

Further, the irradiation light source unit 110 again includes a visible light source unit 112 and a near-infrared light source unit 114, the image detection unit 120 again includes an image detection sensor 122 and a filter unit 124, and an image processing unit 130 again includes a near infrared ray detection value calculation unit 132, a visible ray detection value calculation unit 134, an estimated visible ray detection value calculation unit 136, and a calibration unit 138.

The irradiation light source unit 110 may be implemented as a composite light source device including a visible light irradiation light source and a near-infrared fluorescence excitation light source, and at this time, the visible light irradiation light source may selectively adjust the wavelength of the irradiation light.

To this end, the visible light source unit 112 selectively irradiates visible light having a set wavelength in consideration of the spectral characteristics of the image detection sensor, and the near infrared light source unit 114 irradiates near infrared light having an excitation wavelength to excite the fluorescent material.

2A is a graph showing spectral characteristics of the image detection sensor of FIG. 1, and FIGS. 2B and 2C are diagrams illustrating a white light region irradiated from a visible light source unit and a near infrared region irradiated from a near infrared light source unit, respectively.

In addition, FIG. 2D is a diagram illustrating visible and near-infrared regions when visible and near-infrared rays are simultaneously irradiated from the irradiation light source of FIG. 1. In particular, in FIG. 2D, it can be seen that the visible light region is displayed only for some wavelengths selected by the visible light source unit 112.

The image detection unit 120 may be implemented as a single-chip color camera, and when the visible light irradiation light source and the near-infrared fluorescence excitation light source are simultaneously irradiated, the reflected light and the near-infrared fluorescence of the visible light irradiation light source can be simultaneously detected with a single-chip color camera. have.

The image detection sensor 122 includes pixels for detecting visible light and near-infrared light of some wavelengths of red, green, and blue, arranged according to a set pattern. In general, single-chip CMOS and CCD image detection sensors have a Bayer pattern with a color filter array (CFA), and R, G, and B filters have 50% G, depending on human visual characteristics. R and B are uniformly intersected so that each is 25%, and single color information is detected at each pixel location. 3 is a diagram illustrating an example of a Bayer pattern.

When outputting and displaying a color image, since each pixel of the image detection sensor is combined with only one of R, G, and B filters, one pixel detects only one color, so the missing color information for each pixel is the color value from the surrounding pixels. Color information is completed through demosaicing, which is an image processing process that is interpolated.

Meanwhile, in general, a black and white camera does not have an infrared cut filter, but a color camera incorporates a cut filter for color reproduction. However, recently, due to the development of image sensor and image processing technology, products without infrared cut filter have been released. In this case, in a single-chip RGB camera with no infrared cut-off filter and R, G, and B color filters disposed, the near-infrared region is detected with the same signal sensitivity in all color channels as shown below.

R Channel: R _VIS + R _NIR

G channel: G _VIS + G _NIR

B channel: B _VIS + B _NIR

This fact can be seen in FIG. 2A that the near-infrared region is detected with the same signal sensitivity in all color channels. Accordingly, R, G, and B pixels can also be used as near-infrared channels.

More specifically, I ^CFA When referring to a single color component of each pixel detected by the image detection device, the I ^CFA (i,j) of each pixel is i = 1, 2...M, j = 1, 2... When it is N, it can be expressed as follows.

)을 획득할 수 있다. R _ij , B _ij , and G _ij are the color components of CFA image I ^CFA , and if the color channel is 8 bits, the values of R _ij , B _ij and G _ij are 0-255. An image for outputting the missing color information for each pixel to the display through color value interpolation processing from surrounding pixels (

) Can be obtained.

과

을 다음과 같이 구분하여 획득할 수 있다.At this time, due to the color filter array spectral characteristics of the image detection sensor, each pixel of the CFA image I ^CFA includes not only the visible region of the color filter but also the near infrared region. Therefore, the near infrared signal is separated from each pixel.

and

Can be obtained by dividing as follows.

,

,

는 누락된 색 정보 추정값이다. At this time,

,

,

Is the missing color information estimate.

For example, when observing near-infrared fluorescence, a wavelength less than λ ₂ is the stopband limit wavelength of the color filter array red filter of the image detection sensor along with the near-infrared excitation light. Is irradiated with illumination light for observing surrounding tissues, the R channel of the single-chip RGB camera detects near-infrared fluorescence, and the G and B channels are light reflected from the surrounding tissues by irradiation with the irradiation light ( Reflection light) and near-infrared fluorescence can be detected.

R channel: R _NIR

G channel: G _VIS + G _NIR

B channel: B _VIS + B _NIR

The R channel has only near-infrared values and can be used to calculate estimated near-infrared values of the surrounding G and B channels.

,

는 B채널과 G채널의 추정 근적외선 값이다. 각 화소로부터 근적외선 신호를 분리함으로써 가시광선 신호만을 갖는 CFA image I^CFA 데이터를 추정하여 산출할 수 있다.

,

Is the estimated near-infrared value of the B and G channels. By separating the near-infrared signal from each pixel, CFA image I ^CFA data having only visible light signals can be estimated and calculated.

CFA image having only visible light signal I A background image of an object to be observed may be obtained as follows through demosaicing, an image processing process of interpolating color values from surrounding pixels using ^CFA .

Accordingly, in the present invention, a near-infrared fluorescence image (I _NIR ) and a color background image (I _VIS ) can be simultaneously acquired, and these images can be displayed alone or by fusion (I).

That is, it is possible to acquire a color background image in the visible light region and a fluorescence image in the near-infrared region at the same time as in a system having a single RGB camera and a system having a plurality of cameras or sensors. Moreover, since the near-infrared fluorescence signal is separated, it is possible to produce an emphasis image that emphasizes only the intensity of the near-infrared fluorescence signal.

4 to 6 schematically illustrate the concept of a near-infrared fluorescence image (I _NIR ), a color background image (I _VIS ), and a fusion image (I) of a near-infrared fluorescence image (I _NIR ) and a color background image (I _VIS ), respectively. It is a drawing shown.

The filter unit 124 passes through a near-infrared region excluding visible light and an excitation wavelength, and may be configured as a notch filter. FIG. 2E is a diagram illustrating a detection wavelength region after filtering by the filter of FIG. 1.

The near-infrared ray detection value calculating unit 132 calculates the near-infrared ray detection value of other pixels by using the near-infrared ray detection value obtained from some pixels in which only near-infrared ray is detected among the image detection sensors, and the visible ray detection value calculating unit 134 By using the calculated near-infrared detection value, the visible light detection value of other pixels is calculated, and the estimated visible ray detection value calculating unit 136 uses the calculated visible light detection value of the other pixels. The estimated visible light detection value is calculated.

According to this configuration, by using light of different wavelengths respectively detected from different pixels of a single image detection sensor, while equipped with a single camera, it simultaneously detects the visible and near-infrared wavelengths to simultaneously display surrounding tissue images and near-infrared images. Be able to acquire and provide.

The calibration unit 138 calibrates the detected value from the image detection sensor 122 according to a preset reference. As shown in FIG. 2A, the R, G, and B sensitivities differ depending on the wavelength. For example, even in the case of R, the sensitivity is not 0 even in the 400 to 550 nm area of the shielding area, and the sensitivity of R, G, and B in the near-infrared area is also slightly different. Therefore, it performs a function of calibrating it.

As an example, the visible light of the wavelength irradiated by the irradiation light source unit 110 is blue, the pixel detecting the visible light of the red and green wavelengths of the image detection sensor detects only near infrared rays, and the image processing unit 130 detects near infrared rays Using the value, a near-infrared detection value in a blue visible ray detection pixel may be calculated, and an estimated blue visible ray detection value in a red and green visible ray detection pixel may be calculated.

7 is a diagram for describing an example of color value interpolation performed by the image processing unit of FIG. 1. FIG. 7 shows an example of calculating a near-infrared value of a blue cell by using the near-infrared detection value of a red and green cell around a blue cell.

More specifically, when the visible light irradiation light source and the near-infrared fluorescence excitation light source are simultaneously irradiated, if the wavelength of the irradiation light is λ _EX <λ _{1, the} detected R channel and G channel signals are near-infrared fluorescence signals, and the detected B channels are near-infrared fluorescence and It contains reflected light of visible light irradiation.

Therefore, the near-infrared fluorescence image is obtained by estimating the near-infrared fluorescence signal of the B channel from the near-infrared fluorescence signal of the R channel and G channel, and the reflected light information of the R and G channels is estimated from the visible light signal of the B channel. The background image of the object may be acquired as follows.

When λ _EX <λ ₁ ,

R channel: R _NIR,

G channel: G _NIR

B channel: B _VIS + B _NIR

As another example, the visible light of the wavelength irradiated by the irradiation light source unit 110 is blue and green, the pixel detecting the red visible light of the image detection sensor 122 detects only near-infrared light, and the image processing unit 130 Using the near-infrared detection value, a near-infrared detection value in a blue and green visible ray detection pixel may be calculated, and an estimated blue and green visible ray detection value in a red visible ray detection pixel may be calculated.

More specifically, when the visible light source and the near-infrared fluorescence excitation light source are simultaneously irradiated, if λ ₁ <λ _EX <λ _{2, the} intensity of the near-infrared fluorescence signal of the G channel and the near-infrared fluorescence signal of the B channel from the R channel having only the near-infrared fluorescence signal And calculate the intensity of reflected light of the G and B channels. By estimating the reflected light information of the R channel from the intensity of the reflected light of the G channel and the B channel, a background image of the observation object composed of the reflected light may be obtained as follows.

When λ ₁ <λ _EX <λ ₂ ,

R channel: R _NIR

G channel: G _VIS + G _NIR

B channel: B _VIS + B _NIR

The image output unit 140 outputs image data provided by the image processing unit 130. In this case, the image output unit 140 may display reflected light of the visible light irradiation light source and near-infrared fluorescence at the same time or overlap each other, and at this time, the near-infrared fluorescence signal may be displayed in pseudo color.

Although the present invention has been described by some preferred embodiments, the scope of the present invention should not be limited thereto, and modifications or improvements of the above embodiments supported by the claims should also be reached.

100: Visible and near-infrared image providing system
110: irradiation light source unit
112: visible light source unit
114: near-infrared light source unit
120: image detection unit
122: image detection sensor
124: filter unit
130: image processing unit
132: near-infrared ray detection value calculation unit
134: visible light detection value calculation unit
136: Estimated visible light detection value calculation unit
140: video output unit

Claims (20)

An irradiation light source unit for irradiating visible light and near-infrared light having a preset wavelength on the subject; And
Image data detected from an image detection sensor including pixels having similar sensitivities in the near-infrared region and arranged according to a preset pattern including the Bayer pattern color filter and detecting visible light regions of some wavelengths of red, green, and blue As a visible light and near-infrared image providing system including an image processing unit for processing,
The irradiation light source unit simultaneously irradiates visible light and near-infrared light excluding some wavelength ranges of the red, green, and blue,
The image processing unit,
A near-infrared ray detection value calculator configured to calculate near-infrared ray detection values of other pixels by using near-infrared ray detection values obtained from some pixels in which only near-infrared rays are detected among the image detection sensors;
A visible light detection value calculator configured to calculate a visible light detection value of the other pixels by using the calculated near infrared ray detection value; And
And an estimated visible ray detection value calculator configured to calculate an estimated visible ray detection value of some of the pixels by using the visible ray detection value of the other pixels.
The method according to claim 1,
Visible light and near-infrared image providing system, characterized in that it further comprises an image detector including the image detection sensor.
The method according to claim 2,
Visible and near-infrared image providing system, characterized in that the excluded some wavelength region includes a red visible light region.
The method of claim 3, wherein the irradiation light source unit
A visible light source unit for selectively irradiating visible light of the preset wavelength; And
Visible and near-infrared image providing system comprising a near-infrared light source unit for irradiating near-infrared rays of a preset excitation wavelength to excite fluorescence.
The method according to claim 2,
The image detection unit further comprises a filter unit to pass through a near-infrared region and a visible ray region excluding an excitation wavelength irradiated to the subject.
The method according to claim 1,
A visible and near-infrared image providing system, further comprising an image output unit that outputs image data provided by the image processing unit.
The method of claim 6,
The image output unit converts the near-infrared detection value into a virtual color and outputs the converted value.
The method according to claim 1,
Visible light of a wavelength irradiated by the irradiation light source unit is blue,
In the pixel detecting visible light of red and green wavelengths of the image detection sensor, only near-infrared rays are detected,
The image processing unit calculates a near-infrared ray detection value in a blue visible ray detection pixel by using the near-infrared ray detection value, and calculates an estimated blue visible ray detection value in a red and green visible ray detection pixel. Video delivery system.
The method according to claim 1,
The visible light of the wavelength irradiated by the irradiation light source is blue and green,
In the pixel detecting the red visible light of the image detection sensor, only near-infrared light is detected,
The image processing unit calculates a near-infrared ray detection value in a blue and green visible ray detection pixel by using the near-infrared ray detection value, and calculates an estimated blue and green visible ray detection value in a red visible ray detection pixel. And near-infrared image providing system.
The method according to claim 1,
Visible and near-infrared image providing system, characterized in that it further comprises a calibration unit for calibrating the detected value from the image detection sensor according to a preset reference.
A light source irradiation step of irradiating visible light and near-infrared light having a preset wavelength on the subject; And
Image data detected from an image detection sensor including pixels having similar sensitivities in the near-infrared region and arranged according to a preset pattern including the Bayer pattern color filter and detecting visible light regions of some wavelengths of red, green, and blue An image providing method of a simultaneous system of visible and near-infrared images comprising an image processing step of processing,
The light source irradiation step simultaneously irradiates visible and near-infrared rays excluding some wavelength regions of the red, green, and blue,
The image processing step,
A near-infrared detection value calculating step of calculating near-infrared detection values of other pixels by using near-infrared detection values obtained from some pixels in which only near-infrared rays are detected among the image detection sensors;
A visible light detection value calculation step of calculating a visible light detection value of the other pixels by using the calculated near infrared ray detection value; And
And calculating an estimated visible ray detection value of some of the pixels by using the visible ray detection value of the other pixels.
The method of claim 11,
And an image detection step of detecting visible and near-infrared rays through the image detection sensor.
The method of claim 12,
The visible and near-infrared image providing method, characterized in that some wavelength regions excluded include a red visible ray region.
The method of claim 13, wherein the light source irradiation step
Selectively irradiating visible light of the preset wavelength,
A method for providing visible and near-infrared rays, comprising irradiating near infrared rays having a preset excitation wavelength together with the visible rays to excite fluorescence.
The method of claim 12,
The image detecting step further comprises a filtering step of passing a near-infrared region and a visible ray region excluding an excitation wavelength irradiated to the subject.
The method of claim 11,
And an image output step of outputting image data provided by the image processing step.
The method of claim 16,
In the image outputting step, the near-infrared ray detection value is converted into a virtual color and then output.
The method of claim 11,
Visible light of the wavelength irradiated by the light source irradiation step is blue,
In the pixel detecting visible light of red and green wavelengths of the image detection sensor, only near-infrared rays are detected,
The image processing step includes calculating a near-infrared detection value in a blue visible ray detection pixel using the near-infrared ray detection value, and calculating an estimated blue visible ray detection value in a red and green visible ray detection pixel. Method of providing near-infrared image.
The method of claim 11,
The visible light of the wavelength irradiated by the light source irradiation step is blue and green,
In the pixel detecting the red visible light of the image detection sensor, only near-infrared light is detected,
The image processing step includes calculating a near-infrared detection value in a blue and green visible ray detection pixel using the near-infrared ray detection value, and calculating an estimated blue and green visible ray detection value in a red visible ray detection pixel. Method of providing light and near infrared images.
The method of claim 11,
And a calibration step of calibrating the detected value by the image detection sensor according to a preset reference.

Images