KR101796523B1 - Enhancing method and system of accuracy of depth map under variable illumination - Google Patents

Abstract

The method of the present invention is a method for improving the accuracy of depth information calculated from color images (R, G, B) and infrared images (IR) input by multi-color sensors under variable illumination, A histogram generation step of generating histograms for R, G, B, and IR channels, a histogram generation step for generating R, G, B, and IR channel histograms, An adaptive color selection step of selecting a color of a channel to be used as a blurred color and a focused color used for calculating the depth information by comparing the amount of change in the channel, an outline detection step of detecting an outline , A PSF for generating blurred reference images for the focused color, A conversion step, and calculating a correlation of the image of the blur and the color blur reference image correlation processing step, and a depth map generation step of generating a depth map showing the depth of the step is characterized by the included.

Description

TECHNICAL FIELD [0001] The present invention relates to a method and system for enhancing depth information accuracy under variable illumination,

The present invention relates to a method and system for improving depth information accuracy under variable illumination, and more particularly, to a method and system for calculating depth information using a multi-color sensor with high accuracy even when the type and nature of illumination change. will be.

Due to the rapid development of image processing technology, 3D camera technology including color image and depth image is becoming an issue.

With 3D camera technology,

(1) Stereo camera (Bumblebee2)

(2) IR pattern (Randomized dots) based camera (Kinect, Xtion)

(3) Time of Flight (TOF) camera (Kinect2)

.

It can be applied to various application fields by 3D reconstruction using depth image and XYZ three dimensional space expansion.

For example, the following application fields can be cited.

(1) Stereo 3D image generation - 3D display

(2) De-focusing / Auto-focusing of a digital camera

(3) 3D reconstruction of 3D printing (Reconstruction)

(4) 3D motion recognition (Gesture Recognition)

In the prior art of this field,

(1) Color image and depth information extraction technology using multi-color sensor based on dual aperture

(2) Kernel-based video convolution technology

And the like.

Meanwhile, a multi-color sensor acquires a color (RGB) image and an infrared (IR) image through a dual aperture. A technique of generating a depth map having depth information by using the color image and the infrared image is known. The depth map can be used, for example, for depth image generation.

The depth information is obtained by correlating a color image generated by picking up an object with an infrared image and extracting the maximum similarity. The quality factors in depth information extraction are accuracy and speed.

US Patent Application Publication No. US2013 / 0033579 " Multi-Aperture Image Data "

In the prior art, it is assumed that only one kind of illumination, for example, halogen illumination, is fixedly used when acquiring a color image and an infrared image. As described above, in the depth information acquisition system of the related art, which is specialized only for one kind of illumination, there is a problem that depth information acquired under different types of illumination other than the assumed kind becomes inaccurate. That is, there is a problem that the prior art is susceptible to variable changes such as brightness and color of illumination.

In the actual environment, natural light of variable brightness, incandescent light, fluorescent light, halogen and the like are present, and light of various colors such as Red, Green, Blue, and Yellow is included as shown in FIG. Therefore, in the depth information calculation, it is necessary to adaptively cope with the change of illumination environment. That is, in consideration of the increasingly diverse application fields, it is desirable that the multi-color sensor can be used while maintaining the accuracy under various lighting environments such as natural light, fluorescent light, incandescent light, and halogen.

SUMMARY OF THE INVENTION It is an object of the present invention to provide a method and system for improving depth information accuracy under variable illumination that can improve the accuracy of depth information extraction even under various lighting and color environments.

The method of the present invention for solving the above problem is a method for improving the accuracy of depth information calculated from color images (R, G, B) and infrared images (IR) which are input images by a multi-color sensor under variable illumination A histogram generation step of generating a histogram for each of the R, G, B and IR channels for the input image, a histogram for each of the generated R, G, B and IR channels, An adaptive color selection step of selecting a color of a channel to be used as a blurred color and a focused color used for calculating the depth information by comparing the amount of change of the channel, an outline detection step of detecting the outline , Based on a PSF (Point Spread Function) model generated in advance, a PSF for generating blurred reference images for the focused color A conversion step, and calculating a correlation of the image of the blur and the color blur reference image correlation processing step, and a depth map generation step of generating a depth map showing the depth of the step is characterized by the included.

Here, the stretching of the histogram is preferably performed according to a linear histogram stretching formula .

Here, the linear histogram stretching formula may be expressed as:

Min and Max denote the minimum and maximum brightnesses of respective histograms of R, G, B, and IR channels, and I (N) And newMax may be configured to represent the minimum and maximum brightness of the integrated (R + G + B + IR) histogram.

The histogram generation step may be performed at a time point (t-1), and the adaptive color selection step may be performed at a time point (t).

Meanwhile, the system of the present invention improves the accuracy of depth information calculated from color images (R, G, B) and infrared images (IR) input by multi-color sensors under variable illumination, A histogram generator for generating a histogram for each of the R, G, B, and IR channels, a histogram generator for generating R, G, B, and IR channels, By comparing the amount of change in the channel, an adaptive < RTI ID = 0.0 & gt ; And a color selector, and the contour detecting unit for detecting a contour, based on the PSF PSF model preloaded generated, with respect to the focused color, generating a reference image blur Conversion unit, and the image and calculating a correlation of the reference image correlation processing blur, depth map for generating a depth map showing the depth of the step of the color blur And a generating unit .

Here, the stretching of the histogram is preferably performed according to a linear histogram stretching formula .

Here, the linear histogram stretching formula may be expressed as:

Min and Max denote the minimum and maximum brightnesses of respective histograms of R, G, B, and IR channels, and I (N) And newMax preferably represent the minimum and maximum brightness of the integrated (R + G + B + IR) histogram.

It is preferable that the histogram generation step is performed at a time point (t-1), and the adaptive color selection step is performed at a time point (t).

According to the present invention, the accuracy of extraction of depth information can be improved even in an illumination environment of various brightness and color.

FIG. 1 shows a block diagram of a high-speed depth image extracting apparatus based on a multi-color sensor.
[Fig. 2] shows an input image by illumination of red, green, blue, and yellow colors.
FIG. 3 represents the definition of a formula of histogram stretching.
4 shows the results of (a) histogram and (b) histogram stretching for the input image.
5 compares the pre-applied image and the post-applied image of the histogram stretching for each channel of the input image.
Fig. 6 shows depth map results before (a) application of the histogram stretching and after (b) application.

Hereinafter, a method and system for improving depth information accuracy under variable illumination according to the present invention will be described in detail with reference to the accompanying drawings. However, the same reference numerals are used to denote members performing the same function by the same configuration, even if the drawings are different, and the detailed description thereof may be omitted. Also, to be connected includes the concept of passing through the medium in the middle.

<Method>

The method of improving the depth information accuracy under the variable illumination of the present invention is a technique that can be applied not only to one kind of illumination but also to a variable illumination environment in which various lights such as an incandescent lamp, a fluorescent lamp, a halogen lamp, It is possible to improve the accuracy of the depth information calculated from the color images R, G, and B and the infrared image IR, which are input images 10 by the multi-color sensor. According to the present invention, the image is adaptively corrected using histogram stretching, thereby extracting a uniform degree of depth information even under a variable illumination environment, thereby improving the reliability of depth information.

The method includes a histogram generation step 30 , an adaptive color selection step 40 , an outline detection step 50 , a PSF Conversion step 60 , correlation processing step 70, and depth map generation step 80 are included. For example, the histogram generation step is performed at a time point (t-1), and the adaptive color selection step is performed at a time point (t).

The histogram generation step 30 generates a histogram for each of the R, G, B and IR channels (see FIG. 4) for the R, G, B and IR input images 10 in the Bayer pattern . The histogram shows the brightness contrast of the multi-color image (R, G, B, IR) as the input image. The histogram information includes the minimum, maximum value, average value, and variance value information of 1 to 254 values in brightness.

The adaptive color selection step (40) stretches the histogram using the generated histogram information for each of the R, G, B, and IR channels, By comparing the amount of change of the channel, it is the step of selecting the color of the channel to be used as the blurred color and the color of the channel to be used as the focused color adaptively used for calculating the depth information. According to the stretching, the contrast histogram of the input image is readjusted uniformly.

Conventionally, for an input image input from a multi-channel sensor, the color of a channel to be used as a blurred color is determined to be any one of R, G, and B (mainly R or G) , And IR. However, this method has a problem that accuracy of extracting depth information is changed when illumination is changed.

In the present invention, the histogram is stretched as described above by using the fact that the histogram is changed according to the illumination, and the color of the channel to be used as the focused color and the color of the channel to be used as the blurred color are changed according to the amount of change from the original histogram Therefore, the color of the optimal channel is determined according to the illumination, so that there is an effect that is robust against illumination change.

The histogram stretching formula of FIG. 3 is used for this histogram stretching. The histogram stretching equations are largely divided into linear equations and nonlinear equations. The linear equation uses only the histogram information in the image, and the nonlinear equation is influenced by two experimentally generated constant values.

Here, the nonlinear equation is naturally effective for the environment, in which the value of the optimized parameter is applied as a constant value under an environment where the illumination property is fixed. However, this means that under a variable illumination environment, one constant value can not exhibit optimized performance, and furthermore, in a variable illumination environment, the performance deteriorates. Linear formulas do not have these constraints.

In order to adaptively adapt to the illumination, in the present invention, the histogram stretching is selected to be performed according to the linear histogram stretching formula . For example, the linear histogram stretching formula may be expressed as:

. &Lt; / RTI &gt;

Here, I defines pixels in an image, and I (N) means a converted pixel. Min and Max denote the minimum and maximum brightness of each histogram of the R, G, B and IR channels, and newMin and newMax denote the minimum and maximum brightness of the integrated (R + G + B + IR) .

FIG. 4 is a graph showing the relationship between the original histogram (left side) of the R, G, B and IR channels and the histogram corrected by histogram stretching (right side) for the R, G, Show. Here, the red line indicates a red channel, the green line indicates a green channel, the blue line indicates a blue channel, and the purple line indicates an IR channel. Looking at the stretched histogram, it can be seen that the blue and IR channels, which were clustered in the low brightness region, are mainly spread evenly.

Fig. 5 shows images before and after application of the histogram stretching. It is confirmed that the histograms of the R, G, B, and IR channels are generated differently depending on the illumination of various colors of R, G, B, and Y in the original histogram (Before) before the stretching application. On the other hand, when the histogram after stretching is applied, the histogram is corrected so that the image is corrected adaptively in various variable illumination environments, and the difference of the histograms for each channel of R, G, B, and IR is almost It can be seen that it is not.

The outline detection step (50) is a step of detecting an outline. The contour detection step largely includes noise reduction and contour extraction processing. The noise elimination uses a median filter, a weighted median filter, and a bilateral filter, which are typically used in image processing. This has the effect of eliminating noise and emphasizing boundaries, which is effective in improving the accuracy of contour detection. The contour extraction is performed by a known method.

The PSF The conversion step 60 is a step of generating reference blurred reference images for the focused color based on a PSF (Point Spread Function) model generated in advance. Here, reference images are comparative images for extracting depth information.

The correlation processing step 70 is a step of calculating a correlation between the blurred color image and the blurred reference images. The principle of the correlation is to extract the difference value between the sample and the reference, extract the minimum and maximum value based on the difference, and determine the similarity based on the extracted minimum and maximum values.

The depth map generating step 80 is a step of generating a depth map indicating a depth step. In the depth map generation step, a step corresponding to the reference image obtained by obtaining the maximum similarity degree is represented by depth information in consideration of the extracted correlation relationship, and color mapping is performed thereon. For example, the mapped color information is expressed in red for a closer distance, and blue for a longer distance.

When the depth information is obtained as described above, for example, RGB + IR and a display processing step 90 for outputting depth information may be added.

<Effect>

Fig. 6 shows the result of depth information calculation before and after applying the histogram stretching. Here, the lights R, G, B, and Y represent typical illumination colors. (A) shows the depth map calculated based on the images of the original unretained R, G, B and IR channels. The equipment used in this experiment is a depth map of R, G, The results are inaccurate (close to each other), especially for B-color lighting, resulting in very inaccurate results due to loss of near and far information. On the other hand, the depth map calculated based on the image of the stretched corrected R, G, B, and IR channels of the present invention is shown in (b) , There is not a large difference between them, and it can be seen that they do not show particularly inaccurate results.

Therefore, even under environments of varying illumination brightness and illumination of various colors, the histogram stretching evenly corrects each channel image, so that the depth information obtained from this improves the reliability under any illumination environment.

<Device / system>

The method of the present invention can be implemented as a system by a data processing device including a memory, a functional processing semiconductor, etc., with a data processing unit such as a CPU or a microprocessor as a center. Each step of the method may be implemented corresponding to each part of the system, and in actual implementation, some parts may be integrated into another part, or may be divided into other parts.

A conceptual diagram of an apparatus for extracting depth information in an image, at high speed, based on a multi-color sensor is shown in Fig. 1, the system of the present invention includes a histogram generator 30 , A color selection unit 40 , an outline detection unit 50 , a PSF Conversion unit 60 , a correlation processing unit 70 , a depth map And a generation unit (80) . The detailed operation of each part will be omitted by referring to the operation of each corresponding step of the method of the present invention.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed embodiments, but, on the contrary, is intended to cover various modifications and equivalent arrangements included within the scope of the appended claims.

INDUSTRIAL APPLICABILITY The present invention can be applied to a method and system industry for improving depth information accuracy under variable illumination.

10: input image
20: Memory interface
30: histogram generator
40: adaptive color selector
50: contour detector
60: PSF conversion section
70: Correlation processor
80: Depth map generation unit
90:

Claims (8)

A method for improving the accuracy of depth information calculated by a blurred color and a focused color from an input image under variable illumination,
(R, G, B) and infrared (IR) images, which are input images by a multi-color sensor based on a dual aperture in an environment under variable illumination, A histogram generation step of generating a histogram for each channel of the channel,
The histogram of each channel of the generated R channel, G channel, B channel and IR channel is stretched to adaptively adjust the input image so that the contrast histogram of the input image is uniformly readjusted. Each channel, B channel, and IR channel By comparing the amount of change in the channel, among the colors of the R channel, G channel, B channel, and IR channel of the input image from the multi-color sensor based on the dual aperture under variable illumination, the blurred color An adaptive color selection step of adaptively selecting a color of a channel to be most suitably used and a color of a channel to be most suitably used as a focused color,
An outline detection step of detecting an outline from the blurred color adaptively selected and the image of the focused color,
A PSF conversion step of generating blurred reference images for the focused color based on a PSF model generated in advance;
A correlation processing step of calculating a correlation between the blurred color image and the blurred reference images,
A depth map generation step of generating a depth map indicating a step of the entire depth of the color image by the multi-color sensor under the variable illumination by the depth value of the blurred reference image showing the maximum degree of similarity from the correlation
Wherein the depth information of the variable illumination is obtained by the following method. The method according to claim 1,
The stretching of the histogram is performed according to a linear histogram stretching formula .
Wherein the depth information under the variable illumination is enhanced. The method of claim 2,
The linear histogram stretching formula may include:

Lt; / RTI &gt;
I defines pixels in an image,
I (N) means the converted pixel,
Min and Max represent the minimum and maximum brightness of each histogram of the R, G, B and IR channels,
newMin and newMax represent the minimum and maximum brightness of the integrated (R + G + B + IR) histogram.
Wherein the depth information under the variable illumination is enhanced. The method according to any one of claims 1 to 3,
The histogram generation step is performed at a time point (t-1)
After the adaptive color selection step is performed at time t
Wherein the depth information under the variable illumination is enhanced. A system for improving the accuracy of depth information computed by a blurred color and a focused color from an input image under variable illumination,
(R, G, B) and infrared (IR) images, which are input images by a multi-color sensor based on a dual aperture in an environment under variable illumination, A histogram generator for generating a histogram for each channel of the channel,
The histogram of each channel of the generated R channel, G channel, B channel and IR channel is stretched to adaptively adjust the input image so that the contrast histogram of the input image is uniformly readjusted. Each channel, B channel, and IR channel By comparing the amount of change in the channel, it is possible to obtain a blurred color to be used for calculating the depth information, among the colors of the R channel, G channel, B channel, and IR channel of the input image from the multi-color sensor based on the dual aperture under variable illumination An adaptive color selector for adaptively selecting a color of a channel to be most suitably used as a focused color and a color of a channel to be most suitably used as a focused color,
An outline detecting unit for detecting an outline from the blurred color and the focused color image adaptively selected,
A PSF conversion unit for generating a plurality of blurred reference images with respect to the focused color based on a PSF model generated in advance;
A correlation processor for calculating a correlation between the blurred color image and the plurality of blurred reference images,
A depth map generating unit for generating a depth map indicating a step of the entire depth of the color image by the multi-color sensor under the variable illumination by the depth value of the blurred reference image showing the maximum similarity from the correlation,
Wherein the depth information accuracy enhancement system is a variable depth illumination system. The method of claim 5,
The stretching of the histogram is performed according to a linear histogram stretching formula .
A depth information accuracy enhancement system under variable illumination. The method of claim 6,
The linear histogram stretching formula may include:

Lt; / RTI &gt;
I defines pixels in an image,
I (N) means the converted pixel,
Min and Max represent the minimum and maximum brightness of each histogram of the R, G, B and IR channels,
newMin and newMax represent the minimum and maximum brightness of the integrated (R + G + B + IR) histogram.
A depth information accuracy enhancement system under variable illumination. The method according to any one of claims 5 to 7,
The histogram generation unit is performed at a time point (t-1)
After the adaptive color selection unit is performed at time t
A depth information accuracy enhancement system under variable illumination.

Images