KR101681197B1

KR101681197B1 - Method and apparatus for extraction of depth information of image using fast convolution based on multi-color sensor

Info

Publication number: KR101681197B1
Application number: KR1020150063692A
Authority: KR
Inventors: 최종호; 김호배; 조효원
Original assignee: (주)이더블유비엠
Priority date: 2015-05-07
Filing date: 2015-05-07
Publication date: 2016-12-02
Also published as: KR20160132209A

Abstract

In the extraction method of the image depth information by using the high speed convolution of, for detecting the color image and the infrared image image input step of receiving the form Bayer pattern (Bayer Pattern), a color image within the contour from the multi-color sensor process, the median filter for removing noise (noise Reduction), and border highlighted (boundary Enhancement) (Median Filter) processing steps, and a contour for detecting the contours in the filtered image detection step, and the integral image generation method comprising: based on the infrared image received from the multi-color sensor generates an integral image, and the generated integrated image within at least 4 a fast convolution step of approaching a point of + alpha (alpha is possible from 0) to convolute an image and generating a plurality of blurred images using the convolution, and a fast convolution step of generating a single color (RGB) And a depth image generating step of generating a depth map by measuring a maximum degree of similarity between the blurred images .

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method and apparatus for extracting depth information of an image using a fast convolution based on a multi-

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method and apparatus for extracting depth information of an image using fast convolution based on a multi-color sensor, and more particularly, In the process of generating an image, at least 4 + N (N is possible from 0) in the image without a PSF model and not a method of approaching all pixels within a mask based on a plurality of PSF models for convolution, To a method and an apparatus for reducing memory usage and a method and apparatus for extracting depth information of an image using the same.

Due to the rapid development of image processing technology, 3D camera technology including color image and depth image is becoming an issue.

With 3D camera technology,

(1) Stereo camera (Bumblebee2)

(2) IR Pattern (Randomized dots) based camera (Kinect, Xtion)

(3) Time of Flight (TOF) camera (Kinect2)

.

It can be applied to various application fields by 3D reconstruction using depth image and XYZ three dimensional space expansion.

For example, the following application fields can be cited.

(1) Stereo 3D image generation - 3D display

(2) De-focusing or auto-focusing of a digital camera

(3) 3D Reconstruction of 3D Printing

(4) 3D motion recognition (Gesture Recognition)

In the prior art of this field

(1) Color image and depth information extraction technology using multi-color sensor based on Duar Aperture

(2) Kernel-based video convolution technology

And the like.

US Patent Publication No. US2013 / 0033579

Although a single multi-color sensor based on a dual aperture can be used to extract color images and depth information, it is difficult to process images due to the high computational complexity of extracting depth information and memory usage. . Therefore, it is necessary to reduce the amount of computation and memory usage.

Conventionally, a process of generating a plurality of blurred infrared images using a plurality of PSF (Point Spread Function) models has been performed in a process of extracting depth information. Here, we use memory for a number of PSF models for convolution of images and approach all pixels in kernel based mask (MASK).

However, according to the prior art, (1) the amount of computation is large and the processing speed is slow; and (2) the memory usage is very large.

The present invention reduces the time required to generate a plurality of infrared images convolved using only an integral value of at least 4 + N (N is 0 to 0) in an image without a PSF model, Processing technology was developed.

That is, it performs high-speed convolution based on a multi-color sensor to reduce the amount of computation, shorten the time required, and reduce the memory usage. At this time, the processing speed and the memory usage amount with respect to the conventional system are relatively compared and analyzed.

According to an aspect of the present invention, there is provided a method of extracting depth information of an image using fast convolution, comprising the steps of : inputting a color image and an infrared image from a multi-color sensor in a Bayer pattern; and, the color process for detecting the contour within the image, median filter for removing noise (noise Reduction), and border highlighted (boundary Enhancement) (Median Filter) processing steps, and a contour for detecting the contours in the filtered image detection step, and the integral image generation method comprising: based on the infrared image received from the multi-color sensor generates an integral image, and the generated integrated image within at least 4 a fast convolution step of approaching a point of + alpha (alpha is possible from 0) to convolute an image and generating a plurality of blurred images using the convolution, and a fast convolution step of generating a single color (RGB) And a depth image generating step of generating a depth map by measuring a maximum degree of similarity between the blurred images .

At this time, it is preferable that the 4 + alpha points are symmetrical to each other with respect to the pixel to be processed.

The present invention for achieving the above object, in the extraction assembly of the image depth information by using the high speed convolution of Bayer pattern color image and an infrared image from the multi-color sensor (Bayer Pattern) receiving in the form image input, and a color to the process for detecting the image within the contour, the median filter for removing noise (noise Reduction), and border highlighted (boundary Enhancement) (Median Filter) at least within the processing section and the and the contour detecting unit for detecting a contour line in the filtered image, and based on the infrared image received from the multi-color sensor and integral image generation unit for generating the integrated image generating integral image 4 + α A fast convergence unit for approaching a point of an image (a is possible from 0) to generate a plurality of blurred images using the convolution of the image, a single color (RGB) image and a plurality of blurring And a depth image generating unit for generating a depth map by measuring the maximum similarity between the blurred images .

According to the present invention, an image processing method and apparatus for shortening the time required to generate a plurality of convoluted infrared images and reducing memory usage, and extracting depth information of an image using the method.

FIG. 1 is a conceptual diagram of an apparatus for extracting depth information of an image using a high-speed convolution based on a multi-color sensor.
FIG. 2 shows a process of a median filter.
FIG. 3 shows an RGB image and an IR image input through a multi-color sensor.
FIG. 4 shows the image converted through the median filter and the outline detection result using the image. The mask size represents from 3 x 3 to 13 x 13.
5 shows a method of generating an integral image.
FIG. 6 shows a method of extracting a convolution value for a desired region in an integral image.
[Fig. 7] shows a process of generating an integral image.
FIG. 8 shows an input image received from a multi-color sensor and an integral image of the input image.
FIG. 9 shows a convolution process using an integral-image interpolation method.
10 shows a fast convolution result based on an integral image including an interpolation method. The results are convoluted from the 0th to the 80th.
11 shows data obtained by comparing and analyzing a convolution using a plurality of PSF models of 33 x 33 x 16 banks x 32 bits and a fast convolution based on an integral image.
12 shows the result of the depth information of the image. The image from the minimum bank to the maximum bank is mapped from red to blue.

Hereinafter, the present invention will be described in detail with reference to the accompanying drawings. However, the parts having the same function by the same configuration are denoted by the same reference numerals even if the drawings are different, and the detailed description thereof may be omitted.

A conceptual diagram of an apparatus for extracting depth information of an image using fast convolution based on the multi-color sensor in the present invention is shown in FIG.

A method for extracting depth information of an image using fast convolution, the method comprising: an image input step (1) for inputting a color image and an infrared image from a multi-color sensor in a Bayer pattern; a process for detecting the contours in the image, median filter for removing noise (noise Reduction), and border highlighted (boundary Enhancement) (Median Filter) processing step (2), the integral image generation step and the edge detection step (3) for detecting the contours in the filtered image, and based on the infrared image received from the multi-color sensor to generate the integral image (5) And a high-speed convergence step (6 ) for generating a plurality of blurred images by using a convolution of an image by approaching points of at least 4 + alpha (alpha is possible from 0) in the generated integral image ) and further characterized in that the color of the single (RGB) including an image and a plurality of blurring (blurring) the measures the maximum similarity between the image depth map (depth-map) depth image generating step (4) for generating a . However, the process of the above method may be implemented by software stored in a semiconductor storage device such as a ROM, a RAM, or a flash RAM, and the software may be distributed as a function of each function or module, and may be configured as a device.

In the image input step 1, an RGB image and an IR image can be classified and input using a dual aperture, and an input form is defined as a Bayer pattern. 3 shows an RGB image and an IR image received from a multi-color sensor.

The median filter processing step (2) represents a processing procedure in Fig. The neighboring pixels are collected based on one pixel to perform ascending sorting, and the middle value among the sorted values is mapped to one pixel value as a reference. Here, filtering is preferably performed so as to remove noise and emphasize the boundary lines so that highly reliable contours are detected.

More specifically, neighboring pixel points can be collected by setting a mask size based on a pixel at a point in the input image (u, v). 2, 1, 0, 0, 9, 1, 2, 3, 4, and 5 are obtained by setting the 3x3 mask size based on the pixel position of (u, v) 5, 8}. (U, v) is replaced with the median value ((mask size * mask size) / 2 + 1) th value in the ascending or descending order. 2, 3, 5, 7, 8, and 9} are arranged in ascending order in FIG. 2, and the middle value (fifth value) .

4 shows the outline detection result using the converted image and the filtered image through the median filter.

As the size of the mask in the median filter increases, the noise phenomenon is reduced and the interface is emphasized. FIG. 4 shows median filtering of mask sizes from 3x3 to 13x13 based on the same input image, showing the difference in noise phenomenon and emphasis effect on the boundary surface according to the mask size, and using each filtered image And the outline image derived through the same outline detection step is displayed. The effect of reducing the contour of noise according to the size of each mask and the contour of the boundary are emphasized.

The contour detection step 3 extracts a correlation value between neighboring pixels based on one pixel, extracts a difference value therebetween, and defines a contour line only when the extracted value is greater than the corresponding threshold value . The rest is defined as a black background.

The integral image generating step 5 accumulates data in a memory at intervals of one pixel, and generates an image with accumulated values. FIG. 5 illustrates a method of generating an integral image, FIG. 6 illustrates a method of extracting a convolution value for a desired region in an integral image, and FIG. 7 illustrates a process of generating an integral image. 8 shows the result of the integral image generated based on the input image.

More specifically, FIG. 5 shows a process of generating an integral image. d (i, j) is the integral value of the integral image of (i, j) and A (y, x) is the i and j pixels of the input image. The integrated value of the (i, j) -th integrated image requires a cumulative integral value of 3 points and a pixel of the input image of 1 point, and can be calculated by approaching 4 points. In detail, the integral value of (i, j) is summed with the integral value of (i, j-1) When the pixels of the image are summed, an integral value of d (i, j) is generated.

6 shows a process of convoluting only the corresponding region using only the integral value of four points in the generated integral image. The convoluted value of b (i, j) represents the convoluted value for the region with the magnitude of (w, h) based on the i, j address. The convoluted value of B (i, j) requires an integral value of at least 4 points + α in the integral image, and the integral value of d (i, jw) j), adds the integral value of d (ih, jw) which is the overlapping region, and then averages the result to the size of the convolution region to derive the convolution region.

7 shows a process of obtaining the sum of pixels of a corresponding region by generating an integral image and approaching four points in the integral image.

One point in the input image is defined as a pixel value (Pixel), and one point in the integral image is defined as an integral value (Integral Valus).

The integral image is generated by sequentially accumulating values based on the input image. For example, assuming that the input image is {1, 2, 2, 4, 1}, the integral images are sequentially accumulated values {1, 1 + 2, 1 + 2 + 2, 1 + 2 + 2 + 4 , 1 + 2 + 2 + 4 + 1} = {1, 3, 5, 9, 10}.

And shows the process of deriving the area to be subjected to convolution processing in the integral image only by a four-point approach. If the integral value of 46 in FIG. 7 is subtracted from the integral value of 22 and the integral value of 20, and the overlapping 10 is added, 46-22-20 + 10 = 14, and the sum of the pixel values of the corresponding region in the input image + 2 + 5 + 4 = 14). Finally, averaging is performed to the size of the region to be convoluted.

The fast convolution step 6 performs a convolution by averaging the values and magnitudes extracted from a point of minimum 4 * alpha (alpha is possible from 0 ) in the integral image. Interpolation according to the size of the mask is preferably applied.

9 shows a method in which an interpolation method considering the size of a mask is applied.

The size of the mask is 1, 3, 5, 7, 9, 11 and so on in an odd numbered process considering the symmetrical relationship with respect to the center point. Thus, to derive the convoluted values of 1.1, 1.2, 1.3, 1.4, 1.5 and so on including the convoluted values of even, 2, 4, 6, 8, .

A 3x3 mask and a 5x5 mask, for example, a convolution corresponding to 4x4 is derived. 4 points corresponding to the corners of each mask are extracted, and the value corresponding to the mask of 3x3 is multiplied by the weight of 0.5, and the value corresponding to the 5x5 mask is multiplied by the weight of 0.5 (i.e., average value). Taking as an example the values {1,5,6,21} in the 3x3 mask of FIG. 9 and {0,0,0,46} in the 5x5 mask, the four point values of the 4x4 mask are {0 * 0.5 + 1 * 0.5 + 5 * 0.5, 0 * 0.5 + 6 * 0.5, 46 * 0.5 + 21 * 0.5} = {0.5. 2.5. 3.0. 33.5} and performs convolution.

FIG. 10 shows a blurred result image from steps 0 to 80. In the case of a convolution based on a conventional PSF with a size of 33 x 33, a 16-bass PSF model uses a PSF model memory of 33 x 33 x 16banks x 32bits = 544.4 bytes, and 512 x 384 x 33 x 33 x 16 banks = 3,425,697,792 Clock count is required. In the case of the fast convolution of the present invention, 512 x 384 x 8 (8 points including interpolation) x 16 banks = 25,165,824 clocks are required based on an image of 512 x 384 size without using a PSF memory. Shorten the number of times. Figure 11 shows a comparison of the convolution based on the existing PSF and the fast convergence of a point approach of at least 4 * alpha (alpha is possible from 0).

The depth image generation step 4 generates a depth map by measuring maximum similarity using a plurality of transformed images blurred through high speed convolution with an RGB image input from a multi-color sensor .

Fig. 12 shows the result of the depth information of the image. The image from the minimum bank to the maximum bank is mapped from red to blue.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed embodiments, but, on the contrary, is intended to cover various modifications and equivalent arrangements included within the scope of the appended claims.

INDUSTRIAL APPLICABILITY The present invention can be applied to a method and apparatus for extracting depth information of an image using fast convolution based on a multi-color sensor.

1:
2: Median filter processing unit
3: contour detector
4: Depth image generation unit
5: Integrated image generation unit
6: Fast convolution unit
7: Output section

Claims

A method for extracting depth information of an image using fast convolution,
An image input step of receiving a color (RGB) image and an infrared (IR) image from a multi-color sensor in the form of a Bayer pattern;
A process for detecting a contour in a color image includes a median filter processing step for noise reduction and boundary enhancement,
An outline detection step for detecting an outline in the filtered image,
An integral image generation step of generating an integral image based on an infrared image input from a multi-color sensor,
A fast convolution step of generating a plurality of blurred infrared images by using the extracted values and magnitudes based on four pixels corresponding to the four corners of the generated integral image, Wow,
A depth image generation step of generating a depth map by measuring a maximum similarity between a single color image and a plurality of blurred infrared images
And extracting the depth information of the image using the fast convolution.

The method according to claim 1,
The four pixels corresponding to the four corners of the quadrangle are symmetrical with respect to the pixel to be processed
And extracting depth information of the image using high-speed convolution.

An apparatus for extracting depth information of an image using fast convolution, the apparatus comprising:
An image input unit for receiving a color (RGB) image and an infrared (IR) image from a multi-color sensor in a Bayer pattern,
A process for detecting an outline in a color image includes a median filter processing unit for noise reduction and boundary enhancement,
An outline detection unit for detecting an outline in the filtered image,
An integral image generating unit for generating an integral image based on the infrared image input from the multi-color sensor,
A high speed convolution unit for generating a plurality of blurred infrared images by using the extracted values and magnitudes based on four pixels corresponding to four corners forming the rectangle in the generated integral image, Wow,
A depth image generating unit for generating a depth map (Depth-Map) by measuring a maximum similarity between a single color image and a plurality of blurred infrared images ,
And extracting the depth information of the image using the fast convolution.

The method of claim 3,
The four pixels corresponding to the four corners of the quadrangle are symmetrical with respect to the pixel to be processed
And extracting depth information of the image using high-speed convolution.