KR101629454B1 - Apparatus and method for enlarging depth map - Google Patents

Abstract

A depth image magnifying apparatus and method are disclosed. The depth image enlargement device detects a boundary line of an object in a low resolution depth image and expands the depth image using an asymmetric filter when an object boundary exists in the search window.

Asymmetric filter, depth image, enlargement, expansion

Description

TECHNICAL FIELD [0001] The present invention relates to a depth image enlarging apparatus,

The following embodiments relate to an apparatus and method for enlarging a depth image.

When synthesizing a color image at a certain time using a color image and a depth map, the resolution of the input color image and the depth image should be the same. However, when a low-resolution depth camera is used or a low-resolution depth image is received due to transmission bandwidth limitation, a method of increasing the resolution of the depth image is needed.

The color image and the depth image can be compressed and transmitted together using a compression method such as Joint Multi View Codec (JMVC) in three-dimensional and free-view images. In order to prevent the increase of the bit generation amount, a method of reducing the spatial resolution of the depth image is needed because the bit generation amount is increased by transmitting the depth image together with transmission of the conventional color image. In this method, the transmitting terminal compresses and transmits the reduced depth image, enlarges the original image at the receiving end, and then synthesizes the image at the intermediate or arbitrary time using the color image.

According to an aspect of the present invention, there is provided a depth image enlarging apparatus including a boundary line detecting unit for detecting a boundary line of an object in a depth image having a low resolution, a determination unit for determining whether a boundary line of the detected object exists in the search window, And an image magnifying unit for enlarging the low-resolution depth image using an asymmetric filter if it is determined that the boundary line of the object exists inside the window.

According to an aspect of the present invention, the image enlargement unit may include a filter mask coefficient determination unit for determining a coefficient of a filter mask for the asymmetric filter, a filter mask for matching the coefficient of the determined filter mask and a low- And an interpolation unit interpolating the depth image, wherein the low resolution depth image has a lower resolution than the color image.

According to an aspect of the present invention, the filter mask coefficient determination unit may include a pixel scanning unit that scans an upper left region pixel, an upper right region pixel, a lower left region pixel, or a lower right region pixel based on the interpolation position; And a determination unit for determining a coefficient of the extended filter mask by expanding the filter mask using the zero padding result and the zero padding unit for zero padding.

According to an aspect of the present invention, when the boundary line of the object exists in the scanned pixel area, the zero padding unit sets the coefficient of the filter mask corresponding to the pixel of the boundary line and the pixel outside the boundary line to '0' .

According to an aspect of the present invention, the determination unit expands the filter mask using a portion located at a symmetrical position of a portion set to '0' according to the result of the zero padding, Can be set using the coefficients of the initial filter mask corresponding to the zero padded area.

According to another aspect of the present invention, there is provided a depth image enlarging method comprising the steps of: detecting a boundary of an object in a depth image with a low resolution; determining whether a boundary line of the detected object exists in the search window; And enlarging the low-resolution depth image using an asymmetric filter if it is determined that the boundary line of the object exists in the interior.

According to an aspect of the present invention, the step of enlarging the depth image includes a step of determining a filter mask coefficient for determining a coefficient of a filter mask for the asymmetric filter, And interpolating the depth image by convoluting the depth image, wherein the low resolution depth image has a lower resolution than the color image.

According to an aspect of the present invention, the step of determining the filter mask coefficient may include scanning an upper left area pixel, an upper right area pixel, a lower left area pixel, or a lower right area pixel based on the interpolation position, Analyzing and performing zero padding, and expanding the filter mask using the zero-padded result and determining coefficients of the extended filter mask.

According to an aspect of the present invention, the zero padding may be performed when a boundary of the object exists in the scanned pixel area, and a coefficient of a filter mask corresponding to pixels on the boundary line and pixels outside the boundary line is '0' .

According to an aspect of the present invention, the step of determining the coefficient of the extended filter mask may include expanding the filter mask using a portion located at a symmetrical position of a portion set to '0' according to the result of the zero padding And a coefficient of the extended filter mask portion may be set using a coefficient of an initial filter mask corresponding to the zero padded region.

An embodiment of the present invention can provide an apparatus and method for expanding a depth image of a low resolution to a resolution level of an original color image at a receiving end when a depth image is reduced and transmitted at low resolution.

Also, embodiments of the present invention can provide an apparatus and method for restoring a high-resolution depth image by interpolating pixels for an area including a boundary of an object using an asymmetric filter in a low-depth depth image.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. However, the present invention is not limited to or limited by the embodiments. Like reference symbols in the drawings denote like elements.

BRIEF DESCRIPTION OF DRAWINGS FIG. 1 is a diagram showing a configuration of a depth image magnifying apparatus according to an embodiment of the present invention; FIG.

Referring to FIG. 1, a depth image enlarging apparatus 100 according to an embodiment of the present invention includes a boundary detecting unit 110, a determining unit 120, a first interpolating unit 130, and an image enlarging unit 140 And enlarges the depth image of the resolution lower than the resolution of the color image to the resolution level of the color image.

The boundary detection unit 110 receives the low-resolution depth image and analyzes the input low-resolution depth image to detect the boundary of the object. For example, the boundary detection unit 110 can detect a boundary line of an object included in the analyzed low-resolution depth image by analyzing the low-resolution depth image as a sobel detector or a canny detector.

The determination unit 120 sets a search window at the current pixel position and determines whether a boundary line of the detected object exists in the search window.

The first interpolator 130 interpolates the low-resolution depth image using a symmetric filter and outputs a high-resolution depth image if it is determined that the boundary of the object does not exist in the search window. For example, the first interpolator 130 outputs the low resolution depth image to the high resolution depth image enlarged to the resolution level of the color image through bicubic interpolation or Lanzcos interpolation using the symmetric filter. can do.

If it is determined that the boundary line of the detected object exists in the search window, the image enlarging unit 140 interpolates the low resolution depth image using the asymmetric filter to output the high resolution depth image. The image enlarging unit 140 includes a filter mask coefficient determiner 141 and a second interpolator 142. Hereinafter, the configuration and operation of the filter mask coefficient determination unit 141 will be described in more detail with reference to FIG.

2 is a diagram showing an example of a specific configuration of the filter mask coefficient determination unit shown in FIG.

2, the filter mask coefficient determination unit 141 includes a pixel scanning unit 210, a zero padding unit 220, and a determination unit 230. The filter mask coefficient determination unit 141 determines whether the detected boundary line exists in the search window The filter mask coefficient for the asymmetric filter is determined.

The pixel scanning unit 210 scans the upper left region pixel, the upper right region pixel, the lower left region pixel, or the lower right region pixel based on the interpolation position. That is, the pixel scanning unit 210 scans the pixels according to the order for borderline search in the upper left area, the upper right area, the lower left area, or the lower right area based on the interpolation position.

3 is a diagram showing an example in which an area of a search window is divided around an interpolation position.

Referring to FIG. 3, the pixel scanning unit 210 scans the search window 300 to the upper left region 310, the lower right region 320, the lower right region 330, and the upper right region 340, And scans the pixels included in each of the regions 310, 320, 330, For example, the pixel scanning unit 210 scans a pixel in order from the starting points of the boundary searching in the upper left region 310, the lower left region 320, and the lower right region 330 in order of searching for a boundary, There are no borderline pixels.

4 is a diagram showing an example of a coefficient for the initial filter mask.

Referring to FIG. 4, the initial filter mask is a 6 × 6 symmetric filter having a symmetric coefficient with respect to the center as shown in Equation (1).

5 is a diagram showing an example in which coefficients of a filter mask for an area not including a boundary line are determined.

Referring to FIG. 5, the filter mask coefficient determiner 141 determines that the coefficient of the filter mask for the region not including the boundary line is equal to the coefficient of the initial filter mask. For example, the filter mask coefficient determining unit 141 does not include the boundary line in the upper left region 310, the lower right region 320, and the lower right region 330 shown in FIG. 3, Lt; RTI ID = 0.0 > filter mask. ≪ / RTI > At this time, the upper right region 340 may not be yet pixel-scanned and the filter coefficient may be marked as Not Determined (ND).

6 is a diagram showing an example of pixel scanning for an area including a boundary line among divided areas of a search window.

Referring to FIG. 6, the pixel scanning unit 210 scans a pixel according to the order for boundary detection based on the starting point of the boundary search in the upper right region 600. That is, the pixel scanning unit 210 performs pixel scanning in the upper right region 600 of the edge map of the depth image corresponding to the initial filter mask to check whether the depth image is a boundary line, Search.

The zero padding unit 220 analyzes the scanned pixel and performs zero padding on the scanned pixel. That is, when the boundary line of the object exists in the scanned pixel, the zero padding unit 220 sets the coefficient of the filter mask corresponding to the borderline pixel and the borderline pixel to '0'. The pixel outside the boundary may be a pixel existing after the boundary in the order for searching for the boundary.

7 is a diagram showing an example in which coefficients of a filter mask for an area including a boundary line are determined.

Referring to FIG. 7, when the zero padding unit 220 is a boundary pixel included in the upper right region 600 and a boundary pixel outside the boundary line as shown in FIG. 6, unlike the coefficient of the initial filter mask shown in FIG. 4, 0 '.

The determination unit 230 expands the filter mask using the zero padded result and determines the coefficients of the extended filter mask.

8 is a diagram showing an example of an extended filter mask.

8, the determining unit 230 may extend the filter mask using a portion 820 that is symmetrically located in the zero-padded portion 810, and may multiply the coefficients of the expanded filter mask by the zero- 810, < / RTI >

The determination unit 230 can remove the unstable noise near the boundary in the depth image by increasing the size of the filter mask with respect to the depth image to increase the smoothing.

9 is a diagram showing an example of the coefficients of the extended filter mask.

Referring to FIG. 9, the extended filter mask 920 is 8 × 6, which is greatly expanded in size from 6 × 6, which is the size of the initial pre-expansion filter mask 910. In the initial filter mask shown in FIG. A coefficient corresponding to the position of the zero padded portion shown in FIG. 7 may be used to determine the coefficient of the filter mask for the extended portion.

The second interpolator 142 convolutes the asymmetric filter mask according to the determined filter mask coefficient and the input low resolution depth image. For example, the second interpolator 142 performs a convolution operation on the normalized asymmetric filter mask and the low-resolution depth image by multiplying the coefficients of the determined filter mask by a normalization constant, thereby interpolating the low-resolution depth image It is possible to output a high-resolution depth image enlarged to the resolution level of the color image.

FIGS. 10 and 11 are views showing an example of a case where borderline pixels immediately adjacent to the interpolation position are present. FIG.

10 and 11, the second interpolator 142 can interpolate pixels as an average of two input pixels if there are two symmetrical border pixels among the four pixels immediately adjacent to the interpolation position .

FIG. 12 is a diagram showing an example of an asymmetric filter mask, and FIG. 13 is a diagram showing an example of a mask pixel for an input image.

12 and 13, the second interpolator 142 convolutes the asymmetric filter mask according to the coefficients of the filter mask shown in FIG. 12 and the mask pixels for the low-resolution depth image shown in FIG. 13, By interpolating the depth image, a high-resolution depth image can be output.

As described above, the depth image enlarging apparatus 100 according to an embodiment of the present invention interpolates pixels of an area including a boundary line of an object in a low-resolution depth image using an asymmetric filter, thereby generating a noise boost phenomenon or an artifact .

14 is a flowchart illustrating an operation of a depth image enlargement method according to an embodiment of the present invention.

Referring to FIG. 14, in step S1410, the depth image enlargement apparatus detects an object boundary line for a low-resolution depth image. That is, in step S1410, when the depth image of low resolution is inputted, the depth image enlargement device detects an object boundary line with respect to the inputted low resolution depth image by using an edge detector. For example, in step S1410, the depth image enlargement apparatus can detect an object boundary line at a low-resolution depth image using a Sobel detector or a canny detector.

In step S1420, the depth image enlargement unit sets a search window at a position of the current pixel, and determines whether the boundary of the detected object exists in the set search window.

If the boundary is not present, the depth image enlargement unit performs interpolation on the low-resolution depth image using a symmetric filter in step S1430. For example, in step S1430, if it is determined that the boundary does not exist in the search window, if the depth image enlarging device determines that the boundary is not near the boundary, By performing interpolation on the image, it is possible to enlarge it to a high-resolution depth image.

If the boundary exists, the depth image magnifying apparatus determines a filter mask coefficient in step S1440, and interpolates the depth image using the asymmetric filter according to the determined filter mask coefficient (S1442). That is, in step S1440, if the boundary exists in the search window, the depth image enlargement unit determines the mask coefficient of the asymmetric filter as the current pixel position as the boundary region (S1441) (Step S1442). The depth image is obtained by interpolating the asymmetric filter mask and the low-resolution depth image.

FIG. 15 is a detailed flowchart of a depth image enlargement method for a case where a boundary exists in a search window.

Referring to FIG. 15, in step S1511, when it is determined that a boundary exists in the search window, the depth image enlargement unit calculates a depth image from the start point in the upper left region of the region divided into quadrants based on the interpolation position within the search window The pixels are scanned in order for searching for a borderline.

In step S1512, if it is determined that a border exists in the search window, the depth image enlargement unit searches for a boundary line from the start point in the lower left region among the regions divided into quadrants based on the interpolation position within the search window Scan the pixels as they are.

In step S1513, if it is determined that a border exists in the search window, the depth image enlargement unit searches for a boundary line from the start point in the right lower region among the four divided regions based on the interpolation position within the search window Scan the pixels as they are.

In step S1514, if it is determined that a border exists in the search window, the depth image enlargement unit searches for a boundary line from the start point in the upper right region among the regions divided into quadrants based on the interpolation position within the search window Scan the pixels as they are.

In step S1520, the depth image enlargement unit zero paddes the pixels scanned by the boundary pixels or pixels existing after the boundary pixels. That is, in step S1520, the depth image magnifying apparatus sets the filter mask coefficient to '0' for the pixels whose scanned pixels are present after the boundary pixels or the boundary pixels. For example, in step S1520, the depth image enlargement apparatus does not change the initial mask filter coefficients because there is no boundary line in the upper left region 310, the lower right region 320, and the lower right region 330 as shown in FIG. The coefficient of the filter mask corresponding to the pixel corresponding to the edge map boundary line or the pixel outside the boundary line in the upper right region 600 as shown in FIG. 6 may be changed to '0'.

In step S1530, the depth image enlargement device enlarges the size of the filter mask with respect to the depth image of the low resolution, and sets the coefficients of the extended filter mask. That is, in step S1530, the depth image magnifying apparatus enlarges the size of the filter mask using the zero-padded portion, and adds the coefficients of the expanded asymmetric filter mask to the coefficients of the initial filter mask for the zero- . For example, in step S1530, the depth image enlargement device may expand the size of the filter mask for the low-resolution depth image to an 8x6 asymmetric filter mask larger than the initial mask filter size of 6x6. For example, in step S1530, the depth image enlargement unit may expand a portion of the initial mask filter where the coefficients of the initial mask filter are in a symmetrical position of the zero padded portion, By using the coefficient of the initial mask filter corresponding to the portion of the initial mask filter.

In step S1442, the depth image enlargement device convolutes the determined asymmetric filter mask coefficient and the low-resolution depth image to output a pixel interpolated with a high-resolution depth image. That is, in step S1442, the depth image enlargement device convolutes the determined asymmetric filter mask coefficient and the low-resolution depth image to interpolate the low-resolution depth image to enlarge the low-resolution depth image to the resolution level of the color image Resolution depth image.

As described above, according to the depth image enlarging method according to an embodiment of the present invention, if the size of the filter mask is extended to increase the smoothing depth, the artifact of the synthesized image can be prevented by removing the unstable noise near the boundary in the depth image have.

The depth image enlargement methods according to the present invention may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like, alone or in combination. The program instructions recorded on the medium may be those specially designed and constructed for the present invention or may be available to those skilled in the art of computer software. Examples of computer-readable media include magnetic media such as hard disks, floppy disks and magnetic tape; optical media such as CD-ROMs and DVDs; magnetic media such as floppy disks; Magneto-optical media, and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of program instructions include machine language code such as those produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform operations of the present invention, and vice versa.

While the invention has been shown and described with reference to certain preferred embodiments thereof, it will be understood by those of ordinary skill in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims. This is possible.

Therefore, the scope of the present invention should not be limited to the described embodiments, but should be determined by the equivalents of the claims, as well as the claims.

BRIEF DESCRIPTION OF DRAWINGS FIG. 1 is a diagram showing a configuration of a depth image magnifying apparatus according to an embodiment of the present invention; FIG.

2 is a diagram showing an example of a specific configuration of the filter mask coefficient determination unit shown in FIG.

3 is a diagram showing an example in which an area of a search window is divided around an interpolation position.

4 is a diagram showing an example of a coefficient for the initial filter mask.

5 is a diagram showing an example in which coefficients of a filter mask for an area not including a boundary line are determined.

6 is a diagram showing an example of pixel scanning for an area including a boundary line among divided areas of a search window.

7 is a diagram showing an example in which coefficients of a filter mask for an area including a boundary line are determined.

8 is a diagram showing an example of an extended filter mask.

9 is a diagram showing an example of the coefficients of the extended filter mask.

10 and 11 are views showing an example of a case where boundary pixels immediately adjacent to the interpolation position exist.

12 is a diagram showing an example of an asymmetric filter mask.

13 is a diagram showing an example of a mask pixel for an input image.

14 is a flowchart illustrating an operation of a depth image enlargement method according to an embodiment of the present invention.

15 is a detailed flowchart of a depth image enlargement method in a case where a boundary exists in a search window.

Claims (11)

A boundary line detecting unit for detecting a boundary line of an object in a low resolution depth image; A determination unit for determining whether a boundary of the detected object is present in the search window; And An image enlargement unit for enlarging the low-resolution depth image using an asymmetric filter if it is determined that the boundary of the object exists in the search window, Lt; / RTI > Wherein the asymmetric filter has a row size and a column size different from each other. The method according to claim 1, Wherein the image enlarging unit comprises: A filter mask coefficient determiner for determining a filter mask coefficient for the asymmetric filter; And And interpolating the depth image by convoluting the filter mask according to the determined filter mask coefficient and the depth image of the low resolution, Lt; / RTI > Wherein the low-resolution depth image has a lower resolution than the color image. 3. The method of claim 2, Wherein the filter mask coefficient determination unit determines, A pixel scanning unit that scans the upper left region pixel, the upper right region pixel, the lower left region pixel, or the lower right region pixel based on the interpolation position; A zero padding unit for analyzing the scanned area pixels and performing zero padding; And A decision unit for expanding the filter mask using the zero padded result and determining a coefficient of the extended filter mask, And a depth-of-view enlargement device. The method of claim 3, The zero- And sets a coefficient of a filter mask corresponding to pixels on the boundary line and pixels on the boundary line to '0' when the boundary line of the object exists in the scanned pixel area. 5. The method of claim 4, Wherein, Wherein the filter mask is expanded using a portion in a symmetric position of a portion set to '0' according to the zero-padded result, and the coefficient of the extended filter mask portion is set to an initial filter mask corresponding to the zero- By using the coefficient of the depth image enlarging device. Detecting a boundary line of an object in a low-resolution depth image; Determining whether a boundary of the detected object is present in a search window; And If it is determined that the boundary line of the object exists in the search window, enlarging the low-resolution depth image using an asymmetric filter Lt; / RTI > Wherein the asymmetric filter is different in row size and column size from each other. The method according to claim 6, Wherein the step of enlarging the depth image comprises: Determining a filter mask coefficient that determines a coefficient of a filter mask for the asymmetric filter; And Performing interpolation on the depth image by convoluting the filter mask according to the determined filter mask coefficient and the depth image having the low resolution, Lt; / RTI > Wherein the low-resolution depth image has a lower resolution than the color image. 8. The method of claim 7, Wherein determining the filter mask coefficient comprises: Scanning an upper left region pixel, an upper right region pixel, a lower left region pixel, or a lower right region pixel based on the interpolation position; Analyzing the scanned area pixels and performing zero padding; And Expanding the filter mask using the zero padded result, and determining coefficients of the extended filter mask Wherein the depth image is enlarged. 9. The method of claim 8, Wherein the zero padding comprises: And sets a coefficient of a filter mask corresponding to pixels on the boundary line and pixels on the boundary line to '0' when the boundary line of the object exists in the scanned pixel area. 10. The method of claim 9, Wherein determining the coefficients of the extended filter mask comprises: Wherein the filter mask is expanded using a portion in a symmetric position of a portion set to '0' according to the zero-padded result, and the coefficient of the extended filter mask portion is set to an initial filter mask corresponding to the zero- By using the coefficients of the depth image. A computer-readable recording medium having recorded thereon a program for executing the method according to any one of claims 6 to 10.

Images