KR20110045650A - Apparatus and method for enlarging depth map - Google Patents

Abstract

PURPOSE: An apparatus and method for enlarging a depth map are provided to enlarge the depth image of low definition through a receiver. CONSTITUTION: A border line detection unit(110) detects the border line of an object in a depth image of low definition, and a judgment unit(120) judges the existence of a border line of the detected object inside a search window. If the border line of the object exists inside the search window, the image enlargement unit(140) enlarges the depth image of the low definition through an asymmetric filter.

Description

Depth Image Magnification Apparatus and Method {APPARATUS AND METHOD FOR ENLARGING DEPTH MAP}

The following embodiments are directed to an apparatus and method for enlarging a depth image.

When synthesizing a color image of an arbitrary point of view using a color image and a depth map, the resolution of the color image and the depth image coming into the input should be the same. However, if a low resolution depth camera is used or a low resolution depth image is received due to the limitation of the transmission bandwidth, a method of enlarging the resolution of the depth image is required.

In 3D and free view images, a color image and a depth image may be compressed and transmitted together using a compression scheme such as Joint Multi View Codec (JMVC). Since the bit generation amount is increased by transmitting the depth image together with the transmission of the conventional color image, a method of reducing the spatial resolution of the depth image is needed to prevent the increase in the bit generation amount. This method compresses and transmits the depth image by reducing the resolution of the transmitting end, and enlarges the original resolution at the receiving end, and then synthesizes the image at an intermediate time point or an arbitrary time point using the color image.

Depth image magnification apparatus according to an embodiment of the present invention includes a boundary detection unit for detecting the boundary of the object in a low-depth depth image, a determination unit for determining whether the boundary of the detected object exists in the search box and the search If it is determined that the boundary of the object exists inside the window, an image enlargement unit for enlarging the low-depth depth image using an asymmetric filter.

According to an aspect of the present invention, the image magnification unit convolves a filter mask coefficient determination unit for determining a coefficient of a filter mask for the asymmetric filter and a filter mask according to the coefficient of the determined filter mask and the depth image of the low resolution. The interpolation unit may interpolate the depth image, and the low resolution depth image may have a lower resolution than the color image.

In addition, according to an aspect of the present invention, the filter mask coefficient determining unit analyzes a pixel scanning unit scanning a top left region pixel, a top right region pixel, a bottom left region pixel, or a bottom right region pixel based on an interpolation position, and analyzes the scanned region pixels And a zero padding unit for zero padding and a filter unit for expanding a filter mask using the zero padding result and determining coefficients of the extended filter mask.

According to an aspect of the present invention, when the boundary line of the object exists in the scanned pixel area, the zero padding unit sets the coefficient of the filter mask corresponding to the pixel of the boundary line and the pixel outside the boundary line to '0'. Can be.

Further, according to an aspect of the present invention, the determining unit extends the filter mask by using a symmetrical position of a portion set to '0' according to the zero padding result, and coefficients of the extended filter mask portion May be set using coefficients of an initial filter mask corresponding to the zero-padded region.

In addition, the depth image magnification method according to an embodiment of the present invention comprises the steps of detecting the boundary of the object in the low-resolution depth image, determining whether the boundary of the detected object is present in the search box and the search window If it is determined that there is a boundary of the object therein, expanding the low-resolution depth image using an asymmetric filter.

According to an aspect of the present invention, the enlarging the depth image may include determining a filter mask coefficient for determining a coefficient of a filter mask for the asymmetric filter, and a filter mask and the low resolution according to the determined coefficient of the filter mask. And performing interpolation on the depth image by convolving the depth image, wherein the low resolution depth image may have a lower resolution than the color image.

According to an aspect of the present invention, the determining of the filter mask coefficient may include scanning a top left region pixel, a top right region pixel, a bottom left region pixel, or a bottom right region pixel based on an interpolation position, and determining the scanned region pixel. Analyzing and zero-padding, and extending the filter mask by using the zero-padded result, and determining the coefficients of the extended filter mask.

According to an aspect of the present invention, in the zero padding, when a boundary line of the object exists in the scanned pixel area, a coefficient of a filter mask corresponding to a pixel of the boundary line and an outer pixel of the boundary line is '0'. Can be set.

According to an aspect of the present invention, the determining of the coefficient of the extended filter mask may be performed by extending the filter mask by using a symmetrical position of a portion set to '0' according to the zero padding result. The coefficient of the extended filter mask part may be set using coefficients of an initial filter mask corresponding to the zero padded area.

An embodiment of the present invention may provide an apparatus and a method for enlarging a low resolution depth image to a resolution level of an original color image when a depth image is reduced in transmission.

In addition, an embodiment of the present invention may provide an apparatus and method for reconstructing a high resolution depth image by interpolating a pixel for an area including an object boundary line using an asymmetric filter in a low resolution depth image.

Hereinafter, with reference to the contents described in the accompanying drawings will be described in detail an embodiment according to the present invention. However, the present invention is not limited to or limited by the embodiments. Like reference numerals in the drawings denote like elements.

1 is a view showing the configuration of a depth image enlargement apparatus according to an embodiment of the present invention.

Referring to FIG. 1, a depth image enlargement apparatus 100 according to an exemplary embodiment of the present invention includes a boundary line detector 110, a determiner 120, a first interpolator 130, and an image enlarger 140. The depth image having a resolution lower than that of the color image is enlarged to the resolution level of the color image.

The edge detection unit 110 receives a low resolution depth image and analyzes the input low resolution depth image to detect a boundary of an object. For example, the boundary line detector 110 may detect a boundary line of an object included in the analyzed low resolution depth image by analyzing the low resolution depth image as a sobel detector or a canny detector.

The determination unit 120 sets a search window at the position of the current pixel, and determines whether a boundary line of the detected object exists inside the search window.

If it is determined that the boundary of the object does not exist inside the search window, the first interpolator 130 outputs a high resolution depth image by interpolating the low resolution depth image using a symmetric filter. For example, the first interpolator 130 outputs a high resolution depth image in which the low resolution depth image is enlarged to the resolution level of the color image through bibic cubic interpolation or Lancos coarse interpolation using the symmetric filter. can do.

If it is determined that the boundary of the detected object exists inside the search window, the image enlarger 140 interpolates the low resolution depth image using an asymmetric filter and outputs a high resolution depth image. The image enlarger 140 includes a filter mask coefficient determiner 141 and a second interpolator 142. Hereinafter, the configuration and operation of the filter mask coefficient determiner 141 will be described in more detail with reference to FIG. 2.

FIG. 2 is a diagram illustrating an example of a specific configuration of the filter mask coefficient determination unit illustrated in FIG. 1.

Referring to FIG. 2, the filter mask coefficient determiner 141 includes a pixel scanning unit 210, a zero padding unit 220, and a determination unit 230, and the detected boundary line is present in the search window. The filter mask coefficient for the asymmetric filter.

The pixel scanning unit 210 scans an upper left area pixel, an upper right area pixel, a lower left area pixel, or a lower right area pixel based on the interpolation position. That is, the pixel scanning unit 210 scans pixels in the upper left area, the upper right area, the lower left area, or the lower right area in the order of boundary search based on the interpolation position.

3 is a diagram illustrating an example in which an area of a search window is divided based on an interpolation position.

Referring to FIG. 3, the pixel scanning unit 210 sets the search window 300 as the upper left region 310, the lower left region 320, the lower right region 330, and the upper right region 340 based on the interpolation position. The pixels are equally scanned and pixels included in each of the regions 310, 320, 330, and 340 are scanned. For example, the pixel scanning unit 210 scans the pixels in the order of the boundary search from the starting points of the boundary search in the upper left area 310, the lower left area 320, and the lower right area 330, and scans the pixels. As a result, no borderline pixel exists.

4 is a diagram illustrating an example of coefficients for an initial filter mask.

Referring to FIG. 4, the initial filter mask is a 6 × 6 size symmetric filter as shown in Equation 1 and has a symmetric coefficient with respect to the center.

5 is a diagram illustrating an example in which coefficients of a filter mask are determined for a region not including a boundary line.

Referring to FIG. 5, the filter mask coefficient determination unit 141 determines that the coefficient of the filter mask for the region not including the boundary line is the same as the coefficient of the initial filter mask. For example, the filter mask coefficient determiner 141 does not include a boundary line in the upper left region 310, the lower left region 320, and the lower right region 330 illustrated in FIG. 3, and thus the coefficient of the filter mask is illustrated in FIG. 4. It can be determined to use the same as the coefficient of the initial filter mask. In this case, the upper right region 340 may be marked as not determined (ND: Not Determined) since the pixel coefficient has not been scanned yet.

6 is a diagram illustrating an example of pixel scanning of a region including a boundary line among divided regions of a search window.

Referring to FIG. 6, the pixel scanning unit 210 scans pixels in an order for boundary line search based on a starting point of boundary line search in the upper right region 600. That is, the pixel scanning unit 210 performs pixel scanning in the upper right region 600 of the edge map of the depth image corresponding to the initial filter mask to check whether the depth image is a boundary line, and checks the boundary pixel and the pixels outside the boundary line. Search.

The zero padding unit 220 zero-pads the analyzed pixel. That is, when the boundary line of the object exists in the scanned pixel, the zero padding unit 220 sets the coefficient of the filter mask corresponding to the boundary pixel and the pixel outside the boundary line to '0'. The pixel outside the boundary line may be a pixel existing after the boundary line in the order for searching the boundary line.

7 is a diagram illustrating an example in which coefficients of a filter mask for a region including a boundary line are determined.

Referring to FIG. 7, when the zero padding unit 220 is a boundary pixel included in the upper right region 600 as illustrated in FIG. 6 and a pixel outside the boundary line, unlike the coefficients of the initial filter mask illustrated in FIG. Change to 0 '.

The determination unit 230 expands the filter mask by using the zero-padded result and determines coefficients of the extended filter mask.

8 is a diagram illustrating an example of an extended filter mask.

Referring to FIG. 8, the determiner 230 extends the filter mask by using a portion 820 that is symmetrical to the zero-padded portion 810, and zeroes the coefficients of the expanded filter mask to the zero-padded portion ( The determination is made using the coefficient of the initial filter mask corresponding to 810.

The determiner 230 may remove the unstable noise near the boundary in the depth image by expanding the size of the filter mask with respect to the depth image to increase the smoothing.

9 is a diagram illustrating an example of coefficients of an extended filter mask.

Referring to FIG. 9, the extended filter mask 920 is 8 × 6 that is larger in size than 6 × 6, which is the size of the initial filter mask 910 before expansion. The coefficient of the filter mask for the extended portion may be determined using the coefficient corresponding to the position of the zero padded portion shown in FIG. 7.

The second interpolator 142 convolves the asymmetric filter mask according to the determined coefficient of the filter mask and the input low resolution depth image. For example, the second interpolator 142 multiplies the determined filter mask coefficients by a normalization constant to perform convolution on the normalized asymmetric filter mask and the low resolution depth image to interpolate the low resolution depth image. A high resolution depth image enlarged to a resolution level of a color image may be output.

10 and 11 are diagrams showing an example of a case where boundary pixels immediately adjacent to interpolation positions exist.

10 and 11, when two boundary pixels are symmetrically among four pixels immediately adjacent to the interpolation position, the second interpolator 142 may interpolate the pixels with an average of two input pixels. .

12 is a diagram illustrating an example of an asymmetric filter mask, and FIG. 13 is a diagram illustrating an example of mask pixels for an input image.

12 and 13, the second interpolator 142 convolves an asymmetric filter mask according to the coefficient of the filter mask illustrated in FIG. 12 and a mask pixel for the low resolution depth image illustrated in FIG. By interpolating the depth image, a high resolution depth image may be output.

As described above, the depth image enlargement apparatus 100 according to an exemplary embodiment of the present invention does not generate noise boost or artifact by interpolating pixels of a region including the boundary of an object in a low resolution depth image using an asymmetric filter. You can do that.

14 is a flowchart illustrating an operation of a method for enlarging a depth image according to an exemplary embodiment of the present invention.

Referring to FIG. 14, in operation S1410, the depth image magnifying apparatus detects an object boundary line of a low resolution depth image. That is, in operation S1410, when the depth image of the low resolution image is input, the depth image magnification apparatus detects a boundary of an object with respect to the input low resolution depth image by using an edge detector. For example, in operation S1410, the depth image magnifying apparatus may detect an object boundary in a low resolution depth image using a Sobel detector or a Canny detector.

In operation S1420, the depth image enlargement apparatus sets a search window at a position of a current pixel, and determines whether a boundary of the detected object exists inside the set search window.

If the boundary line does not exist, in step S1430, the depth image enlarging apparatus interpolates the depth image of the low resolution using a symmetric filter. For example, if it is determined in step S1430 that the depth image magnification apparatus is not near the boundary if no boundary line exists in the search box, the depth of the low resolution through bicubic interpolation or lancos interpolation using the symmetric filter is used. By interpolating the image, the image can be enlarged to a high resolution depth image.

If the boundary exists, the depth image enlarging apparatus determines a filter mask coefficient (S1441) and performs interpolation on the low resolution depth image using an asymmetric filter according to the determined coefficient of the filter mask. (S1442). That is, in operation S1440, when the boundary line is present in the search window, the depth image magnification apparatus determines that a current pixel position is a region near the boundary, and determines a mask coefficient of an asymmetric filter (S1441). By performing interpolation on the asymmetric filter mask and the low resolution depth image, a high resolution depth image is obtained (S1442).

FIG. 15 is a detailed flowchart of a method of enlarging a depth image when a boundary line exists in a search window.

Referring to FIG. 15, when it is determined in step S1511 that the depth image enlargement apparatus has a boundary inside the search box, the depth image enlargement apparatus starts from the starting point in the upper left area of the region divided into four quarters based on the interpolation position within the search window. The pixels are scanned in order to search the boundary.

In operation S1512, when it is determined that a boundary line exists in the search window, the depth image magnification apparatus searches for a boundary line from a starting point in a lower left region of the region divided into four quarters based on the interpolation position in the search window. Scan the pixels as they are.

In operation S1513, when it is determined that a boundary line exists in the search window, the depth image magnification apparatus searches for a boundary line from a starting point in a lower right region of the region divided into quarters based on the interpolation position within the search window. Scan the pixels as they are.

In operation S1514, when it is determined that a boundary line exists in the search window, the depth image magnification apparatus searches for a boundary line from a starting point in a top right region of a region divided into quarters based on the interpolation position in the search window. Scan the pixels as they are.

In operation S1520, the depth image zooming apparatus zero-pads the scanned pixel with respect to a boundary pixel or a pixel present after the boundary pixel. That is, in step S1520, the depth image enlargement apparatus sets the filter mask coefficient to '0' for the pixel in which the scanned pixel exists or after the boundary pixel. For example, in operation S1520, the depth image enlargement apparatus does not change the initial mask filter coefficient because there is no boundary between the upper left region 310, the lower left region 320, and the lower right region 330 as shown in FIG. 3. As illustrated in FIG. 6, the coefficient of the filter mask corresponding to the pixel corresponding to the edge map boundary line or the pixel outside the boundary line may be changed to '0' in the upper right region 600.

In operation S1530, the depth image enlarging device extends the size of the filter mask with respect to the low resolution depth image, and sets coefficients of the extended filter mask. That is, in operation S1530, the depth image enlarging device expands the size of the filter mask using the zero-padded portion, and uses the coefficient of the expanded asymmetric filter mask as the coefficient of the initial filter mask with respect to the zero-padded portion. Use to set. For example, in operation S1530, the depth image enlarging device may extend the size of the filter mask to an 8 × 6 asymmetric filter mask larger than 6 × 6, which is the size of the initial mask filter, for the low resolution depth image. For example, in operation S1530, the depth image enlarging device expands a portion of the coefficient of the initial mask filter at a symmetrical position of the zero-padded portion, and zero-pads the coefficient of the expanded portion of the extended filter mask. It can be set using the coefficients of the initial mask filter corresponding to the specified part.

In operation S1442, the depth image enlarging apparatus convolves the determined asymmetric filter mask coefficient and the low resolution depth image to output interpolated pixels as a high resolution depth image. That is, in operation S1442, the depth image enlarging apparatus convolves the determined asymmetric filter mask coefficient and the low resolution depth image to interpolate the low resolution depth image to enlarge the low resolution depth image to the resolution level of the color image. And output the high resolution image.

As described above, in the depth image enlargement method according to an embodiment of the present invention, when the size of the filter mask is increased to smooth the depth image, the artifact of the synthesized image may be prevented by removing the unstable noise near the boundary in the depth image. have.

Depth image magnification methods according to the present invention can be implemented in the form of program instructions that can be executed by various computer means may be recorded on a computer readable medium. The computer readable medium may include program instructions, data files, data structures, etc. alone or in combination. Program instructions recorded on the media may be those specially designed and constructed for the purposes of the present invention, or they may be of the kind well-known and available to those having skill in the computer software arts. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tape, optical media such as CD-ROMs, DVDs, and magnetic disks, such as floppy disks. Magneto-optical media, and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine code generated by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like. The hardware device described above may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

As described above, the present invention has been described by way of limited embodiments and drawings, but the present invention is not limited to the above embodiments, and those skilled in the art to which the present invention pertains various modifications and variations from such descriptions. This is possible.

Therefore, the scope of the present invention should not be limited to the described embodiments, but should be determined not only by the claims below but also by the equivalents of the claims.

1 is a view showing the configuration of a depth image enlargement apparatus according to an embodiment of the present invention.

FIG. 2 is a diagram illustrating an example of a specific configuration of the filter mask coefficient determination unit illustrated in FIG. 1.

3 is a diagram illustrating an example in which an area of a search window is divided based on an interpolation position.

4 is a diagram illustrating an example of coefficients for an initial filter mask.

5 is a diagram illustrating an example in which coefficients of a filter mask are determined for a region not including a boundary line.

6 is a diagram illustrating an example of pixel scanning of a region including a boundary line among divided regions of a search window.

7 is a diagram illustrating an example in which coefficients of a filter mask for a region including a boundary line are determined.

8 is a diagram illustrating an example of an extended filter mask.

9 is a diagram illustrating an example of coefficients of an extended filter mask.

10 and 11 are diagrams showing an example in the case where boundary pixels immediately adjacent to interpolation positions exist.

12 is a diagram illustrating an example of an asymmetric filter mask.

13 is a diagram illustrating an example of mask pixels for an input image.

14 is a flowchart illustrating an operation of a method for enlarging a depth image according to an exemplary embodiment of the present invention.

15 is a detailed flowchart illustrating a method of enlarging a depth image when a boundary exists in a search window.

Claims (11)

A boundary line detector for detecting a boundary line of an object in a low resolution depth image; A determination unit determining whether a boundary of the detected object exists in a search window; And If it is determined that the boundary of the object exists inside the search window, an image enlargement unit for enlarging the low-depth depth image using an asymmetric filter Depth image magnifying apparatus comprising a. The method of claim 1, The image enlargement unit, A filter mask coefficient determiner for determining a coefficient of a filter mask for the asymmetric filter; And An interpolator interpolating the depth image by convolving the filter mask according to the coefficient of the determined filter mask and the low resolution depth image Including, The depth image enlargement apparatus of the low resolution has a lower resolution than the color image. The method of claim 2, The filter mask coefficient determination unit, A pixel scanning unit scanning a top left region pixel, a top right region pixel, a bottom left region pixel, or a bottom right region pixel based on the interpolation position; A zero padding unit configured to analyze and zero pad the scanned area pixels; And A decision unit to expand a filter mask using the zero-padded result and determine coefficients of the extended filter mask Including, a depth image magnification device. The method of claim 3, The zero padding unit, And if a boundary line of the object exists in the scanned pixel area, setting a coefficient of a filter mask corresponding to a pixel of the boundary line and a pixel outside the boundary line to '0'. The method of claim 4, wherein The determination unit, The filter mask is extended using a portion located at a symmetrical position of a portion set to '0' according to the zero padding result, and an initial filter mask corresponding to coefficients of the extended filter mask portion corresponding to the zero-padded region. A depth image magnification apparatus, which is set using a coefficient of. Detecting boundary lines of an object in a low resolution depth image; Determining whether a boundary of the detected object exists within a search window; And If it is determined that a boundary of the object exists inside the search window, enlarging the low-depth depth image using an asymmetric filter Depth image magnification method comprising a. The method of claim 6, Magnifying the depth image, Determining filter mask coefficients that determine coefficients of a filter mask for the asymmetric filter; And Performing interpolation on the depth image by convolving the filter mask according to the coefficient of the determined filter mask and the depth image of the low resolution; Including, The depth image of the low resolution has a lower resolution than the color image, depth image expansion method. The method of claim 7, wherein Determining the filter mask coefficients, Scanning a top left region pixel, a top right region pixel, a bottom left region pixel, or a bottom right region pixel based on the interpolation position; Analyzing and zero padding the scanned area pixels; And Expanding a filter mask using the zero padded result and determining coefficients of the extended filter mask Including, depth image magnification method. The method of claim 8, The zero padding step, And if a boundary line of the object exists in the scanned pixel area, setting a coefficient of a filter mask corresponding to a pixel of the boundary line and a pixel outside the boundary line to '0'. 10. The method of claim 9, Determining the coefficients of the extended filter mask, The filter mask is extended using a portion located at a symmetrical position of a portion set to '0' according to the zero padding result, and an initial filter mask corresponding to coefficients of the extended filter mask portion corresponding to the zero-padded region. The depth image magnification method is set using the coefficient of. A computer-readable recording medium in which a program for executing the method of any one of claims 6 to 10 is recorded.

Images