KR100914171B1 - Apparatus and method for depth based image rendering on mobile broadcasting - Google Patents

Abstract

An apparatus and a method for rendering a depth based image in mobile broadcasting are provided to apply a discontinuous maintenance filter to a depth image, thereby minimizing distortion of the depth image. A discontinuous maintenance filter(111) smoothes a depth image correspondingly to gradient amplitude of the depth image. An adaptive smoothing filter(115) calculates a gradient degree of the depth image transmitted from the discontinuous maintenance filter. Only when gradient amplitude is larger than a preset value, the adaptive smoothing filter applies a filter feature according to a gradient direction to the depth image. The adaptive smoothing filter transmits the depth image to an encoding module(120). The encoding module transmits a reference image and the depth image.

Description

Apparatus and method for depth based image rendering on Mobile broadcasting}

The present invention relates to a depth image based rendering in a mobile terminal for a 3D service in a mobile broadcast. More particularly, in a depth image based rendering, a depth image using a discontinuity preservation filter and a gradient direction based adaptive smoothing filter is provided. A filtering method and apparatus for reducing distortion of a.

The present invention is derived from the research conducted as part of the IT growth engine technology development project of the Ministry of Information and Communication and the Ministry of Information and Telecommunications Research and Development. [Task Management Number: 2007-S-004-01] Development].

3D stereoscopic image is an image medium that can convey the reality of the three-dimensional world, which is a real world, and has advantages such as presence feeling, presence, naturalness, and clarity.

The 3D stereoscopic image can be viewed by using a plurality of display means to view different two-dimensional images on both eyes of the left and right eyes, or by cross-displaying different two-dimensional images at predetermined time intervals on one stereoscopic image display means. The multi-view image having the above viewpoints is displayed to the user so that the user's brain accurately synthesizes the two lights reflected from the object to reproduce the perspective and reality of the 3D image.

The method for generating a 3D stereoscopic image includes a stereo matching method for receiving two binocular images and generating a three-dimensional image, a depth value of one image and each pixel forming the image, There is a depth image based rendering technique that inputs depth values to generate different two-dimensional images.

Depth image-based rendering technique uses less information compared to stereo matching, which greatly increases the transmission efficiency and enables the user to use 3D service according to the user's intention. It is widely used in terminals. However, since portable broadcasting has a very low effective bit rate for data services, it is necessary to improve transmission efficiency of depth images.

In general, smoothing methods have been proposed as preprocessing methods for depth images, and Gaussian smoothing filters have been most widely used. However, the Gaussian smoothing filter not only causes distortion in the binocular image generated due to the distortion of the depth image, but also has a problem that the three-dimensional effect is deteriorated because it does not form a sense of conformity with the boundary of the object.

In order to solve the problem of the Gaussian smoothing filter, an asymmetric filter has been introduced, but it is also difficult to properly use the depth information due to the distortion of the depth image, and there is a problem that the three-dimensional effect is lowered like the Gaussian smoothing filter.

The present invention has been made to improve the prior art as described above, by applying a discontinuity preservation filter according to the size of the gradient to the depth image, to minimize the distortion of the depth image, while smoothing the parts that do not significantly affect the formation of three-dimensional effect Provided are a depth image based rendering method and apparatus for reducing a bit rate required by a. The present invention provides a depth image-based rendering method and apparatus for reducing the number of holes generated during rendering by applying a discontinuity preservation filter by a gradient size to a depth image.

The present invention provides a depth image-based rendering method and apparatus for reducing the number of occurrences of a hole by simultaneously warping the right view image from the left and the left view image from the right.

Depth image-based rendering transmitter according to an embodiment of the present invention includes a discontinuity preservation filter by the gradient size for smoothing and transmitting the depth image corresponding to the gradient size of the depth image; It includes a transmission unit for transmitting a reference image and the depth image.

Depth image-based rendering receiver according to an embodiment of the present invention includes a receiving unit for receiving the reference image and the depth image; And simultaneously generating a right eye view image and a left eye view image having binocular disparity using the reference image and the depth image, and synthesizing a binocular stereoscopic image from the right view image and the left view image. .

Depth image-based rendering transmission method according to an embodiment of the present invention comprises the steps of: smoothing the depth image corresponding to the size of the gradient of the depth image; Calculating an inclination of the smoothed depth image and adaptive smoothing by applying a characteristic of a filter according to an inclination direction to the smoothed depth image only when the inclination magnitude is higher than a predefined value; And encoding and transmitting a reference image and the adaptive smoothed depth image.

Depth image-based rendering receiving method according to an embodiment of the present invention comprises the steps of receiving an encoded reference image and the depth image; Decoding and reconstructing the depth image and the reference image; And simultaneously generating left and right view images using the reconstructed depth image and the reconstructed reference image, and synthesizing the left and right view images.

According to the present invention, by applying the discontinuity preservation filter according to the magnitude of the gradient to the depth image, it is possible to reduce the bit rate required for transmission by smoothing the portions that do not significantly affect the formation of the stereoscopic effect while minimizing distortion of the depth image.

According to the present invention, the number of holes generated during rendering may be reduced by applying a discontinuity preservation filter based on the magnitude of the gradient to the depth image.

According to the present invention, the right view image may be warped simultaneously from the left and the left view image may be simultaneously warped from the right to reduce the number of holes.

Hereinafter, with reference to the accompanying drawings will be described an embodiment of the present invention; If it is determined that the gist of the present invention may be unnecessarily obscured, detailed description thereof will be omitted.

1 is a block diagram of a depth image based rendering apparatus according to an exemplary embodiment of the present invention.

As shown in FIG. 1, the depth image based rendering apparatus according to an embodiment of the present invention performs a preprocessing of a depth image to reduce transmission capacity and distortion of the depth image, and a reference image. A transmitter 100 including a reference stream and an encoding module 120 encoding and transmitting a depth image preprocessed by the preprocessing module 110, and receiving encoded information from the encoding module 120. The simultaneous processing module 220 for generating a stereoscopic image by simultaneously performing four steps of depth image based rendering using the decoding module 210 for decoding the depth image and the reference image and the depth image information decoded by the decoding module 210 is performed. It comprises a receiver 200 that includes.

The preprocessing module 110 adapts the smoothing based on the discontinuity preservation filter 111 and the gradient direction by the gradient size, which preserves and discontinuities in the depth image, which is the main factor that makes the user feel the three-dimensional effect. The filter 115 is applied, and the adaptive smoothing filter 115 based on the gradient direction is sequentially applied to the depth image passing through the discontinuity preservation filter 111.

The discontinuity preservation filter 111 and the adaptive smoothing filter 115 based on the gradient direction commonly use the following Equations 1, 2, and 3.

Here, N (x, y) is a normalization factor and D (x, y) is a depth image value.

As shown in Equation 1, the preprocessing module 110 repeatedly applies a 3 * 3 linear filter determined by w (x, y) reflecting the characteristics of the filter t times. The smoothed value in the step is prevented from becoming smaller than the input image. As the smoothed value becomes smaller than the input image, the dynamic range of the depth image decreases as the number of iterations increases, and the main cause of the deterioration of the stereoscopic effect is to smooth the inside and the outside of the object at the discontinuity boundary formed at the boundary of the object. do.

The adaptive smoothing filter has the effect of spreading the filter characteristics as the number of repetitions increases, so that smoothing action is spread in the direction where w (x, y) is large and smoothing action is suppressed in the direction where w (x, y) is small. And the characteristics of the image are preserved.

The discontinuity preservation filter 111 based on the magnitude of the gradient measures the spatial gradient by applying Equation 4 described below as a step for determining w (x, y).

와

는 하기된 수학식 5를 적용하여 계산되며, 하기된 수학식 5 이외에도 소벨(Sobel) 연산자 등을 이용하여 계산할 수 있다.In this case, the derivative value of each part

Wow

Is calculated by applying Equation 5 below, and can be calculated using the Sobel operator in addition to Equation 5 below.

At this time. I (x, y) is a reference image value.

The discontinuity preservation filter 111 based on the magnitude of the gradient size calculates the magnitude of the gradient vector by applying the following Equation 6 to the spatial gradient calculated by Equation 4 in two steps for determining w (x, y).

The discontinuity preservation filter 111 based on the magnitude of the gradient has three steps for determining w (x, y), and applies w (x, y) to the magnitude of the gradient vector calculated by Equation 6 below. Calculate

In Equation 7, S defines the discontinuity of the depth to be preserved.

The transfer function g for determining w (x, y) in Equation 7 is calculated corresponding to Equation 8 described below.

In this case, K is a reference value for activating the discontinuity of the depth image having a higher value than itself.

The discontinuity preservation filter 111 based on the magnitude of the gradient calculates w (x, y) by applying the transfer function of Equation 8 to Equation 7, and calculates the calculated w (x, y) by Equation 1,2,3. By applying the filter to discontinuity preservation filter, it is possible to reduce the bit rate required for transmission by smoothing the parts that do not greatly affect the stereoscopic shape while minimizing the distortion of the depth image.

The preprocessing module 110 reduces the bit rate required for transmission by smoothing the portions where the depth image to which the discontinuity preservation filter 111 is applied due to the gradient size does not significantly affect the formation of the stereoscopic effect while minimizing the distortion of the input depth image. The depth discontinuity, which plays an important role in forming the three-dimensional effect, is well preserved, and the number of holes generated during rendering is similar to the original depth image, so that the adaptive smoothing filter according to the gradient direction is applied to the image to which the discontinuity preservation filter 111 by the gradient size is applied. 115) applies.

The adaptive smoothing filter 115 according to the gradient direction calculates the gradient direction by applying Equation 9 below as a step for determining w (x, y).

In the adaptive smoothing filter 115 according to the gradient direction, w (x, y) is determined by applying the gradient direction calculated in step 1 to Equation 10 in two steps for determining w (x, y).

As shown in Equation 11, the adaptive smoothing filter 115 according to the gradient direction preserves discontinuity due to the magnitude of the gradient by calculating w (x, y) according to the gradient direction only when the magnitude of the gradient is higher than a predefined value. In the image to which the filter 111 is applied, the hole area is reduced by applying an inclination direction only in an area where a hole is generated during rendering.

Since the adaptive smoothing filter 115 according to the gradient direction occurs only at the depth discontinuity in the transverse direction in consideration of the viewpoint change in the transverse direction, the weight of the filter is closer to the transverse direction as shown in Equation 10. To increase.

The preprocessing module 110 applies the adaptive smoothing filter 115 according to the gradient direction to the image to which the discontinuity preservation filter 111 according to the gradient size is applied, and thus the edge compared to the case of applying the Gaussian smoothing filter as shown in FIG. 2. It can reduce the left and right distortion of. In addition, the smoothing by the maximum operation prevents the smoothing of the inside of the object with depth, and the size of the object is not reduced in the depth image after smoothing. To help.

As shown in FIG. 3, the simultaneous processing module 220 uses the right image horizontal parallax calculation unit 221 to generate a right view image having binocular disparity using the reference image and the depth image, and the reference image and the depth image. Left image horizontal parallax calculation unit 222 for generating a left view image having binocular parallax, a non-occluded area inspection module 224 that checks whether a hole has occurred when generating left and right view images, and fills a hole when a non-occluded area occurs The non-occluded region interpolation module 226 for correcting and the stereoscopic image generation module 228 for synthesizing a binocular stereoscopic image having a size twice as large as that of the reference image for outputting the left and right viewpoint images on a 3D display Include.

을 계산한다. The right image horizontal parallax calculation unit 221 applies the following equation (11) to the received reference image and the depth image to the right view horizontal coordinate value of the binocular stereoscopic image corresponding to the coordinates (x, y) of the reference image.

Calculate

을 계산한다. The left image horizontal parallax calculation unit 222 applies the following equation (12) to the received reference image and the depth image to the left view horizontal coordinate value of the binocular stereoscopic image corresponding to the coordinates (x, y) of the reference image.

Calculate

는 좌, 우 시점 사이의 거리를 나타낸다. Where X is the horizontal size of the reference image, f is the focal length, and D (x, y) is the depth value at that coordinate,

Represents the distance between left and right viewpoints.

The right image horizontal disparity calculator 221 generates a right view image by starting the warping from the left side of the reference image as shown in FIG. 4 according to an occlusion compatible warp order. The horizontal parallax calculator 222 generates a left view image by starting warping from the right side of the reference image as shown in FIG. 4 according to an occlusion compatible warp order.

When each pixel is warped, the distance that each pixel is moved varies depending on the size of the depth value. Thus, a hole, which is a non-occluded region, in which two or more pixels are concentrated in one pixel of the viewpoint image is generated. In this case, it is necessary to determine which pixel has a large depth value among the two pixels and allocate a pixel having a large depth value. In the simultaneous processing module 220, the right view image is warped from the left and the left view image is warped from the right. Since the pixels are drawn later, the pixels with small depth values are automatically erased, reducing the number of holes generated.

The non-occluded area inspection module 224 determines whether a hole, which is a non-occluded area, is generated according to the distance between the coordinates of the previous step and the currently generated coordinates in the process of simultaneously generating the left and right viewpoint images for the given coordinates of the reference image. Then, the left view image and the right view image where no hole is generated are transmitted to the stereoscopic image generating module 228.

The non-occluded area interpolation module 226 interpolates using a linear interpolation method when a hole is generated, and transmits the interpolated area to the stereoscopic image generation module 228.

 The stereoscopic image generating module 228 receives the left view image and the right view image so as to be divided into odd columns and even columns so that the odd columns can be viewed by the right eye and the even columns by the left eye. Compose a stereoscopic stereoscopic image.

The simultaneous processing module 220 includes a right image horizontal parallax calculator 221, a left image horizontal parallax calculator 222, a non-occluded area inspection module 224, a non-occluded area interpolation module 226, and a stereoscopic image generation module 228. ) Can be performed simultaneously using only memory for binocular stereoscopic image storage, thereby suppressing unnecessary memory usage and requiring only one reference image search.

The depth image based rendering apparatus according to the present invention may be applied in other forms according to system characteristics.

Depth image-based rendering apparatus according to an embodiment of the present invention, as shown in Figure 1, the transmitter 100 includes a discontinuity preservation filter 111 by the gradient size and the gradient direction-based adaptive smoothing filter 115 Since the receiver 200 does not perform the preprocessing of the depth image, the receiver 200 is a system suitable for a small terminal having limited computing power.

The depth image-based rendering apparatus according to another embodiment of the present invention includes a discontinuity preservation filter module 130 in which the transmitter 100 applies the discontinuity preservation filter by the gradient size to the depth information, as shown in FIG. 6. The receiver 200 may include an adaptive smoothing filter module 230 that applies an gradient smoothing-based adaptive smoothing filter to the depth information reconstructed by the decoding module 210 to perform different degrees of smoothing according to a parameter given by a user. Therefore, the receiver 200 is a suitable system when the terminal has a high computing power.

7 is a flowchart of a depth image based rendering method according to an embodiment of the present invention.

In operation S100, the transmitter 100 receives a reference image and a depth image from the outside.

In operation S200, the transmitter 200 applies discontinuity preservation filters according to the magnitude of the gradient to the depth image received in operation S100 to smooth the portions that do not significantly affect the formation of the stereoscopic effect while minimizing distortion of the depth image. Reduce the bit rate required for transmission.

In step S300, the transmitter 200 applies the adaptive smoothing filter 115 according to the gradient direction to the depth image to which the discontinuity preservation filter 111 based on the magnitude of the gradient is applied in step S200. Decrease.

In step S400, the encoding module 120 is suitable for transmitting the depth image to which the adaptive smoothing filter 115 according to the gradient direction is applied in step S300 and the reference image received in step S100 through a wired or wireless network. Encode in format.

In operation S500, the transmitter 100 transmits the depth image and the reference image encoded in operation S400 to the receiver 200.

In operation S600, the decoding module 210 decodes and restores the depth image and the reference image received from the transmitter 100.

In step S700, the simultaneous processing module 220 simultaneously generates left and right view images having binocular disparity using the depth image and the reference image reconstructed in step S600, and a hole is generated in the left and right view images. Then, a binocular stereoscopic image is synthesized by correcting the following, and detailed steps will be described later with reference to FIG. 8.

In step S800, the receiver 200 provides the binocular stereoscopic image synthesized in step S700 to the outside and displays it.

8 is a detailed flowchart of the image simultaneous processing step of FIG.

In operation S710, the simultaneous processing module 220 receives the depth image and the reference image reconstructed in operation S600.

In operation S720, the right image horizontal disparity calculator 221 starts warping from the left side of the reference image to generate a right view image, and the left image horizontal disparity calculator 222 starts warping from the right side of the reference image. A left view image is generated.

In operation S730, the non-occluded area inspection module 224 generates a hole that is a non-occluded area according to the distance between the coordinates of the previous step and the currently generated coordinates in the process of simultaneously generating the left and right viewpoint images with respect to the given coordinates of the reference image. It is determined whether this occurred.

In step S740, the non-occluded area interpolation module 226 interpolates the generated holes using linear interpolation.

In operation S750, the stereoscopic image generating module 228 configures the left-view image and the right-view image to be divided into odd columns and even columns to synthesize a binocular stereoscopic image.

In the present invention as described above has been described by the specific embodiments, such as specific components and limited embodiments and drawings, but this is provided to help a more general understanding of the present invention, the present invention is not limited to the above embodiments. For those skilled in the art, various modifications and variations are possible from these descriptions.

Therefore, the spirit of the present invention should not be limited to the described embodiments, and all the things that are equivalent to or equivalent to the claims as well as the following claims will belong to the scope of the present invention. .

1 is a block diagram of a depth image based rendering apparatus according to an exemplary embodiment of the present invention.

2 is a state diagram showing an image without preprocessing, an image to which Gaussian smoothing is applied, and an image to which preprocessing according to the present invention is applied.

3 is a detailed block diagram of the simultaneous processing module of FIG.

4 is a state diagram showing a warping sequence according to the present invention.

5 is a block diagram of a binocular stereoscopic image synthesized according to the present invention.

6 is a configuration diagram of a depth image based rendering apparatus according to another exemplary embodiment of the present invention.

7 is a flowchart of a depth image based rendering method according to an embodiment of the present invention.

8 is a detailed flowchart of the image simultaneous processing step of FIG.

Claims (16)

A discontinuity preservation filter based on a gradient size for smoothing and transmitting the depth image corresponding to the magnitude of the gradient of the depth image; A transmitter for transmitting a reference image and the depth image; And Calculating the inclination of the depth image transmitted by the discontinuity preservation filter by the magnitude of the inclination, and applying the characteristics of the filter according to the inclination direction to the depth image only when the inclination magnitude is higher than a predefined value to transmit to the transmission unit Adaptive Smoothing Filter by Slope Direction Depth image-based rendering transmitter comprising a. delete The method of claim 1, The discontinuity preservation filter by the gradient size measures the spatial gradient, calculates the magnitude of the gradient vector corresponding to the spatial gradient, calculates the characteristics of the filter corresponding to the gradient vector, and uses the characteristics of the filter to determine the depth. A depth image based rendering transmitter, characterized in that the image is smoothed. The method of claim 3, The discontinuity preservation filter by the gradient size is a depth image-based rendering transmitter, characterized in that for measuring the characteristics of the filter corresponding to the gradient vector by applying the discontinuity of the gradient vector size and the depth to be preserved. The method of claim 4, wherein The transfer function for determining the characteristics of the filter is inversely proportional to a real squared value less than 1 divided by a gradient vector size divided by a reference value for activating discontinuity of a depth image having a higher value than that of the filter. Depth image based rendering transmitter. A receiver which receives the reference image and the depth image; A simultaneous processing module which simultaneously generates a right eye view image and a left eye view image having binocular disparity using the reference image and the depth image, and synthesizes a binocular stereoscopic image from the right view image and the left view image; And The adaptive smoothing filter module calculates the inclination of the depth image received by the receiver and applies the characteristic of the filter according to the inclination direction to the depth image only when the inclination magnitude is higher than a predefined value and transmits the characteristic to the depth processing module. Depth image-based rendering receiver comprising a. The method of claim 6, The simultaneous processing module generates the right view image by starting warping from the left side of the reference image, and simultaneously generates the left view image by starting warping from the right side of the reference image. receiving set. The method of claim 6, And the simultaneous processing module interpolates using a linear interpolation method when a hole, which is a non-occluded region, is generated when the right view image and the left view image are generated. The method of claim 6, In the simultaneous processing module, a multiple of a value obtained by dividing a distance between a focal length and a left and right viewpoint by a multiple of a depth value at a corresponding coordinate from an x coordinate value of a reference image is multiplied by a coordinate (x, and calculating the right view horizontal coordinate value of the binocular stereoscopic image corresponding to y) to generate the right view image. The method of claim 6, The simultaneous processing module multiplies the distance between the focal length and the distance between the left and right viewpoints by a multiple of the depth value at the coordinate having the x coordinate reduced from the horizontal size of the reference image by the x coordinate value of the reference image to the x coordinate value of the reference image. Value and a multiple of the sum of the horizontal size of the reference image plus the x coordinate value of the reference image is calculated so that the left view horizontal coordinate value of the binocular stereoscopic image corresponding to the coordinate (x, y) of the reference image is calculated. And a depth image based rendering receiver to generate the left view image. delete Smoothing the depth image corresponding to the magnitude of the gradient of the depth image; Calculating an inclination of the smoothed depth image and adaptive smoothing by applying a characteristic of a filter according to an inclination direction to the smoothed depth image only when the inclination magnitude is higher than a predefined value; And encoding and transmitting a reference image and the adaptive smoothed depth image. The method of claim 12, The smoothing of the depth image may include: Measuring a spatial gradient; Calculating a magnitude of an inclination vector corresponding to the spatial inclination; Calculating a characteristic of the filter corresponding to the gradient vector; And smoothing the depth image using the characteristics of the filter. Receiving an encoded reference image and a depth image; Calculating an inclination of the depth image and applying a characteristic of a filter according to an inclination direction to the depth image only when the inclination magnitude is higher than a predefined value; Decoding and reconstructing the depth image and the reference image; And simultaneously generating left and right view images using the reconstructed depth image and the reconstructed reference image, and synthesizing the left and right view images to generate a binocular stereoscopic image. The method of claim 14, Simultaneously generating the left and right view images using the reconstructed depth image and the reconstructed reference image starts warping from the left side of the reference image to generate the right view image and warps from the right side of the reference image. Depth image-based rendering receiving method characterized in that for generating the left view image. The method of claim 14, Simultaneously generating the left and right view images using the reconstructed depth image and the reconstructed reference image is characterized by interpolating using a linear interpolation method when holes are generated while simultaneously generating the left and right view images. Depth image based rendering method.

Images