KR20150078359A - Method and device of generating multi-view image with resolution scaling function - Google Patents

Abstract

The present invention relates to a method and a device for generating a multi-view image capable of minimizing a warping artifact increase due to the difference between input and output resolutions when scaling horizontal resolution after warping. The method for generating a multi-view image of the present invention comprises: a first step of vertically scaling an input image and a depth image; a second step of calculating disparity information and position information corresponding to interpolation arrangement information from the vertically scaled depth image by time; a third step of warping the position information and the vertically scaled image respectively using the disparity information by time; a fourth step of filling a hole of the image and position information which are warped by time; and a fifth step of outputting a multi-view image by horizontally scaling the filled image by time using the filled position information.

Description

METHOD AND APPARATUS FOR GENERATING MULTI-VIEW IMAGE WITH RESOLUTION SCALING FUNCTION WITH A RESOLUTION SCALING FUNCTION

The present invention relates to a multi-view image generation method and apparatus capable of minimizing an increase in warping artifacts according to input / output resolution differences when scaling horizontal resolution after warping.

The stereoscopic image display apparatus realizes stereoscopic 3D images by separating the left eye image and the right eye image photographed at different angles from each other spatially or temporally on the left and right eyes of viewers, Respectively.

The spectacle method implements the 3D image by displaying the polarizing direction of the left eye image and the right eye image on the display device in a manner that is different from that of the left eye image or the right eye image or by separating the images in a time division manner and separating the left and right images using polarizing glasses or liquid crystal shutter glasses. It is inconvenient to wear special glasses.

In the non-eyeglass system, a 3D image is provided to a viewer by spatially separating left and right images through a 3D optical filter such as a Parallax Barrier or a Lenticular Lens installed in front of or behind a display device, There is an advantage not required. However, in the non-eyeglass system, a stereoscopic viewing area in which a viewer views a normal 3D image by inputting a left eye image and a right eye image in the left and right eyes of a viewer, a right eye image is input in the left eye, There is a disadvantage in that a viewing area in which a 3D image can be viewed is limited because there is a reverse stereoscopic area in which a viewer views an abnormal 3D image. To solve this problem, the spectacles 3D display device displays a multi-view image, thereby widening the viewing area for stereoscopic viewing.

In the multi-view method, stereoscopic image generation is performed by generating an image at a plurality of viewpoints from an input image, and then formatting and outputting the image so that the corresponding view is visible at each viewpoint.

Recently, a Depth-Image-Based Rendering (DIBR) algorithm is known in which a 2D image and a depth information image extracted from a stereo image, i.e., a depth image, are used to generate a multi-view image . The depth image is a representation of the distance information of the objects using the disparity of the corresponding points of the left eye image and the right eye image in terms of grayscale. The artifact of the multi-view image generated by the DIBR algorithm determines the image quality of the stereoscopic image.

The DIBR algorithm requires a technique of scaling the input resolution to the output resolution when generating the multi-view image, and the resolution scaling can be performed before or after the warping.

However, if the horizontal scaling is performed before the warping, the logic size of the warping portion increases exponentially as the horizontal resolution increases.

On the other hand, in order to prevent an increase in the logic size of the warping portion, there is a problem in that when the horizontal scaling is performed after the warping, the resolution of the representable warping disparity decreases, and the warping artifact increases with the increase of the horizontal resolution.

It is an object of the present invention to provide a multi-view image generation method and apparatus capable of minimizing an increase in warping artifacts due to a difference in input / output resolution when scaling a horizontal resolution after warping, .

According to an aspect of the present invention, there is provided a multi-view image generating method including: a first step of vertically scaling an input image and a depth image; A second step of calculating disparity information and position information corresponding to interpolation placement information from the vertically scaled depth image for each viewpoint; A third step of warping each of the position information and the vertically scaled image according to a viewpoint using the disparity information; A fourth step of filling holes of the warped image and the position information according to the viewpoints; And outputting a multi-view image by horizontally scaling the filled image using a view point by using the filled position information.

A multi-view image generating apparatus according to an embodiment of the present invention includes a vertical scaler for vertically scaling an input image and a depth image; A disparity / position converter for calculating disparity information and position information corresponding to the interpolation placement information from the vertically scaled depth image for each viewpoint; A warping unit for warping each of the position information and the vertically scaled image according to a viewpoint using the disparity information; A hole filling unit for filling a hole of the warped image and the position information according to the viewpoint; And a horizontal scaler for horizontally scaling the filled image using a viewpoint using the filled position information to output a multi-view image.

When the horizontal scaler (fifth step) maps each pixel data of the peeled image to correspond to the output resolution, the horizontal scaler shifts and maps the data of each pixel according to the filled position information, And performs horizontal scaling for each viewpoint by interpolating the data.

The disparity / position converter (second step) calculates initial disparity information based on input resolution from the vertically scaled depth image; Dividing the initial disparity information and calculating temporary disparity information for each viewpoint; Separating the temporary disparity information into an integer part and a decimal part, and outputting the integer part as disparity information of each viewpoint; The fractional part is multiplied by the input / output resolution ratio, and then rounded to integers to output as position information of the respective viewpoints.

The disparity / position converter (second step) calculates disparity information based on output resolution from the vertically scaled depth image for each viewpoint; And outputs the quotient and the remainder obtained by dividing the disparity information based on the output resolution reference by the input / output resolution ratio and defines the disparity information and the position information based on the input resolution, respectively.

The disparity / position converter (second step) calculates initial disparity information based on input resolution from the vertically scaled depth image; Dividing the initial disparity information and calculating temporary disparity information for each viewpoint; The temporary disparity information is multiplied with the input / output resolution ratio and then rounded to integerize to calculate disparity information based on the output resolution on a per-view basis.

The hole filling unit (fourth step) performs a hole interpolating process using the valid data adjacent to the hole, the hole having no information generated by the warping in each of the warped image and the warped position information, Lt; / RTI >

The horizontal scalers (the fifth step), as expression mathematical below, showing a mapping position of the adjacent source pixel data (RGB ^'warp _before, RGB' ^warp _next) in said filling images, the adjacent original pixels, each position information ( (RGB ' ^scale _j ) interpolation processing by interpolation processing using the distance between the position' ^warp _before , position ' ^warp _next and the current pixel position j, .

≪ Equation &

The multi-view image generating method and apparatus according to the present invention can minimize some of the warping artifacts due to horizontal scaling by applying some of the disparity information of the output resolution to the input pixel arrangement information in the interpolation processing for horizontal scaling after warping .

That is, in the multi-view image generating method and apparatus according to the present invention, the temporary disparity information of the input resolution is divided into an integer portion and a decimal portion, the integer portion is used as disparity information for warping, and the decimal portion is used for horizontal scaling It is possible to minimize the increase of the warping artifact due to the horizontal scaling after warping by using the position information as the position information for interpolation.

1 is a block diagram showing a prior art DIBR processing apparatus according to the present invention.
FIG. 2 is an exemplary diagram illustrating a multi-view image generation process of the DIBR processing apparatus shown in FIG. 1. FIG.
3 is an exemplary diagram showing loss of warping resolution due to scaling of the DIBR processing apparatus shown in Fig.
4 is a block diagram showing a DIBR processing apparatus according to an embodiment of the present invention.
5 is a flowchart illustrating a DIBR processing method according to an embodiment of the present invention.
6 is an exemplary diagram illustrating an image scaling process using disparity information of the DIBR processing apparatus shown in FIG.
7 is an exemplary diagram showing a comparison result between the prior art and the application result of the DIBR processing apparatus according to the present invention.

Prior to describing the embodiments of the present invention, the prior art will be described in order to facilitate understanding of the DIBR algorithm related to the present invention.

FIG. 1 is a block diagram showing a prior art DIBR processing apparatus according to the present invention, and FIG. 2 is an exemplary diagram showing a multi view image generation process of the DIBR processing apparatus shown in FIG.

Referring to FIGS. 1 and 2, a vertical scaler 10 scales the 2D image and the depth image input from the outside in the vertical direction, and vertically scans the depth image in the vertical direction as shown in FIGS. 2 (a) and 2 (b) And outputs a 2D image and a depth image scaled in the direction.

The disparity converter 20 calculates disparity information indicating the degree to which each pixel is shifted according to the viewpoint of the multi-view using the depth image output from the vertical scaler 10, and outputs the disparity information.

The warping unit 30 warps the 2D image output from the vertical scaler 10 using the point-by-point disparity information output from the disparity converter 20, thereby outputting a warped image by point-in-time. FIG. 2 (c) shows one warped image of the multi-view output from the warping unit 30. FIG.

The hole filling unit (40) inserts appropriate information into the information-free hole present in the warped image output from the warping unit (30) and outputs it. Fig. 2 (d) shows the hole-filled image in the warped image shown in Fig. 2 (c).

The horizontal scaler 50 scales the image output from the hole filling unit 40 in the horizontal direction to generate and output a multi-view image matching the output resolution. FIG. 2 (e) shows an image in which the image shown in FIG. 2 (d) is horizontally scaled.

Thus, the prior art DIBR processing apparatus reduces the logic size of the warping unit 30 by performing vertical scaling, generating a warped image, and then performing horizontal scaling, rather than a structure that performs horizontal scaling before warping There are advantages to be able to.

However, the prior art DIBR processing apparatus has a drawback that the warping artifact is reduced in accordance with an increase in the horizontal resolution as shown in Fig. 2 (e) because the resolution of the warping disparity that can be expressed by horizontal scaling after warping is reduced have.

Referring to FIG. 2 (e), the output image is scaled horizontally four times as much as the original, and the disparity information, which is expressed by a shift of 1 pixel or 2 pixels in the original, 8 pixels, etc., it can be seen that the warping artifact is increased by increasing the vertical disparity deviation in horizontal scaling in the texture having many vertical details.

FIG. 3 is a diagram illustrating an example of loss of the warping resolution due to horizontal scaling when the output horizontal resolution is four times the input horizontal resolution in the DIBR processing technique of the prior art.

3 (a), when the RGB data is warped according to the scaled disparity after horizontally scaling the disparity, since the warping disparity is expressed in units of one pixel, there is no disparity difference in the input / output image Able to know.

Referring to FIG. 3 (b), the DIBR processing apparatus shown in FIG. 1 generates a multi-view image having the same horizontal resolution as an input image and then performs horizontal scaling. It can be seen that the resolution of the representable warping disparity is reduced by expressing the disparity in units of 4 pixels, and it can be seen that the warping artifact can be increased.

For example, when it is assumed that the first disparity of 5.4 pixels based on the input resolution and the second disparity of 5.7 pixels based on the disparity of the first and second pixels representing the vertical line in the input image are respectively calculated, The first disparity is outputted as 5 pixels and the second disparity is outputted as 6 pixels through the rounding of the part. If the first pixel is warped by 5 pixels and the second pixel is shifted by 6 pixels according to the output first and second disparities and then the warped first and second pixels are scaled horizontally four times, The first pixel is shifted by 5 * 4 = 20 pixels relative to the original, but the second pixel is shifted by 6 * 4 = 24 pixels to be represented in the output image, thereby increasing the warping artifact in the output image.

In order to solve artifact problems of the prior art, the present invention attempts to minimize the increase of warping artifacts due to horizontal scaling by applying some of the disparity information as placement information of input pixels in an interpolation process for horizontal scaling after warping.

In other words, according to the present invention, the temporary disparity information of the input resolution is divided into an integer part and a decimal part, and the integer part is used as disparity information for warping, and the decimal part is used as position information for interpolation for horizontal scaling The warping artifact increase can be minimized.

Hereinafter, a DIBR algorithm according to a preferred embodiment of the present invention will be described.

4 is a block diagram showing a DIBR processing apparatus according to an embodiment of the present invention.

4 includes a vertical scaler 110 for vertical scaling of an input 2D image and a depth image, disparity information from a vertically scaled depth image, and position information corresponding to interpolation placement information for each view point A warping unit 130 for warping the position information and the vertically scaled 2D image using disparity information, a warping unit 130 for generating warped image and warped position information generated by warping, A hole filling unit 140 for filling a hole, a horizontal scaler for shifting from a reference position in accordance with the filled position information to map each pixel of the filled image according to the output resolution, and horizontally scaling through interpolation processing 150).

5 is a flowchart illustrating a DIBR processing method according to an embodiment of the present invention.

4 and 5, the vertical scaler 110 scales the 2D image input from the outside and the depth image in the vertical direction, and outputs a vertically scaled 2D image and a depth image. (S110)

The disparity / position converter 120 uses disparity information indicating the degree to which each pixel is shifted at each view of the multi-view using the depth image output from the vertical scaler 110, And outputs the position information to be used in the interpolation process for the viewpoint. (S120)

The disparity / position converter 120 calculates the initial disparity information indicating the difference between the corresponding points of the left-eye image and the right-eye image from the vertically scaled depth image on the basis of the input resolution, and then outputs the initial disparity information to each of the multi- And calculates temporary disparity information at each time point. At this time, the temporary disparity information of each viewpoint may include an integer part and a decimal part by dividing the initial disparity information.

The disparity / position converter 120 separates the temporary disparity information of each viewpoint into an integer part and a decimal part, and then defines an integer part as disparity information (disparity) The fractional part can be defined as position information (position ') at each time point by integrating it with rounding after multiplication with input / output resolution ratio (rate _horizontal ).

On the other hand, the disparity / position converter 120 calculates the disparity information (diparity _out ) based on the output resolution from the vertically scaled depth image, and calculates disparity information based on the input resolution (disparity ') And position information (position') can be calculated for each viewpoint.

As shown in Equation (1), and parity disk / position converter 120 calculates the ratio of input and output resolution _(horizontal rate) by calculating the input horizontal resolution (xsize _in) for the output horizontal resolution (xsize _out). To this end, the disparity / position converter 120 receives input and output horizontal resolution information (xsize _in , xsize _out ) from the outside.

Display a parity / position converter 120 has an output resolution of the disparity information (diparity _out) the resolution rate (rate _horizontal) of the input resolution of the quotient and the remainder divided by the disparity information (disparity ') and position information (position') Respectively. Disparity information (disparity ') and position information (position') are calculated for each pixel at each time point.

For example, Figure 6 with respect to the six pixels (P1 ~ P6) side by side in the horizontal direction as shown in (a), the output resolution of the disparity information _{(disparity out) 0, 1,} 2, 3, 4, 5 is calculated , in assuming a case where the input resolution ratio (rate _horizontal) 4, dividing the disparity information of the output resolution _{(disparity out) 0, 1,} 2, 3, 4, 5 respectively in input and output resolution rate (rate _horizontal) 4 share The corresponding 0, 0, 0, 0, 1, and 1 are output as disparity information (disparity ') of each of the pixels P1 to P6, (Position ') of the respective positions P1 to P6.

The disparity / position converter 120 calculates temporal disparity information of each viewpoint including a fractional part based on the input resolution from the vertically scaled depth image, and then supplies the temporal disparity information to the _rate- And then rounding the result to integer to calculate disparity information (disparity _out ) of the output resolution.

For example, based on the input resolution from the vertical scaling a depth image, assuming a case in which the initial disparity information 1.2 of a pixel for the point in time at which the calculated initial display input resolution ratio parity information 1.2 (rate _horizontal) 4 (1.2 * 4 = 4.8), and the result (4.8) is rounded off to calculate the disparity information (disparity _out ) 5 of the output resolution corresponding to the sixth pixel (P6).

As described above, the disparity / position converter 120 separates the initial disparity information 1.2 into an integer portion and a decimal portion without using Equation 1, and the integer portion 1 divides the initial disparity information 1.2 into an integer portion and a decimal portion, disparity information (disparity ') and defines the output, the fractional part of 0.2 is the output resolution rate (rate _horizontal) 4, and a multiply operation one (0.2 * 4 = 0.8), the sixth pixel (P6) the Integer a first rounded after (Position ') of the position information (position').

The warping unit 130 warps the 2D image (RGB) output from the vertical scaler 10 according to the disparity information (disparity ') output from the disparity / position converter 120 and outputs the warped image . The warping unit 130 warps the position information (position ') output from the disparity / position converter 120 according to the point disparity information (disparity') to output the warped position information at each point in time do. (S130)

For example, as shown in FIG. 6B, the warping unit 130 warps and warps a 2D image (RGB) and position information (position ') according to disparity information (disparity'), (RGB ^warp ) and warped position information (position ^warp ).

The hole filling section 140 fills in the information-free hole, which occurs in the warped image and the warped position information, by performing warping of the warping section 130, and outputs the information. (S140)

The hole filling unit 140 linearly interpolates the valid data (before, next) adjacent to both sides of the hole as shown in the following Equation (2) Image data (RGB ' ^warp ) and appropriate position information (position' ^warp ).

Where L is the length of the hole, d is the distance from the hole to be filled by interpolation to the next valid data, and before / next means the previous / next valid data value, respectively.

The horizontal scaler 150 scales horizontally the warped image (RGB ' ^warp ) from the hole filling unit 140 in accordance with the output resolution in consideration of the warped position information (position' ^warp ) view 5). (S150)

The horizontal scaler 150 scales the warped image from the hole filling unit 140 in the horizontal direction in accordance with the output resolution so as to position the data of each pixel according to the warped position information (position ' ^warp ) The data is shifted from the reference position R and linear interpolation is performed using adjacent pixel data to output a horizontally scaled image reflecting the disparity _out of the desired output resolution.

For example, as shown in Fig. 6 (d), the horizontal scaler 150 scales the warped image (RGB ' ^warp ) in the horizontal direction to map the original pixels P1 to P6, respectively, 1, 2, 3, 2, and 0 from the reference positions R1 to R6 in accordance with the position information (0, 1, 2, 3, 2, 0).

The horizontal scaler 150 calculates the position information indicating the position of the adjacent original pixel (RGB ' ^warp _before , RGB' ^warp _next ) and the position of the adjacent original pixel in the filled image (position (RGB ' ^scale _j ) of the current pixel can be calculated through an interpolation process using the distance between' ^warp _before , position ' ^warp _next and the current pixel position j.

7 is an exemplary diagram showing a comparison result between the prior art and the application result of the DIBR processing apparatus according to the present invention.

Referring to FIG. 7, in the horizontal scaler 50 of the DIBR processing apparatus according to the prior art, when interpolation processing is performed after horizontally scaling only the warped image according to the output resolution as shown in (a) It can be seen that artifacts due to the upper and lower disparity deviations are conspicuous as shown in (b) due to the lowering of the parity resolution.

On the other hand, in the DIBR processing apparatus according to the present invention, in the horizontal scaling step, the horizontal scaler 150 scans the original pixels of the warped image according to the position information using some of the disparity information (decimal part) It can be seen that artifacts can be reduced as compared to the prior art shown in (c), as shown in (d), by performing the interpolation processing after shifting the objects.

As described above, the DIBR processing apparatus and method according to the present invention minimize some of the warping artifacts due to horizontal scaling by applying a part of the disparity information of the output resolution as the arrangement information of the input pixels in the interpolation processing for horizontal scaling after warping have.

That is, in the DIBR processing apparatus and method according to the present invention, the temporary disparity information of the input resolution is divided into integer part and decimal part, the integer part is used as disparity information for warping, and the decimal part is used for interpolation for horizontal scaling It is possible to minimize the increase in warping artifacts due to horizontal scaling after warping by using the position information as placement information.

It will be apparent to those skilled in the art that various modifications and variations can be made in the present invention without departing from the spirit or scope of the general inventive concept as defined by the appended claims and their equivalents. Will be clear to those who have knowledge of.

10, 110: vertical scaler 20: disparity converter
30, 130: warping portion 40, 140: hole filling portion
50, 150: horizontal scaler 120: disparity / position converter

Claims (14)

A first step of vertically scaling an input image and a depth image;
A second step of calculating disparity information and position information corresponding to interpolation placement information from the vertically scaled depth image for each viewpoint;
A third step of warping each of the position information and the vertically scaled image according to a viewpoint using the disparity information;
A fourth step of filling holes of the warped image and the position information according to the viewpoints;
And outputting a multi-view image by horizontally scaling the filled-in image by the viewpoint using the filled position information. The method according to claim 1,
The fifth step
Wherein when mapping each pixel data of the poured image to output resolution, the data of each pixel is shifted and mapped according to the poured position information, and then the mapped pixel data is interpolated, Wherein the horizontal and vertical scaling are performed. The method according to claim 1,
The second step
Calculating initial disparity information based on input resolution from the vertically scaled depth image;
Dividing the initial disparity information and calculating temporary disparity information for each viewpoint;
Separating the temporary disparity information into an integer part and a decimal part and outputting the integer part as disparity information at each time point;
And multiplying the fractional part by an input / output resolution ratio and integrating the fractional part by rounding to output as position information of the respective viewpoints. The method according to claim 1,
The second step
Calculating disparity information based on output resolution from the vertically scaled depth image for each viewpoint;
And outputting a quotient obtained by dividing the disparity information based on the output resolution reference by an input / output resolution ratio and the remainder as the disparity information and the position information based on the input resolution, respectively. The method of claim 4,
The second step
Calculating initial disparity information based on input resolution from the vertically scaled depth image;
Dividing the initial disparity information and calculating temporary disparity information for each viewpoint;
Further comprising the step of multiplying the temporary disparity information by the input / output resolution ratio and then performing rounding to integrate the disparity information based on the output resolution based on the viewpoints. The method according to claim 1,
The fourth step
Characterized in that a hole having no information generated by the warping in each of the warped image and the warped position information according to the viewpoint is filled through a linear interpolation process using valid data adjacent to the hole, How to create an image. The method according to claim 1,
The fifth step
As mathematical expressions and below, wherein the filling the adjacent source pixel data of the image _{^{(RGB 'warp before, RGB'}} warp next) and the adjacent position information showing the mapping position of the original pixels, respectively (position ^'warp _before, position' ^warp _next ) And a current position of the pixel (j), interpolation processing of the current pixel data (RGB ' ^scale _j ) is performed through an interpolation process, and rate is an input / output resolution ratio. Generation method.
≪ Equation &

A vertical scaler for vertically scaling the input image and the depth image;
A disparity / position converter for calculating disparity information and position information corresponding to the interpolation placement information from the vertically scaled depth image for each viewpoint;
A warping unit for warping each of the position information and the vertically scaled image according to a viewpoint using the disparity information;
A hole filling unit for filling a hole of the warped image and the position information according to the viewpoint;
And a horizontal scaler for horizontally scaling the filled-in image by the viewpoint using the filled position information, and outputting a multi-view image. The method of claim 8,
The horizontal scaler
Wherein when mapping each pixel data of the poured image to output resolution, the data of each pixel is shifted and mapped according to the poured position information, and then the mapped pixel data is interpolated, And the horizontal-scaling is performed. The method of claim 8,
The disparity / position converter
Calculating initial disparity information based on input resolution from the vertically scaled depth image;
Dividing the initial disparity information and calculating temporary disparity information for each viewpoint;
Separating the temporary disparity information into an integer part and a decimal part, and outputting the integer part as disparity information of each viewpoint;
Multiplies the fractional part by an input / output resolution ratio, and rounds the integer part to output as position information of the respective viewpoints. The method of claim 8,
The disparity / position converter
Calculating disparity information based on output resolution from the vertically scaled depth image for each viewpoint;
Wherein the dividing unit divides the quotient obtained by dividing the disparity information based on the output resolution reference by the input / output resolution ratio and the remainder is defined as the disparity information and the position information based on the input resolution. The method of claim 11,
The disparity / position converter
Calculating initial disparity information based on input resolution from the vertically scaled depth image;
Dividing the initial disparity information and calculating temporary disparity information for each viewpoint;
Wherein the disparity information based on the output resolution is calculated for each viewpoint by integrating the temporary disparity information by multiplying the temporary disparity information by the input / output resolution ratio, and rounding the resultant. The method of claim 8,
The hole-
Characterized in that a hole having no information generated by the warping in each of the warped image and the warped position information according to the viewpoint is filled through a linear interpolation process using valid data adjacent to the hole, Image generating device. The method of claim 8,
The horizontal scaler
As mathematical expressions and below, wherein the filling the adjacent source pixel data of the image _{^{(RGB 'warp before, RGB'}} warp next) and the adjacent position information showing the mapping position of the original pixels, respectively (position ^'warp _before, position' ^warp _next ) And a current position of the pixel (j), interpolation processing of the current pixel data (RGB ' ^scale _j ) is performed through an interpolation process, and rate is an input / output resolution ratio. Generating device.
≪ Equation &

Images