JP2013114682A - Method for generating virtual image - Google Patents

Abstract

PROBLEM TO BE SOLVED: To facilitate countermeasures to an error in a depth map image in rendering based on a depth image.SOLUTION: The image of a virtual view of a scene is generated on the basis of a set of texture images obtained from the scene and a corresponding set of depth images. A set of candidate depths associated with the respective pixels of the selected image is searched. Costs that estimate the synthesis quality of the virtual image are determined for each candidate depth. The candidate depth having the minimum costs is selected, and the optimal depth of each pixel is obtained. Then, virtual images are synthesized on the basis of the optimal depth of each pixel and the texture images. First depth emphasis and the second depth emphasis are applied prior to view synthesis and during view synthesis so that an error can be corrected, or a noise due to the estimation or acquisition of the fine depth image and rough depth characteristics can be suppressed.

Description

  The present invention relates generally to depth image based rendering (DIBR), and more particularly to a method for enhancing a depth image using a trellis structure.

  The 3D display presents images of different views of the 3D scene for each eye. In a conventional stereo system, the left view and right view images are acquired, encoded, stored or transmitted, and then decoded and displayed. In more advanced systems, virtual images having different viewpoints than existing input views are synthesized to improve (or enhance) 3D features, such as adjusting the perceived depth of a 3D stereo display, and Enables the generation of multiple virtual images for new virtual views and can support multi-view autostereoscopic displays.

  Depth image based rendering (DIBR) is a method of synthesizing virtual images and typically requires a depth image of the scene. Depth images can contain noise, which can result in artifacts in the rendered image, and pixel-level depth images usually do not necessarily represent depth discontinuities that occur at object boundaries This is another source of artifacts in the rendered image.

  As shown in FIG. 1, the prior art view synthesis includes a warping step 110, in which pixels corresponding to virtual locations are based on the geometry of the scene, the reference input images 101 and 102, ie the reference image. The warped image is warped from the texture image and the depth image. In a texture image, each pixel (sample) has a 2D location and brightness, and the brightness can be a color when 3 (RGB) channels are used. In the depth image, each pixel at the 2D location is the depth from the camera to the closest point of the scene.

  During blending 120, for each input viewpoint, the warped images are combined into a single image. The hole filling 130 fills any remaining holes in the blended image and generates a combined virtual image 103. Blending is executed only when there are a plurality of input viewpoints from which a synthesized virtual image is generated.

  The warping step can include forward warping and backward warping. In forward warping, pixels in the reference image are mapped to a virtual image via 3D projection. In backward warping, pixels in the reference image are not directly mapped to the virtual image. Instead, the depth is mapped to the virtual image, and then the warped depth image is used to determine a corresponding pixel in the reference image for each pixel location in the virtual image.

  Most of the pixels in the virtual image are mapped after the warping process. However, some pixels do not have any corresponding mapped depth. This is caused by disocclusion from one viewpoint to another. The mapped depthless pixels are known as holes in the virtual image.

  When multiple input reference images exist, the warping results are merged into a single image using blending. Some holes can be complementarily filled during this step. That is, the holes in the left reference image can have values mapped from the right reference image. In addition, blending can also eliminate mapping conflicts that occur when there are different mapping values from different reference images. For example, a weighted average can be applied or one of the mapping values is selected depending on the proximity of the virtual viewpoint location to the reference image.

  Some holes remain after the blending process. For this reason, final hole filling is necessary. For example, inpainting can be used to propagate surrounding pixel values into the remaining holes. In one implementation, background pixels are spread into small holes.

  Prior art methods cannot handle errors in depth map images. Thus, there is a need for more accurate view synthesis to improve the quality of the composite image so that the composite image is free of boundary artifacts and the composite image geometrically matches the image features present in the input image. Has been.

  View composition is an essential feature for multiple 3D video applications, including free viewpoint navigation and image generation for autostereoscopic displays. For this purpose, a depth image based rendering (DIBR) method is usually applied.

  However, the quality of the rendered image is very sensitive to the quality of the depth image, which is usually estimated by a process that is prone to error. Furthermore, the depth image per pixel is not an ideal representation of a 3D scene, especially along the depth boundary. This expression may cause an unnatural result of a scene having an occlusion area.

  Embodiments of the present invention provide a trellis-based view synthesis method that can overcome the above limitations in depth images and reduce artifacts in rendered images. In this way, for each pixel that needs to be warped, a candidate set of depths is identified based on the estimated depth of that pixel and the neighboring depths. The cost of each candidate depth is quantified based on the composite quality estimate. Next, a candidate depth having the optimal expected quality is selected.

  The method applies first depth enhancement and second depth enhancement prior to and during view synthesis to correct errors or due to estimation or acquisition of dense depth images and sparse depth features. Suppress noise.

It is a block diagram of a conventional view synthesis method. FIG. 4 is a schematic diagram of a view synthesis trellis constructed in accordance with an embodiment of the present invention. FIG. 4 is a schematic diagram of neighboring pixels used to predict the depth of the next pixel, in accordance with an embodiment of the present invention. FIG. 6 is another schematic diagram of neighboring pixels used to predict the depth of the next pixel, in accordance with an embodiment of the present invention. FIG. 6 is another schematic diagram of neighboring pixels used to predict the depth of the next pixel, in accordance with an embodiment of the present invention. FIG. 6 is a schematic diagram of increasing or decreasing depth boundaries assigned different cost functions according to an embodiment of the present invention. 3 is a flowchart of a trellis-based view synthesis method according to an embodiment of the present invention; 4 is a flowchart of a non-iterative method of trellis-based view synthesis according to an embodiment of the present invention. 4 is a flowchart of an iterative method of trellis-based view synthesis according to an embodiment of the present invention. 1 is a block diagram of a system that includes dense depth estimation, sparse depth estimation, and trellis-based view synthesis in accordance with an embodiment of the present invention. FIG. 4 is a block diagram of trellis-based view synthesis based on dense depth images and sparse depth features, according to embodiments of the invention. FIG. 4 is a block diagram of trellis-based view synthesis using a depth enhancement method according to an embodiment of the present invention. It is a figure of the pixel used for color difference cost calculation.

  The depth image can have errors caused by the estimation process or the acquisition process. In addition, the representation of the depth image for each pixel is not always accurate at the depth discontinuity.

  Accordingly, embodiments of the present invention provide trellis-based view synthesis methods that overcome limitations in depth image representation and estimation. The depth image can be acquired by a range camera or can be estimated from stereo display correspondences (stereo disparity correspondences) in the left texture image and the right texture image. The method is applied during the depth image based rendering (DIBR) warping process.

  FIG. 2 shows an example of a trellis 201 constructed for view synthesis according to an embodiment of the present invention. The trellis 201 is constructed for a predetermined number of pixels. In one embodiment, one line of image pixels is configured into a trellis and the warping process is performed line by line. That is, each column of the trellis represents one image pixel having a different depth A-D. A node in each column of the trellis represents a candidate depth mapping for that pixel in the virtual image.

  In the first step, a set of candidate depths (A, B, C, D) 202 is identified for each pixel. The set includes an estimated depth from the input depth image and several other candidate depths based on neighboring depths. The number of candidate depths corresponds to the number of rows in the trellis. In FIG. 2, each pixel has four depths A-D corresponding to four rows in the trellis.

  In the second step, the synthesis quality is estimated using a cost function. Composite quality is a criterion for selecting the optimal candidate depth.

Determining a set of candidate depths In a first step, a set of candidate depths including an estimated depth from an input depth image is identified. In addition to this value, several other candidate depths are identified from neighboring depths. The candidate depth can be used when the estimated depth from the input depth image is not correct, i.e., when the depth causes artifacts or mismatches with the input image. Hereinafter, several methods for obtaining the optimal candidate depth will be described.

  One way to determine the set of candidate depths is to use a predetermined increase and / or decrease with respect to the estimated depth from the input depth image. For example, if the estimated depth is 50, the depth candidate set may include {49, 50, 51}. Incrementing by a factor other than 1 can also be considered. The number of depths can be variable, and is not necessarily symmetrical about the estimated depth. For example, the set can be {46, 48, 50, 52, 54}, or {48, 49, 50, 52, 54}. Candidate depth can also be obtained by a look-up table. In the look-up table, the candidate depth can vary for each estimated depth in some cases.

  A second method for obtaining a set of candidate depths is a method that uses a prediction value based on depths from neighboring pixels. For example, an average depth or a central depth from neighboring depths can be used. A predetermined window size can also be used to determine the number of neighboring pixels considered in the prediction.

  The preferred method includes the previous pixel from the same line in the window. In FIG. 3, there are four pixels 301 in the same line from the left in the window. In FIG. 4, there are four pixels 401 in the same column from the top line in the window. In FIG. 5, a 4 × 4 window of pixel 501 is identified. In another embodiment, the pixels can conform to any shape. As a result of the increase in the number of candidate depths, the computational complexity increases. This is because each candidate is checked and compared.

  In FIG. 2, the number of candidate depths is set to 4 for each pixel. In one example, the depth A (first line from the bottom) represents the estimated depth from the input depth image. Depths B and C (middle rows 2 and 3) are depths obtained by increasing or decreasing depth A by 1, respectively. Depth D (top row) indicates the depth predicted by using the central depth from neighboring pixels as shown in FIG.

View Synthesis Using Dynamic Programming After a set of candidate depths is determined, each node in the trellis is assigned a metric according to a cost function that estimates the synthesis quality. The view synthesis problem is then solved by finding an optimal set of depths across the trellis. For example, dynamic programming is used to solve optimization problems.

  In order to estimate the composite quality, an evaluation function is defined as a cost function. The cost function can depend on whether the warping process is forward warping or backward warping. Without loss of generality, the cost function definition will be described assuming backward warping for the preferred embodiment of the present invention. This definition applies to forward warping.

  In one implementation, the cost function evaluates the mean square error (MSE) between two square blocks of pixels. These blocks are the upper left blocks relative to the pixel location. Let (x, y) represent the current pixel location and (x ', y') represent the warped position using the candidate depth.

  The first block is located at (x-s, ys) to (x, y) in the synthesized virtual image, and the second block is (x'-s, y'-s) to Located at (x ′, y ′). Here, s is a block size. Cropping is applied if a part of the block exceeds the image area.

  An energy function other than MSE can also be used as a cost function. For example, the average absolute error is an effective cost function that estimates the composite quality. Image features or structural similarity measures can also be extracted from the blocks, and a matching process can be used to determine whether the blocks are geometrically consistent.

  Since any artifact in the foreground object is more easily perceived by the human eye, there is a need for a way to synthesize foreground objects in a consistent manner. For this reason, in this method, the upper left block is not always used to obtain the cost metric.

  As shown in FIG. 6, the pixels are classified into three types of areas, namely, a flat area 601, a depth reduction area 602, and a depth increase area 603 as shown in FIG. For pixels that are in the depth-decreasing boundary (right boundary in FIG. 6) or flat area, the upper left block is used. In the case of pixels at the depth increasing boundary (left boundary in FIG. 6), the upper right block is used.

  In some applications, a confidence map can be used in addition to the estimated depth image as input to the synthesis process. When the depth estimator shows high reliability, the cost function of depth from the depth image can be weighted by a coefficient.

System Embodiments Hereinafter, three embodiments shown in FIGS. 7 to 9 will be described for image synthesis based on trellis. These embodiments are arranged in ascending order of complexity. In these figures, “samples” are pixels in various images.

  In the first embodiment as shown in FIG. 7, local optimization is performed with limited complexity. In this embodiment, candidate depth selection does not rely on selection of optimal depth candidates from previous pixels. Thus, candidate depth assignment and pixel evaluation can be performed in parallel. A step-by-step description of this embodiment is described below.

  Each step shown in the various figures may be performed in a processor connected to a memory and input / output interface as known in the art. A virtual image can be rendered and output to a display device. Alternatively, each step can be implemented in the system using means including separate discrete electronics within a video encoder or decoder (codec). More specifically, for video encoding / decoding systems (codecs), images of other views can be predicted using the method for generating virtual images described in the present invention. See, for example, US Pat. No. 7,728,877 “Method and system for synthesizing multiview videos”, incorporated herein by reference.

Step 701: Identify candidate depths for all pixels in the trellis. In this step, the following candidates are determined:
a. Depth A: Select the depth signaled in the depth image of the current pixel. If the pixel is not the first pixel in the line, two further depth candidates are selected as follows:
b. Depth B: Select the depth that is most different from Depth A in the set of depths signaled in the depth images of multiple pixels in front of the same line. The previous pixel is as shown in FIG. Four previous pixels are preferred.
c. Depth C: Unlike the depth B, the depth C is selected from the same line, and is selected between the depths in the same row from the upper line as shown in FIG.
d. Depth D: There is no such candidate depth in this embodiment.

  Step 702: Evaluate the cost for each candidate depth of each pixel.

  Step 703: Compare the cost of all candidate depths for each pixel to find the candidate depth with the lowest cost. Select the corresponding depth for each pixel.

  FIG. 8 illustrates a second embodiment that is also local optimization with limited complexity. In this embodiment, the candidate depth assignment within a trellis column relies on the optimal depth selection of the previous pixel or column in the trellis. The following is a step-by-step description of this embodiment.

  Step 801: The index i is initialized.

  Step 802: Identify the candidate depth of pixel i. In this step, three depth candidates selected in the same manner as the embodiment shown in FIG. 7 are included. However, when deriving depths B and C, the optimal depth from the previous pixel is used, which can be different from what is signaled in the depth image.

  Step 803: Evaluate the cost for each depth candidate of pixel i.

  Step 804: Compare the costs of all depth candidates and determine the minimum cost of pixel i.

  Step 805: If there are more pixels not processed in the trellis, i is incremented by 1 (806) and iterated.

  In the first two embodiments, optimal depth candidates are selected for each row in the trellis by evaluating a local cost function. In the third embodiment, an optimum path across the trellis, which is a combination of depth candidates from a column, is obtained. The route cost is defined as the sum of the node costs in the route.

  Nodes can have different costs in different paths. This embodiment is illustrated in FIG. The procedure has two loops that iterate over i and p. The outer loop spans all possible paths, while the inner loop is for all nodes in one possible path.

  For each potential path, node candidate depths are sequentially identified (901) and evaluated (902) within the path. Depth candidate assignment is obtained as follows. It is determined whether there are more pixels in the path (903).

  If the next node is located in row “depth A”, the node is set to the depth as signaled in the depth image. If the node is located in row “depth B”, then select the depth that is the median from the given depth set of previous pixels in the same line. A given depth of the previous pixel is specified for the current path. If the node is located in row “depth C”, the node is selected as the median depth from the same column of the top line in the image.

  If different paths intersect the same node, a different depth can be assigned to depth B for that same node. Depths A and C remain the same for different paths.

  After all nodes in the path have been evaluated, the path cost is determined as the sum of the node costs (904), and if there are no more paths (905), the path with the minimum cost is used for the final composite result. (906).

View Synthesis Using Sparse Depth In US patent application Ser. No. 13 / 026,750, a related application of the present invention, uses depth images as input to the view synthesis process, where the estimated depth is based on trellis Considered one of several candidate depths in the view synthesis process. In this way, each pixel in the input image is associated with a corresponding depth to form a depth image. These depth images are called dense depth images.

  In contrast, sparse depth features refer to a collection of depths associated with a small subset of pixels in the input texture image. A number of known techniques can be used to determine sparse depth features, including known money-Lucas-Tomasi (KLT) feature trackers. The KLT feature tracker first detects an image, eg, a corner point or salient feature of the left view, and then finds a corresponding feature in another image, eg, the right view.

  As shown in FIG. 10, dense depth estimation 1010 is performed from an input stereo video image (video) 1001, and a dense depth image 1011 corresponding to the left view and right view of the stereo pair is generated. Similarly, sparse depth estimation 1020 is performed from the input stereo video, and a sparse depth feature set 1021 is generated based on the correspondence between the left and right views of the stereo pair.

  Next, as described above with reference to FIGS. 7 to 9, trellis-based view synthesis 1030 is performed using the dense depth image, the sparse depth feature, and the input stereo video, and the virtual image 1002. Is generated.

  For convenience of this description, the sparse depth feature forms a so-called sparse depth image.

  As shown in FIG. 11, the dense depth image 1011 receives a dense depth warping 1110. Dense depth warping 1110 generates a warped dense depth image corresponding to the position of the virtual view. Warping is accomplished by mapping each depth to a corresponding depth in the virtual view according to the virtual view position and the geometry parameters of the scene.

  In the preferred embodiment of the invention, there are two warped dense depth images. One corresponds to the warping of the dense depth image of the left view, and the other corresponds to the warping of the dense depth image of the right view. The depth of the warped dense depth image is a candidate depth for view synthesis based on the trellis.

  Further, the sparse depth feature 1021 receives a sparse depth warping 1120. The sparse depth warping 1120 first generates warped sparse depth features in the virtual view. Sparse depth feature warping is similar to dense depth image warping, but is performed on a smaller subset of features compared to a complete set of pixel locations in the input image. Next, a dense set of depths is determined from a set of warped sparse features using known prior art techniques such as nearest neighbor assignment, linear interpolation, bicubic interpolation and the like.

  Alternatively, interpolation can first be performed on sparse depth features and then mapped to a virtual view. The output of the sparse depth mapping process generates additional candidate depths for trellis-based view synthesis.

  As shown in FIG. 2, multiple candidate depths can be evaluated for view synthesis of each pixel in the virtual view. In a preferred embodiment of the present invention, candidate depths are determined from dense depth images and sparse depth features.

  The trellis structure 1130 in FIG. 11 generates a trellis as shown in FIG. 2, where each column corresponds to one pixel location in the virtual view and each node in one column is used for synthesis. Corresponding to one candidate depth.

  The trellis is constructed for one row of the virtual view image. Each node is associated with one candidate depth and an estimated composite quality metric using disparity candidates. All of the methods for generating candidate depths described above can be used. In addition, candidate depths determined from sparse depth features can be used when creating a trellis.

  After the trellis is built, a minimum cost path through the trellis is determined 1140 according to the embodiment described with respect to FIGS. The resulting depth set is used to warp the input image to the virtual view position (1150). This process is performed for both the left and right input views.

  Finally, blending step 1160 averages the left and right views by a weighting factor that depends on the distance of the left and right views from the reference view. If the virtual view position is closer to the left view, the view warped from the left view has a greater weighting factor than the view warped from the right view. Hole pixels in one warped view are filled with the other warped view if they are not holes in the other warped view. After blending, the final virtual view image is displayed.

Depth Enhancement Using Trellis and Sparse Depth For clarity, only left view processing is shown and described with respect to FIG. The processing for the right view is similar. Optional items are indicated by broken lines.

  The above embodiments of the present invention have described a view synthesis method that utilizes dense depth images 1011 and sparse depth features 1021. In this method, using the estimated depth and sparse depth features, several candidate depths are determined as part of the trellis-based view synthesis process.

  In this embodiment of the invention, a method is used to enhance the quality of the dense depth image based on the dense depth image and sparse depth features. In this context, enhancement refers to error correction or suppression of noise due to dense depth image estimation or acquisition. The enhanced depth image is used in subsequent view synthesis.

  As shown in FIG. 12, the depth enhancement 1201 can be applied to the dense depth image 1011 and the sparse depth feature 1021 corresponding to the acquired left view and right view. The second depth enhancement 1203 corresponds to the virtual view in the view composition 1230. A first depth enhancement is applied to the left view to generate a first enhanced dense depth image (1202). The first depth enhancement can also be applied independently to the right view.

  In this embodiment, the set of depth candidates 202 is selected from the first enhanced depth image. Before the path with the lowest cost through the trellis is selected, the cost of each depth candidate is determined.

  Depth enhancement is followed by occlusion handling 130, ie, if a single view is used, hole filling and warping 110 or extrapolation. This applies only to the first enhanced depth image (s). At this time, the texture image is not used.

  During view synthesis 1230, after occlusion handling, warping and extrapolation to the virtual view, a second depth enhancement 1203 is applied to the first enhanced depth image and the second enhanced depth image at the virtual view position. Is generated. The second enhanced depth image is then used with the texture image (s) during the second warping and extrapolation.

  During both the first depth enhancement and the second depth enhancement, the input depth identified in the dense depth image is always selected as the first candidate. This depth can be inaccurate, which can lead to artifacts or mismatches to the input image. Therefore, additional candidates are considered.

  During the first depth enhancement, alternative depth candidates are selected based on the minimal collocated (color) intensity difference of the previous pixel in the same row in the texture image. The depth of the nearest sparse depth feature is also selected as an alternative candidate.

  During the second depth enhancement, the median depth from the previous five depths in the same row and column is selected as an alternative candidate.

  Following the second depth enhancement, texture images 1221 and 1222 can be warped (1240), blended and extrapolated (1250) to generate virtual image 1002.

  During the first depth enhancement, because of the cost function of the depth candidate, there are three measurements: stereo cost, ie color intensity consistency between the two views, and color difference cost, ie current location and candidate pixel location. And the color difference between and the depth difference cost, i.e. the depth difference between the current pixel and the alternative candidate.

  As shown in FIG. 13, the stereo cost is between the two windows (A, B) in the left color (texture) image and the right color (texture) image, where the location depends on the current pixel location and depth. Defined by mean value separated mean absolute difference (mrMAD).

  FIG. 13 shows an example of the color difference cost. The left color image 1301 is shown at the top and the depth (parallax) image is shown at the bottom. The color difference cost is defined as the MAD of the two surrounding windows 1303 at the current pixel location 1311 and the candidate pixel location 1312. The right image is processed in the same way.

  In depth enhancement B, only the stereo cost and the depth difference cost are considered. In order to determine the stereo cost, the location of two windows in the reference view is determined. This is determined depending on the current pixel location, the candidate depth, and the position of the virtual view relative to the reference view.

  After the candidate parallax cost is determined, this cost is compared to the current parallax cost. If the cost of the candidate parallax is lower than the cost of the current parallax, the current parallax is updated to the candidate parallax. After all the pixels have been processed, an enhanced depth map is output in subsequent steps.

Claims (11)

A method for generating a virtual image of a virtual view of a 3D scene based on a texture image and a depth image, wherein the depth image includes an associated dense depth image and an associated sparse depth feature, The method is
Applying a first depth enhancement to the dense depth image and the sparse depth feature to obtain a first enhanced depth image;
Determining a plurality of candidate depths for each pixel of the first enhanced depth image;
Obtaining a cost for estimating the synthesis quality of the virtual image for each candidate depth;
Selecting the candidate depth with the lowest cost to generate an enhanced depth of the pixel in the first enhanced depth image;
Applying view synthesis to the first enhanced depth image and the corresponding texture image to generate the virtual image; and
Each said step is executed in a processor,
A method for generating virtual images.   The method further comprising: applying a second depth enhancement to the first enhanced depth image for the virtual view to obtain a second enhanced depth image used during the view synthesis. The method according to 1.   The method of claim 1, wherein each of the steps is performed similarly for a left view and a right view of the 3D scene.   The method of claim 1, wherein the first depth enhancement and the second depth enhancement correct errors and suppress noise due to estimation or acquisition of the dense depth image and the sparse depth features. .   The method of claim 1, wherein after the first depth enhancement, occlusion handling is performed on the first enhanced depth image.   The method of claim 1, wherein the minimum cost is determined before the path with the minimum cost through the trellis is selected.   The method of claim 6, wherein the candidate depth is based on a minimum connected luminance difference of previous pixels in the same row of the texture image.   The method of claim 1, wherein the candidate depth includes a nearest sparse depth feature.   The method of claim 1, wherein the minimum cost is based on a stereo cost, ie, color luminance consistency between two views and a color difference cost.   The method of claim 9, wherein the stereo cost is defined by an average separation average absolute difference between two windows in a left texture image and a right texture image.   The method of claim 9, further comprising determining a cost of the candidate depth parallax.

Images