KR101710444B1 - Depth Map Generating Apparatus and Method thereof - Google Patents

Abstract

A depth map generation apparatus and method is disclosed and automatically generates a depth map corresponding to each two-dimensional image of video from normal video. An apparatus for generating a depth map, comprising: an image obtaining unit for obtaining a plurality of two-dimensional images that are continuous in time among input video; A saliency map generator for generating at least one saliency map corresponding to a current two-dimensional image according to an HVP model; A depth map generation unit based on conspicuousness; Dimensional image and a plurality of pre-stored three-dimensional representative structures among the plurality of two-dimensional images, and determining the three-dimensional representative structure having the highest matching degree as the three-dimensional structure of the current two- Dimensional structure matching section; A depth map based on registration; An overall depth map generation unit for generating an overall depth map by combining a depth map based on conspicuousness and a depth map based on matching; And a space-time area smoothing unit for smoothing the composite depth map.

Description

[0001] The present invention relates to a depth map generating apparatus and method,

The present invention relates to an apparatus and method for generating a depth map, and more particularly to an apparatus and method for automatically generating a depth map corresponding to a two-dimensional image of each frame of video, usually of video.

In recent research areas and commercial markets, 3D television is becoming a hot issue. The difference between a three-dimensional television and a conventional two-dimensional television lies in the stereoscopic video it represents. The viewer can feel the depth effect like the actual three-dimensional scene. This effect is based on the visual model theory of the human eye. When a person looks at the real world using two eyes and sees a three-dimensional scene, the images of the two eyes become different. Through two different images of the left and right eyes of an independently projected person, a person will form a three-dimensional scene in the brain.

Currently, however, most media (movies, video) and image acquisition devices (digital cameras, film cameras, etc.) still rely on mono systems using a single camera. When such a medium is directly displayed on a three-dimensional television, it can not exhibit a three-dimensional effect. In order for such a medium to be converted into three-dimensional video, there is a method of employing a large number of persons as a solution and displaying a depth map of each region by hand. The consequences of this transition can be very satisfying, but the drawback is that it requires too much manpower.

There are currently solutions, but all have limitations for use in normal video sequences. For example, one method provides a depth-of-view system that requires computer interaction, but this method does not fully realize unattended supervision for 3D television applications, none. Another way is to simulate the stereoscopic video difference using the motion visual difference assuming that the horizontal motion is performed on the object in the image and the background is stopped, but the above assumption can not be real time in normal video, There is a limit to processing.

Embodiments of the present invention provide a method and apparatus for fully processing various types of video (including still image sequences) without user input.

According to an embodiment of the present invention, there is provided an image processing apparatus including: an image obtaining unit that obtains a plurality of two-dimensional images that are continuous in time among input images; Dimensional structure of the current two-dimensional image; a three-dimensional structure matching unit for calculating each degree of matching of the three-dimensional stereoscopic structure and determining the three-dimensional stereoscopic structure of the highest matching degree as a three- A depth map having a three-dimensional typical structure of a three-dimensional structure of a current two-dimensional image is set as a depth map based on matching corresponding to a current two-dimensional image, and each pixel of the depth map based on the matching is a corresponding And a depth-map-based matching unit for displaying a depth value based on the matching of the pixels.

The depth value due to the match is in the range [0, 1], 0 indicates that the corresponding pixel has the maximum depth, and 1 indicates that the corresponding pixel has the minimum depth.

Wherein the three-dimensional structure matching unit comprises: a plane splitting module that divides the current two-dimensional image into at least one region corresponding to a plane of the matched three-dimensional representative structure; Calculating a depth of each of the regions based on a characteristic distribution of each region, calculating an average value of the characteristics of the regions, calculating a similarity between the two regions based on a norm of a difference between the average values, A matching degree calculating module for calculating a matching degree by a sum of the density of the two regions and the similarity between the two regions; And a three-dimensional structure determination module for determining the three-dimensional representative structure having the highest matching degree according to the degree of matching to the three-dimensional structure of the current two-dimensional image.

에 따라 각 상기 영역 r의 밀도를 계산하고, 여기서

이고, p는 상기 영역의 픽셀이며, I(p)는 픽셀 p의 특성값이고,

는 상기 영역의 상기 픽셀의 특성값의 평균값이며, area(r)는 상기 영역 중 픽셀의 수량이다.The matching degree calculation module

The density of each of the regions r is calculated according to Equation

P is a pixel of the area, I (p) is a characteristic value of the pixel p,

Is an average value of the characteristic values of the pixels in the area, and area (r) is the number of pixels in the area.

에 따라 영역 ri와 영역 rj 사이의 유사성을 계산하고, 여기서

는 상기 영역 중 특성의 평균값이고, |.|는 놈(norm)이다.The matching degree calculation module

The similarity between the region ri and the region rj is calculated according to Equation

Is the average value of the properties in the region, and | .vertline. Is the norm.

The characteristic is a color, a gradient, or a boundary.

The nucleus is a 1-nucleotide, a 2-nucleotide, or an ∞ nucleotide.

According to another embodiment of the present invention, there is provided an image processing apparatus including: an image obtaining unit obtaining a plurality of two-dimensional images that are continuous in time among input video; Dimensional image; and generating at least one saliency map corresponding to a current two-dimensional image of the plurality of two-dimensional images in accordance with the HVP model, wherein each pixel of the saliency map has a saliency A map generator; Dimensional image and a depth map based on the saliency corresponding to the current two-dimensional image using the at least one saliency map, and each pixel of the depth map based on the saliency displays a depth value of the saliency of the corresponding pixel of the current two- A depth map generation unit based on the conspicuousness; Dimensional image and a plurality of pre-stored three-dimensional representative structures among the plurality of two-dimensional images, and determining the three-dimensional representative structure having the highest matching degree as the three-dimensional structure of the current two- Dimensional structure matching section; Dimensional depth of the three-dimensional specimen structure is previously stored, and the depth map having the typical structure of the three-dimensional structure of the determined current two-dimensional image is regarded as an image based on the matching corresponding to the current two- Each pixel of the map representing a matched depth value of a corresponding pixel of a current two-dimensional image; An overall depth map generation unit for generating an overall depth map by combining the depth map based on the conspicuousness and the depth map based on matching and displaying each pixel of the comprehensive depth map an overall depth value of a corresponding pixel of the current two- And a depth map generating unit for generating a depth map.

Wherein the saliency map generator comprises: a feature saliency map generation module for generating a saliency saliency map by identifying a characteristic of a current two-dimensional image; a motion detection unit for detecting a motion between a current two- A motion aberration map generation module for generating an aberration map based on the motion; An object saliency map generation module that generates a target saliency map by identifying an object of a current two-dimensional image; An odd map control module for generating one, any two or all of the conspicuousness maps using any or all of the characteristic saliency map generation module, the motion saliency map generation module and the target saliency map generation module, ; ≪ / RTI >

Wherein the depth map generation unit based on the saliency generates a depth map based on the saliency through the following process: if the saliency map generator generates only the target saliency map, Assigning a constant value in the range to a pixel corresponding to a pixel identified as an object of the two-dimensional image in the depth map based on the sensibility, and assigning 0 to other pixels in the depth map based on the sensibility; [0, 1] according to the conspicuousness of each pixel in the characteristic saliency map or the motional saliency map, the depth map generation unit based on the saliency may generate [0, 1] A value in the range is assigned to each pixel in the depth map based on the conspicuity; 0 indicates that the corresponding pixel has minimum conspicuity; 1 indicates that the corresponding pixel has maximum conspicuity; The depth map generator based on the saliency maps adds the corresponding pixels in the two saliency maps to each other to obtain a normalized value or a relatively large value To a corresponding pixel in the depth map based on conspicuity; Wherein the depth map generation unit based on the saliency maps the constants in the range of (0, 1) to the target in the target salience map in the depth map based on the saliency, when the two saliency maps including the saliency map as the target of the saliency map generation unit are generated To a pixel corresponding to each of the pixels identified as " a ", and assigns corresponding pixel values of the saliency maps other than the saliency map of the object among the two saliency maps to other corresponding pixels in the depth map based on saliency; Wherein when the saliency map generating section generates the entire saliency map, the depth map generating section based on the saliency corresponds to each pixel identified as an object in the target saliency map in the depth map based on the saliency, with a constant in the range of (0, 1) And adds the corresponding pixels among the two conspicuous maps other than the target conspicuousness map to the standardized value or the relatively large value to the corresponding pixels of the depth map based on the conspicuousness.

The pixel values of the depth map based on the depth map and matching based on the conspicuity are [0. 1], where 0 indicates that the corresponding pixel has the maximum depth, and 1 indicates that the corresponding pixel has the minimum depth.

The comprehensive depth map generation unit may select a relatively large value among the corresponding pixels of the depth map based on the depth map based on the conspicuousness and the value normalized by adding the corresponding pixels of the depth map based on the consistency, .

The object in the current two-dimensional image may include a person, a face, or a character.

According to another embodiment of the present invention, there is provided an image processing apparatus including: an image obtaining unit obtaining a plurality of two-dimensional images that are continuous in time among input video; Acquiring an initial depth map corresponding to each two-dimensional image among the plurality of input two-dimensional images, wherein each pixel value in the initial depth map is a depth value of a corresponding pixel in the corresponding two-dimensional image; And a space-time area smoothing unit for smoothing the space in the time domain with respect to the initial depth map.

(X + y (t)) at a time (t +? T) in each pixel P1 (x, y, t) of the current two-dimensional image at time t, based on the HVP model, the smoothness values S (P1, P2) are calculated according to the similarity, distance and depth difference between the values of x, y and t + Δt according to the expected smoothing effect, A smoothing amount calculation module for determining the smoothing amount; Calculates a depth value D '(P1) = D (P1) -S (P1) of the pixel P1 of the current two-dimensional image after smoothing according to the smoothness amount S (P1, P2) The difference between the depth value of the pixel P1 after smoothing and the depth value D '(P2) = D (P2) + S (P1, P2) of the pixel P2 The smoothing module making the absolute value smaller than the absolute value of the difference between the depth value D (P1) of the pixel value P1 before smoothing and the depth value D (P2) of the pixel value P2 .

The smoothed amount calculation module calculates the smoothed amount S (P1, P2) according to D (P1) -D (P2) * N (P1, P2) * C .) Is the pixel is the depth value;

이며;here,

;

I (.) Is the characteristic (color or pattern) value of the pixel, and |. | Is the absolute value.

According to another embodiment of the present invention, there is provided an image processing apparatus including: an image obtaining unit obtaining a plurality of two-dimensional images that are continuous in time among input video; Dimensional image; and generating at least one saliency map corresponding to a current two-dimensional image of the plurality of two-dimensional images according to an HVP model, wherein each pixel of the saliency map includes a salient feature indicative of saliency of a corresponding pixel of a current two- A sex map generation unit; Dimensional image; and a depth map based on the saliency of a corresponding pixel in the current two-dimensional image, wherein the depth map is based on the saliency based on the at least one saliency map, A depth map generation unit based on the conspicuousness; Dimensional image and a plurality of three-dimensional structural matching modules among a plurality of the two-dimensional images and calculating a degree of matching of the three-dimensional representative structure stored in advance among a plurality of the three-dimensional structural matching modules, A three-dimensional structure matching unit fixed by a three-dimensional structure; Dimensional structure is previously stored and a depth map of a three-dimensional structure of a three-dimensional structure determined as a current two-dimensional image is used as a depth map based on a matching degree corresponding to a current two-dimensional image, Wherein each pixel of the map represents a depth value based on a matching degree of a corresponding pixel of a current two-dimensional image; An overall depth map generating unit for generating an overall depth map by combining the depth map based on the conspicuousness and the depth map based on matching and displaying each pixel in the depth map an overall depth value of a corresponding pixel in the current two- And a space-time area smoothing unit for smoothing the space area and the time area with respect to the overall depth map.

According to another embodiment of the present invention, there is provided an image processing method comprising: obtaining a plurality of consecutive two-dimensional images of an input video; Calculating a degree of matching of a current two-dimensional image and a plurality of pre-stored three-dimensional representative structures among the plurality of two-dimensional images, and determining a three-dimensional representative structure having a highest matching degree as a three- ; The depth map of the three-dimensional stereoscopic structure is stored in advance and the depth map of the three-dimensional stereoscopic structure determined by the three-dimensional structure of the current two-dimensional image is used as the depth map based on the matching corresponding to the current two- And each pixel of the due depth map displays a depth value based on a matching degree of a corresponding pixel of the current two-dimensional image.

Determining the current three-dimensional structure of the two-dimensional image comprises: dividing the current two-dimensional image into at least one region corresponding to a plane in the matched three-dimensional representative structure; Calculate a density of each of the regions according to a distribution of the characteristics of the regions; Calculating an average value of the characteristics of each of the regions and calculating a similarity between the two regions according to a difference between the average values; Calculating a matching degree according to a sum of the density of each of the regions and a similarity between the two regions; And fixing the three-dimensional template structure having the highest matching degree to the three-dimensional structure of the current two-dimensional image according to the matching degree.

에 따라 상기 각 영역 r의 밀도를 계산하고, 여기서 여기서

이고, p는 상기 영역의 픽셀이며, I(p)는 픽셀 p의 특성값이고,

는 상기 영역의 상기 픽셀의 특성값의 평균값이며, area(r)는 상기 영역 중 픽셀의 수량이다.

Lt; RTI ID = 0.0 > r, < / RTI >

P is a pixel of the area, I (p) is a characteristic value of the pixel p,

Is an average value of the characteristic values of the pixels in the area, and area (r) is the number of pixels in the area.

를 근거로 영역 ri와 영역 rj 사이의 유사성을 계산하고, 여기서

는 상기 영역의 특성의 평균값이며, |.|는 놈이다.

, The similarity between the region ri and the region rj is calculated, where

Is an average value of the characteristics of the region, and |

According to another embodiment of the present invention, there is provided an image processing method comprising: obtaining a plurality of consecutive two-dimensional images of an input video; Generating at least one saliency map corresponding to a current two-dimensional image of the plurality of two-dimensional images according to an HVP model, each pixel of the saliency map displaying saliency of a corresponding pixel of a current two- ; Wherein the depth map is based on the at least one saliency map and the saliency map corresponding to the current two-dimensional image, and each pixel of the depth map based on the saliency displays a depth value based on the saliency of the corresponding pixel of the current two- step; Calculating a degree of matching of a current two-dimensional image and a plurality of pre-stored three-dimensional representative structures among the plurality of two-dimensional images, and determining a three-dimensional representative structure having a highest matching degree as a three- ; The depth map of the three-dimensional stereoscopic structure is stored in advance and the depth map of the three-dimensional stereoscopic structure determined by the three-dimensional structure of the current two-dimensional image is used as the depth map based on the matching corresponding to the current two- Each pixel of the due depth map displaying a depth value based on a matching degree of a corresponding pixel of a current two-dimensional image; And combining the depth map based on the roughness-based depth map and the matching depth-based map to generate a comprehensive depth map, wherein each pixel in the overall depth map displays an overall depth value of a corresponding pixel in the current two- Provides a map generation method.

Generating the saliency map includes: generating one or any two or all of the characteristic saliency map, the motion saliency map and the target saliency map, generating a characteristic saliency map through the characteristics in the current two-dimensional image Generating an motion saliency map by identifying a motion between the current two-dimensional image and a two-dimensional image that is temporally adjacent to the current two-dimensional image, and generating an object saliency map by identifying an object in the current two-dimensional image .

The generating of the depth map based on the conspicuousness may include: generating only the conspicuity map of the object, the constant value within the range of (0, 1) is converted into the pixel corresponding to the pixel identified as the object in the two- And assigns 0 to other pixels of the depth map based on the conspicuousness; A value within the range of [0, 1] is assigned to each pixel of the depth map based on the conspicuousness according to the conspicuousness of each pixel of the characteristic saliency map or the motional saliency map. , 0 indicates that the corresponding pixel has minimum saliency, and 1 indicates that the corresponding pixel has maximum saliency; Adding a corresponding normalized value or a relatively large value to a corresponding pixel in the depth map based on the conspicuousness by adding corresponding pixels in the two conspicuous maps to each other when the two conspicuousness maps not including the conspicuousness map of the object are generated; When the two conspicuity maps including the conspicuity map of the object are generated, the constants in the range of (0, 1) are given to the pixels corresponding to the pixels identified as the objects of the object conspicuity map in the depth map based on conspicuity, The corresponding pixel values of the consonantity maps other than the consonantity map of the object among the consonantity maps to the other corresponding pixels of the depth map based on the conspicuousness; When the entire saliency map is generated, the constants in the range of (0, 1) are given to the pixels corresponding to the respective pixels identified as the targets of the target saliency map in the depth map based on the saliency, Adding corresponding pixels of the sex map to each other to give a normalized value or a relatively large value to a corresponding pixel in the depth map based on the conspicuousness.

Determining the three-dimensional structure of the current two-dimensional image comprises: dividing the current two-dimensional image into at least one region corresponding to a plane of the matched three-dimensional representative structure; Calculating a density of each of the regions according to a characteristic distribution among the regions; Calculating an average value of the characteristics among the areas, calculating a similarity between the two areas according to a difference between the average values; Calculating a matching degree according to a sum of the density of each of the regions and a similarity between the two regions; And determining the three-dimensional representative structure having the highest matching degree as the three-dimensional structure of the current two-dimensional image according to the matching degree.

It is possible to generate a comprehensive depth map by selecting a relatively large value among the corresponding pixels in the depth map based on the depth map and the corresponding pixel values in the depth map based on the conspicuousness or the corresponding depth map based on the conspicuousness.

According to another embodiment of the present invention, there is provided an image processing method comprising the steps of: acquiring a plurality of two-dimensional images that are sequential in time of an input video; Acquiring a first depth map corresponding to each two-dimensional image of the plurality of input two-dimensional images, wherein each pixel value in the depth map is a depth value of a corresponding one of the corresponding two-dimensional images; And advancing smoothness in a spatial domain and a time domain with respect to the initial depth map.

Wherein the step of advancing the spatial domain and the smoothing over the temporal domain with respect to the initial depth map comprises: based on the HVP model, for each pixel P1 (x, y, t) and time (t + (P1, P2) in accordance with the difference in similarity, distance and depth value between the pixel P2 (x + x, y + y, t + Determining Δx, Δy and Δt according to the smoothing effect; Calculates a depth value D '(P1) = D (P1) -S (P1) of the pixel P1 of the current two-dimensional image after smoothing according to the smoothness amount S (P1, P2) The difference between the depth value of the pixel P1 after smoothing and the depth value D '(P2) = D (P2) + S (P1, P2) of the pixel P2 Making the absolute value smaller than the absolute value of the difference between the depth value D (P1) of the pixel value P1 before smoothing and the depth value D (P2) of the pixel value P2 .

Calculating the smoothed amount S (P1, P2) according to D (P1) -D (P2) * N (P1, P2) * C Value;

이며;here,

;

I (.) Is the characteristic (color or pattern) value of the pixel, and |. | Is the absolute value.

According to another embodiment of the present invention, there is provided an image processing method comprising: obtaining a plurality of consecutive two-dimensional images of an input video; Generating at least one saliency map corresponding to a current two-dimensional image of the plurality of two-dimensional images according to an HVP model, each pixel of the saliency map displaying saliency of a corresponding pixel of a current two- ; Using the at least one saliency map and a depth map based on the saliency corresponding to the current two-dimensional image, each pixel of the depth map based on the saliency displays a depth value based on the saliency of a corresponding pixel in the current two- step; Calculating a degree of matching of a current two-dimensional image and a plurality of pre-stored three-dimensional representative structures among the plurality of two-dimensional images, and determining a three-dimensional representative structure having a highest matching degree as a three- ; The depth map of the three-dimensional stereoscopic structure is stored in advance and the depth map of the three-dimensional stereoscopic structure determined by the three-dimensional structure of the current two-dimensional image is used as the depth map based on the matching corresponding to the current two- Each pixel of the due depth map displaying a depth value based on a matching degree of a corresponding pixel of a current two-dimensional image; And combining the depth map based on the saliency and the depth map based on the matching to generate a comprehensive depth map, wherein each pixel in the overall depth map displays an overall depth value of a corresponding pixel in the current two-dimensional image; And advancing smoothness on the spatial domain and the time domain on the global depth map.

According to an embodiment of the present invention, various types of video can be processed completely automatically without user's input.

1 is a block diagram of an apparatus for generating a depth map according to a first embodiment of the present invention;
FIG. 2 is a schematic view showing an example of a three-dimensional typical structure; FIG.
3 is a block diagram of a three-dimensional structure matching module according to the present invention;
4 is a block diagram of an apparatus for generating a depth map according to a second embodiment of the present invention;
5 is a block diagram of a saliency map generation module according to the present invention;
FIG. 6 illustrates an example of a depth map generated using an apparatus according to a second embodiment of the present invention; FIG.
FIG. 7 is a block diagram of an apparatus for smoothing a depth map according to a third embodiment of the present invention; FIG.
8 is a block diagram of a spatial domain and time domain smoothing module of the present invention;
9 is a block diagram of a spatial domain and time domain smoothing module according to the present invention;
10 is a block diagram of a depth map generating apparatus according to a fourth embodiment of the present invention;
11 is a view showing an example of generating a depth map according to the apparatus of the fourth embodiment of the present invention;
12 is a flowchart of a method of generating a depth map according to a fifth embodiment of the present invention;
13 is a flowchart for determining a three-dimensional structure of a current two-dimensional image of the present invention;
14 is a flowchart of a method of generating a depth map according to a sixth embodiment of the present invention;
15 is a flowchart of a method of advancing smoothness for a depth map according to a seventh embodiment of the present invention;
16 is a flowchart of a method of generating a depth map according to an eighth embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail. The examples are shown in the figures, and the same reference numerals are used for the same components in different videos. Where necessary, redundant description of the same reference numerals is omitted.

FIG. 1 shows an apparatus for generating a depth map according to a first embodiment of the present invention.

Referring to FIG. 1, the depth map generation apparatus 100 includes an image acquisition unit 110, a three-dimensional structure matching unit 120, and a depth-of-match map generation unit 130.

The input of the depth map generating apparatus 100 consists of a video sequence composed of a plurality of images. The image obtaining unit 110 obtains a plurality of two-dimensional images that are continuous in time among the input video. For each image of the plurality of two-dimensional images, the three-dimensional structure mating section 120 obtains a three-dimensional structure that best matches the current image among a plurality of pre-stored three-dimensional representative structures.

Specifically, a similar three-dimensional structure of the current image can be obtained through application to the knowledge of the pre-knowledge of the three-dimensional model structure. A series of examples of three-dimensional representative structures are shown in Fig.

Referring to Fig. 2, the second row shows an example of a three-dimensional stereoscopic structure, and the first row shows an image of a corresponding real scene. In reality, the actual structure of the scene is more complicated than the pre-stored 3D stereoscopic structure, but due to the boundaries of the visual system of the human eye, a complex 3D structure can give viewers various 3D sensations when viewed on a 3D television , A simple three-dimensional structure can generate a depth map of a two-dimensional image, giving viewers more three-dimensional feeling than a conventional two-dimensional video.

In order to obtain a three-dimensional structure that best matches the current image in a plurality of pre-stored three-dimensional representative structures, it is necessary to calculate each matching degree of the current image and a plurality of previously stored three-dimensional representative structures, Dimensional structure of the current two-dimensional image as a three-dimensional structure of the current two-dimensional image.

3 shows a three-dimensional structure matching unit 120 according to the present invention.

3, the three-dimensional structure matching unit 120 includes a plane division module 121, a matching degree calculation module 122, and a three-dimensional structure determination module 123. [

The plane splitting module 121 splits the current image into at least one plane according to one of a plurality of previously stored three-dimensional representative structures. For example, when matching the current image with the first three-dimensional representative structure of the second row of FIG. 2, the three-dimensional representative structure has only one plane, so the entirety of the current image is one region; However, when matching the current image with the fourth three-dimensional structure of the second row of FIG. 2, it is necessary to divide the current image into four regions corresponding to four planes of the four-dimensional three-dimensional representative structure.

Then, the matching degree calculation module 122 calculates the density of each area and the similarity between the two areas through the characteristics (color, gradient, boundary) of each area of the current image and calculates the matching degree S ). (Ri, rj) is the density of each region calculated according to the property of the region, and diff (ri, rj) is the density of the region (similarity between regions calculated according to feature).

The calculation of Dense (ri) is shown in Equation (2) below.

Std (ri) in the equation (2) is a standard distribution of the characteristics in the area ri and is expressed by the following equation (3).

는 영역(ri) 중 픽셀의 특성값의 평균값이며, area(r)은 영역(ri)의 픽셀의 수량이다.In Equation (3), p is a pixel in the region ri, I (p) is a characteristic value of the pixel (p)

(R) is the average value of the pixel values in the region (ri), and area (r) is the number of pixels in the region (ri).

The calculation of Diff (ri, rj) is shown in Equation (4) below.

는 영역 중 픽셀의 특성값의 평균값이고, |.|는 놈(norm)이며, 1-놈

, 2-놈

이며, ∞놈은

이다.In Equation 4,

Is the average value of the characteristic values of the pixels in the region, |. | Is the norm,

, 2-man

And ∞

to be.

Also, Dense (ri) is the density of the characteristic in the region, and the more dense the characteristic of the region, the larger the value of Dense (ri). Diff (ri, rj) is the similarity between the two regions. The larger the difference between the two regions, the larger the diff (ri, rj) value. Therefore, the relatively high matching degree S means that the characteristics in each of the image areas after division have consistency and uniqueness.

In general, each three-dimensional plane has a matched image characteristic, and two different three-dimensional planes have different characteristics. Therefore, when dividing the current image according to different three-dimensional stereoscopic structures and calculating the respective degrees of matching, the higher the arbitrary degree of matching obtained, the more the matching between the current image and the arbitrary three- it means. Accordingly, the three-dimensional structure determination module 123 determines the three-dimensional structure having the highest matching degree according to the degree of matching to the three-dimensional structure of the current two-dimensional image.

Then, the depth map generation unit 130 based on registration generates a depth map based on the matching of the current image according to the determined three-dimensional structure. Each pixel in the depth map due to the matching is in the range [0, 1] and displays the depth map of the corresponding pixel in the current two-dimensional image.

The present embodiment shows a comparatively good effect on an image occupying a main portion of a scene.

4 is an apparatus for generating a depth map according to a second embodiment of the present invention.

4, the depth map generating apparatus 400 includes an image obtaining unit 410, a saliency map generating unit 420, a depth map generating unit 430, a three-dimensional structure matching unit 120, A matching depth based map generating unit 130, and a global depth map generating unit 440. [ The three-dimensional structure matching unit 120, the depth-based depth map generation unit 130, and the module of the same reference numeral in FIG.

The input of the depth map generating apparatus 400 is a video sequence composed of a plurality of images. The image obtaining unit 410 obtains a plurality of two-dimensional images continuously in time among the input video.

From the HVP (Human Visual Perception) model, the viewer feels more interested in the portion of the video that has saliency, and the portion having the saliency generally feels closer to the viewer (i.e., . Therefore, it is possible to identify a characteristic, a motion, or an object having a conspicuousness of a two-dimensional image, and calculate a depth value between [0, 1] according to the identified characteristic, motion, A depth map based on the conspicuousness can be obtained.

In order to identify characteristics, motions, or objects having salience in the image, the saliency map section 420 includes a characteristic saliency generating module 421, a motion saliency map generation module 422, a target saliency map generation module 423 And an affinity map control module 424, as shown in Fig.

The characteristic saliency map generation module 421 identifies characteristics such as the color, gradient, or boundary characteristic of the two-dimensional image. For example, use the Solbe operator or the Prewitt operator to identify gradient properties, and use the Laplacian boundary detection computation to identify boundary characteristics in the image.

The motion aberration map generation module 422 generates an motion aberration map by identifying motion between two images that are adjacent in time.

The target saliency map generation module 423 generates a target saliency map by identifying an object (person, face, or character) in the image, and uses the target identification model based on, for example, a boosting calculation method, And face.

The saliency map control module 424 may use one or both of the characteristic saliency map generation module, the motion saliency map generation module 422 and the target saliency map generation module 423 to generate one, Thereby generating two or all of the saliency maps.

For example, if the image includes a very large number of people in the two-dimensional image in the video sequence, it includes a target among the modules used by the saliency map control module 424 module 423. If the two-dimensional image in the video sequence does not include a person, a face or a character but contains a relatively large number of motions, the target conspicuousness map generation module 423 among the modules used by the conspicuity map control module 424 And includes motion awareness map generation module 422 and the like. That is, the three kinds of modules are used according to the two-dimensional image sequence itself to be processed.

If only the target saliency map is generated, then the depth map generation unit 430 based on the saliency property sets a constant value (e.g. 0.8) within the range of (0, 1) to a pixel identified as the target among the two- And 0 is given to other pixels in the depth map based on the conspicuousness.

If one of the characteristic saliency map or the motion aurorance map is generated, the depth map generation unit 430 based on the saliency maps the value within the range [0, 1] according to the saliency of each pixel in the characteristic saliency map or the motion semi- To each pixel in the depth map based on the conspicuousness. 0 indicates that the corresponding pixel has the minimum saliency, and 1 indicates that the corresponding pixel has the maximum saliency.

For example, [0, 1] ranges depending on the difference between the characteristic values of the center pixel or the center block calculated according to each position in the image at different sizes and the characteristic average value of pixels adjacent to the upper, lower, left, Is assigned to each pixel in the depth map based on the conspicuousness.

For example, assuming that a color characteristic is used to generate a characteristic saliency map, the color characteristic is a (R, G, B) vector. First, calculate and record the difference between the vector of (R, G, B) of a single pixel and the average value of the (R, G, B) vector of neighboring upper, lower, left, and right adjacent pixels; Then, the size is enlarged to 4 * 4 blocks, and the average of the (R, G, B) vector quantities of 16 pixels of each 4 * 4 block is added to calculate the average (referred to as the adjacent average) Calculate the difference between the averages; Subsequently, the difference between the center average of the 8 * 8 block and the adjacent average is computed. Continue the above operation to enlarge the size to the whole image and execute it; Finally, differences in all sizes of each pixel are added together and normalized within the range [0, 1] to obtain a depth map based on conspicuousness.

The depth map generating unit 430 based on the conspicuousness adds the corresponding pixels in the generated two conspicuous maps to the standardized value or the relatively large value as the remarkableity map, To the corresponding pixels in the depth map based on the sex;

The depth map generating unit 430 based on the conspicuity generates a consonant property map including a target conspicuity map and a constant (e.g., 0.8) within the range of (0, 1) To the pixels corresponding to the pixels identified as the respective objects in the depth map, and the corresponding pixel values in the saliency maps other than the target saliency map among the two saliency maps are given to the other corresponding pixels in the depth map based on the saliency.

When the entire saliency map is generated, the depth map generation unit 430 based on the saliency property sets a constant (e.g., 0.8) in the range (0, 1) to a pixel identified as each target in the target saliency map in the depth map based on the saliency And the corresponding pixels in the two conspicuousness maps other than the object conspicuousness map are added to each other to give a normalized value or a relatively large value to the corresponding pixel in the depth map based on the conspicuousness.

The comprehensive depth map generation unit 440 normalizes the depth map based on the conspicuousness by adding the corresponding pixel values in the depth map matched with the conspicuousness or selects a relatively large value among the corresponding pixels in the depth map based on the depth map based on the conspicuousness, Create a map.

Fig. 6 is an effect diagram of generating a comprehensive depth map by combining the depth map based on the conspicuousness and the depth map based on matching of the present invention.

FIG. 7 shows an apparatus 700 for smoothing the depth map according to the third embodiment of the present invention.

Referring to FIG. 7, an apparatus 700 for smoothing a depth map includes an image obtaining unit 710, a depth map obtaining unit 720, and a construction area smoothing unit 730.

The image acquiring unit 710 acquires a plurality of two-dimensional images that are consecutive in time among a video sequence composed of a plurality of input images, and the depth map acquiring unit 720 acquires a depth map of each corresponding two- The initial depth map can be obtained, and each pixel value in the initial image is the depth value of the corresponding pixel in the corresponding two-dimensional image.

According to the HVP model, the human eye is sensitive to a large depth variation of the boundary position in the image, and if there is a rapid depth variation between adjacent frames, the viewer becomes dizziness. Accordingly, a depth map is created that smoothes the spatial region and the time region with respect to the initial depth map, thereby making the viewer comfortable.

8 shows a construction area smoothing part 730 according to the present invention.

Referring to FIG. 8, the construction area smoothing unit 730 includes a smoothing amount calculation module 731 and a smoothing module 732.

FIG. 9 is an example of progressing the spatial domain and time domain smoothing according to the present invention.

8 and 9, the smoothed amount calculation module 731 calculates a smoothed amount of a pixel P1 (x, y, t) in the frame t and a pixel P2 (x + x, y + ? Y, z +? Z) (e.g., color or pattern). If the design of the values of DELTA x, DELTA y and DELTA t is too large, the smoothing becomes too much; Conversely, if the design of the values of DELTA x, DELTA y and DELTA t is too small, the smoothing effect does not appear. Therefore, the values of DELTA x, DELTA y and DELTA t are determined according to the expected smoothing effect to be obtained, and a comparatively appropriate smoothing effect can be obtained, for example, DELTA x = 5, DELTA y = 5 and DELTA t =

According to the HVP model, for example, when the color is used as a characteristic, if the colors of P1 and P2 are similar, the depths of P1 and P2 are similar. Therefore, by adjusting the depths of P1 and P2 according to the colors of P1 and P2, the absolute value of the difference between the depth value D (P1) of the pixel P1 after smoothing and the depth value D (P2) The depth value D (P1) of the pixel P2 and the depth value D (P2) of the pixel P2.

The smoothed amount calculation module 731 calculates the smoothed amount S according to the following equation (5).

In Equation 5, D (.) Is the depth value of the pixel. C (P1, P2) is the difference (i.e., similarity) between the characteristic values of pixels P1 and P2 and N (P1, P2) is the difference between P1 and P2 calculated according to (x, y, It is a street. C (p1, P2) and N (P1, P2) can be calculated according to the following equations (6) and (7).

In Equation 6, I (.) Is the characteristic value of the pixel, and | .vertline. Is the absolute value.

이다.In Equation (7)

to be.

을 계산한다.The smoothing module 732 corrects the depth value of the current two-dimensional image pixel P1 after smoothing according to the smoothed amount S (P1, P2)

.

Each pixel of the current two-dimensional image can obtain the depth map of the current two-dimensional image after smoothing by applying the functions of the smoothing amount calculation module 731 and the smoothing module 732. [

FIG. 10 shows an apparatus 1000 for generating a depth map according to a fourth embodiment of the present invention.

The depth map generation apparatus 1000 includes an image acquisition unit 1010, a saliency map generation unit 420, a depth map generation unit 430, a three-dimensional structure matching unit 120, A total depth map generating unit 440, and a construction area smoothing unit 730. [0033]

The saliency map generation unit 420 and the saliency depth map generation unit 430 are the same as the modules of the same reference numerals shown in Fig. The three-dimensional structure matching unit 120 and the depth-based depth map generation unit 130 are the same as those shown in FIG. The construction area smoothing unit 730 and the module of the same reference numeral shown in Fig. 7 are the same.

The image acquisition device 1010 acquires a plurality of two-dimensional images that are sequential in time among the input video.

11 is an example of a depth map generated by the depth map generating apparatus 1000 according to the present invention. The depth map according to the present invention has a relatively good effect.

12 is a flowchart of a depth map generation method based on matching according to the fifth embodiment of the present invention.

Referring to FIG. 12, step S1210 obtains a plurality of two-dimensional images continuous in time among the input video.

In operation S1220, the degree of matching between the current two-dimensional image and the previously stored three-dimensional representative structure is calculated, and the three-dimensional representative structure having the highest matching degree is determined as the three-dimensional structure of the current image. Although the pre-stored three-dimensional stereoscopic structure is generally simpler than the actual structure of the real scene, due to the characteristics of the visual system of the human eye, a simple three-dimensional stereoscopic structure can generate a depth map of the two- It can provide a much better three-dimensional feeling than 2D video.

In order to obtain a three-dimensional structure having the highest matching degree and a current image obtained from a plurality of pre-stored three-dimensional representative structures, a matching degree of the current two-dimensional image and a plurality of pre-stored three-dimensional representative structures must be calculated.

13 is a flowchart for calculating the matching degree according to the present invention.

Referring to FIG. 13, in step S1221, the current image is divided into at least one plane according to one of a plurality of previously stored three-dimensional representative structures. For example, if there is one plane in the three-dimensional stereoscopic structure matching the current image and in FIG. 2, the whole of the current image is taken as one divided area; However, in the case where the 3D stereoscopic structure having the matching of FIG. 2 with the current image has a plurality of planes, the current image must be divided into a plurality of regions corresponding to the respective planes of the 3D stereoscopic structure.

The step S1221 then uses the characteristics (color, gradient, or boundary) of each region of the current image to determine the density of each region and the density of each region between the two regions using Equations 1, 2, The similarity is calculated and the matching degree S is calculated. The higher the degree of matching S calculated through Equation 1, the better the match between the current image and its segmented based three-dimensional representative structure.

Therefore, in step S1225, the three-dimensional template structure having the highest matching degree is determined as the three-dimensional structure of the current image.

Referring again to FIG. 12, a depth map based on the matching of the current image is generated according to the three-dimensional structure determined in step S1230. Each pixel of the depth map based on the matching is in the range [0, 1] and represents the depth value of the corresponding pixel of the current two-dimensional image. 0 indicates that the corresponding pixel has the maximum depth, and 1 indicates that the corresponding pixel has the minimum depth. This embodiment produces a relatively good effect on the image of the main part pointing to the scene.

FIG. 14 is a flowchart of a depth map based on the conspicuousness according to the sixth embodiment of the present invention.

Referring to FIG. 14, a plurality of continuous two-dimensional images in time are acquired in the video inputted in step S1410.

In step S1420, one of the characteristic saliency map, the motion saliency map and the target saliency map, any two or all of the saliency maps are generated, and the characteristic (color, gradient, or boundary characteristic) Generating an aberrantness map by identifying a motion between two adjacent two-dimensional images in time, identifying an object (person, face, or character) of the two-dimensional image, .

If only the target saliency map is generated in step S1420, a constant value (e.g., 0.8) within the range of (0, 1) in step S1430 is compared with a pixel value And assigns 0 to other pixels in the depth map based on the conspicuousness.

If one of the characteristic saliency maps or the motional saliency maps is generated in step S1420, then in step S1430, a value within the range of [0, 1] according to the saliency of each pixel of the characteristic saliency map or the motion semblance map is set to Is given to each pixel in the depth map based on the conspicuousness. 0 indicates that the corresponding pixel has minimum saliency, and 1 indicates that the corresponding pixel has maximum saliency. For example, in the range [0, 1] according to the difference between the characteristic value of the center pixel or the central block calculated at each position of the image at different sizes, and the average characteristic value of the adjacent pixels in the upper, Value to each pixel of the saliency depth map.

If two saliency maps not including the target saliency map are generated in step S1420, a value obtained by normalizing the sum of the corresponding pixels in the two saliency maps or a relatively large value obtained by adding the corresponding pixels in the two saliency maps to the depth map based on the saliency To the corresponding pixel.

In step S1420, if two conspicuity maps including the target conspicuousness map are generated, then in step S1430, the constants in the range (0, 1) (e.g., 0.8) And assigns the corresponding pixel value of the sensibility map except the target saliency map of the two saliency maps to the other corresponding pixels of the depth map based on the saliency.

If the entire saliency map is generated in step S1420, a constant in the range (0, 1) (e.g., 0.8) in step S1430 corresponds to a pixel of each identified target of the target saliency map in the depth map based on saliency And adds a normalized value or a relatively large value to the corresponding pixel of the depth map based on the conspicuousness by adding the corresponding pixels of the two conspicuousness maps excluding the target conspicuousness map.

Step S1440 and step S1220 in Fig. 12 are the same, and step S1450 and step S1230 in Fig. 12 are the same.

In step S1460, the depth map based on the conspicuousness generated in step S1430 and the corresponding pixel value in the depth map based on the matching created in step S1450 are added to normalize or to the depth map based on the conspicuousness, Thereby generating a comprehensive depth map by selecting a relatively large value of the corresponding pixels.

15 is a flowchart of a method of smoothing a depth map according to a seventh embodiment of the present invention.

Referring to FIG. 15, a continuous two-dimensional image is acquired in time among the video sequences composed of the plurality of images input in step S1510.

The first depth map corresponding to each two-dimensional image input in step S1520 is obtained, and each pixel value in the initial depth map represents the depth value of the corresponding pixel in the corresponding two-dimensional image.

9, a pixel P1 (x, y, t) in the frame t and a pixel P2 (x +? X, y +? Y, t +? T ) (For example, color or pattern). If the design of the values of DELTA x, DELTA y and DELTA t is too large, the smoothing becomes too much; Conversely, if the design of the values of DELTA x, DELTA y and DELTA t is too small, the smoothing effect does not appear. Therefore, the values of DELTA x, DELTA y and DELTA t are determined according to the expected smoothing effect to be obtained, and a comparatively appropriate smoothing effect can be obtained, for example, DELTA x = 5, DELTA y = 5 and DELTA t = According to the HVP model, for example, when the color is used as a characteristic, if the colors of P1 and P2 are similar, the depths of P1 and P2 are similar. Therefore, by adjusting the depths of P1 and P2 according to the colors of P1 and P2, the absolute value of the difference between the depth value D '(P1) of the pixel P1 of smooth water and the depth value D' (P2) is smaller than the difference between the depth value D (P1) of the pixel P2 and the depth value D (P2) of the pixel P2.

The smoothing amount S is calculated according to Equations (5), (6), and (7).

Then, the depth value D '(P1) = D (P1) -S (P1) of the current two-dimensional image pixel P1 after smoothing is calculated according to the smoothed amount S (P1, P2).

Compute the smoothed amount (S) for each pixel of the current two-dimensional image, smooth it, and obtain a depth map of the current two-dimensional image after smoothing.

16 is a flowchart of a method of generating a depth map according to an eighth embodiment of the present invention.

Referring to FIG. 16, a continuous two-dimensional image in time among the video sequences composed of a plurality of images input in step S1610 is obtained. Steps S1620, S1630, S1640, S1650, and S1660 correspond respectively to steps S1420, S1430, S1440, S1450, (S1460). Steps S1670 and S1523 in Fig. 15 are the same.

Also, the depth map generation method according to an embodiment of the present invention may be implemented in the form of a program command that can be executed through various computer means and recorded in a computer readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like, alone or in combination. The program instructions recorded on the medium may be those specially designed and constructed for the present invention or may be available to those skilled in the art of computer software. Examples of computer-readable media include magnetic media such as hard disks, floppy disks and magnetic tape; optical media such as CD-ROMs and DVDs; magnetic media such as floppy disks; Magneto-optical media, and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of program instructions include machine language code such as those produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

While the invention has been shown and described with reference to certain preferred embodiments thereof, it will be understood by those of ordinary skill in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims. This is possible. Therefore, the scope of the present invention should not be limited to the described embodiments, but should be determined by the equivalents of the claims, as well as the claims.

100, 400, 1000 depth map generator
110, 410, 710, and 1010,
120 three-dimensional structure matching section
121 plane splitting module
122 Matching degree calculation module
123 3D Structure Confirmation Module
130 depth map generation section
420 < SEP >
421 Characteristic Remarkability Map Generation Module
422 Motion Existence Map Generation Module
423 Target Remarkability Map Generation Module
424 Remarkability map control module
430 depth map generation section
440 < / RTI >
700 Smoothing device
720 depth map acquisition unit
730 Construction area Smooth part
731 Smoothness Calculation Module
732 Smoothing module

Claims (73)

An image acquiring unit acquiring a plurality of two-dimensional images successive in time among the input video;
Dimensional image and a plurality of pre-stored three-dimensional representative structures among the plurality of two-dimensional images, and determining the three-dimensional representative structure of the highest matching degree as a three-dimensional structure of the current two- A structure matching section; And
Dimensional image of the three-dimensional structure of the current two-dimensional image is set as a depth map based on the matching corresponding to the current two-dimensional image, and the depth map of the three- Each pixel of the depth map based on the depth map represents a depth value based on the matching of the corresponding pixels of the current two-dimensional image;
Lt; / RTI >
Wherein the three-dimensional structure matching unit comprises:
A plane splitting module that divides the current two-dimensional image into at least one region corresponding to a plane of the matched three-dimensional representative structure;
Calculating a depth of each of the regions based on the characteristic distribution of each region, calculating average values corresponding to the characteristics of the regions, calculating a similarity between the two regions from a norm of a difference between the average values, A matching degree calculating module for calculating a matching degree by a sum of the density of each of the areas and a similarity between the two areas; And
A three-dimensional structure determination module for determining a three-dimensional representative structure having the highest matching degree according to the degree of matching to a three-dimensional structure of a current two-dimensional image;
And a depth map generation unit for generating depth map information. The method according to claim 1,
Wherein the depth value due to registration is in the range [0, 1], 0 indicates that the corresponding pixel has the maximum depth, and 1 indicates that the corresponding pixel has the minimum depth. delete The method according to claim 1,
The matching degree calculation module

, Where r is each of the regions,

P is a pixel of the area, I (p) is a characteristic value of the pixel p,

Is an average value of the characteristic values of the pixels in the area, and area (r) is a number of pixels in the area. The method according to claim 1,
The matching degree calculation module

The similarity between the region ri and the region rj is calculated according to Equation

Is a mean value of the characteristics in the region, and | is a normal depth map generator. 6. The method according to any one of claims 4 to 5,
Wherein the feature is a color, gradient, or boundary depth map. 6. The method of claim 5,
Wherein the genome is a 1-genome, a 2-genome, or a ∞ genome. An image acquiring unit acquiring a plurality of two-dimensional images successive in time among the input video;
Dimensional image; and generating at least one saliency map corresponding to a current two-dimensional image of the plurality of two-dimensional images in accordance with the HVP model, wherein each pixel of the saliency map has a saliency A map generator;
Dimensional image and a depth map based on the saliency corresponding to the current two-dimensional image using the at least one saliency map, and each pixel of the depth map based on the saliency displays a depth value of the saliency of the corresponding pixel of the current two- A depth map generation unit based on the conspicuousness;
Dimensional image and a plurality of pre-stored three-dimensional representative structures among the plurality of two-dimensional images, and determining the three-dimensional representative structure having the highest matching degree as the three-dimensional structure of the current two- Dimensional structure matching section;
Dimensional depth of the three-dimensional specimen structure is previously stored, and the depth map having the typical structure of the three-dimensional structure of the determined current two-dimensional image is regarded as an image based on the matching corresponding to the current two- Each pixel of the map representing a matched depth value of a corresponding pixel of a current two-dimensional image; And
An overall depth map generating unit for generating an overall depth map by combining the depth map based on the conspicuousness and the depth map based on the matching, and each pixel of the comprehensive depth map displaying the comprehensive depth value of the corresponding pixel of the current two- dimensional image;
Lt; / RTI >
Wherein the three-dimensional structure matching unit comprises:
A plane splitting module that divides the current two-dimensional image into at least one region corresponding to a plane of the matched three-dimensional representative structure;
Calculating a density of each of the regions according to a characteristic distribution of the regions; Calculating similarities between the two regions according to the difference between the average values by calculating average values corresponding to the characteristics of the regions; A matching degree calculation module for calculating a matching degree according to the density of each area and the similarity between the two areas; And
A three-dimensional structure determination module for determining a three-dimensional representative structure having the highest matching degree based on the matching degree to a three-dimensional structure of a current two-dimensional image;
And a depth map generation unit for generating depth map information. 9. The method of claim 8,
Wherein the saliency map generator comprises:
A characteristic saliency map generation module that generates a characteristic saliency map by identifying a characteristic of a current two-dimensional image;
A motion aberration map generation module that generates a motion aberration map by identifying a motion between a current two-dimensional image and a two-dimensional image that is temporally adjacent to a current two-dimensional image;
An object saliency map generation module that generates a target saliency map by identifying an object of a current two-dimensional image;
An odd map control module for generating one, any two or all of the conspicuousness maps using any or all of the characteristic saliency map generation module, the motion saliency map generation module and the target saliency map generation module, ;
And a depth map generation unit for generating depth map information. 10. The method of claim 9,
The depth map generation unit based on the conspicuousness generates a depth map based on the conspicuousness through the following processing:
When the saliency map generating unit generates only the target saliency map, the depth map generating unit based on the saliency corresponds to a pixel identified as a target of the two-dimensional image in the depth map based on the salience of a constant value within the range of (0, 1) And assigns 0 to other pixels in the depth map based on the conspicuousness;
[0, 1] according to the conspicuousness of each pixel in the characteristic saliency map or the motional saliency map, the depth map generation unit based on the saliency may generate [0, 1] A value in the range is assigned to each pixel in the depth map based on the conspicuity; 0 indicates that the corresponding pixel has the minimum saliency; and 1 indicates that the corresponding pixel has the maximum saliency;
The depth map generating unit based on the saliency map may compare the values obtained by normalizing the corresponding pixels in the two saliency maps to each other or by comparing them with each other Assigning a large value to a corresponding pixel in the depth map based on the conspicuousness;
Wherein the depth map generation unit based on the saliency maps the constants in the range of (0, 1) to the target in the target salience map in the depth map based on the saliency, when the two saliency maps including the saliency map as the target of the saliency map generation unit are generated To a pixel corresponding to each of the pixels identified as " a ", and assigns corresponding pixel values of the saliency maps other than the saliency map of the object among the two saliency maps to other corresponding pixels in the depth map based on saliency;
Wherein when the saliency map generating section generates the entire saliency map, the depth map generating section based on the saliency corresponds to each pixel identified as an object in the target saliency map in the depth map based on the saliency, with a constant in the range of (0, 1) And assigns a larger value to a corresponding pixel of the depth map based on the conspicuousness by comparing the values obtained by normalizing the corresponding pixels among the two conspicuous maps other than the target conspicuousness map or by normalizing them. delete 9. The method of claim 8,
The matching degree calculation module

, Where r is each of the regions,

P is a pixel of the area, I (p) is a characteristic value of the pixel p,

Is an average value of the characteristic values of the pixels in the area, and area (r) is a number of pixels in the area. 9. The method of claim 8,
The matching degree calculation module

The similarity between the region ri and the region rj is calculated according to Equation

Is a mean value of the characteristics of the region, and | 14. The method according to any one of claims 12 to 13,
Wherein the feature is a color, gradient, or limit. 14. The method of claim 13,
Wherein the genome is a 1-genome, a 2-genome, or a ∞ genome. 9. The method of claim 8,
The pixel values of the depth map based on the depth map and matching based on the conspicuity are [0. 1], where 0 indicates that the corresponding pixel has a maximum depth, and 1 indicates that the corresponding pixel has a minimum depth. 9. The method of claim 8,
The comprehensive depth map generation unit compares the depth map based on the conspicuousness with the corresponding pixels in the depth map based on the value obtained by standardizing the sum of the corresponding pixels of the depth map based on matching and the depth map based on the conspicuousness, A depth map generator for generating a depth map. 10. The method of claim 9,
Wherein the object in the current two-dimensional image includes a person, a face, or a character. An image acquiring unit acquiring a plurality of two-dimensional images successive in time among the input video;
Dimensional image and a plurality of pre-stored three-dimensional representative structures among the plurality of two-dimensional images, and determining the three-dimensional representative structure of the highest matching degree as a three-dimensional structure of the current two- A structure matching section;
Dimensional image of the three-dimensional structure of the current two-dimensional image is set as a depth map based on the matching corresponding to the current two-dimensional image, and the depth map of the three- Each pixel of the depth map based on the depth map represents a depth value based on registration of a corresponding pixel of the current two-dimensional image; And
A space-time area smoothing unit for smoothing the space map and the time space with respect to the depth map based on the matching;
Lt; / RTI >
Wherein the three-dimensional structure matching unit comprises:
A plane splitting module that divides the current two-dimensional image into at least one region corresponding to a plane of the matched three-dimensional representative structure;
Calculating a density of each of the regions according to a characteristic distribution of the regions; Calculating similarities between the two regions according to the difference between the average values by calculating average values corresponding to the characteristics of the regions; A matching degree calculation module for calculating a matching degree according to the density of each area and the similarity between the two areas; And
A three-dimensional structure determination module for determining a three-dimensional representative structure having the highest matching degree based on the matching degree to a three-dimensional structure of a current two-dimensional image;
The depth map smoothing device comprising: 20. The method of claim 19,
The space-time area smoothing unit includes:
(X + x, y + y, t + t) of the two-dimensional image at each pixel P1 (x, y, t) and time (t + (P1, P2) according to the difference in the similarity, distance, and depth values between the two values of Δx, Δy and Δt, and calculates the amount of smoothening to determine Δx, Δy and Δt according to the expected smoothing effect module;
Calculates a depth value D '(P1) = D (P1) -S (P1) of the pixel P1 of the current two-dimensional image after smoothing according to the smoothness amount S (P1, P2) The difference between the depth value of the pixel P1 after smoothing and the depth value D '(P2) = D (P2) + S (P1, P2) of the pixel P2 A smoothing module for making the absolute value smaller than the absolute value of the difference between the depth value D (P1) of the pixel value P1 before smoothing and the depth value D (P2) of the pixel value P2;
Wherein the depth map smoothing device further comprises: 21. The method of claim 20,
The smoothed amount calculation module calculates the smoothed amount S (P1, P2) according to (D (P1) -D (P2)) * N (P1, P2) * C (.) Is the pixel is the depth value;

here,

;

I (.) Is the characteristic (color or pattern) value of the pixel, and |. | Is the absolute value. 22. The method of claim 21,
Wherein the characteristic is a color or a pattern. An image acquiring unit acquiring a plurality of two-dimensional images successive in time among the input video;
Dimensional image, the at least one saliency map corresponding to a current two-dimensional image of the plurality of two-dimensional images in accordance with the HVP model, wherein each pixel of the saliency map includes a salient feature indicating the saliency of the corresponding pixel of the current two- A sex map generation unit;
Dimensional image; and a depth map based on the saliency of a corresponding pixel in the current two-dimensional image, wherein the depth map is based on the saliency based on the at least one saliency map, A depth map generation unit based on the conspicuousness;
Dimensional image and a plurality of pre-stored three-dimensional representative structures among the plurality of two-dimensional images, and determining the three-dimensional representative structure having the highest matching degree as the three-dimensional structure of the current two- Dimensional structure matching section;
Wherein the depth map of the three-dimensional stereoscopic structure having the three-dimensional structure determined as the current two-dimensional image is set as a depth map based on the matching degree corresponding to the current two-dimensional image, Wherein each pixel of the depth map represents a depth value based on a matching degree of a corresponding pixel of a current two-dimensional image;
An overall depth map generating unit for generating an overall depth map by combining the depth map based on the conspicuousness and the depth map based on matching and displaying each pixel in the depth map an overall depth value of a corresponding pixel in the current two- And
A space-time area smoothing unit for smoothing the space area and the time area with respect to the overall depth map;
Lt; / RTI >
Wherein the three-dimensional structure matching unit comprises:
A plane splitting module that divides the current two-dimensional image into at least one region corresponding to a plane of the matched three-dimensional representative structure;
Calculate a density of each of the regions based on a characteristic distribution of the regions; Calculating similarities between the two regions based on a difference between the average values by calculating average values corresponding to the respective characteristics; A matching degree calculation module for calculating a matching degree based on the density of each area and the sum of similarities between the two areas; And
A three-dimensional structure determination module for determining a three-dimensional representative structure having the highest matching degree based on the matching degree to a three-dimensional structure of a current two-dimensional image;
And a depth map generation unit for generating depth map information. 24. The method of claim 23,
Wherein the saliency map generator comprises:
A characteristic saliency map generation module that generates a characteristic saliency map by identifying a characteristic of a current two-dimensional image;
A motion aberration map generation module that generates a motion aberration map through a motion between a current two-dimensional image and a current two-dimensional image in time;
An object saliency map generation module that generates a target saliency map by identifying an object of a current two-dimensional image;
Wherein one or any two or all of one of the characteristic saliency map generation module, the motion aurorance map generation module and the target saliency map generation module is used to generate one or any two or all of the saliency maps Map control module;
And a depth map generation unit for generating depth map information. 25. The method of claim 24,
The depth map generation unit based on the conspicuousness generates a depth map based on the conspicuousness through the following processing:
If the saliency map generator generates only a saliency map of a target object, the depth map generator based on the saliency may set a constant value in the range of (0, 1) to a pixel identified as an object in the two- Assigns 0 to the corresponding pixels, and assigns 0 to other pixels in the depth map based on the conspicuousness;
[0, 1] according to the conspicuousness of each pixel of the characteristic saliency map or the motional saliency map, when the saliency map generator generates one of the characteristic saliency map or the motion aurora saliency map, A value in the range is assigned to each pixel in the depth map based on the conspicuity; 0 indicates that the corresponding pixel has minimum conspicuity; 1 indicates that the corresponding pixel has maximum conspicuity;
The depth map generating unit based on the saliency map may compare the values obtained by normalizing the corresponding pixels of the two saliency maps to each other or by comparing them with each other Assigning a large value to the corresponding pixel of the depth map based on the conspicuousness;
Wherein the depth map generator based on the saliency maps the constants in the range of (0, 1) to the angle of the target saliency map in the depth map based on the saliency, when the two saliency maps including the saliency map as the target of the saliency map generation unit are generated To a pixel corresponding to a pixel identified as an object, and assigns corresponding pixel values of the conspicuity maps other than the conspicuity map of the object among the two conspicuity maps to other corresponding pixels of the depth map based on conspicuity;
Wherein when the saliency map generating unit generates the entire saliency map, the depth map generating unit based on the saliency matches a constant within the range of (0, 1) with each pixel identified as the target of the target saliency map in the depth map based on saliency And adds the corresponding pixels of the two conspicuous maps other than the target conspicuousness map to each other to standardize the values or compares them with each other to give a larger value to the corresponding pixels of the depth map based on the conspicuousness. delete 24. The method of claim 23,
The matching degree calculation module

, Where r is each of the regions,

P is a pixel of the area, I (p) is a characteristic value of the pixel p,

Is an average value of the characteristic values of the pixels in the area, and area (r) is a number of pixels in the area. 28. The method of claim 27,
The matching degree calculation module

The similarity between the region ri and the region rj is calculated according to Equation

Is a mean value of the characteristics of the region, and | 29. The method according to any one of claims 27 to 28,
Wherein the feature is a color, gradient, or boundary depth map. 29. The method of claim 28,
Wherein the genome is a 1-genome, a 2-genome, or a ∞ genome. 24. The method of claim 23,
The pixel values of the depth map based on the depth map and matching based on the conspicuity are [0. 1], where 0 indicates that the corresponding pixel has a maximum depth, and 1 indicates that the corresponding pixel has a minimum depth. 24. The method of claim 23,
The comprehensive depth map generation unit compares the depth map based on the conspicuousness with the corresponding pixels of the depth map based on the sum of the corresponding pixels of the depth map based on the matching and the corresponding pixels of the depth map based on the matching, A depth map generator for generating a global depth map. 24. The method of claim 23,
Wherein the object in the current two-dimensional image includes a person, a face, or a character. 24. The method of claim 23,
The space-time area smoothing unit includes:
(X + x, y + y, t + t) of the two-dimensional image at each pixel P1 (x, y, t) and time (t + (P1, P2) according to the difference in the similarity, distance, and depth values between the two values of Δx, Δy and Δt, and calculates the amount of smoothening to determine Δx, Δy and Δt according to the expected smoothing effect module;
Calculates a depth value D '(P1) = D (P1) -S (P1) of the pixel P1 of the current two-dimensional image after smoothing according to the smoothness amount S (P1, P2) The difference between the depth value of the pixel P1 after smoothing and the depth value D '(P2) = D (P2) + S (P1, P2) of the pixel P2 A smoothing module for making the absolute value smaller than the absolute value of the difference between the depth value D (P1) of the pixel value P1 before smoothing and the depth value D (P2) of the pixel value P2;
The depth map generating apparatus further comprising: 35. The method of claim 34,
The smoothed amount calculation module calculates the smoothed amount S (P1, P2) according to (D (P1) -D (P2)) * N (P1, P2) * C (.) Is the pixel is the depth value;

here,

;

I (.) Is the characteristic (color or pattern) value of the pixel, and |. | Is the absolute value. 36. The method of claim 35,
Wherein the feature is a color or a pattern. Obtaining a plurality of consecutive two-dimensional images of the input video;
Calculating a degree of matching of a current two-dimensional image and a plurality of pre-stored three-dimensional representative structures among the plurality of two-dimensional images, and determining a three-dimensional representative structure having a highest matching degree as a three- ;
The depth map of the three-dimensional stereoscopic structure is stored in advance and the depth map of the three-dimensional stereoscopic structure determined by the three-dimensional structure of the current two-dimensional image is used as the depth map based on the matching corresponding to the current two- Each pixel of the due depth map displaying a depth value based on a matching degree of a corresponding pixel of a current two-dimensional image;
Lt; / RTI >
Determining the three-dimensional structure of the current two-dimensional image comprises:
Dividing a current two-dimensional image into at least one region corresponding to a plane in the matched three-dimensional representative structure;
Calculating a density of each of the regions according to the distribution of the characteristics of the regions; Calculating similarities between the two regions according to a difference between the average values by calculating average values corresponding to the respective characteristics; Calculating a matching degree according to a sum of the density of each of the regions and a similarity between the two regions; And
Determining a three-dimensional structure having a highest degree of matching as a three-dimensional structure of a current two-dimensional image according to the degree of matching;
/ RTI > 39. The method of claim 37,
Wherein the depth value based on the degree of matching is in the range [0, 1], 0 indicates that the corresponding pixel has the maximum depth, and 1 indicates that the corresponding pixel has the minimum depth. delete 39. The method of claim 37,

, Where r is each of the regions,

P is a pixel of the area, I (p) is a characteristic value of the pixel p,

Is an average value of the characteristic values of the pixels in the area, and area (r) is the number of pixels in the area. 39. The method of claim 37,

, The similarity between the region ri and the region rj is calculated, where

Is a mean value of the characteristics of the region, and | 42. The method according to any one of claims 40 to 41,
Wherein the characteristic is a color, gradient, or boundary depth map. 42. The method of claim 41,
Wherein the genome is a 1-genome, a 2-genome, or a ∞ genome. Obtaining a plurality of consecutive two-dimensional images of the input video;
Generating at least one saliency map corresponding to a current two-dimensional image of the plurality of two-dimensional images according to an HVP model, each pixel of the saliency map displaying saliency of a corresponding pixel of a current two- ;
Wherein the depth map is based on the at least one saliency map and the saliency map corresponding to the current two-dimensional image, and each pixel of the depth map based on the saliency displays a depth value based on the saliency of the corresponding pixel of the current two- step;
Calculating a degree of matching of a current two-dimensional image and a plurality of pre-stored three-dimensional representative structures among the plurality of two-dimensional images, and determining a three-dimensional representative structure having a highest matching degree as a three- ;
The depth map of the three-dimensional stereoscopic structure is stored in advance and the depth map of the three-dimensional stereoscopic structure determined by the three-dimensional structure of the current two-dimensional image is used as the depth map based on the matching corresponding to the current two- Each pixel of the due depth map displaying a depth value based on a matching degree of a corresponding pixel of a current two-dimensional image; And
Combining the depth map based on the conspicuousness and the depth map based on the matching to generate a comprehensive depth map, each pixel in the overall depth map displaying an overall depth value of a corresponding pixel in the current two-dimensional image;
Lt; / RTI >
Determining the three-dimensional structure of the current two-dimensional image comprises:
Dividing a current two-dimensional image into at least one region corresponding to a plane of the matched three-dimensional representative structure;
Calculating a density of each of the regions according to a characteristic distribution among the regions; Calculating average values corresponding to the characteristics of each of the regions and calculating a similarity between the two regions according to a difference between the average values; Calculating a matching degree according to a sum of the density of each of the regions and a similarity between the two regions; And
Determining a three-dimensional structure having the highest matching degree as a three-dimensional structure of a current two-dimensional image according to the degree of matching;
And a depth map generating step of generating a depth map. 45. The method of claim 44,
Generating the saliency map includes:
A characteristic saliency map, a motion saliency map and a target saliency map, generating a characteristic saliency map through the characteristics in the current two-dimensional image, and generating a current two-dimensional image and a current two- Generating an motion saliency map by identifying an image motion between two adjacent two-dimensional images in time, and generating an object saliency map by identifying an object in the current two-dimensional image. 46. The method of claim 45,
The depth map based on the conspicuousness may be generated by:
If only the object's saliency map is generated, a constant value in the range of (0, 1) is given to a pixel corresponding to a pixel identified as an object in the two-dimensional image of the depth map based on the sensibility, and 0 is set as a depth map Lt; / RTI >
A value within the range of [0, 1] is assigned to each pixel of the depth map based on the conspicuousness according to the conspicuousness of each pixel of the characteristic saliency map or the motional saliency map. , 0 indicates that the corresponding pixel has minimum saliency, and 1 indicates that the corresponding pixel has maximum saliency;
When two conspicuous maps that do not include the conspicuity map of the object are generated, the corresponding pixels in the two conspicuity maps are added together to give a standardized value or a larger value to the corresponding pixels in the depth map based on the conspicuousness ;
When the two conspicuity maps including the conspicuity map of the object are generated, the constants in the range of (0, 1) are given to the pixels corresponding to the pixels identified as the objects of the object conspicuity map in the depth map based on conspicuity, The corresponding pixel values of the consonantity maps other than the consonantity map of the object among the consonantity maps to the other corresponding pixels of the depth map based on the conspicuousness;
When the entire saliency map is generated, the constants in the range of (0, 1) are given to the pixels corresponding to the respective pixels identified as the targets of the target saliency map in the depth map based on the saliency, Adding the corresponding pixels of the sex map to each other to normalize or compare the values with each other to assign a larger value to a corresponding pixel in the depth map based on the conspicuousness. delete 45. The method of claim 44,

, Where r is each of the regions,

P is a pixel of the area, I (p) is a characteristic value of the pixel p,

Is an average value of the characteristic values of the pixels in the area, and area (r) is the number of pixels in the area. 45. The method of claim 44,

The similarity between the region ri and the region rj is calculated according to Equation

Is a mean value of the characteristics of the region, and | A method according to any one of claims 48 to 49,
Wherein the characteristic is a color, gradient, or boundary depth map. 50. The method of claim 49,
Wherein the genome is a 1-genome, a 2-genome, or a ∞ genome. 45. The method of claim 44,
The pixel values of the depth map based on the depth map based on the conspicuousness are in the range [0, 1], 0 indicates that the corresponding pixel has the maximum depth, and 1 indicates that the corresponding pixel has the minimum depth To generate a depth map. 50. The method of claim 49,
A depth map based on the depth map based on the sum of the corresponding pixel values in the depth map based on the conspicuousness and matching, or a depth map based on the conspicuousness, and a corresponding depth pixel on the matching map. To generate a depth map. 46. The method of claim 45,
Wherein the object in the current two-dimensional image comprises a person, a face, or a character. Obtaining a plurality of temporally continuous two-dimensional images of the input video;
Calculating a degree of matching of a current two-dimensional image and a plurality of pre-stored three-dimensional representative structures among the plurality of two-dimensional images, and determining a three-dimensional representative structure having a highest matching degree as a three- ;
The depth map of the three-dimensional stereoscopic structure is stored in advance and the depth map of the three-dimensional stereoscopic structure determined by the three-dimensional structure of the current two-dimensional image is used as the depth map based on the matching corresponding to the current two- Each pixel of the due depth map displaying a depth value based on a matching degree of a corresponding pixel of a current two-dimensional image; And
Advancing smoothness in a spatial domain and a time domain with respect to the depth map based on the matching;
Lt; / RTI >
Determining the three-dimensional structure of the current two-dimensional image comprises:
Dividing a current two-dimensional image into at least one region corresponding to a plane of the matched three-dimensional representative structure;
Calculating a density of each of the regions according to a characteristic distribution among the regions; Calculating average values corresponding to the characteristics of each of the regions and calculating a similarity between the two regions according to a difference between the average values; Calculating a matching degree according to a sum of the density of each of the regions and a similarity between the two regions; And
Determining a three-dimensional structure having the highest matching degree as a three-dimensional structure of a current two-dimensional image according to the degree of matching;
And a depth map generating step of generating a depth map. 56. The method of claim 55,
Wherein the step of smoothing in spatial and temporal regions with respect to the depth map based on the matching comprises:
(X + x, y + y, t + t) of the two-dimensional image at each pixel P1 (x, y, t) and time (t + Calculating a smoothed amount S (P1, P2) according to the difference between the similarity, distance and depth value between the reference points A and B, and determining Δx, Δy and Δt according to the expected smoothing effect;
Calculates a depth value D '(P1) = D (P1) -S (P1) of the pixel P1 of the current two-dimensional image after smoothing according to the smoothness amount S (P1, P2) The difference between the depth value of the pixel P1 after smoothing and the depth value D '(P2) = D (P2) + S (P1, P2) of the pixel P2 Making the absolute value smaller than the absolute value of the difference between the depth value D (P1) of the pixel value P1 before smoothing and the depth value D (P2) of the pixel value P2; Generation method. 57. The method of claim 56,
(P1, P2) according to the following equation (D (P1) -D (P2)) * N (P1, P2) * C Depth value;

here,

;

I (.) Is the characteristic (color or pattern) value of the pixel, and |. | Is the absolute value. 58. The method of claim 57,
Wherein the feature is a color and a pattern. Obtaining a plurality of consecutive two-dimensional images of the input video;
Generating at least one saliency map corresponding to a current two-dimensional image of the plurality of two-dimensional images according to an HVP model, each pixel of the saliency map displaying saliency of a corresponding pixel of a current two- ;
Using the at least one saliency map and a depth map based on the saliency corresponding to the current two-dimensional image, each pixel of the depth map based on the saliency displays a depth value based on the saliency of a corresponding pixel in the current two- step;
Calculating a degree of matching of a current two-dimensional image and a plurality of pre-stored three-dimensional representative structures among the plurality of two-dimensional images, and determining a three-dimensional representative structure having a highest matching degree as a three- ;
The depth map of the three-dimensional stereoscopic structure is stored in advance and the depth map of the three-dimensional stereoscopic structure determined by the three-dimensional structure of the current two-dimensional image is used as the depth map based on the matching corresponding to the current two- Each pixel of the due depth map displaying a depth value based on a matching degree of a corresponding pixel of a current two-dimensional image; And
Combining the depth map based on the conspicuousness and the depth map based on the matching to generate a comprehensive depth map, each pixel in the overall depth map displaying an overall depth value of a corresponding pixel in the current two-dimensional image; And
Advancing smoothness on the spatial domain and the time domain on the global depth map;
Lt; / RTI >
Determining the three-dimensional structure of the current two-dimensional image comprises:
Dividing a current two-dimensional image into at least one region corresponding to a plane of the matched three-dimensional representative structure;
Calculating a density of each of the regions according to a characteristic distribution among the regions; Calculating average values corresponding to the characteristics of each of the regions and calculating a similarity between the two regions according to a difference between the average values; Calculating a matching degree according to a sum of the density of each of the regions and a similarity between the two regions; And
Determining a three-dimensional structure having the highest matching degree as a three-dimensional structure of a current two-dimensional image according to the degree of matching;
And a depth map generating step of generating a depth map. 60. The method of claim 59,
Generating the saliency map includes:
A characteristic saliency map, a motion saliency map and a target saliency map, generating a characteristic saliency map through the characteristics in the current two-dimensional image, and generating a current two-dimensional image and a current two- A method of generating a depth map, the method comprising: generating an aberration map by identifying an image motion between two adjacent two-dimensional images in time, and generating an object saliency map by identifying an object in the current two-dimensional image. 60. The method of claim 59,
To create a depth map based on conspicuity:
If only the object's saliency map is generated, a constant value in the range of (0, 1) is given to the pixel corresponding to the pixel identified as the object of the two-dimensional image in the depth map based on the sensibility, and 0 is set as the depth map To other pixels of the pixel;
A value within the range of [0, 1] is assigned to each pixel of the depth map based on the conspicuousness according to the conspicuousness of each pixel of the characteristic saliency map or the motional saliency map. , 0 indicates that the corresponding pixel has minimum saliency, and 1 indicates that the corresponding pixel has maximum saliency;
When two conspicuous maps that do not include the conspicuity map of the object are generated, the corresponding pixels of the two conspicuity maps are added together to give a standardized value or a larger value to a corresponding pixel in the depth map based on the conspicuousness ;
When the two conspicuous maps including the conspicuousness map of the object are generated, the constants in the range of (0, 1) are given to the pixels corresponding to the pixels identified as the objects of the object conspicuity map in the depth map based on conspicuity, The corresponding pixel value of the saliency map other than the saliency map of the object among the surname maps to the other corresponding pixels in the depth map based on the saliency;
When the entire saliency map is generated, a constant in the range of (0, 1) is given to a pixel corresponding to the pixel identified as the target of the target saliency map in the depth map based on saliency, and two conspicuous A corresponding pixel in the map is added to the standardized value, or a larger value is compared with each other to give a corresponding pixel to the corresponding pixel in the depth map based on the conspicuousness. delete 60. The method of claim 59,

, Where r is each of the regions,

P is a pixel of the area, I (p) is a characteristic value of the pixel p,

Is an average value of the characteristic values of the pixels in the area, and area (r) is the number of pixels in the area. 60. The method of claim 59,

The similarity between the region ri and the region rj is calculated according to Equation

Is a mean value of the characteristics of the region, and | 65. The method according to any one of claims 63 to 64,
Wherein the feature is a color and a pattern. 65. The method of claim 64,
Wherein the genome is a 1-genome, a 2-genome, or a ∞ genome. 60. The method of claim 59,
The pixel values of the depth map based on the depth map based on the conspicuousness are in the range [0, 1], 0 indicates that the corresponding pixel has the maximum depth, and 1 indicates that the corresponding pixel has the minimum depth To generate a depth map. 60. The method of claim 59,
A depth map based on the depth map based on the sum of the corresponding pixel values in the depth map based on the conspicuousness and matching, or a depth map based on the conspicuousness, and a corresponding depth pixel on the matching map. To generate a depth map. 60. The method of claim 59,
Wherein the object in the current two-dimensional image comprises a person, a face, or a character. 60. The method of claim 59,
Wherein the step of smoothing the spatial domain and the temporal domain with respect to the global depth map comprises:
(X + x, y + y, t + t) of the two-dimensional image at each pixel P1 (x, y, t) and time (t + Calculating a smoothed amount S (P1, P2) according to the difference between the similarity, distance and depth value between the reference points A and B, and determining Δx, Δy and Δt according to the expected smoothing effect;
Calculates a depth value D '(P1) = D (P1) -S (P1) of the pixel P1 of the current two-dimensional image after smoothing according to the smoothness amount S (P1, P2) The difference between the depth value of the pixel P1 after smoothing and the depth value D '(P2) = D (P2) + S (P1, P2) of the pixel P2 Making the absolute value smaller than the absolute value of the difference between the depth value D (P1) of the pixel value P1 before smoothing and the depth value D (P2) of the pixel value P2; Generation method. 71. The method of claim 70,
(P1, P2) according to the following equation (D (P1) -D (P2)) * N (P1, P2) * C Depth value;

here,

;

I (.) Is the characteristic (color or pattern) value of the pixel, and |. | Is the absolute value. 72. The method of claim 71,
Wherein the feature is a color and a pattern. A computer-readable recording medium recording a program for performing the method of any one of claims 37, 44, 55, and 59.

Images