WO2012172853A1 - Apparatus for generating three-dimensional image, method for generating three-dimensional image, program, and recording medium - Google Patents

Abstract

When the position of a vanishing point can be estimated, a depth model of an image is generated on the basis of the vanishing point, and when the position of the vanishing point cannot be estimated, the depth model of the image is generated on the basis of the degree of prominence, whereby a three-dimensional image can be generated with a more natural feeling of depth. An apparatus for generating a three-dimensional image (1) is provided with: a vanishing point estimation unit (20) for estimating a vanishing point from an image to be processed; a depth model generator (30) for generating a different depth model on the basis of whether it was possible or impossible for the vanishing point estimation unit (20) to estimate the vanishing point; and a viewpoint image generator (40) for generating the image presented to the right eye and the image presented to the left eye on the basis of the depth model generated by the depth model generator (30), the image to be processed, and assumed viewing condition information. If it was possible for the vanishing point estimation unit (20) to estimate the vanishing point, the depth model generator (30) generates the depth model on the basis of the vanishing point, and if it was impossible for the vanishing point estimation unit (20) to estimate the vanishing point, the depth model is generated on the basis of the degree of prominence of each pixel within the image to be processed.

Description

Stereoscopic image generating apparatus, stereoscopic image generating method, program, and recording medium

The present invention relates to a stereoscopic image generation apparatus, a stereoscopic image generation method, a program, and a recording medium that add binocular stereoscopic information to a 2D image to generate a 3D image.

In recent years, with the spread of 3DTV (3D Television) and the start of 3D digital broadcasting, the environment for viewing 3D video at home is being prepared. However, with the improvement of the 3D video playback environment, it has been pointed out that there is a shortage of 3D video content. As an approach to resolving such a shortage of content, 2D / 3D conversion (2D to 3D conversion), in which binocular stereoscopic information is artificially added to a 2D image to generate a 3D image, has attracted attention.

As a technique for realizing 2D / 3D conversion, for example, a technique disclosed in Patent Document 1 is known. This patent document 1 is provided with a basic depth model indicating depth values of three basic images, and the input image depth model is changed by changing the composition ratio of the three basic depth models according to the pattern of the input image. A stereoscopic image generation apparatus that generates an image to be presented to the left eye / right eye from the generated depth model and an input image is disclosed.

Japanese Patent No. 4214976 (Japanese Patent Laid-Open No. 2005-151534)

However, in the stereoscopic image generating device disclosed in Patent Document 1, it is difficult to express a depth model of an image that does not match the assumed basic depth model. For example, when there is a vanishing point outside the screen, there is a problem that it cannot be expressed even if the composition ratio of the three basic depth models is changed. For this reason, it was not possible to generate a stereoscopic image with a natural sense of depth.

The present invention was made to solve the above-described problems, and when the vanishing point position can be estimated by a geometric depth cue, a depth model of the image is generated based on the vanishing point, If the vanishing point position cannot be estimated due to a geometric depth cue, a more natural depth sensation can be obtained by generating a depth model of the image based on the degree of saliency representing the attractiveness in the image based on human visual characteristics. It is an object of the present invention to provide a stereoscopic image generating apparatus, a stereoscopic image generating method, a program, and a recording medium that can generate a certain stereoscopic image.

In order to solve the above-described problem, a first technical means of the present invention is a stereoscopic image generation apparatus that generates binocular stereoscopic information by adding binocular stereoscopic information to a 2D image, and estimates a vanishing point from the processing target image. Vanishing point estimating means, depth model generating means for generating a different depth model based on whether the vanishing point can be estimated by the vanishing point estimating means, the depth model generated by the depth model generating means, and the processing target Based on the image and the assumed viewing condition information, a viewpoint image generation unit that generates a right eye presentation image and a left eye presentation image is provided, and the depth model generation unit can estimate the vanishing point by the vanishing point estimation unit. A depth model is generated based on the vanishing point, and if the vanishing point cannot be estimated by the vanishing point estimating means, a depth model is generated based on the saliency of each pixel in the processing target image. It is obtained by and generating a.

The second technical means includes, in the first technical means, reduced image generation means for generating a reduced image having a predetermined image size from the processing target image, and the reduced image is converted into the vanishing point estimation means and the depth model generation. And an enlarged depth model generating means for generating an enlarged depth model having the same image size as the processing target image from the depth model of the reduced image generated by the depth model generating means. .

According to a third technical means, in the first technical means, the depth model of the processing target image generated by the depth model generation means is smoothed in the spatial direction, and the depth model of the processing target image smoothed in the spatial direction is used. And the depth model smoothed in the spatio-temporal direction of the comparison target image in the past than the processing target image, the depth model of the processing target image is smoothed in the time direction, and the spatio-temporal of the processing target image A spatio-temporal direction smoothing means for generating a depth model smoothed in the direction is provided.

In a fourth technical means according to any one of the first to third technical means, the assumed viewing condition information includes a pixel pitch of a display for displaying the 3D image, an image size of the display, a viewer to the display And a parallax range representing the depth of the 3D image, and a baseline length that is a distance between the left and right virtual viewpoints.

According to a fifth technical means, in any one of the first to fourth technical means, the saliency of each pixel in the processing target image is a point where the color difference between the target pixel and its peripheral pixels is large, or the target The higher the color difference between the pixel and the whole image, or the greater the color difference between the local region including the target pixel and its surrounding region, the higher the calculation.

A sixth technical means is the fifth technical means, wherein when the vanishing point cannot be estimated by the vanishing point estimating means, the depth model generating means has a high saliency of each pixel in the processing target image. The depth model is generated so that is on the near side.

A seventh technical means is any one of the first to sixth technical means, wherein the vanishing point estimating means estimates the vanishing point of the processing target image from the straight line information in the processing target image. An inter-frame vanishing point that estimates a vanishing point of the processing target image based on a point estimation unit, the processing target image, a comparison target image that is earlier than the processing target image, and a vanishing point position in the comparison target image; And an estimation means.

The eighth technical means comprises scene change detection means for detecting whether or not a scene change has occurred between the processing target image and the comparison target image in the seventh technical means, and the scene change detection means When a change is detected, the intra-frame vanishing point estimating means is selected, and when no scene change is detected by the scene change detecting means, the inter-frame vanishing point estimating means is selected. .

A ninth technical means includes, in the eighth technical means, storage means for storing vanishing point information including the position of the vanishing point of the comparison target image, and the vanishing point information of the comparison target image is stored in the storage means. The inter-frame vanishing point estimating means is selected, and when the vanishing point information of the comparison target image is not stored in the storage means, the intra-frame vanishing point estimating means is selected. It is what.

The tenth technical means is any one of the seventh to ninth technical means, wherein the comparison target image is an image immediately before the processing target image.

An eleventh technical means is a stereoscopic image generation method by a stereoscopic image generation apparatus that adds binocular stereoscopic information to a 2D image and generates a 3D image, and the stereoscopic image generation apparatus detects a vanishing point from a processing target image. A vanishing point estimating step, a depth model generating step for generating a different depth model based on whether or not the vanishing point can be estimated in the vanishing point estimating step, and a depth model generated in the depth model generating step; A viewpoint image generation step of generating a right eye presentation image and a left eye presentation image based on the processing target image and the assumed viewing condition information, and the depth model generation step includes a vanishing point in the vanishing point estimation step. Can be estimated, a depth model is generated based on the vanishing point, and when the vanishing point cannot be estimated in the vanishing point estimating step, It is obtained by and generating a depth model based on the saliency of each pixel in the.

The twelfth technical means is a program for causing a computer to execute the stereoscopic image generating method in the eleventh technical means.

The thirteenth technical means is a computer-readable recording medium recording the program in the twelfth technical means.

According to the present invention, when the vanishing point position can be estimated by the geometric depth cue, a stereoscopic image in which the depth feeling by the geometric depth cue is emphasized is generated by generating the image depth model based on the vanishing point. Can be generated.
Further, according to the present invention, when the vanishing point position cannot be estimated by the geometric depth cue, by generating the image depth model from the saliency representing the attractiveness in the image based on the human visual characteristics, It is possible to generate a stereoscopic image in which the sense of depth of a portion that is noticed by a person is emphasized.

It is a block diagram which shows the structural example of the stereo image production | generation apparatus which concerns on embodiment of this invention. It is a flowchart for demonstrating the operation example of the frame unit of the stereo image production | generation apparatus which concerns on embodiment of this invention. It is the schematic of the scene change detection based on a brightness | luminance histogram. It is a block diagram which shows the structural example of the scene change detection part which concerns on embodiment of this invention. It is a flowchart for demonstrating the operation example of the scene change detection part which concerns on embodiment of this invention. It is a block diagram which shows the structural example of the vanishing point estimation part which concerns on embodiment of this invention. It is a flowchart for demonstrating the operation example of the vanishing point estimation part which concerns on embodiment of this invention. It is a flowchart for demonstrating the operation example of the vanishing point estimation part in a flame | frame which concerns on embodiment of this invention. It is a figure which shows an example of the image corresponding to the flow of FIG. It is the schematic for demonstrating the straight line detection by Hough transform. It is a flowchart for demonstrating the operation example of the vanishing point estimation part between frames which concerns on embodiment of this invention. It is a figure which shows an example of the image corresponding to the flow of FIG. It is a figure which shows an example of the range to which an intra-frame vanishing point estimation means and an inter-frame vanishing point estimation means are applied in the same scene. It is a block diagram which shows the structural example of the depth model production | generation part which concerns on embodiment of this invention. It is a flowchart for demonstrating the operation example of the depth model production | generation part which concerns on embodiment of this invention. It is a figure which shows an example of a basic depth model in case there exists a vanishing point in the screen which concerns on embodiment of this invention. It is a figure which shows an example of the basic depth model in case there exists a vanishing point outside the screen which concerns on embodiment of this invention. It is a figure which shows an example of the process which calculates | requires the depth model based on the vanishing point which concerns on embodiment of this invention. It is a figure which shows an example of the process which calculates | requires the depth model based on the saliency which concerns on embodiment of this invention. It is a bird's-eye view of camera (viewpoint) arrangement for generating a viewpoint image. It is a block diagram which shows the structural example of the viewpoint image generation part which concerns on embodiment of this invention. It is a flowchart for demonstrating the operation example of the viewpoint image generation part which concerns on embodiment of this invention. It is a figure which shows an example of the process which produces | generates the viewpoint image which concerns on embodiment of this invention. It is a figure which shows the parallax vector of a cross direction and a spreading | diffusion direction. It is a block diagram which shows the modification of the depth model production | generation part which concerns on embodiment of this invention. It is a flowchart for demonstrating the operation example of the depth model production | generation means based on the saliency in the modification of the depth model production | generation part which concerns on embodiment of this invention. It is a figure which shows an example of the process which calculates | requires the depth model (modification) based on the saliency which concerns on embodiment of this invention. It is a figure which shows an example of the depth model corresponding to an example of the modification of the depth model used as a reference | standard in the depth model (modification) based on the saliency which concerns on embodiment of this invention. It is a block diagram which shows the structural example of the stereo image production | generation apparatus (1st modification) which concerns on embodiment of this invention. It is a flowchart for demonstrating the operation example of the frame unit of the stereo image production | generation apparatus (1st modification) which concerns on embodiment of this invention. It is a block diagram which shows the structural example of the stereo image production | generation apparatus (2nd modification) which concerns on embodiment of this invention. It is a flowchart for demonstrating the operation example of the frame unit of the stereo image production | generation apparatus (2nd modification) which concerns on embodiment of this invention. It is a block diagram which shows the structural example of the spatio-temporal direction smoothing part which concerns on embodiment of this invention. It is a flowchart for demonstrating the operation example of the spatiotemporal direction smoothing part which concerns on embodiment of this invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In the drawings, portions having the same function are denoted by the same reference numerals, and repeated description is omitted.
FIG. 1 is a block diagram illustrating a schematic configuration example of a stereoscopic image generating apparatus according to the present invention. In the figure, reference numeral 1 denotes a stereoscopic image generating apparatus. The stereoscopic image generation apparatus 1 includes a scene change detection unit 10, a vanishing point estimation unit 20, a depth model generation unit 30, and a viewpoint image generation unit 40. FIG. 2 is a flowchart for explaining an operation example in units of frames of the stereoscopic image generating apparatus 1 according to the present invention.

2, first, the stereoscopic image generation apparatus 1 in FIG. 1 uses an input image at time t (hereinafter also referred to as a processing target image F (t)) as a scene change detection unit 10, a vanishing point estimation unit 20, a depth. It outputs to the model production | generation part 30 and the viewpoint image production | generation part 40 (step S11 of FIG. 2).

(About the scene change detection unit 10)
The scene change detection unit 10 in FIG. 1 corresponds to the scene change detection means of the present invention, and the input processing target image F (t) and the image input immediately before the processing target image F (t) ( Hereinafter, a predetermined image feature amount is calculated from the comparison target image F (t-1)), the similarity of the calculated image feature amount is compared, and the segment points (scene change) of the images that are continuous in time series , And scene change information S (t) indicating the presence or absence of a scene change in the processing target image F (t) is output to the vanishing point estimation unit 20 and the depth model generation unit 30 (step S12 in FIG. 2). Here, as an example of a predetermined image feature amount, scene change detection based on a luminance histogram representing the appearance frequency of the luminance value of an image will be described with reference to FIGS.

As shown in FIG. 3, the scene change detection based on the luminance histogram is performed based on the luminance histograms H _L (t) and H _L (t−1) from the processing target image F (t) and the comparison target image F (t−1). ) And the similarity d (H _L (t), H _L (t−1)) of the calculated luminance histogram is compared with a predetermined threshold value to determine whether or not there is a scene change in the processing target image F (t) Is to determine.

As shown in FIG. 4, the scene change detection unit 10 includes a luminance histogram generation unit 101, a buffer 102, a histogram similarity calculation unit 103, and a scene change determination unit 104. FIG. 5 is a flowchart for explaining an operation example of the scene change detection unit 10. In FIG. 5, the luminance histogram generation unit 101 in FIG. 4 acquires luminance information from the input processing target image F (t), and a luminance histogram H _L (t) that represents the appearance frequency of the luminance value from the acquired luminance information. And the calculation result (luminance histogram H _L (t)) is output to the buffer 102 and the histogram similarity calculation unit 103 (step S21 in FIG. 5).

The buffer 102 in FIG. 4 stores the luminance histogram H _L (t) of the processing target image F (t) in order to detect a scene change in the image F (t + 1) immediately after the processing target image F (t). (Step S22 in FIG. 5). The histogram similarity calculation unit 103 in FIG. 4 inputs the luminance histogram H _L (t) of the input processing target image F (t) and the luminance histogram H of the comparison target image F (t−1) read from the buffer 102. _The similarity d (H _L (t), H _L (t−1)) is calculated from _L (t−1) by the equation (1), and the calculation result is output to the scene change determination unit 104 (FIG. 5). Step S23).

Here, in Expression (1), W represents the number of pixels per line of the image, H represents the number of lines of the image, v represents the luminance value, V represents the number of gradations of the luminance value, and H _L (v | t) represents the appearance frequency of the luminance value v on the image F (t) at time t. In equation (1), the range of values taken by the similarity d (H _L (t), H _L (t−1)) is 0 to 2, and the closer the value is to 0, the more similar the shape of the histogram is. The closer the value is to 2, the more different the shape of the histogram is.

The scene change determination unit 104 in FIG. 4 performs threshold determination based on the similarity d (H _L (t), H _L (t−1)) of the input histogram and a predetermined threshold d _th , and the expression (2 ), Scene change information S (t) indicating the presence / absence of a scene change in the processing target image F (t) is set and output to the outside (step S24 in FIG. 5).

That is, when the similarity d (H _L (t), H _L (t−1)) is smaller than the threshold value d _th , the scene change determination unit 104 determines that there is no scene change, and the scene change information S (t) Set “0” to. In other cases, it is determined that there is a scene change, and “1” is set in the scene change information S (t).

As described above, according to the scene change detection unit 10, the predetermined image feature amount is calculated from the processing target image F (t) and the comparison target image F (t-1), and the similarity of the calculated image feature amount is compared. Thus, it is possible to detect segment points (scene changes) of images that are continuous in time series.

(About the vanishing point estimation unit 20)
Returning to FIG. 1, the vanishing point estimation unit 20 corresponds to the vanishing point estimation unit of the present invention, and the input scene change information S (t) of the processing target image F (t) and the vanishing point estimation unit 20. Vanishing point estimating means (intraframe vanishing point estimating means for estimating vanishing point position from straight line in image, feature between images based on previous vanishing point information VP (t−1) stored internally An inter-frame vanishing point estimating means for estimating the vanishing point position in the current frame from the point correspondence and the vanishing point position of the previous frame) is selected, and the processing target image F (t) input by the selected vanishing point estimating means is selected. The vanishing point position is estimated, and vanishing point information VP (t) describing the result is output to the depth model generating unit 30 (step S13 in FIG. 2). Here, the “vanishing point” is a point where, when two parallel lines in a three-dimensional space are projected (projected) onto a plane, those lines always converge to one point.

Subsequently, the vanishing point estimation unit 20 in the present embodiment will be described in detail. As illustrated in FIG. 6, the vanishing point estimation unit 20 includes a switching unit 201, a switching unit 202, an intraframe vanishing point estimation unit 21, an interframe vanishing point estimation unit 22, a buffer 203, and a buffer 204. The intra-frame vanishing point estimation unit 21 in FIG. 6 includes an edge detection unit 211, a straight line detection unit 212, and a vanishing point identification unit 213. This intra-frame vanishing point estimation unit 21 corresponds to the intra-frame vanishing point estimation means of the present invention, and estimates the vanishing point of the processing target image F (t) from the straight line information in the processing target image F (t). 6 includes a feature point detection unit 221, a corresponding point calculation unit 222, a transformation matrix calculation unit 223, and a vanishing point position calculation unit 224. This inter-frame vanishing point estimation unit 22 corresponds to the inter-frame vanishing point estimation means of the present invention, and is compared with the processing target image F (t) and the processing target image F (t) in the past compared image F (t−1). ) And the position of the vanishing point in the comparison target image F (t−1), the vanishing point of the processing target image F (t) is estimated. FIG. 7 is a flowchart for explaining an operation example of the vanishing point estimation unit 20.

In FIG. 7, the vanishing point estimation unit 20 in FIG. 6 estimates vanishing points based on the input scene change information S (t) and the previous vanishing point information VP (t−1) read from the buffer 204. A means is selected (step S31 in FIG. 7). Specifically, when there is a scene change (“S (t) = 1”), or when the vanishing point information VP (t−1) indicates that there is no vanishing point in the previous frame, that is, the vanishing point. When the information VP (t−1) is “vp_num = 0” (Yes in step S31 in FIG. 7), the switching unit 201 in FIG. 6 outputs an image input destination, and the switching unit 202 in FIG. 6 outputs vanishing point information. The original is switched to the intra-frame vanishing point estimation unit 21 (step S32 in FIG. 7), and then the intra-frame vanishing point estimation unit 21 estimates the vanishing point position from the straight line in the image, and the result (vanishing point information) VP (t)) is output (step S33 in FIG. 7).

(Intraframe vanishing point estimation unit 21)
Here, the intra-frame vanishing point estimation unit 21 will be described in detail. FIG. 8 is a flowchart for explaining an operation example of the intra-frame vanishing point estimation unit 21. FIG. 9 is a diagram illustrating an image example corresponding to the flow of FIG. In step S331 in FIG. 8, the edge detection unit 211 in FIG. 6 calculates edge point information Edge (t) used for straight line detection from the input processing target image F (t) (see FIG. 9A). . Specifically, first, a differential operator is applied to each color component (for example, RGB (Red (red), Green (green), Blue (blue))), and a gradient vector of each color component i in the x direction and the y direction. G _i (x, y | t) = (ΔG _ix (t), ΔG _iy (t)) (i = 1, 2, 3) is calculated. Note that i = 1, 2, and 3 are an R component, a G component, and a B component, respectively.

Subsequently, the edge detection unit 211 calculates the edge strength E (x, y | t) by performing the calculation of Expression (3) for each pixel of the coordinates (x, y).

Subsequently, the edge detection unit 211 performs the calculation of Expression (4) for each pixel of the coordinates (x, y), so that the edge strength is locally maximum from the edge strength E (x, y | t). Are extracted as edge points, and edge point information Edge (t) describing the result is output to the straight line detection unit 212.

In other words, the edge detection unit 211 detects an edge when the edge strength E (x, y | t) has a maximum value (Local Maxima) within the range of the window size W1 × W2 with the coordinates (x, y) as the center. “1” is set to the point Edge (x, y | t), and “0” is set to the edge point Edge (x, y | t) otherwise. Here, W1 represents the size of the window in the x direction, and W2 represents the size of the window in the y direction.

Proceeding to step S332 in FIG. 8, the straight line detection unit 212 in FIG. 6 applies straight line information L (t) to the input edge point information Edge (t) by applying Hough transform (see FIG. 9B). To get. Here, the straight line detection by the Hough transform will be described with reference to FIG. In FIG. 10A, feature points A, B, and C on a certain straight line L are edge points that become “Edge (x, y | t) = 1” obtained by the edge detection unit 211. . First, in the Hough transform, a straight line L on the image space shown in FIG. 10A is expressed using polar coordinate expressions (ρ, θ). ρ represents the distance of a perpendicular drawn from the origin of the image space to the straight line L, and θ represents the angle formed by the perpendicular to the x axis of the image space. The range of ρ is ρ ≧ 0, and the range of θ is 0 ≦ θ <2π.

A straight line L that passes through the feature points A, B, and C on the image space of FIG. 10A is expressed by Expression (5) using parameters (ρ ₀ , θ ₀ ).

Further, when a straight line group passing through each of the feature points A, B, and C is mapped to the parameter space, it is represented as a curve a, a curve b, and a curve c on the parameter space in FIG. 10B. That is, the intersection (ρ ₀ , θ ₀ ) of the curve a, the curve b, and the curve c is detected as a straight line L passing through the feature points A, B, and C on the parameter space.

As described above, the Hough transform is applied to the edge point where “Edge (x, y | t) = 1”, and the curves having a predetermined threshold value or more intersect on the parameter space, and the number of intersections increases. NL polar coordinates (ρs, θs) (s = 1,..., NL) are extracted as a straight line Ls, and data describing the polar coordinates (ρs, θs) is set as straight line information L (t). The range of NL is 0 ≦ NL ≦ NL _max . Here, NL _max is a predetermined constant representing the upper limit value of the number of straight lines to be extracted. If the straight line extraction condition is not satisfied and no straight line is detected, NL = 0.

Returning to FIG. 8 again, the vanishing point identifying unit 213 in FIG. 6 obtains the number of straight lines NL from the input straight line information L (t), compares the number of straight lines NL with a predetermined threshold value, and compares the magnitude of the step 334. In step S335, it is determined whether or not the vanishing point position is estimated (step S333). When the number of straight lines NL is smaller than the threshold ThL (≧ 2) (or in the following case) (No in step S333), it is determined that there is no geometric depth clue sufficient to estimate the vanishing point, and the process proceeds to step S336. When the number of straight lines NL is greater than or equal to the threshold ThL (≧ 2) (or larger) (Yes in step S333), the vanishing point identifying unit 213 determines that there is a sufficient geometric depth clue for vanishing point estimation. From the input straight line information L (t), the straight line Li (i = 1,..., NL) and the straight line Lj (j) satisfying the conditions of the expressions (6) and (7) with respect to the angle θ representing the straight line. = 1,..., NL), the intersection P _ij (i ≠ j) is calculated by the matrix operation of the equation (8), and the intersection information is acquired (step S334). It should be noted that when the intersection of the straight lines is obtained, the overlap calculation between the straight line Li and the straight line Lj once selected is not performed. Note that the condition of Expression (6) represents that the selected two straight lines are not parallel, and the condition of Expression (7) is the horizontal direction (| θ−π | ≈π / 2) and the vertical direction (| θ−π | ≈0 or | θ−π | ≈π).

Subsequently, the vanishing point identifying unit 213 assumes that the distribution model of the acquired intersection point P _ij of the straight line is expressed using a mixed model GMM (Gaussian Mixture Model) of Kc Gaussian distributions shown in Expression (9). Then, the parameters (wi, μi, Σi) (i = 1,..., Kc) of the distribution model are acquired by the EM (Expectation-Maximization) algorithm, and the position of the vanishing point is determined (step S335).

In Equation (9), P (x) represents the probability that the vector x (coordinates of the intersection Pij) will appear. Kc represents the number of classes (number of Gaussian distributions), wi represents a weighting coefficient of class i Gaussian distribution, and the sum of the weighting coefficients is 1. Μi represents a class i average vector (class i barycentric coordinates), Σi represents a class i covariance matrix, and D represents the number of dimensions of the vector x. N (x | μi, Σi) in Expression (9) represents a Gaussian distribution (normal distribution) of class i, and is expressed using an average vector μi and a covariance matrix Σi. That is, the vanishing point identifying unit 213 determines the average vector μi (center of gravity coordinates) of the distribution of the top N (≧ 1) classes having a large weighting coefficient wi as the vanishing point position. Hereinafter, for the sake of simplicity, the number of vanishing points will be described as N = 1, but the present invention is not limited to this.

Subsequently, the vanishing point identifying unit 213 sets vanishing point information VP (t) as shown in FIG. 9C based on the result of step S333 or step S335 (step S336). The vanishing point information VP (t) is expressed as data in Table 1, for example.

In Table 1, vanishing point information VP (t) is “vp_time” indicating time t (or a number (frame number) assigned to a frame of an image), “vp_num” indicating the number of detected vanishing points, And a list representing the position “vp_pos [n]” of the n vanishing points detected.

Returning to step S31 in FIG. 7 again, when there is no scene change (“S (t) = 0”) and the vanishing point information VP (t−1) indicates that there is a vanishing point in the previous frame ( (“Vp_num> 0” of the vanishing point information VP (t−1)) (No in step S31), the switching unit 201 in FIG. 6 is the input destination of the image, and the switching unit 202 in FIG. 6 is the output source of the vanishing point information. Then, switching to the inter-frame vanishing point estimation unit 22 is performed (step S34), and then the inter-frame vanishing point estimation unit 22 and the input processing target image F (t) and the previous image F (t stored in the buffer 203 ( From t-1), the correspondence between feature points between images is obtained, and the vanishing point position in the processing target image F (t) is estimated from the correspondence and the previous vanishing point information VP (t-1). The result (vanishing point information VP (t)) is output (step Flop S35). That is, the vanishing point estimation unit 20 includes storage means (buffer 204 in FIG. 6) that stores vanishing point information VP (t−1) including the position of the vanishing point of the comparison target image F (t−1). Whether or not the frame has a vanishing point is determined by storing the vanishing point information VP (t−1) of the previous frame in this storage means, and the vanishing point information VP (t−1) is “vp_num> 0”. It is determined by whether or not there is.

(About the inter-frame vanishing point estimation unit 22)
Here, details of the inter-frame vanishing point estimation unit 22 will be described. FIG. 11 is a flowchart for explaining an operation example of the inter-frame vanishing point estimation unit 22. FIG. 12 is a diagram illustrating an example of an image corresponding to the flow of FIG. In step S351 in FIG. 11, the feature point detection unit 221 in FIG. 6 receives the input processing target image F (t), the previous image F (t−1), as shown in FIG. NK feature points Ks (s = 1,..., NK) used for obtaining the correspondence between the images are detected, and the coordinates (x _{Ks, t} , y _{Ks, t} ) of the feature points Ks are detected. The described feature point information K (t) is output to the corresponding point calculation unit 222 in FIG. 6 (step S351).

A feature point is a point extracted as a part or vertex of an object's edge based on a change in color or luminance between pixels. For example, a secondary expressed using a gradient vector Gi (x, y) (i = x, y) of luminance in the x and y directions within the range of the local region S with the pixel (x, y) as the center. A first eigenvalue λ1 and a second eigenvalue λ2 of the moment matrix A (Expression (10)) are obtained, and a pixel (x, y) that satisfies the condition shown in Expression (11) is detected as a feature point.

That is, a characteristic point is obtained when the smaller eigenvalue of the first eigenvalue λ1 and the second eigenvalue λ2 of the second moment matrix A is greater than (or greater than) the predetermined threshold λth (for example, non-patent literature). 1). In Equation (10), the coefficient w (u, v) represents a weighting coefficient for a pixel (x + u, y + v) that is separated from the pixel (x, y) by u in the x direction and v in the y direction. A value obtained by normalizing the value of the secondary Gaussian distribution within the range of the local region S determined so as to satisfy the condition (12) is used.

Proceeding to step S352 in FIG. 11, the corresponding point calculation unit 222 in FIG. 6, as shown in FIG. 12B, inputs the processing target image F (t) and the previous one read from the buffer 203. Based on the image F (t−1) and the feature point information K (t) of the processing target image F (t) acquired in step S351, each feature point Ks (s = 1) of the processing target image F (t). ,..., NK) is calculated by optical flow at a position (x _{Ks, t −1} , y _{Ks, t −1} ) where the previous image F (t−1) is on the image F (t−1). Corresponding point information Q (t, t−1) describing the positions at time t and time t−1 is output to the transformation matrix calculation unit 223 in FIG. 6 (step S352).

The position (x _{Ks, t−1} , y _{Ks, t−1} ) at which each feature point Ks of the processing target image F (t) is on the previous image F ( _t−1 ) is, for example, an expression It can be acquired by solving the constraint condition of the optical flow by the gradient method shown in (13) with respect to (x _{Ks, t−1} , y _{Ks, t−1} ) (for example, see Non-Patent Document 2).

Here, in Expression (13), Gi (x, y | t) (i = x, y, t) is a gradient vector in the x direction, y direction, and t direction (time direction) related to the luminance of the image F (t). S represents a local area of a predetermined size centered on the feature point Ks.

Proceeding to step S353 in FIG. 11, the transformation matrix calculation unit 223 in FIG. 6 uses the feature point Ks (s = 1,..., NK) from the corresponding point information Q (t, t−1) acquired in step S352. ) Is calculated from the position on the previous image F (t−1) to the position on the processing target image F (t), and information describing the conversion matrix H is shown in FIG. The data is output to the vanishing point position calculation unit 224 (step S353). Note that the correspondence between two images (F (t), F (t−1)) can be expressed by Expression (14) using the transformation matrix H (see, for example, Non-Patent Document 3). . In the equation (14), the symbol “˜” represents an equivalence relation, and means that they are equal by allowing a constant multiple difference.

The transformation matrix H is called projective transformation because it can express general transformation. Here, if it is assumed that the correspondence between images can be expressed as parallel movement, Expression (14) is expressed as Expression (15).

In the equation (15), coefficients tx and ty represent amounts of movement in the x and y directions, respectively. Assuming that the correspondence between images can be expressed as affine transformation including translation, rotation, and enlargement / reduction, Expression (14) is expressed as Expression (16).

Coefficients a, b, c, and d in Expression (16) represent enlargement / reduction and rotation, and coefficients tx and ty are the same as those in Expression (15). The coefficients h _ij (i, j = 1, 2, 3) of the transformation matrix H in the equations (14), (15), and (16) are the constraint conditions and corresponding point information Q ( It is calculated by solving simultaneous equations derived from t, t-1) by the method of least squares. If there is not a sufficient number of corresponding points and the conversion matrix H cannot be calculated, a predetermined conversion matrix H ₀ is used.

Proceeding to step S354 in FIG. 11, the vanishing point position calculation unit 224 in FIG. 6 determines that “the vanishing point position at time t is the vanishing point position on the previous image F (t−1). Using the transformation matrix H calculated by the transformation matrix calculation unit 223, the vanishing point position at the time t is calculated (step 354), assuming that the position is projected onto the processing target image F (t). Based on the result, vanishing point information VP (t) is set (step S355). FIG. 12C shows an image example when the vanishing point on the image F (t−1) is projected onto the image F (t) by the transformation matrix H.

7 again, the buffer 203 in FIG. 6 deletes the previous image F (t−1) and stores the input processing target image F (t). Also, the buffer 204 in FIG. 6 deletes the previous vanishing point information VP (t−1), and the vanishing point information input from the intra-frame vanishing point estimation unit 21 or the inter-frame vanishing point estimation unit 22. VP (t) is stored, and the vanishing point estimation process for the processing target image F (t) is terminated (step S36).

As described above, according to the vanishing point estimation unit 20 of the present embodiment, as shown in FIG. 13, in the same scene (a time series image group correlated in the spatial direction and the time direction), the same from the first frame F (t ₀ ). Until the frame F (t ₀ + k−1) in which the vanishing point is first detected in the scene, the vanishing point is estimated by the intra-frame vanishing point estimating means (intra-frame vanishing point estimating unit 21). From the next frame F (t ₀ + k) of the frame F (t ₀ + k−1) where the vanishing point is detected to the last frame F (t ₀ + N) in the same scene, the inter-frame vanishing point estimation means (inter-frame Since the vanishing point is estimated by the vanishing point estimation unit 22), the vanishing point is more robust to the camera work than the case where the vanishing point is estimated by the intra-frame vanishing point estimation unit, and the vanishing point fluctuation is suppressed and stabilized. The vanishing point can be estimated.

(About the depth model generation unit 30)
Returning to FIG. 2 again, the depth model generation unit 30 in FIG. 1 corresponds to the depth model generation means of the present invention, and generates different depth models based on whether or not the vanishing point can be estimated by the vanishing point estimation unit 20. To do. That is, the depth model generation unit 30 is based on the vanishing point information VP (t), based on the visual model of the depth model creating means (first depth model creating means for creating the depth model from the vanishing point position, human visual characteristics). Second depth model creating means for creating a depth model from the saliency representing the attractiveness in the image), and the depth value of each pixel in the processing target image F (t) is set by the selected depth model creating means. Then, the depth model D (t) representing the depth value of each pixel is output to the viewpoint image generation unit 40 (step S14 in FIG. 2).

Subsequently, the depth model generation unit 30 in the present embodiment will be described in detail. As illustrated in FIG. 14, the depth model generation unit 30 includes a switching unit 301, a switching unit 302, an area dividing unit 303, a distance calculating unit 304, a saliency calculating unit 305, and a depth value setting unit 306. FIG. 15 is a flowchart for explaining an operation example of the depth model generation unit 30.

15, the depth model generation unit 30 in FIG. 14 selects a depth model creation means based on the input vanishing point information VP (t) (step S41). That is, when there is a vanishing point in the current frame (“vp_num> 0” of vanishing point information VP (t)) (Yes in step S41), the switching unit 301 in FIG. 14 transfers the image input destination to the region dividing unit 303. The switching unit 302 in FIG. 14 switches the output source of the data input to the depth value setting unit 306 to the distance calculation unit 304, and the first depth model creation means based on the vanishing point is selected (step S42).

In step S43, the distance calculation unit 304 in FIG. 14 calculates the distance Dist (x, y) between the coordinates of the vanishing point of the vanishing point information VP (t) and the coordinates of each pixel, and describes the result. The distance information Dist (t) is output to the depth value setting unit 306 (step S43). Specifically, the distance Dist (x, y) between the coordinates of the vanishing point and the coordinates of each pixel is calculated based on any one of Expression (17), Expression (18), and Expression (19). In Expressions (17), (18), and (19), Δx and Δy represent the distance in the x direction and the distance in the y direction between each pixel and the vanishing point, respectively.

Here, FIG. 16 shows an example of the distance information Dist (t) based on the equations (17), (18), and (19) when the vanishing point VP is present in the screen. FIG. 16A shows an example in which the vanishing point VP is in the screen. BD1a in FIG. 16B represents equation (17), BD1b in FIG. 16C represents equation (18), and BD1c in FIG. 16D represents distance information Dist (t) based on equation (19). . FIG. 17 shows an example of the distance information Dist (t) based on the equations (17), (18), and (19) when the vanishing point VP exists outside the screen. FIG. 17A shows an example in which the vanishing point VP is outside the screen. BD2a in FIG. 17B represents equation (17), BD2b in FIG. 17C represents equation (18), and BD2c in FIG. 17D represents distance information Dist (t) based on equation (19). . In FIG. 16 and FIG. It is assumed that the white part is closest and becomes farther away as it becomes blacker.

Proceeding to step S44, the region dividing unit 303 in FIG. 14 has a plurality of similar feature amounts (feature value values are within a predetermined range) by region division (clustering) of the processing target image F (t). Into a set of pixels (region; class). For example, the region dividing unit 303 divides the image into a plurality of regions by clustering in the feature amount space. The clustering by the feature amount space means that each pixel of the image space is mapped to the feature amount space (for example, color, edge, motion vector), and the K-means method, the Mean-Shift method, or the K nearest neighbor in the feature amount space. Clustering is performed by a technique such as a search method (approximate K nearest neighbor search method). After the clustering processing in the feature amount space is finished, the pixel value in the class is replaced with a pixel value (for example, an average value) that is a representative value of each region, and the pixel value in the original image space is replaced with each region. Is assigned to all pixels in each region, and region information R (t) describing the result is output to the depth value setting unit 306 (step S44).

Proceeding to step S45, the depth value setting unit 306 in FIG. 14 sets the depth value of each pixel based on the input distance information Dist (t) and region information R (t). Specifically, as shown in Expression (20), the average value of the distances Dist (x, y) of the pixels in each area indicated by the area information R (t) is scaled, and the depth value D _{base serving} as a reference is scaled. A value shifted by (x, y) is set as the depth value D (x, y) of each pixel (step S45).

In Expression (20), D _max is the upper limit value of the depth value, D _min is the lower limit value of the depth value, Dist _max is the maximum value of the distance information Dist (t), and Dist _min is the minimum value of the distance information Dist (t). The value D _base (x, y) is a predetermined constant for adjusting the reference value of the depth value of each pixel (the depth value as the farthest view). Here, an example of the depth model based on the vanishing point is shown in FIG. In FIG. 18, an image A represents an example of the processing target image F (t), and an image B is a region division result (region division information R (t)) of the processing target image F (t) obtained by the region division unit 303. For example, the image C represents an example of the vanishing point VP of the processing target image F (t), and the image D represents an example of the distance information Dist (t) of the processing target image F (t) obtained by the distance calculation unit 304. The image E is an example of the depth model obtained by the depth value setting unit 306 based on the region division information R (t) of the image B and the distance information Dist (t) of the image D. In the image E of FIG. 18, the bright part represents the near side, and the dark part represents the back.

Returning to step S41 in FIG. 15 again, when there is no vanishing point in the current frame (“vp_num = 0” in vanishing point information VP (t)) (No in step S41), the switching unit 301 in FIG. The switching unit 302 in FIG. 14 switches the output source of the data input to the saliency calculating unit 306 to the saliency calculating unit 306, and the depth model creating means based on the saliency is selected. (Step S46).

The saliency calculating unit 305 in FIG. 14 calculates the saliency M (t) representing the attractiveness in the image based on the human visual characteristics from the input processing target image F (t) (step S47). Examples of portions that are easy for people to pay attention to are locations where the color difference between the pixel of interest and its surrounding pixels is large (local color difference), locations where the color difference between the pixel of interest and the entire image is large, or a local region including the pixel of interest There is a portion (global color difference) where the color difference between the image and the surrounding area is large. Color difference is a quantitative representation of perceptual differences in color. As a color space for evaluating color difference, CIELAB color space (CIE 1976 L * a * b *) is a uniform color space. Also referred to as space). Based on the visual characteristics of the person, the saliency M (x, y) of each pixel is calculated by Equation (21).

Here, in Expression (21), ΔE _local represents a local color difference, ΔE _global represents a _global color difference, and coefficients α and β represent predetermined weight coefficients. That is, Expression (21) expresses the saliency by a linear sum of the local color difference and the global color difference. The local color difference ΔE _local is calculated by equation (22), and the global color difference ΔE _global is calculated by equation (23). In Equations (22) and (23), L * represents a lightness index, a * represents red-green perceptual chromaticity, and b * represents yellow-blue perceived chromaticity. Note that a CIELV color space (also referred to as CIE 1976 L * u * v * color space) may be used as the color space for evaluating the color difference. Note that the coefficient w (u, v) in the equation (22) is the same as that in the equation (12), and thus the description thereof is omitted.

Proceeding to step S48, the depth value setting unit 306 in FIG. 14 sets the depth value of each pixel by the calculation of Expression (24) based on the input saliency M (t), and describes the result. The depth model D (t) is output (step S48).

In Formula (24), D _max is the upper limit value of the depth value, D _min is the lower limit value of the depth value, M _max is the maximum value of the saliency M (t), M _min is the minimum value of the saliency M (t), D _base (x, y) is a predetermined constant for adjusting the reference value of the depth value of each pixel (the depth value as the farthest view). That is, the saliency M (x, y) of each pixel is scaled by the equation (24), and the value shifted by the reference depth value D _base (x, y) is the depth value D (x, y) of each pixel. Set as. Here, an example of the depth model based on the saliency is shown in FIG. In FIG. 19, an image A represents an example of the processing target image F (t), an image B represents an example of the saliency M (t) of the processing target image F (t) obtained by the saliency calculating unit 305, and the image C represents an example of a reference depth model (D _base ), and the image D is created by combining the depth model (D _base ) of the image C with the saliency M (t) of the image B in the depth value setting unit 306. It is an example of a depth model.

In the image B of FIG. 19, a bright part (white) represents a part that is easily noticed by a person (high attractiveness), and a dark part (black) represents a part that is difficult for human attention (low attractiveness). Moreover, in the image D of FIG. 19, it represents that a bright part is a near side, and a dark part represents that it is the back. As shown in the image D of FIG. 19, the depth model creation means based on the saliency superimposes the scaled saliency on the reference depth surface (D _base ), and the region of interest and its surrounding region By emphasizing the difference in relative depth, a pseudo depth feeling is perceived.

When the depth model is generated based on the saliency, the depth of the portion with high saliency (high attraction) is on the near side on the reference depth surface, and the saliency is low (low attraction) ) Set the depth of the part to the back side. Thereby, the relative depth difference between the region of interest and the surrounding region is emphasized, and a pseudo depth sensation can be perceived. In other words, when the saliency of the attention area is higher than the saliency of the surrounding area, the depth is set to be relatively closer to the surrounding area. Further, when the saliency of the attention area is equal to the saliency of the surrounding area, the depth is set to be relatively the same. Further, when the saliency of the attention area is lower than the saliency of the peripheral area, the depth is set to be relatively deep with respect to the peripheral area.

As described above, according to the depth model generation unit 30, when the vanishing point position can be estimated by the geometric depth clue, the depth model of the image is created based on the vanishing point, thereby generating the geometric depth clue. It is possible to create a depth model that emphasizes the sense of depth. In addition, when the vanishing point position cannot be estimated due to the geometric depth cue, the depth model of the image is created from the saliency representing the attractiveness in the image based on the visual characteristics of the person, so that It is possible to create a depth model that emphasizes the sense of depth.

(About the viewpoint image generation unit 40)
Returning to FIG. 2 again, the viewpoint image generation unit 40 of FIG. 1 corresponds to the viewpoint image generation means of the present invention, and each pixel represented by the depth model D (t) based on the assumed viewing condition information set in advance. From the depth value of each pixel, each pixel on the reference image F (t) (input image; processing target image) and the viewpoint image Fi (t) (i = 1, r; Fr: right eye presentation image, Fl: left eye presentation image) ) Calculates a disparity vector (shift amount) representing a shift amount to the corresponding pixel on the above, and based on the pixel on the reference image F (t) and the corresponding calculated disparity vector, each viewpoint image Fi (t) (I = 1, r) is generated (step S15).

Here, “assumed viewing condition information” is information for generating a stereoscopic image (left-eye presentation image, right-eye presentation image) to be presented to the viewer, and the pixel pitch (pixel) of the display that displays the stereoscopic image. Distance) μ, display image size, distance between viewer and display for displaying stereoscopic image (assumed viewing distance) f, parallax range (range of parallax vector) representing depth of stereoscopic image, baseline length t (viewpoint) This represents the distance between the virtual right viewpoint Cr of the image Fr (t) and the virtual left viewpoint Cl of the viewpoint image Fl (t).

FIG. 20 shows an overhead view of an example of a camera (viewpoint) arrangement for generating a viewpoint image based on the assumed viewing condition information. In the example of FIG. 20, assuming that a parallel image is captured, a camera on the virtual right viewpoint Cr and a camera on the virtual left viewpoint Cl are arranged in parallel with the camera on the reference viewpoint Cc in the x-axis direction, respectively. Is observing a point of interest P in a three-dimensional space. Further, the position of the attention point P projected on the image plane Ic of the reference viewpoint Cc is Xc, the position of the attention point P projected on the image plane Il of the virtual left viewpoint Cl is X1, and the image plane Ir of the virtual right viewpoint Cr. Let Xr be the position of the point of interest P projected above. In FIG. 20, the distance (focal length or viewing distance) f to the image plane corresponding to each viewpoint, the distance Z in the z direction from the viewpoint to the point of interest P, the reference viewpoint Cc and each virtual viewpoint (Cr, Cl). The geometrical relationships between the positions Xl and Xc, Xr and Xc of the target point P projected on each image plane using the distance t / 2 in the x direction are expressed by equations (25) and (26), respectively. The

From the above, the disparity vector (shift amount) di (i = 1, r) representing the amount of deviation between the pixel on the reference image F (t) and the corresponding pixel on the viewpoint image Fi (t) (i = 1, r). ) Is derived from equations (27) and (28) obtained by modifying equations (25) and (26).

Note that the variable μ in the equations (27) and (28) represents the pixel pitch. That is, a reference image F (t), a depth model D (t) that is a relative depth value, and a function (Z = z (D (t))) that converts the depth model D (t) into an absolute depth value Z are given. For example, each viewpoint image Fi (t) (i = 1, r) can be generated based on Expression (27) and Expression (28).

Hereinafter, the viewpoint image generation unit 40 will be described based on the above concept. FIG. 21 is a block diagram illustrating a configuration example of the viewpoint image generation unit 40. FIG. 22 is a flowchart for explaining an operation example of the viewpoint image generation unit 40. FIG. 23 is a diagram for explaining a viewpoint image generation example in the viewpoint image generation unit 40. First, as illustrated in FIG. 21, the viewpoint image generation unit 40 includes a disparity vector calculation unit 401, a texture shift unit 402, a gap filling unit (also referred to as an occlusion compensation unit) 403, and a floating window superimposing unit 404. .

In FIG. 22, the disparity vector calculation unit 401 in FIG. 21 converts the input assumed viewing condition information, the depth model D (t), and the depth model D (t) into an absolute depth value (Z = z (D ( t))) and the disparity vectors di (i = i = i) from the equations (27) and (28) to the respective pixels on the reference image F (t) and the corresponding pixels on each viewpoint image Fi (t). l, r) is calculated, and the result is output to the texture shift unit 402 (step S51 in FIG. 22). In addition to calculating the disparity vector of each pixel based on Expressions (27) and (28), the disparity vector calculation method is based on the assumed viewing condition information in advance as shown in the LUT of FIG. You may calculate using the lookup table which derives a parallax vector from the depth value (relative depth value) set based on. Note that g (D) in FIG. 23B represents a function for converting the depth value D into a disparity vector. Here, the spreading direction and the crossing direction of the disparity vector (shift amount) in FIG. 23B will be described with reference to FIG. In FIG. 24, let P be a certain point of interest, Pr be the point of interest P projected onto the display surface when viewed from the right eye, and Pl be the point of interest P projected onto the display surface when viewed from the left eye. At this time, as shown in FIG. 24A, the disparity vector in the spreading direction is such that a certain point of interest P is located behind the display surface, and the disparity vector from Pr to Pl on the display surface, or Pl This is a case where the value of the parallax vector from to Pr becomes positive. Similarly, as shown in FIG. 24B, the disparity vector in the cross direction is such that a certain point of interest P is located in front of the display surface, and the disparity vector from Pr to Pl on the display surface, or Pl to Pr. This is a case where the value of the parallax vector to becomes negative. When the value of the parallax vector is zero, the attention point P is located on the display surface.

Subsequently, the texture shift unit 402 in FIG. 21 converts each pixel (x, y) of the reference image F (t) to each viewpoint image Fi (t) based on the corresponding disparity vector di (i = 1, r). ) (I = 1, r) and the corresponding pixel value of the pixel (u, v), and the generated viewpoint image is output to the gap filling unit 403 (step S52 in FIG. 22). When the pixel value is set, texture shift is performed from a pixel whose disparity vector value has a value on the spreading direction side (for example, d2 on the LUT in FIG. 23B).

For example, in FIG. 23A, a case where a virtual left viewpoint viewpoint image Fl (t) (left eye presentation image) is generated from the reference image iF and the depth model iD is considered. In the depth model iD, the depth value of the white portion is D1 and the depth value of the black portion is represented by D2. At this time, when texture shift is performed based on the LUT in FIG. 23B, first, each pixel of the layer L2 having the depth value D2 in FIG. 23A is shifted by d2 in the spreading direction. After that, when each pixel of the layer L1 having the depth value D1 in FIG. 23A is shifted by d1 in the intersecting direction, a defective region Gs1 that is not located at the left end / right end of the screen and a defective region Gl1 that is located at the left end / right end of the screen A viewpoint image oF1 having is obtained. Here, the missing area (occlusion area) represents an area in which no pixel value is set because there is no corresponding pixel on the reference image in the viewpoint image oF1 in FIG.

Subsequently, the gap filling unit 403 in FIG. 21 in the input viewpoint image Fi (t) (i = 1, r), the missing region that is not located at the screen edge (for example, the viewpoint image oF1 in FIG. 23A). The pixel of Gs1) is interpolated from the pixel group located around the defect area, and the interpolated viewpoint image Fi (t) (i = 1, r) is output to the floating window superimposing unit 404 (step S53 in FIG. 22). . Note that, for example, linear interpolation, a median filter, or a known image restoration method (see, for example, Non-Patent Document 6) is used as a method for interpolating a pixel in a defective region.

Subsequently, the floating window superimposing unit 404 of the input viewpoint images Fi (t) (i = 1, r) has a defect area located at the screen edge (for example, the viewpoint image oF1 in FIG. 23A). In Gl1), the maximum width W1 of the defect region is obtained. Subsequently, a floating window (black band) having a width W2 (= αW1) is inserted into the right end and the left end of each viewpoint image, and the result is output (step S54 in FIG. 22). W2 is a value obtained by scaling W1 with a predetermined constant α. Further, the viewpoint image after the floating window is inserted is, for example, the viewpoint image oF2 of FIG. In the viewpoint image oF2 in FIG. 23A, floating windows fw1 and fw2 are inserted at the left end and the right end of the screen, respectively. The reason for inserting the floating window is that the position or shape of a certain target is extremely different in the image presented to the left eye / right eye (for example, the generated viewpoint image is located at the left end / right end of the screen). This is to suppress a visual field struggle that alternately perceives left and right retinal images caused by the inability to see both eyes as one object.

As described above, according to the present embodiment, when the vanishing point position can be estimated by the geometric depth cue, the depth model of the image is generated based on the vanishing point, so that the depth sensation by the geometric depth cue can be obtained. An enhanced stereoscopic image can be generated. In addition, when the vanishing point position cannot be estimated due to the geometric depth cue, the depth model of the image is generated from the saliency representing the attractiveness in the image based on the human visual characteristics, thereby A stereoscopic image in which the sense of depth is emphasized can be generated.

(Modification of depth model generation unit 30 (depth model generation unit 30a))
In the embodiment described above, the depth model generation unit 30 scales the saliency M (x, y) of each pixel according to the equation (24) as an example of the depth model creation unit based on the saliency, and becomes a reference depth value D. _{Although the case} where the value shifted by _base (x, y) is set as the depth value D (x, y) of each pixel has been described, the present invention is not limited to this. For example, the configuration of the depth model generation unit 30 may be changed so that the switching unit 301 is removed as illustrated in FIG. 25 and the image F (t) is input to the region division unit 303 and the saliency calculation unit 305. . That is, the depth model generation unit 30a includes a switching unit 302, an area dividing unit 303, a distance calculating unit 304, a saliency calculating unit 305, and a depth value setting unit 306.

An operation example of the depth model creating means based on the saliency in this case will be described with reference to FIG. Note that the operation of the depth model creation means based on the vanishing point is the same as that in steps S42 to S45 in FIG. 15, and will not be described here.

First, the saliency calculating unit 305 in FIG. 25 calculates the saliency M (t) from the processing target image F (t) by the same processing as step S47 in FIG. 15, and the result is sent to the depth value setting unit 306. This is output (step S47 ′ in FIG. 26). Subsequently, the area dividing unit 303 in FIG. 25 divides the processing target image F (t) into areas by processing similar to that in step S44 in FIG. 15, and sets area information R (t) describing the result to the depth value setting. It outputs to the part 306 (step S48 'of FIG. 26).

Thereafter, the depth value setting unit 306 in FIG. 25, based on the input saliency M (t) and the area information R (t), each area information R (t) indicates, as shown in Expression (29). The average value of the saliency M (x, y) of the pixels in the region is scaled, and the value shifted by the reference depth value D _base (x, y) is used as the depth value D (x, y) of each pixel. Setting is made (step S49 'in FIG. 26).

In Expression (29), D _max is the upper limit value of the depth value, D _min is the lower limit value of the depth value, M _max is the maximum value of the saliency M (t), and M _min is the minimum value of the saliency M (t). The value D _base (x, y) is a predetermined constant for adjusting the reference value of the depth value of each pixel (the depth value as the farthest view). Here, an example of the depth model based on the saliency in the modification is shown in FIG. In FIG. 27, image A represents an example of the processing target image F (t), and image B is an example of the region division result (region information M (t)) of the processing target image F (t) obtained by the region dividing unit 303. The image C represents the saliency M (t) of the processing target image F (t) obtained by the saliency calculating unit 305, the image D represents an example of a reference depth model (D _base ), and the image E Is an example of the depth model obtained by the depth value setting unit 306 based on the region information R (t) of the image B, the saliency M (t) of the image C, and the depth model (D _base ) as a reference of the image D. is there. In the image C of FIG. 27, a bright part (white) represents a part that is easily noticed by a person (high attraction), and a dark part (black) represents a part that is difficult for a person to pay attention (low attraction). In the image E of FIG. 27, the bright part represents the near side, and the dark part represents the back. In addition, regarding the reference depth model (D _base ), in the image D of FIG. 27, a plane having the same depth value is given as an example, but the present invention is not limited to this. For example, the plane equation shown in the following equation (30) may be determined in advance, and the reference depth value D _base (x, y) may be set by the coordinates (x, y) of each pixel. An example of the depth model serving as a reference expressed by the equation (30) is shown in an image A of FIG. Image A in FIG. 28 is a depth model that serves as a reference in which the coefficients a, b, and c in Expression (30) are set so that the depth is closer to the lower end and the depth is closer to the upper end. The result of the depth model D (t) created when the image A in FIG. 28 is input instead of the image D in FIG. 25 is shown in an image B in FIG.

As described above, when the vanishing point position cannot be estimated due to the geometric depth cue, the depth model generation unit 30a and the saliency representing the attractiveness in the image based on the visual characteristics of the person and the image segmentation result ( By setting a uniform depth value for each region based on (region information), it is possible to generate a depth model that suppresses errors in the depth context.

(Modification of scene change detection unit 10)
In the above embodiment, the scene change detection unit 10 has been described using the luminance histogram as the image feature amount used for scene change detection, but the present invention is not limited to this. For example, instead of the luminance histogram, a color histogram representing the appearance frequency of each color component, an average error of inter-frame differences, and a motion vector distribution may be used as the image feature amount.

(Modification of the edge detection unit 211)
In the above embodiment, the edge detection unit 211 has described edge detection in which a point (Local Maxima) where the edge intensity is locally maximum in the image space is extracted as an edge point, but the present invention is not limited to this. For example, a known edge detection method such as Canny Edge detection may be used. Further, as a differential operator (edge detector), a known method such as a Sobel filter, a Prewitt filter, a LoG filter (Laplacian of Gaussian), or a DoG filter (Difference of Gaussian) is used. May be.

(Modification of Vanishing Point Identification Unit 213)
In the said embodiment, although the vanishing point identification part 213 demonstrated the case where Gaussian distribution was used as a distribution model used for a mixed model, this invention is not limited to this. For example, an exponential distribution family (Laplace distribution, beta distribution, Bernoulli distribution, etc.) may be used for the distribution model. The vanishing point identifying unit 213 may determine the number of classes Kc used in the mixed model as a predetermined value and determine the value as in the following example. The vanishing point identifying unit 213 sets a predetermined class number Kc ′ as the class number Kc, and performs clustering by the K-means method. Thereafter, when there is a class Ci and a class Cj where the distance between classes satisfies a predetermined threshold value or less (or less), the vanishing point identifying unit 213 merges the class Ci and the class Cj to form a new class Ck ′. Process. The vanishing point identifying unit 213 determines the number of classes Kc (≦ Kc ′) by repeating this process until the number of classes converges to a constant value. Note that the method used by the vanishing point identifying unit 213 to estimate the distribution model of the intersection is not limited to a parametric estimation method such as a mixed model, but is also a Mean-shift method, a K-means method, a K nearest neighbor search method (approximate K). A non-parametric estimation method such as a nearest neighbor search method may be used.

(Modification of the area dividing unit 303)
In the above embodiment, the case in which the region dividing unit 303 performs clustering in the feature amount space has been described, but the present invention is not limited to this, and clustering in the image space may be performed. Clustering in the image space is a method for performing region division based on the similarity between pixels or pixel groups (regions) constituting the region in the original image space without mapping to the feature amount space. is there. For example, the region dividing unit 303 uses (a) pixel combination method, (b) region growth method (also referred to as Region Growing method), and (c) region division integration method (also referred to as Split & Merge method) in the image space. Clustering may be performed.

(Modification of saliency calculation unit 305)
In the above embodiment, the case where the saliency calculating unit 305 calculates the saliency based on the local color difference and the global color difference has been described. However, the present invention is not limited to this, and the local color difference (expression ( The saliency may be calculated based on one of the indicators of the first term ΔE _local ) in 21) or the global color difference (second term ΔE _{global in} equation (21)). Alternatively, the color difference may be calculated using only the lightness index L * without using the red-green perceptual chromaticity a * and the yellow-blue perceptual chromaticity b *. In this case, in a human visual characteristic, a portion having a large brightness contrast (contrast difference) represents high attractiveness. Further, the local color difference ΔE _local and the global color difference ΔE _global are expressed by the following equation (31), using the lightness difference ΔL *, chroma difference ΔC *, and hue difference ΔH * based on the CIE method, respectively. You may obtain | require by Formula (32) (for example, refer nonpatent literature 4).

Note that the coefficient w (u, v) in the equation (31) is the same as that in the equation (12). Further, the coefficients l, c, h in the expressions (31) and (32) are predetermined weighting coefficients. The method of obtaining the saliency is not limited to the color difference, and the saliency may be calculated based on a plurality of image feature amounts such as a color difference, an edge gradient, and a motion vector (for example, see Non-Patent Document 5).

(First Modification of Stereoscopic Image Generation Device 1)
In the above embodiment, the input / output image size of the scene change detection unit 10, the vanishing point estimation unit 20, and the depth model generation unit 30 of the stereoscopic image generation device 1 is assumed to be the same as the input image F (t). However, the present invention is not limited to this. For example, in order to reduce the amount of computation and the memory size, images input to the scene change detection unit 10, the vanishing point estimation unit 20, and the depth model generation unit 30 are reduced in advance to a predetermined image size, and the depth You may add and implement the process which expands the depth model output from the model production | generation part 30 to input image size. That is, the first modification (stereoscopic image generating apparatus 2) of the stereoscopic image generating apparatus 1 includes a reduction processing unit 50, a scene change detecting unit 10, a vanishing point estimating unit 20, and a depth model generating unit as shown in FIG. 30, an enlargement processing unit 60, and a viewpoint image generation unit 40. The reduction processing unit 50 corresponds to reduced image generation means of the present invention, and generates a reduced image having a predetermined image size from the input image F (t). Then, the generated reduced image is input to the vanishing point estimation unit 20 and the depth model generation unit 30. The enlargement processing unit 60 corresponds to the enlarged depth model generation unit of the present invention, and generates an enlarged depth model having the same image size as the input image F (t) from the depth model of the reduced image generated by the depth model generation unit 30. .

An example of the operation of the stereoscopic image generating apparatus 2 will be described with reference to FIG. Each operation (step S63, step S64, step S65, step S67 in FIG. 30) of the scene change detection unit 10, the vanishing point estimation unit 20, the depth model generation unit 30, and the viewpoint image generation unit 40 in FIG. Each operation of the scene change detection unit 10, the vanishing point estimation unit 20, the depth model estimation unit 30, and the viewpoint image generation unit 40 of the stereoscopic image generation apparatus 1 shown in FIG. 1 (step S12 in FIG. Since it is the same as S13, step S14, and step S15), description thereof is omitted.

30, first, the stereoscopic image generation device 2 in FIG. 29 outputs the input image at time t to the reduction processing unit 50 and the viewpoint image generation unit 40 (step S61 in FIG. 30).

29 reduces the input processing target image F (t) to a predetermined image size, converts the reduced image Fd (t) to the scene change detection unit 10, the vanishing point estimation unit 20, and It outputs to the depth model production | generation part 30 (step S62 of FIG. 30). Note that image reduction is performed using, for example, a nearest neighbor method, a bilinear method, or a bicubic method.

29 enlarges the input depth model D (t) to the image size of the input image F (t), and outputs the enlarged depth model Du (t) to the viewpoint image generation unit 40 (FIG. 30 step S66). The depth model is enlarged using, for example, the nearest neighbor method, the bilinear method, or the bicubic method.

According to the stereoscopic image generating apparatus 2, scene change detection processing, vanishing point estimation processing, and depth model generation processing are performed using a reduced image having an image size smaller than the input image, and therefore, compared with the stereoscopic image generating apparatus 1 of FIG. 1. Thus, the memory size and the amount of calculation can be reduced.

(Second Modification of Stereoscopic Image Generation Device 1)
In the above embodiment, when the vanishing point position can be estimated by the geometric depth cue, the stereoscopic image generation apparatus 1 generates a depth model of the image based on the vanishing point and estimates the vanishing point position by the geometric depth cue. When it is not possible, the depth model of the image is generated based on the saliency representing the attractiveness in the image based on the visual characteristics of the person. For this reason, in the frames before and after the depth model generation unit is switched, the depth model is different in the time direction, so that the change in parallax (depth) is considered to be large. Similarly, in the frames before and after the scene change occurs, since the depth model is different in the time direction, it is considered that the change in parallax (depth) increases. Therefore, in the second modification of the stereoscopic image generating device 1 (stereoscopic image generating device 3 in FIG. 31), the spatio-temporal direction in which the depth model is smoothed in the spatiotemporal direction in order to reduce the change in parallax in the temporal direction. A smoothing unit 70 is provided between the depth model generation unit 30 and the viewpoint image generation unit 40. That is, the stereoscopic image generation device 3 includes a scene change detection unit 10, a vanishing point estimation unit 20, a depth model generation unit 30, a spatio-temporal direction smoothing unit 70, and a viewpoint image generation unit 40 as illustrated in FIG. Is done. The spatio-temporal direction smoothing unit 70 corresponds to the spatio-temporal direction smoothing means of the present invention, and includes a spatial direction smoothing unit 701, a time direction smoothing unit 702, and a buffer 703 as shown in FIG. The The spatiotemporal direction smoothing unit 70 smoothes the depth model D (t) of the processing target image F (t) generated by the depth model generation unit 30 in the spatial direction, and the image F (t) smoothed in the spatial direction. Based on the depth model Ds (t) and the depth model Dt (t−1) smoothed in the spatio-temporal direction of the comparison target image F (t−1) in the past than the image F (t), the image F The depth model Ds (t) of (t) is smoothed in the time direction, and the depth model Dt (t) smoothed in the spatio-temporal direction of the image F (t) is generated.

An example of the operation of the stereoscopic image generating apparatus 3 will be described with reference to FIGS. In addition, each operation | movement (step S72, step S73, step S74, step S76 of FIG. 33) of the scene change detection part 10, the vanishing point estimation part 20, the depth model generation part 30, and the viewpoint image generation part 40 of FIG. Each operation of the scene change detection unit 10, the vanishing point estimation unit 20, the depth model estimation unit 30, and the viewpoint image generation unit 40 of the stereoscopic image generation apparatus 1 shown in FIG. 1 (step S12 in FIG. Since it is the same as S13, step S14, and step S15), description thereof is omitted.

(About the spatio-temporal direction smoothing unit 70)
The spatiotemporal direction smoothing unit 70 in FIG. 31 performs a smoothing process in the spatiotemporal direction on the depth model D (t) of the input processing target image F (t), and the result (smoothed depth model Dt ( t) is output (step S75 in FIG. 33).

Specifically, the spatial direction smoothing unit 701 in FIG. 32 smoothes the depth model D (t) in the spatial direction using a one-dimensional smoothing filter in the order of the horizontal direction, the vertical direction, or the vertical direction and the horizontal direction. The result (depth model Ds (t)) is output to the time direction smoothing unit 702 (step S81 in FIG. 34). As the one-dimensional smoothing filter, for example, a one-dimensional Gaussian filter is used.

32 is based on the input depth model Ds (t) smoothed in the spatial direction and the smoothed depth model Dt (t−1) of the previous frame stored in the buffer 703. Then, the smoothed depth model Dt (t) is generated by the following equation (33), and the result is output to the buffer 703 and the outside (step S82 in FIG. 34). The coefficient α in the equation (33) is a predetermined value between 0 and 1.

32 deletes the smoothed depth model Dt (t−1) of the previous frame and stores the input smoothed depth model Dt (t) of the current frame (step S83 in FIG. 34).

According to the stereoscopic image generating device 3, by changing the depth model in the spatio-temporal direction, the change in parallax (depth) in the frames before and after the depth model generating means is switched and in the frames before and after the scene change occurs. Can be reduced.

As described above, the embodiments of the stereoscopic image generating apparatus according to the present invention have been mainly described. However, the present invention may be in the form of a stereoscopic image generating method by the stereoscopic image generating apparatus 1. Moreover, it is good also as a form of the program for making a computer perform this stereo image production | generation method.

That is, a part of the stereoscopic image generating apparatus 1 in the above-described embodiment may be realized by a computer. In that case, a program for realizing this control function may be recorded on a computer-readable recording medium, and the program recorded on this recording medium may be read into a computer system and executed. Here, the “computer system” is a computer system built in the stereoscopic image generation apparatus 1 and includes an OS and hardware such as peripheral devices. The “computer-readable recording medium” refers to a storage device such as a flexible medium, a magneto-optical disk, a portable medium such as a ROM or a CD-ROM, and a hard disk incorporated in a computer system. Further, the “computer-readable recording medium” is a medium that dynamically holds a program for a short time, such as a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line, In such a case, a volatile memory inside a computer system serving as a server or a client may be included and a program that holds a program for a certain period of time. The program may be a program for realizing a part of the functions described above, and may be a program capable of realizing the functions described above in combination with a program already recorded in a computer system.

Further, a part or all of the stereoscopic image generating apparatus 1 in the above-described embodiment may be realized as an integrated circuit such as an LSI (Large Scale Integration). Each functional block of the stereoscopic image generating apparatus 1 may be individually made into a processor, or a part or all of them may be integrated into a processor. Further, the method of circuit integration is not limited to LSI, and may be realized by a dedicated circuit or a general-purpose processor. Further, in the case where an integrated circuit technology that replaces LSI appears due to progress in semiconductor technology, an integrated circuit based on the technology may be used.

As described above, the embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to that described above, and various design changes and the like can be made without departing from the scope of the present invention. It is possible to

1, 2, 3, 3D image generation device, 10, scene change detection unit, 20, vanishing point estimation unit, 21, intra-frame vanishing point estimation unit, 22, interframe vanishing point estimation unit, 30, depth model generation unit, 40: viewpoint image generation unit, 50 ... reduction processing unit, 60 ... enlargement processing unit, 70 ... spatiotemporal direction smoothing unit, 101 ... luminance histogram generation unit, 102, 203, 204, 703 ... buffer, 103 ... histogram similarity Calculation unit 104... Scene change determination unit 201, 202, 301, 302... Switching unit 211 211 Edge detection unit 212 Straight line detection unit 213 Vanishing point identification unit 221 Feature point detection unit 222 Point calculation unit, 223 ... transformation matrix calculation unit, 224 ... vanishing point position calculation unit, 303 ... area division unit, 304 ... distance calculation unit, 305 ... saliency calculation unit, 306 ... depth value setting unit, 01 ... disparity vector calculation unit, 402 ... texture shift unit, 403 ... gap filling section, 404 ... floating window superimposing unit, 701 ... spatial direction smoothing unit, 702 ... time direction smoothing unit.

Claims (13)

1. A stereoscopic image generating apparatus that adds binocular stereoscopic information to a 2D image and generates a 3D image,
   Vanishing point estimating means for estimating vanishing points from the processing target image;
   A depth model generating means for generating a different depth model based on whether or not the vanishing point can be estimated by the vanishing point estimating means;
   A viewpoint image generation unit that generates a right eye presentation image and a left eye presentation image based on the depth model generated by the depth model generation unit, the processing target image, and the assumed viewing condition information;
   When the vanishing point can be estimated by the vanishing point estimating unit, the depth model generating unit generates a depth model based on the vanishing point, and when the vanishing point cannot be estimated by the vanishing point estimating unit, A three-dimensional image generation apparatus that generates a depth model based on the saliency of each pixel in the processing target image.
2. A reduced image generating means for generating a reduced image of a predetermined image size from the processing target image;
   The reduced image is input to the vanishing point estimating means and the depth model generating means, and an enlarged depth model having the same image size as the processing target image is generated from the depth model of the reduced image generated by the depth model generating means. The stereoscopic image generation apparatus according to claim 1, further comprising an enlarged depth model generation unit.
3. The depth model of the processing target image generated by the depth model generation means is smoothed in the spatial direction, the depth model of the processing target image smoothed in the spatial direction, and a comparison target image in the past than the processing target image. Based on the depth model smoothed in the spatiotemporal direction, the depth model of the processing target image is smoothed in the temporal direction, and the depth model smoothed in the spatiotemporal direction of the processing target image is generated. The stereoscopic image generating apparatus according to claim 1, further comprising a smoothing unit.
4. The assumed viewing condition information includes the pixel pitch of the display that displays the 3D image, the image size of the display, the distance from the viewer to the display, the parallax range that represents the depth of the 3D image, and between the left and right virtual viewpoints The three-dimensional image generation apparatus according to any one of claims 1 to 3, further comprising a base line length that is a distance.
5. The degree of saliency of each pixel in the processing target image is a location where the color difference between the target pixel and its surrounding pixels is large, a location where the color difference between the target pixel and the entire image is large, or a local region including the target pixel and its The three-dimensional image generation apparatus according to any one of claims 1 to 4, wherein a part having a larger color difference from the surrounding area is calculated to be higher.
6. When the vanishing point cannot be estimated by the vanishing point estimating unit, the depth model generating unit generates the depth model such that a location where the saliency of each pixel in the processing target image is high is on the near side. The stereoscopic image generating apparatus according to claim 5, wherein the apparatus is a stereoscopic image generating apparatus.
7. The vanishing point estimating means includes an intra-frame vanishing point estimating means for estimating a vanishing point of the processing target image from straight line information in the processing target image, and a comparison target image that is earlier than the processing target image and the processing target image. 7. The inter-frame vanishing point estimating means for estimating the vanishing point of the processing target image based on the vanishing point position in the comparison target image. The three-dimensional image generation device described.
8. A scene change detecting means for detecting whether or not a scene change has occurred between the processing target image and the comparison target image, and when a scene change is detected by the scene change detecting means, the in-frame vanishing point estimating means; The stereoscopic image generating apparatus according to claim 7, wherein when a scene change is detected and no scene change is detected by the scene change detection unit, the inter-frame vanishing point estimation unit is selected.
9. A storage unit that stores vanishing point information including the position of the vanishing point of the comparison target image, and the vanishing point information of the comparison target image is stored in the storage unit; The stereoscopic image generating apparatus according to claim 8, wherein, when the vanishing point information of the comparison target image is not stored in the storage unit, the intra-frame vanishing point estimation unit is selected.
10. 10. The three-dimensional image generation apparatus according to claim 7, wherein the comparison target image is an image immediately before the processing target image.
11. A stereoscopic image generation method by a stereoscopic image generation apparatus that adds binocular stereoscopic information to a 2D image and generates a 3D image,
    The three-dimensional image generation device, a vanishing point estimation step for estimating a vanishing point from the processing target image,
    A depth model generation step for generating a different depth model based on whether or not the vanishing point could be estimated in the vanishing point estimation step;
    A viewpoint image generation step of generating a right eye presentation image and a left eye presentation image based on the depth model generated in the depth model generation step, the processing target image, and the assumed viewing condition information;
    In the depth model generation step, when the vanishing point can be estimated in the vanishing point estimation step, a depth model is generated based on the vanishing point, and the vanishing point cannot be estimated in the vanishing point estimation step. A depth model is generated based on the saliency of each pixel in the processing target image.
12. A program for causing a computer to execute the stereoscopic image generation method according to claim 11.
13. A computer-readable recording medium on which the program according to claim 12 is recorded.

Images