JP5431393B2 - Stereo image generating apparatus and method - Google Patents

Description

  Embodiments described herein relate generally to a parallax image generation apparatus and method.

  In recent years, development of consumer-use stereoscopic display devices has become active, while many images are two-dimensional images. Therefore, a method for generating a stereoscopic image from a two-dimensional image has been proposed. In order to generate a stereoscopic image, it may be necessary to generate a viewpoint image that is not included in the original image. In that case, it is necessary to interpolate a pixel or the like in a portion (hereinafter referred to as a hidden surface area) that is hidden behind an object in the original image.

  Therefore, a method for interpolating the pixel values of the hidden surface area has been proposed. There is a technique for generating a pixel value of a hidden surface area generated when generating a three-dimensional image from a two-dimensional image based on a pixel value corresponding to a pixel at an end of a partial image adjacent to the hidden surface area. In the above-described prior art, when interpolating the pixel value of the hidden surface area in the parallax image, the pixel value representing the object on the near side may be interpolated even though the hidden surface area is the back side area. .

JP 2004-295859 A

  The problem to be solved by the present invention is to generate a parallax image with less influence of the hidden surface.

In order to solve the above-described problem, the stereoscopic image generation apparatus according to the embodiment includes two units for generating a parallax image having a viewpoint different from the image from at least one image and depth information corresponding to the image. For each set including the above viewpoints, a larger evaluation value is calculated from the depth information as the hidden surface area generated when the parallax image is created is larger, and is calculated for the viewpoint set. A selection unit that selects the set of viewpoints with which the evaluation value is minimized, and so as to suppress a change in viewpoint on the time axis when the image is a moving image compared to a case where the image is not a moving image _, Using the selected set of viewpoints and the set of viewpoints respectively selected in a plurality of consecutive frames before the image including at least the previous frame of the image, the corrected viewpoint of the image Comprising a viewpoint control unit for generating and outputting a set, the parallax image of the view corresponding to the combination of the viewpoint, which is the modified output from the viewpoint control unit, and generate unit that generates from the image.

The figure which shows the stereo image production | generation apparatus of 1st Embodiment. The figure explaining the relationship between the pixel position of an image, and the coordinate of a horizontal / vertical direction. The conceptual diagram which shows the relationship between parallax amount and a depth value. The figure explaining a viewpoint axis. The figure which shows a viewpoint axis | shaft at the time of image | photographing a some camera side by side vertically and horizontally. The figure showing an example of an input image and the depth information corresponding to it. The figure which shows the distribution of the depth in the plane which passes along line segment MN, and the group of a viewpoint. The figure which shows the example of the parallax image produced | generated from the input image. The figure which shows operation | movement of a stereo image production | generation apparatus. The figure which shows an example of the detailed operation | movement which a calculation part performs. The figure which shows the example of a change of the detailed operation | movement which a calculation part performs. The conceptual diagram which shows the relationship between a viewpoint and a hidden surface area. The conceptual diagram which shows the relationship between a viewpoint and a hidden surface area. The figure which shows the example of a change of the detailed operation | movement which a calculation part performs. The figure which shows the stereo image production | generation apparatus of 2nd Embodiment. The figure which shows the three-dimensional image generation apparatus of 3rd Embodiment.

  Hereinafter, embodiments of the present invention will be described. In addition, the same code | symbol is attached | subjected to the structure and process which mutually perform the same operation | movement, and the overlapping description is abbreviate | omitted.

(First embodiment)
The stereoscopic image generation apparatus according to the present embodiment generates a parallax image at a viewpoint different from the input image from at least one input image and depth information corresponding to the input image. The parallax image generated by the stereoscopic image generating apparatus according to the present embodiment may be any method as long as stereoscopic display is possible. Either field sequential or frame sequential may be used, but in this embodiment, a frame sequential case is illustrated. Further, the input image may be not only a two-dimensional image but also a stereoscopic image.

  The depth information may be depth information prepared in advance by the image provider. Further, it may be depth information estimated from an input image by some estimation method.

  Further, the depth information may be information obtained by processing the dynamic range such as compression / decompression. There are various methods for supplying the input image and the depth information. For example, there is a method of acquiring at least one input image and depth information corresponding to the input image through a tuner or by reading information stored on an optical disc. Further, a method may be used in which a depth value is estimated at a stage before a two-dimensional image or a stereoscopic image with parallax is supplied from the outside and input to the stereoscopic image generation apparatus.

  FIG. 1 is a diagram illustrating a stereoscopic image generating apparatus according to the present embodiment. The stereoscopic image generation apparatus includes a calculation unit 101, a selection unit 102, and a parallax image generation unit 103. The stereoscopic image generation device generates a parallax image with a viewpoint different from that of the input image from the depth information corresponding to the input image. A viewer can perceive a stereoscopic image by displaying parallax images having parallax.

  The calculation unit 101 calculates an evaluation value that increases as the hidden surface area generated when a parallax image is created for each set of candidate viewpoints using depth information (only). The calculated evaluation value is sent to the selection unit 102 in association with the combination information. Note that the calculation unit 101 does not need to actually create a parallax image, and only needs to be able to estimate the area of the hidden surface area generated by the set of assumed viewpoints. In the present embodiment, the size of the hidden surface area indicates the total number of pixels belonging to the hidden surface region. Any number of viewpoints may be included in one set as long as the number is two or more. The candidate viewpoint indicates a virtual position where an image is captured in advance.

  The selection unit 102 selects one candidate viewpoint set based on the evaluation value calculated by the calculation unit 101 for each set. As a selection method, it is preferable to select a set of candidate viewpoints having the smallest evaluation value. Among the plurality of candidate viewpoints, the viewpoint that generates the smallest shade area when the parallax image is created is selected as the viewpoint for generating the parallax image.

  The parallax image generation unit 103 generates a parallax image of a viewpoint corresponding to the set of viewpoints selected by the selection unit 102.

  Hereinafter, the virtual viewpoint where the input image is taken is referred to as a first viewpoint. Note that the first viewpoint may include a plurality of viewpoints, such as when the input image has a plurality of images captured from a plurality of viewpoints. A viewpoint included in the viewpoint set selected by the selection unit 102 is referred to as a second viewpoint set. The parallax image generation unit 103 generates a virtual image taken from the position of the second viewpoint.

  FIG. 2 is a diagram illustrating the relationship between the pixel position of an image (including an input image and a parallax image) and the horizontal and vertical coordinates. The pixel positions of the image are displayed as gray circles, and the horizontal and vertical axes are described. In this way, each pixel position is assumed to be an integer position in horizontal and vertical coordinates. Hereinafter, unless otherwise specified, the vector starts from the upper left corner (0, 0) of the image.

  FIG. 3 is a conceptual diagram illustrating the relationship between the amount of parallax and the depth value. The x axis is an axis along the horizontal direction of the screen. The z axis is an axis along the depth direction. The greater the depth, the deeper the position from the imaging position. z = 0 indicates a virtual position on the display surface. The straight line DE is on the display surface. Point B indicates the first viewpoint. Point C indicates the second viewpoint. In FIG. 3, since it is assumed that the viewing position is viewed in parallel to the screen, the straight line DE and the straight line BC are parallel. Let b be the distance between points BC. The object is disposed at a point A having a depth Za. Here, the depth Za is a vector whose depth direction is positive. A point D indicates a position where the object is displayed on the input image. The pixel position on the screen of the point D is represented by a vector i. Point E indicates a position when the object is displayed on the generated parallax image. That is, the length of the line segment DE is the amount of parallax.

  FIG. 3A is a diagram illustrating a relationship between a depth value and a parallax amount when an object behind the screen is displayed. FIG. 3B is a diagram illustrating the relationship between the depth value and the amount of parallax when an object on the near side of the screen is displayed. In FIGS. 3A and 3B, the positional relationship between the point D and the point E on the x-axis is reversed. In order to reflect the positional relationship between the point D and the point E, a disparity vector d (i) starting from the point D and ending at the point E is defined. The element value of the disparity vector follows the x axis. If the parallax vector is defined as shown in FIG. 3, the parallax amount at the pixel position i is represented by a vector d (i).

Now, assuming that the vector from the viewer to the screen is Zs, the triangle ABC and the triangle ADE are similar to each other, so | Za + Zs |: | Za | = b: | d (i) | When this is solved for d (i), the following equations are established to set the x-axis and z-axis as shown in FIG.
That is, the disparity vector d (i) is uniquely determined from the depth Za (i) at the pixel position i. Therefore, in the following description, a portion described as a disparity vector can be read as a depth value.

  Next, the viewpoint will be described with reference to FIGS.

  FIG. 4 is a diagram illustrating the viewpoint axis. FIG. 4A shows the relationship between the screen and the viewpoint viewed from the same direction as in FIG. Point L indicates the position of the left eye, point R indicates the position of the right eye, and point B indicates the shooting position of the input image. A viewpoint axis that is positive in the right direction in the figure passing through the points L, B, and R and whose origin is the point B is defined.

4B also shows the relationship between the screen and the viewpoint viewed from the same direction as in FIG. In this case, the viewpoint axis when the images taken from the two viewpoints of the point S and the point T are input is positive in the right direction of the figure as in FIG. It is the axis with the origin at the middle, that is, the midpoint between points S and T. This axis is parallel to the straight line BC in FIG. On the viewpoint axis, coordinates obtained by normalizing the average human interocular distance to 1 instead of the real space distance are used. In such a definition, in FIG. 4, the point R is located at 0.5 on the viewpoint axis, and the point L is located at -0.5. In the following description, the viewpoint is represented by coordinates on the viewpoint axis. In this way, Expression (1) can be written as Expression (2) as a function corresponding to the viewpoint (scale).

In this way, for example, the pixel value I (i, 0.5) at the pixel position i of the image viewed from the viewpoint 0.5 can be expressed as in Expression (3).

  Moreover, although the case where the input image is a video from one viewpoint has been illustrated, the viewpoint axis can be set similarly even when two or more parallax images are provided. The viewpoint axis can be set assuming that the image for the left eye is taken from -0.5 on the viewpoint axis and the image for the right eye is taken from 0.5. Further, even when an image captured by arranging a plurality of cameras in the vertical and horizontal directions as shown in FIG. 5 is input, it is possible to determine viewpoint axes such as the v-axis and h-axis in FIG.

  The relationship between the number of pixels belonging to the hidden surface area and the viewpoint will be described with reference to FIGS.

  FIG. 6 is a diagram illustrating an example of an input image and depth information corresponding to the input image. The depth information indicates a position closer to the viewer side as it is blacker.

  FIG. 7 is a diagram showing a set of depth distribution and viewpoints in a plane passing through the line segment MN assumed on the input image of FIG. A line indicated by a bold line represents a depth value. Assume a set of two viewpoints (L, R) and (L ', R'). L and L ′ represent the left eye viewpoint, and R and R ′ represent the right eye viewpoint. The distance between L and R and the distance between L 'and R' are preferably average interocular distances, respectively. When the parallax image viewed from the viewpoint set (L, R) is generated, the hidden surface area 701 is geometrically generated when viewed from the viewpoint R as shown in FIG. The hidden surface area is an area that is behind another object or surface from a certain viewpoint and is not visible in the input image.

  FIG. 8A is a diagram showing parallax images at viewpoints L and R generated from the input image of FIG. A parallax image including the hidden surface area 701 is generated. Since the input image does not include information on the hidden surface area, it is necessary to interpolate pixel values estimated by some method. However, it is difficult to correctly predict the pixel value of the hidden surface area, and there is a high possibility that image quality degradation will occur.

  FIG. 8B is a diagram illustrating parallax images at the viewpoints L ′ and R ′ generated from the input image in FIG. 6. In the case of the example in FIG. 7, when a parallax image viewed from the viewpoint set (L ′, R ′) is generated, a hidden surface area does not occur. Therefore, it is possible to generate a parallax image without including the hidden surface area as shown in FIG. That is, it can be seen that the total number of pixels belonging to the hidden area changes by adaptively changing the set of viewpoints according to the input depth information.

  FIG. 9 is a diagram for explaining the operation of the stereoscopic image generating apparatus.

The calculation unit 101 sets a set of candidate viewpoints according to the viewpoint axis (S901). For example, when generating a parallax image for the left eye and a parallax image for the right eye, each set consists of two viewpoints. The set Ω indicates a set of candidate viewpoints. Note that a set of candidate viewpoints may be determined in advance. An example in which the set Ω is set in the following case will be described.
Ω = {(-0.5, 0.5), (-1.0, 0.0), (0.0, 1.0)}
Although an example in which a set of three viewpoints is used as a candidate has been described, any number may be used as long as it is a plurality. Note that the amount of calculation is required as the number of candidates increases. Therefore, it is preferable to set the number of candidates according to the allowable calculation amount. In addition, by including the same viewpoint as the viewpoint from which the input image was captured in one of the viewpoint set elements, it is possible to reduce the amount of computation in the subsequent stage parallax image generation. For example, (−1.0, 0.0) and (0.0, 1.0) in the above example include the same viewpoint as the viewpoint from which the input image was captured.

  The calculating unit 101 calculates the evaluation value E (ω) for each viewpoint set ω of the set Ω (S902). As described above, the evaluation value E (ω) is a value that increases as the number of pixels belonging to the hidden surface area increases. There are various methods for calculating the evaluation value E (ω). One is to use the disparity vector described above, assign an input pixel value to the position indicated by the disparity vector, and calculate the number of pixels to which no pixel value is assigned. A specific calculation method will be described later with reference to FIG.

  The calculation unit 101 determines whether or not the evaluation values of all the viewpoint sets set in S901 have been calculated (S903). If not calculated (S903, No), the process returns to S902 and the viewpoint sets for which evaluation values have not been calculated. The evaluation value E (ω) is calculated. When the evaluation values of all the viewpoint sets set in S901 have been calculated (S903, Yes), the process proceeds to S904.

  The selection unit 102 selects a set of viewpoints used for generating a parallax image based on the evaluation value calculated in S902 (S904). It is preferable to select a set of viewpoints having the smallest evaluation value.

  The parallax image generation unit 103 generates a parallax image corresponding to the set of viewpoints selected in S904. For example, if the viewpoint set selected in S904 is (0.0, 1.0), a parallax image corresponding to the viewpoint 1.0 is generated (S905). Note that since the image corresponding to the viewpoint 0.0 is an input image, there is no need to generate it again in S905.

  The parallax image generated in S905 is output, and the process for one input image is terminated.

  FIG. 10 is a diagram illustrating an example of a detailed operation of S902 performed by the calculation unit 101.

The calculation unit 101 initializes E (ω) to 0 (S9021). Using the input depth information, create a map function Map (i, ω _j ) that represents whether each pixel in the image of a certain viewpoint ωj in the set of viewpoints ω to be processed is a hidden surface area . Map (i, ω _j ) = OCCLUDE is set as an initial value for the pixel i∈P on the image corresponding to the viewpoint ωj (S9022). Here, P indicates all the pixels of the input image. OCCLUDE means that the pixel indicated on the left side belongs to the hidden area.

Next, the value of the position shifted by the disparity vector d (i, ω _j ) obtained by converting the depth information with respect to the pixel i∈P is calculated as Map (i + d (i, ω _j ), ω _j ) = NOT_OCCLUDE (S9023). Here, NOT_OCCLUDE means that the pixel indicated by the left side does not belong to the hidden surface area.

Then, it is determined whether the processing in step S9022~S9023 the elements omega _j of all omega has been completed (S9024). If the processing of S9022 to S9023 has not been completed for all ω elements ω _j (S9024, NO), the process proceeds to step S9022.

If the processing of S9022~S9023 has ended the elements omega _j for all ω (S9024, YES), for all pixels i∈P the _{Map (i, ω j),} Map (i, ω j) = OCCLUDE It is determined whether or not (S9025). If Map (i, ω _j ) = OCCLUDE (S9025, Yes), the number of pixels is added to E (ω) (S9026). That is, among all the pixels iεP of the map function Map (i, ω _j ), the number of pixels with Map (i, ω _j ) = OCCLUDE is added to E (ω). If Map (i, ωj) = OCCLUDE is not satisfied (Map (i, ωj) = NOT_OCCLUDE) (S9025, NO), the process of step S9026 is not performed, and the process proceeds to step S9027.

As a result, in the parallax image I (i, ω _j ), an evaluation value E (ω) indicating that the pixel value is not assigned by the parallax vector, that is, the number of pixels belonging to the hidden surface area is obtained.

  In step S9027, it is determined whether or not the processing of steps S9025 to S9026 has been completed for all pixels i (S9027). If the processing of steps S9025 to S9026 has not been completed for all the pixels i (S9027, NO), the process proceeds to step S9025.

When the process of step S9025~S9026 is for all pixels i finished (S9027, YES), it is determined whether or not the processing of steps S9025~S9026 the elements omega _j of all omega has been completed (S9028). If the processing of S9025 to S9026 has not been completed for all ω elements ω _j (S9028, NO), the process proceeds to step S9025.

When the processing of S9025 to S9026 has been completed for all ω elements ω _j (S9028, YES), the processing of step S902 is terminated.

(Modification 1)
FIG. 11 is a diagram describing another example of the evaluation value calculation method performed by the calculation unit 101. In the process of FIG. 10, the number of hidden area pixels is simply calculated. When pixel values are assigned around the hidden surface area pixels, interpolation can be performed. For this reason, the pixels in the hidden surface area may not cause large image quality degradation. Conversely, when hidden surface area pixels are concentrated, it is difficult to estimate the pixel value of the hidden surface area because measures such as interpolation cannot be taken. An evaluation value calculation method taking this difficulty into account is shown in FIG. The calculation unit 101 may perform processing by replacing steps S9025 to S9028 in FIG. 10 with steps S1101 to S1108.

First, the internal variable weight is initialized to 0 (S1101). Map (i, ω _j) for all pixels i∈P _{of, Map (i, ω j)} = determines a whether OCCLUDE (S1102). When Map (i, ω _j ) is OCCLUDE for the pixel iεP (S1102, Yes), the weight is incremented, and then the weight is added to E (ω) (S1103). When Map (i, ω _j ) is not OCCLUDE (S1102, NO), the weight is set to 0 (S1104). At this time, the order in which the pixel i is selected follows the raster scan scanning order. That is, in this operation, the weight value increases as it is determined in S1102 that the pixels selected in the raster scan scan order are consecutively belonging to the hidden surface area, and E (ω) in S1103. The amount of increase increases.

Then, it is determined whether or not the processing of steps S1102 to S1104 has been completed for all the pixels i (S1105). If the processing of steps S1102 to S1104 has not been completed for all the pixels i (S1105, NO), the process proceeds to step S1102. When the process of step S1102~S1104 has been completed for all pixels i (S1105, YES), the processing of step S1102~S1104 the elements omega _j of all omega determines whether or not it is completed (S1106). If the processing of S1102 to S1104 has not been completed for all ω elements ω _j (S1106, NO), the process proceeds to step S1102.

When the processing of S1102 to S1104 is completed for all ω elements ω _j (S1106, YES), the processing ends.

  Although the raster scan order is described here, other areas such as Hilbert scanning may be changed to the order of scanning with a single stroke. Further, instead of E (ω) + = weight in S1103, E (ω) + = 2 ^ weight may be used so that the evaluation value becomes larger as the hidden surface region continues.

  In addition, when the hidden surface area appears near the center of the screen, the image quality deterioration tends to be noticeable subjectively. In consideration of this, instead of E (ω) ++ in S1103 in FIG. 11, E (ω) + = exp (− (Norm (ic))) is used, and the closer the pixel position i is to the center of the screen, the more evaluation is performed. It is also possible to increase the value. Here, c is a vector representing the center position of the screen. Norm () is a function representing a vector norm value, and uses a general L1 norm or L2 norm. Similarly, E (ω) + = weight * exp (−Norm ((ic))) may be given in place of E (ω) + = weight in S1103 in FIG. .

(Modification 2)
In addition, if the set of viewpoints is discontinuous in time, the temporal connection of the video is lost, which leads to a significant deterioration in image quality subjectively. Therefore, an evaluation value E (ω) that allows a selection unit 102 (to be described later) to select a viewpoint set as close as possible to the viewpoint set ω ^t-1 when a parallax image is created one frame before in time. The derivation method of can also be considered. Specifically, instead of E (ω) ++ in S9026 of FIG. 10, E (ω) + = (1-exp (− (Norm (ω−ω ^t−1 ))))) It is also possible to make the evaluation value smaller as is closer to the viewpoint set ω ^t−1 selected at the time of creating the previous frame. As described above, Norm () uses a general L1 norm or L2 norm. Similarly, instead of E (ω) + = weight in S1103 in FIG. 11, E (ω) + = weight * (1-exp (-(Norm (ω-ω ^t-1 )))) You may give the same effect.

  It can be seen from the geometric relationship that the size of the hidden surface area can be obtained from the first derivative of the depth value of the adjacent pixel in the given depth information. This will be described below.

FIG. 12 shows a view when viewed from the vertical direction, similarly to FIG. 3 and the like. Point A is the viewpoint at the position α on the viewpoint axis. That is, the length of the line segment AE is bα when the interocular distance is b. Point C and point D represent pixel positions, and in this case, point C and point D are adjacent pixels. Also, the screen is arranged at the origin of the coordinate z axis that the depth value follows. Here, it is assumed that the negative value of the z-axis does not fall below −Zs. This is because it will not jump out of the viewer's position. According to the z-axis, the depth value z (C) of the point C and the depth value z (D) of the point D are represented. The origin of the viewpoint axis is point E. A bold line represents a given depth value. In this case, it can be seen that the length of the line segment BC becomes the size of the hidden surface area. The length of this line segment BC is obtained by using the similar relationship between ΔAEF and ΔBCF.
It can be written as However, α is 0 or more. This is because, as shown in the figure, the hidden surface region when the gradient z (C) -z (D) of z (C) and z (D) is negative occurs when α> 1.

FIG. 13 is another conceptual diagram showing the relationship between the viewpoint and the hidden surface area. The figure at the time of turning right and left in FIG. 12 is shown. However, the positions of the points C and D are interchanged in order to make the positional relationship between the points C and D the same as in FIG. That is, the depth value gradient z (C) -z (D)> 0 is shown. In this case, if the geometric relationship is used as before, the length of the line segment BD is
It can be written as However, α <0. To summarize these relationships, the length L (α, i, j) of the line segment representing the size of the hidden surface area is
Can be written. Pixel positions i and j are adjacent pixels.

FIG. 14 is a diagram for explaining a method of calculating the evaluation value E (ω) when L is used. As shown in step S501 to step S503 in FIG. 14, the calculation unit 101 calculates all the viewpoints included in the element ω of the set Ω and all the pixels iεP.
The evaluation value can also be calculated by. In this case, however, j indicates the pixel position of the next position when i is scanned in the raster scan order. Further, Equation 7 is such that the evaluation value E (ω) increases as the size of the hidden surface region increases.
It may be as follows. Here, Pow (x, y) is a function that returns the value of x raised to the power of y. Furthermore, the position vector c indicating the center position of the screen is used so that the evaluation value E (ω) increases as the hidden surface area in the center of the screen increases.
It may be as follows. Also, make sure that the set of viewpoints selected does not change significantly over time.
An expression such as may be used.

The selection unit 102 receives the evaluation value E (ω) of each element of the set Ω defined by the calculation unit 101 as input, and determines one viewpoint set ω_sel. ω_sel is
In this way, a set of viewpoints having the smallest evaluation value E (ω) at each viewpoint of the set Ω is set. Alternatively, ω_sel is set to ω closest to ω ^t−1 from a set of viewpoints where E (ω) <Th with respect to a predetermined threshold Th. Here, the predetermined threshold Th is preferably determined such that the number of pixels belonging to the hidden surface area is 0.1% with respect to the number of pixels of the entire screen.

  The parallax image generation unit 103 acquires the determined viewpoint set ω_sel in the depth information, the input image, and the selection unit 102, generates a parallax image according to the parallax vector corresponding to the viewpoint set, and outputs the parallax image. At this time, a hidden surface area is generated unless the evaluation value is zero. Any existing method may be used as a method for giving a pixel value to the generated hidden surface area.

  According to the first embodiment, an evaluation value that increases as the number of pixels of the hidden surface area that appears when a parallax image is generated from a plurality of second viewpoint sets set in advance is increased. By selecting the second set of viewpoints to be minimized and generating a parallax image when virtually taken from the set of second viewpoints, it becomes possible to reduce the number of pixels belonging to the hidden surface area, It becomes possible to improve the quality of parallax images.

  In addition, the above example is a case where one two-dimensional image is input. However, when a parallax image for left eye and right eye prepared in advance by the provider is input, There are uses for generating images from different shooting positions. That is, the input parallax image has to output a stereoscopic image with the amount of parallax already determined on the provider side, while the viewer side can view a powerful stereoscopic image with a larger amount of parallax. This is an application for needs such as wanting to reduce the amount of parallax or reducing fatigue caused by viewing a stereoscopic image. Therefore, it is possible to meet the above needs of viewers by generating depth information for input left and right parallax images by methods such as stereo matching and generating parallax images with a large or small dynamic range of depth. become.

(Second Embodiment)
In the first embodiment, an evaluation value that increases as the number of pixels of a hidden surface area that appears when a parallax image is generated from a set of a plurality of second viewpoints set in advance is increased, and the evaluation value is minimized. The second viewpoint set was selected. In that case, the viewpoint may change rapidly in time series. When a moving image is input, if a sudden change in viewpoint occurs, the temporal connection of stereoscopic video is lost, giving the viewer a sense of discomfort. In the present embodiment, the above problem is solved by gradual change of the viewpoint.

  FIG. 15 is a diagram illustrating a stereoscopic image generating apparatus according to the present embodiment. The stereoscopic image generating apparatus according to the present embodiment is different from the first embodiment in that the viewpoint control unit 201 is further provided.

  The viewpoint control unit 201 acquires the viewpoint set ω_sel selected by the selection unit 102, and uses the viewpoint set at the time of creation of the past parallax image that is internally held, thereby correcting the corrected viewpoint set ω_cor. It is sent to the parallax image generation unit 103. A method of deriving the corrected viewpoint set ω_cor will be described.

A set of viewpoints of parallax images created n frames before is represented by ω ⁽ⁿ⁾ . Here, ω ⁽⁰⁾ is ω_sel. ω_cor is derived by the following FIR filter.
Here, a _i is a filter coefficient, and a coefficient having a characteristic that the FIR filter becomes a low-pass filter is set.

Further, ω_cor can be derived by the following equation using a first-order delay.
ω_cor = h * ω ⁽⁰⁾ + (1-h) ω ⁽¹⁾
h represents a time constant. The range is 0 <h <1.

  In order to reduce the amount of calculation, it is conceivable that at least one of the viewpoints of ω_col is fixed at the shooting position of the input image.

  As described above, in the second embodiment, it is possible to realize a stereoscopic video generation apparatus that suppresses a sudden change in the viewpoint to provide a temporal connection of stereoscopic video and does not give the viewer a sense of incongruity.

(Third embodiment)
In the second embodiment, since the number of pixels belonging to the hidden surface area changes gradually until the viewpoint set becomes smaller, the number of pixels belonging to the hidden surface area does not necessarily decrease during this transition. On the other hand, at the timing of the scene change, even if the parallax position is changed abruptly, the viewer does not feel uncomfortable. Therefore, in the present embodiment, a stereoscopic video generation method with a more appropriate set of viewpoints is provided by increasing the change in the parallax position at the timing when the moving image scene changes.

  FIG. 16 is a diagram illustrating a configuration of the present embodiment. Compared to FIG. 1, the difference is that it further includes a detection unit 301 and a viewpoint control unit 302.

  The detection unit 301 detects a scene change in the input image. When the occurrence of a scene change is detected before the detection target frame, a DETECT signal is sent to the viewpoint control unit 302. If no occurrence of a scene change is detected, a NONE signal is sent to the viewpoint control unit 302.

When the viewpoint control unit 302 receives a NONE signal from the detection unit 301, the viewpoint control unit 302 performs the same processing as the parallax position control unit 201. On the other hand, when the DETECT signal is received, ω ⁽⁰⁾ is set to ω_cor instead of the output of the FIR filter. When deriving ω_col in a first-order lag system, the time constant is set to h ₁ . However, 1> h ₁ > h.

  As described above, in the third embodiment, at the timing when the moving image scene changes, it is possible to realize a stereoscopic video generation apparatus with a more appropriate set of viewpoints by increasing the change in the parallax position. Although several embodiments of the present invention have been described, these embodiments are presented by way of example and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the scope of the invention. These embodiments and modifications thereof are included in the scope and gist of the invention, and are included in the invention described in the claims and the equivalents thereof.

101 ... calculation unit, 102 ... selection unit, 103 ... parallax image generation unit,
201, 302 ... viewpoint control unit,
301... Detection unit

Claims (5)

In an apparatus for generating a parallax image of a viewpoint different from the image from at least one image and depth information corresponding to the image,
For each set including two or more viewpoints, a calculation unit that calculates an evaluation value that becomes a larger value from the depth information as the hidden surface area generated when a parallax image is created increases;
A selection unit that selects the viewpoint set that minimizes the evaluation value calculated for the viewpoint set;
When the image is a moving image, the image includes _the selected set of viewpoints and at least a previous frame of the image so as to suppress changes in the viewpoint on the time axis compared to a case where the image is not a moving image. A viewpoint control unit that generates and outputs a set of corrected viewpoints for the image using a plurality of viewpoint sets respectively selected in a plurality of previous consecutive frames;
A generation unit that generates a parallax image of a viewpoint corresponding to the corrected set of viewpoints output from the viewpoint control unit, from the image;
A three-dimensional image generating apparatus. The viewpoint control unit, when before Symbol image is a moving image, a set of a plurality of viewpoints that are respectively selected in a plurality of frames of the image from the previous contiguous containing at least previous frame of the image, the image The stereoscopic image generation apparatus according to claim 1, wherein a linear sum of the viewpoint set selected for the viewpoint is derived as the corrected viewpoint set.   The stereoscopic image generating apparatus according to claim 2, wherein the coefficient of the linear sum has a characteristic of becoming a low-pass filter. When the image is a moving image, further comprising a detection unit for detecting a scene change,
Stereoscopic image generating apparatus according to claim 1 to 3 noise crab 1 wherein increasing the change in aspect before and after said detected scene change. In a method for generating a parallax image of a viewpoint different from the image from at least one image and depth information corresponding to the image,
For each set including two or more viewpoints, calculating an evaluation value that becomes a larger value from the depth information as the hidden surface area generated when a parallax image is created increases;
Selecting the viewpoint set that minimizes the evaluation value calculated for the viewpoint set;
When the image is a moving image, the image includes the selected set of viewpoints and at least a previous frame of the image so as to suppress changes in the viewpoint on the time axis compared to a case where the image is not a moving image. Generating and outputting a modified set of viewpoints for the image using a set of viewpoints each selected in a plurality of previous consecutive frames;
Generating a parallax image of a viewpoint corresponding to the modified set of viewpoints from the image;
A method for generating a stereoscopic image.