JP2014003521A - Depth estimation data generating apparatus, pseudo stereoscopic image generating apparatus, depth estimation data generation method, and depth estimation data generation program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To generate depth estimation data that brings a natural three-dimensional effect from an input non-stereoscopic image of one view point while preventing portions forming bright spots by reflecting light from being viewed excessively as if they protrude.SOLUTION: A synthesis section synthesizes a plurality of basic depth models with a synthesis ratio corresponding to values calculated at an upper higher frequency component evaluation section and at a lower higher frequency component evaluation section, and generates synthesized depth data. A bright spot interpolation part interpolates bright spot parts detected by a bright spot detector with peripheral pixels and generates an image signal for depth addition. An addition section adds a predetermined signal component of the image signal for depth addition to the synthesized depth data to generate depth estimation data.

Description

  The present invention relates to a depth estimation data generation device and a pseudo stereoscopic image generation device that generate depth estimation data from an image (non-stereo image) to which depth information is not given explicitly or implicitly like a stereo image. The present invention relates to a depth estimation data generation method and a depth estimation data generation program.

  In a stereoscopic display system, a normal still image or moving image, that is, depth information for representing a stereoscopic image, is explicitly or implicitly like a stereo image in order to enable viewing of a non-stereo image by pseudo-stereoscopic vision. In addition, a process of generating a pseudo three-dimensional image from an image (non-stereo image) that is not given in FIG. As an example of such a technique, there is a pseudo-stereoscopic image generation apparatus disclosed in Patent Document 1, for example. In order to determine a scene structure that is as realistic as possible, the technique described in Patent Document 1 uses a plurality of basic depth models that indicate depth values for each of a plurality of basic scene structures. The composition ratio is determined according to the high-frequency component evaluation value of the input non-stereo image from the high-frequency component evaluation unit and the high-frequency component evaluation unit at the bottom of the screen, and multiple types of basic depth models are synthesized according to the composition ratio . The adder superimposes the synthesized basic depth model and the R signal of the non-stereo image to generate final depth estimation data, and performs processing based on the depth estimation data on the image signal of the non-stereo image. Thus, an image signal of another viewpoint image that gives a three-dimensional feeling is generated.

JP 2006-185033 A

  In the method of Patent Document 1, a combined depth model is generated by combining a plurality of types of basic depth models, and final depth estimation data is generated by superimposing an R signal on the combined depth model. As a signal to be superimposed on the synthesized depth model, a depth signal that depends only on the level of the R signal is superimposed on any picture, and therefore there is a portion that reflects light and forms a bright spot. Only the bright spot portion may appear to protrude excessively, resulting in a sense of separation of objects and a mismatch in positional relationship, and further improvement has been desired.

  Even though the present invention has been made in view of the above problems, a depth estimation data generating apparatus, a pseudo three-dimensional object that prevents a portion where only a portion that forms a bright spot by reflecting light from appearing to protrude excessively is produced, and a natural three-dimensional effect is provided. An object is to provide an image generation device, a depth estimation data generation method, and a depth estimation data generation program.

  In order to achieve the above object, the present invention is a depth estimation data generation apparatus that generates depth estimation data from a non-stereo image to which depth information is not given explicitly or implicitly like a stereo image. A storage unit (14, 15, 16) for storing a plurality of basic depth models indicating depth values for each of a plurality of basic scene structures, and for estimating the scene structure of the non-stereo image A calculation unit (12, 13) that calculates a composite ratio between the plurality of basic depth models by using a statistic for a pixel value in a predetermined region in the screen of the stereoscopic image, and the plurality of read out from the storage unit A synthesis unit (17) that synthesizes the depth model based on the synthesis ratio calculated by the calculation unit and generates synthesis depth data, and a bright spot that detects a bright spot portion from the non-stereo image When the bright part is detected by the output part (110) and the bright spot detection part, the pixel value of the detected bright spot part is replaced with the pixel value of the peripheral pixel of the pixel of the bright spot part, and the depth A bright spot interpolation unit (120) that generates an image signal for addition, and an addition unit (19) that generates the depth estimation data by adding a predetermined signal component of the image signal for depth addition to the depth data. There is provided a depth estimation data generation device (10) characterized by the above.

  In order to achieve the above object, the present invention provides a bright spot evaluation value calculation section (111) in which the bright spot detection section calculates a bright spot evaluation value at each pixel according to the level of each pixel of the input signal. The bright spot evaluation value of the evaluation target pixel is compared with whether or not the bright spot evaluation value of the evaluation target pixel is larger than a predetermined value, and determined to be true if large, and the bright spot evaluation value of the evaluation target pixel is A comparison unit (114) for comparing with a surrounding average that determines whether or not it is larger than the average value of the bright spot evaluation values in the pixels around the bright spot evaluation target pixel, and including the evaluation target pixel. A comparison unit (116) of variance values for comparing whether or not the variance value of the bright spot evaluation values in the peripheral pixels is smaller than a predetermined value, and judging that the value is true if the variance value is small, the high luminance comparison unit, Judgment result of either the comparison unit with the ambient average or the comparison unit of the variance value If a true, the bright spot detection unit provides the depth estimation data generating apparatus described above, wherein the detecting the evaluation target pixels as bright spots.

  In order to achieve the above object, the present invention provides a pseudo-stereoscopic image that generates a pseudo-stereoscopic moving image from a non-stereoscopic image without depth information being given explicitly or implicitly like a stereo image. The image generation device (1), which shifts the texture of the non-stereo image by an amount corresponding to the depth of the corresponding portion according to the depth estimation data supplied from any one of the depth estimation data generation devices described above. A different viewpoint image generation unit (50) that generates a different viewpoint image to be a left-eye image and / or a right-eye image, and the different viewpoint image generated by the different viewpoint image generation unit and the non-stereo image Provided is a pseudo-stereoscopic image generation apparatus characterized in that one is used as a left-eye image and the other is output as a right-eye image.

  In order to achieve the above object, the present invention provides a depth estimation data generation method for generating depth estimation data from a non-stereo image in which depth information is not given explicitly or implicitly like a stereo image. In order to estimate the scene structure of the non-stereo image, a composite ratio between the plurality of basic depth models is calculated by using a statistic for a pixel value in a predetermined region in the screen of the non-stereo image. Calculating step, combining a plurality of basic depth models based on the combining ratio calculated in the calculating step, generating combined depth data, and detecting a bright spot from the non-stereo image Step, and when a bright spot is detected by the bright spot detection unit, the pixel value of the detected bright spot portion is replaced with a pixel value of a peripheral pixel of the bright spot portion, and an image for depth addition Depth estimation comprising: a bright spot interpolation step for generating a signal; and an addition step for generating a depth estimation data by adding a predetermined signal component of the image signal for depth addition to the synthesized depth data Provide a data generation method.

  In order to achieve the above object, the present invention realizes a computer with a function of generating depth estimation data from a non-stereoscopic image to which depth information is not given explicitly or implicitly like a stereo image. A depth estimation data generation program for generating a plurality of basic depth models by using a statistic for a pixel value in a predetermined region in a screen of the non-stereo image in order to estimate a scene structure of the non-stereo image A calculation step for calculating a combination ratio between the two, a combination step for combining a plurality of basic depth models based on the combination ratio calculated in the calculation step to generate combined depth data, and a bright spot portion from the non-stereo image When detecting a bright spot in the bright spot detecting step and the bright spot detecting section, the pixel value of the detected bright spot portion is used as a peripheral pixel of the bright spot portion. A bright spot interpolation step of generating a depth addition image signal by replacing with a pixel value; an addition step of adding a predetermined signal component of the depth addition image signal to the synthesized depth data to generate the depth estimation data; A depth estimation data generation program characterized by comprising:

  According to the present invention, a depth estimation data generation device, a pseudo-stereoscopic image generation device, and a depth estimation that prevent a natural reflection of only a portion that reflects light and forms a bright spot are prevented from appearing to protrude excessively. A data generation method and a depth estimation data generation program can be provided.

It is a block diagram for demonstrating the structure of a pseudo-stereoscopic image production | generation apparatus. It is a figure which shows an example of the three-dimensional structure of basic depth model type 1. FIG. It is a figure which shows an example of the three-dimensional structure of basic depth model type 2. FIG. It is a figure which shows an example of the three-dimensional structure of basic depth model type 3. FIG. It is a figure explaining the basic depth model synthetic | combination ratio determination conditions. It is an example of an image for explaining a bright spot. It is a figure for demonstrating the structure of a bright spot detection and interpolation part. It is a flowchart for demonstrating operation | movement of a bright spot detection and interpolation part. It is a figure for demonstrating the object pixel which calculates | requires the average of the luminescent point evaluation value of a surrounding pixel. It is a figure for demonstrating the object pixel used as the origin for interpolating a brightness | luminance part. It is a figure for demonstrating operation | movement of a brightness | luminance part detection and interpolation part.

Next, embodiments of the present invention will be described in detail with reference to the drawings.
FIG. 1 is a block diagram for explaining the configuration of a pseudo stereoscopic image generating apparatus according to the present invention. As shown in FIG. 1, the pseudo-stereoscopic image generation apparatus 1 according to the present embodiment includes a depth estimation data generation unit 10 and another viewpoint image generation unit 50.

  The depth estimation data generation unit 10 includes an image input unit 11, a high-frequency component evaluation unit 12 at the top of the screen, a high-frequency component evaluation unit 13 at the bottom of the screen, frame memories 14, 15 and 16, a synthesis unit 17, It has a bright spot detection / interpolation unit 18 and an addition unit 19, and generates depth estimation data.

  The image input unit 11 includes a frame memory. After temporarily storing an image signal for one frame that is an input non-stereo image signal, the image signal for the one frame is stored in a high-frequency component evaluation unit 12 at the top of the screen. , And supplied to the high frequency component evaluation unit 13 and the bright spot detection / interpolation unit 18 at the bottom of the screen. The non-stereoscopic image signal input to the image input unit 11 is a one-viewpoint image signal, and here, as an example, is a right-eye image signal to be displayed as a right-eye image. The signal system of the image input to the image input unit is not limited, but in the present embodiment, the input image is described as an RGB signal as an example.

  The high frequency component evaluation unit 12 at the top of the screen obtains a high frequency component in an area corresponding to about 20% of the top of the image signal for the right eye for one frame, and calculates it as a high frequency component evaluation value at the top of the screen. Then, the high frequency component evaluation unit 12 at the top of the screen supplies the high frequency component evaluation value at the top of the screen to the synthesis unit 17. The high-frequency component evaluation unit 13 at the bottom of the screen obtains a high-frequency component in an area corresponding to the lower 20% region of the screen in the right-eye image signal for one frame, and calculates it as a high-frequency component evaluation value at the bottom of the screen. To do. Then, the high frequency component evaluation unit 13 at the bottom of the screen supplies the high frequency component evaluation value at the bottom of the screen to the synthesis unit 17.

  On the other hand, the frame memory 14 stores basic depth model type 1, the frame memory 15 stores basic depth model type 2, and the frame memory 16 stores basic depth model type 3 images in advance. The images of these basic depth model types 1 to 3 indicate scene images serving as a basis for generating pseudo stereoscopic image signals by generating depth estimation data based on non-stereo image signals, respectively. The frame memories 14 to 16 function as a storage unit that stores a plurality of basic depth models indicating depth values for each of a plurality of scene structures serving as a basis of the present invention.

  That is, the basic depth model type 1 image is a depth model image having a spherical concave surface, and an image having a three-dimensional structure as shown in FIG. In many cases, this basic depth model type 1 image is used. This is because, in a scene in which no object exists, setting the center of the screen to the farthest distance can provide a three-dimensional effect with less discomfort and a comfortable depth feeling.

  In addition, the above basic depth model type 2 image is obtained by replacing the upper part of the basic depth model type 1 image with an arch-shaped cylindrical surface instead of a spherical surface. It is an image of a model in which a cylindrical surface (axis is a vertical direction) and a lower surface is a concave surface (spherical surface).

  Furthermore, the basic depth model type 3 image is a cylindrical surface that has a three-dimensional structure as shown in FIG. 4, with the upper part being a plane, the lower part being continuous from the plane, and going down toward the front. The upper part is a plane image and the lower part is a cylindrical surface (the axis is the horizontal direction). The images of the basic depth model types 1 to 3 stored in the frame memories 14 to 16 constituting the basic depth model type generating unit are supplied to the synthesis unit 17.

  The synthesizing unit 17 firstly evaluates the high frequency component at the top of the screen supplied from the high frequency component evaluation unit 12 at the top of the screen and the high frequency component at the bottom of the screen supplied from the high frequency component evaluation unit 13 at the bottom of the screen. Based on the value, the composition ratio k1 of the basic depth model type 1, the composition ratio k2 of the basic depth model type 2, and the composition ratio of the basic depth model type 3 by a predetermined method without considering the scene of the image Calculate k3. The total value of the three synthesis ratios k1 to k3 is always “1”.

  FIG. 5 shows an example of conditions for determining the composition ratio. FIG. 5 shows a high-frequency component evaluation value at the top of the screen indicated by the horizontal axis (hereinafter abbreviated as the high-frequency component evaluation value at the top) and a high-frequency component evaluation value at the bottom of the screen indicated by the vertical axis (hereinafter, the high frequency component at the bottom It is shown that the composition ratio is determined based on the balance between each value of the component evaluation value (abbreviated as component evaluation value) and the predesignated values tps, tpl, bms, and bml. The condition for determining the composition ratio is an example, and the present invention is not limited to this.

  In FIG. 5, regions where a plurality of types are described are synthesized linearly according to the high frequency component evaluation value. For example, in the region of “type 1/2” in FIG. 5, the value of the basic depth model type 1 is the ratio of (upper high-frequency component evaluation value) and (lower high-frequency component evaluation value) as follows. A ratio between Type 1 and Type 2 which is a value of basic depth model type 2 is determined, and Type 3 which is a value of basic depth model type 3 is not used for determining the ratio.

Type1: Type2: Type3
= (Upper high-frequency component evaluation value-tps): (tpl-Upper high-frequency component evaluation value): 0
In FIG. 5, in the “Type 1/2/3” region, the average of Type 1/2 and Type 1/3 is adopted, and the value of Type 1/2/3 is determined as follows.

Type1: Type2: Type3
= (Upper high-frequency component evaluation value-tps) + (Lower high-frequency component evaluation value-bms): (tpl-Upper high-frequency component evaluation value): (bml-Lower high-frequency component evaluation value)
The synthesis ratios k1, k2, and k3 are calculated by the following equations.

k1 = Type1 / (Type1 + Type2 + Type3)
k2 = Type2 / (Type1 + Type2 + Type3)
k3 = Type3 / (Type1 + Type2 + Type3)
Subsequently, the combining unit 17 combines the images of the basic depth model types 1 to 3 at the ratio indicated by the combining ratios k1 to k3 calculated as described above, and calculates the depth data of the curved surface constituting the background (hereinafter referred to as combining). (Also referred to as depth data). The synthesis unit 17 functions as a calculation unit and a synthesis unit of the present invention.
The adding unit 19 adds the combined depth data supplied from the combining unit 17 and the R signal component of the image signal supplied from the bright spot detection / interpolation unit 18 to generate depth estimation data.

  One of the reasons for using the R signal component is that the size of the R signal component is uneven in the original image in an environment that is close to direct light and in which the change in the brightness level (brightness) of the texture is not large. According to the rule of thumb that there is a high probability of matching. That is, this R signal component is a signal component indicating a signal level substantially corresponding to the unevenness of the original image of the input non-stereo image. Note that a texture is an element constituting an image, and is composed of a single pixel or a group of pixels.

Another reason for using the R signal component is that red and warm colors are advanced colors in chromaticity, and the depth is recognized in front of the cold color system. It is possible to emphasize the stereoscopic effect.
Here, when a bright spot exists in the input image, the R signal component of the bright spot portion becomes larger than the surrounding area. Therefore, when the R signal component of the image having the bright spot is added to the synthesized depth data, Only the point portion may appear to protrude excessively, leading to a sense of separation of objects and a mismatch in positional relationship. FIGS. 6A and 6B are images of the same object. FIG. 6A shows a bright spot and FIG. 6B shows no bright spot. If the R signal component of the image of FIG. 6A is used as depth data as it is, the R component of the bright spot portion is larger than the R component of the portion that is not the bright spot, and therefore the depth estimation data in which only the bright spot portion protrudes excessively is obtained. Generated.

In order to prevent the generation of excessive depth data at the bright spot portion, the bright spot detection / interpolation unit 18 detects the bright spot in the input image, and interpolates the detected bright spot portion pixels with the peripheral pixels. To generate an image signal for depth addition.
FIG. 7 shows an example of the configuration of the bright spot detection / interpolation unit 18. The bright spot detection / interpolation unit 18 includes a bright spot detection unit 110 and a bright spot interpolation unit 120. The bright spot detection unit 110, based on the input non-stereo image, a bright spot evaluation value calculation unit 111 that calculates a bright spot evaluation value for evaluating a bright spot, a high brightness comparison unit 112, and a surrounding average value detection Unit 113, ambient average comparison unit 114, ambient variance detection unit 115, variance value comparison unit 116, high luminance comparison unit 112, ambient average comparison unit 114, and variance value comparison unit 116. A bright spot evaluation unit 117 that evaluates bright spots based on the comparison result.

FIG. 8 is a flowchart for explaining the processing flow of the bright spot detection / interpolation unit. The operation of the bright spot detection / interpolation unit 18 will be described below with reference to FIGS.
In step s1 of FIG. 8, the bright spot evaluation signal generation unit 111 calculates a bright spot evaluation value Y for evaluating the bright spot by the following formula based on the input non-stereo image signal (RGB signal).
Y = (R * a + G * b + B * c) / (a + b + c)
R, G, and B are the R, G, and B values of the input signal, and a, b, and c are appropriate values that weight the R, G, and B values when determining the bright spot evaluation value. The objective is to evaluate the bright spot more correctly by setting the weighted values of the R, G, and B values as the bright spot evaluation value Y. For example, a = 5, b = 9, and c = 2. . Appropriate values for a, b, and c may be obtained by experiment, or a combination of several numerical values may be stored and selected according to the image content.

In step s2, the high luminance comparison unit 112 detects “high luminance” which is one of the features of the bright spot.
Specifically, when the bright spot evaluation value Y of the evaluation target pixel is larger than a predetermined threshold value Thy (Y> Thy), it is determined that the brightness is high.

  When it is determined in step s2 that the luminance is high (s2 / yes), the process proceeds to step s3, and the comparison unit 114 with the ambient average is one of the features of the bright spot, “brighter than surrounding pixels”. This feature is detected. First, the average Yr_ave of the bright spot evaluation values Y in the surrounding pixels is obtained by the surrounding average value detection unit 113. If the pixel to be evaluated is a bright spot, it is highly possible that the immediate surrounding pixels are also bright spots. The target pixel for average value detection is shown in FIG. In FIG. 9, pixels indicated by diagonal lines indicate evaluation target pixels, pixels indicated by black indicate pixels for average value detection, and pixels indicated by white indicate pixels that are not subjected to average value detection. The average value detection target pixel is a pixel that is 3 to 6 pixels left and right from the evaluation target pixel in the horizontal direction, and is also a pixel that is 3 to 6 lines above and below the current pixel in the vertical direction. The average luminance evaluation value Yr_ave of the surrounding pixels is supplied to the comparison unit 114 and the surrounding dispersion detection unit 115 with the surrounding average. The above-described average value detection target pixel is an example, and the present invention is not limited to this.

  The average Yr_ave of the luminance evaluation values of the surrounding pixels detected by the surrounding average value detection unit 113 and the luminance evaluation value Y of the evaluation target pixel are compared by the comparison unit 114 with the surrounding average. That is, when the bright spot evaluation value Y of the evaluation target pixel is larger than the average Yr_ave of the luminance evaluation values of the surrounding pixels and the difference is larger than the predetermined threshold value Thave ((Y−Yr_ave)> Thave), It is determined that the luminance is high.

  If it is determined in step s3 that the luminance is higher than the surrounding pixels (s3 / yes), the process proceeds to step s4. In step s4, the dispersion value comparison unit 116 detects a feature that is one of the features of the bright spot, that is, the surface on which the bright spot is generated is “a smooth surface in the same object as the periphery”. To do. The surrounding dispersion detection unit 115 uses the average Yr_ave of the luminance evaluation values of the surrounding pixels received from the surrounding average value detection unit 113 and the luminance evaluation value Y of the evaluation target pixel received from the luminance evaluation value calculation unit 111, and Find the variance Yr_s. The target range for obtaining the variance is the range used for obtaining the average Yr_ave of the luminance evaluation values of the surrounding pixels. The obtained ambient variance Yr_s is supplied to the variance value comparison unit 116.

  When the evaluation target pixel is a smooth surface in the same object as the surrounding area, the variance Yr_s is a small value. The variance comparison unit 116 compares the threshold value Ths with the following formula, and when the variance Yr_s is smaller than the threshold value Ths (Yr_s <Ths), the evaluation target pixel is a smooth surface in the same object as the surrounding area. Judge that there is.

  If it is determined in step s4 that the variance value is smaller than the threshold (s4 / yes), the bright spot evaluation unit 117 determines that the pixel to be evaluated is a bright spot, and the process proceeds to step s5. The processing order of steps s2, s3, and s4 in FIG. 8 need not be limited to this, and the processing of steps s2, s3, and s4 is performed at the same time, and only when all the determinations are yes, step s5 is performed. It is good also as a structure to advance.

  The bright spot interpolation unit 120 includes an interpolation pixel creation unit 121 and a switch 122. In step s5, the interpolation pixel creation unit 121 creates an interpolation pixel for interpolating the bright spot portion. Interpolated pixels are created based on surrounding pixels that are slightly away from the pixel to be interpolated. An example of the target pixel that is the source of the interpolation pixel is shown in FIG. In FIG. 10, pixels indicated by diagonal lines indicate pixels to be interpolated that are determined as bright spots, pixels indicated by black indicate pixels for which an average is obtained, and pixels indicated by white indicate pixels that are not subject to an average. Yes. The pixels whose averages are to be obtained are four pixels shown in black, which are three pixels away from the interpolation target pixels in the horizontal direction and the vertical direction, and the average of the pixel values of these pixels is calculated to calculate the pixels of the interpolation pixels. Value. The above-described pixel for which the average is obtained is an example, and the present invention is not limited to this.

In step s6, the switch 122 replaces the pixel determined as the bright spot by the bright spot evaluation unit with the pixel created by the interpolation pixel creation unit 121, and ends the process.
When it is determined to be no in any of steps s1, s2, and s3, the process is terminated without interpolating the pixels. By sequentially performing the above processing for all the pixels in the screen, it is possible to detect and interpolate the bright spot portion in the screen and generate a depth addition image signal.

  Returning to FIG. The image signal for depth addition generated at the bright spot portion by the bright spot detection / interpolation section is sent to the addition section 19. The adding unit 19 adds the R component of the image signal for depth addition received from the bright spot detection / interpolation unit 18 to the combined depth model received from the combining unit 17 to obtain final depth estimation data (Depth ) Is generated and output.

An operation example of the bright spot detection / interpolation unit 18 will be described with reference to FIG.
In order to simplify the description, a case where only two pixels serving as bright spots exist in one horizontal line of pixels near the center of the screen is illustrated as an example. In the input image of FIG. 11, the luminance evaluation value Y of the pixel shown in white is high, represents a bright spot, and the luminance evaluation value Y of the pixel shown in black is low. In addition, the luminance evaluation values Y of the pixels in the lines other than the illustrated lines are all the same values as the pixels shown in black, and the combined depth data received from the combining unit 17 has the same value (for example, Depth = 0) in the screen. ).

  (A) shows the level of the depth signal generated by the conventional method that does not interpolate the bright spot. The depth level of the pixel portion shown in black is 30 and the depth level of the pixel portion shown in white is 90. It shows that there is. Here, the greater the depth level, the shallower the depth or the direction of jumping out from the screen. In (A), the depth signal is shallower or jumps out of the screen by two pixels shown in white. It represents.

(B) shows the evaluation result in the high luminance comparison unit 112, and only the pixel portion shown in white is determined to be high luminance (yes).
(C) shows the evaluation result in the comparison unit 114 with the ambient average, and it is determined that only the pixel portion shown in white has higher luminance (yes) than the surrounding pixels, as in (B).
(D) shows the evaluation result in the variance value comparison unit 116, and only the pixel portion shown in white and the pixel portion shown in black adjacent to the pixel shown in white are in the same object as the surroundings. It is determined that the surface is (yes).
(E) shows the bright spot detection result. From the evaluation results of (B), (C), and (D), it is determined that only the pixel portion shown in white is a bright spot (yes).

(F) is the level of the interpolation depth signal created by the interpolation pixel creation unit 121. Since the pixel value of the interpolation pixel is calculated from the average of a total of four pixels that are three pixels apart vertically and horizontally with respect to the target pixel to be interpolated, in the case where all four pixels are shown in black, the depth level by the averaged pixel value is In the case where one pixel including white is included, the depth level based on the averaged pixel value is 45.
(G) is a depth signal created based on a pixel obtained by interpolating the bright spot detection portion in the interpolation pixel creation unit. Only the pixel determined as the bright spot in (E), that is, the portion corresponding to the pixel indicated by white of the input pixel is interpolated. As a result, the depth level of the entire line is 30.

  As described above, when the depth signal (A) generated by the conventional method and the depth signal (G) according to the present embodiment are compared, in the depth signal according to the present embodiment, the unevenness difference between the bright pixels and the surrounding pixels is different. Is suppressed, and it is possible to prevent only the bright spot portion from popping out excessively and causing a sense of separation of objects.

An image of another viewpoint can be generated based on the depth estimation data generated by the depth estimation data generation unit 10. For example, when moving the viewpoint to the left, for objects that are displayed in front of the screen, the closer the object, the closer to the viewer (in the nose side), the more visible the texture of the corresponding part on the inside, that is, the right. Just move.
As for objects to be displayed at the back of the screen, the closer the object is to the outside of the viewer, the corresponding texture is moved to the left by an amount corresponding to the depth. A stereo pair is formed by using this as the left-eye image and the original image as the right-eye image.

Returning to FIG. 1, the different viewpoint image generation unit 50 will be described.
The different viewpoint image generation unit 50 includes a texture shift unit 51, an occlusion compensation unit 52, and a post processing unit 53, and outputs a left eye image 54 and a right eye image 55 as stereo pair images.
The texture shift unit 51 performs texture shift of the right eye image by an amount corresponding to the depth estimation data received from the depth estimation data generation unit 10, and generates an image of a viewpoint different from the right eye viewpoint.

A portion where there is no texture, that is, occlusion may occur due to a change in the positional relationship in the image due to texture shift. For such a part, the occlusion compensation unit 52 fills in the corresponding part of the input image, or a well-known document (Kunio Yamada, Kenji Mochizuki, Kiyoharu Aizawa, Takahiro Saito: “Image texture divided by the region competition method” Occlusion compensation based on the statistic of ”, Eiji Jakugaku, Vol. 56, No. 5, pp. 863-866 (2002.5)).
The image subjected to occlusion compensation by the occlusion compensation unit 52 is subjected to post-processing such as smoothing by the post-processing unit 53, thereby generating a left-eye image 54 by reducing noise generated in the previous processing, A stereo pair is formed by using the right eye image 55 as the image. These left eye image 54 and right eye image 55 are output to the stereo display device 2.

Regarding the generation of the stereo pair image, the left-eye image may be generated as an original image and the right-eye image may be generated as a different viewpoint image by reversing left and right.
In the above processing, a stereo pair image is formed in which either the right-eye image or the left-eye image is used as the input image and the other is generated as another viewpoint image. That is, it is also possible to configure a stereo pair image using another viewpoint image moved to the right and another viewpoint image moved to the left.
In this embodiment, an example with two viewpoints is described as the different viewpoint image generation unit. However, when displaying on a display device capable of displaying two or more viewpoints, the number of different viewpoints according to the number of viewpoints is displayed. It is also possible to configure a multi-viewpoint image generation device that generates an image.

The stereo display device 2 includes a projection system using polarized glasses, a projection system or display system combining time-division display and liquid crystal shutter glasses, a lenticular stereo display, an anaglyph stereo display, a head mounted display, and the like. .
In addition, as described above, it is possible to construct a multi-viewpoint stereoscopic image display system using a display device that can display two or more viewpoints. Further, the present stereoscopic display system may be configured to be equipped with an audio output. In this case, for image content that does not have audio information, such as still images, an aspect in which an environmental sound suitable for an image is added can be considered.

In the present embodiment, the unit of the number of images to be counted is described as a frame, but may be realized in units of fields.
Note that the present invention is not limited to the case where the depth data generation unit and the stereo pair generation unit configured as shown in FIG. 1 are configured by hardware, and can also be realized by software using a computer program. In this case, the computer program may be taken into the computer from a recording medium or may be taken into the computer via a network.

DESCRIPTION OF SYMBOLS 1 Pseudo three-dimensional image generation apparatus 10 Depth estimation data generation part 11 Image input part 12 High frequency component evaluation part 13 of screen upper part High frequency component evaluation part 14, 15, 16 Frame memory 17 Synthesis | combination part 18 Bright spot detection and interpolation Unit 19 addition unit 50 different viewpoint image generation unit 110 bright spot detection unit 111 bright spot evaluation value calculation unit 112 high brightness comparison unit 113 ambient average value detection unit 114 comparison unit with ambient average 115 ambient dispersion detection unit 116 comparison of dispersion values Unit 117 bright spot evaluation unit 120 bright spot interpolation unit 121 interpolation pixel creation unit 122 switch

Claims (5)

A depth estimation data generation device that generates depth estimation data from a non-stereo image that is not given depth information explicitly or implicitly like a stereo image,
A storage unit for storing a plurality of basic depth models indicating depth values for each of a plurality of basic scene structures;
A calculation unit that calculates a composite ratio between the plurality of basic depth models by using a statistic for a pixel value in a predetermined region in a screen of the non-stereo image in order to estimate a scene structure of the non-stereo image; ,
Combining the plurality of depth models read from the storage unit based on the combination ratio calculated by the calculation unit, and generating combined depth data;
A bright spot detection unit for detecting a bright spot part from the non-stereo image;
When the bright spot part is detected by the bright spot detection unit, the pixel value of the detected bright spot part is replaced with the pixel value of the peripheral pixel of the pixel of the bright spot part, and a bright addition image signal is generated. A point interpolation unit;
A depth estimation data generation apparatus comprising: an addition unit configured to add a predetermined signal component of the image signal for depth addition to the depth data to generate the depth estimation data. The bright spot detection unit
A bright spot evaluation value calculation unit that calculates a bright spot evaluation value in each pixel according to the level of each pixel of the input signal;
A high-intensity comparison unit that compares whether or not the bright spot evaluation value of the pixel to be evaluated is greater than a predetermined value, and determines true if it is greater;
Compare whether or not the luminescent spot evaluation value of the pixel to be evaluated is larger than the average value of the luminescent spot evaluation values in the surrounding pixels of the pixel to be evaluated and compare with the ambient average that is determined to be true if it is larger And
A dispersion value comparison unit for comparing whether or not the dispersion value of the bright spot evaluation value in the surrounding pixels including the evaluation target pixel is smaller than a predetermined value,
If any of the determination results of the high-luminance comparison unit, the comparison unit with the ambient average, and the comparison unit of the variance value is true, the bright spot detection unit detects the evaluation target pixel as a bright spot. The depth estimation data generation device according to claim 1, wherein Depth information is not given explicitly or implicitly like a stereo image, and is a pseudo stereoscopic image generating device that generates a pseudo stereoscopic moving image from a non-stereo image,
The left-eye image and / or the image by shifting the texture of the non-stereo image by an amount corresponding to the depth of the corresponding portion in accordance with the depth estimation data supplied from the depth estimation data generation device according to claim 1 or 2. A different viewpoint image generation unit that generates another viewpoint image to be a right eye image,
One of the different viewpoint images generated by the different viewpoint image generation unit and the non-stereo image is output as a left-eye image, and the other is output as a right-eye image. A depth estimation data generation method for generating depth estimation data from a non-stereo image in which depth information is not given explicitly or implicitly like a stereo image,
A calculation step of calculating a synthesis ratio between the plurality of basic depth models by using a statistic for a pixel value in a predetermined region in a screen of the non-stereo image to estimate a scene structure of the non-stereo image; ,
Combining a plurality of basic depth models based on the combining ratio calculated in the calculating step, and generating combined depth data;
A bright spot detection step for detecting a bright spot portion from the non-stereoscopic image;
When detecting a bright spot in the bright spot detection step, a bright spot interpolation step of generating a depth addition image signal by replacing the pixel value of the detected bright spot portion with a pixel value of a peripheral pixel of the bright spot portion When,
A depth estimation data generation method comprising: an addition step of adding a predetermined signal component of the depth addition image signal to the combined depth data to generate the depth estimation data. A depth estimation data generation program for causing a computer to realize a function of generating depth estimation data from a non-stereo image that is not given depth information explicitly or implicitly like a stereo image,
A calculation step of calculating a synthesis ratio between the plurality of basic depth models by using a statistic for a pixel value in a predetermined region in a screen of the non-stereo image to estimate a scene structure of the non-stereo image; ,
Combining a plurality of basic depth models based on the combining ratio calculated in the calculating step, and generating combined depth data;
A bright spot detection step for detecting a bright spot portion from the non-stereoscopic image;
When detecting a bright spot in the bright spot detection step, a bright spot interpolation step of generating a depth addition image signal by replacing the pixel value of the detected bright spot portion with a pixel value of a peripheral pixel of the bright spot portion When,
A depth estimation data generation program, comprising: an addition step of adding a predetermined signal component of the depth addition image signal to the combined depth data to generate the depth estimation data.