JP2012060381A - Depth data generation device, pseudo stereoscopic image signal generation device, depth data generation method, pseudo stereoscopic image signal generation method, depth data generation program, and pseudo stereoscopic image signal generation program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a depth data generation device and the like capable of generating a video signal of a pseudo stereoscopic image of higher accuracy from a video signal of a non-stereoscopic image without increasing a storage capacity.SOLUTION: A color frequency detection unit 11 defines an angle range in a rotation direction whose center is the intersection of an X axis and a Y axis in a color difference plane as one color axis range, sets plural color axis ranges, and detects a color frequency indicating the appearance frequency of each pixel in each field or each frame of a video signal of a non-stereoscopic image included in each color axis range. A data processing unit 12 calculates a shift limit to be a limit of a shift amount in a shift amount characteristic set for each color axis range based on the color frequency, and sets the shift limit to the shift amount characteristic. Based on the shift amount characteristic and the saturation of each pixel in the video signal of the non-stereoscopic image, a depth data estimation unit 13 calculates a shift amount that is depth data of each pixel for each color axis range.

Description

  The present invention relates to a depth data generation apparatus, a depth data generation method, and depth data generation for generating depth data from a video signal of a non-stereo image that is not given depth information either explicitly or implicitly as a stereo image. The present invention relates to a program, a pseudo stereoscopic image signal generation device that generates a pseudo stereoscopic image signal using the depth data, a pseudo stereoscopic image signal generation method, and a pseudo stereoscopic image signal generation program.

  A normal still image or moving image, that is, a pseudo-stereoscopic video signal from a non-stereoscopic video signal that is not given depth information to express a three-dimensional image, either explicitly or implicitly like a stereo image As a pseudo-stereoscopic image signal generation device that performs processing for generating a three-dimensional image data creation method based on a perspective method based on the premise that a tube having a cross-section with a contour line according to depth is known, for example. (For example, refer to Patent Document 1). In addition, in order to determine a scene structure that is as realistic as possible, a non-stereoscopic image that is scaled to a predetermined image size using multiple types of basic depth models that indicate depth values for each of the multiple types of basic scene structures. Is used to determine the composition ratio according to the high-frequency component evaluation value of the input non-stereo image from the upper high-frequency component evaluation unit and the lower high-frequency component evaluation unit, and multiple types of basics according to the composition ratio After the depth model is synthesized, the adder adds the synthesized basic depth model and the R signal of the non-stereo image, and then scales again to generate final depth data. Processing based on this depth data Is applied to a non-stereo image video signal to generate and output a video signal of another viewpoint image that gives a stereoscopic effect (for example, a special stereo image signal generation device is known). References 2).

Japanese Patent Laid-Open No. 9-185712 JP 2006-186511 A

  However, since the 3D image data creation method described in Patent Document 1 creates 3D image data based on the perspective method, it is actually applied to all scenes of various input non-stereo images. There is a problem that the effect is limited because the perspective structure estimation is not always suitable. Further, even when perspective structure estimation is suitable, there is a problem that it is not easy to automatically construct a correct depth structure model and realize stereoscopic vision without a sense of incongruity.

  In addition, in the depth data generation device described in Patent Document 2, since a plurality of basic basic depth models are stored in advance, when trying to generate a video signal of a pseudo three-dimensional image with higher accuracy, There is a problem that the number of basic depth models increases and the storage capacity increases.

  The present invention has been made in view of the above, and a depth data generation device capable of generating a more accurate pseudo-stereoscopic image signal from a non-stereoscopic image signal without increasing the storage capacity. An object of the present invention is to provide a pseudo stereoscopic image signal generation device, a depth data generation method, a pseudo stereoscopic image signal generation method, a depth data generation program, and a pseudo stereoscopic image signal generation program.

  In order to achieve the above object, according to an aspect of the present invention, there is provided a depth data generating device having the following configurations (1) and (2), a pseudo stereoscopic image signal generating device having the following configuration (3), and the following (4): , (5) Depth Data Generation Method, (6) Pseudo Stereo Image Signal Generation Method, (7), (8) Depth Data Generation Program, (9) A pseudo stereoscopic image signal generation program having the following configuration is provided.

(1)
In the color difference plane in which the level of one color difference signal of the first and second color difference signals in the video signal of the non-stereo image is the X axis, and the level of the other color difference signal is the Y axis orthogonal to the X axis, the X axis An angle range in the rotation direction centering on the intersection of the image and the Y axis is defined as one color axis range, and a plurality of the color axis ranges are set, and the video signal of the non-stereoscopic image included in each color axis range is set. Color frequency detection means for detecting a color frequency representing the frequency of appearance of each pixel for each field or frame;
The shift in the shift amount characteristic that is a characteristic with respect to the saturation of the shift amount that is depth data for generating a pseudo stereoscopic image signal from the video signal of the non-stereo image, which is set according to each color axis range A shift limit calculating means for calculating a shift limit to be an amount limit value based on the color frequency, and setting the shift limit in the shift amount characteristic;
Saturation calculating means for calculating the saturation of each pixel in the video signal of the non-stereo image;
Shift amount calculating means for calculating the shift amount of each pixel for each color axis range using the shift amount characteristic and the saturation calculated by the saturation calculating unit. Depth data generator.

(2)
The depth data generation device according to (1), wherein the shift limit calculation unit increases the shift limit as the color frequency increases in a predetermined range of the color frequency.

(3)
(1) or the depth data generation device according to (2);
Based on the shift amount calculated by the depth data generation device and the video signal of the non-stereo image, the stereo of the left-eye image signal and the right-eye image signal is shifted by shifting the texture of the video signal of the non-stereo image. And a stereo pair generating device that generates a pair as a pseudo stereoscopic image signal.

(4)
In the color difference plane in which the level of one color difference signal of the first and second color difference signals in the video signal of the non-stereo image is the X axis, and the level of the other color difference signal is the Y axis orthogonal to the X axis, the X axis An angle range in the rotation direction centering on the intersection of the image and the Y axis is defined as one color axis range, and a plurality of the color axis ranges are set, and the video signal of the non-stereoscopic image included in each color axis range is set. Detecting a color frequency representing the appearance frequency of each pixel for each field or frame;
The shift in the shift amount characteristic that is a characteristic with respect to the saturation of the shift amount that is depth data for generating a pseudo stereoscopic image signal from the video signal of the non-stereo image, which is set according to each color axis range Calculating a shift limit that is an amount limit value based on the color frequency, and setting the shift limit to the shift amount characteristic;
Calculating the saturation of each pixel in the video signal of the non-stereo image;
Calculating the shift amount of each pixel for each color axis range using the shift amount characteristic and the saturation calculated in the step of calculating the saturation. Data generation method.

(5)
The depth data generation according to (4), wherein the step of setting the shift limit in the shift amount characteristic increases the shift limit as the color frequency increases in the predetermined range of the color frequency. Method.

(6)
In the color difference plane in which the level of one color difference signal of the first and second color difference signals in the video signal of the non-stereo image is the X axis, and the level of the other color difference signal is the Y axis orthogonal to the X axis, the X axis An angle range in the rotation direction centering on the intersection of the image and the Y axis is defined as one color axis range, and a plurality of the color axis ranges are set, and the video signal of the non-stereoscopic image included in each color axis range is set. Detecting a color frequency representing the appearance frequency of each pixel for each field or frame;
The shift in the shift amount characteristic that is a characteristic with respect to the saturation of the shift amount that is depth data for generating a pseudo stereoscopic image signal from the video signal of the non-stereo image, which is set according to each color axis range Calculating a shift limit that is an amount limit value based on the color frequency, and setting the shift limit to the shift amount characteristic;
Calculating the saturation of each pixel in the video signal of the non-stereo image;
Calculating the shift amount of each pixel for each color axis range using the shift amount characteristic and the saturation calculated in the step of calculating the saturation;
Based on the shift amount calculated in the step of calculating the shift amount and the video signal of the non-stereo image, the texture of the video signal of the non-stereo image is shifted to thereby generate a left-eye image signal and a right-eye image signal. Generating a stereo pair as a pseudo stereoscopic image signal. A pseudo stereoscopic image signal generating method comprising:

(7)
In the color difference plane in which the level of one color difference signal of the first and second color difference signals in the video signal of the non-stereo image is the X axis, and the level of the other color difference signal is the Y axis orthogonal to the X axis, the X axis An angle range in the rotation direction centering on the intersection of the image and the Y axis is defined as one color axis range, and a plurality of the color axis ranges are set, and the video signal of the non-stereoscopic image included in each color axis range is set. Detecting a color frequency representing the appearance frequency of each pixel for each field or frame;
The shift in the shift amount characteristic that is a characteristic with respect to the saturation of the shift amount that is depth data for generating a pseudo stereoscopic image signal from the video signal of the non-stereo image, which is set according to each color axis range Calculating a shift limit that is an amount limit value based on the color frequency, and setting the shift limit to the shift amount characteristic;
Calculating the saturation of each pixel in the video signal of the non-stereo image;
Depth for causing a computer to execute the step of calculating the shift amount of each pixel for each color axis range using the shift amount characteristic and the saturation calculated in the step of calculating the saturation Data generation program.

(8)
The depth data generation according to (7), wherein the step of setting the shift limit in the shift amount characteristic increases the shift limit as the color frequency increases in a predetermined range of the color frequency. program.

(9)
In the color difference plane in which the level of one color difference signal of the first and second color difference signals in the video signal of the non-stereo image is the X axis, and the level of the other color difference signal is the Y axis orthogonal to the X axis, the X axis An angle range in the rotation direction centering on the intersection of the image and the Y axis is defined as one color axis range, and a plurality of the color axis ranges are set, and the video signal of the non-stereoscopic image included in each color axis range is set. Detecting a color frequency representing the appearance frequency of each pixel for each field or frame;
The shift in the shift amount characteristic that is a characteristic with respect to the saturation of the shift amount that is depth data for generating a pseudo stereoscopic image signal from the video signal of the non-stereo image, which is set according to each color axis range Calculating a shift limit that is an amount limit value based on the color frequency, and setting the shift limit to the shift amount characteristic;
Calculating the saturation of each pixel in the video signal of the non-stereo image;
Calculating the shift amount of each pixel for each color axis range using the shift amount characteristic and the saturation calculated in the step of calculating the saturation;
Based on the shift amount calculated in the step of calculating the shift amount and the video signal of the non-stereo image, the texture of the video signal of the non-stereo image is shifted to thereby generate a left-eye image signal and a right-eye image signal. A pseudo-stereoscopic image signal generation program for causing a computer to execute the step of generating a stereo pair of

  According to the present invention, it is possible to generate a pseudo-stereoscopic image video signal with higher accuracy from a non-stereoscopic image video signal without increasing the storage capacity.

It is a block diagram which shows the structural example of embodiment of the depth data generation apparatus and pseudo | simulation stereoscopic image signal generation apparatus which concern on this invention. It is explanatory drawing which shows the predetermined | prescribed determination area | region provided in the screen which is 1 field or 1 frame which a color frequency detection part detects color frequency. It is a block diagram which shows the structural example of a color frequency detection part. It is a figure which shows an example of the color axis range of each color in the color difference plane at the time of a color frequency detection part detecting a color frequency. FIG. 5 is a graph showing the frequency amount of the color frequency shown in FIG. 4. It is explanatory drawing which shows the structural example of a data processing part. It is a figure which shows the example of calculation of the gain _Glim . It is a figure which shows the structural example of a shift limit processing circuit. It is a block diagram which shows the structural example of a depth data estimation part. It is explanatory drawing which shows an example of the color difference plane which makes X-axis (horizontal axis) BY and Y-axis (vertical axis) RY. It is explanatory drawing of the correction function used in the shift amount calculation part of a depth data estimation part. It is a figure which shows an example of a shift amount characteristic.

  Next, embodiments of a depth data generation device, a pseudo stereoscopic image signal generation device, a depth data generation method, a pseudo stereoscopic image signal generation method, a depth data generation program, and a pseudo stereoscopic image signal generation program according to the present invention will be described. In the following description, the pseudo stereoscopic image signal generation device includes a depth data generation device and a stereo generation device.

<Embodiment>
Hereinafter, a depth data generation device and a pseudo stereoscopic image signal generation device according to the present invention will be described with reference to the drawings.

  FIG. 1 is a block diagram illustrating a configuration example of an embodiment of a depth data generation device and a pseudo stereoscopic image signal generation device according to the present invention.

  As illustrated in FIG. 1, the pseudo stereoscopic image signal generation device according to the present embodiment includes a depth data generation device 1 and a stereo pair generation device 2, and the depth information is explicit or is a stereo image. Thus, the left-eye image is obtained from video signals (luminance signal Y, color difference signals BY, RY) of a non-stereo image composed of a plurality of time-sequential images that are not given implicitly. A stereo pair (pseudo stereoscopic image signal) composed of the signal and the right-eye image signal is generated, and the stereo pair is output to the stereo display device 3.

  The depth data generation apparatus 1 calculates a shift amount, which is depth data for generating a pseudo stereoscopic image signal from a non-stereo image video signal, and includes a color frequency detection unit (color frequency detection means) 11, data A processing unit 12, a depth data estimation unit 13, and a storage unit 14 are included.

  The color frequency detection unit 11 uses the color difference signal level of one of the first and second color difference signals in the video signal of the non-stereo image as the X axis and the color difference signal as the Y axis orthogonal to the X axis. In the plane, an angular range in the rotation direction centering on the intersection of the X axis and the Y axis is defined as one color axis range, and a plurality of color axis ranges are set, and the video signal included in each color axis range is set. A color frequency representing the appearance frequency of each pixel for each field or frame is detected. Details of the color frequency detection unit 11 will be described later.

  Based on the color frequency detected by the color frequency detection unit 11, the data processing unit 12 uses the depth data estimation unit 13 (described later) to determine the shift amounts S1 to Sn for each of the 1 to n-axis color axis ranges. The limit lim is calculated. The data processing unit 12 sets shift points p1 to p4 described later for each color axis range. Details of the data processing unit 12 will be described later.

  The depth data estimation unit 13 calculates the saturation of each pixel in the video signal of the non-stereo image, and uses the calculated saturation and a shift amount characteristic described later for each pixel for each of the 1 to n-axis color axis ranges. Shift amounts S1 to Sn, which are depth data, are calculated. Details of the depth data estimation unit 13 will be described later.

The storage unit 14 stores in advance an initial value limit ₀ of the shift limit for each color axis range used in the data processing unit 12. Of course, the initial value lim ₀ may be stored in the data processing unit 12 in advance, and in this case, the storage unit 14 can be omitted.

  The stereo pair generation device 2 includes a texture shift unit 21 that shifts the texture of the video signal of the non-stereo image based on the shift amounts S1 to Sn generated by the depth data estimation unit 13 of the depth data generation device 1, and an occlusion. An occlusion compensation unit 22 that compensates, a post processing unit 23 that performs post processing, and a stereo pair generation unit 24 that generates a stereo pair of a left eye image signal and a right eye image signal are provided.

  The stereo display device 3 displays a pseudo three-dimensional image on the screen based on the stereo pair generated by the stereo pair generation device 2.

<< Example of Configuration and Operation of Color Frequency Detection Unit 11 >>
Next, an example of the configuration and operation of the color frequency detection unit 11 will be described.

  FIG. 2 is an explanatory diagram illustrating a predetermined determination area fa provided in a screen f that is one field or one frame in which the color frequency detection unit 11 performs color frequency detection.

  In other words, in the present embodiment, the color frequency detection unit 11 specifies, for example, a color range and performs the color frequency detection from the level of the video signal as a color frequency detection frame, so that the entire screen f is one field or one frame. Instead, a predetermined determination area fa provided in the screen f is set. Note that the area of the predetermined determination area fa is arbitrary, and the screen f can be one field or one frame.

  FIG. 3 is a block diagram illustrating a configuration example of the color frequency detection unit 11.

  As shown in FIG. 3, the color frequency detection unit 11 includes a hue comparator 111, a saturation comparator 112, a logical product (AND) gate 113, a counter 114, and a register 115.

  Here, the hue comparator 111 has a specified hue range in the video signal, that is, L0−L1 ≦ hue <L0 + L2 (where L0, L1, and L2 are arbitrary hue values and are shown in FIG. It is determined whether or not the expression of the hue line is satisfied. The hue comparator 111 outputs “1” when the hue satisfies L0−L1 ≦ hue <L0 + L2, and outputs “0” when the hue does not satisfy.

  The saturation comparator 112 determines whether or not the saturation in the video signal is larger than a predetermined threshold (for example, a value equal to a shift point p1 described later is preferable, but not limited thereto). The saturation comparator 112 outputs “1” when it is larger than the predetermined threshold, and outputs “0” when it is smaller than the predetermined threshold.

  A logical product (AND) gate 113 takes a logical product (AND) of the output of the hue comparator 111 and the output of the saturation comparator 112 and outputs the result to the counter 114.

  The counter 114 counts up the output of the AND gate 113 and increments the count value by one. Thus, the count value of the counter 114 indicates the total number of pixels that are included in the hue level region that satisfies the condition of the hue comparator 111 and whose saturation level satisfies the condition of the saturation comparator 112. This counter value becomes color frequency data.

  The register 115 holds the count value of the counter 114 and outputs the count value to the data processing unit 12 as the color frequency.

  As described above, the color frequency detection unit 11 is included in a hue level region that satisfies the predetermined condition (L0−L1 ≦ hue <L0 + L2) of the hue comparator 111, which is counted by the counter 114 and held in the register 115, and The total number of pixels whose saturation level satisfies a predetermined condition (shift point p1 <saturation) of the saturation comparator 112 is detected, and the counter value is supplied to the data processing unit 12 as color frequency data. That is, the color frequency detection unit 11 describes the saturation level in the color axis range of each color for each color axis range (also referred to as a color range) of 1 to n-axis colors described later that satisfies the condition of the hue comparator 111. The total number of pixels larger than the shift point p1 is output as color frequency data (CHx, x = 1 to n).

  FIG. 4 is a diagram illustrating an example of the color axis range of each color on the color difference plane when the color frequency detection unit 11 illustrated in FIG. 1 detects the color frequency.

  That is, as shown in FIG. 4, the color frequency detection unit 11 sets the level of one color difference signal BY of the color difference signals RY and BY in the video signal as the X axis, and the other color difference signal RY. In the color difference plane whose level is the Y axis perpendicular to the X axis, an angular range in the rotation direction around the intersection of the X axis and the Y axis is defined as one color axis range. The color frequency detection unit 11 sets a plurality (1 to n) of the color axis ranges, and the saturation level for each field or frame of the video signal included in the 1 to n axis color axis ranges is described later. A color frequency representing the appearance frequency of pixels larger than p1 is detected.

  FIG. 5 is a graph showing the frequency amount of the color frequency shown in FIG.

  5, the horizontal axis indicates the color axis range of 1 to n axes, which is the angle range in the orthogonal coordinates of the color difference plane shown in FIG. 4, while the vertical axis indicates the determination region shown in FIG. The value of the color frequency detected within each color axis range of 1 to n axes in fa is taken. Here, as described above, the value of the color frequency represents the appearance frequency of a pixel whose saturation level is greater than the shift point p1.

<< Example of Configuration and Operation of Data Processing Unit 12 >>
Next, an example of the configuration and operation of the data processing unit 12 will be described.

  FIG. 6 is an explanatory diagram illustrating a configuration example of the data processing unit 12.

  As shown in FIG. 6, the data processing unit 12 includes a gain calculator 121, a shift limit calculator (shift limit calculation means) 122, and a shift point setter 123.

The gain calculator 121 determines the color range from the color frequency detection unit 11 based on the color frequency data CH1, CH2,..., CHn from the 1 to n axis color axis ranges. The gain G _lim is calculated.

As shown in the following equation (1), the shift limit calculator 122 multiplies the gain G _lim for each color axis range calculated by the gain calculator 121 by the shift limit initial value lim ₀ to obtain the color axis range. Every time, a shift limit lim which is the maximum shift amount (shift amount limit value) is calculated. The shift limit calculator 122 sets the calculated shift limit lim as the shift limit lim in the shift amount characteristics used in the shift amount calculation units 1341 to 134n of the depth data estimation unit 13 described later.

lim = lim ₀ × G _lim (1) Here, details of the relationship between the color frequency, the gain G _lim, and the shift limit lim which is the maximum shift amount will be described later. In the embodiment, the gain G _lim is increased so that the shift limit lim increases as the color frequency increases.

  The shift point setting unit 123 sets the shift points p1 to p4 for each color axis range in the shift amount calculation units 1341 to 134n of the depth data estimation unit 13. As will be described later, the shift points p1 to p4 are set according to hues such as warm colors and cold colors. The shift point setting unit 123 receives color axis information indicating each of the 1 to n axis color axis ranges from the color frequency detection unit 11, and the shift point setting unit 123 determines the color axis range based on the color axis information. Shift points p1 to p4 are set according to the hue.

  The number of shift points is not limited to 4 points (p1 to p4), and may be 3 points or 5 points or more.

<< Example of Operation of Gain Calculator 121 of Data Processing Unit 12 >>
FIG. 7 is a diagram illustrating a calculation example of the gain G _lim by the gain calculator 121. As described above, the gain G _lim calculated by the gain calculator 121 is multiplied by the shift limit initial value lim ₀ by the shift limit calculator 122.

In FIG. 7, the horizontal axis indicates the color frequency, and the vertical axis indicates the gain _Glim . When the gain G _lim is 1, the value of the shift limit lim calculated by the shift limit calculator 122 is the initial value lim ₀ . The initial value limit ₀ of the shift limit is stored in the storage unit 14 for each color axis range.

As shown in FIG. 7, the gain calculator 121, in a predetermined range of color frequency (r1 and r2), in accordance with the color frequency (color area) is increased to increase the gain _{G lim.} The gain calculator 121 sets the gain _Glim to 0 when the color frequency is less than r1, and sets the gain _Glim to the maximum value r3 when the color frequency is greater than r2. The maximum color frequency is 1, and the values of r1 and r2 are, for example, r1 = 0.2 and r2 = 0.9. The value of r3 is, for example, r3 = 1.4. r1, r2, and r3 are values that the gain calculator 121 can variably set. Note that the minimum value of the gain _Glim may not be zero. r3 is set to a value larger than 1.

The gain calculator 121 changes the value of the shift limit lim calculated by the shift limit calculator 122 according to the color frequency by using such a value of the gain G _lim , and each color of 1 to n axes The shift amounts S1 to Sn in the axial range are controlled.

  Here, if the shift limit lim is changed for each field or frame, when a scene change or the like occurs, the shift limit lim changes steeply, and there is a possibility that a problem occurs in the case of a moving image. Therefore, there is a case where it is desired to moderate the change of the shift limit lim to some extent. Even if the change in the color frequency changes sharply, if the shift limit lim is changed gradually, and as a result, the shift amounts S1 to Sn to be applied to the video signal finally change gradually, then in the case of a moving image. There is no problem.

  FIG. 8 is a diagram showing a configuration of the shift limit processing circuit 1221 for moderating the change of the shift limit lim as described above. The shift limit processing circuit 1221 is provided in the shift limit calculator 122.

  As shown in FIG. 8, the shift limit processing circuit 1221 includes an adder 1222, a register 1223, a multiplier 1224, and a multiplier 1225.

The adder 1222, the register 1223, and the multiplier 1224 constitute a leaky integration circuit 1226 in the time direction. The register 1223 stores a shift limit value lim _k−1 one field before or one frame before.

When the current field or frame shift limit value lim _k is input to the leaky integration circuit 1226, the multiplier 1224 performs an operation of lim _k−1 × 15/16. The adder 1222 adds the current field or frame shift limit value lim _k and the value calculated by the multiplier 1224 (lim _k−1 × 15/16), and lim = lim _k + (lim _{k −1} × _15/16 ) is calculated, and the calculation result is stored in the register 1223.

  Then, the multiplier 1225 performs an operation of lim ′ = lim × (1/16), and uses this value (lim ′) as the value of the current shift limit of the depth data estimation unit 13 shown in FIG. The shift amount calculation units 1341 to 134n are set.

  As a result, static processing can be changed to dynamic processing, and a change in shift limit lim between fields or frames can be moderated.

<< Example of Configuration and Operation of Depth Data Estimation Unit 13 >>
Next, an example of the configuration and operation of the depth data estimation unit 13 will be described.

  FIG. 9 is a block diagram illustrating a configuration example of the depth data estimation unit 13.

  As shown in FIG. 9, the depth data estimation unit 13 includes an LPF 131, an angle calculation unit 132, a saturation calculation unit (saturation calculation unit) 133, and a shift amount calculation unit for each color axis range of 1 to n axes ( Shift amount calculation means) 1341 to 134n.

  When the color difference signals RY and BY of the video signal are input, the LPF 131 removes noise and outputs it to the angle calculation unit 132 and the saturation calculation unit 133.

  The angle calculation unit 132 calculates the angle T of the color difference signals RY and BY for each pixel.

  The saturation calculation unit 133 calculates a saturation VC that is the magnitude of the vector of the color difference signals RY and BY for each pixel. An equation for calculating the saturation VC, which is the magnitude of the color difference signals RY and BY, is shown in the following equation (2).

VC = {(R−Y) ² + (B−Y) ² } ^1/2 (2) Formula Here, the color difference signals R−Y and B−Y are expressed as B−Y as shown in FIG. 10, for example. It can be represented by a color difference plane with the X axis (horizontal axis) and RY as the Y axis (vertical axis). In this color difference plane, the rotation direction represents hue and the radial direction represents saturation.

Further, L0, L1, and L2 at angles θ ₀ , θ ₁ , and θ ₂ (θ ₁ <θ ₀ <θ ₂ ) represent equal hue lines. The equi-hue line L0 is a correction center line and is set by the correction center angle θ ₀ , and the angles θ _{1 to} θ ₂ are correction regions. The values of the equi-hue lines L0, L1, and L2 are used in the hue comparison in the hue comparator 111 of the color frequency detection unit 11 shown in FIG.

As shown in FIG. 9, the shift amount calculation units 1341 to 134n for the respective color axis ranges of 1 to n axes include the angle T of the color difference signals RY and BY and the saturation calculation unit 133 from the angle calculation unit 132, as shown in FIG. Saturation VC, which is the magnitude of the color difference signals RY and BY, is input from the angle θ ₀ , θ ₁ , θ ₂ , or angle θ ₀ and the correction range R that determine the respective correction regions. Is entered.

Here, the angles θ ₀ , θ ₁ , and θ ₂ supplied to the shift amount calculation units 1341 to 134n are θ ₀₁ , θ ₁₁ , θ _{21 to} θ _0n , θ _1n , and θ _2n , respectively. The angles θ _{01 to} θ _0n are collectively referred to as an angle θ ₀ , the angles θ _{11 to} θ _1n are collectively referred to as an angle θ ₁ , and the angles θ _{21 to} θ _2n are collectively referred to as an angle θ ₂ . It should be noted that the correction areas of the angles θ _{1 to} θ ₂ need to be set so as not to overlap.

  Next, functions of the shift amount calculation units 1341 to 134n will be further described with reference to FIGS.

  FIG. 11 is an explanatory diagram of the correction function M12 used in the shift amount calculation units 1341 to 134n of the depth data estimation unit 13 of the present embodiment shown in FIG.

11, a function indicating a large part of the angle than isochromatic line L1 of FIG. 10 and F _L1, expressed this example in the C language, the following equation (3).

F _L1 = T−θ ₁ ;
if (F _L1 <0) F _L1 = 0; Formula (3) This formula (3) has the characteristics shown in FIG.

Similarly, the function representing the angle of portion smaller than an equal hue line L2 in FIG. 10 and F _L2, Expressing this in C language, the following equation (4).

F _L2 = θ ₂ −T;
if (F _L2 <0) F _L2 = 0; Formula (4) This formula (4) has the characteristics shown in FIG.

  The shift amount calculation units 1341 to 134n generate a correction function M12 in order to obtain a shift amount that is depth data based on the following equation (5).

M12 = Min (F _L1 , F _L2 ); Equation (5) The correction function M12 expressed by the equation (5) is F _L1 expressed by the above equation (3) and F expressed by the equation (4). The smaller of _L2 is selected. Therefore, the correction function M12 represented by the aforementioned formula (5), a triangular characteristic with a vertex angle theta ₀ as shown in FIG. 11 (C). The characteristic of the correction function M12 is not limited to a triangular shape, and may be trapezoidal by setting an upper limit value, or may be a cosine function. The correction function M12 needs to set the outside of the angles θ ₁ and θ ₂ outside the correction range to 0.

By the way, when θ ₁ = θ ₀ −R and θ ₂ = θ ₀ + R, the angle T calculated by the angle calculator 132 is a value of 0 to 360 °, and the angles θ ₁ and θ ₂ are 0 °. Or, when crossing 360 °, discontinuity occurs in the angle value.

  Therefore, the shift amount calculation units 1341 to 134n correct the angle T used in the above formulas (3) and (4) in advance using the following formula (6) expressed in C language, for example.

if (T−θ ₁ ) ≧ 360) T = T-360;
if (θ ₂ −T) ≧ 360) T = T + 360; Expression (6) The shift amount calculation units 1341 to 134n have the angle θ ₁ and the angle θ ₂ cross 0 ° or 360 ° according to the expression (6). even if, when included angle T in the range of the angle theta ₁ through? _2, by correcting the angle T in successive values, in the range of the angle theta ₁ through? ₂ angle T value It is possible to prevent the discontinuity from occurring. Therefore, the correction function M12 having the characteristics shown in FIG. 11C can be generated correctly.

As described above, the shift amount calculation units 1341 to 134n set the correction angle R to 180 ° when θ ₁ = θ ₀ −R and θ ₂ = θ ₀ + R are set as correction regions centered on the angle θ ₀ . A correction region up to the vicinity, that is, θ ₀ ± 180 ° can be set.

  Here, since the shift amount calculation units 1341 to 134n are provided according to the number n of colors to be corrected, respectively, as shown in the following equation (7), the color shown by the above equation (2) is used. The value of the correction function M12 of the characteristic shown in FIG. 11C expressed by the above equation (5) is added to the saturation VC which is the magnitude of the color difference signals RY and BY from the degree calculation unit 133. To calculate the corrected saturation VC ′.

VC ′ = VC × M12 (7) Then, the shift amount calculation units 1341 to 134n use the corrected saturation VC ′ calculated by the above equation (7) as a parameter, that is, the color to be corrected, that is, 1 to n. The shift amounts S1 to Sn, which are depth data, are calculated for each color axis range of the axis.

At that time, as shown in FIG. 9, the shift amount calculation units 1341 to 134n for each color axis range have predetermined correction center angles θ ₀ and angles θ _{1 to} θ ₂ (or correction center angles) corresponding to the colors to be corrected. θ ₀ , correction range R), and shift points p 1 to p 4 from the data processing unit 12 (see FIG. 6) in order to vary the degree of shift amounts S 1 to Sn, which are depth data for each color axis range of 1 to n axes. , And a shift limit lim indicating the maximum shift amount is input and set independently.

  Therefore, the shift amount calculation units 1341 to 134n independently calculate the shift amounts S1 to Sn, which are depth data, for each of the color axis ranges of the 1 to n axes based on the following equation (8). .

if (VC ′ ≦ p1) S = 0;
if (VC ′ ≦ p2) S = lim × (VC′−p1) / (p2−p1);
if (VC ′ ≦ p3) S = lim;
if (VC ′ ≦ p4) S = lim × (p4−VC ′) / (p4−p3);
else S = 0; Equation (8) This equation (8) has characteristics as shown in FIGS.

  12A to 12C are diagrams illustrating an example of the shift amount characteristic that is a characteristic with respect to the saturation of the shift amounts S1 to Sn. 12A to 12C, the horizontal axis represents the saturation VC ′ corrected in the above equation (7), and the vertical axis represents the shift amount S which is depth data. 12A to 12C, p1 to p4 indicate the values of the shift points p1 to p4.

  First, FIG. 12A shows a characteristic in which a positive shift amount increases in the case of medium saturation as an example. That is, in FIG. 12A, the shift amount S is 0 when the saturation VC ′ is from 0 to p1, and the shift amount S increases as the saturation VC ′ increases from p1 to p2, and from p2 to p3. The shift amount S becomes an upper limit shift limit lim until the shift amount S decreases with increasing saturation VC ′ from p3 to p4, and the shift amount S becomes 0 when greater than p4. In FIG. 12A, p2 and p3 are set so that the shift amount S becomes the shift limit lim in the middle saturation.

  In FIG. 12B, the shift amount S is 0 when the saturation VC ′ is from 0 to p1, and the shift amount S increases as the saturation VC ′ increases from p1 to p2 of high saturation. To p3 and p4, the shift amount S becomes the shift limit lim which is the upper limit (limit value). FIG. 12B shows the case where p3 and p4 have the same value, but these may be different values. In this case, the shift amount S decreases as the saturation VC ′ increases from p3 to p4, and when larger than p4, the shift amount S may be zero.

  The shift amount characteristic shown in FIG. 12B is a characteristic for generating a pseudo-stereoscopic image in which the protrusion (protrusion) becomes larger as the color becomes darker in the case of warm colors (red, magenta, etc.). .

  FIG. 12C shows a characteristic in which the negative shift amount increases in the case of medium saturation. That is, in FIG. 12C, the shift amount S is 0 when the saturation VC ′ is from 0 to p1, and the shift amount S decreases as the saturation VC ′ is decreased from p1 to p2, and from p2 to p3. The shift amount S becomes a shift limit lim which is the lower limit (limit value) until the shift amount S increases as the saturation VC ′ increases from p3 to p4, and the shift amount S becomes 0 when greater than p4. . In FIG. 12C, p1 to p4 similar to those in FIG.

  The shift amount characteristic shown in FIG. 12C is a characteristic that generates a pseudo-stereoscopic image that has the largest saturation (concaveness) in intermediate saturation in the case of a cold color shade (blue, cyan, etc.). is there.

  In the present embodiment, the following two points are taken into account as factors for determining the shift amount characteristics of FIGS.

  The first point is that red and warm colors are forward colors in chromaticity, and the depth is recognized in front of the cold color system. Therefore, the three-dimensional effect can be enhanced by arranging the red and warm color portions in front. On the contrary, blue and cold colors are receding colors, and the depth is recognized deeper than the warm color system. Therefore, it is possible to enhance the stereoscopic effect by arranging the blue and cold portions in the back.

  The second point is that the color is deep, that is, high saturation, the depth is recognized in the foreground, and conversely, the light color, that is, low saturation, the depth is recognized in the back. There is a feature. This is due to the phenomenon of "atmosphere perspective" where light transmitted from a distance is diffused by the atmosphere, and nearby colors are highly saturated, but distant colors are reduced in saturation and gather in similar hues. It is.

  Based on the above two points, the shift amount calculation units 1341 to 134n generate the shift amount S that is depth data based on the shift amount characteristics of FIGS. The shift amount characteristics are set in the shift amount calculation units 1341 to 134n by setting the shift limit lim and the shift points p1 to p4 by the shift limit calculator 122 and the shift point setting unit 123 of the data processing unit 12, respectively. Is done. The shift amount calculation units 1341 to 134n calculate shift amounts S1 to Sn, which are depth data of each pixel, for each color axis range, using the shift amount characteristics and saturation. The shift limit lim and the shift points p1 to p4 can be freely set by the data processing unit 12 independently for each color axis range, that is, for the shift amount calculation units 1341 to 134n of the depth data estimation unit 13.

  Note that the shift amount characteristics described above are examples of the present embodiment, and the present invention is not limited to these characteristics.

  Through the above processing, the depth data estimation unit 13 generates shift amounts S1 to Sn that are depth data of the color axis ranges of 1 to n axes, and outputs them to the stereo pair generation device 2.

  As described above, in the depth data generation device 1 of the present embodiment, the color frequency representing the appearance frequency of each pixel for each field or frame of the video signal included in each color axis range from the video signal of the non-stereo image. The shift limit lim in the shift amount characteristic for each color axis range is calculated based on the color frequency, and the depth data of each pixel is calculated for each color axis range using the saturation and shift amount characteristic of each pixel. A certain shift amount S1 to Sn is calculated. This eliminates the need to store a plurality of types of basic depth models in advance, and does not increase the storage capacity, and is highly accurate from a video signal of a non-stereoscopic image in which depth information is not given explicitly or implicitly. A video signal of a pseudo three-dimensional image can be generated.

  As a result, according to the present embodiment, a non-stereoscopic image is considered in consideration of human visual characteristics from a video signal of a non-stereoscopic image in which depth information is not given explicitly or implicitly like a stereo image. It is possible to estimate the depth of each color according to the color tone, color shading, and color area, and generate a more accurate pseudo three-dimensional image signal for various input scenes. It becomes possible.

≪Example of operation of stereo pair generation device 2≫
In the stereo pair generation device 2 of the present embodiment shown in FIG. 1, another viewpoint is based on the shift amounts S1 to Sn that are depth data generated by the depth data generation device 1 and the input video signal. An image signal is generated. For example, when moving the viewpoint to the left, for objects that are displayed in front of the screen, the closer the object, the closer to the viewer (the nose side) the more visible, the amount of texture corresponding to the depth on the inside, that is, the right Just move. On the other hand, as for the object to be displayed deeper than the screen, the closer the object is to the outside of the person viewing the image, the corresponding part of the texture is moved to the left by an amount corresponding to the depth. A stereo pair is formed by using this as the left-eye image and the original image as the right-eye image.

  That is, in the stereo pair generation device 2 of the present embodiment shown in FIG. 1, first, the texture shift unit 21 outputs each of the 1 to n-axis color axis ranges output from the depth data estimation unit 13 of the depth data generation device 1. With respect to the shift amounts S1 to Sn as depth data, the textures of the video signal corresponding to the shift amount S pixels, for example, rightward, in order from the values with the smaller shift amounts S1 to Sn, that is, those located in the back. Shift to. If the shift amount S, which is depth data, is negative, the shift is made to the left by the shift amount S pixels.

  The texture shift operation of the video signal based on the shift amount S in the texture shift unit 21 corresponds to the shift of the texture of the video signal of the non-stereo image. In other words, this is a process of moving each pixel of the non-stereo image to the left and right according to the value of the shift amount S that is depth data.

  Here, there is a case where a portion where no texture exists, that is, an occlusion occurs due to a change in the positional relationship in the image due to the shift. For such a part, the occlusion compensation unit 22 fills the video signal with a video signal around the corresponding part of the video signal, or a known document (Kunio Yamada, Kenji Mochizuki, Kiyoharu Aizawa, Takahiro Saito: “Division by region competition method” Occlusion compensation based on the statistics of the texture of the image that has been processed ", the Journal of Emotionology, Vol.56, No.5, pp.863-866 (2002.5)).

  The image that has been subjected to occlusion compensation by the occlusion compensation unit 22 is subjected to post-processing such as smoothing by the post-processing unit 23 to reduce noise generated in the previous processing, thereby being a different viewpoint image signal. A left eye image signal is generated.

  Then, the stereo pair generation unit 24 generates a stereo pair of the left-eye image signal and the right-eye image signal as a pseudo-stereoscopic image signal by using the input video signal as the right-eye image signal, and this stereo pair is sent to the stereo display device 3. Is output. Note that the stereo pair may be reversed left and right to form a stereo pair in which the input video signal is the left eye image signal and the generated different viewpoint image signal is the right eye image signal. In the above processing, a stereo pair is formed in which one of the right-eye image signal and the left-eye image signal is used as an input video signal, and the other is generated as another viewpoint image signal. It is also possible to form a stereo pair using a signal, that is, using another viewpoint image signal moved to the right and another viewpoint image signal moved to the left.

<< Example of operation of stereo display device 3 >>
The stereo display device 3 of the present embodiment shown in FIG. 1 includes, for example, a projection system using polarized glasses, or a projection system or display system that combines time-division display and liquid crystal shutter glasses, a lenticular stereo display, An anaglyph-type stereo display, a head-mounted display, or the like, or a projector system using two projectors corresponding to each image of a stereo image. Are displayed on a display or the like to display a pseudo stereoscopic image.

  As described above, by combining the depth data generation device 1, the stereo pair generation device 2, and the stereo display device 3 described above, depth information is given explicitly or implicitly like a stereo image. It is possible to estimate the depth of each color according to the color of the non-stereo image, the shade of the color, and the color area from the video signal of the non-stereo image taking into account human visual characteristics, and various scenes to be input On the other hand, it is possible to display a video signal of a pseudo three-dimensional image with higher accuracy and allow the user to stereoscopically view it as a three-dimensional image.

  Therefore, according to the pseudo stereoscopic image signal generation device of the present embodiment, depth data is generated from a video signal of a non-stereo image to which depth information is not given explicitly or implicitly. By applying the above processing to a non-stereo image video signal, a different viewpoint image signal that gives a stereoscopic effect is generated, and a pseudo stereoscopic image signal is generated from the non-stereo image video signal and the different viewpoint image signal. Therefore, by taking into account human visual characteristics and estimating the depth according to the color of the non-stereo image, the shade of the color, and the area of the color, it is possible to simulate more accurately for various input scenes. It is possible to generate a video signal of a three-dimensional image.

  In the description of the above embodiment, the stereo pair generation device 2 has been described with the two viewpoints of the left-eye image signal and the right-eye image signal. However, the present invention is not limited to this, and display of two or more viewpoints is possible. In the case of display on a simple display device, the number of different viewpoint image signals may be generated according to the number of viewpoints.

  In addition, as described above, it is possible to construct a multi-viewpoint stereoscopic image display system using a display device that can display two or more viewpoints. Further, the present stereoscopic display system may be configured to be equipped with an audio output. In this case, for image content that does not have audio information, such as still images, an aspect in which an environmental sound suitable for an image is added can be considered.

  In the present embodiment, as shown in FIG. 1, the depth data generating device 1, the stereo pair generating device 2, and the stereo display device 3 are configured by hardware. However, in the present invention, the hardware The depth data generation device 1, the stereo pair generation device 2, and the stereo display device 3 are configured by, for example, a CPU and software of a computer program for operating the CPU as described above. Of course, the function may be achieved. In this case, the computer program may be taken into the computer from a recording medium, or may be taken into the computer via a network.

  Although the present invention has been described above with reference to one embodiment, the above embodiment exemplifies an apparatus and method for embodying the technical idea of the present invention. The material, shape, structure, arrangement, etc. of the constituent articles are not specified as follows. The technical idea of the present invention can be variously modified within the scope of the claims.

DESCRIPTION OF SYMBOLS 1 Depth data generation apparatus 11 Color frequency detection part 12 Data processing part 122 Shift limit calculator 13 Depth data estimation part 133 Saturation calculation part 1341-134n Shift amount calculation part 14 Storage part 2 Stereo pair generation apparatus 21 Texture shift part 22 Occlusion Compensation unit 23 Post processing unit 24 Stereo pair generation unit 3 Stereo display device

Claims (9)

In the color difference plane in which the level of one color difference signal of the first and second color difference signals in the video signal of the non-stereo image is the X axis, and the level of the other color difference signal is the Y axis orthogonal to the X axis, the X axis An angle range in the rotation direction centering on the intersection of the image and the Y axis is defined as one color axis range, and a plurality of the color axis ranges are set, and the video signal of the non-stereoscopic image included in each color axis range is set. Color frequency detection means for detecting a color frequency representing the frequency of appearance of each pixel for each field or frame;
The shift in the shift amount characteristic that is a characteristic with respect to the saturation of the shift amount that is depth data for generating a pseudo stereoscopic image signal from the video signal of the non-stereo image, which is set according to each color axis range A shift limit calculating means for calculating a shift limit to be an amount limit value based on the color frequency, and setting the shift limit in the shift amount characteristic;
Saturation calculating means for calculating the saturation of each pixel in the video signal of the non-stereo image;
Shift amount calculating means for calculating the shift amount of each pixel for each color axis range using the shift amount characteristic and the saturation calculated by the saturation calculating unit. Depth data generator.   The depth data generation apparatus according to claim 1, wherein the shift limit calculation unit increases the shift limit as the color frequency increases in a predetermined range of the color frequency. The depth data generation device according to claim 1 or 2,
Based on the shift amount calculated by the depth data generation device and the video signal of the non-stereo image, the stereo of the left-eye image signal and the right-eye image signal is shifted by shifting the texture of the video signal of the non-stereo image. And a stereo pair generating device that generates a pair as a pseudo stereoscopic image signal. In the color difference plane in which the level of one color difference signal of the first and second color difference signals in the video signal of the non-stereo image is the X axis, and the level of the other color difference signal is the Y axis orthogonal to the X axis, the X axis An angle range in the rotation direction centering on the intersection of the image and the Y axis is defined as one color axis range, and a plurality of the color axis ranges are set, and the video signal of the non-stereoscopic image included in each color axis range is set. Detecting a color frequency representing the appearance frequency of each pixel for each field or frame;
The shift in the shift amount characteristic that is a characteristic with respect to the saturation of the shift amount that is depth data for generating a pseudo stereoscopic image signal from the video signal of the non-stereo image, which is set according to each color axis range Calculating a shift limit that is an amount limit value based on the color frequency, and setting the shift limit to the shift amount characteristic;
Calculating the saturation of each pixel in the video signal of the non-stereo image;
Calculating the shift amount of each pixel for each color axis range using the shift amount characteristic and the saturation calculated in the step of calculating the saturation. Data generation method.   5. The depth data generation according to claim 4, wherein the step of setting the shift limit in the shift amount characteristic increases the shift limit as the color frequency increases in a predetermined range of the color frequency. Method. In the color difference plane in which the level of one color difference signal of the first and second color difference signals in the video signal of the non-stereo image is the X axis, and the level of the other color difference signal is the Y axis orthogonal to the X axis, the X axis An angle range in the rotation direction centering on the intersection of the image and the Y axis is defined as one color axis range, and a plurality of the color axis ranges are set, and the video signal of the non-stereoscopic image included in each color axis range is set. Detecting a color frequency representing the appearance frequency of each pixel for each field or frame;
The shift in the shift amount characteristic that is a characteristic with respect to the saturation of the shift amount that is depth data for generating a pseudo stereoscopic image signal from the video signal of the non-stereo image, which is set according to each color axis range Calculating a shift limit that is an amount limit value based on the color frequency, and setting the shift limit to the shift amount characteristic;
Calculating the saturation of each pixel in the video signal of the non-stereo image;
Calculating the shift amount of each pixel for each color axis range using the shift amount characteristic and the saturation calculated in the step of calculating the saturation;
Based on the shift amount calculated in the step of calculating the shift amount and the video signal of the non-stereo image, the texture of the video signal of the non-stereo image is shifted to thereby generate a left-eye image signal and a right-eye image signal. Generating a stereo pair as a pseudo stereoscopic image signal. A pseudo stereoscopic image signal generating method comprising: In the color difference plane in which the level of one color difference signal of the first and second color difference signals in the video signal of the non-stereo image is the X axis, and the level of the other color difference signal is the Y axis orthogonal to the X axis, the X axis An angle range in the rotation direction centering on the intersection of the image and the Y axis is defined as one color axis range, and a plurality of the color axis ranges are set, and the video signal of the non-stereoscopic image included in each color axis range is set. Detecting a color frequency representing the appearance frequency of each pixel for each field or frame;
The shift in the shift amount characteristic that is a characteristic with respect to the saturation of the shift amount that is depth data for generating a pseudo stereoscopic image signal from the video signal of the non-stereo image, which is set according to each color axis range Calculating a shift limit that is an amount limit value based on the color frequency, and setting the shift limit to the shift amount characteristic;
Calculating the saturation of each pixel in the video signal of the non-stereo image;
Depth for causing a computer to execute the step of calculating the shift amount of each pixel for each color axis range using the shift amount characteristic and the saturation calculated in the step of calculating the saturation Data generation program.   8. The depth data generation according to claim 7, wherein the step of setting the shift limit in the shift amount characteristic increases the shift limit as the color frequency increases in a predetermined range of the color frequency. program. In the color difference plane in which the level of one color difference signal of the first and second color difference signals in the video signal of the non-stereo image is the X axis, and the level of the other color difference signal is the Y axis orthogonal to the X axis, the X axis An angle range in the rotation direction centering on the intersection of the image and the Y axis is defined as one color axis range, and a plurality of the color axis ranges are set, and the video signal of the non-stereoscopic image included in each color axis range is set. Detecting a color frequency representing the appearance frequency of each pixel for each field or frame;
The shift in the shift amount characteristic that is a characteristic with respect to the saturation of the shift amount that is depth data for generating a pseudo stereoscopic image signal from the video signal of the non-stereo image, which is set according to each color axis range Calculating a shift limit that is an amount limit value based on the color frequency, and setting the shift limit to the shift amount characteristic;
Calculating the saturation of each pixel in the video signal of the non-stereo image;
Calculating the shift amount of each pixel for each color axis range using the shift amount characteristic and the saturation calculated in the step of calculating the saturation;
Based on the shift amount calculated in the step of calculating the shift amount and the video signal of the non-stereo image, the texture of the video signal of the non-stereo image is shifted to thereby generate a left-eye image signal and a right-eye image signal. A pseudo-stereoscopic image signal generation program for causing a computer to execute the step of generating a stereo pair of

Images