JP2011216076A - Image processor, image processing method, and image display device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To generate a depth calculation method to be used to generate a parallax image representing unevenness of an object with a three-dimensional effect by depth distribution in a screen calculated by a plurality of methods.SOLUTION: A parallax image generation device includes a first depth calculating part, a second depth calculating part and a third combining part. The first depth calculating part includes a first estimating part for estimating distribution information in accordance with a specific pixel value pattern, a second estimating part for estimating distribution information on the basis of the movement of a first image, and a first combining part for combining the distribution information estimated by the first estimating part with the distribution information estimated by the second estimating part to generate first depth information. The second depth calculating part generates second depth information showing relative unevenness of each object in the first image. The third combining part combines the first depth information with the second depth information by a method different from the first combining part to generate third depth information.

Description

  The present disclosure relates to an image processing device, an image processing method, and an image display device.

  There is a technique for estimating depth information in a screen based on a two-dimensional image (including a still image and a moving image) and generating a parallax image according to the estimated depth information.

  A method of blending models representing the depth of the entire screen according to the high frequency components of the image, and adding the red signal that is one of the color signals of the image to the blended depth model to obtain the final depth Is disclosed (for example, Patent Document 1). Further, a method is disclosed in which the depth of the entire screen is estimated by three methods, and the final depth is obtained by weighted averaging of the three estimated depths (for example, Non-Patent Document 1).

JP 2005-151534 A

C.-C. Cheng, C.-T. Li, Y.-M.Tsai and L.-G. Chen, "A Quality-Scalable Depth-Aware Video Processing System", SID2009

  An object of the present invention is to generate depth information used to generate a parallax image representing the unevenness of an object while maintaining a stereoscopic effect due to the depth distribution in the screen obtained by a plurality of methods.

  In order to solve the above problem, a parallax image generation device according to an aspect of the present invention includes a first depth calculation unit, a second depth calculation unit, and a third synthesis unit. A first depth calculating unit configured to estimate distribution information according to a specific pixel value pattern; a second estimating unit configured to estimate distribution information based on a motion of the first image; and A first combining unit that combines the distribution information estimated by the second estimating unit to generate first depth information. The second depth calculation unit generates second depth information indicating relative unevenness for each object in the first image. The third synthesis unit generates the third depth information by synthesizing the first depth information and the second depth information by a method different from that of the first synthesis unit.

The figure which shows the parallax image generation apparatus of 1st Embodiment. The figure which shows operation | movement of the parallax image generation apparatus of 1st Embodiment. (A) The figure which shows the example of the depth model of distribution information (b) The figure which shows the example of the depth model of uneven | corrugated information. The figure explaining the method of calculating a parallax vector from depth information. It is a figure explaining the method to produce | generate a parallax image from a parallax vector. The figure which shows the example of depth information. The figure which shows the example of 1st depth information and 2nd depth information. The figure which shows the example of 3rd depth information. The figure which shows the example of the conversion model from depth information to parallax.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. Note that, in the following embodiments, the same numbered portions are assumed to perform the same operation, and repeated description is omitted.

(First embodiment)
The present embodiment relates to an apparatus that generates at least one parallax image having parallax from a first image that is a two-dimensional image (including a still image and a moving image). In the following embodiment, a method for generating two parallax images is illustrated, but the present invention is not limited to this. For example, in the case of applications such as stereoscopic viewing with the naked eye, the number of parallax images corresponding to the number of viewpoints may be generated.

FIG. 1A is a diagram illustrating a parallax image generation device according to the present embodiment. The parallax image generation device includes a first depth calculation unit 100, a second depth calculation unit 200, a synthesis unit 300, and a generation unit 400.
The first depth calculation unit 101 includes N (N ≧ 2) or more estimation units 101 (1) to 101 (N) and a synthesis unit 103.
The estimation units 101 (1) to 101 (N) are distributions indicating the distribution of the depth in the first image in the reproduction range in the three-dimensional space for each of the estimation units 101 (1) to 101 (N) by different methods. Estimate information.
The synthesizing unit 103 obtains first depth information by synthesizing a plurality of distribution information estimated by the estimating units 101 (1) to 101 (N).
The second depth calculation unit 200 obtains second depth information indicating relative unevenness in the object region in the first image. The object area indicates an area where an object exists in the first screen.

  The distribution information indicates an absolute depth within the depth reproduction range of the entire screen, while the unevenness information indicates a relative depth relationship within the object. In other words, the depth range in which the distribution information is reproduced may be any as long as it is within the reproduction range, whereas the depth range in which the unevenness information is reproduced is narrower than the reproduction range.

The synthesizing unit 300 synthesizes the first depth information and the second depth information to obtain third depth information. Note that the synthesis unit 300 synthesizes the depth by a method different from that of the synthesis unit 103. Details will be described later.
The generation unit 400 generates a parallax image based on the third depth information and the first image. Any existing method may be used as the method for generating the parallax image from the depth.

FIG. 1B is a diagram illustrating a modification of the parallax image generation device according to the present embodiment. The second depth calculation unit 200 is different from the second depth calculation unit 201 in that it includes M (M ≧ 2) or more estimation units 201 (1) to 201 (M) and a synthesis unit 203 (see FIG. 1). It is different from the example of a).
The estimation unit 201 (m) detects a target object in the first image, and estimates unevenness information according to the type of the detected object for each detected object (1 ≦ m ≦ M). Further, the estimation units 201 (1) to 201 (M) may detect a common object by M types of methods and estimate unevenness information of the detected object by the same method. In addition, the estimation units 201 (1) to 201 (M) may detect a common object by a common method and estimate unevenness information of the detected object by M types of methods.

  The synthesizing unit 203 synthesizes the plurality of unevenness information estimated by the estimating units 201 (1) to 201 (M).

Next, the operation of the parallax image generation device according to the present embodiment will be described, but the example of FIG.
FIG. 2 is a diagram illustrating an operation of the parallax image generating device according to the present embodiment. A depth reproduction range in a three-dimensional space reproduced by a parallax image generated in each of the following steps is 0 ≦ z ≦ Z. Here, z represents depth information. z = 0 represents the foremost side (the depth is small) within the reproduction range, and z = Z represents the farthest side (the depth is large) within the reproduction range.

  The estimation units 101 (1) to 101 (N) are distributions indicating the distribution of the depth in the first image in the reproduction range in the three-dimensional space for each of the estimation units 101 (1) to 101 (N) by different methods. Information is estimated (S21 (1) to (N)). In S21 (n), the estimation unit 101 (n) estimates distribution information (1 ≦ n ≦ N). As a method of estimating the distribution information, a method of preparing a predetermined depth model and appropriately using them according to the first image may be used.

FIG. 3A is a diagram illustrating an example of a depth model of distribution information. The x axis and the y axis indicate positions in the image. Moreover, the area hatched in white indicates that the depth is on the far side, and that the depth is closer to the near side as it approaches black. Moreover, the method of detecting the motion between the 1st image and the 2nd image which should be displayed at the time different from a 1st image, and estimating distribution information based on the detected motion may be used. Also, a method of estimating distribution information according to a specific pixel value pattern may be used. The distribution information estimation method in S21 (n) may be any method as long as it can estimate the depth distribution information in the entire screen of the first image. Here, the depth zn at the pixel position (x, y) in the distribution information estimated by the nth (1 ≦ n ≦ N) S21 (n) is defined as follows. Note that 0 ≦ zn (x, y) ≦ Z.
The synthesizing unit 103 synthesizes a plurality of distribution information estimated by the estimating units 101 (1) to 101 (N) to obtain first depth information (S22). In the depth synthesis step, N pieces of depth information are synthesized to obtain one first depth information. In computer graphics, when combining two depths, the one close to the camera is selected and combined. This is based on the concept of ray tracing, and is based on the principle that the light entering the eye is not finally shielded. Here too, the depth on the near side is selected and synthesized in the same manner. The smaller the depth value, the closer to the front, the first depth information is obtained by selecting the minimum value as in the following equation.
zc (x, y) = min {z1 (x, y),..., zN (x, y)}
Here, zc (x, y) is the first depth information at the pixel position (x, y). Further, it means an operation of selecting a minimum value from min {a, b, c ...} and {a, b, c ...}. Although not preferable in terms of ray tracing, the first depth information may be obtained by averaging N pieces of depth information.

  The estimation units 201 (1) to (M) detect a target object in the first image, and estimate unevenness information according to the type of the detected object for each detected object (S23 (1) ) To S23 (M)). In S23 (m), the estimation unit 201 (m) estimates unevenness information (1 ≦ m ≦ M). The estimation unit 201 detects an object to be detected in the first image. The second depth information is calculated by assigning a relative unevenness model determined in advance according to the type of the detected object in the object region. In the present embodiment, a method of detecting the position and size of a face existing in the first image and assigning a predetermined relative depth model to the detected position will be exemplified. By doing so, it is possible to set appropriate unevenness in the object region.

  FIG. 3B is a diagram illustrating an example of a face and torso model. The x axis and the y axis indicate positions in the image. Moreover, the area hatched in white indicates that the depth is on the far side, and that the depth is closer to the near side as it approaches black. Further, not only the face but also objects may be detected and a depth model corresponding to them may be assigned. Note that the object to be detected may be, for example, a person, an animal, a car, etc., regardless of the face. In this embodiment, an example of detecting a vehicle other than a face will be described.

  Here, the unevenness information at the pixel position (x, y) estimated by S23 (m) is defined as follows. −Zr ≦ rm (x, y) ≦ Zr where rm (x, y) = 0 is a state in which there is no relative unevenness, and −Zr is a state in which the object region is relatively convex toward the near side, Zr Is relatively concave on the back side. Here, Zr can be determined based on the size of the detected object. For example, since the relative depth of a larger object looks larger, Zr is increased, and a smaller object is decreased. Alternatively, Zr may be determined based on the first depth information at the position of the object. When the first depth information indicates the near side, Zr is increased, and when the first depth information is the back, Zr is decreased.

  The synthesizing unit 203 synthesizes a plurality of unevenness information estimated by the estimating units 201 (1) to 201 (M) to obtain second depth information (S24). In computer graphics, when combining two depths, the one close to the camera is selected and combined. This is based on the concept of ray tracing, and is based on the principle that the light entering the eye is not finally shielded. Also in S24, the depth on the near side is selected and combined as in S22. The depth information can be achieved by selecting the minimum value as follows because a small value represents the near side. rc (x, y) = min {r1 (x, y),..., rM (x, y)} where rc (x, y) is the second depth information at the pixel position (x, y). . Although not correct in terms of ray tracing, an average of M unevenness information may be used.

  Further, the unevenness information may be synthesized based on the first depth information. If the first depth information at the centroid position of each object giving unevenness information is the centroid depth information, the centroid depth information is the foremost in the region where the overlap occurs between the objects (for example, a person and a car). You may make it select the uneven | corrugated information which becomes.

  The synthesizer 300 synthesizes the first depth information and the second depth information to obtain third depth information (S24). Note that the synthesis unit 300 synthesizes the depth by a method different from that of the synthesis unit 103 and the synthesis unit 203.

The first depth information and the second depth information are added to express the depth by adding relative unevenness in the object area to the first depth information indicating the absolute distribution of the depth in the first image. Thus, the third depth information is obtained.
zf (x, y) = zc (x, y) + rc (x, y)
Here, zf (x, y) is the third depth information at the pixel position (x, y). Further, multiplication may be used instead of addition. zf (x, y) = zc (x, y) · rc (x, y)
By combining the first depth information and the second depth information, the depth reproduction range 0 ≦ z ≦ Z in the three-dimensional space to be reproduced may be deviated. Therefore, it is preferable to adjust the third depth information so as to be within the reproduction range.
The generation unit 400 generates a parallax image based on the third depth information and the first image (S25). First, a disparity vector is calculated from the third depth information. Hereinafter, an example of a method for generating a parallax image will be described in detail.

  FIG. 4 is a diagram illustrating a method for calculating a disparity vector from depth information. A disparity vector can be calculated using the triangle connecting the right eye, the left eye, and the object, and the right disparity / left disparity / object on the screen and the similarity of the two triangles. Here, zf is the final depth information, d [cm] is the magnitude of the parallax vector, b [cm] is the interocular distance, zs [cm] is the distance to the screen, z0 [cm], and the depth in real space Distance Lz [cm].

Conversion from pixel size to real space [cm] is performed as follows.
dpixel = (screen resolution [pixel] / screen size [cm]) · d [cm]
The depth information zf is in the range of 0 to Z (may be 0 to 1), where 0 represents the near side and Z represents the back. However, this value is virtual and needs to be converted into an actual distance. The depth information is converted into the real space distance using the real space distance Lz. Conversion from depth to real space is performed using the following conversion formula. γ = Lz / zmax [cm] where zmax = Z. Then, the distance z ′ from the screen to the object is as follows. z ′ = γzf−z0
The following relationship holds from the similarity of triangles.
d: b = (z ′): (zs + z ′) Formula (1)
d (zs + z ′) = bz ′ (2)
z = {(b−d) z0 + dzs} / {γ (b−d)} (3)

  The depth conversion model is an inversely proportional model to the disparity vector. However, for example, a model of a function that approximates an inversely proportional relationship by a partial proportional relationship may be used.

Stereoscopic parameters b, zs, z0, and Lz can be arbitrarily determined based on the stereoscopic vision desired to be provided. For example, zs is determined according to the actual screen position, and z0 is increased when it is desired to increase the pop-out. The depth can be determined by LZ.
If the stereoscopic parameter is determined, the disparity vector can be calculated from the depth information according to the following depth disparity vector conversion model obtained by modifying Expressions (1) to (3). The unit of d is [cm]
d: b = (z ′): (zs + z ′)
d = b {z ′ / (zs + z ′)}

FIG. 5 is a diagram illustrating a method of generating a parallax image from the calculated parallax vector. If the first image is obtained from an intermediate viewpoint between the left eye and the right eye, the left parallax image and the right parallax image can be generated from parallax vectors dL and dR obtained by multiplying the parallax vector d by -1/2 and 1/2.
dL = (− 1/2) d, dR = (1/2) d

  The left parallax image can be generated by moving the pixel value It (x, y) of the first image according to dL. The right parallax image can be generated by moving the pixel value It (x, y) of the first image according to dR. A region that is not assigned a pixel value (hereinafter referred to as a hole) may occur by simply moving. Therefore, peripheral pixel values may be assigned to the holes. Although an example of generating two parallax images has been described here, the same processing may be performed for multi-parallax images.

Next, the third depth information by the parallax image generating device of the present embodiment will be described in detail.
For example, consider a case where an image in which a person is present in a room is the first image. In that case, for example, a plurality of distribution information is estimated by applying a model of the depth of the room, obtaining a motion vector, and the like, and these are combined to obtain the first depth information. Further, a person in the first image is detected to estimate facial unevenness information (second depth information). It is necessary to synthesize the estimated first depth information and second depth information. However, the reproduction range of the uneven information is significantly narrower than the depth distribution information of the entire screen. Therefore, if the first depth information and the second depth information are synthesized by a method similar to the synthesis method used when obtaining the first depth information and / or the second depth information, a stereoscopic effect due to the depth distribution in the screen is obtained. Unevenness of the object cannot be expressed while holding it.

6 to 8 below, the horizontal axis indicates the pixel position, and the vertical axis indicates the depth information.
FIG. 6 is a diagram illustrating an example of depth information. FIG. 6A shows a test pattern for depth information. In FIG. 6, 0 indicates the farthest side, and 10 indicates the foremost side. The depth information described above reverses the magnitude relationship between the back side and the near side. Distribution information 1 and distribution information 2 show examples of distribution information obtained by the estimation unit 101 (n), respectively. The unevenness information 1 and the unevenness information 2 indicate examples of unevenness information obtained by the estimation unit 201 (m).

  FIG. 6B shows an average of the distribution information 1, distribution information 2, unevenness information 1, and unevenness information 2 of the test pattern of FIG. This shows the result of selecting the maximum value by position and compositing. When the average is taken, the dynamic range of the depth indicated by the distribution information is compressed, and the unevenness information is buried in the whole. On the other hand, when the maximum value is taken, the dynamic range of the depth of the entire room is maintained, but the depth of the face is completely buried. This is because the absolute depth distribution information in the entire screen and the uneven depth information in the object are combined together. “Absolute” means an absolute depth with respect to the depth reproduction range 0 ≦ z ≦ Z. “Relative” means a relative depth in a local object area on the screen. The reproduction range of relative depth information is different from the reproduction range of absolute depth information. Therefore, if the absolute depth information and the relative depth information are combined by the same method, the combination fails because the meanings of the mutual values are different.

  FIG. 7A is a diagram illustrating first depth information obtained by combining the distribution information 1 and the distribution information 2 from the depth information test pattern illustrated in FIG. FIG. 7B is a diagram illustrating an example of the second depth information synthesized from the unevenness information 1 and the unevenness information 2. The combining method indicates that the first depth information and the second depth information are combined by selecting depth information having the closest depth at the same position.

FIG. 8 shows third depth information obtained by combining the first depth information and the second depth information shown in FIG. The third depth information was synthesized by adding the first depth information and the second depth information.
Screens obtained by a plurality of methods by using different methods for the synthesis method for obtaining the first depth information and the second depth information and the synthesis method for synthesizing the first depth information and the second depth information. It is possible to generate a parallax image that expresses the unevenness of the object while maintaining the stereoscopic effect due to the depth distribution.

(Second Embodiment)
FIG. 9 is a diagram illustrating the parallax image generating device of the present embodiment. The parallax image generation device of this embodiment is different from the parallax image generation device of FIG. 1B in that it has a function of adjusting depth information.

  The adjustment unit 102 (n) adjusts the distribution information estimated by the estimation unit 101 (n). For example, the estimation unit 101 (1) estimates the distribution information in the range of 0 ≦ z ≦ Z, but the estimation unit 101 (2) outputs the value in the range of 100 ≦ z ≦ Z. If the depth information ranges are different, they need to be adjusted to the same range. Therefore, the adjustment unit 102 (n) adjusts, for example, by fixing one point of depth information (hereinafter referred to as a fixed point) (for example, Z) to widen or narrow the depth range. In addition, when the reliability of a certain estimation unit 101 (n) is low, it is possible to reduce the influence of depth information with low reliability by adjusting the depth range to the back side.

  The adjustment unit 202 (m) adjusts the second depth information estimated by the estimation unit 201 (m). The method of adjustment may be the same method as that of the adjustment unit 102 (n). Adjustment can be performed in the same manner as the depth information adjustment step. In addition, the characteristic of relative uneven | corrugated information can be effectively utilized by making a fixed point into 0.

First depth calculation unit 100, estimation unit 101 (n), adjustment unit 102 (n), adjustment unit 202 (m), second depth calculation unit 200, synthesis unit 300, and generation unit 400

Claims (21)

Estimated by a first estimation unit that estimates distribution information according to a specific pixel value pattern, a second estimation unit that estimates distribution information based on the motion of the first image, and the first and second estimation units. A first depth calculating unit comprising: a first combining unit that combines the distribution information to generate first depth information;
A second depth calculation unit that generates second depth information indicating relative unevenness of each object in the first image;
A third combining unit that combines the first depth information and the second depth information by a method different from the first combining unit to generate third depth information;
An image processing apparatus comprising:   The image processing apparatus according to claim 1, wherein the second depth calculation unit detects a face as an object and generates second depth information indicating a relative unevenness of the detected face.   The second depth calculation unit detects the position of the face, assigns a relative depth model determined in advance according to the detected position, and generates the second depth generation unit. The image processing apparatus according to claim 2.   The second depth calculation unit detects at least one of a person, an animal, and a car as an object, and generates second depth information indicating a relative unevenness of the detected object. Item 8. The image processing apparatus according to Item 1.   The second estimation unit detects a motion between the first image and a second image to be displayed at a time different from the first image, and estimates distribution information based on the detected motion. The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus.   The image processing apparatus according to claim 1, further comprising: a generation unit that generates a parallax image based on the third depth information and the first image.   The image processing apparatus according to claim 6, wherein the generation unit generates the parallax images for left eye and right eye.   The image processing apparatus according to claim 6, wherein the distribution information indicates an absolute depth within a depth reproduction range in an entire screen on which the parallax image is displayed.   The first combining unit selects the distribution information having a depth closer to the same position in the first image from the distribution information estimated by the first and second estimation units. Item 9. The image processing apparatus according to any one of Items 1 to 8. The first depth calculator is
A first adjustment unit that adjusts the distribution information estimated by the first estimation unit;
The image processing apparatus according to claim 1, further comprising: a second adjustment unit that adjusts the distribution information estimated by the second estimation unit.   The image processing apparatus according to claim 1, wherein the third synthesis unit generates the third depth information by adding the first depth information and the second depth information. . An image display device that generates at least one parallax image having parallax with each other using a first image and displays the generated parallax image,
Estimated by a first estimation unit that estimates distribution information according to a specific pixel value pattern, a second estimation unit that estimates distribution information based on the motion of the first image, and the first and second estimation units A first synthesis unit that synthesizes the distributed information to generate first depth information;
A second depth calculation unit that generates second depth information indicating relative unevenness of each object in the first image;
A third combining unit that combines the first depth information and the second depth information by a method different from the first combining unit to generate third depth information;
A generating unit configured to generate the parallax image based on the third depth information and the first image;
An image display device comprising: a display unit that displays the generated parallax image.   The image display device according to claim 12, wherein the second depth calculation unit detects a face as an object and generates second depth information indicating a relative unevenness of the detected face.   The second depth calculation unit detects the position of the face, assigns a relative depth model determined in advance according to the detected position, and generates the second depth generation unit. The image display device according to claim 13.   The second depth calculation unit detects at least one of a person, an animal, and a car as an object, and generates second depth information indicating a relative unevenness of the detected object. Item 13. The image display device according to Item 12.   The second estimation unit detects a motion between the first image and a second image to be displayed at a time different from the first image, and estimates distribution information based on the detected motion. The image display device according to claim 12, wherein the image display device is an image display device.   The image display device according to any one of claims 12 to 16, wherein the distribution information indicates an absolute depth within a depth reproduction range in the entire screen on which the parallax image is displayed.   The first combining unit selects the distribution information having a depth closer to the same position in the first image from the distribution information estimated by the first and second estimation units. Item 18. The image display device according to any one of Items 12 to 17. The first depth calculator is
A first adjustment unit that adjusts the distribution information estimated by the first estimation unit;
The image display device according to claim 12, further comprising: a second adjustment unit that adjusts the distribution information estimated by the second estimation unit.   The image display according to claim 12, wherein the third combining unit generates the third depth information by adding the first depth information and the second depth information. apparatus. Estimating distribution information according to a specific pixel value pattern;
Estimating distribution information based on movement of the first image;
Combining the estimated distribution information to generate first depth information;
Generating second depth information indicating relative irregularities for each object in the first image;
An image processing method comprising: combining the first depth information and the second depth information by a method different from the first combining unit to generate third depth information.