JP2014021744A - Image processing device, image processing method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To enable reduction of an information amount of information used for generating a depth value representing the position of an object in a depth direction in an image.SOLUTION: On the basis of a quadratic curve representing a model of a depth value representing the position in a depth direction of an object such as an in-face area, an out-of-face area, a neck area, an in-body area, and an out-of-body area, a depth generation unit generates a depth value of the object such as an in-face area, an out-of-face area, a neck area, an in-body area, and an out-of-body area in an image. The present technique can be applied to, for example, an image processing device for generating a depth map of an image.

Description

  The present technology relates to an image processing device, an image processing method, and a program, and in particular, an image that can reduce an information amount of information used for generating a depth value representing a position in a depth direction of an object in an image. The present invention relates to a processing device, an image processing method, and a program.

  When generating a three-dimensional (3D) image from a two-dimensional (2D) image, the image processing apparatus uses, for example, a depth map including a depth value representing a position in the depth direction of each pixel of the two-dimensional image. Specifically, the image processing apparatus shifts each pixel of the two-dimensional image in the left-right direction based on the depth value of each pixel constituting the depth map, and the left-eye image and the right-eye image in the three-dimensional image. Is generated. As a result, when a viewer who views a 3D display or the like that displays a three-dimensional image views the left-eye image with the left eye and the right-eye image with the right eye, parallax occurs, and the viewer feels the depth.

  By the way, various methods have been devised as a method for generating the depth value of an object in an image. For example, a method of generating a depth value of a person's face in an image using three-dimensional shape information of four or more feature points that do not exist on the same plane of a general human face has been devised ( For example, see Patent Document 1).

JP 2007-4767

  However, the information amount of the three-dimensional shape information of each feature point is relatively large, and a relatively large memory is required to hold the three-dimensional shape information.

  The present technology has been made in view of such a situation, and makes it possible to reduce the amount of information used for generating the depth value of an object in an image.

  An image processing apparatus according to one aspect of the present technology includes a generation unit that generates a depth value of the object in an image based on a quadratic curve representing a model of a depth value representing a position of the object in the depth direction. It is.

  An image processing method and a program according to an aspect of the present technology correspond to an image processing apparatus according to an aspect of the present technology.

  In one aspect of the present technology, the depth value of the object in the image is generated based on a quadratic curve representing a model of the depth value representing the position of the object in the depth direction.

  According to an aspect of the present technology, it is possible to reduce the amount of information used for generating the depth value of an object in an image.

It is a block diagram which shows the structural example of one Embodiment of the image processing apparatus to which this technique is applied. It is a figure which shows the example of site | part information. It is a block diagram which shows the structural example of the depth production | generation part of FIG. It is a figure which shows the example of a horizontal quadratic curve. It is a figure explaining the change of the depth value accompanying zoom-in imaging. It is a figure which shows the relationship between the size of a face image, and the range of the horizontal quadratic curve after correction | amendment. It is a figure explaining the production | generation of a depth value in case all the angles of a face image are 0 degree | times. It is a figure explaining the production | generation of a depth value in case all the angles of a face image are 0 degree | times. It is a figure which shows the example of the depth value of each pixel of a person area | region when all the angles of a face image are 0 degree | times. It is a figure explaining the production | generation of a depth value in case the yaw angle and pitch angle of a face image are other than zero. It is a flowchart explaining the depth map production | generation process of the image processing apparatus of FIG. It is a flowchart explaining the detail of the depth value production | generation process of FIG. It is a flowchart explaining the other example of a depth map production | generation process. It is a block diagram which shows the structural example of the hardware of a computer.

<One embodiment>
[Configuration Example of Embodiment of Image Processing Apparatus]
FIG. 1 is a block diagram illustrating a configuration example of an embodiment of an image processing apparatus to which the present technology is applied.

  1 includes a face detection unit 11, an in-face region determination unit 12, an out-of-face region determination unit 13, a neck region determination unit 14, a torso region determination unit 15, an torso region determination unit 16, and a boundary determination. The unit 17, the depth generation unit 18, the selection unit 19, and the smoothing processing unit 20 are configured. The image processing apparatus 10 generates a depth map of an image input as an input image from the outside.

  Specifically, the face detection unit 11 of the image processing apparatus 10 detects a face image in an input image input from the outside. The face detection unit 11 converts the face image information representing the position and size of the face image into an in-face region determination unit 12, an out-of-face region determination unit 13, a neck region determination unit 14, a torso region determination unit 15, and an torso region. It supplies to the determination part 16.

  The face detection unit 11 detects the pitch angle and yaw angle of the detected face image as the face image angle. Note that the pitch angle of the face image is an angle that is rotated about an axis that is parallel to the line connecting the left and right eyes of the person and passes through the center of the person's head. The yaw angle is an angle that is perpendicular to the axis and rotates about an axis that passes substantially through the center of the head. Details of the pitch angle and yaw angle detection method of the face image are described in, for example, Japanese Patent Application Laid-Open No. 2010-3021. The face detection unit 11 supplies the detected angle of the face image and the size of the face image to the depth generation unit 18.

  Based on the face image information supplied from the face detection unit 11, the in-face region determination unit 12 sets an ellipse region having a size based on the size of the face image as the in-face region centered on the center of the face image. For example, the intra-facial region determination unit 12 sets an elliptical region whose short axis is one time the horizontal length of the face image and whose long axis is 1.5 times the vertical length of the face image as the intra-face region. Set. The intra-facial region determination unit 12 supplies intra-facial region information indicating the position and size of the intra-facial region to the boundary determination unit 17.

  The out-of-face area determination unit 13 is based on the face image information supplied from the face detection unit 11 and has a size based on the size of the face image centered on the center of the face image and is larger than the in-face area. Is set to the area outside the face. The out-of-face area determination unit 13 supplies out-of-face area information indicating the position and size of the out-of-face area to the boundary determination unit 17.

  Based on the face image information supplied from the face detection unit 11, the neck region determination unit 14 sets a rectangular region having a size based on the size of the face image below the face image as the neck region. The neck region determination unit 14 supplies neck region information representing the position and size of the neck region to the boundary determination unit 17.

  Based on the face image information supplied from the face detection unit 11, the torso region determination unit 15 has a trapezoidal region below the neck region, a rectangular region below the trapezoidal region, and a size based on the size of the face image. To the body region.

  For example, the torso region determination unit 15 has a trapezoidal region in which the upper side and the lower side below the neck region are proportional to the horizontal length of the face image, and the height is proportional to the vertical length of the face image. And a rectangular area under the trapezoidal area, the horizontal length of which is proportional to the horizontal length of the face image, and the vertical length of which is proportional to the vertical length of the face image, Set to the inner area. The body region determination unit 15 supplies body region information indicating the position and size of the body region to the boundary determination unit 17.

  The torso-outside region determination unit 16 is based on the face image information supplied from the face detection unit 11 and is a size based on the size of the face image, which is larger than the torso region below the neck region, The rectangular area below the trapezoidal area is set as the body outside area. The torso outside region determination unit 16 supplies the torso outside region information indicating the position and size of the torso outside region to the boundary determination unit 17.

  The boundary determination unit 17 includes the in-face region information from the in-face region determination unit 12, the out-of-face region information from the out-of-face region determination unit 13, the neck region information from the neck region determination unit 14, and the in-body region determination unit 15. A region that is at least one of an in-face region, an out-of-face region, a neck region, a torso region, or an out-of-torso region based on the torso region information and the torso region information from the torso region determining unit 16 Let it be a person area.

  Further, the boundary determination unit 17 determines each pixel in the person area as a final in-face area based on the in-face area information, the out-of-face area information, the neck area information, the in-body area information, and the out-to-body area information. The pixel is determined as any one of the pixels in the region outside the face, the neck region, the body region, or the region outside the body. That is, the boundary determination unit 17 functions as a setting unit and divides and sets a person area into a plurality of areas.

  In the following, when it is not necessary to particularly distinguish each of the in-face region, the out-of-face region, the neck region, the torso region, and the torso region, these are collectively referred to as a part region. Similarly, in-face area information, out-of-face area information, neck area information, torso area information, and torso area information are collectively referred to as part area information.

  The boundary determination unit 17 supplies part information representing the position and size of the final part region for each part region to the depth generation unit 18. In addition, the boundary determination unit 17 generates a person area flag indicating that it is a person area for each pixel in the person area, and is not a person area for each pixel outside the person area in the input image. A person region flag representing this is generated. The boundary determination unit 17 supplies the person region flag of each pixel of the input image to the selection unit 19.

  The depth generation unit 18 stores a quadratic curve representing a model of the depth value of the object in each region. Based on the size and angle of the face image supplied from the face detection unit 11, the stored quadratic curve, and the part information supplied from the boundary determination unit 17, the depth generation unit 18 Generate pixel depth values. The depth generation unit 18 supplies the generated depth value to the selection unit 19.

  For each pixel of the input image, the selection unit 19 determines the depth value supplied from the depth generation unit 18 based on the person area flag supplied from the boundary determination unit 17 or the depth for the background image input from the outside. Select a value. The selection unit 19 generates a depth map by assigning the depth value of each selected pixel to the pixel value of the depth map of the pixel. Then, the selection unit 19 supplies the generated depth map to the smoothing processing unit 20.

  The smoothing processing unit 20 smoothes the depth map supplied from the selection unit 19 using a general spatial filter. Thereby, the big change of the depth value which generate | occur | produces at the boundary of each site | part area | region can be suppressed.

[Example of part information]
FIG. 2 is a diagram illustrating an example of the part information generated by the boundary determination unit 17 in FIG.

  In the example of FIG. 2, a person area 41 exists in the center of the image 40. The boundary determination unit 17 first determines, based on the intra-facial region information, a pixel in the intra-facial region represented by the intra-facial region information among pixels in the person region 41 as a final intra-facial region 42 pixel. To do.

  Next, based on the out-of-face region information, the boundary determination unit 17 determines pixels other than the pixels in the in-face region 42 among the pixels in the out-of-face region represented by the out-of-face region information. To decide. Then, based on the neck area information, the boundary determination unit 17 determines a pixel other than the in-face area 42 and the out-of-face area 43 among the pixels in the neck area represented by the neck area information as the final neck area 44. To do.

  Next, the boundary determination unit 17 selects pixels other than the in-face region 42, the out-of-face region 43, and the neck region 44 among the pixels in the torso region represented by the torso region information based on the torso region information. Is determined as the final body region 45. Finally, the boundary determination unit 17 determines the in-face region 42, the out-of-face region 43, the neck region 44, and the torso among the pixels in the out-of-torso region represented by the out-of-torso region information based on the out-of-torso region information. Pixels other than the region 45 are determined as the final torso region 46.

[Configuration example of depth generator]
FIG. 3 is a block diagram illustrating a configuration example of the depth generation unit 18 of FIG.

  The depth generation unit 18 in FIG. 3 includes a storage unit 61, a size correction unit 62, an angle correction unit 63, and a generation unit 64.

  The storage unit 61 of the depth generation unit 18 stores a quadratic curve of each part region. Specifically, the storage unit 61 includes a horizontal quadratic curve that is a quadratic curve representing a model of the relationship between the depth value of the object in each part region and the horizontal position in the part region, and the object in each part region. A vertical quadratic curve that is a quadratic curve representing a model of the relationship between the depth value and the position in the vertical direction in the region is stored.

  The size correction unit 62 reads out the horizontal quadratic curve and the vertical quadratic curve of each region from the storage unit 61, and based on the size of the face image supplied from the face detection unit 11, the horizontal quadratic curve and the vertical 2 Correct the quadratic curve. The size correction unit 62 supplies the corrected horizontal quadratic curve and vertical quadratic curve to the angle correction unit 63.

  The angle correction unit 63 corrects the horizontal quadratic curve and the vertical quadratic curve of the in-face region and the out-of-face region supplied from the size correction unit 62 based on the angle of the face image supplied from the face detection unit 11. . The angle correction unit 63 supplies the corrected horizontal quadratic curve and vertical quadratic curve of the in-face region and the out-of-face region to the generation unit 64. In addition, the angle correction unit 63 supplies the horizontal quadratic curve and the vertical quadratic curve of the neck region, the body region, and the body outside region supplied from the size correction unit 62 to the generation unit 64 as they are.

  The generation unit 64 sequentially sets each pixel in the person area as a processing target pixel. The generation unit 64 determines the region of the pixel to be processed based on the region information supplied from the boundary determination unit 17. The generating unit 64 is based on the horizontal quadratic curve and the vertical quadratic curve of the region of the pixel to be processed among the horizontal quadratic curve and the vertical quadratic curve supplied from the angle correction unit 63. Depth values corresponding to the horizontal and vertical positions of the pixel are generated. The generation unit 64 supplies the generated depth value of each pixel in the person area to the selection unit 19 in FIG.

[Example of horizontal quadratic curve]
FIG. 4 is a diagram illustrating an example of a horizontal quadratic curve stored in the storage unit 61 of FIG.

In FIG. 4, the horizontal axis represents the position j in the horizontal direction when the center coordinate in the horizontal direction of the part region is 0. The vertical axis represents the depth value d _h of the object in the part region at position j. In this specification, the depth value is a value when the depth value for the background image is set to 0, and is assumed to be larger as representing the near side position and smaller as representing the far side position.

As shown in FIG. 4, the horizontal quadratic curve represents a model of the relationship between the depth value d _h of the object in the part region and the horizontal position j in the part region. Specifically, the horizontal quadratic curve is defined by the following equation (1).

As shown in FIG. 4, ofst in equation (1) is the distance from the depth value for the background image to the depth value d _h represented by the horizontal quadratic curve, and jvol is the depth value d represented by the horizontal quadratic curve. The range of _h . Moreover, jw is a half value of the range of the position j in the horizontal direction effective in the horizontal quadratic curve.

  Although only the horizontal quadratic curve is described in FIG. 4, the vertical quadratic curve is horizontal except that the position j replaces the vertical position i when the central coordinate in the vertical direction of the region is 0. Since it is the same as a quadratic curve, description is abbreviate | omitted.

Here, the distance from the depth value of the background image to the depth value d _v representing the vertical quadratic curve is the same as the distance of the horizontal quadratic curve ofst. In the following, the range of depth values d _v representing the vertical quadratic curve (range) is referred to as range IVOL, the half value of the range of positions i of the effective vertical in the vertical quadratic curve, that the value iw .

[Description of correction based on size of face image of horizontal quadratic curve and vertical quadratic curve]
FIG. 5 is a diagram for explaining a change in depth value accompanying zoom-in photographing.

  As shown in FIG. 5, the range of the depth value corresponding to the horizontal position of the person area 81A in the normally photographed image 81 is the horizontal position of the person area 82A in the image 82 photographed at a predetermined zoom magnification. It is larger than the range of depth values corresponding to. As described above, when the size of the person area increases due to zoom-in shooting or shooting from a relatively close distance, the range of depth values corresponding to the position in the horizontal direction increases. On the other hand, although not shown, when the size of the person area is reduced by zoom-out shooting or shooting from a relatively long distance, the range of depth values corresponding to the position in the horizontal direction is reduced.

  Although not shown, the depth value range corresponding to the position in the vertical direction also changes according to the size of the person area in the same manner as the depth value range corresponding to the position in the horizontal direction. Accordingly, the size correction unit 62 corrects the range jvol of the horizontal quadratic curve and the range ivol of the vertical quadratic curve based on the size of the face image corresponding to the size of the person area.

  FIG. 6 is a diagram showing the relationship between the size of the face image and the range jvol of the horizontal quadratic curve corrected by the size correction unit 62.

  In FIG. 6, the horizontal axis represents the horizontal length of the face image, and the vertical axis represents the horizontal quadratic curve range jvol corrected by the size correction unit 62.

  As shown in FIG. 6, the range jvol of the horizontal quadratic curve corrected by the size correction unit 62 is proportional to the horizontal length of the face image. That is, the size correction unit 62 corrects the range jvol based on the horizontal length of the face image so that the range jvol is proportional to the horizontal length of the face image. Although not shown, the size correction unit 62 corrects the range ivol to be proportional to the vertical length of the face image based on the vertical length of the face image, similarly to the range jvol. .

  Note that the horizontal quadratic curve range jvol and the vertical quadratic curve range ivol stored in the storage unit 61 may be default values, for example.

  As described above, the size correction unit 62 corrects the range jvol and the range ivol to be larger as the size of the face image is larger, based on the size of the face image. Therefore, the image processing apparatus 10 can generate a depth map that reflects the difference in depth value between a short-distance person and a long-distance person that humans feel in the real world. As a result, a more natural 3D image can be generated using this depth map.

  In addition, the size correction unit 62 can easily generate a more natural depth map only by correcting the range jvol and the range ivol based on the size of the face image. On the other hand, when generating a depth map based on three-dimensional shape information of a general human face as in the prior art, the face corresponding to the three-dimensional shape information is enlarged or reduced to obtain three-dimensional shape information. It is necessary to match the face size corresponding to the face size in the input image with high accuracy. Therefore, in the conventional case, complicated processing is required to generate a more natural depth map.

  In the example of FIG. 6, the size correction unit 62 linearly converts the range jvol and the range ivol based on the size of the face image, but may perform nonlinear conversion according to human visual characteristics.

[Description of depth value generation]
7 and 8 are diagrams for explaining generation of depth values when the angles of the face images are all 0 degrees.

  As shown in FIGS. 7 and 8, when the angles of the face images are all 0 degrees, the angle correction unit 63 generates the horizontal quadratic curve and the vertical quadratic curve supplied from the size correction unit 62 as they are. To supply.

  As illustrated in FIG. 7, the generation unit 64 generates a horizontal quadratic curve and a vertical quadratic curve supplied from the angle correction unit 63 when the region of the pixel to be processed is the in-face region 42. Of these, the horizontal quadratic curve 101 and the vertical quadratic curve 102 of the in-face region 42 are selected. Then, the generating unit 64 generates the depth value of the pixel to be processed using the selected horizontal quadratic curve 101 and vertical quadratic curve 102.

  Specifically, when the horizontal quadratic curve 101 of the in-face region 42 is defined by the following equation (2) and the vertical quadratic curve 102 is defined by the following equation (3), the generation unit 64: The depth value of the pixel to be processed is generated by the equation (4).

In equations (2) to (4), ofst ₁ is a distance ofst common to the horizontal quadratic curve 101 and the vertical quadratic curve 102. In equation (2), d _h1 is the horizontal quadratic curve 101. Depth value represented by. In the equations (2) and (4), jvol ₁ is the range jvol of the horizontal quadratic curve 101, and jw ₁ is the value jw of the horizontal quadratic curve 101.

In Expression (3), d _V1 is a depth value represented by the vertical quadratic curve 102, and in Expression (4), d ₁ is a depth value of the in-face region 42. Further, in Expression (3) and Expression (4), ivol ₁ is the range ivol of the vertical quadratic curve 102, and iw ₁ is the value iw of the vertical quadratic curve 102.

  As shown in FIG. 7, when the region of the pixel to be processed is the out-of-face region 43, the generation unit 64 includes the horizontal quadratic curve and the vertical quadratic curve supplied from the angle correction unit 63. Then, the horizontal quadratic curve 103 and the vertical quadratic curve 104 of the out-of-face region 42 are selected. Then, the generation unit 64 generates the depth value of the pixel to be processed using the selected horizontal quadratic curve 103 and vertical quadratic curve 104.

  Specifically, when the horizontal quadratic curve 103 of the out-of-face region 42 is defined by the following equation (5) and the vertical quadratic curve 104 is defined by the following equation (6), the generation unit 64: The depth value of the pixel to be processed is generated by the equation (7).

In Expressions (5) to (7), ofst ₂ is a distance ofst common to the horizontal quadratic curve 103 and the vertical quadratic curve 104. In Expression (5), d _h2 is the horizontal quadratic curve 103. Depth value represented by. In the equations (5) and (7), jvol ₂ is the range jvol of the horizontal quadratic curve 103, and jw ₂ is the value jw of the horizontal quadratic curve 103.

In Expression (6), d _V2 is a depth value represented by the vertical quadratic curve 104, and in Expression (7), d ₂ is a depth value of the out-of-face region 42. Further, in Expression (6) and Expression (7), ivol ₂ is the range ivol of the vertical quadratic curve 104, and iw ₂ is the value iw of the vertical quadratic curve 104.

In the example of FIG. 7, since the range jvol ₂ is set to be larger than the range jvol ₁ , a depth value closer to the actual human face depth value can be generated.

That is, the generation unit 64 generates the depth value of the in-face region 42 using the horizontal quadratic curve 101 defined by the range jvol ₁ , and sets the depth value of the out-of-face region 43 to a range jvol that is larger than the range jvol _1. It generated using a horizontal quadratic curve 103 defined by _2. Accordingly, the curvature corresponding to the depth value of the out-of-face region 43 is larger than the curvature corresponding to the depth value of the in-face region 42. Here, generally, when the face of a person is viewed from the front, the curvature of the side of the face corresponding to the out-of-face region 43 is stronger than the front of the face corresponding to the in-face region 42. Therefore, it can be said that the generation unit 64 can generate a depth value closer to the depth value of the actual person's face.

In addition, since the range jvol ₂ is set to be larger than the range jvol ₁ , the difference between the depth values of the area outside the person area and the outside face area 43 becomes large. As a result, in the 3D image generated using the depth map, the viewer can easily separate the person image and the background image by the position in the depth direction. That is, a 3D image having a large stereoscopic effect can be generated using the depth map.

  As shown in FIG. 8, when the region of the pixel to be processed is the body region 45, the generation unit 64 includes the horizontal quadratic curve and the vertical quadratic curve supplied from the angle correction unit 63. The horizontal quadratic curve 121 and the vertical quadratic curve 122 of the body region 45 are selected. Then, the generation unit 64 generates the depth value of the pixel to be processed using the selected horizontal quadratic curve 121 and vertical quadratic curve 122 as in the case of the in-face region 42 described with reference to FIG. .

  Furthermore, as illustrated in FIG. 8, when the region of the pixel to be processed is the torso-outside region 46, the generation unit 64 includes the horizontal quadratic curve and the vertical quadratic curve supplied from the angle correction unit 63. The horizontal quadratic curve 123 and the vertical quadratic curve 124 of the outer body region 46 are selected. Then, the generation unit 64 generates the depth value of the pixel to be processed using the selected horizontal quadratic curve 123 and vertical quadratic curve 124 as in the case of the out-of-face region 43 described with reference to FIG. .

  Although illustration is omitted, when the part region of the pixel to be processed is the neck region 44, the generation unit 64 is similar to the case of other part regions, and the horizontal quadratic curve and the vertical 2 of the neck region 44 are the same. The depth value of the pixel to be processed is generated using the next curve. In the horizontal quadratic curve and the vertical quadratic curve of the neck region 44, the depth value of the neck region 44 is smaller than the depth value of the out-of-face region 43. Thereby, in the 3D image generated by using the depth map, the chin and the neck can be separated, and a 3D image approximated to an actual person can be generated.

  FIG. 9 is a diagram illustrating an example of the depth value of each pixel of the person area generated as described above.

  In FIG. 9, the x-axis represents the horizontal position of each pixel in the input image, the y-axis represents the vertical position of each pixel in the input image, and the z-axis represents the depth value of each pixel. In FIG. 9, the depth value of the area outside the person area is also shown as the depth value 0 for the background image.

Next, FIG. 10 illustrates generation of pixel depth values in the in-face region 42 and the out-of-face region 43 when the yaw angle of the face image is θ _j other than 0 and the pitch angle is θ _i other than 0. It is a figure explaining.

As shown in FIG. 10, when the yaw angle of the face image is θ _j other than 0, the angle correction unit 63 uses the yaw angle θ _j as shown in the following equation (8) to shift amount shift _j. Ask for. In this specification, it is assumed that the yaw angle θ _j is a positive value when the face image is facing left, and the yaw angle θ _j is a negative value when the face image is facing right.

Then, as shown in FIG. 10, the angle correction unit 63 converts the horizontal quadratic curve 101 (FIG. 7) into a horizontal quadratic curve 141 defined by the following equation (9) based on the shift amount shift _j . to correct.

In Expression (9), d _h1 ′ is a depth value represented by the horizontal quadratic curve 141.

Here, the horizontal quadratic curve 143 that is the corrected horizontal quadratic curve 103 is expressed by the following equation ( ₂ ) using the depth value d _h2 ′ represented by the horizontal quadratic curve 143 and the distance ofst ₂ of the horizontal quadratic curve 103: 10).

d _h2 '-ofst ₂ = a (jb)
(10)

At this time, two intersections (Hb, d _Hb ) and intersections (Hc, d _Hc ) of the horizontal quadratic curve 141 and the horizontal quadratic curve 143 which is the corrected horizontal quadratic curve 103 are expressed by the following equation (11). It is represented by

  In Expression (11), −Hb, that is, Hc, is half the length of the short axis of the in-face region 42.

  By solving the simultaneous equations of Expression (11), a and b are expressed by the following Expression (12).

Here, since the intersection (Hb, d _Hb ) and the intersection (Hc, d _Hc ) are also points on the horizontal quadratic curve 141, d _Hb and d _Hc can be obtained by the above-described equation (9). . Accordingly, the angle correction unit 63 _obtains d _Hb and d _Hc by the above-described equation (9), and corrects the horizontal quadratic curve 103 to a horizontal quadratic curve 143 defined by the following equation (13).

As shown in FIG. 10, when the pitch angle of the face image is θ _i other than 0, the angle correction unit 63 uses the value iw ₁ , the pitch angle θ _i , and the range ivol ₁ to shift the shift amount shift. _The shift amount shift _i is obtained in the same manner as _j . In this specification, when the face image is upward, the pitch angle θi is a positive value, when the face image is downward, it is assumed the pitch angle theta _i is a negative value.

As shown in FIG. 10, the angle correction unit 63 corrects the vertical quadratic curve 102 (FIG. 7) to the vertical quadratic curve 142 in the same manner as the horizontal quadratic curve 101 based on the shift amount shift _i . Further, the angle correction unit 63 corrects the vertical quadratic curve 104 to the vertical quadratic curve 144 in the same manner as the horizontal quadratic curve 143.

When the pitch angle and the yaw angle are other than 0, the generation unit 64 uses the following equation (14) based on the horizontal quadratic curve 141 and the vertical quadratic curve 142 to calculate the pixel depth of the in-face region 42. Generate the value d ₁ '.

Further, the generation unit 64 generates the depth value d ₂ ′ of the pixel in the out-of-face region 43 based on the horizontal quadratic curve 143 and the vertical quadratic curve 144 by the following equation (15).

In Expression (15), d _Vb and d _Vc are depth values corresponding to the vertical positions Vb and Vc of the two intersections of the vertical quadratic curve 142 and the vertical quadratic curve 144. The position −Vb is the same as the position Vc, and is half the length of the long axis of the in-face region 42.

As described above, the angle correction unit 63 is based on the yaw angle θ _j and the pitch angle θ _i of the face image, and the in-face region when the yaw angle and the pitch angle are predetermined values (0 in the present embodiment). 42 and the horizontal quadratic curve and the vertical quadratic curve of the region 43 outside the face are corrected. Therefore, the depth generation unit 18 uses the horizontal quadratic curve and the vertical quadratic curve of the in-face region 42 and the out-of-face region 43 when the yaw angle and the pitch angle are predetermined values, and the in-face region 42 and the out-of-face region. The depth value of the region 43 can be a natural depth value corresponding to the angle of the face image.

  On the other hand, when a depth map is generated using three-dimensional shape information of a general human face as in the prior art, in order to generate a depth value according to the angle of the face image, the angle of the face image Each three-dimensional shape information needs to be held. Therefore, the amount of information used for generating the depth map increases.

[Description of processing of image processing apparatus]
FIG. 11 is a flowchart for explaining the depth map generation processing of the image processing apparatus 10 of FIG. This depth map generation process is started when an image is input to the image processing apparatus 10 from the outside as an input image.

  In step S11 of FIG. 11, the face detection unit 11 of the image processing apparatus 10 detects a face image in an input image input from the outside. The face detection unit 11 converts the face image information of the detected face image into an in-face region determination unit 12, an out-of-face region determination unit 13, a neck region determination unit 14, a torso region determination unit 15, and an torso region determination unit. 16 is supplied.

  In step S12, the face detection unit 11 detects the pitch angle and yaw angle of the detected face image as the angle of the face image. The face detection unit 11 supplies the detected angle of the face image and the size of the face image to the depth generation unit 18.

  In step S <b> 13, the in-face region determination unit 12 determines an ellipse region having a size based on the size of the face image centered on the center of the face image based on the face image information supplied from the face detection unit 11. And the face area information of the face area is supplied to the boundary determination unit 17.

  In step S <b> 14, the out-of-face area determination unit 13 has a size based on the size of the face image centered on the center of the face image based on the face image information supplied from the face detection unit 11. A larger ellipse area is set as the out-of-face area. The out-of-face area determination unit 13 supplies out-of-face area information of the out-of-face area to the boundary determination unit 17.

  In step S15, the neck region determination unit 14 sets a rectangular region having a size based on the size of the face image below the face image as the neck region based on the face image information supplied from the face detection unit 11. The neck area information of the neck area is supplied to the boundary determination unit 17.

  In step S <b> 16, the body region determination unit 15, based on the face image information supplied from the face detection unit 11, has a trapezoid region below the neck region and a size below the trapezoid region, based on the size of the face image. The rectangular area is set as the body area. The body area determination unit 15 supplies the body area information of the body area to the boundary determination unit 17.

  In step S17, the torso outside region determination unit 16 is based on the face image information supplied from the face detection unit 11, and is a size based on the size of the face image and larger than the torso region below the neck region. The region and the rectangular region below the trapezoidal region are set as the body outside region. The torso outside region determination unit 16 supplies the torso outside region information of the torso outside region to the boundary determination unit 17.

  In step S <b> 18, the boundary determination unit 17 determines the part region based on the part region information supplied from the in-face region determination unit 12, the neck region determination unit 14, the torso region determination unit 15, and the torso region determination unit 16. Is determined as a person area.

  In step S19, the boundary determination unit 17 determines each pixel in the person area based on the part area information as a final in-face area, an out-of-face area, a neck area, and a torso area, as described in FIG. The final region is determined by determining the pixel in the region outside the body. The boundary determination unit 17 supplies the determined part information on the final part region to the depth generation unit 18.

  In step S <b> 20, the boundary determination unit 17 generates a person area flag representing a person area for each pixel in the person area, and the person area for each pixel outside the person area in the input image. A person area flag representing that is not generated is generated. The boundary determination unit 17 supplies the person region flag of each pixel of the input image to the selection unit 19.

  In step S <b> 21, the depth generation unit 18 performs a depth value generation process for generating a depth value of each pixel in the person area. Details of the depth value generation processing will be described with reference to FIG.

  In step S <b> 22, the selection unit 19 determines a pixel that has not yet been determined as a pixel to be processed among the pixels constituting the input image as a processing target image. In step S <b> 23, the selection unit 19 determines whether or not the person area flag of the image to be processed represents a person area.

  If it is determined in step S23 that the person area flag of the image to be processed represents a person area, the process proceeds to step S24. In step S24, the selection unit 19 assigns the depth value of the pixel to be processed of the person area supplied from the depth generation unit 18 as the pixel value of the pixel in the depth map corresponding to the pixel to be processed, and performs the process in step S26. Proceed to

  On the other hand, if it is determined in step S23 that the person area flag of the image to be processed represents that it is not a person area, the process proceeds to step S25. In step S25, the selection unit 19 assigns a background image depth value input from the outside as the pixel value of the pixel of the depth map corresponding to the pixel to be processed, and advances the process to step S26.

  In step S <b> 26, the selection unit 19 determines whether all the pixels of the input image have been processed. When it is determined in step S26 that all the pixels of the input image have not yet been processed, the selection unit 19 returns the process to step S22 until all the pixels of the input image are processed. The processes in steps S22 to S26 are repeated.

  On the other hand, when it is determined in step S26 that all the pixels of the input image are the processing target pixels, the selection unit 19 smoothes the depth map in which the depth values are assigned as the pixel values of all the pixels. To supply.

  In step S27, the smoothing processing unit 20 smoothes and outputs the depth map supplied from the selection unit 19 using a general spatial filter, and ends the processing.

  FIG. 12 is a flowchart for explaining details of the depth value generation processing in step S21 of FIG.

  In step S <b> 41 of FIG. 12, the size correction unit 62 reads out the horizontal quadratic curve and the vertical quadratic curve of each region from the storage unit 61. In step S42, based on the size of the face image supplied from the face detection unit 11, the range jvol of the horizontal quadratic curve and the range ivol of the vertical quadratic curve of each read region are corrected.

  The size correction unit 62 supplies the corrected horizontal quadratic curve and vertical quadratic curve to the angle correction unit 63. The angle correction unit 63 supplies the horizontal quadratic curve and the vertical quadratic curve of the neck region 44, the body region 45, and the body outside region 46 supplied from the size correction unit 62 to the generation unit 64 as they are.

In step S <b> 43, the angle correction unit 63 determines whether the yaw angle of the face image supplied as the face image angle from the face detection unit 11 is zero. When it is determined in step S43 that the yaw angle of the face image is not 0, in step S44, the angle correction unit 63 is based on the non-zero yaw angle θ _j and the above-described equations (8) and (9). And the horizontal quadratic curve of the in-face region 42 and the out-of-face region 43 is corrected by Equation (13). Then, the angle correction unit 63 supplies the corrected horizontal quadratic curves of the in-face region 42 and the out-of-face region 43 to the generation unit 64, and the process proceeds to step S45.

  On the other hand, when it is determined in step S43 that the yaw angle of the face image is 0, the angle correction unit 63 skips the process of step S44 and the in-face region 42 and the out-of-face region supplied from the size correction unit 62. 43 horizontal quadratic curves are supplied to the generator 64 as they are. Then, the process proceeds to step S45.

In step S45, the angle correction unit 63 determines whether the pitch angle of the face image supplied as the face image angle from the face detection unit 11 is zero. When it is determined in step S45 that the pitch angle of the face image is not 0, in step S46, the angle correction unit 63, based on the non-zero pitch angle θ _i , as described in FIG. 42 and the vertical quadratic curve of the out-of-face area 43 are corrected. Then, the angle correction unit 63 supplies the corrected vertical quadratic curve of the in-face region 42 and the out-of-face region 43 to the generation unit 64, and the process proceeds to step S47.

  On the other hand, when it is determined in step S45 that the pitch angle of the face image is 0, the angle correction unit 63 skips the process of step S46 and the in-face area 42 and the out-of-face area supplied from the size correction unit 62. 43 vertical quadratic curves are supplied to the generation unit 64 as they are. Then, the process proceeds to step S47.

  In step S47, the generation unit 64 determines a pixel that has not yet been processed as a pixel to be processed among the pixels in the person area. In step S <b> 48, the generation unit 64 determines the region of the pixel to be processed based on the region information supplied from the boundary determination unit 17.

  In step S49, the generation unit 64 is based on the horizontal quadratic curve and the vertical quadratic curve of the region of the pixel to be processed among the horizontal quadratic curve and the vertical quadratic curve supplied from the angle correction unit 63. Depth values corresponding to the horizontal and vertical positions of the pixel to be processed are generated.

  In step S50, the generation unit 64 determines whether all the pixels in the person area have been processed. If it is determined in step S50 that all pixels in the person area have not yet been processed, the process returns to step S47, and steps S47 to S50 are performed until all pixels in the person area are processed. The process is repeated.

  On the other hand, when it is determined in step S50 that all the pixels in the person area have been processed, the generation unit 64 sets the depth values of all the pixels in the generated person area to the selection unit 19 in FIG. To supply. And a process returns to step S21 of FIG. 11, and progresses to step S22.

  As described above, the image processing apparatus 10 generates the depth value of the input image based on the quadratic curve representing the depth value model. Therefore, the amount of information used to generate the depth value can be reduced as compared with the conventional case where the depth value is generated based on the three-dimensional shape information of each feature point of a general human face. it can. As a result, the storage capacity of the storage unit 61 can be small.

  Further, since the image processing apparatus 10 generates the depth value for each part region based on the quadratic curve for each part region, compared to the case where the depth value is generated based on one quadratic curve of the entire person region. , Depth value freedom is increased. As a result, a more natural depth value can be easily generated. Furthermore, since the image processing apparatus 10 sets the part region based on the face image, the part region can be easily set.

  Further, in the image processing apparatus 10, since all the information used for generating the depth value of the object in each part area is a quadratic curve, the depth generation part 18 shares a calculation part for calculating the depth value between the part areas. Can be

  The image processing apparatus 10 sets five part regions in the input image and generates the depth value based on the quadratic curve for each part region. However, the number of part regions set in the input image is as follows. The number is not limited to five. For example, the image processing apparatus 10 may have a mode in which a predetermined part region is not set. This mode is set by an instruction from the user, for example.

[Another example of depth value generation processing]
FIG. 13 is a flowchart for explaining a depth map generation process when the image processing apparatus 10 has a neck area exclusion mode in which a neck area is not set. This depth map generation process is started when an image is input to the image processing apparatus 10 from the outside as an input image.

  The processing in steps S71 to S74 in FIG. 13 is the same as the processing in steps S11 to S14 in FIG.

  In step S75, the neck area determination unit 14 determines whether or not the neck area exclusion mode is set. If it is determined in step S75 that the neck area exclusion mode is not set, the neck area determination unit 14 advances the process to step S76.

  On the other hand, when it is determined in step S75 that the neck area exclusion mode is set, the neck area determination unit 14 skips step S76 and advances the process to step S77.

  The processes in steps S76 to S88 are the same as the processes in steps S15 to S27 in FIG. 13, but when the neck area exclusion mode is set, the person area and the part area do not include the neck area.

[Description of computer to which this technology is applied]
The series of processes described above can be executed by hardware or can be executed by software. When a series of processing is executed by software, a program constituting the software is installed in the computer. Here, the computer includes, for example, a general-purpose personal computer capable of executing various functions by installing various programs by installing a computer incorporated in dedicated hardware.

  FIG. 14 is a block diagram illustrating an example of a hardware configuration of a computer that executes the above-described series of processes using a program.

  In a computer, a central processing unit (CPU) 201, a read only memory (ROM) 202, and a random access memory (RAM) 203 are connected to each other by a bus 204.

  An input / output interface 205 is further connected to the bus 204. An input unit 206, an output unit 207, a storage unit 208, a communication unit 209, and a drive 210 are connected to the input / output interface 205.

  The input unit 206 includes a keyboard, a mouse, a microphone, and the like. The output unit 207 includes a display, a speaker, and the like. The storage unit 208 includes a hard disk, a nonvolatile memory, and the like. The communication unit 209 includes a network interface and the like. The drive 210 drives a removable medium 211 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

  In the computer configured as described above, the CPU 201 loads, for example, the program stored in the storage unit 208 to the RAM 203 via the input / output interface 205 and the bus 204 and executes the program. Is performed.

  The program executed by the computer (CPU 201) can be provided by being recorded on the removable medium 211 as a package medium or the like, for example. The program can be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

  In the computer, the program can be installed in the storage unit 208 via the input / output interface 205 by attaching the removable medium 211 to the drive 210. The program can be received by the communication unit 209 via a wired or wireless transmission medium and installed in the storage unit 208. In addition, the program can be installed in the ROM 202 or the storage unit 208 in advance.

  The program executed by the computer may be a program that is processed in time series in the order described in this specification, or in parallel or at a necessary timing such as when a call is made. It may be a program for processing.

  The embodiments of the present technology are not limited to the above-described embodiments, and various modifications can be made without departing from the gist of the present technology.

  For example, the image processing apparatus 10 may generate the depth value of an object in an area other than the person area (building area, sky area, landscape area, animal area, car area, etc.) in the image. Good. In this case, an area suitable for the area is set as the part area. For example, when the image processing apparatus 10 generates a depth value of an object in a car area, a windshield area, a tire area, a hood area, and the like are set as the part areas.

  Further, the image processing apparatus 10 may detect and set a part region from the input image instead of setting the part region based on the position and size of the face image.

  Further, the image processing apparatus 10 sets an inner region and an outer region in regions other than the face and body regions (such as a neck region), and generates depth values based on two types of quadratic curves. Also good. Further, the number of divisions of the face and body regions is not limited to 2, and may be 3 or more. As the number of divisions increases, the degree of freedom of the depth value increases and a more natural depth value can be generated.

  Further, the image processing apparatus 10 detects an angle of a body image other than a face image, a neck image, and the like, and according to the angle, a horizontal quadratic curve of a corresponding partial region, similar to the in-face region and the out-of-face region. The vertical quadratic curve may be corrected.

  Furthermore, the image processing apparatus 10 may not correct the horizontal quadratic curve and the vertical quadratic curve based on the size of the face image and the angle of the face image. Further, the generation unit of the depth value may not be a pixel unit.

  Further, for example, the present technology can take a configuration of cloud computing in which one function is shared by a plurality of devices via a network and is jointly processed.

  Furthermore, each step described in the above flowchart can be executed by one apparatus or can be shared by a plurality of apparatuses.

  In addition, when a plurality of processes are included in one step, the plurality of processes included in the one step can be executed by being shared by a plurality of apparatuses in addition to being executed by one apparatus.

  The present technology can be configured as follows.

(1)
An image processing apparatus comprising: a generation unit configured to generate a depth value of the object in an image based on a quadratic curve representing a depth value model representing a position of the object in the depth direction.
(2)
A setting unit configured to divide and set the region of the object in the image into a plurality of regions;
The image processing apparatus according to (1), wherein the generation unit generates a depth value of the object in the image based on the quadratic curve for each of the areas set by the setting unit.
(3)
The image processing apparatus according to (2), wherein the setting unit sets a region outside the object and a region inside the object.
(4)
The quadratic curve represents a model of the relationship between the depth value of the object and the position in the object,
The image generation unit according to any one of (1) to (3), wherein the generation unit generates the depth value corresponding to a position of the object in the image based on the quadratic curve. Processing equipment.
(5)
The quadratic curve represents a horizontal quadratic curve representing a model of the relationship between the depth value of the object and the horizontal position of the object, and a model of the relationship between the depth value of the object and the vertical position of the object. A vertical quadratic curve,
The generation unit generates the depth value corresponding to each position in the horizontal direction and the vertical direction of the object in the image based on the horizontal quadratic curve and the vertical quadratic curve. The image processing apparatus according to 4).
(6)
An angle correction unit for correcting the quadratic curve based on the angle of the object in the image;
The image processing device according to any one of (1) to (5), wherein the generation unit generates a depth value of the object in the image based on the quadratic curve corrected by the angle correction unit. .
(7)
A size correction unit for correcting the quadratic curve based on the size of the object in the image;
The image processing apparatus according to any one of (1) to (6), wherein the generation unit generates a depth value of the object in the image based on the quadratic curve corrected by the size correction unit. .
(8)
The image processing apparatus according to any one of (1) to (7), wherein the object includes at least a face.
(9)
The image processing device
An image processing method including a generation step of generating a depth value of the object in an image based on a quadratic curve representing a model of a depth value representing a position in the depth direction of the object.
(10)
Computer
The program for functioning as a production | generation part which produces | generates the depth value of the said object in an image based on the quadratic curve showing the model of the depth value showing the position of the depth direction of an object.

  DESCRIPTION OF SYMBOLS 10 Image processing apparatus, 17 Boundary determination part, 18 Depth generation part, 62 Size correction part, 63 Angle correction part

Claims (10)

An image processing apparatus comprising: a generation unit configured to generate a depth value of the object in an image based on a quadratic curve representing a depth value model representing a position of the object in the depth direction. A setting unit configured to divide and set the region of the object in the image into a plurality of regions;
The image processing apparatus according to claim 1, wherein the generation unit generates a depth value of the object in the image based on the quadratic curve for each region set by the setting unit. The image processing apparatus according to claim 2, wherein the setting unit sets a region outside the object and a region inside the object. The quadratic curve represents a model of the relationship between the depth value of the object and the position in the object,
The image processing apparatus according to claim 1, wherein the generation unit generates the depth value corresponding to a position for each position in the object in the image based on the quadratic curve. The quadratic curve represents a horizontal quadratic curve representing a model of the relationship between the depth value of the object and the horizontal position of the object, and a model of the relationship between the depth value of the object and the vertical position of the object. A vertical quadratic curve,
The generation unit generates the depth value corresponding to each position in the horizontal direction and the vertical direction of the object in the image based on the horizontal quadratic curve and the vertical quadratic curve. 5. The image processing apparatus according to 4. An angle correction unit for correcting the quadratic curve based on the angle of the object in the image;
The image processing apparatus according to claim 1, wherein the generation unit generates a depth value of the object in the image based on the quadratic curve corrected by the angle correction unit. A size correction unit for correcting the quadratic curve based on the size of the object in the image;
The image processing apparatus according to claim 1, wherein the generation unit generates a depth value of the object in the image based on the quadratic curve corrected by the size correction unit. The image processing apparatus according to claim 1, wherein the object includes at least a face. The image processing device
An image processing method including a generation step of generating a depth value of the object in an image based on a quadratic curve representing a model of a depth value representing a position in the depth direction of the object. Computer
The program for functioning as a production | generation part which produces | generates the depth value of the said object in an image based on the quadratic curve showing the model of the depth value showing the position of the depth direction of an object.

Images