JP2012173916A - Imaging device and image processing information generating program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a simply-constituted imaging device that is available even in an environment with many light disturbances.SOLUTION: An imaging device 1 takes an image of a subject A to be illuminated by a light L, and generates depth information showing a depth value of the subject A, reflection characteristic information showing a reflection characteristic of the subject A and illumination information showing conditions of the light L as image processing information. The imaging device 1 comprises: a camera C for taking an image that can be treated as a stereo image; the light L; and an image processing information generator 100 for using the image taken by the camera C to generate the image processing information. The camera C comprises: a lens including two polarizing filters with mutually-perpendicular polarization directions; a polarizing beam splitter; and two imaging elements for imaging light that is split by the polarizing beam splitter.

Description

  The present invention relates to a technique for generating image processing information such as depth information indicating a depth value of a subject, reflection characteristic information indicating a texture of a subject surface, and light source information indicating a lighting condition.

  Conventionally, image processing such as image recognition, image synthesis, video production, computer graphics (CG), and virtual reality (VR) has been widely performed. When performing this image processing, image processing information such as depth information indicating the depth value of the subject, reflection characteristic information indicating the texture of the subject surface, and light source information indicating illumination conditions (light source conditions) is important. Therefore, a method for generating such image processing information has been proposed.

  As a method of generating image processing information, a method of estimating a depth value using a stereo camera has been proposed (Non-Patent Document 1). The technique described in Non-Patent Document 1 estimates a depth value from a shift amount of images obtained by photographing the same subject from different viewpoints (the surface portion of the same subject exists in the image).

  In addition, a method for estimating a depth value using a camera-coded pupil has been proposed (Non-Patent Document 2). The technique described in Non-Patent Document 2 has a camera-coded special pupil (filter) built in the lens, and takes a blurred image in which the camera code is convoluted according to the depth value. The method described in Non-Patent Document 2 estimates a depth value by solving an inverse problem from this blurred image.

  And the method of estimating a depth value is proposed by the principle of triangulation (nonpatent literature 3). The technique described in Non-Patent Document 3 estimates the depth based on the principle of triangulation by irradiating a subject with range sensor slit light or camera-coded pattern light by an active technique such as light cutting. is there.

  Furthermore, a method for measuring the reflection characteristics of the subject surface has been proposed (Non-Patent Document 4). The technique described in Non-Patent Document 4 irradiates a subject with light while changing the constrained illumination condition, and measures the reflected light.

Heiko Hirschmuller (2008), Stereo Processing by Semi-Global Matching and Mutual Information, IEEE Transactions on Pattern Analysis and Machine Intelligence, Volume 30 (2), February 2008, pp.328-341. Image and Depth from a Conventional Camera with a Coded Aperture, http://groups.csail.mit.edu/graphics/CodedAperture/CodedAperture-LevinEtAl-SIGGRAPH07.pdf High-speed range finder with practical application 4.1 Optical cutting method, Journal of the Robotics Society of Japan Vol.23 No.3, pp.278-281, 2005, http://www.space-vision.jp/japanese/pdf/R15.pdf OGM: BRDF measuring device, http://www.dressingsim.com/new/product/LSC/OGM/index.html

  However, the methods described in Non-Patent Documents 1 to 3 cannot generate reflection characteristic information on the subject surface (that is, the texture of the subject) and light source information. Further, the method described in Non-Patent Document 3 is difficult to use in an environment where there is a lot of light disturbance such as outdoors because it is necessary to irradiate the subject with pattern light by an active method and measure the reflected light. is there. Furthermore, the method described in Non-Patent Document 4 cannot acquire the depth information of the subject, and it is necessary to measure the shape or surface normal of the subject beforehand with a range sensor or the like.

  As described above, no prior art has been proposed that can generate depth information, reflection characteristic information, and light source information together. If the methods described in Non-Patent Documents 1 to 4 are combined, the method for generating these pieces of image processing information together can be easily realized. However, as described below, it is not realistic to combine the methods described in Non-Patent Documents 1 to 4.

  As described above, the method described in Non-Patent Document 1 requires a stereo camera, the method described in Non-Patent Document 2 requires a camera incorporating a special pupil, and the method described in Non-Patent Document 4 is a range. Each technique requires a unique device, such as requiring a sensor. Therefore, when the methods described in Non-Patent Documents 1 to 4 are combined, the configuration of the imaging apparatus becomes very large and complicated, which is not realistic. For this reason, there is a request from the user that a method for generating these pieces of image processing information collectively with a simple configuration, and in particular, there is a strong demand for downsizing the camera body.

  SUMMARY OF THE INVENTION Accordingly, it is an object of the present invention to provide an imaging technique that solves the above-described problems and can collectively generate depth information, reflection characteristic information, and light source information in an environment with a lot of light disturbances, and has a simple configuration. .

  In order to solve the above-described problem, an imaging device according to the first invention of the present application images a subject illuminated by a light source, depth information indicating a depth value of the subject, reflection characteristic information indicating a reflection characteristic of the subject, and a light source An imaging device that generates light source information indicating a direction and a light amount of the imaging device, an imaging unit, a parallax amount calculation unit, a depth value conversion unit, an imaging unit position calculation unit, a diffuse reflected light component image generation unit, A specular reflection light component image generation unit, a light source direction calculation unit, a light source information generation unit, and a reflection coefficient calculation unit are provided.

  According to such a configuration, the imaging unit of the imaging apparatus transmits the incident light from one of the polarizing elements and the light transmitting element composed of two polarizing elements whose polarization directions are orthogonal to each other, and from the other of the polarizing elements. A spectroscopic element that reflects incident light, a first image sensor that images incident light that has passed through the spectroscopic element, and a second image sensor that captures incident light reflected by the spectroscopic element are provided.

  Here, due to the optical conditions of the imaging means, in the two captured images captured by the first image sensor and the second image sensor, the subject position is deviated from the in-focus position, resulting in blurring. Depending on the parallax, parallax occurs. That is, both captured images can be handled as stereo images.

  In addition, the imaging device has a simpler configuration of imaging means than a stereo camera that generates a stereo image. Furthermore, the imaging apparatus does not need to irradiate the subject with the pattern light when performing imaging with the imaging unit.

  In addition, the imaging apparatus uses, for example, matching between two captured images obtained by capturing the subject with the first imaging element and the second imaging element by the parallax amount calculation unit, and the parallax amount for each corresponding pixel corresponding to the two captured images. Is calculated. Then, the imaging apparatus uses the depth value conversion unit to convert the parallax amount calculated by the parallax amount calculation unit into a depth value for each corresponding pixel using a preset conversion formula to generate depth information. Here, the depth information indicates the distance from the imaging means to the subject surface. In other words, connecting the position indicated by the depth value of the target pixel and the position indicated by the depth value of the neighboring pixel indicates the subject surface. Therefore, the imaging apparatus uses the depth value conversion unit to calculate a normal direction orthogonal to the subject surface represented by the depth values of the target pixel and the neighboring pixels of the target pixel, using the depth information.

  In addition, the imaging apparatus calculates the position of the imaging unit by performing an optimization process on the depth value calculated by the depth value conversion unit by the imaging unit position calculation unit. Then, the imaging apparatus selects, with the diffuse reflected light component image generation unit, the smaller of the corresponding pixels in the two captured images as the minimum luminance value, and diffusion indicating the minimum luminance value for each corresponding pixel A reflected light component image is generated. Furthermore, the imaging apparatus selects, as the maximum luminance value, the larger luminance value of the corresponding pixels in the two captured images as the maximum luminance value by the specular reflection component image generation unit, and the maximum luminance value and the minimum luminance value for each corresponding pixel. A specular reflection light component image showing the difference between and is generated.

  Here, for example, when specular reflection light polarized in an oblique 45 degree direction is incident, the first image sensor and the second image sensor have the same amount of specular reflection light. Therefore, even if the minimum luminance value is selected. It is difficult to suppress the specular reflection light component. That is, the effect of suppressing the specular reflection light component is brought about by shifting from 45 degrees. In order to reduce this influence by the optical system, the configuration of the imaging means becomes large and complicated, which is not practical. Therefore, the diffuse reflected light component image generation unit selects the minimum luminance value and calculates the diffuse reflected light component. Thereafter, the specular reflection light component image generation unit calculates the specular reflection light component using the diffuse reflection light component.

  The diffuse reflected light component image is an image composed of pixels with smaller luminance values among the corresponding pixels in the two captured images, and shows the diffuse reflected light component. The specular reflection light component image is an image composed of pixels in which the difference between the maximum luminance value and the minimum luminance value becomes the luminance value, and indicates the specular reflection light component.

In addition, the imaging apparatus calculates the incident angle of the reflected light at the subject from the normal direction calculated by the depth value conversion unit and the position of the imaging unit by the light source direction calculation unit, and the direction indicated by the calculated incident angle is the light source Calculate as the direction of. Here, for example, the light source direction calculation unit calculates the direction of the light source from the normal direction of the surface of the subject and the position of the imaging means, according to the law of reflection.
By

  In addition, the imaging apparatus calculates the light amount of the light source based on the specular reflection component image generated by the specular reflection component image generation unit and the direction of the light source by the light source information generation unit, and the calculated light amount and direction of the light source Generate as information. Here, for example, the light source information generation unit divides a virtual sphere having a predetermined radius into a lattice shape, and obtains the light source intensity of each light beam for each lattice vertex. Then, the light source information generation unit calculates the maximum value among the light source intensities of each light ray, and normalizes between zero and the maximum light amount over each lattice vertex included in the phantom sphere, so that the light amount of the light source Is calculated.

  Further, the imaging apparatus performs an optimization process by applying a reflection model to the light source information generated by the light source information generation unit, the specular reflection light component image, and the diffuse reflection light component image by the reflection coefficient calculation unit. The specular reflection coefficient and the diffuse reflection coefficient are calculated as reflection characteristic information.

  The imaging apparatus according to the second invention of the present application further includes a light source control unit that turns on and off the light source, and the light source information generation unit calculates the light amount of the light source when the light source is turned on and when the light source is turned off. It is preferable to calculate light source information represented by an absolute value by calculating a weight from a light amount when the light source is turned off by a predetermined calculation formula, and multiplying the calculated weight by the light amount when the light source is turned off.

  The imaging apparatus according to the third aspect of the present invention further includes a subject region extraction unit that extracts, from one of the two captured images, a region included in a depth extraction range in which the depth value of the corresponding pixel is set in advance as a subject region. It is preferable.

  In addition, the imaging apparatus according to the fourth invention of the present application detects a face area from one of two captured images by face area detection processing, calculates an average value of depth values of all pixels included in the detected face area, and calculates It is preferable to further include a depth extraction range calculation unit that sets a depth extraction range by adding and subtracting a predetermined value to the average value.

  In the imaging apparatus according to the first invention of the present application, a general computer includes a parallax amount calculation unit, a depth value conversion unit, an imaging means position calculation unit, a diffuse reflection component image generation unit, a specular reflection component image generation unit, It can also be realized by an image processing information generation program that functions as a light source direction calculation unit, a light source information generation unit, and a reflection coefficient calculation unit.

  According to the first aspect of the present invention, the configuration of the imaging unit is simple, and depth information, reflection characteristic information, and light source information can be collectively generated using only a captured image captured by the imaging unit. Therefore, according to the first invention of the present application, a simple configuration can be realized without requiring a unique device for generating each piece of information for each of depth information, reflection characteristic information, and light source information. Furthermore, according to the first invention of the present application, it is not necessary to irradiate the subject with the pattern light when performing imaging with the imaging means, and therefore, it can be used even in an environment with a lot of light disturbance.

  According to the second invention of the present application, since the light source information is represented by an absolute value, the actual brightness of the light source can be known from the light source information. Furthermore, according to the second aspect of the present invention, when image processing is performed in another imaging environment using the light source information, it is not necessary to convert the light source information for the other imaging environment. That is, according to the second invention of the present application, the consistency of the light source information can be easily maintained.

According to the third aspect of the present invention, since the subject region can be extracted in addition to the depth information, the reflection characteristic information, and the light source information, the convenience of the imaging apparatus can be improved.
According to the fourth aspect of the present invention, since the depth extraction range can be calculated, it is not necessary to manually set this, and the convenience of the imaging apparatus can be further improved and the face area of the subject can be accurately extracted.

It is the schematic which shows the structure of the imaging device which concerns on embodiment of this invention. (A) is a figure which shows the structure of the camera of FIG. 1, (b) is a figure explaining the lens with which the camera of FIG. 1 is provided. It is a figure explaining the reason which can handle the captured images 1 and 2 as a stereo image, (a) is a figure which paid its attention to the 1st image sensor, (b) is a figure which paid attention to the 2nd image sensor. It is a block diagram which shows the structure of the image processing information generation apparatus of FIG. (A) And (b) is a figure explaining conversion by the depth value conversion part of FIG. It is a figure explaining conversion by the depth value conversion part of FIG. It is a figure explaining calculation of the normal direction by the depth value conversion part of FIG. It is a figure explaining calculation of the normal direction by the depth value conversion part of FIG. It is a figure explaining the production | generation of the diffuse reflection light component image by the diffuse reflection light component image generation part of FIG. It is a figure explaining the production | generation of the illumination information by the illumination information generation part of FIG. It is a flowchart which shows operation | movement of the image processing information generation apparatus of FIG.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings as appropriate. In each embodiment, means having the same function and the same member are denoted by the same reference numerals, and description thereof is omitted.

[Configuration of imaging device]
With reference to FIG. 1, a configuration of an imaging apparatus 1 according to an embodiment of the present invention will be described.
The imaging device 1 captures a subject A illuminated by an illumination (light source) L, generates depth information, reflection characteristic information, and illumination information (light source information) as image processing information. Imaging means) C and an image processing information generating apparatus 100.

The camera (imaging means) C images the subject A. The camera C is connected to the image processing information generation device 100 via the signal cable 9 _C , and outputs a captured image (moving image) to the image processing information generation device 100.

  The subject A is a subject to be imaged by the imaging device 1. In FIG. 1, a cylindrical object is illustrated as the subject A, but it may be a person, an animal, a landscape, or the like. Further, the number of subjects in the imaging apparatus 1 is not limited to one, and may be two or more.

In this embodiment, the illumination L will be described as an example of the light source.
The illumination L is a light source that illuminates the subject A, and is disposed around the camera C. The illumination L is, for example, a white LED (Light Emitting Diode) that can be turned on and off at high speed (in units of one frame second). The illumination L, via a signal cable 9 _C, and is connected to the image processing information generating unit 100.
In the present embodiment, since the light source is the illumination L, the image processing information generation device 100 described later generates illumination information as the light source information.

[Camera configuration]
Here, the configuration of the camera C will be described with reference to FIG.
As shown in FIG. 2A, the camera C includes a lens (light transmission element) 11, a polarization BS (spectral element) 12, a first imaging element 13, and a second imaging element 14.
In FIG. 2A, the vertical polarization component and the horizontal polarization component of the incident light are indicated by arrows.

  The lens (light transmitting element) 11 has two polarizing filters (polarizing elements) 11a and 11b whose polarization directions are orthogonal to each other. For example, as illustrated in FIG. 2B, the lens 11 includes a polarizing filter 11 a on the left side of the lens surface of the convex lens and a polarizing filter 11 b on the right side of the lens surface as viewed from the outside of the camera C. In this case, the lens 11 can be formed, for example, by bonding two thin-film polarizing filters to the lens surface of a general convex lens.

  In the lens 11, the ratio of the polarizing filters 11a and 11b is 1: 1, but the present invention is not limited to this. For example, in the lens 11, the ratio of the polarization filters 11 a and 11 b is calculated by taking into consideration the sensitivity of the first image sensor 13 and the second image sensor 14 and the performance of the lens 11, and the accuracy of block matching by the parallax amount calculation unit 101 described later. Can be changed within the range that can be maintained.

  As shown in FIG. 2A, the polarizing filter 11a passes only the vertical polarization component of the incident light from the subject A. Further, the polarization filter 11b passes only the lateral polarization component of the incident light from the subject A.

  The polarization BS (spectral element) 12 is a beam splitter that transmits incident light (vertical polarization component) from the polarization filter 11a and reflects incident light (lateral polarization component) from the polarization filter 11b. . The light transmitted through the polarized light BS12 reaches the image pickup surface of the first image pickup device 13. Further, the light reflected by the polarized light BS12 reaches the imaging surface of the second imaging element 14.

  The first image pickup device 13 picks up an image of light (longitudinal polarization component) transmitted through the polarized light BS12, and is, for example, a CCD or a CMOS sensor. Then, the first image sensor 13 outputs an image obtained by imaging the subject A to the image processing information generation device 100.

  The second image sensor 14 images light reflected by the polarized light BS12 (laterally polarized light component), and is, for example, a CCD or CMOS sensor. Then, the second image sensor 14 outputs an image obtained by capturing the subject A to the image processing information generation device 100.

  Hereinafter, each frame image of the moving image captured by the first image sensor 13 is referred to as “captured image 1”, and each frame image captured by the second image sensor 14 is referred to as “captured image 2”. Since these captured images 1 and 2 can be handled as stereo images, the reason will be described.

<Reasons for handling as stereo images>
Of the incident light from the subject, at least the diffuse reflected light component is dispersed in the polarization direction. As a result, the diffuse reflected light component is imaged only on the polarizing filter 11 a or the polarizing filter 11 b under the same condition as the aperture shape of the lens 11. In other words, the diffusely reflected light component is shielded substantially evenly by the polarizing filter 11a and the polarizing filter 11b because the polarization direction is not biased.

  On the other hand, the specular reflection component in which a light beam from a certain light source is reflected on the subject surface is incident on the camera C while satisfying the condition of incident angle = reflection angle (except when the subject condition such as a concave mirror is satisfied). ). Therefore, the specular reflection light component has an intensity ratio of P-polarized light and S-polarized light that is not 1: 1 with a Brewster angle that differs for each material (refractive index of subject A). (Except when the position of the camera C is not a light source). For this reason, if specular reflection is caused on the surface of the subject A, the first image sensor 13 and the second image sensor 14 have different amounts of incident light.

  Due to the above optical system conditions, the first image sensor 13 and the second image sensor 14 are blurred in the captured images 1 and 2 due to the displacement of the subject A with respect to the in-focus position. And since parallax arises according to this blur, the captured images 1 and 2 can be handled as a stereo image.

A more specific description will be given with reference to FIG. 3 (see FIG. 2 as appropriate).
Thereafter, a round object at the focal position of the lens 11 is indicated by “◯”, a triangular object at the rear side of the focal position of the lens 11 is indicated by “Δ”, and a square on the front side of the focal position of the lens 11 is indicated. Let the square object be “□”.

  In FIG. 3A, attention is paid to the first image sensor 13, and the polarization filter 11b is a filter for the first image sensor 13, and therefore the polarization filter 11b is illustrated by dots. In FIG. 3B, attention is paid to the second image sensor 14, and the polarization filter 11a is a filter for the second image sensor 14, and therefore the polarization filter 11a is illustrated by dots.

  Consider the case where the lens 11 is a general convex lens. Since the subject “◯” is at the focal position of the lens 11, it forms an image on the imaging surface of the first imaging element 13. In addition, since the subject “Δ” is located behind the focal position of the lens 11, an image is formed on the front side of the imaging surface of the first imaging element 13. Furthermore, since the subject “□” is on the near side of the focal position of the lens 11, an image is formed on the back side of the imaging surface of the first imaging element 13.

  Next, the lens 11 includes polarizing filters 11a and 11b, the polarizing filter 11a transmits incident light from the subjects “◯”, “Δ”, and “□”, and the polarizing filter 11b includes the subjects “O” and “Δ”. Consider the case where the incident light from “□” is shielded.

  In this case, as shown in FIG. 3A, in the subject “Δ”, the optical path of the incident light that has passed through the polarizing filter 11 a once intersects in front of the imaging surface of the first imaging element 13, so that the first imaging is performed. It reaches the right side of the imaging surface of the element 13. In addition, in the subject “□”, incident light that has passed through the polarizing filter 11 a reaches the left side of the imaging surface of the first imaging element 13. Accordingly, the first image sensor 13 captures the captured image 1 in which the subjects “□”, “◯”, and “Δ” are arranged in order from the left side.

  In FIG. 3B, the polarizing filter 11b transmits the incident light from the subjects “◯”, “Δ”, “□”, and the polarizing filter 11a from the subjects “O”, “Δ”, “□”. To block the incident light.

  For this reason, in the subject “Δ”, the optical path of the incident light that has passed through the polarization filter 11 b once intersects before the imaging surface of the second imaging element 14 and reaches the left side of the imaging surface of the second imaging element 14. In addition, in the subject “□”, incident light that has passed through the polarization filter 11 b reaches the right side of the imaging surface of the second imaging element 14. Therefore, the second image sensor 14 captures the captured image 2 in which the subjects “Δ”, “◯”, and “□” are arranged in order from the left side.

  As described above, in the captured images 1 and 2, the subjects “Δ” and “□” are imaged so as to be shifted from the center of the image to the left and right as the focal position of the lens 11 is shifted to the near side or the far side. Therefore, the captured images 1 and 2 generate parallax according to the positions of the subjects “◯”, “Δ”, and “□”, and can be handled as stereo images.

[Configuration of Image Processing Information Generating Device]
The configuration of the image processing information generation apparatus 100 will be described with reference to FIG. 4 (see FIGS. 1 and 2 as appropriate).
In FIG. 4, a part of the configuration of the camera C is omitted for easy viewing of the drawing.

  As shown in FIG. 4, the image processing information generation apparatus 100 generates image processing information, and includes a parallax amount calculation unit 101, a depth value conversion unit 110, and a camera parameter calculation unit (imaging means position calculation unit). 120, a depth information storage unit 130, a reflection characteristic estimation unit 140, a parameter setting unit 150, a depth extraction range calculation unit 160, a subject area extraction unit 170, and an illumination control unit 180.

The image processing information generation apparatus 100 is mounted on, for example, a computer that includes an image input board for captured images 1 and 2 and a general-purpose I / O board for illumination control.
Note that the illumination control unit 180 is a configuration necessary when the illumination information is represented by an absolute value, and will be described in a modification example described later.

  The parallax amount calculation unit 101 receives the captured images 1 and 2 from the camera C and calculates the parallax amount for each corresponding pixel corresponding to the captured images 1 and 2. The parallax amount calculation unit 101 calculates the parallax amount as a shift amount for each piece of each part in the captured images 1 and 2 by block matching, for example. Then, the parallax amount calculation unit 101 outputs the calculated parallax amount to the depth value conversion unit 110.

  Here, the parallax amount calculation unit 101 can improve the accuracy of matching by performing block matching for high-frequency spatial frequency components or block matching using color information.

In the present embodiment, the parallax amount calculation unit 101 uses block matching, but the present invention is not limited to this. That is, the parallax amount calculation unit 101 can apply various matching methods for calculating the parallax amount from the shift amount of the stereo image, such as a gradient method.
This gradient method is described, for example, in the document “Detection of optical flow by generalized gradient method: Measurement of object motion under non-uniform illumination, Information Processing Society of Japan, Journal of Information Processing Society of Japan, Computer Vision and Image Media 49 (SIG_6 (CVIM — 20)), 1-12, 2008-03-15, http://ci.nii.ac.jp/naid/110006684635 ”.

  The depth value conversion unit 110 receives the parallax amount from the parallax amount calculation unit 101, converts the parallax amount into a depth value for each corresponding pixel using a conversion formula, and generates depth information. Further, the depth value conversion unit 110 calculates a normal direction orthogonal to the subject surface represented by the depth values of the target pixel and the neighboring pixels of the target pixel. Then, the depth value conversion unit 110 outputs the generated depth information and the normal direction of the subject surface to the depth information storage unit 130. Further, the depth value conversion unit 110 outputs the generated depth information to the camera parameter calculation unit 120.

<Specific example of conversion by depth value conversion unit>
A specific example of conversion by the depth value conversion unit 110 will be described with reference to FIGS. 5 and 6 (see FIG. 4 as appropriate).
The blur center position of the subject is the position of the center of gravity of the opening in the lens 11 (that is, the polarization filter 11a or the polarization filter 11b). Here, the opening in the lens 11 will be described as a semicircle.

In this case, as shown in FIG. 5A, the center of gravity position G (x _G , y _G ) is on the target axis (y axis), so x _G = 0. Therefore, the distance y _G from the x axis is obtained. In a thin band (shown by hatching) at the position of height y, the area moment about the x-axis is (2x) yΔy. When the entire semicircle is divided into thin bands, the area moment about the x-axis of the semicircle can be expressed by the following equation (1).

  When x included in the integral term of the equation (1) is expressed as a function of y, the following equation (2) is established.

Here, when Expression (3) is substituted into Expression (1), the following Expression (4) is obtained. Then, the area of the semicircle, because it is? Pa ^2/2, the distance _{y G} can be expressed by the following equation (5).

  In FIG. 5A, a is the opening radius of the lens 11. The center of gravity calculation is described in, for example, the document “http://www.mech.kanagawau.ac.jp/lab/urata_lab/class/MathMachAna/2003/mach08txt.pdf”.

  Next, as shown in FIG. 5B, a case where the polarizing filter 11a is an aperture in the optical system of the camera C is considered. In this case, the diameter of the lens 11 is Φa = 2a, and the gravity center position G1 of the polarizing filter 11a is 4a / 3π from the center position of the blur. In addition, the diameter Φb = 2b of the imaging surface of the first imaging element 13 (where b is the radius of the first imaging element 13 and is defined by equation (3) described later), The barycentric position G2 of the region where the incident light from the filter 11a reaches is 4b / 3π from the center position of the blur. Therefore, when the blur radius is c, in the captured images 1 and 2 at the position of 4c / 3π, the parallax amount can be expressed as 2 × 4c / 3π.

  As shown in FIG. 6, in the lens 11, the following formula (6) is established from the lens formula. Here, o is the principal point of the lens 11, f is the focal point of the lens 11, p is the end point of the lens 11, a 'is an image of a, b' is an image of b, F Is the focal length (distance between the principal point and the focal point), A is the distance (depth value) from the object plane (subject surface) to the principal point, B is the distance from the principal point to the image plane, m Is the magnification.

Here, the imaging surface of the first imaging element 13 is located on the right side f. Therefore, from FIG. 6, the relationship between the center position of the blur and the parallax amount Dp can be expressed by the following equations (7) and (8). In Expressions (7) and (8), op ^→ is the aperture diameter of the lens 11 (that is, the diameter of the polarizing filter 11a), and Dp is the amount of parallax.

Here, since the magnification m of Expression (6) is known from the specification of the lens 11 and the aperture diameter op is also known, the parallax amount calculation unit 101 calculates the parallax amount Dp. Therefore, the depth value conversion unit 110 can convert the parallax amount Dp into the depth value A using Expressions (6) to (8).
The conversion formula is set in advance in the depth value conversion unit 110 because the relationship between the parallax amount and the absolute depth value, which is the distance from the camera C, varies depending on the design and setting of the camera C.

  In addition, although Formula (6)-Formula (8) is a case where a focus position is after an imaging surface (the back side of an imaging surface), even when a focal position is before an imaging surface, it can convert similarly. At this time, since the conversion formula differs before and after the imaging surface, it is necessary to select the conversion formula. In this case, for example, the depth value conversion unit 110 may select the conversion formula on the condition that the parallax is 0 because the direction in which the parallax is generated is reversed with the parallax being 0 as a boundary.

<Calculation of normal direction by depth value conversion unit>
Next, calculation of the normal direction by the depth value conversion unit 110 will be described with reference to FIGS. 7 and 8 (see FIG. 4 as appropriate).

For example, as shown in FIG. 7 the lower part, and the pixel of interest P _C, and the neighboring pixel P _L adjacent to the left of the target pixel P _C, the depth values of the neighboring pixels P _R that is adjacent to the right side of the pixel of interest P _C determined Suppose. At this time, as shown in FIG. 7 upper stage with respect to the depth value of the pixel of interest P _C, the depth values of the neighboring pixels P _R has shown the front side, the depth values of the neighboring pixels P _L is shows a back side Suppose that In this case, the subject surface α represented by the depth values of the target pixel P _C and the neighboring pixels P _R and P _L is inclined with respect to the camera C such that the right side is close to the camera C and the left side is far from the camera C. Yes. Thus, the pixel of interest P _C and the neighboring pixel P _R, the CPU recognizes the direction of the subject surface alpha from the depth values of the P _L, it can be obtained in the normal direction β of the object surface alpha. The normal direction β of the subject surface α is based on the geometric position of the camera C.

Thus, the depth value conversion unit 110, while moving the pixel of interest P _C to each part of the subject to repeat it obtains the normal direction β of the surface for each site of the subject. Although attention is paid to a part of the subject in FIG. 7, the subject is often imaged three-dimensionally as shown in FIG.

  For this reason, consider a case where the normal direction of the surface of the subject is calculated using a polyhedron as the subject. In FIG. 8, a surface 1 that is the bottom surface of the polyhedron is the subject surface. The polyhedron has vertices P1 to P8 on each surface, and is a convex polygon. These vertices P1 to P8 are given clockwise when viewed from the outside of the polyhedron. The viewpoint indicates the position of the camera C. The line-of-sight vector is a vector indicating the line of sight of the camera C (the optical axis of the lens 11). The normal vector is a vector indicating the normal of the surface 1. In this case, the depth value conversion unit 110 can calculate the normal N of the surface 1 as an outer product of two lines as shown in the following equation (9).

Here, it is defined by P ₁ = (x ₁ , y ₁ , z ₁ ), P ₂ = (x ₂ , y ₂ , z ₂ ), P ₃ = (x ₃ , y ₃ , z ₃ ). Therefore, in FIG. 8, the direction of the normal line N is the normal direction of the subject surface.
This method is described, for example, on the homepage "http://nis-lab.is.su-tokyo.ac.jp/nis/CG/cgtxt/furoku/fu1.htm#Appendix%EF%BC%91" Has been.

Returning to FIG. 4, the description of the configuration of the image processing information generation apparatus 100 will be continued.
The camera parameter calculation unit 120 receives the depth information from the depth value conversion unit 110, performs optimization processing on the depth value indicated by the depth information, and calculates the position and direction of the camera C. The camera parameter calculation unit 120 calculates the position and direction of the camera C as camera parameters by performing an optimization process such as a least square method, for example. Then, the camera parameter calculation unit 120 outputs the calculated camera parameters to the depth information storage unit 130 and the reflection characteristic estimation unit 140.

The camera parameter calculation method is described in, for example, the document “Integration of 3D feature point maps restored from images into the world coordinate system, The Institute of Electronics, Information and Communication Engineers, 2010 IEICE General Conference, http: //www.kameda-lab.org/research/publication/2010/201003_IEICE/201003IEICE-d_12_077-HayashiM.pdf ”.
The camera parameter calculation unit 120 can also use an optimization method such as Newton's method, Powell's method, or conjugate gradient method.

  The depth information storage unit 130 receives depth information and the normal direction of the subject surface from the depth value conversion unit 110, receives camera parameters from the camera parameter calculation unit 120, and stores the depth information and normal direction. is there. For this reason, the depth information storage unit 130 includes a storage device 13a and a storage device control unit 13b. The storage device 13a is, for example, a memory that stores depth information. The storage device control unit 13b controls writing to the storage device 13a and reading from the storage device 13a.

  Here, since the camera parameters (the direction and position of the camera C) are input, the storage device control unit 13b uses the world coordinates based on the depth information, the normal direction of the subject surface, and the luminance values of the captured images 1 and 2. Are aligned in the virtual space and stored in the storage device 13a.

  In the present embodiment, the depth information, the normal direction of the subject surface, and the luminance values of the captured images 1 and 2 are stored on the voxel space provided in the storage device 13a. Further, the depth information is subjected to a voting process (+1 voting process) on the voxel coordinates from which the depth value is obtained, and is stored as a label indicating the presence / absence of the object surface and auxiliary information for polygonization.

  The reflection characteristic estimation unit 140 receives the captured images 1 and 2 from the camera C, receives camera parameters from the camera parameter calculation unit 120, and refers to the depth information stored in the depth information storage unit 130 and the normal direction of the subject surface. Thus, the reflection characteristic information and the illumination information are generated. Therefore, the reflection characteristic estimation unit 140 includes a diffuse reflection light component image generation unit 141, a specular reflection light component image generation unit 142, an illumination direction calculation unit (light source direction calculation unit) 143, and an illumination information generation unit (light source information). Generation section) 144 and a reflection coefficient calculation section 145.

  The diffuse reflected light component image generation unit 141 selects the smaller luminance value of the corresponding pixels in the captured images 1 and 2 as the minimum luminance value, and the diffuse reflected light component image indicating the minimum luminance value for each corresponding pixel. Is generated.

  Here, the captured image 1 does not include a horizontal polarization component, and the captured image 2 does not include a vertical polarization component. Further, as described above, for example, when specular reflection light polarized in an oblique 45 degree direction is incident, the first image sensor 13 and the second image sensor 14 have the same amount of specular reflection light. Even if the value is selected, it is difficult to suppress the specular reflection light component. That is, the effect of suppressing the specular reflection light component is brought about by shifting from 45 degrees. In order to reduce this influence by the optical system, the configuration becomes large and complicated, which is not practical.

  Further, when the luminance value by the specular reflection light component of various polarization directions is obtained for a certain time, there is a possibility that the state is probabilistically orthogonal to the polarization filters 11a and 11b or close to orthogonal. In this case, the specular reflection light component is shielded by one of the polarization filters 11a and 11b. For this reason, the diffuse reflection light component image generation unit 141 selects and collects the minimum luminance values of the corresponding coordinates among the corresponding pixels in the captured images 1 and 2 to thereby collect the diffuse reflection light that suppresses the specular reflection light component. Generate component images. As a result, the diffuse reflected light component image generation unit 141 improves the separation accuracy between the diffuse reflected light component and the specular reflected light component compared to the prior art, and the diffuse reflected light component is less likely to be included and highly accurate. Can be extracted.

  In the example of FIG. 9, the diffuse reflected light component image generation unit 141 sets the pixel located at the upper left in the captured images 1 and 2 as the corresponding pixel. In addition, the diffuse reflected light component image generation unit 141 selects the smaller one of the two corresponding pixels as the minimum luminance value. Then, the diffuse reflected light component image generation unit 141 sets the selected minimum luminance value as the luminance value of the pixel (for example, the upper left pixel) at the same position as the corresponding pixel. The diffuse reflected light component image generation unit 141 performs such processing repeatedly on the corresponding pixels while moving the corresponding pixels in the captured images 1 and 2 to generate a diffuse reflected light component image. To do. That is, the diffuse reflected light component image is an image composed of pixels with smaller luminance values among the corresponding pixels in the captured images 1 and 2 and indicates the diffuse reflected light component.

  The specular reflection light component image generation unit 142 selects the largest luminance value among the corresponding pixels in the captured images 1 and 2 as the maximum luminance value, and calculates the difference between the maximum luminance value and the minimum luminance value for each corresponding pixel. The specular reflection component image shown is generated.

  Similar to the diffuse reflection light component image generation unit 141, the specular reflection light component image generation unit 142 selects, as the maximum luminance value, the larger luminance value among the corresponding pixels located in the upper left of the captured images 1 and 2. Further, the diffuse reflection light component image generation unit 141 refers to the diffuse reflection light component image generated by the diffuse reflection light component image generation unit 141 and acquires the minimum luminance value of the corresponding pixel. Then, the diffuse reflected light component image generation unit 141 determines the difference between the selected maximum brightness value and the minimum brightness value acquired from the diffuse reflected light component image as a pixel at the same position as the corresponding pixel (for example, the upper left pixel). Luminance value. The specular reflection component image generation unit 142 repeatedly performs such processing on all corresponding pixels while moving the corresponding pixels in the captured images 1 and 2 to generate a specular reflection component image. That is, the specular reflection component image is an image composed of pixels in which the difference between the maximum luminance value and the minimum luminance value becomes the luminance value, and indicates the specular reflection component.

  The illumination direction calculation unit (light source direction calculation unit) 143 calculates the incident angle of the reflected light from the subject A from the normal direction of the subject surface and the position of the camera C, and uses the calculated direction of the incident angle as the illumination direction. Is to be calculated.

  Here, from the position of the camera C and the normal direction of the subject surface, the emission angle of the reflected light reflected by the subject A can be calculated by the law of reflection. In other words, the intensity of the specular reflected light becomes extremely small as the line of sight of the camera C deviates from the reflected light emission angle. From this, if the illumination direction calculation unit 143 is specular reflection, the incident angle is equal to the reflection angle. Therefore, the direction indicated by the incident angle of the reflected light indicates the direction of the illumination L irradiated with the light that is the source of the reflected light. Calculate as direction.

  The illumination information generation unit (light source information generation unit) 144 is based on the specular reflection component image generated by the specular reflection component image generation unit 142 and the direction of the illumination L calculated by the illumination direction calculation unit 143. The light quantity is calculated, and the calculated light quantity and direction of the illumination L are generated as illumination information.

<Generation of illumination information by illumination information generation unit>
With reference to FIG. 10, generation of illumination information by the illumination information generation unit 144 will be described (see FIG. 4 as appropriate).
Here, the illumination information generation unit 144 determines the difference between the minimum luminance value and the maximum luminance value (= intensity of the specular reflection light component indicated by the specular reflection light component image) and the surface of the subject based on the direction of the illumination L. The light quantity of the illumination L on the line formed by the direction of the illumination L can be acquired. In order to simplify the calculation, consider a case where a virtual sphere Vb having a certain radius is set in advance and the virtual sphere is previously divided into a lattice shape as shown in FIG. In this case, the illumination information generation unit 144 stores the acquired light amount of the illumination L in association with each lattice vertex.

  At this time, a plurality of reflected lights may enter each lattice vertex due to reflection on the surface of the subject. In the example of FIG. 10, two reflected lights are incident on the lattice vertex Sp1. In this way, when a plurality of reflected lights are incident on a certain lattice vertex, there is a high possibility that the illumination L exists in the direction indicated by the incident angle of the reflected light having the maximum intensity among these reflected lights. Therefore, the illumination information generation unit 144 selects, as the light quantity of the illumination L at the lattice vertex, the one having the maximum intensity among the plurality of reflected lights incident on each lattice vertex. And the illumination information generation part 144 normalizes the light quantity of the illumination L in each lattice position from zero to the maximum light quantity, and memorize | stores it as illumination information with the direction of the illumination L. FIG. That is, since the illumination information is normalized, it is represented by a relative value.

This illumination information is equivalent to an HDR omnidirectional image that is generally used when rendering a high-quality CG. Therefore, the illumination information is effective information for improving the reality when a separate CG component is rendered and combined during shooting.
The lighting information is described in detail on the homepage “http://ict.debevec.org/˜debevec/Research/HDR/”, for example.

Returning to FIG. 4, the description of the configuration of the image processing information generation apparatus 100 will be continued.
The reflection coefficient calculation unit 145 performs an optimization process that applies a reflection model to the illumination information generated by the illumination information generation unit 144, the specular reflection light component image, and the diffuse reflection light component image, to thereby obtain a specular reflection coefficient. The diffuse reflection coefficient is calculated as reflection characteristic information.

  Here, by the processing up to the illumination information generation unit 144, the position of the camera C (that is, the principal point position of the lens 11), the position of the subject A in the virtual space, the normal of the subject surface, and the incident light on the subject Is known. Here, the incident light corresponds to the lattice positions arranged in a lattice shape on the virtual spherical surface. Further, it is assumed that the illumination L has no directivity and emits light rays in all directions from the lattice position. That is, the illumination L is assumed to have no bias in light distribution characteristics such as a spotlight.

  From the above conditions, the reflection coefficient calculation unit 145 can calculate the reflection characteristic information by performing an optimization process to which the reflection model is applied. Here, the reflection coefficient calculation unit 145 calculates the specular reflection coefficient, the diffuse reflection coefficient, and the order of the Cos function as the reflection characteristic information, for example, by the Markert method to which the Phong model is applied.

  Note that the reflection coefficient calculation unit 145 is not limited to a reflection model or an optimization method. For example, the reflection coefficient calculation unit 145 can use a reflection model such as BRDF (Bidirectional Reflectance Distribution Function) or an optimization method such as a Newton method, a Powell method, or a conjugate gradient method.

  The parameter setting unit 150 receives necessary setting parameters for the image processing information generation apparatus 100 and outputs the input setting parameters to each unit of the image processing information generation apparatus 100. Here, for example, since a predetermined value to be added to or subtracted from the depth extraction range described later is input as the setting parameter, the parameter setting unit 150 outputs this predetermined value to the depth extraction range calculation unit 160.

The depth extraction range calculation unit 160 receives the captured image 2 from the camera C, receives a predetermined value from the parameter setting unit 150, and detects a face region from the captured image 2 by face region detection processing. Then, the depth extraction range calculation unit 160 calculates the average value of the depth values of all the pixels included in the detected face area, and calculates the depth extraction range by adding or subtracting a predetermined value to the calculated average value. More specifically, the depth extraction range calculation unit 160 sets a value obtained by adding a predetermined value to the calculated average value as an upper limit of the depth extraction range, and calculates a value obtained by subtracting the predetermined value from the calculated average value. Calculated as the lower limit of. Thereafter, the depth extraction range calculation unit 160 outputs the calculated depth extraction range to the subject region extraction unit 170.
Note that the depth extraction range calculation unit 160 can use a method of extracting a skin color region, for example, as face region detection processing.

  The subject region extraction unit 170 receives the captured image 2 from the camera C, receives the depth extraction range from the depth extraction range calculation unit 160, and refers to the depth information stored in the depth information storage unit 130. Thus, an area whose depth value is included in the depth extraction range is extracted as a subject area.

  That is, the subject area extraction unit 170 determines whether or not a depth value corresponding to a certain pixel of the captured image 2 is included in the depth extraction range. Then, when the depth value is included in the depth extraction range, the subject region extraction unit 170 extracts the pixel as a pixel of the subject region. The subject area extraction unit 170 repeats this process for all the pixels of the captured image 2 to extract and output the subject area.

  As described above, the subject region extraction unit 170 extracts a subject placed at a specific position in the captured image 2 by selecting a depth value region in the depth extraction range in the captured image 2 from the depth value. Is possible. This extraction of the subject area is a process that is highly needed in video production, particularly video synthesis.

  Note that the subject region extraction unit 170 is not particularly limited in the subject region extraction method. For example, the image processing information generation apparatus 100 includes two variable resistance knobs (not shown) for inputting an upper limit and a lower limit of the depth value. Then, the subject area extraction unit 170 may manually set the upper limit and the lower limit of the depth value by operating these variable resistance knobs.

  Note that the depth extraction range calculation unit 160 and the subject region extraction unit 170 perform processing on the captured image 2, but may perform processing on the captured image 1. In this case, which of the captured images 1 and 2 is the target may be input to the parameter setting unit 150 as a parameter.

  In addition, although this embodiment demonstrated the illumination L as a light source, this invention is not limited to this. That is, in the present invention, the light source information may be obtained using natural light (for example, sunlight) other than the illumination L as a light source.

  In the present embodiment, it has been described that the camera C captures a moving image and the image processing information generation apparatus 100 processes each frame image of the moving image. However, the present invention is not limited to this. That is, the camera C may capture a still image, and the image processing information generation apparatus 100 may process the still image.

[Operation of Image Processing Information Generating Device]
The operation of the image processing information generation apparatus 100 in FIG. 4 will be described with reference to FIG. 11 (see FIG. 4 as appropriate).
Here, it is assumed that captured images 1 and 2 are input to the image processing information generation apparatus 100.

  In the image processing information generation device 100, the parallax amount calculation unit 101 calculates the parallax amount for each corresponding pixel (step S1). Further, the image processing information generation apparatus 100 converts the parallax amount into a depth value for each corresponding pixel using a conversion formula by the depth value conversion unit 110, and calculates the normal direction of the subject surface (step S2).

  In the image processing information generation apparatus 100, the camera parameter calculation unit 120 performs optimization processing such as a least square method to calculate the position and direction of the camera C as camera parameters (step S3).

  In the image processing information generation apparatus 100, the diffuse reflected light component image generation unit 141 selects the smaller luminance value of the corresponding pixels in the captured images 1 and 2 as the minimum luminance value, and the minimum luminance for each corresponding pixel A diffuse reflected light component image indicating the value is generated (step S4).

  In the image processing information generation apparatus 100, the specular reflection light component image generation unit 142 selects the larger luminance value of the corresponding pixels in the captured images 1 and 2 as the maximum luminance value, and the maximum luminance value for each corresponding pixel. A specular reflection light component image indicating the difference between the minimum luminance value and the minimum luminance value is generated (step S5).

  In the image processing information generation apparatus 100, the illumination direction calculation unit 143 calculates the incident angle of the reflected light from the subject from the normal direction of the subject surface and the position of the camera C, and the direction of the calculated incident angle is the direction of illumination. (Step S6).

  In the image processing information generation device 100, the illumination information generation unit 144 calculates the light amount of the illumination L based on the specular reflection component image and the direction of the illumination L, and generates the calculated light amount and direction of the illumination L as illumination information. (Step S7).

  In the image processing information generation device 100, the reflection coefficient calculation unit 145 performs an optimization process that applies a reflection model to the illumination information, the specular reflection light component image, and the diffuse reflection light component image, and the specular reflection coefficient. Then, the diffuse reflection coefficient is calculated (step S8).

  The image processing information generating apparatus 100 detects a face area by the face area detection process from the captured image 2 by the depth extraction range calculation unit 160, calculates an average value of depth values of all pixels included in the detected face area, A depth extraction range is obtained by adding or subtracting a predetermined value to the calculated average value (step S9).

  In the image processing information generation device 100, the subject region extraction unit 170 refers to the depth information stored in the depth information storage unit 130 and extracts, from the captured image 2, a region whose depth value is included in the depth extraction range as a subject region. (Step S10).

  As described above, in the imaging apparatus 1 according to the embodiment of the present invention, the camera C is simple, and the depth information, the reflection characteristic information, and the illumination information are obtained using the captured images 1 and 2 captured by the camera C. Can be generated together. Therefore, the imaging device 1 does not require a unique device for generating each piece of information for each of depth information, reflection characteristic information, and light source information, and can simplify the configuration of the camera C. Furthermore, since the imaging apparatus 1 does not need to irradiate the subject with pattern light when imaging with the camera C, the imaging apparatus 1 can be used even in an environment with a lot of light disturbance.

  In the prior art, in order to acquire depth information and reflection characteristic information, it is necessary to use a large measuring device. However, the imaging apparatus 1 can reduce the size of the camera C to the same level as a home-use handy camera. In addition, even when it is difficult to acquire depth information that is highly accurate and has no omission, the imaging apparatus 1 can extract the subject area with the quality required for video production by the image processing information generation apparatus 100. Furthermore, the imaging device 1 is suitable for video production such as television programs and movies because the camera C itself can capture moving images with a simple configuration.

  Further, in the prior art, a great deal of labor and high skill are required to perform video production such as video synthesis and changing illumination conditions after imaging. These can only be expected to reduce labor to some extent even if imaging is performed assuming video production at the time of imaging. For this reason, although it is rare that a user performs video production and there is a need for video production, the motivation of video production is difficult due to the necessity of a great deal of labor and expertise. However, the imaging apparatus 1 can greatly save the generation processing of image processing information with the same level of effort as that of normal camera imaging, and can increase the degree of freedom of video production. As a result, the imaging apparatus 1 allows the user to perform desired video production without requiring specialized knowledge and great effort for video production. In other words, the imaging device 1 enables video production without depending on the skill and sense of the user.

  Although the imaging apparatus 1 according to the embodiment of the present invention has been described, the present invention is not limited to the above-described embodiment, and various modifications can be made. Hereinafter, the imaging device 1 according to the modification of the present invention will be described with respect to differences from the first embodiment.

(Modification: Lighting information expressed in absolute values)
As shown in FIG. 1, the imaging apparatus 1 performs control to repeatedly turn on and off the illumination L in accordance with the timing of imaging, and is expressed as a relative value using illumination information when the illumination is on and when the illumination is off. Convert the lighting information into absolute values.

Here, the illumination L is connected to the image processing information generation apparatus 100 via the signal cable _9L . The illumination L is repeatedly turned on and off in synchronization with the imaging timing of the camera C in accordance with a synchronization signal input from the illumination control unit 180 described later.

  The illumination control unit (light source control unit) 180 in FIG. 4 turns on and off the illumination L. The illumination control unit 180 outputs a synchronization signal that synchronizes lighting and extinguishing of the illumination L and imaging of the camera C to the illumination L and the camera C at regular intervals (for example, one frame second).

  That is, at the timing when this synchronization signal is input, the camera C performs imaging. In addition, at the timing when the synchronization signal is input, the illumination L is turned off if it is turned on, and turned on if it is turned off.

  In addition to the camera parameters, the reflection characteristic estimation unit 140 receives captured images 1 and 2 when the illumination is turned on and captured images 1 and 2 when the illumination is turned off from the camera C. Each means of the reflection characteristic estimation unit 140 performs the same processing as in the first embodiment on each of the captured images 1 and 2 when the illumination is turned on and when the illumination is turned off.

  Similarly to the first embodiment, the illumination information generation unit 144 includes illumination information when the illumination is turned on (light quantity and direction of the illumination L when the illumination is turned on), and illumination information when the illumination is turned off (the amount of light of the illumination L when the illumination is turned off and Direction). Then, the illumination information generation unit 144 calculates a scale (weight) from a light amount when the illumination is turned on and when the illumination is turned off using a predetermined calculation formula, and multiplies the calculated scale by the light amount when the illumination is turned off.

<Conversion of illumination information by illumination information generation unit>
Hereinafter, the conversion of the illumination information by the illumination information generation unit 144 will be specifically described.
Here, the actually measured light quantity value of the illumination L measured in advance is I, the distance from the illumination L to the subject A is Lg, the incident angle of the reflected light with respect to the normal of the subject surface is θ, and the reflected light is reflected on the subject surface. Let Kd be the reflection coefficient of the reflected portion. This incident angle θ is known because the illumination direction calculation unit 143 has already calculated it.

  Further, the distance (depth information) from the camera C to the subject A is known. Therefore, the imaging device 1 captures the illumination L as a subject and generates a distance (depth information) from the camera C to the illumination L, so that the positional relationship among the camera C, the subject A, and the illumination L becomes clear. The distance Lg can be obtained.

  In this case, the reflected light amount Ir is a difference between the luminance when the illumination is turned on and the luminance when the illumination is turned off, and can be expressed by the following equation (10).

Here, from the illumination information when the illumination is extinguished, the amount of light when the illumination to the reflecting portion is extinguished is _defined as I _omni . Further, from the illumination information at the time of lighting, the amount of light at the time of lighting the reflecting portion is set to I _total . In this case, for the diffuse reflected light component, the luminance I _on when the illumination is turned _on and the luminance I _off when the illumination is turned _off can be expressed by the following equation (11).

  Substituting equation (10) into equation (11) and dividing each term by the reflection coefficient Kd yields equation (12) below.

  Here, if the normal vector VA orthogonal to the subject surface is (ax, ay, az) and the straight vector VB from the center of the lens 11 to the reflection part is (bx, by, bz), the incident angle θ Can be expressed by the following equation (14) from equation (13), which is the inner product of the vectors VA and VB. Note that sqr is a function that returns a square root.

As described above, the illumination information generation unit 144 generates illumination information represented by a relative value. For this reason, some scale (weight) s is applied to the light _amounts I _omni and I _total in Expression (11). That is, Formula (12) is represented by the following calculation formula (15).

Here, the light _amounts I _omni and I _total , the light amount actual measurement value I, the incident angle θ, and the distance Lg are known. Therefore, the illumination information generation unit 144 can obtain the value of the scale s by solving the calculation formula (15). The illumination information generation unit 144 can convert the illumination information represented by a relative value into an absolute value by multiplying the light amount I _omni by the scale s.

At this time, since there is a time difference between lighting and extinguishing of the illumination L, a positional deviation may occur between the captured images 1 and 2 when the illumination is lit and the captured images 1 and 2 when the illumination is extinguished. Therefore, the illumination information generation unit 144 can correct this misalignment by performing block matching or optical flow on the captured images 1 and 2 when the illumination is turned on and the captured images 1 and 2 when the illumination is turned off. preferable.
In this modification, for example, it is assumed that there is no position shift because the time difference is as small as about 1 frame second (that is, 1/30 second).

Moreover, when using the illumination information of a relative value, since the illumination L and the illumination control part 180 are unnecessary, it is preferable to set it as the structure which can select the attachment or detachment of the illumination L, or the presence or absence of the process by the illumination information generation part 144.
In addition, since it is the same as that of 1st Embodiment except the illumination information generation part 144 and the illumination control part 180, detailed description was abbreviate | omitted.

  As described above, the imaging device 1 according to the modification of the present invention can convert illumination information into an absolute value. As a result, the imaging apparatus 1 can be used in an environment with a lot of light disturbances as in the above-described embodiment, and a simple configuration can be realized. Furthermore, since the illumination information is represented by an absolute value, the imaging apparatus 1 can know the actual brightness of the illumination L from the illumination information. Furthermore, in the imaging device 1, when performing image processing in another imaging environment using the illumination information, it is not necessary to convert the illumination information for the other imaging environment. That is, the imaging apparatus 1 can easily maintain the consistency of the illumination information even when the illumination information is shared between different imaging environments.

In the embodiment, the image processing information generation apparatus of the imaging apparatus according to the present invention has been described as an independent apparatus. However, in the present invention, a general computer is operated by a program that functions as each unit of the image processing information generation apparatus. It can also be operated. This program may be distributed via a communication line, or may be distributed by writing in a recording medium such as a CD-ROM or a flash memory.

1 Imaging device 11 Lens (light transmission element)
11a, 11b Polarizing filter (polarizing element)
12 Polarized BS (spectral element)
13 First imaging device 14 Second imaging device 100 Image processing information generation device 101 Parallax amount calculation unit 110 Depth value conversion unit 120 Camera parameter calculation unit (imaging means position calculation unit)
130 depth information storage unit 140 reflection characteristic estimation unit 141 diffuse reflection light component image generation unit 142 specular reflection light component image generation unit 143 illumination direction calculation unit (light source direction calculation unit)
144 Illumination information generation part (light source information generation part)
145 reflection coefficient calculation unit 150 parameter setting unit 160 depth extraction range calculation unit 170 subject region extraction unit 180 illumination control unit C camera (imaging means)
L Illumination (light source)

Claims (5)

An imaging device that images a subject illuminated by a light source and generates depth information indicating a depth value of the subject, reflection characteristic information indicating a reflection characteristic of the subject, and light source information indicating a direction and a light amount of the light source. There,
A light transmitting element composed of two polarizing elements whose polarization directions are orthogonal to each other; a spectroscopic element that transmits incident light from one of the polarizing elements and reflects incident light from the other of the polarizing elements; and the spectroscopic element An imaging means comprising: a first imaging element that images incident light transmitted through the element; and a second imaging element that images incident light reflected by the spectroscopic element;
A parallax amount calculating unit that calculates a parallax amount for each corresponding pixel corresponding to the two captured images from two captured images obtained by capturing the subject with the first image sensor and the second image sensor;
The depth information is generated by converting the parallax amount calculated by the parallax amount calculation unit into a depth value for each corresponding pixel using a preset conversion formula, and the depth value between the target pixel and a neighboring pixel of the target pixel A depth value conversion unit that calculates a normal direction perpendicular to the subject surface represented by:
An imaging unit position calculation unit that performs optimization processing on the depth value calculated by the depth value conversion unit and calculates a position of the imaging unit;
A diffuse reflected light component image that generates a diffuse reflected light component image indicating the minimum brightness value for each of the corresponding pixels by selecting the smaller of the corresponding pixels in the two captured images as the minimum brightness value A generator,
A specular reflection component image indicating a difference between the maximum luminance value and the minimum luminance value for each of the corresponding pixels is selected as the maximum luminance value of the corresponding pixels in the two captured images. A specular reflection component image generation unit to generate,
A light source direction that calculates an incident angle of reflected light from the subject from the normal direction calculated by the depth value conversion unit and the position of the imaging unit, and calculates the direction indicated by the calculated incident angle as the direction of the light source A calculation unit;
Light source information generation for calculating the light amount of the light source based on the specular light component image generated by the specular light component image generation unit and the direction of the light source, and generating the calculated light amount and direction of the light source as the light source information And
The light source information generated by the light source information generation unit, the specular reflection light component image, and the diffuse reflection light component image are subjected to optimization processing using a reflection model, and the specular reflection coefficient and the diffuse reflection coefficient are calculated. A reflection coefficient calculation unit for calculating the reflection characteristic information;
An imaging apparatus comprising: A light source controller for turning on and off the light source;
The light source information generation unit
The light amount of the light source is calculated when the light source is turned on and the light source is turned off, the weight is calculated from the light amount when the light source is turned on and the light source is turned off by a predetermined calculation formula, and the calculated light weight is multiplied by the calculated light amount The imaging apparatus according to claim 1, wherein the light source information represented by an absolute value is calculated.   The subject area extracting unit further extracts, as a subject area, an area included in a depth extraction range in which a depth value of the corresponding pixel is set in advance from one of the two captured images. Item 3. The imaging device according to Item 2.   A face area is detected from one of the two captured images by a face area detection process, an average value of depth values of all pixels included in the detected face area is calculated, and a predetermined value is added to or subtracted from the calculated average value. The imaging apparatus according to claim 3, further comprising a depth extraction range calculation unit that sets the depth extraction range. A light transmitting element composed of two polarizing elements whose polarization directions are orthogonal to each other; a spectroscopic element that transmits incident light from one of the polarizing elements and reflects incident light from the other of the polarizing elements; and the spectroscopic element The subject illuminated by a light source is imaged by an imaging means comprising a first imaging element that images incident light that has passed through the element and a second imaging element that images incident light reflected by the spectroscopic element, and the subject In order to generate depth information indicating the depth value of the object, reflection characteristic information indicating the reflection characteristic of the subject, and light source information indicating the direction and light amount of the light source,
A parallax amount calculation unit that calculates a parallax amount for each corresponding pixel corresponding to the two captured images from two captured images obtained by capturing the subject with the first imaging element and the second imaging element;
The depth information is generated by converting the parallax amount calculated by the parallax amount calculation unit into a depth value for each corresponding pixel using a preset conversion formula, and the depth value between the target pixel and a neighboring pixel of the target pixel Depth value conversion unit for calculating a normal direction orthogonal to the subject surface represented by
An imaging unit position calculation unit that performs optimization processing on the depth value calculated by the depth value conversion unit and calculates a position of the imaging unit;
A diffuse reflected light component image that generates a diffuse reflected light component image indicating the minimum brightness value for each of the corresponding pixels by selecting the smaller of the corresponding pixels in the two captured images as the minimum brightness value Generator,
A specular reflection component image indicating a difference between the maximum luminance value and the minimum luminance value for each of the corresponding pixels is selected as the maximum luminance value of the corresponding pixels in the two captured images. Specular reflection component image generation unit to generate,
A light source direction that calculates an incident angle of reflected light from the subject from the normal direction calculated by the depth value conversion unit and the position of the imaging unit, and calculates the direction indicated by the calculated incident angle as the direction of the light source Calculation part,
Light source information generation for calculating the light amount of the light source based on the specular light component image generated by the specular light component image generation unit and the direction of the light source, and generating the calculated light amount and direction of the light source as the light source information Part,
The light source information generated by the light source information generation unit, the specular reflection light component image, and the diffuse reflection light component image are subjected to optimization processing using a reflection model, and the specular reflection coefficient and the diffuse reflection coefficient are calculated. A reflection coefficient calculation unit for calculating the reflection characteristic information;
An image processing information generation program for functioning as

Images