JP5541653B2 - Imaging apparatus and control method thereof - Google Patents

Description

  The present invention relates to an imaging apparatus that generates a distance image of an imaging target and a control method thereof.

  In the following description, an image configured with a distance to an imaging target as a pixel value is referred to as a “distance image”. Conventionally, a plurality of methods are known as a method of an imaging device that measures the distance image. Here, an imaging device that does not require a scanning operation at the time of measurement and acquires a distance image by one shooting is provided. The method will be described.

  Various methods have been proposed as an imaging device for obtaining a distance image. For example, a method using a time of flight (TOF) until an irradiated light beam is reflected by an imaging object and returned, or a plurality of images can be obtained. There is a method for obtaining the depth by the stereo method from the images. In addition, a pattern light such as a modulated wave is projected to obtain the depth, or the light collected by the main optical system is separated by the sub-optical system to form two images and focused from the respective parallaxes. There is a method called a distance measuring sensor that detects a state. In this distance sensor system, information on relative depth is acquired with reference to the focal plane.

  Polarized light refers to a state where the vibration direction of light is biased in a specific direction. And the polarization component of light can be measured by using the polarization optical system which permeate | transmits only the light of a specific vibration direction. In addition, it is not possible to directly estimate the distance information to the object (imaging target) based on the polarization information, but it is possible to know the azimuth of the normal of the object surface by analyzing the polarization state and obtaining the polarization azimuth angle. it can. In addition, if the degree of polarization is obtained, the slope of the normal can be estimated. It is also possible to distinguish the polarization component from the non-polarization component and remove the reflected light component to improve the measurement accuracy of the distance image.

  A technique for estimating the shape of the observation object by such polarization analysis is called “Shape from polarization”, and a plurality of related techniques have been proposed. For example, the following Non-Patent Document 1 discloses a technique for estimating a shape by performing polarization analysis on a glossy metal surface. In order to perform polarization analysis, conventionally, as in the method shown in Non-Patent Document 1, a linear polarization filter is disposed in front of the imaging optical system, and the direction of polarization is mechanically rotated to specify a specific direction. It was necessary to transmit only a polarization component and to photograph a plurality of images. In another prior art, a liquid crystal polarizing filter is also used. In this case, the direction of polarization is electrically changed. In any case, these methods require a plurality of imaging operations and are difficult to use in combination with other optical measurement methods in terms of real-time characteristics.

  On the other hand, Patent Document 1 below discloses an apparatus that can perform polarization analysis from only one captured image. For example, as shown in FIG. 3 to be described later, the imaging apparatus described in Patent Document 1 includes a light receiving element 123 and a polarization optical system 122 including an array of polarizers 301 equal in number to the pixels of the light receiving element 123. . In the polarizing optical system 122, polarizers (301a to 301d) having different polarization directions are regularly arranged (only a part of the polarizers is illustrated in FIG. 3). In addition, light of different polarization components is recorded in the pixels 302a to 302d of the light receiving element 123 corresponding to the polarizers 301a to 301d (in FIG. 3, only a part of the light receiving elements is illustrated. Shown). If imaging is performed with such an imaging apparatus, information is read from only predetermined pixels and an image is reconstructed, an image of any polarization component in four directions can be obtained. By using such a configuration, it is possible to acquire data of polarization components in a plurality of directions necessary for polarization analysis by one imaging.

JP 2007-86720 A JP 2006-46960 A JP 55-155331 A

O. Morel, F. Meriaudeau, C. Stolz and P. Gorria, Polarization imaging applied to 3D inspection of specular metallic surfaces, Machine Vision Applications in Industrial Inspection XIII, Proceedings of SPIE, vol. 5679, pp. 178-186, 2005 . M. Bass edit., Handbook of optics Vol.II, Optical Society of America, McGraw-Hill, New York, 1995. B. K. P. Horn and M. J. Brooks, edit., Shape from Shading, The MIT Press, Cambridge, Massachusetts, 1989. S. Geman and D. Geman, Stochastic relaxation, Gibbs distribution, and the Bayesian restoration of images, IEEE Trans. On PAMI, Vol.6, No.6, pp.721-741, 1985. Y. Boykov and G. Funka-Lea.Graph Cuts and Efficient N-D Image Segmentation.International Journal of Computer Vision, vol. 70, no. 2, pp. 109-131, 2006. C. Tomasi and R. Manduchi, "Bilateral Filtering for Gray and Color Images", Proceedings of the 1998 IEEE International Conference on Computer Vision, Bombay, India, 1998. P. J. Rousseeuw and A. M. Leroy. Robust Regression and Outlier Detection. John Wiley & Sons, New York, 1987. R.J.Rousseeuw, Least median of squares regression, J. American Stat.Assoc., Vol.79, pp.871-880, 1984. de Bruijn, N. G. "A Combinatorial Problem." Koninklijke Nederlandse Akademie v. Wetenschappen vol. 49, pp. 758-764, 1946.

  However, in some conventional distance image capturing methods, the measurement accuracy may deteriorate under certain conditions due to the measurement principle.

  For example, in a method using TOF, near infrared light or the like is often used as irradiation light, but spatial resolution may be reduced due to diffuse reflection or multiple reflection. In particular, the accuracy may decrease due to multiple reflections between the measurement object and the floor surface, and this phenomenon is prominent when measured close to the measurement object. In addition, external light contamination and reflection in the environment is a disturbance factor.

  The stereo method is a method for obtaining a distance by the principle of triangulation by obtaining regions and corresponding points having high correlation between images. For this reason, the distance can be measured only for a region where a point or line that is a clue to correspondence exists, and the distance cannot be measured in a region where the luminance distribution does not change or changes slowly. In addition, although the uneven surface of the object may be shielded by parallax, the distance of the shielded area cannot be measured. In the stereo method, the measurement value may be lost due to such a cause, and it is necessary to supplement the region where the loss has occurred by some method.

  Further, since the method using pattern light is based on the principle of triangulation like the stereo method, the distance cannot be measured in an area that is shaded by the pattern light. In addition, there is a method that uses a modulated wave whose intensity is spatially changed as the pattern light. However, when a pattern having a different reflectance with respect to the pattern light exists on the object surface, this method may reduce the measurement accuracy.

  Further, in the method using the distance measuring sensor, the characteristic is similar to the method of the stereo method, and the measurement accuracy is low in the low contrast region.

  In addition, as a characteristic common to any of the above four methods, it is difficult to measure a surface having a zero-order reflected light component such as a metal surface or a glossy surface.

  In order to compensate for the decrease in accuracy as described above, it is possible to perform imaging using a plurality of distance measurement methods, complement the measurement results, and complementarily improve the accuracy. However, since the accuracy decreases with respect to the 0th-order reflected light component among the systems, the disturbance is effectively effective for increasing the disadvantages such as the size and complexity of the apparatus due to the combination of the systems. May not be removed.

  The present invention has been made in view of such problems. When generating a distance image representing the distance to the imaging target, the present invention is caused by disturbances such as light reflection and gloss of the imaging target, measurement principles, and the like. An object of the present invention is to provide an imaging apparatus that realizes reduction of an error in the distance to be transmitted and a control method thereof.

An imaging apparatus of the present invention includes a plurality of polarizers having different polarization directions, and receives a polarization optical system that polarizes light from an imaging target and each light polarized by the plurality of polarizers having different polarization directions. A distance measuring element for measuring the distance to the imaging target, a polarization analyzing element for receiving polarization polarized light by the polarizer of a predetermined polarization direction, and performing polarization analysis, Based on the light received by the distance measuring element, distance information acquisition means for acquiring distance information to the imaging target, and by performing the polarization analysis based on the polarization information of the light received by the polarization analysis element, Surface shape information acquisition means for acquiring surface shape information of the imaging target, and distance image generation means for generating a distance image of the imaging target based on the distance information and the surface shape information .

An imaging apparatus control method according to the present invention includes a polarizing optical system that includes a plurality of polarizers having different polarization directions and polarizes light from an imaging target, and each of the lights polarized by the plurality of polarizers having different polarization directions. A device for measuring the distance to the object to be imaged, and a device for polarization analysis for performing polarization analysis by receiving light polarized by the polarizer in a predetermined polarization direction a method for controlling an imaging apparatus including bets, based on the light the distance measuring device has received, a distance information acquiring step of acquiring the distance information to the imaging target, light the ellipsometer element has received of by performing the ellipsometry based on the polarization information, and the surface shape information acquiring step of acquiring the surface shape information of the imaging target, based on the distance information and the surface shape information, the imaging target distance image And a range image generating step of generating.

  According to the present invention, when generating a distance image representing a distance to an imaging target, it is possible to reduce a distance error caused by disturbances such as light reflection of the imaging target and its gloss, measurement principles, and the like. .

1 is a schematic diagram illustrating an outline of the present invention and an example of a schematic configuration of a three-dimensional image capturing apparatus. It is a schematic diagram which shows an example of schematic structure of the three-dimensional image imaging device which concerns on the 1st Embodiment of this invention. It is a schematic diagram which shows an example of a detailed structure of the polarization optical system and the 2nd light receiving element inside the 2nd imaging means shown in FIG. It is a schematic diagram which shows the other example of schematic structure of the three-dimensional image imaging device which concerns on the 1st Embodiment of this invention. FIG. 5 is a schematic diagram illustrating an example of a detailed configuration of a polarization optical system and a light receiving element inside the first imaging unit and the second imaging unit illustrated in FIG. 4. It is a schematic diagram which shows an example of the azimuth and inclination of the normal of the object surface of an imaging target. It is a schematic diagram which shows the example of an observation of a polarization azimuth and a polarization degree about a three-dimensional object. It is a schematic diagram which shows an example of the result of having produced | generated the distance image based on the measured distance information and ellipsometry information. It is a schematic diagram which shows an example of the inclination of the normal line of the object surface of an imaging target. In Formula (5), it is a schematic diagram which shows an example of the relationship between the polarization degree p when n = 1.6 and k = 5, and the inclination φ of the normal of the object surface. It is a flowchart which shows the 1st Embodiment of this invention and shows an example of the process sequence of the polarization analysis process by a polarization analysis means. It is a flowchart which shows the 1st Embodiment of this invention and shows an example of the process sequence of an area | region division process. It is a flowchart which shows the 1st Embodiment of this invention and shows an example of the process sequence of the integration process of the distance information and surface shape information by a distance image generation means. It is a flowchart which shows the 1st Embodiment of this invention and shows the other example of the process sequence of the integration process of the distance information and surface shape information by a distance image generation means. It is a schematic diagram which shows an example of schematic structure of the three-dimensional image imaging device which concerns on the 2nd Embodiment of this invention. FIG. 9 is a schematic diagram illustrating the principle of triangulation distance calculation using pattern light in the distance calculation unit according to the second embodiment of this invention. It is a schematic diagram which shows the 2nd Embodiment of this invention and shows an example of the missing area | region in the case of the distance calculation by a pattern light in a distance calculation means. It is a flowchart which shows the 2nd Embodiment of this invention and shows an example of the process sequence of the integration process of the distance information and surface shape information by a distance image generation means. It is a schematic diagram which shows an example of schematic structure of the three-dimensional image imaging device which concerns on the 3rd Embodiment of this invention. It is a schematic diagram which shows the 3rd Embodiment of this invention and shows an example of the principle of a ranging sensor. It is a schematic diagram which shows an example of schematic structure of the three-dimensional image imaging device which concerns on the 4th Embodiment of this invention. It is a schematic diagram which shows the other example of schematic structure of the three-dimensional image imaging device which concerns on the 4th Embodiment of this invention.

  The best mode (embodiment) for carrying out the present invention will be described below with reference to the drawings. In the following description, an example in which, for example, a three-dimensional image imaging device that generates a distance image for performing three-dimensional distance measurement is applied as the imaging device according to the present invention will be described.

-Outline of the present invention-
First, before describing specific embodiments of the present invention, an overview of the present invention will be described.

FIG. 1 is a schematic diagram illustrating an outline of the present invention and an example of a schematic configuration of a three-dimensional image capturing apparatus 100.
A three-dimensional image capturing apparatus 100 shown in FIG. 1 includes a first image capturing unit 110, a second image capturing unit 120, a distance calculating unit 130, a polarization analyzing unit 140, and a distance image generating unit 150. Has been.

  The first imaging unit 110 includes a first imaging optical system 111 that forms an image of the input light 101a from the imaging target, a ranging optical system 112, and a first light receiving element 113. An image for distance measurement is taken.

  The second imaging unit 120 includes a second imaging optical system 121 that forms an image of the input light 101b from the imaging target, a polarization optical system 122, and a second light receiving element 123, and includes a polarization component. An image for measurement is taken.

  The distance calculating unit 130 calculates distance information 103 related to the distance to the imaging target from the measurement data 102 obtained from the image captured by the first imaging unit 110.

  The polarization analysis unit 140 analyzes the polarization component data 104 obtained from the image captured by the second imaging unit 120 and outputs the analysis result as polarization analysis information 105. Here, the polarization component data 104 refers to image data polarized in a specific direction obtained by passing through a polarizer (for example, 301 in FIG. 3 described later). The polarization analysis information 105 refers to secondary information related to polarization such as the degree of polarization and the polarization azimuth obtained by analyzing the polarization component data 104.

  The distance image generation unit 150 generates a distance image 106 representing the distance to the imaging target based on the distance information 103 calculated by the distance calculation unit 130 and the polarization analysis information 105 output from the polarization analysis unit 140.

  Further, in this specification, for convenience, information obtained by a conventional distance measuring means is called distance information, and information obtained by integrating and estimating distance information and ellipsometry information is called a distance image. Distinguish.

  As described above, in the three-dimensional image capturing apparatus 100 according to the present invention, the distance information 103 obtained from the distance measurement image captured by the first image capturing unit 110 and the second image capturing unit 120 are captured. The ellipsometric information 105 obtained from the image for measuring the polarization component is used. That is, in the present invention, by integrating two pieces of information having different optical characteristics, that is, the distance information 103 and the ellipsometric information 105, the missing region where the measurement value is missing or the region where the error due to the disturbance is generated is displayed on the other side. It complements with information. As a result, the 3D image capturing apparatus 100 according to the present invention can generate a distance image in which measurement errors of the conventional distance measuring method are reduced, for example, the measurement accuracy of the glossy surface to be imaged is improved.

  That is, according to the three-dimensional image capturing apparatus 100 according to the present invention, when generating a distance image representing the distance to the imaging target, the distance due to the light reflection of the imaging target, disturbance such as its gloss, the measurement principle, etc. The reduction of the error can be realized.

  Further, as the first imaging unit 110, for example, an image that uses an image of time of flight (TOF) of light, an image that uses pattern light, or an image related to distance information by a stereo method, etc. Can be applied. The first imaging unit 110 is not limited to this mode, and any system can be applied as long as it captures an image for distance measurement. The first imaging unit 110 may be a passive measurement method or an active measurement method.

  In addition, the first light receiving element 113 and the second light receiving element 123 can be mixed on a panel of one light receiving element. It is also possible to alternately perform distance measurement and polarization component measurement using one type of light receiving element. Furthermore, distance measurement and polarization component measurement can be simultaneously performed using one type of light receiving element.

  Further, as the polarization analysis information 105, for example, information on the polarization azimuth angle and polarization degree of linearly polarized light can be used. Note that the polarization analysis information 105 is not limited to this linearly polarized information, and may be an embodiment using, for example, circularly polarized component information. In the three-dimensional image capturing apparatus 100 according to the present invention, the polarization azimuth angle coincides with the direction of the normal of the object surface to be imaged, and the degree of polarization depends on the inclination of the normal of the object surface to be imaged. The surface shape of the imaging target is restored from the two attributes. Then, when restoring the surface shape of the imaging target, for example, by using distance information, it is possible to eliminate the ambiguity of the polarization azimuth and uniquely determine the polarization azimuth.

  Further, the polarization analysis information 105 is used as information on the polarization component and the non-polarization component, and the two components are separated to remove the reflection or reflected light component on the surface of the imaging target. For example, the distance image generation unit 150 compares the distance information 103 with the polarization analysis information 105 (for example, surface shape information of the imaging target) obtained by analyzing the polarization component data 104. For example, the distance image generation unit 150 causes discontinuity of distribution in one of the distance information 103 and the ellipsometric information 105, and removes a component that does not affect the other as a disturbance component.

  Further, for example, the distance image generating unit 150 analyzes the polarization analysis data 105 (for example, the surface shape of the imaging target) by analyzing the missing portion of the distance information 103 (for example, the missing region 1701 shown in FIG. 17) and the polarization component data 104. Information).

  For example, the distance image generation unit 150 generates the distance image 106 using the distance information 103 and the polarization analysis information 105 (for example, surface shape information of the imaging target) obtained by analyzing the polarization component data 104. At this time, for example, the distance image 106 is generated by minimizing the energy function using the continuity of the distance information 103 and the ellipsometry information 105 (for example, surface shape information of the imaging target) as a regular term.

  In addition, for example, the distance image generation unit 150 analyzes the relative polarization of the imaging target so that the polarization analysis information 105 (for example, surface shape information of the imaging target) obtained by analyzing the polarization component data 104 matches the distance information 103. Estimate the height. At this time, for example, the ellipsometric information 105 (for example, surface shape information of the imaging target) is used to divide the image into a plurality of regions so that the surface shape information matches the distance information 103 for each region. The relative height of the imaging target is estimated.

  Further, the distance image generation unit 150 integrates the distance information 103 and the polarization analysis information 105 (for example, surface shape information of the imaging target) obtained by analyzing the polarization component data 104 by weighting the reliability. Thus, the distance image 106 is generated. Further, the distance image generation unit 150 estimates the refractive index of the imaging target based on the distance information 103 and the polarization analysis information 105 (for example, surface shape information of the imaging target) obtained by analyzing the polarization component data 104. Also do.

  Further, the three-dimensional image capturing apparatus 100 according to the present invention is configured to include a recording unit (not shown in FIG. 1) that records the distance image 106 as a moving image as a measurement result by the distance image generation unit 150. ing.

-Specific embodiments based on the outline of the present invention-
Next, specific embodiments based on the outline of the present invention described above will be described.

(First embodiment)
First, a first embodiment of the present invention will be described.
FIG. 2 is a schematic diagram illustrating an example of a schematic configuration of the three-dimensional image capturing apparatus 100 according to the first embodiment of the present invention. In addition, the same code | symbol is attached | subjected about the structure similar to the structure shown in FIG. 1, and the 3D image imaging device 100 shown in FIG. 2 is described as 3D image imaging device 100-1A in the following description.

  A three-dimensional image capturing apparatus 100-1A shown in FIG. 2 includes a first image capturing unit 210, a second image capturing unit 220, a distance calculating unit 130, a polarization analyzing unit 140, a shape restoring unit 230, and a distance image. The generation unit 150 is included.

  The first imaging unit 210 includes an imaging optical system 211, a beam splitter 212, a filter 213, a first light receiving element 113, a light projecting unit 214, and a time-of-flight measuring unit 215. An image for distance measurement is taken.

  The second imaging means 220 includes an imaging optical system 211, a beam splitter 212, a filter 221, a polarization optical system 122, and a second light receiving element 123, and is used for measuring polarization components. This image is taken.

  As described above, in the first imaging unit 210 and the second imaging unit 220 shown in FIG. 2, the imaging optical system 211 and the beam splitter 212 are configured in common. That is, the three-dimensional image pickup apparatus 100-1A shown in FIG. 2 is configured such that the first imaging optical system 111 and the second imaging optical system 121 shown in FIG. The optical axes are the same.

  As the first imaging means 210 for distance measurement, for example, the same one as the TOF method generally used in Patent Document 2 is used.

  In the three-dimensional image capturing apparatus 100-1A shown in FIG. 2, the flow until the distance image 106 is generated is as follows.

  First, the modulated irradiation light 201 is projected from the light projecting unit 214 of the first imaging unit 210 to the imaging target. Then, the input lights a and b (101a and 101b) reflected and returned from the imaging target are collected by the imaging optical system 211, divided by the beam splitter 212, and a part of the light is used for distance measurement. The first imaging means 210 is guided inside.

  The light guided to the inside of the first imaging means 210 is passed through a filter 213 that allows only the band of the irradiation light 201 to pass through, and then forms an image on the first light receiving element 113. The time-of-flight measurement unit 215 compares the reference waveform 202 from the light projecting unit 214 with the modulation waveform 203 of the light received by the first light receiving element 113 for each pixel of the first light receiving element 113, and the irradiation light 201. Time-of-flight information 204 from when the camera reaches the imaging target and returns to the imaging apparatus is calculated. Thereafter, the distance calculation unit 130 calculates distance information 103 related to the distance to the imaging target from the flight time information 204.

  The wavelength of the irradiation light 201 is preferably near-infrared light that is sensitive to a commonly used imaging device (light receiving device) and is not sensitive to humans. However, other wavelengths or laser light may be used. A combination of these wavelengths may also be used. As a modulation method, light whose intensity is modulated may be used, frequency-modulated light, or pulse-modulated light may be used.

  The beam splitter 212 guides a certain proportion of light into the second imaging unit 220 for measuring the polarization component. Here, it is desirable that a non-polarizing beam splitter is used as the beam splitter 212 and the polarization state does not change before and after transmission.

  The light guided to the inside of the second imaging means 220 is blocked by the filter 221 in the band of the input light a (101a) derived from the irradiation light 201, and the input light b (101b) mainly derived from the ambient light. Passes through the filter 221.

  FIG. 3 is a schematic diagram illustrating an example of a detailed configuration of the polarization optical system 122 and the second light receiving element 123 inside the second imaging unit 220 illustrated in FIG. 2.

  As shown in FIG. 3, a plurality of linearly polarized light polarizers 301, which are minute regions having different polarization directions, are arranged in an array on the polarizing optical system 122. In the example shown in FIG. 3, the four pixels (302a to 303d) of the light receiving element 123 are set as one block unit, and four types having different polarization directions of 0 degree, 45 degrees, 90 degrees, and 135 degrees are provided for each block unit. Linear polarizers 301a to 301d are provided. In this way, by reading out signals only from the pixels 302 corresponding to the polarizer 301 in a specific direction, it is possible to obtain an image of any one of the four types of polarization components. Note that the rules for the arrangement of the polarizer array in the polarization optical system 122 are not limited to those shown in FIG. 3, and other arrangements or no regularity may be used. There is an effect of preventing the occurrence of interference fringes in the polarization component image (polarized image). In addition, in order to perform polarization analysis, which will be described later, in general, there may be different types of polarizers 301 in three or more directions.

  Subsequently, the polarization analysis means 140 performs a polarization state analysis process based on the polarization component data 104 obtained in this way by the second imaging means 220 and outputs the analysis result as polarization analysis information 105. . The shape restoration unit 230 performs a restoration process of the surface shape of the imaging target based on the polarization analysis information 105 output from the polarization analysis unit 140, and outputs the restoration process result as the surface shape information 205. Then, the distance image generating unit 150 integrates these based on the distance information 103 calculated by the distance calculating unit 130 and the surface shape information 205 output from the shape restoring unit 230, and calculates the distance from the imaging target. A distance image 106 is generated.

  FIG. 4 is a schematic diagram illustrating another example of the schematic configuration of the three-dimensional image capturing apparatus 100 according to the first embodiment of the present invention. 1 and 2 are denoted by the same reference numerals, and in the following description, the three-dimensional image pickup device 100 shown in FIG. 4 is described as the three-dimensional image pickup device 100-1B. To do.

  In the three-dimensional image pickup apparatus 100-1B shown in FIG. 4, the first image pickup means 410 for distance measurement and the second image pickup for polarization component measurement are provided without providing the beam splitter 212 (and further the filter) shown in FIG. The optical path to the means 420 is completely the same.

  That is, the first imaging unit 410 is configured to include an imaging optical system 211, a polarizing optical system 421, a light receiving element 422, a light projecting unit 214, and a time-of-flight measuring unit 215, and the second imaging unit 420 includes The imaging optical system 211, the polarization optical system 421, and the light receiving element 422 are included. Other configurations are the same as those of the three-dimensional image capturing apparatus 100-1A shown in FIG.

FIG. 5 is a schematic diagram illustrating an example of a detailed configuration of the polarization optical system 421 and the light receiving element 422 inside the first imaging unit 410 and the second imaging unit 420 illustrated in FIG. 4.
In the case of the three-dimensional image capturing apparatus 100-1B shown in FIG. 4, for example, the polarization optical system 421 and the light receiving element 422 as shown in FIG. In the embodiment shown in FIG. 5A, the polarization component pixels 502 (502a to 502d) and the ranging pixels 503 for measuring the flight time are arranged on the same surface of the light receiving element 422 according to a certain rule. Have been mixed. Here, only one set of a plurality of polarization component pixels 502 (502a to 502d) and distance measurement pixels 503 is indicated in FIG. 5A. The polarization component pixels 502 a to 502 d receive the input light that has passed through the polarizers 501 a to 501 d of the polarization optical system 421, respectively. The ranging pixels 503 are arranged at the center positions of the polarization component pixels 502a to 502d, and uniformly receive the input light that has passed through the polarizers 501a to 501d. In the embodiment shown in FIG. 5A, the amount of light received per pixel is reduced, but only one light receiving element 422 needs to be provided, and the apparatus can be downsized.

In the case of the three-dimensional image capturing apparatus 100-1B shown in FIG. 4, for example, another aspect of the polarization optical system 421 and the light receiving element 422 as shown in FIG.
In the mode shown in FIG. 5B, the polarization optical system 421 includes polarizers 511a to 511c and distance measuring optical elements 512, and correspondingly, the light receiving element 422 includes polarization component pixels 513a to 513c and distance measurement. A pixel 514 is configured. Here, only one set of a plurality of polarization component pixels 513 (513a to 513d) and distance measurement pixels 514 is indicated in FIG. 5B.

  In the embodiment shown in FIG. 5B, the input light passing through the polarizers 511a to 511c is polarized by the polarizer and received by the polarization component pixels 513a to 513d, respectively. Further, the remaining input light passes through the ranging optical element 512 and is received by the ranging pixel 514 without being polarized. The optical element 512 for distance measurement may be a filter that passes only a band of irradiation light for distance measurement, or may be a simple gap. Note that the polarizer 511 in the mode shown in FIG. 5B has only three directions, but the polarization analysis can be performed if there are polarization components in the three directions.

  Further, in the three-dimensional image capturing apparatus 100-1B shown in FIG. 4, both the measurement of the flight time and the measurement of the polarization component are performed by one type of light receiving element 422. For example, both measurements can be performed alternately at different timings. In this case, for example, in imaging of the first frame, the irradiation light 201 whose intensity and frequency are modulated is projected onto the imaging target and received by the light receiving element 422. The flight time measuring unit 215 calculates the flight time information 204 by comparing the phase difference between the modulation waveform 401 measured by the light receiving element 422 and the reference waveform 202 for each pixel of the light receiving element. Then, the distance calculation means 130 calculates distance information 103 related to the distance to the imaging target for each pixel of the light receiving element from the flight time information 204. In the first frame, the modulated light that has passed through any polarizer of the polarization optical system 421 is handled without distinction. In the next one frame, the illumination light 201 is not illuminated by the light projecting unit 214, and the polarization component that has passed through the polarization optical system 421 using ambient light is received by the light receiving element 422 and polarized by the polarization analyzing unit 140. Perform analysis. As described above, by repeating the measurement operation alternately, it is possible to measure the time of flight and the polarization component with one type of light receiving element 422.

  Further, both the measurement of the time of flight and the measurement of the polarization component can be simultaneously performed by one type of light receiving element 422. In this case, the modulated irradiation light 201 is projected onto the imaging target, and the input light is received by the light receiving element 422. The signal measured by the light receiving element 422 is sent to the time-of-flight measurement unit 215 as a modulation waveform 401 without distinguishing the polarization direction, and the time-of-flight measurement unit 215 compares the modulation waveform 401 with the reference waveform 202, Flight time information 204 is calculated. Then, the distance calculation unit 130 calculates distance information 103 related to the distance from the flight time information 204 to the imaging target. At the same time, the modulation waveform 401 is sent from the light receiving element 422 to the polarization analyzing means 140 as polarization component data. At this time, a waveform integrator (not shown in FIG. 4) is provided inside the polarization analyzing means 140, and the polarization analyzing means 140 calculates the average value of the unit time of the modulation waveform 401 for each pixel of the light receiving element 422. The polarization analysis described later is performed based on the calculated value. As described above, by performing the time-of-flight measurement and the polarization component measurement at the same time, it is possible to obtain an effect of reducing an error due to movement or deviation of the imaging target and the three-dimensional image imaging device 100-1B.

  Next, a procedure until the distance image 106 is generated by integrating the distance information 103 and the polarization component data 104 will be described in detail.

<Obtain distance information>
The distribution of the distance information 103 obtained by the distance calculation means 130 in the TOF method is D (x, y). D includes errors due to scattering and multiple reflection. For convenience of calculation, D ′ is defined as the following equation (1) using the maximum distance D _max during observation.

  Hereinafter, D ′ with the sign of D inverted is used as distance information D.

<Polarization analysis>
Here, a procedure for performing polarization analysis and estimating the surface shape of the imaging target will be described.
First, the polarization analysis means 140 obtains the polarization azimuth angle and the polarization degree by polarization analysis of the polarization component data 104. Several methods are known for obtaining the polarization azimuth angle and the polarization degree. In the present embodiment, a general method using polarization component data in four directions will be described.

First, the polarization analysis unit 140 generates four polarization component images by collecting the measured polarization component data 104 for each type of light receiving element. This polarization component image is a distribution image of the intensity of polarization components with linear polarization directions of 0 degrees, 45 degrees, 90 degrees, and 135 degrees, and this distribution is represented by P ₀ (x, y), P ₄₅ (x, y), P ₉₀ (x, y), and P ₁₃₅ (x, y). Here, when D (x, y) of the distance information 103 is compared with P _i (x, y), if either data point sequence is sparse with respect to the other, linear interpolation is performed. The number of data point sequences should be the same.

<Calculation of polarization azimuth>
Next, the polarization azimuth angle θ is obtained by the following equation (2). Here, for details, for example, the method according to Non-Patent Document 1 and Non-Patent Document 2 can be used.

If there is no measurement error, the polarization azimuth angle θ coincides with the normal direction of the measurement surface.
FIG. 6 is a schematic diagram illustrating an example of the azimuth and inclination of the normal of the object surface to be imaged.
Here, the orientation of the normal of the measurement surface is a perspective projection of the normal n of the surface of the imaging target 602a onto the imaging surface 601a of the three-dimensional image capturing apparatus 100, as shown in FIG. Is the azimuth angle θ of n ′ on the imaging plane.

  FIG. 7 is a schematic diagram illustrating an observation example of the polarization azimuth angle and the polarization degree of a three-dimensional object. FIG. 8 is a schematic diagram illustrating an example of a result of generating the distance image 106 based on the measured distance information 103 and the ellipsometric information 105.

  Here, FIG. 7A shows an observation example in which the polarization azimuth angle is observed for a three-dimensional object. Here, the polarization azimuth angle is displayed with the luminance of the image. FIG. 7A shows an example in which the object having the shape shown in FIG. 8A is observed from directly above.

  In general, the reflection of a light source or gloss reflection is one of the factors that cause an error in the measurement of the distance image 106, but the information on the polarization azimuth has a characteristic that hardly affects. Therefore, if the polarization azimuth angle is used, it is possible to measure the normal direction of the surface without being affected by reflection or reflection.

<Calculation of polarization degree>
Next, the degree of polarization ρ (linear degree of polarization) is obtained by the following equation (3). Here, for details, for example, the method according to Non-Patent Document 2 can be used.

  Here, an example of observation of the degree of polarization is shown in FIG. Here, the magnitude of the degree of polarization is displayed by the luminance of the image.

<Estimation of tilt angle>
FIG. 9 is a schematic diagram illustrating an example of the inclination of the normal line of the object surface to be imaged.
As shown in FIGS. 6B and 9, the inclination of the normal of the observation surface is defined as φ. At this time, it is known that there is a relationship expressed by the following equation (4) between the inclination φ of the normal of the observation surface and the polarization degree ρ.

However, n in the equation (4) is the refractive index of the material on the surface of the imaging target. When the imaging target is a non-transparent object, it is necessary to use the complex refractive index n ^ = n-ik instead of the refractive index n. Here, k is an attenuation coefficient. If the complex refractive index is sufficiently large and | n ^ | ² = n ² + k ² >> 1, it can be assumed that there is an approximate expression with good properties shown in the following formula (5). It is shown in Reference 1.

FIG. 10 is a schematic diagram illustrating an example of the relationship between the degree of polarization ρ and the inclination φ of the normal of the object surface when n = 1.6 and k = 5 in the equation (5).
Polarization when the surface of the object is directly facing the imaging means (when the imaging surface of the imaging means and the observation surface are located in parallel, and the normal of the observation surface coincides with the optical axis of the imaging means) The degree is ρ (0) = 0. The degree of polarization ρ increases with an increase in the tilt angle φ of the observation surface. The degree of polarization ρ decreases after reaching a maximum value at an angle and reaches ρ (90) = 0 at 90 degrees. The angle φ ⁰ having the maximum value is called a Brewster angle. In general, the inclination angle φ with respect to the degree of polarization ρ is a bivalent function as shown in FIG. 10, and is not uniquely determined, and two values φ ⁺ and φ ⁻ correspond to one ρ.

<Restoring the surface shape>
Next, a method for estimating the surface shape of the imaging target from the ellipsometry result will be described below.
In general, since the complex refractive index of the observation surface to be imaged cannot be known in advance, an appropriate value is estimated and represented as the complex refractive index value. Here, for example, n = 1.6 and k = 5. The Brewster angle φ ^{0 at} this time is 79.39. As will be described later, a method of estimating the optical parameters n and k of the refractive index from the distance information D and the polarization degree ρ is also conceivable. Further, in order to avoid the multivalent nature of the bivalent function, the degree of polarization is approximated as the following equation (6) to obtain a monovalent function.

Here, in the equation (6), α is an appropriate positive coefficient. Next, a reference table is prepared by numerically solving the inverse function ρ ⁻¹ . This makes it possible to calculate the inclination φ of the normal of the object surface with respect to an arbitrary degree of polarization ρ.

  Subsequently, when the inclination of the object surface in the x and y directions is (v, w), the following equation (7) can be calculated from the normal inclination φ and the normal direction θ.

There is a positive / negative sign because there is one direction that can be taken by the normal in the direction θ, and it is not uniquely determined. However, this can be easily determined from the sign of the gradient _Dx (x, y), _Dy (x, y) of the distance information. Alternatively, the inner product of (D _x, D _y ) and (v, w) may be calculated and determined from the sign. In this way, the ambiguity in estimating the normal direction θ can be eliminated by one of the advantages of the three-dimensional image capturing apparatus 100 according to the present embodiment that handles both the distance information 103 and the ellipsometric information 105. is there.

  Subsequently, the shape of the surface of the entire imaging target is estimated from the value of (v, w) obtained for each pixel of the light receiving element.

  By integrating (v, w) over the entire area of the image, the relative height distribution H (x, y) of the imaging target can be restored, but this method is vulnerable to noise. A distribution that minimizes the energy function shown in the following equation (8) is obtained and set as H (x, y).

  As a method for minimizing the energy function shown in the equation (8), a solution by a relaxation method using the following equation (9) is known (for details, see Non-Patent Document 3, for example).

  This is calculated a certain number of times or until the value converges to obtain H (x, y). The above is the estimation of the surface shape by ellipsometry.

  FIG. 11 is a flowchart illustrating an example of the processing procedure of the polarization analysis processing by the polarization analysis unit 140 according to the first embodiment of this invention.

As described above, the polarization analysis unit 140 performs the following steps S101 to S104.
First, in step S101, the polarization degree ρ (x, y) and the polarization azimuth angle θ (x, y) are obtained from the polarization component data 104. Subsequently, in step S102, the normal inclination of the observation surface φ is obtained from the degree of polarization ρ using the function ρ (φ). Subsequently, in step S103, (v, w) which are inclinations of the object surface in the x direction and the y direction are obtained from the obtained normal plane inclination φ and polarization azimuth angle θ. Subsequently, in step S104, the relative height distribution H (x, y) of the observation target is obtained from (v, w) by the relaxation method.

  As described above, the normal slope φ and the degree of polarization ρ are not approximated by a monovalent function, and for example, a method of restoring the shape using a bivalent function is also conceivable. For example, during the calculation of the relaxation method of the above-described equation (8), the value of H may be updated sequentially by minimizing the function as in the following equation (10).

However, φ ⁻ and φ ⁺ in the equation (10) are inclinations of the normal line of the divalent observation surface corresponding to the polarization degree ρ. H (φ ⁻ ) and H (φ ⁺ ) indicate the height H when the normal angle is obtained as φ ⁺ and the height H when the normal angle is obtained as φ ⁻ .

<Integration of distance information and ellipsometry information>
Next, a method for integrating the relative height distribution obtained by ellipsometry, that is, H (x, y), which is the surface shape information of the object, and the distance information D (x, y) will be described.

<Removal of disturbance components>
For example, when measuring the glossy surface or the like, the distance image generating unit 150 first removes outliers derived from the 0th-order reflection component or the like in the distance information 103. In this case, the information on the polarization azimuth angle θ is less influenced by the reflection component, so that discontinuity appears only in the distance information 103, and the polarization azimuth angle θ is continuous. Utilizing this property, here, for example, a method of removing a disturbance component by a method using the linear process of German et al. Described in Non-Patent Document 4 will be described.

  As an example, the energy function is set as shown in the following equation (11).

Here, D ^{^} in Equation (11) is an estimated value of the distance obtained by removing the disturbance component from D. Also, l = {0, 1}, which is a state variable representing discontinuity such that 1 is taken when the pixel is a disturbance component and 0 is taken when the pixel is not a disturbance component. Also, λ ₁ , λ ₂ , and λ ₃ are positive coefficients of each regular term. The regular term is a term that makes the estimated value D ^{^ the} same as D from the left, a term that makes it easy to discriminate from disturbance when the surface shape is smooth, and a term that suppresses the frequency of discriminating from disturbance. Note that ∇ ² is a Laplacian operator and is defined as the following equation (12) using a second-order differential value.

  Further, in order to minimize the equation (12), the state variable 1 is used as a continuous function and approximated as the following equation (13).

Then, the operation formulas of u and D ^{^} are defined as in the following formulas (14) and (15), and repeated calculation is performed to obtain u and distance D ^{^} that give the minimum energy value.

  However, η is a coefficient that determines the speed of the motion equation, and τ is a coefficient of the sigmoid continuous function, and each determines an appropriate value. In addition, as an example of another method that is simpler and has a low calculation cost, the following method can be given. An energy function for each pixel is defined using the second-order differential value of D (x, y) and H (x, y) as shown in the following equation (16).

  Then, the value of this energy function is compared with a threshold value δ shown in the following equation (17) to determine whether or not there is a disturbance.

  Then, the region determined to be a disturbance is removed and filled (complemented) with the median value of the surrounding pixel information. Alternatively, linear interpolation may be performed using surrounding pixel information. It should be noted that the discriminant function may be expressed by the following equation (18) so that the surface shape information H includes an error due to disturbance and the distance information D can cope with a situation in which the surface shape information H is not affected by the disturbance.

  Equation (18) is a function that takes a large value when either D or H has a discontinuous value and the other is smooth.

<Division of area>
In general, polarization analysis cannot be performed on surfaces that are not visible from the three-dimensional image capturing apparatus 100 and that have an inclination of more than vertical. For this reason, the surface shape information H restored by ellipsometry is information that represents only the surface shape with no height difference between objects, as shown in FIG. 8C. Therefore, the distance image generation unit 150 corrects the height of the surface shape H by dividing the region in the image and integrating the surface shape information 205 and the distance information 103 for each region.

  At this time, it is desirable to divide the region at H and D discontinuous points. There are many methods for area division, but one example is the use of a graph cut method as shown in Non-Patent Document 5, for example.

  The graph cut method is a method of dividing a region by solving a maximum flow rate minimum cut problem. In the graph cut method, it is necessary to define the weights of edges between nodes with the pixel i and the pixel j as nodes of the graph. For example, the weight of the side between the node i and the node j is defined as the following equation (19) using the height difference between the pixels and the spatial distance between the pixels.

Note that λ ₁ and λ ₂ shown in the equation (19) are appropriate coefficients. Further, dist (i, j) is a spatial distance between the pixel i and the pixel j, and is, for example, a Euclidean distance between the pixels. Based on this, segmentation is performed such that if the graph cut method is used as a whole, the region is cut at a boundary with a large difference in elevation.

  FIG. 12 is a flowchart illustrating an example of the processing procedure of the area division process according to the first embodiment of this invention. For example, the distance image generation unit 150 performs region division processing according to steps S201 to S205 below.

  First, in step S201, the farthest pixel and its peripheral area (neighboring area) are used as seeds for the background area. Subsequently, in step S202, segmentation is performed using the closest pixel in the undivided pixel region and its vicinity as a seed for the foreground region. Subsequently, in step S203, the background area and the foreground area are divided by the graph cut algorithm. In step S204, the area divided as the foreground area is stored as the i-th area, and the area is excluded from the image. Subsequently, in step S205, it is determined whether or not all the pixel regions are divided. If all the pixel regions are not divided, i ← i + 1 is set, and the process returns to step S202. Repeat the process. On the other hand, as a result of the determination in step S205, if all the pixel regions are divided, the processing of the flowchart shown in FIG.

  Further, as another example of the region dividing method, a method using the above-described line process for removing disturbance components may be used. Further, instead of the graph cut, for example, a method of quantizing the distance information D by applying the bilateral filter shown in Non-Patent Document 6 a plurality of times may be used.

<Height correction for surface shape information>
In the case of the three-dimensional image capturing apparatuses 100-1A and 100-1B according to the present embodiment, the average of the areas for each of the areas D = {D ¹ , D ² ,..., D ⁿ } divided by the method described above. Find a height. As an example, the median value of the region is used. Similarly, for H, a median is calculated for each region for each region H = {H ¹ , H ² ,..., H ⁿ }. Using these, the surface shape H to ⁱ (x, y) of the region i whose height is corrected is obtained using the following equation (20).

This is performed for all regions H. However, med means to take a median value. Through the above calculation, H to ⁱ (x, y) to which height information is added is obtained from H (x, y) that does not include absolute height information.

  Here, the height is corrected using the median value of the region. However, for example, using the M-estimator method of robust estimation shown in Non-Patent Document 7, the energy function of the following equation (21) is expressed. Fitting that minimizes may be performed.

Here, q (e) in the equation (21) is a positive definite even function having a unique minimum value when e = 0. If q (e) = e ² , the equation (21) is equivalent to the least square method. For example, if the following equation (22) is used as the function q, it is possible to reduce the influence of outliers in fitting.

Here, k in the equation (22) is a constant. Further, the function q is obtained when h ⁱ that minimizes the energy function is analytically obtained. On the other hand, when h ⁱ that minimizes the energy function cannot be obtained analytically, the energy function is minimized by iterative calculation of the differential equation using the following equation (23) to obtain h ⁱ . Here, η is an appropriate coefficient.

<Integration of distance information and surface shape information>
Next, the distance image generation means 150 integrates the distance information D (103) and the surface shape information H to (205) after height correction. At this time, for example, using the robust estimation M-estimator method shown in Non-Patent Document 7, an estimated value that gives the minimum value of the energy function shown in the following equation (24) for each point on the image. Find D ^{^} .

However, q of (24) Formula is the above-mentioned positive definite even function. The method for minimizing the energy function is the same as the method described above. The function g is a regular term that suppresses the discontinuity of D ^{^} . As an example of the function g, the following equation (25) can be considered.

  For example, as an example of the function g, the following equation (26) is also conceivable.

Note that, as an example of an aspect other than the robust estimation method used here, the least square error may be used. As another example, the estimated distance value D ^{^} may be simply the geometric average or arithmetic average result of the distance information D and the surface shape information H. As another example, the value of the surface shape information H˜ may be used as the estimated distance value D ^{^} as it is. As another example, the average value obtained by weighting the distance information D and the surface shape information H˜ with reliability may be used as the estimated distance value D ^{^} . Here, as the reliability, for example, the continuity of the distance information may be used to give a high reliability weight to the higher continuity value. As an example, the following equation (27) is obtained.

By the above method, the distance image generation means 150 outputs the estimated D ^{^} as the distance image 106. As described above, an example of the generated distance image 106 is shown in FIG. FIG. 8A shows the original three-dimensional shape of the imaging target, FIG. 8B shows the distance information D (103) measured by TOF, and FIG. 8C obtained by ellipsometry. Surface shape information H (205). FIG. 8D shows the removal of the disturbance and the region division, estimating the height of the surface shape information H for each region and obtaining H˜, and then integrating the distance information D and the surface shape information H. This is a restored range image. In FIG. 8D, it can be seen that the error and disturbance as in the distance information D shown in FIG. 8B are removed, and the surface shape is restored.

  FIG. 13 is a flowchart illustrating an example of a processing procedure of integration processing of distance information D (103) and surface shape information H (205) by the distance image generation unit 150 according to the first embodiment of this invention.

The distance image generation unit 150 performs the following steps S301 to S304 as described above.
First, in step S301, the disturbance of the distance information D is removed using a line process. Subsequently, in step S302, the region is divided at the discontinuous points of the distance information D and the surface shape information H. Subsequently, in step S303, the surface shape information H is fitted to the distance information D for each region. Subsequently, in step S304, for example, by robust estimation, a value that fits both the surface shape information H and the distance information D is obtained, and this is used as the integration result (ie, the distance image 106).

<Integration of shapes without segmentation>
In the method of estimating the distance information D (103), a method that does not perform region division will be described below as another derivative aspect with lower calculation cost. Here, the same framework as the line process shown by the equations (11) to (15) is used as a method for removing disturbance components.

  First, distance information D (x, y) and surface shape information H (x, y) are fitted to obtain a height difference h (x, y). Here, it is assumed that a state variable indicating whether or not a pixel is located at a discontinuous boundary is l (x, y). When h (x, y) takes a smooth value, a low energy value is given, and when straddling a discontinuous boundary, the smoothness constraint is canceled. Furthermore, the restriction | limiting is put so that the pixel considered as a discontinuous point may be reduced as much as possible. As an example, the energy function is defined as the following equation (28).

In the equation (28), λ ₁ , λ ₂ , and λ ₃ are positive coefficients of each regular term. In order to obtain the parameters h (x, y) and l (x, y) that minimize this, l (x, y) is approximated by a continuous value as described in the equation (13), and the energy function is What is necessary is just to obtain | require by the iterative method by making partial differentiation with each parameter, and setting up an operational equation. Note that an interaction term between boundaries may be added to the equation (28) to prevent the generation of isolated points or unnecessary branches at the boundaries.

  FIG. 14 shows a first embodiment of the present invention, and is a flowchart showing another example of the processing procedure of the integration processing of the distance information D (103) and the surface shape information H (205) by the distance image generating means 150. is there.

In this example, the distance image generation unit 150 performs the following steps S401 to S402 as described above.
First, in step S401, the line shape is used to fit the surface shape information H to the distance information D so that continuity is preserved and a gap of height is allowed at the discontinuous points. Subsequently, in step S402, the surface shape information H fitted to the distance information D is used as an integration result of the distance information D and the surface shape information H (that is, the distance image 106).

<Estimation of distance information without surface shape estimation>
In the present embodiment, when the distance image 106 is estimated from the distance information and the ellipsometric result, the surface shape is not estimated by segmentation and ellipsometry as another derivative aspect with a lower calculation cost. A method is also conceivable. As a specific procedure, a region where the distance information D (103) is discontinuous and the ellipsometric information is continuous is detected, and is removed as a disturbance and complemented. This method is low in calculation cost and has an effect of removing disturbance mainly due to reflection components. As an example, an evaluation value of continuity of the distance information D is provided as in the following equation (29).

  Then, the region where the evaluation value shown in the equation (29) exceeds a predetermined threshold is determined as a disturbance and removed, and is compensated with the median value of the surrounding distance information. Finally, the obtained distance information D is output as a distance image 106.

<Estimation of refractive index from polarization degree and distance information>
In the above-described example, an appropriate value is adopted as the complex refractive index of the object surface to be imaged. However, an aspect in which the complex refractive index is estimated from the polarization degree ρ and the distance information D is also conceivable. . In that case, first, region division is performed by a method such as graph cut, and it is assumed that the complex refractive index in the region is uniform. Here, since the optical parameters are two values of n and k as described from the equations (4) to (5), the values of the polarization degree ρ and the distance information D at two points in the region are substituted. The complex refractive index can be found analytically. However, in this method, in order to be strongly influenced by measurement errors, minimization is performed by the following equation (30) using a plurality of sets (ρ _i , D _i ) of measurement values for a plurality of points in the region. Do.

Here, φ ^{^} _i is the inclination of the normal of the object surface at the pixel i obtained from the distance information D, and φ _i is the inclination of the normal of the object surface when the degree of polarization is ρ _i . As an example of a method of minimizing the expression (30), for example, (φ ^{^} _i −φ _i ) ² = 0 is set, and the values of ρ _i and φ ^{^} _i are randomly determined from the region being observed by a predetermined number F. Sampling is performed, and a relational expression of n and k of F sets is established. Then, the obtained two variable solutions are used to vote (reflect) in the two-dimensional space. As a result, the values n and k that have obtained the most votes are used as estimated values.

  At this time, the LMedS estimation method shown in Non-Patent Document 8 may be used. In this case, the calculation of the estimated value described above is repeated N times, and among the N estimated values, the optical parameter values n and k are evaluated according to the LMedS criterion of the following equation (31), and the best parameter value (lowest) A value that gives a reference value) may be selected as n and k.

  However, in equation (31), med represents an operation that takes a median value. This makes it possible to estimate optical parameters robustly against noise.

(Second Embodiment)
Next, a second embodiment of the present invention will be described.
FIG. 15 is a schematic diagram illustrating an example of a schematic configuration of a 3D image capturing apparatus 100 according to the second embodiment of the present invention. In FIG. 15, the same components as those of the three-dimensional image capturing apparatus 100 according to the first embodiment are denoted by the same reference numerals. In the following description, the three-dimensional image capturing apparatus 100 shown in FIG. It describes as the image pick-up device 100-2.

  The three-dimensional image capturing apparatus 100-2 of the second embodiment integrates the distance information 103 of the imaging target measured by the pattern light method and the polarization component data 104 measured by the polarization optical system, and generates the distance image 106. To get.

  15 includes a first imaging unit 1510, a second imaging unit 220, a distance calculation unit 130, a missing region storage unit 1520, a polarization analysis unit 140, and a shape. The reconstruction unit 230 and the distance image generation unit 150 are configured.

  The first imaging unit 1510 includes an imaging optical system 211, a beam splitter 212, a filter 213, a first light receiving element 113, a pattern generation unit 1511, a light projecting unit 1512, and a modulation analysis unit 1513. Configured to capture an image for distance measurement.

  The second imaging means 220 includes an imaging optical system 211, a beam splitter 212, a filter 221, a polarization optical system 122, and a second light receiving element 123, and is used for measuring polarization components. This image is taken. As described above, in the first imaging unit 1510 and the second imaging unit 220 shown in FIG. 15, the imaging optical system 211 and the beam splitter 212 are configured in common.

  In the present embodiment, the distance measurement method using the pattern light method can use the same method as the general pattern light method. The pattern generation unit 1511 of the first imaging unit 1510 generates a predetermined modulation pattern 1501. Here, as a modulation method, various modes such as a method in which a plurality of trigonometric functions are superimposed and intensity modulation is performed, a pulse modulation method, or a coding method using a de_Bruijn sequence shown in Non-Patent Document 9, for example, are used. Can be considered.

Here, in this example, a bit string [c ₁ , c ₂ ,..., C _n ] (c∈ {0, 1}) according to a binary de_Bruijn sequence is used as a code string. The pattern light projecting unit 1512 generates vertical stripe-shaped pattern light 1502 in which each bit corresponds to the light intensity, and projects it onto the imaging target. For example, when the range of the intensity of the irradiated light is [0 to 1], the bits 0 and 1 correspond to the light intensities 0.5 and 1. The light projecting unit 1512 provides a parallax with respect to the imaging optical system 211.

  The input lights a and b (101a and 101b) collected through the imaging optical system 211 are divided into two by a non-polarizing beam splitter 212, and only the wavelength of the irradiation light is passed through the filter 213 of the first imaging means 1510. The light is passed through to form an image on the first light receiving element 113.

Then, the modulation analysis unit 1513 checks whether there is a homologous pattern (bit string) in the modulation waveform 203 of the image captured by the first light receiving element 113 while referring to the reference waveform 1503, and decodes one row at a time. . At this time, the modulation analysis unit 1513, each pixel of each line _{[x 1, x 2, ...} , x m] 1 one single to be detected at that location code of _{{c 1, c 2, ...} , c n} Is output to the distance calculation means 130 as detection pattern position information 1504.

  At this time, depending on the shape of the object to be imaged, a shaded portion that is not exposed to irradiation light is formed. As described above, if the bits 0 and 1 of the pattern light are associated with the light intensities 0.5 and 1, the shaded portion can be detected as an area having a luminance value equal to or less than a predetermined threshold value. . Then, the modulation analysis unit 1513 stores this portion in the missing area storage unit 1520 as missing area information 1505. Then, such decoding is performed over the entire range of the image.

  Next, the distance calculation unit 130 calculates the distance to the imaging target from the distribution of the pattern light observed on the imaging target. Specifically, the distance calculation unit 130 calculates the distance to the imaging target based on the principle of triangulation based on the position where the code is detected.

FIG. 16 is a schematic diagram illustrating the principle of triangulation distance calculation using pattern light in the distance calculation unit 130 according to the second embodiment of this invention.
In FIG. 16, an appropriate surface is a reference surface 1601. Further, FIG. 16 shows a position 1602 on the image. It is assumed that a code (c _{3 in} FIG. 16) of a certain pattern light 1502 is projected from the light projecting unit 1512 at an angle θ. Here, when the imaging target does not exist, a deviation in the x direction between the position of the code projected and observed on the reference surface 1601 and the position of the code observed on the surface of the imaging target is denoted by Δx. At this time, the height Δz of the object to be imaged from the reference plane 1601 is obtained by the following equation (32).

  FIG. 17 is a schematic diagram illustrating a second embodiment of the present invention and an example of a missing region when the distance calculation unit 130 calculates a distance using pattern light. As illustrated in FIG. 17, a missing region 1701 of the pattern light 1502 is generated due to the parallax between the light projecting unit 1512 of the pattern light 1502 and the first imaging unit 1510.

  Here, the configuration of the polarization optical system 122 and the method for analyzing the polarization component and restoring the surface shape of the imaging target are the same as those described in the first embodiment.

  Next, the distance image generation unit 150 integrates the distance information 103 and the surface shape information 205 to generate the distance image 106. The integration of the distance information 103 and the surface shape information 205 is performed by the same process as described in the first embodiment. However, in the case of the second embodiment, the missing area 1701 due to the parallax as shown in FIG. 17 may occur. Therefore, when the missing area information 1505 is stored in the missing area storage unit 1520, Using the missing area information 1505, the missing area 1701 is complemented.

  At this time, since there is no parallax in the measurement of the polarization component, the missing region 1701 due to the parallax can be complemented using the surface shape information 205. In the present embodiment, the following method is used as an example of a complementing method.

  In this embodiment, first, the distance information D (x, y) is moved by moving H (x, y) in the z direction only for the pixel n in the vicinity of the missing area 1701 of the distance information D (x, y). Fit to. Then, the difference h between the distance information D and the height H is obtained from the fitted result. Subsequently, the missing area 1701 is complemented with H (x, y) + h. Specifically, the processing shown in the following equation (33) is performed.

  Here, in Expression (33), med means that the median is taken, and n is a pixel within a predetermined distance of the missing region 1701. However, n does not include the missing region 1701. Also, null is a pixel in the missing area 1701.

  FIG. 18 is a flowchart illustrating an example of a processing procedure of integration processing of distance information D (103) and surface shape information H (205) by the distance image generation unit 150 according to the second embodiment of this invention.

The distance image generation means 150 performs the following steps S501 to S505 as described above. Here, steps S501 to S503 are basically the same processing as steps S301 to S303 shown in FIG. 13, and step S505 is basically the same processing as step S304 shown in FIG. .
First, in step S501, the disturbance of the distance information D is removed using a line process. Subsequently, in step S502, the region is divided at the discontinuous points of the distance information D and the surface shape information H. Subsequently, in step S503, the surface shape information H is fitted to the distance information D for each region. Subsequently, in step S504, when there is a missing region 1701 in the distance information D, the missing region 1701 is complemented using the surface shape information H. Subsequently, in step S505, a value that fits both the surface shape information H and the distance information D is obtained by, for example, robust estimation, and this is set as an integration result (that is, the distance image 106).

(Third embodiment)
Next, a third embodiment of the present invention will be described.
FIG. 19 is a schematic diagram illustrating an example of a schematic configuration of a 3D image capturing apparatus 100 according to the third embodiment of the present invention. In FIG. 19, the same components as those of the three-dimensional image capturing apparatus 100 according to the first embodiment are denoted by the same reference numerals. In the following description, the three-dimensional image capturing apparatus 100 shown in FIG. This is described as an image capturing apparatus 100-3.

  The three-dimensional image capturing apparatus 100-3 according to the third embodiment integrates the distance information 103 of the imaging target measured by the distance measuring sensor method and the polarization component data 104 measured by the polarization optical system 122 to obtain a distance image. 106 is obtained. Here, the distance measuring sensor according to the present embodiment has a configuration of a general distance measuring sensor as disclosed in Patent Document 3, for example.

  19 includes a first imaging unit 1910, a second imaging unit 1920, a distance calculating unit 130, a polarization analyzing unit 140, a shape restoring unit 230, and a distance image. The generation unit 150 is included.

  The first imaging means 1910 includes an imaging optical system 1911, a beam splitter 1912, a separator lens 1913, first light receiving elements 1914a and 1914b, and a parallax detection unit 1915, and is used for distance measurement. This image is taken. Here, the separator lens 1913 constitutes a distance measuring optical system, for example. In the example shown in FIG. 19, the separator lens 1913 is configured by an optical system (1913a and 1913b) that connects two images. However, in the present invention, the separator lens 1913 is not limited to this, and two or more image What is necessary is just to be comprised with the optical system. Further, the parallax detection unit 1915 detects the parallax information 1904 from, for example, the imaging positions of the two or more images.

  The second imaging unit 1920 includes an imaging optical system 1911, a beam splitter 1912, a polarization optical system 122, and a second light receiving element 123, and captures an image for measuring polarization components. To do. As described above, in the first imaging unit 1910 and the second imaging unit 1920 shown in FIG. 19, the imaging optical system 1911 and the beam splitter 1912 are configured in common.

<Principle of distance sensor>
Here, the principle of the distance measuring sensor will be described with reference to FIGS.
FIG. 20 is a schematic diagram showing an example of the principle of the distance measuring sensor according to the third embodiment of the present invention.

  When the imaging target is observed, the input light 1901 collected by the imaging optical system 1911 is divided by the beam splitter 1912 and guided to the distance measuring sensor of the first imaging means 1910. Subsequently, the light guided to the first imaging unit 1910 is divided into light passing through the right side and the left side of the lens by the separator lenses 1913 and 1913b of the separator lens 1913. Then, the light divided into two by the separator lens 1913 forms an image on the first light receiving elements 1914a and 1914b, respectively.

  Of the two received light signals 1903a and 1903b detected by the first light receiving elements 1914a and 1914b, in this example, the received light signal 1903a is used as a reference signal, and the received light signal 1903b is used as a reference signal. Assume that the reference signal 1903a is measured at the position C of the first light receiving element 1914a. At this time, if the imaging target is located on the focal plane, the reference signal 1903b is also detected at the position C on the first light receiving element 1914b.

  When the imaging target is behind the focal plane, the reference signal 1903b is detected at a position shifted to the A side (FIG. 20) on the first light receiving element 1914b. When the imaging target is ahead of the focal plane, the reference signal 1903b is detected at a position shifted to the E side (FIG. 20) on the first light receiving element 1914b. If the parallax detection unit 1915 obtains the amount of parallax for each pixel on the light receiving element 1914, the parallax detection unit 1915 makes a relative relative to the focal plane at each position corresponding to A to E in the image captured by the second light receiving element 123. Depth can be obtained. However, distance measurement sensors generally have limited detection accuracy due to the fact that the base line length is limited to the lens diameter or less, the received light signal will be blurred if it is far from the focal plane, and it will be difficult to measure the amount of parallax. .

Here, FIG. 20 will be described.
First, the imaging optical system 1911 is focused on the imaging target. Here, the focusing may be performed automatically by using an automatic focusing mechanism which is generally used widely, or may be performed manually. Subsequently, the light is split and guided to the first imaging unit 1910 and the second imaging unit 1920 by the beam splitter 1912.

  In FIG. 20, a plurality of images are formed on the light receiving elements 1914a and 1915b by the separator lenses 1913a and 1913b. The received light signals 1903a and 1903b received by the light receiving elements 1914a and 1915b are sent to the parallax detection unit 1915, where the parallax is detected. The parallax detection unit 1915 extracts an AC component having an appropriate wavelength from the two light reception signals (1903a and 1903b), and then compares the waveforms to detect a phase difference as parallax. At this time, the parallax detection unit 1915 may calculate the cross-correlation and obtain the phase difference without extracting the AC component.

  Then, the distance calculation unit 130 calculates a relative distance from the focal plane based on the parallax information 1904 related to the parallax (phase difference) detected by the parallax detection unit 1915. Here, the distance calculation means 130 stores in advance a correspondence table between the lens extension amount 1902 of the imaging optical system 1911 and the distance information to the focal plane. The distance calculation means 130 calculates distance information from the lens extension amount 1902 to the current focal position using this correspondence table. Then, the distance calculation unit 130 calculates the distance information 103 that is the absolute distance for each pixel in the imaging target by adding the calculated distance information and the relative distance information obtained from the parallax information 1904.

  Further, regarding the measurement of the polarization component by the second imaging unit 1920 and the polarization analysis by the polarization analysis unit 140, the same method as that shown in the first embodiment is performed. Further, the same method as that shown in the first embodiment is used for removing the disturbance due to the reflection component.

  Then, the distance image generating unit 150 integrates the surface shape information 205 based on the ellipsometric information 105 and the distance information 103 obtained by the distance calculating unit 130 to generate the distance image 106. As a specific method for generating the distance image 106, the same method as that described in the first embodiment is used.

(Fourth embodiment)
Next, a fourth embodiment of the present invention will be described.
FIG. 21 is a schematic diagram illustrating an example of a schematic configuration of a 3D image capturing apparatus 100 according to the fourth embodiment of the present invention. In FIG. 21, the same components as those of the three-dimensional image capturing apparatus 100 according to the first embodiment are denoted by the same reference numerals. In the following description, the three-dimensional image capturing apparatus 100 shown in FIG. This is described as the original image capturing apparatus 100-4A.

  A three-dimensional image capturing apparatus 100-4A shown in FIG. 21 integrates distance information 103 of an imaging target measured by a stereo method and polarization component data 104 measured by a polarization optical system 122 to obtain a distance image 106. It is.

  A three-dimensional image capturing apparatus 100-4A illustrated in FIG. 21 includes a first image capturing unit 2110, a second image capturing unit 2120 (120), a parallax detection unit 2130, a missing area storage unit 2140, and a distance calculation unit 130. A polarization analyzing unit 140, a shape restoring unit 230, and a distance image generating unit 150.

  The first imaging unit 2110 includes a first imaging optical system 111 and a first light receiving element 113. The second imaging unit 2120 includes a second imaging optical system 121, a polarization optical system 122, and a second light receiving element 123.

The input light a (101a) collected by the first imaging optical system 111 is received by the first light receiving element 113. The luminance information 2101a detected by the first light receiving element 113 becomes a reference image for distance measurement by the stereo method. The input light b (101b) collected by the second imaging optical system 121 that also serves to measure the polarization component and also the distance measurement passes through the polarization optical system 122 and then forms an image on the second light receiving element 123. To do. Of the signals measured by the second light receiving element 123, the polarization component data 104 is sent to the polarization analysis means 140. At the same time, in the second light receiving element 123, an image obtained by averaging the polarization components is sent to the parallax detection means 2130 as luminance information 2101b. The image (luminance information 2101b) obtained by averaging the polarization components becomes a reference image for distance measurement by the stereo method.

  Subsequently, the parallax detection unit 2130 compares the luminance information 2101a output from the first imaging unit 2110 and the luminance information 2101b output from the second imaging unit 2120 to obtain a corresponding point, for each pixel. The parallax is calculated (detected). Here, as a method for searching for corresponding points, a general stereo method may be used. The parallax detection unit 2130 stores the missing region 1701 as the missing region 1701 regarding the region where the luminance contrast does not exist or the change is gradual and the corresponding point is not detected, or the region where the shielding due to the parallax peculiar to the stereo method exists. Missing area information 2103 is stored.

  Subsequently, the distance calculation unit 130 calculates distance information 103 related to the distance to the imaging target based on the parallax information 2102 from the parallax detection unit 2130. The generation of the ellipsometric information 105 by the ellipsometric means 140 and the generation of the surface shape information 205 by the shape restoring means 230 and the method for integrating the distance information 103 and the surface shape information 205 are the same as in the first embodiment. The method can be used. In addition, as a method for complementing the missing area 1701 based on the missing area information 2103, the same method as in the second embodiment can be used.

FIG. 22 is a schematic diagram illustrating another example of the schematic configuration of the three-dimensional image capturing apparatus 100 according to the fourth embodiment of the present invention. In addition, the same code | symbol is attached | subjected about the structure similar to the structure shown in FIG. 21, and the 3D image imaging device 100 shown in FIG. 22 is described as 3D image imaging device 100-4B in the following description.
A three-dimensional image capturing apparatus 100-4A illustrated in FIG. 21 includes a polarization optical system in both the first image capturing unit 2210 and the second image capturing unit 2220.

  22 includes a first imaging unit 2210, a second imaging unit 2220, polarization analysis units 140a and 140b, a parallax detection unit 2230, and a missing area storage unit 2140. The distance calculation means 130, the polarization analysis means 140, the shape restoration means 230, and the distance image generation means 150 are configured.

  In the three-dimensional image pickup apparatus 100-4B shown in FIG. 22, the polarization analysis means 140a and 140b perform polarization analysis on the polarization component data 104a and 104b measured by the two imaging means (2210 and 2220), respectively. Then, the polarization analysis means 140a and 140b obtain the polarization degree ρ (x, y) and the polarization azimuth angle φ (x, y), respectively, to generate the polarization analysis information 105a and 105b, and send them to the parallax detection means 2230. send.

  The parallax detection unit 2230 uses the polarization analysis information 105a and 105b of the degree of polarization and the polarization azimuth to search for corresponding points between the two images and detect parallax. Then, the parallax detection unit 2230 transmits the parallax information 2102 related to the detected parallax to the distance calculation unit 130. The method for calculating the distance information 103 from the parallax information 2102 in the distance calculation unit 130 and the method for integrating the distance information 103 and the surface shape information 205 by polarization analysis in the distance image generation unit 150 are shown in FIG. It is the same as the method described.

  In the three-dimensional image capturing apparatus 100-4B shown in FIG. 22, there is an imaging target having a surface characteristic such that a change in luminance is poor but a polarization component is largely changed, or there is a reflection due to a high reflectance. There is an effect of improving the accuracy of distance measurement for the imaging target. Specifically, there is an effect of improving the accuracy of distance measurement by increasing clues of correct corresponding points for parallax detection for such an imaging target. In this embodiment, parallax is detected by searching for corresponding points using only the degree of polarization and the polarization azimuth angle. However, parallax may be detected by adding luminance information obtained by averaging polarization components as a clue.

(Other embodiments of the present invention)
Each of the steps in FIGS. 11 to 14 and FIG. 18 showing the control method of the three-dimensional image capturing apparatus 100 according to each embodiment of the present invention described above is performed by, for example, a program stored in a RAM or ROM by the CPU of the computer. It can be realized by executing. This program and a computer-readable recording medium recording the program are included in the present invention.

  In addition, the present invention can be implemented as, for example, a system, apparatus, method, program, storage medium, or the like. You may apply to the apparatus which consists of one apparatus.

  In the present invention, a software program (in the embodiment, a program corresponding to the flowcharts shown in FIGS. 11 to 14 and 18) for realizing the functions of the above-described embodiments is directly or remotely supplied to the system or apparatus. Including what to do. The present invention also includes a case where the system or the computer of the apparatus is achieved by reading and executing the supplied program code.

  Accordingly, since the functions of the present invention are implemented by computer, the program code installed in the computer also implements the present invention. In other words, the present invention includes a computer program itself for realizing the functional processing of the present invention.

  In that case, as long as it has the function of a program, it may be in the form of object code, a program executed by an interpreter, script data supplied to the OS, and the like.

  Examples of the recording medium for supplying the program include a flexible disk, hard disk, optical disk, magneto-optical disk, MO, CD-ROM, CD-R, and CD-RW. In addition, there are magnetic tape, nonvolatile memory card, ROM, DVD (DVD-ROM, DVD-R), and the like.

  As another program supply method, a browser on a client computer is used to connect to an Internet home page. The computer program itself of the present invention or a compressed file including an automatic installation function can be downloaded from the homepage by downloading it to a recording medium such as a hard disk.

  It can also be realized by dividing the program code constituting the program of the present invention into a plurality of files and downloading each file from a different homepage. That is, a WWW server that allows a plurality of users to download a program file for realizing the functional processing of the present invention on a computer is also included in the present invention.

  In addition, the program of the present invention is encrypted, stored in a storage medium such as a CD-ROM, distributed to users, and key information for decryption is downloaded from a homepage via the Internet to users who have cleared predetermined conditions. Let It is also possible to execute the encrypted program by using the downloaded key information and install the program on a computer.

  Further, the functions of the above-described embodiments are realized by the computer executing the read program. In addition, based on the instructions of the program, an OS or the like running on the computer performs part or all of the actual processing, and the functions of the above-described embodiments can also be realized by the processing.

  Further, the program read from the recording medium is written in a memory provided in a function expansion board inserted into the computer or a function expansion unit connected to the computer. Thereafter, the CPU of the function expansion board or function expansion unit performs part or all of the actual processing based on the instructions of the program, and the functions of the above-described embodiments are realized by the processing.

  Note that each of the above-described embodiments is merely a specific example for carrying out the present invention, and the technical scope of the present invention should not be construed as being limited thereto. . That is, the present invention can be implemented in various forms without departing from the technical idea or the main features thereof.

  The three-dimensional image capturing apparatus 100 according to the present invention may be used as, for example, a three-dimensional measuring apparatus for parts that are necessary in the automation of the manufacturing process. In addition, since it is possible to record the distance image as a moving image, it may be used as a security camera.

100: 3D imaging device, 101a, 101b: input light, 102: measurement data, 103: distance information, 104: polarization component data, 105: polarization analysis information, 106: distance image, 110: first imaging means, 111: first imaging optical system, 112: distance measuring optical system, 113: first light receiving element, 120: second imaging means, 121: second imaging optical system, 122: polarization optical system, 123: second light receiving element, 130: distance calculating means, 140: polarization analyzing means, 150: distance image generating means

Claims (12)

A polarization optical system that includes a plurality of polarizers having different polarization directions and polarizes light from an imaging target ;
A distance measuring element for receiving each light polarized by a plurality of polarizers having different polarization directions and measuring a distance to the imaging target;
A polarization analyzing element for receiving light polarized by the polarizer of a predetermined polarization direction and performing polarization analysis;
Distance information acquisition means for acquiring distance information to the imaging target based on the light received by the distance measuring element ;
Surface shape information acquisition means for acquiring surface shape information of the imaging target by performing the polarization analysis based on polarization information of light received by the element for polarization analysis;
It said distance information and on the basis of said surface shape information, the image pickup apparatus characterized by having the distance image generating means for generating a distance image of the imaging target. Furthermore, it has a light projection means for projecting irradiation light on the imaging target ,
The distance information acquisition unit measures a flight time until the irradiation light projected from the light projecting unit is reflected by the imaging target and reaches the element for distance measurement, and based on the measured flight time. the imaging apparatus according to claim 1, characterized in that to calculate the distance to the imaging target. Furthermore, it has a light projection means for projecting pattern light onto the imaging target,
It said distance information obtaining means, the image pickup apparatus according to claim 1, characterized in that to calculate the distance to the imaging target from the distribution of said light is projected to the imaging target the pattern light. The distance image generation means has a state variable representing discontinuity of at least one of the distance information and the surface shape information, and based on a result of estimating the state variable, the distance information and the surface information by integrating the surface shape information, the image pickup apparatus according to any one of claims 1 to 3, characterized in that to generate the range image. Said distance image generating means, when generating the range image, the distance information and, according to claim 1 or any one of the 4 and removing the disturbance component by comparing the surface shape information The imaging device according to item. Furthermore, it has a missing area judging means for judging whether or not a missing area exists in the distance information,
The distance image generating means supplements the missing area with the surface shape information when generating the distance image when the missing area determining means determines that a missing area exists in the distance information. The imaging device according to any one of claims 1 to 5, wherein Furthermore, it has a missing area judging means for judging whether or not a missing area exists in the distance information,
The distance image generation means uses the missing area stored in the missing area storage means when generating the distance image when the missing area determination means determines that a missing area exists in the distance information. The imaging device according to claim 1, wherein the imaging device is supplemented. It said distance image generating means, by integrating with the distance information for each of a plurality of regions of the surface shape information, according to any one of claims 1 to 7, characterized in that to generate the range image Imaging device. It said distance image generating means sets the reliability based on the continuity of the distance information, and based on the corresponding set reliability of claims 1 to 8, characterized in that to generate the range image The imaging device according to any one of the above. The distance image generation unit estimates a refractive index of the imaging target based on the distance information and the polarization information, and restores a surface shape of the imaging target from the polarization information using the estimated refractive index. The imaging apparatus according to claim 1, wherein the distance image is generated. Furthermore, the imaging apparatus according to any one of claims 1 to 10, characterized in that have a recording means for continuously recording said distance image as a moving image. A polarizing optical system that includes a plurality of polarizers having different polarization directions and that polarizes light from the imaging target; and receiving each light polarized by the plurality of polarizers having different polarization directions; A method for controlling an imaging apparatus, comprising: a distance measuring element for measuring a distance; and a polarization analyzing element for receiving light polarized by the polarizer in a predetermined polarization direction and performing polarization analysis. And
A distance information acquisition step for acquiring distance information to the imaging target based on light received by the distance measuring element ;
A surface shape information acquisition step for acquiring surface shape information of the imaging target by performing the polarization analysis based on polarization information of light received by the element for polarization analysis;
Control method for an imaging apparatus, wherein said distance information and on the basis of said surface shape information, and a range image generating step of generating a distance image of the imaging target.

Images