JP5595211B2 - Three-dimensional shape measuring apparatus, three-dimensional shape measuring method, and computer program - Google Patents

Description

  The present invention relates to a three-dimensional shape measuring apparatus, a three-dimensional shape measuring method, and a computer program.

  Various methods have been proposed as a three-dimensional shape measurement method. Roughly classified, there are a passive type in which shape measurement is performed only with an imaging device without using an illumination device, and an active type in which the illumination device and the imaging device are used in combination. The active type is more robust with respect to the measurement target than the passive type, and enables highly accurate distance measurement. In the active type, even when there is little surface texture information to be measured, shape measurement is performed using the projected illumination pattern as a clue, so that it is not easily affected by the surface texture information. Typical examples of the active type include a method based on triangulation, a method based on phase information change, and a method based on projection pattern defocus information.

  Among these methods, a method using a change in pattern phase information is called a fringe pattern projection method, and one of representative methods is a Fourier transform method (see Non-Patent Document 1). However, in this method, since the phase change that can be measured is limited to a specific range, if a shape change outside the range occurs, the phase is folded back and it is necessary to connect the phases. Therefore, if the measurement target has a smooth shape, phase connection can be achieved relatively easily using phase continuity, but if the measurement target has a sharp shape change or is a discontinuous measurement target, the phase connection It becomes difficult.

  Further, as a method based on the defocus information of the projection pattern, a shape measurement method using defocus by the imaging optical system (see Patent Document 1) or a shape measurement method using defocus by the projection optical system (Non-Patent Document 2). See). In the latter case, when the imaging optical system is pan focus, the amount of blur increases in proportion to the amount of deviation from the focal position of the projection optical system. This blur is evaluated as the spread of the pattern on the image, and the shape is measured. Specifically, 24 types of patterns obtained by finely shifting a rectangular projection pattern are projected and imaged. The defocus amount is calculated by changing the luminance value by arranging the luminance value of each pixel of the image in time series for each pattern. Although this method can calculate the absolute value of the depth, it needs to increase the number of pattern projections, and is not suitable for real-time measurement.

Japanese Patent No. 3481631

M. Takeda and K. Mutoh, "Fourier transform profilometry for automatic measurement of 3-D object shapes," Appl.Opt. Vol. 22, No. 24, P. 3977 to 3982 (1983). Li Zhang and Shree Nayar, "Projection defocus analysis for scene capture and image display," ACM Trans. On Graphics, Vol. 25, 3, P. 907-915 (July 2006).

  As described above, a three-dimensional shape measurement technique that can measure a measurement object having a rapid shape change or a discontinuous measurement object to a detailed shape with a small number of projection patterns has not been proposed. Therefore, an object of the present invention is to enable high-precision shape measurement of a measurement object whose shape change is abrupt or a discontinuous measurement object with a small number of projection patterns.

The present invention for solving the above problems includes a pattern projecting unit that projects a pattern having periodicity onto a measurement space, and
Imaging means for photographing the measurement space on which the pattern is projected,
The measurement space is defined by a reference plane, a projection range of the pattern projection unit, and an imaging range of the imaging unit, and the imaging unit images the measurement space so as to be focused on the reference plane,
A three-dimensional shape measuring apparatus for measuring a three-dimensional shape of a measurement target existing in the measurement space,
Phase information calculating means for calculating phase information of the pattern from a change in brightness of the pattern of the photographed image obtained by the photographing means;
Phase information correction means for correcting the calculated phase information;
A blur amount calculating means for calculating a blur amount of the pattern in the captured image;
3D shape calculation means for calculating the 3D shape of the measurement object based on the phase information and the amount of blur when the relative positional relationship between the measurement object and the 3D shape measurement device does not change; With
The phase information correction unit converts the blur amount into depth information of the measurement space, calculates a phase order corresponding to the depth information, corrects the phase information based on the order,
The three-dimensional shape calculation means calculates the three-dimensional shape of the measurement target based on the corrected phase information.

  According to the present invention, it is possible to perform highly accurate shape measurement of a measurement object whose shape change is abrupt or a discontinuous measurement object with a small number of projection patterns.

FIG. 3 is a diagram illustrating a configuration example of a three-dimensional shape measurement apparatus 100 according to the first embodiment. FIG. 4 is a diagram illustrating an example of a projection pattern of the pattern projection unit 1 according to the first embodiment. The figure which shows the example which image | photographed the scene, without displaying a pattern on the pattern projection part. The figure which shows an example of the picked-up image at the time of pattern projection in the condition of FIG. The figure for demonstrating the defocus amount of a projection pattern. The figure which shows the structural example of the apparatus in the case of performing calibration. The figure which shows an example of a calibration board. The flowchart which shows an example of a calibration process. The figure which shows the example of imaging | photography of the calibration board by which the pattern was projected. 6 is a graph showing a correspondence example between a contrast value and a depth value from the photographing unit 2; The figure which shows the structural example in the case of performing detailed calibration. 5 is a flowchart illustrating an example of a three-dimensional shape measurement process according to the first embodiment. The figure for demonstrating the principle of a Fourier-transform method. The figure which shows the calculation result example of the folded phase (phi) _w (x, y). The figure which shows an example of the calculation result of a contrast value and a standard deviation value. The figure for demonstrating the process of phase information (phi) (x, y) calculation. The figure for demonstrating the correction method of phase information. The figure which shows the structural example of the three-dimensional shape measuring apparatus 200 of Embodiment 2. FIG. FIG. 6 is a diagram illustrating an example of a projection pattern of a pattern projection unit 1 according to the second embodiment. 10 is a flowchart illustrating an example of a three-dimensional shape measurement process in the second embodiment. The figure for demonstrating the principle of the phase calculation by a phase shift method. The figure which shows the projection pattern of Embodiment 3, and the calculation result of phase (phi) w (x, y). The figure which shows the projection pattern of Embodiment 4, and the calculation result of phase (phi) w (x, y). FIG. 10 is a diagram illustrating a configuration example of a three-dimensional shape measurement apparatus 300 according to a fifth embodiment. FIG. 10 is a diagram illustrating a projection pattern and a method for calculating local arrangement information according to the fifth embodiment.

  DESCRIPTION OF EXEMPLARY EMBODIMENTS Hereinafter, preferred embodiments of the invention will be described with reference to the accompanying drawings. In addition, in each figure, the same reference number is attached | subjected about the same member and the overlapping description is abbreviate | omitted.

[Embodiment 1]
FIG. 1 shows a schematic top view of a shape measuring apparatus 100 according to Embodiment 1 of the present invention. It arrange | positions so that the optical center O of the imaging | photography part 2 may become the origin of a world coordinate system (X, Y, Z). Further, the Z axis is made coincident with the optical axis of the photographing unit 2, and the X axis is set to the right in the horizontal direction, and the Y axis is set to the vertical direction. In Embodiment 1, it arrange | positions so that the optical axis of the pattern projection part 1 and the optical axis of the imaging | photography part 2 may become parallel. The optical center C of the pattern projection unit 1 is disposed at a position separated from the optical center O of the photographing unit 2 by a distance D on the X axis. The horizontal half field angle of the pattern projection unit 1 is θ _X , and the vertical half field angle is θ _Y. In addition, the horizontal half field angle of the photographing unit 2 is ψ _X , and the vertical half field angle is ψ _Y. In FIG. 1, measurement objects are a cylinder 41 located at a distance Z1 from the optical center O of the photographing unit 2 and a prism 42 located at a distance Z2 from the optical center O of the photographing unit.

  The pattern projection unit 1 includes an image display element 11 and a projection optical system 12. For the pattern projection unit 1, a projection apparatus having an imaging optical system capable of imaging a pattern having periodicity at a specific position such as a liquid crystal projector is used. The focus of the pattern projection unit 1 is adjusted to the reference plane 3 of the scene. The distance from the optical center C of the pattern projection unit 1 to the reference plane 3 is L. It is desirable to set the reference plane 3 at the rearmost of the measurement range. This is because when the measurement target exists both near and behind the reference plane, it can be distinguished whether the projection pattern is blurred because it is in front of the in-focus position or because it is behind. It is because it becomes impossible. A measurement space in which depth measurement is possible is defined by a projection range of the pattern projection unit 1, a shooting range of the shooting unit 2, and a range 40 indicated by a thick solid line in which three ranges closer to the shooting unit than the reference plane overlap. However, it is not possible to measure the part where the occlusion due to the object occurs within the range.

An example of a pattern projected by the pattern projection unit 1 is shown in FIG. As shown in FIG. 2A, the projection pattern is a pattern having periodicity such that the intensity of illumination varies sinusoidally between the maximum intensity I _max and the minimum intensity I _min . In the first embodiment, a case where the pattern projection unit 1 projects one type of pattern will be described. The pitch is adjusted to be p on the reference plane 3. An example of a sinusoidal pattern at a position shifted from the reference plane 3 is as shown in FIG. The blur occurs due to the influence of the positional deviation from the reference plane 3 which is the focal position of the projection optical system. Since the amount of blur generation depends on the defocus amount, in the first embodiment, the defocus amount is used as the reference amount of the blur amount, and the defocus amount is calculated to calculate the blur amount. Due to the occurrence of defocusing, the maximum intensity I _max decreases and the minimum intensity I _min increases. That is, the contrast C described by the following formula (1) is lowered.
C = (I _max -I _min ) / (I _max + I _min ) (1)
In addition, as the amount of defocus generation increases, the projection pattern approaches gray, and thus the standard deviation of image luminance decreases. The standard deviation SD is described by the following formula (2).
Here, N is the total number of pixels in a certain image area, I _ave is the average value of image brightness in a certain image area, and I _i is the brightness of each pixel in the image area.

  Returning to the description of FIG. 1, the photographing unit 2 includes an imaging element 21 and a photographing lens 22. A photoelectric conversion element such as a CMOS sensor or a CCD can be used for the imaging element 21. Let Fc be the focal length of the taking lens 22. It is assumed that various camera parameters such as distortion are calibrated in advance. Since defocusing by the projection optical system of the pattern projection unit 1 and defocusing by the imaging optical system of the imaging unit 2 are separated, it is necessary to perform imaging under conditions that allow the defocusing of the imaging optical system to be ignored. Specifically, it is necessary to sufficiently reduce the aperture of the photographing unit 2 and increase the depth of field. However, it should be noted that if the diaphragm is too narrow, the image will be blurred due to the influence of diffraction.

The front depth of field and the rear depth of field of the photographing unit 2 are calculated from the following equations (3) and (4), respectively.
φ _r × Fno × Z _s ² / (Fc ² + φ _r × Fno × Z _s ) (3)
φ _r × Fno × Z _s ² / (Fc ² −φ _r × Fno × Z _s ) (4)
Here, φ _r is the allowable circle of confusion, Fno is the F value (aperture value), Z _s is the subject distance, and Fc is the focal length of the imaging optical system. The sum of the front depth of field and the rear depth of field is the depth of field of the imaging optical system.

For example, let us consider a case where a camera having a lens of 1/2 inch, VGA, and a focal length of 16 mm is used to shoot a scene with a depth range of 30 cm that is 1 m away. If the permissible circle of confusion φ _r is 0.01 mm and the pixel pitch of the image sensor is 0.01 mm, the F value needs to be set larger than 4. On the other hand, the blur limit due to diffraction is calculated by the following Airy equation (5).
1.22 × λ × Fno (5)
Here, λ is the wavelength of light. When calculating at 780 nm, which is said to be the longest wavelength in visible light, if the pixel pitch of the image sensor is 0.01 mm, the influence of diffraction is eliminated if the F value is smaller than F11. That is, under the above conditions, it is necessary to set the F value between F4 and F8.

  If a scene composed of a cylinder 41 and a prism 42 is photographed without displaying a pattern on the pattern projection unit 1, a photographed image 60 shown in FIG. 3 is obtained. The image 60 includes a cylinder 61 and a prism 62. The x-axis of the image coordinates is the right direction toward the reference plane 3, and the y-axis is the downward direction as well. Further, the origin o of the image coordinates is the upper left.

  Next, when the pattern of FIG. 2 is projected by the pattern projection unit 1, an image 63 shown in FIG. 4 is obtained. The image 63 includes a cylinder 64 and a prism 65. However, in FIG. 4, defocusing by the optical system of the pattern projection unit 1 is not considered. In addition, sinusoidal intensity fluctuations are also shown as binary values in a simplified manner. Actually, the contrast of the pattern decreases due to the defocus of the projection optical system.

Here, the defocus amount of the projection pattern projected by the pattern projection unit 1 depends on the positional deviation amount from the focal position and the aperture diameter of the projection optical system. This will be described with reference to FIG. FIG. 5A shows a schematic diagram of the projection optical system and the projection image when the aperture diameter is A1 and FIG. 5B is the aperture diameter A2. The relationship between A1 and A2 is A1> A2. In FIG. 5A, the real image of the image display element 11 is 13. For simplicity, the projection optical system 12 is shown as a single lens. The focal length of the projection optical system 12 is Fp, and the distance between the image display element 11 and the projection optical system 12 is d. The distance di from the lens to the imaging plane Pf is derived from Expression (7) obtained by modifying the imaging formula represented by Expression (6) below.
1 / Fp = 1 / d + 1 / di (6)
di = d · Fp / (d−Fp) (7)
The image spreads on a plane P1 that is a front of the image plane Pf by α. When the aperture diameter is A, the spread B of the image is described geometrically by the following formula (8).

B = α · A / di (8)
It can be seen that the image spread B, that is, the defocus amount increases as the positional shift amount α from the imaging position increases. The relationship calculated geometrically for simplicity is described. However, considering the wave optics, it is necessary to consider the influence of diffraction on the spread of the image, but it is omitted here. That is, the distance from the reference plane 3 to the measurement target can be estimated by evaluating the defocus amount of the projection pattern by the pattern projection unit 1. On the other hand, it can be seen from equation (8) that the defocus amount increases as the aperture diameter A increases. FIG. 5A shows the image spread B1 when the aperture diameter is A1, and FIG. 5B shows the image spread B2 when the aperture diameter is A2. B1> B2.

  The larger the aperture diameter, the larger the defocus change with respect to the positional deviation, so the distance estimation accuracy is improved. However, if the defocus amount exceeds a certain amount, it becomes impossible to calculate the amount of change due to the positional deviation, and therefore there is a limit to the distance range in which the defocus amount can be estimated. This distance range is wider as the opening diameter is smaller. Therefore, when it is desired to have a wide measurable distance range, it is necessary to reduce the opening diameter to some extent. However, if the distance range is wide, the defocus amount change with respect to the positional deviation is small, so the sensitivity is lowered, and the detailed shape of the object cannot be measured.

  The shape measuring apparatus 100 according to the first embodiment requires calibration for acquiring the relationship between the defocus amount and the depth position before performing shape measurement. First, a simple calibration method will be described with reference to FIGS. FIG. 6A shows a schematic top view of an apparatus for performing calibration, and FIG. 6B shows a schematic side view. The focus of the pattern projection unit 1 is adjusted to the reference plane 3 as in the configuration of FIG. Let Zo be the distance from the imaging unit 2 to the closest point of the shootable range. The calibration board 50 is a board used for calibration, and is made of a flat plate as shown in FIG. 7, and dots are drawn at regular intervals. Further, it is assumed that the calibration board 50 is inclined as shown in FIG. 6B, and the world coordinate values (X, Y, Z) of each dot are known. This is realized by using a known camera calibration method.

  With reference to FIG. 8, the calibration process in the shape measuring apparatus 100 will be described. First, in S01, the pattern projection unit 1 performs pattern projection. Specifically, a pattern whose intensity changes sinusoidally as shown in FIG. 2 is projected onto the calibration board 50. Next, in S02, the photographing unit 2 photographs the calibration board 50 on which the pattern is projected. At this time, for example, a captured image 51 as shown in FIG. 9 is obtained. The image 51 includes a calibration board 50. In FIG. 9, for the sake of simplicity, the number of patterns is reduced, but the defocus of the projection pattern increases toward the lower part of the image close to the photographing unit 2. Next, in S03, the defocus amount calculation unit 8 calculates the defocus amount of the projection pattern. The contrast value or the standard deviation value is used to calculate the defocus amount.

First, the contrast value calculation method will be described. Since the calibration board 50 is arranged as shown in FIG. 6B, the depth values at the same y coordinate of the image are considered to be equal. Therefore, a region including a pattern having a plurality of cycles such as a region 53A in the image 51 is considered. In the region, the contrast C is calculated from the image luminance according to the above equation (1). In Expression (1), I _max is the maximum luminance in the image area, and I _min is the minimum luminance in the image area. For example, calculation is performed for all y coordinates while shifting the region in the y direction as in the image region 53B. However, if a calibration board dot is included in the image area, it causes noise, so it is better to exclude that area.

  Next, a method for calculating the standard deviation value will be described. Similar to the contrast value, a region including a pattern having a plurality of cycles such as the region 53A is considered. In the case of the standard deviation value, the SD value is calculated by the above formula (2) in the area. Calculations are performed for all y coordinates while shifting this region in the y direction.

  Returning to the description of FIG. 8, in S04, depth interpolation processing is performed. Specifically, assuming that the calibration board 50 is a plane, the depth value between dots is calculated by interpolation processing. If the calibration board 50 is substantially flat, it can be calculated by linear interpolation. If there is some distortion, non-linear interpolation may be preferable. Next, in S05, a table of depth values and defocus amounts is created. Specifically, the correspondence between the defocus amount calculated in S03 and the depth value of the calibration board 50 obtained by the interpolation process is obtained. For example, the contrast value C of the photographed image is high at a depth close to the reference plane 3, and the contrast value of the photographed image decreases as the distance to the photographing unit 2 is approached, so that a curve shown in FIG. 10 is drawn. FIG. 10 shows a case where a contrast value is used as the defocus amount. The curve may be approximated by a function using a least square method or the like.

  This method assumes that the defocus does not vary over the entire image screen. This process is established when the influence of the aberration of the projector is small and the fluctuation of the defocus amount according to the position of the projection pattern can be ignored. When the influence of the curvature of field is large, the defocus amount varies between the central portion and the peripheral portion of the projection image, and thus strict calibration may be required. In order to perform calibration precisely, it is necessary to obtain the relationship between the defocus amount and the depth value for each pixel of the image.

This will be described with reference to FIG. In this method, since the calibration board 50 is moved in the Y-axis direction, a calibration board moving unit 54 is required as shown in FIG. Assuming that the width for operating the calibration board 50 is Ys and the number of shifts is N, the shift pitch Sp is described by the following equation (9).
Sp = Ys / N (9)
The smaller the shift pitch Sp of the calibration board 50 is, the more accurate the calibration is. In the strict calibration process in this case, an additional process is performed before and after the process corresponding to the flow shown in FIG. First, the calibration board 50 is moved to the initial position before the pattern photographing in S01. Thereafter, S01 to S05 in FIG. 8 are executed. In the case of strict calibration, the number of times of photographing is counted, and each time the processing of S05 is completed, it is determined whether or not the number of times of photographing has reached a predetermined number N of times that calibration should be finished. If the number of photographing is less than the predetermined number N, the calibration board 50 is moved by the shift pitch Sp, and then S01 to S06 are repeated. When the number of times of photographing reaches the predetermined number N, the calibration is finished. If such processing is performed, since the correspondence between the defocus amount and the depth position is obtained for all the pixels of the image, the accuracy of calibration is improved.

  Next, a procedure for shape measurement by the shape measurement apparatus 100 according to the first embodiment will be described. The shape measurement is performed according to the flowchart shown in FIG. The processing corresponding to the flowchart is realized by the pattern projection unit 1, the imaging unit 2, the phase information calculation unit 7, the defocus amount calculation unit 8, and the three-dimensional shape calculation unit 9 executing corresponding processing programs. .

  First, in S10, an image photographing process including pattern projection (S11) and image photographing (S12) is performed. In the pattern projection of S10, the pattern projection unit 1 projects a pattern onto the scene to be photographed. In S <b> 12, the photographing unit 2 captures an image of the scene to be photographed. Subsequently, in S20, a phase information calculation step including Fourier transform (S21), fundamental frequency component extraction (S22), shift to the origin (S23), inverse Fourier transform (S24), and phase information extraction (S25) is performed. This is performed by the calculation unit 7.

The calculation of the phase information in the first embodiment is based on the Fourier transform method. Therefore, the principle of the Fourier transform method disclosed in Non-Patent Document 1 will be described with reference to FIG. FIG. 13 shows an optical arrangement when the pattern projection unit 1 and the imaging unit 2 are parallel. The reference plane 3 is located at a position L away from the optical axis C of the pattern projection unit 1. Further, it is assumed that there is a object plane M in a position apart Z _s from the optical axis C of the pattern projection unit 1. Let H be the distance from the reference plane 3 to the object plane M. The projection unit projects a sinusoidal pattern that periodically fluctuates at a pitch p onto the reference surface. An image g _B (x, y) observed on the imaging surface is expressed by the Fourier series as shown in Expression (10).
Here, f ₀ is the fundamental frequency of the observed fringe pattern and is described by the following equation (11).
f ₀ = 1 / p (11)
p is the pitch of the pattern on the reference plane, A _n is the maximum amplitude of the intensity. When the scene has a measurement target, the phase of the observed fringe changes according to the distance from the reference plane to the measurement target. For example, it is assumed that there is a point projected onto the position B on the reference plane and imaged on the image plane. Here, if the object is separated from the reference plane by a distance H, the object is projected onto the point M, and therefore, the object is observed in a state shifted by the distance S _M on the image plane. Since this distance changes depending on x and y, equation (10) is modified as equation (12) below.
Here, φ (x, y) = 2πf ₀ S (x, y). In addition, it is necessary to consider the fluctuation of the reflectance r (x, y) of the object surface. Therefore, the equation (12) is transformed into the following equation (13).
The distorted pattern image given by equation (13) can be interpreted as follows. It is a multidimensional signal having a phase φ (x, y), an amplitude r (x, y), and a spatial carrier frequency nfo. Since the phase has information of a three-dimensional shape, it becomes a problem of how to obtain φ (x, y) by separating the non-uniform reflection amplitude r (x, y).
Equation (13) is rewritten as the following equation (14).
Here, q _n (x, y) = A _n r (x, y) · exp (inφ (x, y)). When y is fixed by the FFT algorithm and the one-dimensional Fourier transform of Expression (14) is calculated, the following Expression (15) is obtained.
Here, G (f, y) and Q _n (f, y) are Fourier spectra related to the variable x with y of g (x, y) and q _n (x, y) fixed. In most cases, all the spectra Q _n (f−nf ₀ , y) are separated from each other by the fundamental frequency f ₀ .
Therefore, the spectral component Q ₁ (f−nf ₀ y) when n = 1 is selected, and inverse Fourier transform is performed to obtain a complex signal described by the following equation (16).
Further, the complex signal on the reference plane is given by the following equation (10).
From the equations (16) and (17), the following equation (18) is given.
Here, * represents a complex conjugate. Since | A ₁ | ² r (x, y) in equation (18) is a real function, the following equation (19) is obtained when the complex logarithm is calculated.
By such calculation, it is possible to separate the reflectance r (x, y) of the object surface, which is an unnecessary component, and the phase information φ (x, y) to be extracted.

Next, a method for converting the extracted phase information into height will be described. From the similar relationship between ΔBCO and ΔBMB ′ in FIG. 13, the following equation (20) is established.
S _M = D · H / (H + Z _s ) (20)
Further, when Expression (20) is substituted into φ (x, y) = 2πf ₀ S (x, y) in Expression (12), the following Expression (21) is established.
H = Z _s φ (x, y) / {2πf ₀ D−φ (x, y)} Expression (21)
However, the phase information that can be observed is φ _w (x, y) folded between −π and + π. It is necessary to implement phase connection, such as by using phase continuity. This corresponds to determining the order m of the following equation (21).
φ (x, y) = 2π · m + φ _w (x, y) (21)
In normal phase connection, the continuity of the shape is assumed. However, in a scene where there are a plurality of objects as shown in FIG. 1, since the shape of the objects is discontinuous, the assumption does not hold and phase connection cannot be performed correctly. The above is the principle of phase information calculation and depth measurement by the Fourier transform method.

The process of the phase information calculation step (S20) based on the principle of the Fourier transform method will be described. In the Fourier transform of S21, the captured image is Fourier transformed for each scanning line. The fundamental frequency component extraction of S22, extracts the filters only data near the fundamental frequency f ₀ in the frequency domain of the data after the Fourier transform. The fundamental frequency f ₀ is obtained from the pitch p on the reference plane of the projected vertical stripe pattern by Expression (11). The shift to S23 in the origin, the data near f ₀ the extracted shifted to the origin. Data in the vicinity of f ₀ shifted by the origin is subjected to inverse Fourier transform in S24, and phase information is extracted in the phase information extraction in S25.

FIG. 14 shows an example of a result of calculating the folded phase information φ _w (x, yi) based on the captured image 63 on which the pattern is projected in FIG. The phase information from the data on the scan line yi of the image is shown in the lower graph. Although the detailed shape change is obtained from the phase information of the cylinder 64 and the prism 65, it can be seen that the phase cannot be connected because the position of the object is discontinuous.

  Next, in S30, a defocus amount calculation step including defocus amount calculation (S31) and defocus amount reliability calculation (S32) is executed. In the defocus amount calculation of S31, the contrast value or the standard deviation value in the region near the target pixel is used for the calculation of the defocus amount.

  FIG. 15A shows a calculation method and result in the case of a contrast value. A region 66 within a predetermined range near the target pixel is an example of a contrast value calculation region. The contrast value of equation (1) is calculated within this defocus amount calculation area. The calculation area is raster scanned, and the contrast value is calculated in the entire area of the image. However, the contrast calculation area needs to be performed in a range wider than the projection pattern pitch p on the reference plane. This is because if the above condition is not satisfied, the maximum brightness value and the minimum brightness value may not be included in the region, so that the contrast cannot be evaluated correctly. In FIG. 15A, the calculation result of the contrast at the position where the y coordinate of the image is yi is described in the lower graph. Since the pattern is in an imaging relationship on the reference plane that is the focus area, the contrast is high. Since defocusing occurs between the prism 64 at the front and the cylinder 64 at the front, the contrast is lowered. Since the defocusing of the pattern is larger in the cylinder 64 in front, the contrast value is lower.

  On the other hand, FIG. 15B shows a calculation method and result in the case of the standard deviation value. A region 67 within a predetermined range near the target pixel in the figure is a standard deviation value calculation region. The standard deviation value of Expression (2) is calculated within this standard deviation value calculation area. The calculation area is raster scanned, and the standard deviation value is calculated in the entire area of the image. However, the standard deviation value calculation region needs to be performed in a range wider than the projection pattern pitch p on the reference plane. This is because if the above condition is not satisfied, there is a possibility that the variation in luminance within the region cannot be correctly evaluated. The calculation result of the standard deviation value is described in the graph below. Since the pattern is in an imaging relationship on the reference plane that is the in-focus area, the variation in image luminance increases and the standard deviation value increases. Since defocusing occurs in the prism 65 at the front and the cylinder 64 at the front, the variation in image luminance is reduced and the standard deviation value is reduced. Since the cylinder 64 at the front has a larger defocus amount, the standard deviation value is lower.

  Regardless of whether the contrast value or the standard deviation value is used, the contrast value and the standard deviation value are higher than the value corresponding to the original depth at the boundary between the objects or at the portion where the direction of the surface changes abruptly. There are three possible causes for this. The first is a difference in reflectance caused by a difference in the surface direction of the object, the second is a difference in distance from the projection unit to the object, and a third is a difference in distance from the object to the imaging unit 2.

  First, the difference in reflectance caused by the difference in the surface direction of the object, which is the first reason, will be described. The surface becomes brighter as the incident direction of light from the projection unit to the object and the direction of the photographing unit 2 are closer to the regular reflection angle. Conversely, the farther from the specular direction, the darker the surface. This does not occur if the object surface is fully diffused, but this phenomenon occurs on other surfaces. Next, the difference in distance from the projection unit to the object will be described. The intensity of light from the projection unit attenuates in proportion to the distance. For this reason, the smaller the distance from the projection unit to the object, the higher the incident light intensity, and the brighter the object surface. Conversely, the greater the distance from the projection unit to the object, the lower the incident light intensity, and the darker the object surface.

Next, the difference in distance from the object to the photographing unit 2 will be described. The reflected light intensity from the object is also attenuated in proportion to the distance. The smaller the distance from the object to the photographing unit 2, the brighter the object surface. On the other hand, the larger the distance from the object to the photographing unit 2, the darker the object surface.
In an actual scene, these three phenomena occur in combination. Such a portion is considered to have low reliability of the defocus amount calculation result. Therefore, the reliability is evaluated in the defocus amount reliability calculation in S32.

The defocus amount reliability P (x, y) is calculated using the following equation (22).
P (x, y) = Df (x, y) −Df (x−1, y) Expression (22)
Df (x, y) stores a defocus value at x = x and y = y. The above equation (22) corresponds to taking the difference (differentiation) of the defocus value. That is, the absolute value of P (x, y) increases when the defocus value change amount is large. If the absolute value of P (x, y) is equal to or less than a predetermined threshold, the reliability is high and the defocus amount is used as it is. On the other hand, if it is larger than the threshold value, the reliability is assumed to be low, and correction processing is performed. In the following, this threshold value is represented by Pt, which is a reference for evaluating reliability. The correction process is performed in the phase information correction (S43) in the phase connection process of S40. Details will be described later.

  FIGS. 16B and 16C are graphs showing the calculation result of the defocus value Df and the calculation result of the defocus reliability P (x, y) in the image 63. In the region where the change in the defocus value is large, the absolute value of the defocus reliability increases, and the reliability of the defocus value decreases.

In S40, the three-dimensional shape calculation unit 9 executes a phase connection process including phase order calculation (S41), phase connection (S42), and phase information correction (S43). In the phase order calculation in S41, first, the defocus amount Df (x, y) is converted into the depth value Z (x, y) from the result of the calibration performed in advance. The phase φ (x, y) is obtained using the following equation (23) obtained by modifying the equation (21) from the depth value Z (x, y) (using the relationship of Z = L−H).
φ (x, y) = 2πf ₀ D (1−Z (x, y) / L) (23)
The order m is calculated from the obtained φ (x, y) using the following equation (24).
m = Int [φ (x, y) / 2π] (24)
Here, Int [] means to cut off the decimal point. FIG. 16D shows a graph of the calculated order m. In the region where the reliability of the defocus value is lower than the reference, the phase order is not calculated. In the graph, the region is indicated by hatching. In the phase connection in S42, the phases are connected based on the calculated phase order m and the phase information extraction result φ _w (x, y) in S25. Specifically, the following formula (25) is used.
φ _m (x, y) = 2π · m + φ _w (x, y) Expression (25)
Expression (25) is the same form as Expression (21), but includes a region where the reliability of the defocus value is lower than the reference, and thus is different from the original φ (x, y). In order to distinguish, the phase when the reliability of the defocus value is lower than the reference is described as φ _m (x, y).

FIG. 16E shows a graph of the calculation result of φ _m (x, y). The region where the reliability of the defocus value is lower than the standard is indicated by hatching. In the phase information correction in S43, the missing information in the region where the reliability of the defocus value of φ _m (x, y) is lower than the reference is corrected to obtain the original φ (x, y). Missing correction information using the continuity of the phase phi _w before connection. Note that FIG. 16F is a graph reflecting the order m in FIG.

Next, the correction method will be specifically described with reference to FIG. FIG. 17A is a graph showing the phase before connection corresponding to FIG. FIG. 17B shows a graph of the phase φ _w (x, yi) and dφ _w (x, yi) / dx, which is the differential value (difference value) of the phase φ _w (x, yi). A phase discontinuity occurs at a position where the absolute value of the differential value of the phase is large, x = x _V1 , x _V2 , x _V3 . Further, the surface inclination changes at the position x = x _I1 where the sign of the phase differential value is reversed. These points are called singular points of phase change.

Since the phase discontinuity does not occur in the region other than the singular point where the phase change is small, the continuity of the shape is guaranteed. Therefore, it is possible to correct a region lacking φ _m (x, yi) using the continuity of the phase φ _w (x, yi). The graph of φ _m (x, yi) in FIG. 17C is blank because a graph in a region where the reliability of the defocus value is lower than the reference is not drawn. In such a case, correction is performed using the continuity of φ _w from the vicinity of the blank toward the singular point. Specifically, this operation is continued until the singular point crosses in the direction of the arrow in the figure. The corrected phase is shown as a corrected phase φ (x, yi) in FIG.

Finally, in the shape calculation step of S50, the three-dimensional shape calculation unit 9 calculates the shape. The shape calculation step includes shape information calculation (S51). In the shape information calculation of S51, phase information φ (x, y) is converted into depth information Z (x, y) based on the following equation (26) obtained by modifying equation (23).
Z (x, y) = L (1−φ (x, y) / (2πf ₀ D)) (26)
FIG. 17E shows the shape information finally calculated. It can be seen that detailed shape measurement can be performed on an object at a discontinuous depth by only one pattern projection. As described above, in the present embodiment, shape information can be calculated.

[Embodiment 2]
FIG. 18 shows a schematic diagram of a shape measuring apparatus 200 according to Embodiment 2 of the present invention. Compared with the configuration of the first embodiment shown in FIG. 1, a projection pattern switching unit 14 is added. In the second embodiment, a phase shift method is used to acquire phase information. In the phase shift method, it is necessary to project three or more types while shifting the phase of the sinusoidal pattern shown in FIG. In the second embodiment, four types of sinusoidal patterns are projected while shifting the phase by π / 2.

  Examples of these four types of patterns are as shown in FIG. FIG. 19 shows cases where the phase shift amounts are 0, π / 2, π, and 3π / 2, respectively. By projecting such four types of patterns having different phases, it is possible to obtain phase information that is hardly affected by the target texture. Further, in the calculation of the defocus amount in the defocus amount calculation unit 8, it is possible to calculate the defocus amount with high reliability. However, since there are four types of patterns to be projected, there is a possibility that the subject moves during shooting while switching the patterns. Therefore, in order to cope with a fast-moving subject, a projection unit capable of switching patterns at high speed and a photographing unit 2 having a high frame rate are required.

  As a projection unit capable of switching patterns at high speed, a projector using a DMD as a display element, a projector using a liquid crystal display element having high-speed response, and the like can be considered. As the photographing unit 2 having a high frame rate, a commercially available high-speed camera may be used. If a high-speed camera is used, image acquisition at 100 fps or 1000 fps is also possible. Further, since the observable phase is 0 to 2π as in the Fourier transform method, phase connection is necessary when a phase change exceeding the phase is generated. Since the pre-calibration performs the same operation as the shape measuring apparatus 100 of the first embodiment, the description thereof is omitted. It is assumed that the relationship between the defocus amount and the depth position is acquired by calibration.

  A procedure for shape measurement by the shape measurement apparatus 200 according to the second embodiment will be described. The shape measurement is performed according to the flowchart shown in FIG. In the processing corresponding to the flowchart, the pattern projection unit 1, the photographing unit 2, the phase information calculation unit 7, the defocus amount calculation unit 8, the three-dimensional shape calculation unit 9, and the projection pattern switching unit 14 execute corresponding processing programs. It is realized by doing.

  The image capturing process of S10 'includes pattern projection (S11), image capturing (S12), number of captured images determination (S13), and pattern phase π / 2 shift (S14). The image capturing process in S10 'differs from the image capturing process in the first embodiment in that the number of captured images is determined in S13 and the pattern phase π / 2 shift in S14 is added.

  In S11, the projection unit 1 projects a reference phase pattern. In the image shooting in S12, the shooting target scene is imaged by the shooting unit 2. In S13, it is determined whether or not the number of shots is less than four. If the number of shots is four or more, the process proceeds to the next phase information calculation step (S20 ') and defocus amount calculation step (S30'). If the number of shots is less than 4, the process proceeds to S14. In S14, the phase of the pattern is shifted by π / 2 using the projection pattern switching unit 14. Thereafter, pattern projection in S11 and image photographing in S12 are performed. In this way, in the image capturing process of S10 ', four images having different projection pattern phases by π / 2 are obtained.

The subsequent S20 ′ phase information calculation step includes phase information calculation (S26) using a phase shift image. The process performed in S26 will be described with reference to FIG. In FIG. 21, the horizontal axis represents the phase and the vertical axis represents the image luminance value. Here, the luminance value of the pixel at the same coordinate position (x, y) is extracted for each captured image with phase shift amounts of 0, π / 2, π, and 3π / 2, and plotted at the corresponding position. When the image coordinates are (x, y), the luminance values at 0, π / 2, π, 3π / 2 are respectively represented by I ₀ (x, y), I _{π / 2} (x, y), I _π ( x, y) Let I _{3π / 2} (x, y). At this time, the phase information φ _w (x, y) at the position of the image coordinates (x, y) can be calculated by the following equation (27).
φ _w (x, y) = tan ⁻¹ {I _{3π / 2} (x, y) −I _{π / 2} (x, y)} / {I ₀ (x, y) −I _π (x, y)} ... Formula (27)
In the Fourier transform method, the processing is completed only with the luminance fluctuation according to the change in the projection pattern of the pixel of interest, so that it is possible to calculate the phase that is strong against the fluctuation of the target texture.

  In the subsequent S30 ′, a defocus amount calculation step including defocus amount calculation (S31), defocus amount reliability calculation (S32), processing number determination (S33), image change (S34), and result integration (S35) is performed. This is performed by the defocus amount calculation unit 8. In the defocus amount calculation step of S31, as in the first embodiment, the contrast value or the standard deviation value in the vicinity region of the target pixel is used for calculating the defocus amount, and thus the description thereof is omitted. The defocus amount reliability calculation in S32 also calculates the difference (differentiation) of the defocus value, as in the first embodiment, and a description thereof will be omitted.

  In the process number determination S33 of S33, it is determined whether or not the process number is less than four. If the number of processed sheets is 4 or more, the process proceeds to S35 as a result integration. When the number of processed sheets is less than 4, the process proceeds to S34 image change. In S34, the processing target is changed to a captured image of a projection pattern for which defocus amount calculation is not performed. In S35, the calculation result of the defocus amount and the calculation result of the defocus reliability performed with different projection patterns are integrated.

  Here, an image having a phase shift amount of 0 is set as an initial image, and a region having a defocus reliability lower than the reference is extracted. If the defocus reliability of this region is equal to or higher than the reference for images with different phase shift amounts, the defocus value at that time is overwritten on the result of the initial image. By repeating this operation for images having different phase shift amounts and integrating defocus amounts having a defocus reliability equal to or higher than the reference, a region where the defocus reliability is lower than the reference is reduced. By integrating the results of the captured images of a plurality of patterns, it is possible to select data in a region with high defocus reliability, and thus it is possible to calculate a defocus amount with high accuracy. In addition, regarding the phase connection process of S40 and the shape information calculation process of S50, since the same operation as Embodiment 1 is performed, description is abbreviate | omitted.

  According to the second embodiment described above, although the number of shots is increased to four, it is possible to calculate a phase that is resistant to texture fluctuation and to calculate a defocus amount with high reliability.

[Embodiment 3]
The shape measuring apparatus according to the third embodiment of the invention has the same configuration as that of the second embodiment shown in FIG. The patterns projected by the pattern projection unit 1 are two types of rectangular wave patterns shown in FIGS. If the period of the rectangular wave pattern is p and the rectangular wave in FIG. 22A is the reference phase, the pattern in FIG. 22B is shifted in phase by p / 2. While the projection pattern switching unit 14 switches the pattern projected by the pattern projection unit 1, the imaging unit 2 captures an image of the shooting target scene on which each pattern is projected.

  The defocus amount calculation unit 8 calculates the defocus amount at each pixel of the captured image using the contrast value represented by Expression (1), the standard deviation value represented by Expression (2), and the like.

Next, a method of calculating phase information in S20 ′ of FIG. 20 by the phase information calculation unit 7 in the present embodiment will be described. Here, the pattern of FIG. 22 (a) is projected and the image taken is I _Rp (x, y), and the pattern of FIG. 22 (b) is projected and the image taken is I _Rn (x, y).

First, I _Rp (x, y) and I _Rn (x, y) are compared in size to generate a binarized image B (x, y). When I _Rp (x, y)> = I _Rn (x, y), B (x, y) = 1, and when I _Rp (x, y) <I _Rn (x, y), B (x, y) ) = 0. FIG. 22C shows a generation example of the binarized image B (x, y).

  Next, an edge image E is generated from the binarized image B (x, y) by the following equation (28).

E (x, y) = B (x, y) −B (x + 1, y) (28)
The value of 1 or −1 is given to the edge position by the calculation of Expression (28). Next, the edge positions at both ends of the target pixel x are searched. This is done by searching for the closest 1 or -1 value on the edge image E. Assuming that the searched left edge position is E _l and the right edge position is E _r , the phase φ _w (x, y) is calculated by the following equation (29).

φ _w (x) = 2π · (x−E _l ) / (E _r −E _l ) −π (29)
By calculating this with respect to all y coordinates, phase information φ _w (x, y) is obtained.

In the three-dimensional shape calculation unit 9, the phase order m is determined by Expression (24) based on the defocus amount calculated by the defocus amount calculation unit 8. Further, the phase information φ _w (x, y) calculated by the phase information calculation unit 7 according to the equation (25) is connected to obtain the connection phase φ _m (x, y). Phase correction is performed as necessary, and depth information Z (x, y) is calculated using equation (26) to obtain the three-dimensional shape information of the subject.

  According to the third embodiment described above, although the number of shots is increased to two, it is possible to calculate a phase that is strong against texture fluctuation of the subject.

[Embodiment 4]
The shape measuring apparatus according to the fourth embodiment of the invention has the same configuration as that of the second embodiment shown in FIG. However, in the present embodiment, the patterns projected by the pattern projection unit 1 are two types of triangular waves shown in FIGS. The triangular wave in FIG. 23B is obtained by inverting the brightness in FIG. While the projection pattern switching unit 14 switches the pattern projected by the pattern projection unit 1, the imaging unit 2 captures an image of the shooting target scene on which each pattern is projected.

  The defocus amount calculation unit 8 calculates the defocus amount in each pixel of the captured image using the contrast value represented by Expression (1), the standard deviation value represented by Expression (2), and the like.

Next, a method of calculating phase information in S20 ′ of FIG. 20 by the phase information calculation unit 7 in the present embodiment will be described. Here, it is assumed that the captured image when the pattern of FIG. 23A is captured is I _Tp (x, y), and the captured image when the pattern of FIG. _{23B is} captured is I _Tn (x, y). . At this time, the intensity ratio image I _ratio is calculated by the following equation (30).

I _ratio = I _Tp (x, y) / I _Tn (x, y) (30)
The calculation result of the intensity ratio image I _ratio is shown in FIG. When the period of the triangular wave is p, the phase information φ _w (x, y) is calculated by the following equation (31).

φ _w (x, y) = 2π · I _ratio · (1−x / p−int (x / p)) − π (31)
Here, int () means an operation of truncating the decimal part. The value of the intensity ratio image I _ratio is normalized by Expression (31) so as to change linearly between −π to π. An example of the calculation result of φ _w (x, y) is shown in FIG. In the three-dimensional shape calculation unit 9, the phase order m is determined by Expression (24) based on the defocus amount calculated by the defocus amount calculation unit 8. Further, the phase information φ _w (x, y) calculated by the phase information calculation unit 7 according to the equation (25) is connected to obtain the connection phase φ _m (x, y).

  Phase correction is performed as necessary, and depth information Z (x, y) is calculated using equation (26) to obtain the three-dimensional shape information of the subject.

  According to the fourth embodiment described above, although the number of shots is increased to two, it is possible to calculate a phase that is strong against texture fluctuation of the subject.

[Embodiment 5]
A schematic top view of a shape measuring apparatus 300 according to Embodiment 5 of the present invention is shown in FIG. The difference from the configuration of the first embodiment shown in FIG. 1 is that, instead of the phase information calculation unit 7, a local arrangement information calculation unit 71 that calculates the local arrangement information of the pattern from the brightness change of the pattern of the captured image. It is a point used.

The pattern projected by the pattern projecting unit 1 is a random pattern shown in FIG. The imaging unit 2 captures an image of a scene to be photographed on which a pattern having randomness shown in FIG. Let the captured image at this time be I _Rand (x, y).

The local arrangement information calculation unit 71 performs four processes: binarization processing, encoding processing, correlation calculation processing, and candidate selection processing. In the binarization processing, the captured image I _Rand (x, y) is binarized by threshold processing according to the brightness of the image. In the encoding process, an encoded image Code (x, y) is generated based on the result of the binarization process. The sign “1” is assigned to the bright part and the sign “0” is assigned to the dark part. In the correlation calculation process, first, as shown in FIG. 25 (b), a predetermined range (-Rc to + Rc) near the target pixel of the encoded image is cut out. Then, the correlation calculation with the random pattern RandCode (x) is performed on the clipped region by the following equation (32).
According to the above equation, the smaller the correlation value Core (x), the higher the correlation. FIG. 25C shows an example of the result of the correlation calculation. In the candidate selection process, a plurality of candidates are selected based on the correlation calculation result. In the example shown in FIG. 25C, two candidates, Min1 and Min2, are selected. In such a case, since a plurality of candidates are selected, the depth cannot be uniquely determined.

  Therefore, in order to narrow down candidates, the defocus amount obtained by the defocus amount calculation unit 8 is used. The defocus amount calculation unit 8 calculates the defocus amount at each pixel of the captured image using the contrast value represented by Expression (1), the standard deviation value represented by Expression (2), and the like. The relationship between the position on the captured image, the defocus amount, and the position on the projection pattern is obtained by prior calibration. Therefore, in the example shown in FIG. 25C, the area is narrowed down to an area Cand surrounded by a dotted frame. Thereby, one candidate Min1 is selected from the two candidates, and the depth is obtained. The depth information Z (x, y) of the shooting target scene is calculated by performing the above process on the entire range of the image.

  According to the fifth embodiment described above, it is possible to measure the three-dimensional shape of the subject even if the pattern to be projected is not a periodic pattern.

[Embodiment 6]
The configuration of the shape measuring apparatus according to the sixth embodiment of the invention is the same as that of FIG. In the first to fifth embodiments, the focus of the pattern projection unit 1 is adjusted to the reference plane 3, and the defocus amount is calculated using blurring by the projection optical system 12. However, in the sixth embodiment, the focus is adjusted to the reference plane 3 using the imaging optical system of the imaging unit 2, and the defocus amount is calculated using the blur caused by the imaging optical system.

  In order to separate the defocusing by the imaging optical system of the imaging unit 2 and the defocusing by the projection optical system of the pattern projection unit 1, it is necessary to project under conditions that allow the defocusing of the projection optical system to be ignored. Specifically, it is necessary to sufficiently stop the aperture of the pattern projection unit 1 and deepen the depth of field. According to the measurement range, the F value of the projection optical system is appropriately set using the calculation formulas (3) and (4) for the depth of field and the calculation formula (5) for the limit of blur due to diffraction.

Since the focus of the imaging unit 2 is adjusted to the reference plane 3, the defocus amount increases as the distance from the reference plane increases. The defocus amount calculation unit 8 calculates the defocus amount in each pixel of the captured image using the contrast value represented by Expression (1), the standard deviation value represented by Expression (2), and the like. The phase information calculation unit 7 calculates the phase information φ _w (x, y) from the captured image of the subject on which the sine wave is projected, by the Fourier transform method, as in the first embodiment.

In the three-dimensional shape calculation unit 9, the phase order m is determined by Expression (24) based on the defocus amount calculated by the defocus amount calculation unit 8. Further, the phase φ _w (x, y) calculated by the phase information calculation unit 7 according to the equation (25) is connected to obtain the connection phase φ _m (x, y). In the sixth embodiment, a method of calculating one phase of a sine wave as a projection pattern and calculating the phase by the Fourier transform method has been described as an example. However, a method of projecting a random pattern with the same configuration as that of the fifth embodiment can also be used. In addition, a method of projecting a plurality of sine waves having different phases, a method of projecting a rectangular wave, a method of projecting a triangular wave, etc. can be used with the same configuration as in the second to fourth embodiments to which a pattern switching unit is added. .

  The sixth embodiment described above enables three-dimensional shape measurement even when the imaging optical system does not satisfy the pan focus condition.

[Embodiment 7]
The configuration of the shape measuring apparatus according to the seventh embodiment of the invention is the same as that of FIG. In the first to fifth embodiments, the focus of the pattern projection unit 1 is adjusted to the reference plane 3, and the defocus amount is calculated using blurring by the projection optical system 12. In the sixth embodiment, the focus of the imaging unit 2 is adjusted to the reference plane 3, and the defocus amount is calculated using the blur caused by the imaging lens 22. However, in the seventh embodiment, the defocus amount is calculated using a composite blur that is a combination of the blur caused by the projection optical system of the pattern projection unit 1 and the blur caused by the imaging optical system of the imaging unit 2.

The pattern projection unit 1 and the imaging unit 2 are focused on the reference plane 3 or behind the reference plane 3. By doing so, since the composite blur increases as it moves away from the reference plane, it is not necessary to calculate the defocus amount by separating the blur of the pattern projection unit 1 and the imaging unit 2. The defocus amount calculation unit 8 calculates the defocus amount in each pixel of the captured image using the contrast value represented by Expression (1), the standard deviation value represented by Expression (2), and the like. The phase information calculation unit 7 calculates the phase information φ _w (x, y) from the captured image of the subject on which the sine wave is projected, by the Fourier transform method, as in the first embodiment.

In the three-dimensional shape calculation unit 9, the phase order m is determined by Expression (24) based on the defocus amount calculated by the defocus amount calculation unit 8. Further, the phase information φ _w (x, y) calculated by the phase information calculation unit 7 according to the equation (25) is connected to obtain the connection phase φ _m (x, y). In the seventh embodiment, a method of calculating one phase of a sine wave as a projection pattern and calculating a phase by a Fourier transform method has been described as an example. However, a method of projecting a random pattern with the same configuration as that of the fifth embodiment can also be used. In addition, a method of projecting a plurality of sine waves having different phases, a method of projecting a rectangular wave, a method of projecting a triangular wave, etc. can be used with the same configuration as in the second to fourth embodiments to which a pattern switching unit is added. .

  According to the seventh embodiment described above, three-dimensional shape measurement can be performed even if the imaging optical system and the projection optical system do not satisfy the pan focus condition.

  The three-dimensional shape measuring apparatus according to the present invention can be used for various applications. In the field of CG or the like, modeling of an actual object can be performed. This makes it possible to automate modeling that has been performed manually in the past, and to improve the efficiency of CG production. Since it is possible to calculate the distance with high accuracy with a small number of patterns, it can also be used as a vision for a robot hand that requires a high-speed response. In addition, since shape measurement with a relatively small number of patterns is possible, shape measurement can be performed even on a moving subject. When combined with a small projector, it can be mounted on a small imaging device such as a digital camera, and can be used as input information for advanced processing using a three-dimensional shape.

  In addition, the present invention can realize the same processing as in the first to seventh embodiments with a computer program. In this case, each of the components such as blocks 7 to 9 and 14 in FIGS. 1 and 18 may be functioned by a function or a subroutine executed by the CPU. In general, the computer program is stored in a computer-readable storage medium such as a CD-ROM, and is set in a reading device (such as a CD-ROM drive) included in the computer and copied or installed in the system. Become executable. Therefore, it is obvious that such a computer readable storage medium is also within the scope of the present invention.

(Other examples)
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, etc.) of the system or apparatus reads the program. It is a process to be executed.

Claims (14)

Pattern projection means for projecting a pattern having periodicity onto a measurement space;
Imaging means for photographing the measurement space on which the pattern is projected,
The measurement space is defined by a reference plane, a projection range of the pattern projection unit, and an imaging range of the imaging unit, and the imaging unit images the measurement space so as to be focused on the reference plane, and the measurement space A three-dimensional shape measuring apparatus for measuring a three-dimensional shape of a measurement object existing in
Phase information calculating means for calculating phase information of the pattern from a change in brightness of the pattern of the photographed image obtained by the photographing means;
Phase information correction means for correcting the calculated phase information;
A blur amount calculating means for calculating a blur amount of the pattern in the captured image;
3D shape calculation means for calculating the 3D shape of the measurement object based on the phase information and the amount of blur when the relative positional relationship between the measurement object and the 3D shape measurement device does not change; With
The phase information correction unit converts the blur amount into depth information of the measurement space, calculates a phase order corresponding to the depth information, corrects the phase information based on the order,
The three-dimensional shape calculation unit is characterized in that the three-dimensional shape of the measurement target is calculated based on the corrected phase information. The blur amount calculation means calculates a change amount of the blur amount, excludes a blur amount whose change amount is higher than a threshold,
The three-dimensional shape measurement apparatus according to claim 1, wherein the phase information correction unit converts a blur amount whose change amount is equal to or less than the threshold value into the depth information. The pattern projected by the pattern projecting means is one type of pattern having periodicity,
The phase information calculation means performs Fourier transform on the captured image for each scanning line, extracts data in the vicinity of the fundamental frequency of the periodic pattern from the data after Fourier transform, and performs inverse Fourier transform on the extracted data The three-dimensional shape measuring apparatus according to claim 1, wherein the phase information is obtained. The pattern projected by the pattern projecting unit is a pattern having three or more types of periodicity whose phases are shifted,
The imaging means performs imaging for each of the three or more types of projection patterns,
The phase information calculation unit acquires a luminance value at the same coordinate position for each of the captured images obtained for the three or more types of projection patterns, and each acquired luminance value and a phase shift amount corresponding to the luminance value The three-dimensional shape measurement apparatus according to claim 1, wherein the phase information is calculated based on a relationship between the phase information and the phase information. The blur amount calculating unit calculates a change amount of the blur amount for each of the captured images obtained for the three or more types of projection patterns, and integrates the blur amount whose change amount is equal to or less than a threshold value.
The three-dimensional shape measurement apparatus according to claim 3, wherein the phase information correction unit converts the integrated blur amount into the depth information.   6. The three-dimensional image according to claim 1, wherein the blur amount calculation unit calculates the blur amount based on a contrast value within a predetermined range in the vicinity of the target pixel of the captured image. Shape measuring device.   6. The tertiary according to claim 1, wherein the blur amount calculating unit calculates the blur amount based on a standard deviation value within a predetermined range in the vicinity of the target pixel of the captured image. Original shape measuring device.   The three-dimensional shape measurement apparatus according to claim 1, wherein the pattern having periodicity is a rectangular wave.   The three-dimensional shape measuring apparatus according to claim 1, wherein the pattern having periodicity is a triangular wave.   The three-dimensional shape measuring apparatus according to any one of claims 1 to 9, wherein the blur amount is calculated based on a defocus amount. Pattern projection means for projecting a pattern having periodicity onto a measurement space;
Imaging means for photographing the measurement space on which the pattern is projected,
The measurement space is defined by a reference plane, a projection range of the pattern projection unit, and an imaging range of the imaging unit, and the imaging unit images the measurement space so as to be focused on the reference plane, and the measurement space A three-dimensional shape measurement method using a three-dimensional shape measurement device for measuring a three-dimensional shape of a measurement target existing in
A phase information calculating unit calculating phase information of the pattern from a change in brightness of the pattern of the captured image obtained by the imaging unit;
A phase information correcting unit correcting the calculated phase information;
A step of calculating a blur amount of the pattern in the captured image;
When the relative positional relationship between the measurement target and the three-dimensional shape measurement apparatus does not change, the three-dimensional shape calculation unit calculates the three-dimensional shape of the measurement target based on the phase information and the amount of blur. It includes a step of, a,
In the step of correcting the phase information, the blur amount is converted into depth information of the measurement space, a phase order corresponding to the depth information is calculated, and the phase information is corrected based on the order,
In the step of calculating the three-dimensional shape, the three-dimensional shape of the measurement target is calculated based on the corrected phase information.   The computer program for operating a computer provided with the pattern projection means which projects a pattern, and an imaging means as each means of the three-dimensional shape measuring apparatus of any one of Claim 1 thru | or 10. Pattern projection means for projecting a random pattern onto the measurement space;
Imaging means for photographing the measurement space on which the pattern is projected,
The measurement space is defined by a reference plane, a projection range of the pattern projection unit, and an imaging range of the imaging unit, and the imaging unit images the measurement space so as to be focused on the reference plane, and the measurement space A three-dimensional shape measuring apparatus for measuring a three-dimensional shape of a measurement object existing in
Local arrangement information calculating means for calculating a plurality of local arrangement information of the pattern from a change in brightness of the pattern of the captured image obtained by the imaging means;
A blur amount calculating means for calculating a blur amount of the pattern in the captured image;
Based on one local arrangement information selected based on the amount of blur among the plurality of local arrangement information when the relative positional relationship between the measurement object and the three-dimensional shape measurement apparatus does not change. And a three-dimensional shape calculating means for calculating the three-dimensional shape of the measurement object. Pattern projection means for projecting a random pattern onto the measurement space;
Imaging means for photographing the measurement space on which the pattern is projected,
The measurement space is defined by a reference plane, a projection range of the pattern projection unit, and an imaging range of the imaging unit, and the imaging unit images the measurement space so as to be focused on the reference plane, and the measurement space A three-dimensional shape measurement method using a three-dimensional shape measurement device for measuring a three-dimensional shape of a measurement target existing in
A local arrangement information calculating step in which a local arrangement information calculating unit calculates a plurality of pieces of local arrangement information of the pattern from a change in brightness of the pattern of the captured image obtained by the imaging unit;
A blur amount calculating step of calculating a blur amount of the pattern in the captured image;
When the relative positional relationship between the measurement object and the three-dimensional shape measurement device does not change, the three-dimensional shape calculation means selects one of the plurality of local arrangement information selected based on the blur amount A three-dimensional shape calculating step for calculating a three-dimensional shape of the measurement target based on local arrangement information;
A three-dimensional shape measuring method characterized by comprising: