JP2002202122A - Calibration method for two-dimensional distance image sensor - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a calibration method in which a distance according to the direction of the beam of light of each element can be measured precisely in a distance image sensor which uses a time-of-flight measuring principle and which comprises a plurality of two-dimensionally arranged elements. SOLUTION: The calibration method for the distance image sensor which comprises a light receiving part containing the plurality of two-dimensionally arranged elements and which uses the time-of-flight measuring principle is provided with a step which finds a beam-of-light parameter prescribing the direction of the beam of light incident on each element and a step which finds a distance parameter up to an object according to the output signal of each element.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

The present invention relates to a method of calibrating a distance image sensor of a time-of-flight system.

[0002]

2. Description of the Related Art As a range image sensor that generates a range image by performing three-dimensional measurement, reference 1: R. Miyagawa and
T. Kanade, “CCD-Based Range-Finding Sensor,” IEEE
Transactions on Electron Devices, Vol.44, No.10, O
ctober 1997, pp. 1648-1652. In this sensor system, a light source irradiates a pulse light to an object whose three-dimensional shape is to be measured, and a charge amount related to a delay time of light reflected from the object using a photoelectric element arranged two-dimensionally. The three-dimensional shape of the object is measured based on the relationship between the measured charge amount and the delay time of light. Practically, such an element is composed of a fine element such as a CCD or a CMOS.
By arranging an optical component such as a lens between an element such as a CD or a CMOS and an object, it is necessary to enlarge a measurement area corresponding to an angle of view of a camera. Therefore, in order to actually calculate the phase difference of the pulsed light emitted from the light source and perform accurate shape measurement or distance measurement, it is necessary to correspond to each element including the optical components arranged in front of the photoelectric element. It is necessary to determine the beam direction.

[0003]

[0005] However, Document 1
Is basically configured as a one-dimensional line sensor. Further, since calibration is performed using a board on a plane, a light beam direction corresponding to each element is defined. Is not necessarily calibrated to calculate the distance at.

The present invention has been made in view of such a problem, and an object of the present invention is to provide a range image having a plurality of two-dimensionally arranged elements utilizing a time-of-flight measurement principle. It is an object of the present invention to provide a calibration method that can accurately measure a distance (range) according to a light beam direction of each element in a sensor.

[0005]

In order to achieve the above object, a first aspect of the present invention is to provide a distance image utilizing a time-of-flight measurement principle having a light receiving section including a plurality of elements arranged two-dimensionally. A method for calibrating a sensor, the method including a step of obtaining a light beam parameter defining a direction of a light beam incident on each of the above-described elements, and a step of obtaining a distance parameter to an object according to an output signal of each of the above-described elements. The above configuration corresponds to the first embodiment described below.

According to a second aspect of the present invention, in the method for calibrating a two-dimensional range image sensor, a result of measuring a member having a known three-dimensional shape is used to obtain the light ray parameter or the distance parameter. The above configuration corresponds to the first embodiment described below.

Further, a third invention is directed to the second invention according to the second invention.
In the calibration method of the three-dimensional range image sensor, the measurement of the member having the known three-dimensional shape is performed by changing the distance between the member and the two-dimensional range image sensor a plurality of times. The above configuration corresponds to the first embodiment described below.

A fourth invention is directed to the second invention according to the third invention.
In the method for calibrating a three-dimensional distance image sensor, the member having a known three-dimensional shape includes a member having a marker having a plurality of holes or irregularities, and a member having a planar portion having no marker. I have. The above configuration corresponds to the first embodiment described below.

Further, a fifth invention is directed to the second invention according to the first invention.
In the calibration method of the three-dimensional distance image sensor, the step of obtaining the distance parameter uses a conversion table. The above configuration corresponds to the first embodiment described below.

A sixth invention is directed to the second invention according to the first invention.
In the method for calibrating a three-dimensional range image sensor, in addition to the range image sensor, a shooting device for shooting a member having a known shape is further arranged, and a positional relationship between the range image sensor and the shooting device is defined. The parameters and the parameters relating to the photographing device are calculated.
The above configuration corresponds to a third embodiment described below.

Further, a seventh invention is directed to the second invention according to the second invention.
In the calibration method of the three-dimensional distance image sensor, the light ray parameter and the distance parameter are calculated by three-dimensionally moving the member having the known three-dimensional shape. The above configuration corresponds to a third embodiment described below.

[0012]

DESCRIPTION OF THE PREFERRED EMBODIMENTS [First Embodiment] FIG. 1 is a diagram showing a part of a device configuration of a range sensor 1 as a range image sensor according to the present invention. In FIG. 1, a range sensor 1 is configured to photograph an object 2, and an input / output signal 17 is transmitted to the computer 3. As basic components of the range sensor 1, a timing controller 4 for controlling the overall timing of the range sensor 1, a pulse light source unit 5 for generating pulse light 12 for time of flight, and irradiating the pulse light 12 to the object 2 Light source optical system unit 6 for adjusting to pulse light 13 for the purpose, pulse light 14 reflected from object 2
Sensor unit optical system unit 8 for imaging the image on the sensor element array 7, the sensor element array 7 receiving the pulse light 15, the pulse light generated by the timing controller 4, and the pulse light input by the sensor element array 7. Signal processing of the pulsed light 15 having a time difference from
And a signal processing unit 9 for converting the signal into an input / output signal 16 that can be input, and an input / output interface 10 for outputting the signal processed as described above to the computer 3 and inputting an instruction from the computer 3.

The timing controller 4 receives a command from the computer 3 through the input / output interface 10, generates a timing signal for generating pulsed light, and resets and distributes the charge amount in the sensor element array 7. Generate a timing signal. As the sensor element array 7, a CCD as disclosed in Document 1 is used.
There are a device using a device, a device using a CMOS device, and the like. However, any other device may be used as long as it can store a charge amount related to an optical delay time.

In such a photoelectric conversion element of the sensor element array 7, a signal generated by the timing controller 4 is accumulated by the element for a delay time of light received by the photoelectric conversion element (for example, two gates). In principle, a value proportional to the light delay time (or a value of a certain function system with the light delay time as a variable) is sent out as an electric signal by taking the difference in the amount of charge to be distributed using.

However, regarding the calibration method described in the present embodiment, these elements may be elements that output a function f (t) of the optical delay time t as an electric signal. Here, f (t) may be a monotonically increasing or monotonically decreasing function related to t, and it should be noted here that a method other than the method described in Reference 1 can output such characteristics. .

Hereinafter, mathematical terms used in the present invention will be described.
Will be explained. FIG. 2 shows the structure of the sensor element array 7 in mathematical form.
It is shown in a typical way. The sensor element array 7 is two-dimensional
Are assumed to be uniformly distributed in a shape (not necessarily
It may not be original and may be one-dimensional). Sensor element
Is defined by the index (i, j)
And Here, i and j are integers, and i = 1, 2,.
_iJ = 1,2, ..., N_j It is. At this time, the pulse light source
The light pulse emitted from the knit 5 is reflected by the object 2
And the optical system 8 for the sensor element array in the range sensor 1
After forming an image through the sensor element array 7 (i, j)
To reach. After the pulsed light is emitted, the sensor element
The time to reach i7 is t_ij, Sensor element (i,
Let j) be the charge_ijThen f_ijIs f_ij(T_ij) At t_ij
Is a function of Of course f_ijIndicates slight variations in device characteristics
And the relative position between the light source and the sensor element.
So f_ijIs a function dependent on (i, j).

The calibration parameters for generating the distance image (range data) calculated in the present invention are stored in the signal processing unit 9 or the computer 3, and each sensor element unit is stored on the basis of the parameter values. Is converted to range data. However, the process of calculating (estimating) the calibration parameter itself described in the present invention is performed by a data processing device such as the computer 3.

On the other hand, the range sensor 1 has a unique reference coordinate system, and this coordinate system is now represented by R (x _R , y _R ,
z _R ). Calibration of the range sensor 1 in the present invention is performed by, when an output signal f _ij of each element (i, j) in the sensor element array 7 is given, the reflection on the object 2 corresponding to the f _ij. A method for calculating coordinate values (x _R , y _R , z _R ) of the point where the point has occurred in the R coordinate system and parameters for the calculation are estimated.

Here, the index in the sensor element array 7 is
(I, j), the physical on the sensor element array
Not only is the location determined, but its physical location and
Ray direction defined by the optical element unit 8 for the element array
Can be determined. That is, the element of each sensor element array
Focus the element and the focus of the optical system on the child (i, j)
There must be a reflection point on the object on a straight line (ray)
No. That is, for (i, j), a unique straight line L
Exists in the R coordinate system, and the straight line L is unique in the R coordinate system.
Can be determined. Therefore, the reflection point on the object
(X_R, Y_R, Z _R) Must be on the straight line L
Constraints and f_ijBy using the value of (x
_R, Y_R, Z_R) Can be determined.

In the following discussion, as the definition of the R coordinate system,
An R coordinate system is defined so as to correspond to the sensor element array 7. That the focal position of the sensor element array optical system unit 8 and the origin of the R, parallel to x _R-axis and the index i of the sensor element, and to take y _R-axis parallel to the index j, z _R axis to these It is assumed to be orthogonal.

Actually, an optical system lens is often used in the optical system unit 8 for the sensor element array, and it is also important to determine the straight line (light ray) L including the distortion of the lens. In some cases.

If the lens distortion of the optical system is negligible, the relationship with the point (x _R , y _R , z _R ) on the straight line corresponding to (i, j) is

[0023]

(Equation 1)

Can be represented by Where (α _u , α
_v ) is a parameter representing the magnification in the i and j directions,
(U _0, v ₀₎ represents the (i, j) z _{intersections R-axis} is the tolerance is a sensor array plane and the optical axis defining the.

If the lens distortion cannot be ignored, the equation becomes complicated,

[0026]

(Equation 2)

These formulations are described in reference 2 (J. We
ng, P. Cohen, and M. Herniou, “Camera calibration wi
th distortion models and accuracy evaluation, ”IEE
E Transactions on Pattern Analysis and Machine Int
elligence, Vol.14, No.10, October 1992, pp.965-98
0.), the description is omitted here.

In practice, it is difficult to directly define (measure) the R coordinate system from the outside of the range sensor 1. Therefore, a reference coordinate system is set separately from this coordinate system, and the reference coordinate system is set. It is convenient to calibrate the range sensor 1 while considering the relationship between the range sensor 1 and the R coordinate system.

Here, the world coordinate system W (x _W , y _W , z _W ) is considered as another reference coordinate system.
Assuming that a coordinate transformation parameter to a coordinate system is represented by a rotation matrix R and a translation vector T,

[0030]

(Equation 3)

Here, the parameters defining R are:
A Yaw (φ _x ) -Pitch (φ _y ) -Roll (φ _z ) angle or the like can be used, and the translation vector T is (t _x , t _y ,
_tz ) can be used.

Then, equation (3) becomes

[0033]

(Equation 4)

S = (α _u , α _v , u
₀ , v ₀ ; k ₁ , g ₁ , g ₂ , g ₃ , g ₄ ) are camera internal parameters, and e = (φ _x , φ _y , φ _z ; t _x , _ty ,
Let t _z ) be called the camera extrinsic parameter. Where R is

[0035]

(Equation 5)

Is represented by

Therefore, a more realistic calibration
Is such a parameter group p = (φ_x, Φ_y, Φ
_z; T_x, T_y, T_z; Α_u, Α_v, U₀, V₀;
k₁, G ₁, G_Two, G_Three, G_FourBy estimating
Consider performing calibration in the future.

Before describing the specific method of calibration, first, the internal parameters s = (α _u ,
α _v , u ₀ , v ₀ ; k ₁ , g ₁ , g ₂ , g ₃ , g ₄ ) and external parameters e = (φ _x , φ _y , φ _z ; t _x , _ty ,
_tz ) is known, the range sensor 1
The following describes how the distance image can be generated by the following.

As shown in FIG. 3, the world coordinate system W
In z _W = z ^m become the reference plane plate 18 (m = 1,2, ..., N
_M ), and when a pulsed light is applied to this flat plate,
The signal output from each element (i, j) is _defined as f _ij ^m . First, each element (i, j) obtains an intersection between a prescribed light beam and a reference plane plate of z _W = z ^{m in} a range sensor coordinate system. This involves estimating the ray for (i, j) with respect to (i, j) using equation (2).

[0040]

(Equation 6)

Z _R that satisfies is calculated. It should be noted here that z _W is known (set value) on the reference plane plate, and the unknowns in this equation are z _R , x _W , and y _W.
Individual. On the other hand, since the constraint order of the equation is also 3, a solution exists. From equation (8),

[0042]

(Equation 7)

By deriving and solving a linear equation represented by the following equation, z _R can be calculated. here,

[0044]

(Equation 8)

In this way, the output f _ij ^{m of the} sensor element
Reflection point for R coordinate system for _{(x R, y R, z} R)
Can be obtained as a measured value.

For each element (i, j), different criteria
The reflection point (x _R ^m, Y_R ^m, Z_R ^m)When
Sensor output signal f^mGiven that
Using the three-dimensional position (x_R,
y_R, Z_R) Can be calculated. For example, in FIG.
As shown, the sequence of measurement points (f^m, Z_R ^mAgainst)
Prepare a lookup table as a conversion table
Good. Then z_R ^mAnd z _R ^{m + 1}Is partially linear approximation
Assume that it is small enough to be possible. At this time, any
Z for output signal f_RIs calculated by the following method.
Can be. That is, given an arbitrary output signal f,
Come

[0047]

(Equation 9)

Thus, z _R can be calculated. After calculating the z _R In this way, it is possible to calculate the three-dimensional position in range sensor coordinate system by the equation (13).

The method described so far is generally a look-up table type distance data calculation method.
In the description so far, the lookup table processing has been performed by the partial linear approximation processing, but it is needless to say that nonlinear approximation such as a spline method may be used. Furthermore repeatedly measure f ^m when creating the table, by calculating the representative value of the measured value (e.g., a median or average value), it is possible to reduce the noise contained in the f ^m. In the discussion below, the measurement parameter group necessary to create the look-up table (f ^_m, z _{R m)}
( _M = 1, 2,..., N _M ) is called a distance parameter.

[0050]

(Equation 10)

Hereinafter, how to calculate such a ray parameter and a distance parameter will be described with reference to an embodiment.

FIG. 5 is a diagram showing an algorithm that embodies the calibration method. Step 1 is to calculate a ray parameter and a coordinate conversion parameter that define a ray direction for each element (i, j). Step 2 is to calculate and look up a distance parameter for defining a lookup table related to a distance. This is a step of creating an up table.

Step 1 FIG. 6 is a diagram showing an apparatus for realizing a method for calculating a ray parameter and a coordinate conversion parameter. Range sensor 1
Are arranged on the parallel rail 19, and the calibration board 20 is also arranged on the parallel rail 19. The parallel rail 19 and the calibration board 20 are defined in the world coordinate system, and the calibration board 2
0 is the parallel rail 19 on the parallel rail 19 in the world coordinate system.
It can be moved in parallel to. Here, for the sake of simplicity, it is assumed that the parallel rail 19 can move parallel to the z-axis of the world coordinate system. That is, the calibration board 20 can be arranged at a position indicated by z _W = z ^k .

FIG. 7 is a diagram showing the configuration of the calibration board 20. In the calibration board 20, there are calibration patterns of large circle holes and small circle holes as markers. The part having no marker is constituted by a white flat part. The five great circles are arranged symmetrically with respect to the x direction and the y direction in the world coordinate system to facilitate identification of the calibration pattern, and in the world coordinate system of all holes (patterns). Can be calculated in advance. The number of all holes When N _A, the center position in the world coordinate system of the ^{_{holes, (x a k, y a}} k, z a k) (a = 1,2, ..., N A)
It can be expressed as. Of course, it is assumed that the number of sensor elements is sufficiently larger than the number of holes.

FIG. 8 is a diagram for calculating a ray parameter.
It is a figure showing an algorithm. In step 1-1,
Index parameter k that controls body flow
Is initialized with 1. This parameter is
Parameter that specifies the position where the
is there. In step 1-2, z in the world coordinate system_W= Z^kof
The calibration board 20 is arranged at the position. Stay
In step 1-3, calibration is performed with the range sensor 1.
The board 20 is photographed and the signal output f from each element is taken. _ij ^kEnter
Power. In step 1-4, f_ij ^kIs a two-dimensional array
To analyze, extract and identify patterns, and carry
The center of the position of the hole in the blation pattern (u_a ^k, V_a ^k)
Is calculated. In step 1-5, the extracted and identified
The three-dimensional position of the turn in the world coordinate system (x_a ^k, Y_a ^k,
z_a ^k) To the two-dimensional position (u_a ^k, V_a ^k) And register.
In step 1-6, the value of k is increased by one. Step 1-
7, the index k is set to a predetermined N_KYo
Check whether it is greater than If not big
If so, the process proceeds to step 1-2. If big, step
Move to 1-8. In step 1-8, step 1-
5 three-dimensional position (x_a ^k, Y_a ^k, Z_a ^k) Is secondary
Original position (u_a ^k, V_a ^k)

[0056]

[Equation 11]

Hereinafter, Steps 1-4 and 1-8 which require more detailed explanation in the present algorithm will be described.

Step 1-4 For example, it is assumed that the number of sensor elements is N _i × N _j and is represented by a signal f _ij output from each element. At this time, f _ij is recognized as an abnormal value when the hole in the calibration board 20 is in the ray direction, and when other than the hole is in the ray direction,
A normal value related to the reflection from the calibration board 20 will be transmitted. Therefore, the entire output of the sensor element array can be defined as a two-dimensional array of f _ij . Light rays reflected by the plane portion having no hole on the calibration board 20 are near (i + 1, j), (i-1, j), (i, j-1), and (i-1, j) near the element (i, j).
A normal value is indicated by (i, j + 1) or the like. Therefore, the center position of the hole can be obtained by analyzing the values of the two-dimensional array formed by f _ij . Since such a method is a method often used for extracting a two-dimensional connected region in the image processing and the method, the description of the method is omitted here. It should be noted here that although there are large and small holes on the calibration board 20, it is easy to identify the holes on the f _ij two-dimensional array by finding the larger circle first. Can be.

[0059] By doing so, the calibration pattern (a = 1,2, ..., N A) position _{^{(u a k, v a k}} ) on the two-dimensional array with respect to seeking, in step 1-5 it can be determined that the position in the world coordinate system _{^{(x a k, y a k}} , z a k) a. Here _{^{(u a k, v a k}} ) is kept also subjected also possible that it is not necessarily at the grid point (integer value), ie real numbers.

Step 1-8

(Equation 12)

In this step, first, up to step 1-7
Unify the two-dimensional and three-dimensional coordinate value pairs obtained in
You. Specifically, all measurement point pairs are rearranged, and (u
_b, V _b, X_b, Y_b, Z_b) In a compact form
Represent. Where b of the index is
The index of a and k are integrated, and all
The fixed point is represented by one index. here
b = 1,2, ..., N_BAnd In step 1-8
Are these (u_b, V_b, X_b, Y_b, Z_b)
hand,

[0062]

(Equation 13)

Although this method is described in detail in Document 2, it will not be described in detail. However, in the camera calibration method, parameters (α _u , α _v , u ₀ , v ₀₎ related to the pinhole among the camera internal parameters. )
And the lens distortion parameters (k ₁ , g
₁ , g ₂ , g ₃ , g ₄ ) and further, (φ _x , φ _y , φ _z ; t _x , _ty , _tz ) relating to camera external parameters.

[0064]

[Equation 14]

Step 2 FIG. 9 is a diagram showing an apparatus relating to the calculation of the distance parameter and the creation of the look-up table in step 2.
The range sensor 1 is similar to the parallel rail 1
9 is mounted. On the parallel rail 19, a flat white board 21 is further installed. Similar to the calibration board 20 of step 1, the flat white board 21 can be moved in parallel and can be arranged so that the z position of the world coordinate system is z _W = z ^m . Now, the movement position of the whiteboard 21 is represented by z _W = z ¹ ,
z ^2, ..., and zN _M.

FIG. 10 is a diagram showing details of the algorithm of step 2. In step 2-1, 1 is substituted as the initial value of the index m used in the present algorithm. At step 2-2, installing the whiteboard 21 in a position of z _W = z ^m in the world coordinate system. Step 2-3
Then, a pulse light is irradiated from the range sensor 1 and a signal obtained by converting the pulse light reflected by the whiteboard 21 by each element (i, j) of the range sensor 1 is output signal f _ij.
Record as ^m . In step 2-4, the value of m is increased by one. At step 2-5, and compares m and N _M, if m is not greater than N _M, the process proceeds to step 2-2. Otherwise, the process proceeds to step 2-6.

[0067]

(Equation 15)

Here, step 2-6 which is important in the present algorithm will be described.

Step 2-6

(Equation 16)

(F _ij ^m , z _R ^m ) thus obtained is
It is created as a table as shown in FIG. By using the table created in this way, the distance measurement in the range sensor coordinate system can be performed by the method described above.

[0071]

[Equation 17]

(1) Variation in characteristics of each element is corrected,
(2) Distortion correction of an optical system unit arranged in the range sensor 1 can also be performed, and even the range sensor 1 having a two-dimensional element structure based on time-of-flight measurement principle can accurately perform three-dimensional measurement. A calibration method can be provided.

In the method described so far, the calibration board 20 as shown in FIG. 7 is arranged at a plurality of z positions. Of course, several variations of the calibration board 20 can be considered.

For example, as shown in FIG. 12, a plurality of boards may be prepared in which the size and interval of the pattern in the calibration board 20 are changed according to the position of z. In the calibration board 20 as shown in FIG. 12, the expected angle of view of the range sensor 1 is first assumed, and the hole of the pattern having the almost same size can be always extracted on the sensor element surface at the expected angle of view. It does. In this way, by designing the size and interval of the pattern while taking the angle of view of the range sensor 1 into consideration, the size of the holes in the element array (i, j) becomes substantially constant, and the extraction thereof becomes easier. In addition, the ability to detect holes can be improved.

In the description so far, the position of z _W = z ^k or z _W = z ^m is set by moving the calibration board or white board, but these boards are fixed and the range sensor 1 is fixed. Obviously, the same effect can be obtained even if a method of relatively moving is adopted.

[Second Embodiment] The first embodiment is effective when the number of elements in the array is larger than the number of patterns (holes) in the calibration board. However, when the number of elements in the array is extremely small, the accuracy of measuring the center of the hole may decrease. A method that can cope with such a case is the method described in the second embodiment. FIG. 13 is a diagram illustrating an apparatus configuration for realizing the second embodiment. In the figure, the range sensor 1 is fixed in the world coordinate system, and the calibration board 22 is composed of a white board having one circular hole. The calibration board 22 is mounted on an X stage 23, a Y stage 24, and a Z stage 25. These X Stage 2
3, the Y stage 24 and the Z stage 25 are controlled by using the control signal 26 generated by the computer 3 so that the center position of the hole in the calibration board 22 can be moved to an arbitrary position in the world coordinate system. Have been. More specifically, the center position of the hole in the calibration board 22 can be set to a plurality of predetermined (x _W , y _W , z _W ). That is, in the x, y, and z directions,

[0077]

(Equation 18)

It is possible to move to the position of the combination.

FIG. 14 shows the steps of the algorithm.
It is a flowchart. In step 3-1, the world coordinates
Let the coordinate value of z in the system be z_W= Z^kAnd where
Adjust to the z-coordinate of the center of the hole in the vibration pattern
As described above, the calibration board 2
Move 2 In step 3-2, the preset
X_W= X^m, Y_W= YⁿCalibration
Move so that the center of the hole, which is the pattern, comes. Stay
In steps 3-1 and 3-2, the center of the hole is (x^m, Y ⁿ,
z^k).

In step 3-4, the range sensor 1
Emits a pulsed light, and
The reflected light is input to the sensor element array, and as a result f_ij ^kTo
Output. This is the same as described in the first embodiment.
It is. In step 3-4, f_ij ^kSecondary consisting of
Extract the hole pattern from within the original array. This extraction method
Are the same as in the first embodiment. Thus extracted
Position the center of the hole at (u_a, V_a). Also in this position
The center 3D position of the hole in the corresponding world coordinate system (x _a, Y
_a, Z_a).

[0081]

[Equation 19]

By averaging, that is, by taking the average of effective measured values, more accurate measurement is performed.

In steps 3-5 and 3-6, X
It is checked whether the movement has been completed a predetermined number of times on the Y stage and the Z stage. If not, the process proceeds to step 3-2 or step 3-1. If not, proceed to the next step 3-7.

[0084]

(Equation 20)

Although the above description has been made using the calibration board 22 having a hole in a plane, various variations can be considered.
FIG. 15 shows one example. As shown in FIG. 15, a member having a known three-dimensional shape can be considered as the reference member 26 instead of the calibration board 22. In this example, four triangular planes A,
B, C, and D are arranged, and these planes intersect at one point.

Using the reference member 26 instead of the calibration board 22, the center of the calibration (u _a , v _a ) as the intersection point of the planes A, B, C, and D is moved while moving on the XYZ stage. To f _ij ^k
Extract from On the other hand _{(x a, y a, z} a) can be calculated from the amount of movement of the XYZ stage. Where f
intersection from ^{_{ij k (u a, v a}} ) As a method of calculating the carries a plurality of planes fitting the value of f _ij ^k from the vicinity calculation of f _ij ^k, by finding the intersection of the extracted plane equation , Just do it.

As described above, in the second embodiment, it is possible to simultaneously estimate a ray parameter and a distance parameter using the XYZ stage. First
Calibration is possible even with an array having a smaller number of elements than the sensor element array used in the embodiment. Further, step 1 and step 2 of the first embodiment can be integrated, which is a method suitable for automation of calibration.

[Third Embodiment] In the first and second embodiments, the calibration method of the range sensor 1 has been described. On the other hand, in many actual scenes, the distance image measured by the range sensor 1 and the normal CCD / CMO
It is often required to map a texture image on a distance image (or vice versa) by mapping a texture image (color or monochrome image) obtained from the S camera.

Hereinafter, this method will be described. FIG.
FIG. 9 is a diagram illustrating a configuration of a third embodiment. The configuration of the apparatus is almost the same as that of the first embodiment, except that a texture photographing camera (imaging apparatus) 27 for photographing a texture is mounted on the range sensor 1.

FIG. 17 is a flowchart showing the processing flow of this embodiment. Basically, steps 4-6 and 4-9 are newly added to the processing flow shown in FIG. 8, and these are used to adjust the calibration parameters (light ray parameters and coordinate conversion parameters) of the camera for texture photographing. This is a step for calculation.

In step 4-1, the entire flow is controlled.
Initialize the index parameter k to be rolled to 1.
You. This parameter is used to place the calibration board.
This is a parameter to specify the position to place. Step 4-
In 2, the z of the world coordinate system_W= Z ^kCalibrate to the position
Place an option board. In step 4-3, the range
Image the calibration board with sensor 1
Signal output from_ij ^kEnter In step 4-4
Is f_ij ^kIs analyzed as a two-dimensional array to extract patterns
And identify the holes in the calibration pattern
Center of position (u_a ^k, V_a ^k) Is calculated. Step 4-
In 5, the pattern extracted and identified by the range sensor 1
Three-dimensional position (x_a ^k, Y_a ^k, Z_a ^k)
Two-dimensional position (u_a ^k, V_a ^k) And register.

In step 4-6, the calibration pattern is photographed by the texture photographing camera, and the center of the hole of the calibration pattern is measured from the photographed image. The location within the image of the hole (u _c ^k, v
_c ^k ). On the other hand, the three-dimensional position (x _c ^k , y _c ^k , z _c ^k ) of the pattern corresponding to the extracted hole in the world coordinate system
And registers with the two-dimensional position ^{_{(u c k, v c k}} ).

In step 4-7, the value of k is increased by one.
In step 4-8, the index k is determined in advance.
N_KCheck if it is greater than. If large
If not, the process proceeds to step 4-2. If it ’s big,
Move to step 4-9. In step 4-9,
The three-dimensional position (x_a ^k, Y_a ^k,
z _a ^k) And two-dimensional position (u_a ^k, V_a ^k)

[0094]

(Equation 21)

Further, in step 4-10, step 4
Three-dimensional position of the texture picture camera registered -6 using _{^{(x c k, y c k}} , z c k) and two-dimensional position _{^{(u c k, v c k}} ) the camera internal texture for shooting The parameters and coordinate conversion parameters (external camera parameters) R ′ and T ′ from the world coordinate system to the texture photographing camera coordinate system and the coordinate conversion parameter _c H _s from the range sensor coordinate system to the texture photographing camera coordinate system are calculated. .

Hereinafter, a camera calibration method that requires a more detailed description of the present algorithm, and step 4
-6, step 4-10 will be described.

First, the mathematics required for camera calibration will be described. The coordinate system C and the world coordinate system W of the camera for texture photographing are defined, and the three-dimensional position (x _W , y _W , z _W ) in the world coordinate system W corresponds to the position (s, t) in the camera image. Assuming that

[0098]

(Equation 22)

The following relationship holds.

Here, R ′ (ψ _x , ψ _y , ψ _z ) =
(R _ij ′) and t ′ = (t _x ′, t _y ′, t _z ′) respectively represent a rotation matrix component and a translation vector component of the coordinate transformation from the world coordinate system to the texture photographing camera coordinate system. Represent. (Ψ _x , ψ _y , ψ _z ) represents the Yaw-Pitch-Roll angle in R ′.

In the camera calibration, g = (β
_s , β _t , s ₀ , t ₀ , a ₁ , b ₁ , b ₂ , b ₃ ,
b ₄ ) is a camera internal parameter, and h = (ψ _x , ψ _y , ψ
_{z, t x ', t y} ', t z ') is referred to as the extrinsic camera parameters.

Now, the image points (s ⁱ , t ⁱ ) corresponding to the three-dimensional point group (x _W ⁱ , y _W ⁱ , z _W ⁱ ) given in the world coordinate system
Given (i = 1, 2,..., N), the parameter g = (β _s , β _t , s ₀ , t ₀ , a ₁ , b ₁ , b ₂ , b
₃ , b ₄ ) and h = (ψ _x , ψ _y , ψ _z , t _x ′, t _y ′,
The method of calculating t _z ') is described in Reference 1 as described above.
And so on.

On the other hand, these parameters are calculated
And a three-dimensional point (x_W, Y _W, Z_W) From te
Calculate corresponding point (s, t) in camera coordinate system for texture shooting
To do this, the following method may be used. That is,
(1) First

[0104]

(Equation 23)

In step 4-6, a calibration pattern is photographed by a texture photographing camera. In the image, the plane portion serving as the background appears white, and the hole portion serving as the calibration pattern is easily distinguishable from the white plane portion. That is, by using the basic image processing algorithm, the center coordinates of the hole can be easily calculated from the captured image. Now, the two-dimensional coordinate value _{^{(s d k, t d k}} ). Here, k is an index representing the position of the calibration board, and d is an index representing the recognized calibration pattern (hole). Thus measured _{^{(s d k, t d k}} ) to the position (x _d ^k of the calibration pattern (hole) in the corresponding world coordinate system,
y _d ^k , z _d ^k ).

On the other hand, in step 4-10, step 4
( _Sd ^k , t _d ^k ) calculated at −6 and (x _d ^k , y
_d ^k , z _d ^k ), and the camera internal parameters g = (β _s , β _t , s ₀ , t ₀ , a ₁ , b ₁ , b ₂ ,
b ₃ , b ₄ ) and camera external parameters h = (Ψ _x ,
_{Ψ y, Ψ z, t x} ', t y', calculates a t _z ').

[0107]

(Equation 24)

[0108]

(Equation 25)

Is given by Therefore, the coordinate transformation matrix from the range sensor 1 to the texture photographing camera coordinate system is

[0110]

(Equation 26)

Can be calculated. By this coordinate conversion matrix, a point (x _R ,
y _R , z _R ) are converted to points (x
_C , y _C , z _C ) by the following equation.

[0112]

[Equation 27]

It is assumed that Here, R ″ and T ″ represent the components of the rotation matrix and the translation vector in the coordinate transformation from the range sensor coordinate system to the texture photographing camera coordinate system.

In the following, how to combine a texture image and a distance image from the parameters calculated in this way will be briefly described with reference to FIGS. 18 and 19.

First, a description will be given with reference to FIG. Now, based on f measured by the sensor element 28 (i, j) in the range sensor 1, the lookup table processing is performed as described above, that is,

[0116]

[Equation 28]

Thus, the three-dimensional measurement points (x _R , y _R , z _R ) in the range sensor coordinate system are calculated. All 3D measurement point 29 thereof ^{_{(x R a, y R a}} , z R a) to.
Here, a = 1, 2,..., N. These points, the coordinate transformation parameters from range sensor coordinate system to the texture photographing camera coordinate system (R ", T") by using, the camera coordinate system for the texture photographing point ^{_{(x C a, y C a}} , z _C ^a )
It is mapped by the following formula.

[0118]

(Equation 29)

Subsequently, the camera internal parameters g = (β _s , β _t , s ₀ , t) described above for the texture photographing camera.
_{0, a 1, b 1,} b 2, b 3, b 4) and formula (20) (2
1) by using (22), calculates the ^{_{(x C a, y C a}} , z C a) corresponding point 30 in the camera image for (s ^a, t ^a). Thus, a set of discrete image measurement points and ranging points in a camera image {(s ^a , t ^a ), (x _C ^a , y _C ^a , z _C ^a )}
Is used to perform texture mapping.

Specifically, there is the following method. FIG.
This will be described with reference to FIG. For each pixel (k, m) of the texture photographing camera, three nearest image measurement points (s ^b , t ^b ) 34 (b = 1, 2, 3) are searched for, and the measurement points (x _C ^b) , y _C ^b, searches the z _C ^b) 31. At this time, the plane formed by the distance measuring points is π32. Now the this plane _{n x x C + n y y} C + n z z C = d (29). A distance measuring point for (k, m) may be newly defined based on the intersection point 33 of the plane π32 and the light beam created by the camera pixel point (k, m). In particular,

[0121]

[Equation 30]

Can be obtained. That is, since distance information (x _C , y _C , z _C ) can be added to each pixel point of the texture photographing camera, this is equivalent to texture mapping or fusing a distance image and a texture image. .

As described above, according to the third embodiment, a camera and a photographing device for photographing a texture image and the like are arranged in addition to the range sensor 1, and the calibration patterns are simultaneously recognized, so that the both are recognized. Can be measured, and the distance image and the texture image generated by the range sensor 1 can be fused.

[0124]

According to the first aspect of the present invention, by calculating the light beam parameters defined by each element, even if the optical system of the range sensor has a distortion component (lens distortion), Calibration can be performed accurately. Further, since the light ray parameter and the distance parameter are calculated for each element, it is also possible to correct variations in the time-of-flight characteristics appearing between a plurality of elements.

According to the second aspect of the invention, in addition to the effect of the first aspect, a member having a three-dimensionally known shape can be used as a reference. Calculation of the power parameter becomes easy. As a result, the three-dimensional shape of the object calculated by the entire device can be measured using coordinate values in an absolute sense that is not relative, and accurate three-dimensional measurement can be performed.

According to the third aspect of the present invention, in addition to the effect of the second aspect of the present invention, calculation of parameters used for calibration is facilitated by taking a plurality of images.

According to the invention described in claim 4, in addition to the effect of the invention described in claim 3, recognition of a pattern used for calibration (recognition of a marker such as a hole).
Therefore, manual intervention is reduced, and more accurate and easy calibration can be realized.

According to the fifth aspect of the present invention, in addition to the effects of the first aspect of the present invention, by using a conversion table, calibration parameters are efficiently stored and distance conversion is performed. Can be used for

According to the sixth aspect of the present invention, in addition to the effects of the first aspect, a camera or a photographing device for photographing a texture image or the like is arranged in addition to the range sensor to perform calibration. By simultaneously recognizing the motion patterns, the positional relationship between the two can be measured, and the distance image and the texture image generated by the range sensor can be combined.

According to the seventh aspect of the present invention, in addition to the effect of the second aspect, by moving the three-dimensionally shaped member three-dimensionally, the ray parameter and the distance parameter can be reduced. It is possible to estimate at the same time.

[Brief description of the drawings]

FIG. 1 is a range sensor 1 according to a first embodiment of the present invention.
FIG. 3 is a diagram showing a part of the device configuration of FIG.

FIG. 2 is a diagram mathematically showing a structure of a sensor element array 7;

FIG. 3 is a diagram for explaining how the range sensor 1 can generate a distance image.

It is a diagram showing an internal configuration of a look-up table for the FIG. 4 measurement point sequence _{^{(f m, z R m)}} .

FIG. 5 is a flowchart illustrating an algorithm that embodies a calibration method.

FIG. 6 is a diagram showing an apparatus for realizing a method for calculating a ray parameter and a coordinate conversion parameter.

FIG. 7 is a diagram showing a configuration of a calibration board.

FIG. 8 is a diagram showing an algorithm for calculating a ray parameter.

FIG. 9 is a diagram showing an apparatus relating to calculation of a distance parameter and creation of a look-up table, which are steps 2;

FIG. 10 is a flowchart showing details of the algorithm of step 2;

FIG. 11 is a diagram showing a table of (f _ij ^m , z _R ^m ).

FIG. 12 is a diagram showing a configuration in which the size and interval of a pattern in a calibration board are changed according to the position of z.

FIG. 13 is a diagram showing an apparatus configuration for realizing a second embodiment of the present invention.

FIG. 14 is a flowchart showing the steps of the algorithm of the second embodiment.

FIG. 15 is a diagram for describing an embodiment in which a member having a known three-dimensional shape is used instead of the calibration board.

FIG. 16 is a diagram showing a configuration of a third embodiment of the present invention.

FIG. 17 is a flowchart illustrating a processing flow according to the third embodiment.

FIG. 18 is a diagram (part 1) for explaining how to combine a texture image and a distance image from calculated parameters.

FIG. 19 is a diagram (part 2) for explaining how to combine a texture image and a distance image from calculated parameters.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 1 Range sensor 2 Object 3 Computer 4 Timing controller 5 Pulse light source unit 6 Optical system unit for pulse light source 7 Sensor element array 8 Optical system unit for sensor element array 9 Signal processing unit 10 Input / output interface 11 Timing signal 12 Pulse light 13 Pulse Light 14 Pulsed light 15 Pulsed light 16 I / O signal 17 I / O signal

 ──────────────────────────────────────────────────続 き Continued on the front page F term (reference) 2F065 AA06 BB28 DD00 FF12 FF32 FF61 JJ03 JJ09 JJ26 PP12 RR07 SS03 SS13 UU04 UU05 2F112 AD01 BA06 CA08 FA03 FA45 GA10 5B057 BA02 BA17 BA19 BA26 CA12 CA16 CD14 CH07 DA07 DB20 DB04 AB20 AD01 BA40 CA03 CA31 CA67 DA01 DA07 EA08 FA03

Claims (7)

[Claims] 1. A method for calibrating a distance image sensor using a time-of-flight measurement principle having a light receiving section including a plurality of elements arranged two-dimensionally, wherein a direction of a light beam incident on each of the elements is defined. A step of obtaining a parameter of a light beam to be performed, and a step of obtaining a distance parameter to an object according to an output signal of each element. 2. The method for calibrating a two-dimensional distance image sensor according to claim 1, wherein a result of measuring a member having a known three-dimensional shape is used for obtaining the ray parameter or the distance parameter. 3. The measurement of a member whose three-dimensional shape is known,
3. The method according to claim 2, wherein the distance between the member and the two-dimensional distance image sensor is changed a plurality of times. 4. The member having a known three-dimensional shape includes a member having a plurality of holes or a marker having irregularities, and a member having a planar portion not having the marker. Calibration method for a two-dimensional range image sensor. 5. The method according to claim 1, wherein the step of obtaining the distance parameter uses a conversion table.
A calibration method for a two-dimensional range image sensor according to claim 1. 6. In addition to the distance image sensor, a photographing device for photographing a member having a known shape is further arranged, and a parameter defining a positional relationship between the distance image sensor and the photographing device; 2. The method for calibrating a two-dimensional range image sensor according to claim 1, wherein parameters related to the apparatus are calculated. 7. The two-dimensional distance image sensor according to claim 2, wherein the light ray parameter and the distance parameter are calculated by three-dimensionally moving the member having a known three-dimensional shape. Calibration method.