JPH0727565B2 - Image processing method and apparatus thereof - Google Patents

Description

Description: TECHNICAL FIELD The present invention relates to a method and apparatus for processing image information converted into digital data. More specifically, the same object in a three-dimensional space is different. The present invention relates to a method and an apparatus for detecting an amount of rotation of an image pickup device and an image enlargement ratio in a three-dimensional space by processing an image picked up by an image pickup device that rotates arbitrarily at a position.

[Conventional technology]

In order to measure the position or shape of an object in a three-dimensional space using an image of the object, it is necessary to obtain the accurate position and orientation of the image pickup device that performs the image pickup.

Conventionally, a method is known in which an image of a point whose position in a three-dimensional space is known is calculated from a position on an image obtained by capturing an image with an image capturing device. An example of such a technique is the paper “Camera” by OD Faugeras.G. Toscani et al.
Calibration for 3D Domputer Vision Proc.Int.Works
hop on Industrial Applications of Machine Vision a
nd Machine Intelligence ”.

In this method, a flat plate on which dots are drawn is set at a specific position, and the rotation amount or magnification of the image pickup position is measured from the position on the image taken.

Here, in order to simplify the explanation, it is assumed that only the rotation amount and the enlargement ratio of the image pickup apparatus are unknown, and all values such as other positions are known.

The coordinates of a point p in the three-dimensional space are set to (x, y, z), and the projection point of the point p on the image is set to P (X, Y). Here, for simplification, the imaging device uses the origin (0,0,
0). In addition, the focal length of the imaging device is f, x axis,
The amounts of rotation around the y-axis and z-axis are θ, φ, and α, respectively. At this time, the relationship between the coordinate values of the point p and the point P is expressed by the following formula (1), where p ′ (x ′, y ′, z ′) is a temporary point obtained by expansion according to the rotation in the three-dimensional space. It is expressed by equation (2).

Here, Rx, Ry, and Rz are matrices that represent rotations and are given by the following equation (3).

Further, E (f) is a matrix representing the enlargement ratio of the image and is given by the following equation (3 ').

Next, the positions of three or more n points are determined in the three-dimensional space.
The positions of these n points in three-dimensional space are p ₁ (x ₁ , y ₁ ,
Let z ₁ ), p ₂ (x ₂ , y ₂ , z ₂ ) ... _pn (x _n , y _n , z _n ). Further, the coordinates on the image obtained by picking up these points are defined as P ₁ (X ₁ , Y ₁ ), P ₂ (X ₂ , Y ₂ ) ... P _n (X _n , Y _n ), respectively.

Here, the points converted by each rotation according to the equations (1) and (2) are P ₁ ′ (X ₁ ′, Y ₁ ′), P ₂ ′ (X ₂ ′, Y ₂ ′) ...
When P _n ′ (X _n ′, Y _n ′), the following equation (4) is established.

However, if 1 ≦ i ≦ n each rotation amount (angle) θ, φ, α is correctly obtained, all i
The following equation (5) holds for

However, since they do not completely match due to measurement errors, etc., θ, φ, and α are calculated by the least squares method so that the value of this equation is minimized by assuming the equation (6) below. Ask.

[Problems to be Solved by the Invention] In such a conventional method, it is necessary to actually install an object at a predetermined point in a three-dimensional space. However, in a device that moves during operation or a device in which an object to be watched is often changed, it is necessary to move the image pickup device frequently. Therefore, in the conventional method, it is necessary to set an object at a predetermined position and take an image each time the imaging device is moved.

However, it is frequently and practically difficult to set an object at a predetermined point each time the device is moved.
In particular, it is difficult for a robot or the like to use a high-speed moving device in an unmanned environment or outdoors where it is used.

The present invention has been made in view of such circumstances,
It is not necessary to set the target object at a fixed point, and even if the position and shape of the target object are completely unknown by multiple imaging devices, the rotation amount and focal length (magnification ratio) of the imaging device can be determined only by imaging the target object. An object is to provide a detectable image processing method and its apparatus.

[Means for Solving the Problems]

In the image processing method and apparatus of the present invention, the position and shape of the object is completely unknown, the position of the object is specified in each image on an image obtained by imaging with a plurality of imaging devices at different positions. The rotation amount and the focal length of the image pickup apparatus, that is, the magnification of the image is detected by the coordinate value.

[Action]

In the present invention, basically, the rotation amount and the focal length (enlargement ratio) of the image pickup device itself can be detected by using only the images picked up by a plurality of image pickup devices. It is not necessary to image an object whose position or shape is default.

[Principle of Invention]

Hereinafter, first, an image processing method according to the present invention, that is, a method of detecting the rotation amount of each image pickup device using images picked up by a plurality of image pickup devices will be described.

In brief, the method of the present invention is such that, when a plurality of image pickup devices having the same focal length are installed in parallel and imaged, the coordinate values in the image of the same point in the three-dimensional space are imaged by the image pickup device. Matching points are used in the direction orthogonal to the axis connecting the two.

The present invention can be applied to three or more imaging devices, but here, for convenience of description, a case of using two imaging devices arranged on the left and right will be described.

First, as shown in FIG. 2, a reference state of the image pickup apparatus is determined. Assume the X, Y, Z Cartesian coordinates. The optical axes of the two image pickup devices are on the XZ plane, are parallel to each other, and are oriented in the Z axis direction. The imaging planes of the two imaging devices are X
-In the Y plane, the magnification is equal. The distance between the two image pickup devices is 2a.

In this reference state, the point p (x, y,
coordinate value P on each of the left and right images obtained by imaging z)
_L (X _L , Y _L ) and P _R (X _R , Y _R ) are given by the following equation (6 ′), where the focal length is 1.

From this equation (6 '), since a> 0, the following equation (7) is obtained.

Next, let the coordinate values on the image obtained by the actual image pickup device be P _L ′ (X _L ′, Y _L ′) and P _R ′ (X _R ′, Y _R ′), respectively. Further, the difference between the rotation amounts of the left and right imaging devices with respect to the reference state is set as θ _L , φ _L , α _L and θ _R , φ _R , α _R as values for the respective x, y, z axes. Furthermore,
Let the focal lengths be f _l and f _r , respectively. Similar to the equation (1), the following equation (8) is obtained, and the following equation (8 ') is obtained for the image pickup device located on the left side.

From the above equations (8) and (8 '), the following equation (9) is obtained.

From this equation (9), both sides are divided by Z _L ′ to obtain the following equation (10).

Therefore, y / z can be expressed by a function for P _L ′ (X _L ′, Y _L ′), that is, the following equation (11).

However, The same applies to the image pickup device located on the right side (1
Equation 1 ') is obtained.

Here, from equation (7) Therefore, the following equation (12) is established.

As a result, α _L , φ _L , θ _L , α _R , φ _R , θ _R , f satisfying this equation for a plurality of corresponding points (representing the same point in three-dimensional space) between the left and right images. By determining _L and f _R , it is possible to obtain the amount of rotation with respect to the reference state of the image pickup device. However, since α _L and α _R are rotations on the axis connecting the imaging devices and are not independent of each other, either one of the values or only the difference between them can be obtained. Since the focal lengths f _L and f _R are not independent as is clear from the formula, the ratio F (= f _L / f _R ) of the two can be obtained.

In the actual image processing, the least squares method is used to find the optimum value in order to eliminate the error.

Although the above description is for the case where two image pickup devices are used, it is clear that the above equation (12) can be expanded to three or more, and the present invention can be applied to a plurality of image pickup devices. Is.

〔Example〕

 Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

FIG. 1 is a block diagram showing the configuration of an image processing apparatus used for carrying out the method of the present invention based on the above-described principle.

In the following description, the number of imaging devices is two, but the device of the present invention can be applied to more imaging devices as shown in the drawing.

In the figure, reference numeral 1 denotes an image pickup unit composed of a plurality of image pickup devices,
It is installed at a predetermined point in the three-dimensional space and images the object.

The captured image is sent to the image processing unit 2 in the figure. The image processing unit 2 processes the given image to extract feature points,
It is sent to the corresponding point detection unit 3 in the figure.

The corresponding point detection unit 3 compares the characteristic points of the images obtained by the plurality of imaging devices, selects the corresponding points based on the similarity of the characteristics,
The coordinate value in each image is sent to the matrix calculation unit 4 in the figure.

By the way, since the equation shown in the equation (12) cannot be analytically solved, the matrix calculation unit 4 obtains the solution by the iterative method using the first-order approximation equation. The first-order approximation formula on the left-hand side of the formula (12) is expressed by the following formula (13).

The derivation process is shown below.

First is the left side of the denominator _{Rx (-θ L) Ry (-φ} L) Rz (-
α _L ) will be developed as an example.

From equation (3) θ _L φ _L α _L is close if Sinxx, the cosx1 since _{sinθ L θ L cosθ L 1 sinφ} L φ L cosφ L 1 sinα L φ L cosα L 1 sufficiently 0. Substituting this into equation (12a) Here, if θ _L φ _L α _L is sufficiently close to 0, the following relationship holds.

│α _L │ ＞＞ ｜ θ _L φ _L ｜ 1 ＞＞ ｜ θ _L φ _L α _L ｜ ｜ φ _L ｜ ＞＞ ｜ θ _L α _L ｜ ｜ θ _L ｜ ＞＞ ｜ φ _L α _L ｜ 1 ＞＞ | Θ _L α _L | As can be seen from this equation, the value of the term obtained by multiplying two or more variables can be ignored. Then the formula (12
b) can be approximated as follows.

Therefore, the left side of equation (12) is approximated by the following equation.

Similarly, the approximate expression on the right side is as follows.

This equation (13) is modified to determine the error e as follows.

Further, for the reason described in the section of the principle of the invention, θ _{R is set} to 0 for convenience in this embodiment. Furthermore, the following equation (14) is obtained by deleting terms of the first order and higher.

e '≒-(Y _L ′ -Y _R ′ +1) θ _L + X _L ′ Y _R ′ φ _L + X _L ′
α _L −X _R ′ Y _L ′ φ _R −X _R ′ α _R + (Y _L ′ −Y _R ′) + Y _L ′ d ...
(14) However, the d = F-1 matrix calculation unit 4 calculates the term of Expression (14) for each point,
This is sent to the matrix storage unit 5. The matrix storage unit 5 has 42 registers for representing the value of each term as the value of a 6 × 7 matrix, and θ _L ,
φ _L , α _L , φ _R , α _R , d terms and θ _L , φ _L , α _L ,
φ _R , α _R , d, 1 (that is, (Y _L ′ −Y _R ′) in the equation (14))
Find the product of each term and add it to each register.

When the addition is completed for all corresponding points, this value is sent to the rotation amount / magnification rate calculation unit 6, and the solution is obtained by using the matrix as simultaneous linear equations. As a result, the rotation amounts (angles) θ _L , φ _L , α _L , φ _R , α _{R of} the imaging device and the ratio d of the enlargement ratios.
Is obtained. However, this is an error because it is obtained as a solution of the first-order approximation formula. Therefore, the obtained solution is sent to the error evaluation unit 7, the accuracy of the obtained solution is obtained from the magnitude of the rotation amount, and if the error is equal to or less than a certain value, the obtained solution is output.

If the error is equal to or larger than a certain value, the solution representing the rotation amount is sent to the image processing unit 2 and the original image is converted by the equation (10).
As a result, the apparent rotation amount of the image becomes extremely small. The same processing is performed again on this image to obtain the rotation amount. This process is repeated until the amount of rotation obtained as a solution becomes a certain value or less, and finally a correct value is obtained.

[Verification of the capability of the device of the present invention]

As shown in FIG. 3, from two image pickup devices (P _L 1, P _R 1),
(P _L 2, P _R 2), (P _L 3, P _R 3), (P _L 4, P _R 4), (P _L 5, P
6 points for each of _R 5) and (P _L 6, P _R 6) are obtained.

These points are the coordinates on the image of the same point in the three-dimensional space. In order to verify the capability of this device, a random number was generated, and the image pickup device was moved by an angle and focal length proportional to the random number. The imaging device is (-20.0,0.0,0.0) cm in the left and the right device is (20.
0,0.0,0.0) cm, and each point is (-50,
-50,300), (50, -50,300), (-50,50,300), (50,5
It is located at each coordinate position of (0,300), (-50, -50,500), (50,50,500) cm.

The coordinate values in the resulting 6-point image were input to this apparatus, and the rotation amount and focal length of the image pickup apparatus were detected. The results are shown in FIG. Here, the rotation angle and focal length (1.0
Centered on) was varied within the range of values shown on the horizontal axis. Furthermore, 1000 trials were performed for each focal length. The vertical axis shows the number of times that a correct value was not obtained for each 1000 trials. As is clear from FIG. 4, in the device of the present invention, a correct value is always obtained if the rotation is within the range of ± 0.35 radian. The error that can be obtained is 10
^-6 radians or less.

Next, an example in which the device of the present invention is applied to the environment of building blocks will be shown.

An image as shown in FIG. 5 is obtained by picking up an image of the environment of three blocks with two image pickup devices. In the conventional method, the amount of rotation of the image pickup device cannot be accurately obtained. Therefore, when the processing is performed as it is, it is difficult to obtain the correct position and shape as shown in FIG.

6 (a) is a bird's eye view, FIG. 6 (b) is its front surface, FIG. 6 (c) is its front surface, and FIG. 6 (d) is its side surface.

Therefore, this image was input to the device of the present invention, the rotation angle of the image pickup device was obtained, and the image was corrected.

FIG. 7 shows an image output from the image processing unit 2. The corresponding point detection unit 3 pays attention to the connecting points of the lines in the image,
Extract the same points between the two images. As a result, a relative shift in the up-down direction (Y-axis direction) is obtained with respect to the connection points of both images.

FIG. 8 is a schematic diagram showing this error.

This value is sent to the matrix calculation unit 4 and the amount of rotation is obtained by repeating the process 3 to 8 times. The image was corrected with this value, and the resulting three-dimensional image is shown in FIG.

9 (a) is a bird's eye view, and FIG. 9 (b) is a front view thereof.
The same (c) shows the front and the same (d) shows the side.

As is clear from this figure, it is clear that the apparatus of the present invention enables remarkably accurate correction as compared with that before the correction.

〔The invention's effect〕

As described above in detail, according to the present invention, since it is possible to detect the rotation amount and the enlargement ratio of the image pickup device only from the image obtained by the image pickup device, the predetermined point is set every time the device is moved. There is no need for complicated processing such as positioning the object.

[Brief description of drawings]

FIG. 1 is a block diagram showing the structure of an image processing apparatus according to the present invention, FIG. 2 is a schematic diagram for explaining the principle of the method of the present invention, and FIG. 3 is an image obtained by the image pickup apparatus of FIG. FIG. 4 is a schematic diagram showing an actual rotation amount and a magnifying power detection result by the device of the present invention, FIG. 5 is a schematic diagram of an example of a situation of imaging symmetry by the device of the present invention, and FIG. Fig. 7 is a schematic diagram showing the image processing result by the method of Fig. 7, Fig. 7 is a schematic diagram showing the result of capturing the situation shown in Fig. 5 by two image capturing devices, and Fig. 8 is the two images shown in Fig. 7. FIG. 9 is a schematic diagram showing a state of deviation in the Y-axis direction, and FIG. 9 is a schematic diagram showing an image processing result by the apparatus of the present invention. 1 ... Imaging unit, 2 ... Image processing unit, 3 ... Corresponding point detection unit, 4 ...
Matrix calculation unit 5 ... Matrix storage unit, 6 ... Rotation amount / magnification ratio calculation unit

 ─────────────────────────────────────────────────── ─── Continuation of the front page (56) Bibliography Proceedings of the 37th National Convention of IPSJ ▲ III ▼ P. 1523-1524

Claims (2)

[Claims] 1. A rotation amount and an enlargement ratio of each image pickup device when the same recognition target located in a three-dimensional space is imaged by a plurality of image pickup devices which are located at different positions and rotate around three orthogonal axes. In the image processing method described above, assuming that the orthogonal coordinate system X, Y, Z, the optical axes of any two image pickup devices are oriented in the Z-axis direction and are located in the Z-X plane, and The image pickup plane is located on the XY plane, and X from the reference state in which the enlargement ratios of the two image pickup devices are equal,
The rotation amounts of the two image pickup devices around the Y and Z axes are θ _L and φ, respectively.
_L , α _L and θ _R , φ _R , α _R, and enlargement ratios of the two image pickup devices are f _l and f _r , respectively, and the images recognized by the two image pickup devices are associated with each other. Each corresponding point of the target is extracted and the coordinate value on each image is detected (X _L ′,
Y _L ′) and (X _R ′, Y _R ′) However, An image processing method characterized by calculating a rotation amount θ _L , φ _L , α _L , φ _R , α _R and a magnification rate from the following. 2. A rotation amount and an enlargement ratio of each image pickup device when the same recognition target located in a three-dimensional space is picked up by a plurality of image pickup devices that rotate at different positions and rotate around three orthogonal axes. In the image processing device, the two images picked up by any two image pickup devices are made to correspond to each other, and the corresponding points of the recognition target are extracted to obtain the coordinate values on each image as (X _L ′, Y _L ′). And (X _R ′, Y _R ′) and a corresponding coordinate detection unit X, Y and Z, the optical axes of the arbitrary two image pickup devices are oriented in the Z-axis direction, and Z
-Positioned on the X plane, and the imaging planes of the two imaging devices are X
The rotation amounts of the two image pickup devices around the X, Y, and Z axes from the reference state, which are located on the Y plane and have the same enlargement ratio of the two image pickup devices, respectively, θ _L , φ _L , α _L, and θ _R , φ _R , α
_{Let R} be the enlargement ratios of the two image pickup devices, f _l and f _r , respectively. However, (Left side) − (right side) = e ′ approximate expression e′≈− (Y _L ′ −Y _R ′ +1) θ _L + X _L ′ Y _R ′ φ _L + X _L ′
α _L −X _R ′ Y _L ′ φ _R −X _R ′ α _R + (Y _L ′ −Y _R ′) + Y _L d However, a matrix calculation unit that calculates each term of f _l / f _r = 1 + d, Θ _L , φ _L , α _L , which are the calculation results of the matrix calculation unit,
Each term of φ _R , α _R and θ _L , φ _L , α _L , φ _R , α _R , d, 1
A matrix accumulator that accumulates a matrix of products with each term of Eq. And a rotation amount θ _L , φ _L , α _L , φ _R , α by solving the values of the matrix accumulated in the matrix accumulator as simultaneous linear equations.
A rotation amount calculation unit that calculates the ratio of _R and the enlargement ratio, and an error evaluation unit that evaluates the error of the solution obtained by the rotation amount calculation unit and repeats the above calculation until the error becomes a predetermined value or less. An image processing device characterized by the above.