JPH11161792A - Three-dimensional information restoration device/method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To highly precisely obtain local inclination on the surface of an object at high speed from plural pictures by calculating the parallax inclination of a point from the luminance inclination of respective points in a small area at the periphery of an arbitrary point on a calculated picture and from the calculated parallax of the point. SOLUTION: A picture input part 1 inputs the plural pictures from the different viewpoints by using plural cameras. A picture accumulation part 2 accumulates stereo pictures (f) and (g) inputted by using a picture memory. A luminance inclination calculation part 3 calculates the luminance inclination of the x-direction of the respective stereo pictures (f) and (g). A parallax calculation part 4 makes points correspond to the same point in a three-dimensional space and calculates the dislocation of the corresponding points on the picture, namely, parallax. A parallax inclination calculation part 5 assumes that parallax in a window linearly changes against displacement from the center of the window and calculates the inclination of parallax in the window from luminance inclinations fx and gx calculated in the luminance inclination calculation part 3 and parallax calculated in the parallax calculation part 4.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to three-dimensional information restoration that can be used for, for example, visual control of a mobile robot, object recognition, image monitoring, CG modeling, and the like, for determining the distance to a target from a plurality of viewpoint images and the surface gradient. The present invention relates to an apparatus and a method thereof.

[0002]

2. Description of the Related Art In recent years, researches on intelligent robots capable of flexibly responding to changes in the surrounding environment have been actively conducted. In order to control such an intelligent robot, information on the distance to an object is required. As a sensor for obtaining a distance to an object, there is a laser range finder for directly obtaining a distance by irradiating a laser. The laser range finder is characterized by extremely high measurement accuracy, but generally has a high cost. Also,
Since it is necessary to irradiate a laser, there is a problem that use in an environment where humans exist is very dangerous.

As a method for obtaining distance information from an image, stereo vision is typical. In stereo vision, a plurality of cameras having a known relative positional relationship are arranged as shown in FIG. 1 to obtain a plurality of images having different viewpoints. In this method, points that are the same point in the three-dimensional space are associated with each other on the plurality of images, and the three-dimensional coordinates of the point are calculated using the principle of triangulation. Since stereo vision uses only a TV camera, the cost is relatively low, and there is no danger unlike a laser range finder.

The most important problem in stereo vision is correspondence. That is, it is very difficult to find points corresponding to the same point in different images in a three-dimensional space, and this is a subject that is being actively studied at present. The association is basically performed in the following procedure.

(1) As shown in FIG. 2, a square window is set around an arbitrary point P (x, y) on the left image.

(2) Point P '(x + d, y) on the right image
Set a window around the.

(3) Calculate the similarity (or dissimilarity) of the luminance patterns in the windows around points P and P '.

(4) A certain range (dmin ≦ d ≦ dmax)
Steps (2) and (3) are performed for d, and a change in similarity (or dissimilarity) with d is obtained as shown in FIG.

(5) A point around the point P (x, y) in which the luminance pattern is most similar in the window is associated with each other, and the displacement of the position is defined as the parallax of the point P.

As a similarity (or dissimilarity) evaluation scale, a luminance correlation, a sum of absolute values of luminance differences, a sum of squares of luminance differences, and the like are used. When the parallax of a certain point P on the image is obtained by performing this association, the three-dimensional position of the point P can be calculated using parameters such as a relative positional relationship between a plurality of cameras and a focal length of a camera lens. .

When a robot moving on a plane detects a free space in front of itself and travels, the free space (plane area) and the obstacle area (uneven area) are separated by a gradient. In such an application, a local gradient (normal vector) around the point is required instead of the three-dimensional position of the point.

The equation of the small plane K around the point P (Xp, Yp, Zp) in FIG. 4A is expressed as (p, q) when Z = p (X-Xp) + q (Y-Yp) + Zp. ) Is called the gradient of point P. The gradient (p, q) of the point P can be obtained by calculating the three-dimensional position of the point near the point P and applying the plane K ′ to the point P and points in the vicinity thereof.

For example, as shown in FIG. 4B, the three-dimensional positions of Q and R of a point near the point P are calculated, and the points P, Q and R are calculated.
Is applied to obtain the gradient (p, q) of the point P.

However, the three-dimensional position information obtained by stereo vision includes errors due to various factors, and there is a problem that accuracy is deteriorated by the method of applying a plane. Especially,
If the measurement distance is large, the measurement accuracy of stereo vision deteriorates remarkably, so that it is very difficult to obtain the surface gradient of a distant object. Further, to find the gradient of a certain point, it is necessary to calculate the disparity of nearby points. However, since the calculation cost of the disparity is generally large, there is a problem that this method requires processing time.

[0015]

As described above, the conventional method for obtaining the gradient of an arbitrary point on an image has problems in accuracy and calculation cost.

Accordingly, the present invention has been made in view of the above circumstances, and relates to a three-dimensional information restoration apparatus and method capable of obtaining a local gradient of an object surface from a plurality of images at high speed and with high accuracy.

[0017]

According to a first aspect of the present invention, there is provided an image input means for inputting a plurality of images from a relatively different viewpoint with respect to an object, and a plurality of images input by the image input means. Any point P on at least one of the images
Brightness gradient calculating means for calculating a brightness gradient of each point in a minute area around the image, and calculating a parallax of the point P on at least one of a plurality of images input by the image input means. A parallax calculating unit, a luminance gradient of each point in a minute area around the point P calculated by the luminance gradient calculating unit, and the point P calculated by the parallax calculating unit.
A parallax gradient calculating means for calculating the parallax gradient of the point P from the parallax of the three-dimensional information.

According to a second aspect of the present invention, the point P based on at least one of the different viewpoints is determined from the parallax and the parallax gradient of the point P calculated by the parallax calculating means and the parallax gradient calculating means, respectively. 2. The three-dimensional information restoration apparatus according to claim 1, further comprising a conversion unit that calculates a three-dimensional position and a three-dimensional gradient.

According to a third aspect of the present invention, the parallax gradient calculating means calculates the luminance gradient of each point in a minute area around the point P calculated by the luminance gradient calculating means, and calculates the luminance gradient by the parallax calculating means. And the disparity of the point P on the assumption that the disparity in the minute area around the point P changes linearly with respect to the displacement on the image from the point P. 3. The three-dimensional information restoration apparatus according to claim 1, wherein the gradient is calculated.

According to a fourth aspect of the present invention, the parallax gradient calculating means calculates the luminance gradient of each point in a minute area around the point P calculated by the luminance gradient calculating means, and calculates the luminance gradient by the parallax calculating means. A simultaneous equation relating to the parallax gradient of the point P obtained from the parallax of the point P, and the parallax in the minute area around the point P changes linearly with respect to the displacement on the image from the point P. 3. The three-dimensional information restoration apparatus according to claim 1, wherein the three-dimensional information restoration apparatus calculates the parallax gradient of the point P by solving the simultaneous equations.

According to a fifth aspect of the present invention, the brightness of each point in a minute area around an arbitrary point P on at least one image among a plurality of images from a viewpoint relatively different from the object. A luminance gradient calculating step of calculating a gradient, a parallax calculating step of calculating a parallax of the point P on at least one of the plurality of images, and a calculation of the point P calculated in the luminance gradient calculating step. A three-dimensional parallax gradient calculating step of calculating a parallax gradient of the point P from a luminance gradient of each point in the surrounding minute area and the parallax of the point P calculated in the parallax calculating step. This is an information restoration method.

According to a sixth aspect of the present invention, the point P based on at least one of the different viewpoints is determined from the parallax and the parallax gradient of the point P calculated in the parallax calculating step and the parallax gradient calculating step, respectively. 3D position and 3
The three-dimensional information restoration method according to claim 5, further comprising a conversion step of calculating a dimensional gradient.

According to a seventh aspect of the present invention, in the parallax gradient calculating step, the luminance gradient of each point in the minute area around the point P calculated in the luminance gradient calculating step is calculated in the parallax calculating step. And the disparity of the point P on the assumption that the disparity in the minute area around the point P changes linearly with respect to the displacement on the image from the point P. 7. The three-dimensional information restoration method according to claim 5, wherein a gradient is calculated.

According to an eighth aspect of the present invention, in the parallax gradient calculating step, the luminance gradient of each point in a minute area around the point P calculated in the luminance gradient calculating step is calculated. A simultaneous equation relating to the parallax gradient of the point P obtained from the parallax of the point P, and the parallax in the minute area around the point P changes linearly with respect to the displacement on the image from the point P. The three-dimensional information restoration method according to claim 5 or 6, wherein the parallax gradient of the point P is calculated by solving the simultaneous equations and obtaining the parallax gradient.

According to a ninth aspect of the present invention, the brightness of each point in a minute area around an arbitrary point P on at least one image among a plurality of images from different viewpoints with respect to the object. A brightness gradient calculation function for calculating a gradient, a disparity calculation function for calculating a parallax of the point P on at least one of the plurality of images, and a calculation of the point P calculated in the brightness gradient calculation function. A program for realizing a parallax gradient calculating function of calculating a parallax gradient of the point P from the luminance gradient of each point in the surrounding minute area and the parallax of the point P calculated by the parallax calculating function is stored. Characteristic 3
It is a recording medium for the dimension information restoration program.

According to the first, fifth, and ninth aspects of the present invention, a plurality of images selected from image input means such as a TV camera can be used.
A parallax gradient at an arbitrary position on an image can be obtained at high speed and with high accuracy.

According to the second and sixth aspects of the present invention, the three-dimensional position and the three-dimensional gradient of the parallax gradient can be obtained from the parallax gradient. And its practical effect is enormous.

[0028]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Examples, with reference to the drawings, a description will be given of an embodiment of the present invention.

FIG. 5 shows a schematic configuration of this embodiment in the embodiment.

Here, an image input unit 1, an image storage unit 2,
Brightness gradient calculator 3, parallax calculator 4, parallax gradient calculator 5,
It comprises a conversion unit 6 and an output unit 7. Among these devices, the image input unit 1 can be realized by a plurality of TV cameras,
Brightness gradient calculator 3, parallax calculator 4, parallax gradient calculator 5,
The conversion unit 6 and the output unit 7 store programs for realizing the functions of these devices in a personal computer on a hard disk,
FD. It is realized by being stored in a CD-ROM. The image storage unit 2 includes a hard disk of this personal computer,
Implemented in memory.

(Image Input Unit 1) The image input unit 1 uses a plurality of cameras as shown in FIG. 1 to input a plurality of images from different viewpoints. In the present embodiment, as shown in FIG. 6, two cameras are arranged so that their image planes are on the same plane,
Stereo images f (x, y) and g (x, y) are input.
Here, f is a left image and g is a right image.

(Image Storage Unit 2) The image storage unit 2 stores stereo images f and g input by the image input unit 1 using an image memory.

(Luminance Gradient Calculator 3) Luminance Gradient Calculator 3
Calculates the luminance gradient in the x direction of each pixel of the stereo images f and g stored in the image storage unit 2.

The luminance gradient in the x direction of f (x, y) and g (x, y) is calculated from the following equation, for example.

Fx (x, y) = {− f (x−1, y) + f (x + 1, y)} / 2 gx (x, y) = {− g (x−1, y) + g (x + 1, y) y)｝ / 2 (1) (Parallax calculation unit 4) The parallax calculation unit 4 associates a point corresponding to the same point in the three-dimensional space, and shifts the position of the corresponding point on the image, that is, Calculate parallax.

As shown in FIG. 2, (2) around one point in the left image
w + 1) × (2w + 1) window, and the sum of squares of the difference in luminance in the left and right image windows for each parallax in a certain search range (dmin ≦ d ≦ dmax)

(Equation 1) Is calculated.

Then, a change in D (d) with respect to the parallax d as shown in FIG. 3 is obtained. If D (d) is small, the patterns in the windows are similar, so D (d)
The parallax that minimizes (d) is the required parallax d ^* .

(Parallax gradient calculating section 5) The parallax gradient calculating section 5 assumes that the parallax in the window changes linearly with respect to the displacement (Δx, Δy) from the center of the window, and the luminance gradient calculating section 3 calculates the parallax. The parallax gradient in the window is calculated from the calculated luminance gradients fx and gx and the parallax calculated by the parallax calculator 4.

The disparity of the point (x + Δx, y + Δy) in the window centered on the point (x, y) on the left image f is represented by d + Δ
Assuming that d is d and the invariance of the brightness of the corresponding point is assumed, f (x + Δx, y + Δy) = g (x + Δx + d + Δd, y + Δy) (3)

Assuming that Δd is sufficiently small, and Taylor expansion of the right side, f (x + Δx, y + Δy) = g (x + Δx + d, y + Δy) + gx (x + Δx + d, y + Δy) Δd (4) where the parallax in the window is Assuming a linear change with respect to the displacement (Δx, Δy) from (x, y),
Assuming that the gradients of the parallax in the x and y directions are α and β, respectively, Δd = αΔx + βΔy (5) Substituting this into equation (4) and rearranging:

(Equation 2) Here, Φ (Δx, Δy) = gx (x + Δx + d, y + Δy) Ψ (Δx, Δy) = f (x + Δx, y + Δy) −g (x
+ Δx + d, y + Δy) Equation (6) gives (2w + 1) × (2w) around the point (x, y).
+1) (= N) at any point in the window (x + Δxi
, Y + Δyi) (i = 1, 2,..., N),

(Equation 3) Obtains a parallax gradient (α, β).

(Conversion unit 6) In the conversion unit 6, the parallax calculation unit 4
And the disparity obtained by the disparity gradient calculator 5 and the gradient of the disparity are 3
Convert to dimensional position and gradient.

In general, when a camera as shown in FIG. 6 is positioned, a stereo camera coordinate system O-XYZ having the origin at the middle point of the left and right camera lenses is set, and a certain point P (X, Y) in a three-dimensional space is set. , Z) to the left and right images, respectively (xl
, Yl), (xr, yr), the camera interval is B, the focal length of the camera lens is F, and the parallax is d (= xl−xr).
given that,

(Equation 4) Holds. Using the above equation, a three-dimensional position with respect to the stereo coordinate system can be obtained from the position of the corresponding point on the image.

A method for obtaining the three-dimensional gradient (p, q) from the parallax gradient (α, β) will be described below. As shown in FIG.
The equation of the plane of the small area π around the point P (Xp, Yp, Zp) in the dimensional space is expressed as Z = p (X−Xp) + q (Y−Yp) + Zp (11) Point P ′ (Xp + ΔX, Yp + Δ) in the minute area
Since Y, Zp + ΔZ also satisfies the equation of this plane, Zp + ΔZ = p (Xp + ΔX−Xp) + q (Yp + ΔY−Yp) + Zp (12) Therefore, ΔZ = pΔX + qΔY (13) The point P (Xp, Yp, If the parallax of Zp) is dp,

(Equation 5) Then, the three-dimensional gradient (p, q) can be obtained from the parallax gradient (α, β) using Expression (21).

(Output unit 7) The output unit 7 outputs the three-dimensional position and gradient of each point in the image obtained by the conversion unit 6.

As described above, the three-dimensional position and gradient of an arbitrary point P on an image can be obtained from a plurality of images obtained by the stereo camera.

[0046] In the change example (modified example 1) above embodiment, the image input unit 1, the two cameras, such that each image plane rests on the same plane (at this time two cameras in parallel FIG. 7)
The camera may be arranged at an angle as shown in FIG.

It is also possible to obtain images from different viewpoints by using three or more cameras as shown in FIG. 7B or by moving one camera as shown in FIG. 7C. Good.

(Modification 2) Although the parallax calculation unit 4 calculates the parallax using the sum of squares of the difference in luminance in the window as an evaluation function, the correlation coefficient C may be used as an evaluation function.

(2w + 1) × (2w + 1) (= N) of the point (x, y) on the left image f and the point (x + d, y) on the right image
Can be calculated from the following equation.

[0050]

(Equation 6) (Modification 3) The parallax gradient calculating unit 5 uses only the luminance gradient gx of the right image g when calculating the parallax gradient, but may use the luminance gradient fx of the left image f.

A point on the right image g corresponding to a point (x + Δx, y + Δy) on the left image f is defined as (x ′ + Δx, y ′ + Δy)
Assuming that the parallax of these points is d + Δd, assuming the invariance of the luminance of the corresponding points, f (x′−Δx−d−Δd, y ′ + Δy) = g (x ′ + Δx, y ′) + Δy) (23) Here, since x ′ = x + d, y ′ = y, f (x + Δx−Δd, y + Δy) = g (x + d + Δx, y + Δy) (24) When Taylor expansion is performed on the left side, f (x + Δx, y + Δy) −fx (X + Δx, y + Δy) Δd = g (x + Δx + d, y + Δy) (25) Here, by substituting equation (5) and rearranging,

(Equation 7) Where Φ ′ (Δx, Δy) = fx (x + Δx, y + Δy)） (Δx, Δy) = f (x + Δx, y + Δy) −g (x
+ Δx + d, y + Δy) Equations (6) and (26) represent arbitrary points (x + Δxi, y + Δyi) (i = 1, 2, 2) in a minute area around the point (x, y).
……, N)

(Equation 8) From the above, α and β can be obtained.

(Modification 4) Although the parallax gradient calculating unit 5 uses the assumption that the luminance of the corresponding point is invariant, the parallax gradient can be calculated by using a more general assumption of a linear change in luminance. As described above, assuming that the disparity of the point (x + Δx, y + Δy) in the window centered on the point (x, y) on the left image f is d + Δd, and assuming a linear change in luminance, Equation (3) is obtained.
Is as follows: a × f (x + Δx, y + Δy) + b = g (x + Δx + d + Δd, y + Δy) (28) Similarly, when the right side is Taylor-expanded and further substituted by equation (5),

(Equation 9) Here, Φ (Δx, Δy) = gx (x + Δx + d, y + Δy) F (Δx, Δy) = f (x + Δx, y + Δy) G (Δx, Δy) = g (x + Δx + d, y + Δy) (30) Equation (29) Is also an arbitrary point (x + Δxi, y + Δyi) (i = 1, 2,...) In a minute area around the point (x, y).
N), so

(Equation 10) , The parallax gradient (α, β) and the parameters (a, b)
Get.

In addition, modifications can be made without departing from the spirit of the present invention.

[0054]

According to the present invention, a parallax gradient at an arbitrary position on an image can be obtained at high speed and with high accuracy from a plurality of images selected from a TV camera or the like. Further, since the three-dimensional position and the three-dimensional gradient of the position can be obtained from the parallax gradient, it can be applied to traveling control of a mobile robot and object recognition, and the practical effect is enormous.

[Brief description of the drawings]

FIG. 1 is a diagram for explaining stereo vision.

FIG. 2 is a diagram for explaining parallax calculation.

FIG. 3 is an example of a change in dissimilarity with respect to parallax.

FIG. 4 is a diagram for explaining a conventional method of calculating a surface gradient.

FIG. 5 is an overall configuration of the present invention.

FIG. 6 is a diagram illustrating stereoscopic viewing in a modified example.

FIG. 7 is a diagram illustrating an image input unit.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 1 Image input part 2 Image storage part 3 Luminance gradient calculation part 4 Parallax calculation part 5 Parallax gradient calculation part 6 Conversion part 7 Output part

Claims (9)

[Claims] An image input means for inputting a plurality of images from a relatively different viewpoint with respect to an object, and at least one image among the plurality of images input by the image input means. Brightness gradient calculating means for calculating a brightness gradient of each point in a minute area around an arbitrary point P, and a brightness gradient calculating means for calculating a brightness gradient of at least one of a plurality of images input by the image input means. Parallax calculating means for calculating parallax, From the luminance gradient of each point in a minute area around the point P calculated by the luminance gradient calculating means, and from the parallax of the point P calculated by the parallax calculating means, A three-dimensional information restoring device comprising a parallax gradient calculating means for calculating a parallax gradient of the point P. 2. From the parallax and the parallax gradient of the point P calculated by the parallax calculating means and the parallax gradient calculating means, respectively:
The three-dimensional information restoration apparatus according to claim 1, further comprising a conversion unit that calculates a three-dimensional position and a three-dimensional gradient of the point P based on at least one of the different viewpoints. 3. The method according to claim 2, wherein the parallax gradient calculating unit calculates a luminance gradient of each point in a minute area around the point P calculated by the luminance gradient calculating unit, and calculates a luminance gradient of the point P calculated by the parallax calculating unit. Calculating the parallax gradient of the point P on the assumption that the parallax in the minute area around the point P changes linearly with respect to the displacement on the image from the point P using parallax. 3. The method according to claim 1, wherein
The three-dimensional information restoring device according to claim 1. 4. The parallax gradient calculating means, comprising: a luminance gradient of each point in a minute area around the point P calculated by the luminance gradient calculating means; and a luminance gradient of the point P calculated by the parallax calculating means. The point P obtained from the parallax
Is calculated on the assumption that the parallax in the minute area around the point P changes linearly with respect to the displacement on the image from the point P. 3. The three-dimensional information restoration device according to claim 1, wherein a parallax gradient of the point P is calculated. 5. A luminance for calculating a luminance gradient of each point in a minute area around an arbitrary point P on at least one of a plurality of images from a viewpoint relatively different from an object. A gradient calculating step; a parallax calculating step of calculating a parallax of the point P on at least one of the plurality of images; and the point P calculated in the luminance gradient calculating step.
A parallax gradient calculating step of calculating a parallax gradient of the point P from a luminance gradient of each point in a minute area around the point P and a parallax of the point P calculated in the parallax calculating step. Dimension information restoration method. 6. The three-dimensional position of the point P based on at least one of the different viewpoints, based on the parallax and the parallax gradient of the point P calculated in the parallax calculating step and the parallax gradient calculating step, respectively. 6. The three-dimensional information restoration method according to claim 5, further comprising a conversion step of calculating a dimensional gradient. 7. The point P calculated in the luminance gradient calculating step, wherein the parallax gradient calculating step is
Using the luminance gradient of each point in the micro area around the point P and the parallax of the point P calculated in the parallax calculation step, the parallax in the micro area around the point P is 7. The three-dimensional information restoration method according to claim 5, wherein the parallax gradient of the point P is calculated on the assumption that the gradient changes linearly with respect to a displacement on an image. 8. The point P calculated in the luminance gradient calculating step, wherein the parallax gradient calculating step is performed.
A simultaneous equation relating to the parallax gradient of the point P obtained from the luminance gradient of each point in the micro area around the point P and the parallax of the point P calculated in the parallax calculation step is expressed by a micro area around the point P. Is calculated on the assumption that the disparity in the image changes linearly with respect to the displacement on the image from the point P, and solves the simultaneous equations to calculate the disparity gradient of the point P. Item 3. The method for restoring three-dimensional information according to item 5 or 6. 9. A luminance for calculating a luminance gradient of each point in a minute area around an arbitrary point P on at least one of a plurality of images from a viewpoint relatively different from an object. A gradient calculating function, a parallax calculating function of calculating a parallax of the point P on at least one of the plurality of images, and a minute area around the point P calculated by the luminance gradient calculating function. A program for realizing a parallax gradient calculating function of calculating a parallax gradient of the point P from a luminance gradient of each point and a parallax of the point P calculated by the parallax calculating function. Recording medium for information restoration program.