JP3503121B2 - Distance calculation device and method - Google Patents

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a distance calculation apparatus and method suitable for use in obtaining distance information, particularly depth information, in a robot eye or the like.

[0002]

2. Description of the Related Art Binocular stereoscopic vision is known as a method for calculating the distance to a target object from a two-dimensional image taken by an image input device such as a camera. FIG. 6 shows the principle of binocular stereoscopic vision. The figure shows a state in which the positional relationship between the camera and the target object (the target point in this case) is viewed from above. L _L and L _R represent the lenses of the cameras arranged at a predetermined distance in the horizontal direction, and I _L and I _R represent the imaging surfaces of the left and right cameras, respectively. f _L , f
_R is an imaging plane I _L, a center of the I _R, the imaging plane I _L, I _R
The upper position is represented by the distance from these center positions f _L and f _R.

An intersection of straight lines T _L and T _R passing through the center positions f _L and f _R and perpendicular to the image pickup planes I _L and I _R is defined as P ₁ and is called a gazing point. In this case, the imaging plane I _L, images of different target point P ₂ is the distance from the I _R and gaze point P _1, each imaging plane I _L, occur at different positions in the I _R. As described above, the positional deviation of the images in the horizontal direction caused by the difference in distance from the gazing point P ₁ is called parallax. The parallax amount δ is
Under the condition that the distance between P ₁ and P ₂ is not so large, it can be approximated as follows. δ≈b (β ₁ −β ₂ ) = b (α ₂ −α ₁ ) ≈ab ΔD / D ² (1)

Here, a is the distance between the lenses L _L and L _R of the two cameras, b is the distance between the respective lenses L _L and L _R and the imaging planes I _L and I _R , and D is from the lenses L _L and L _R. gazing point in the depth direction of the distance to P _1, [Delta] D is the distance in the depth direction from the gaze point P ₁ to the target point P _2. Since a, b, and D are determined by the placement of the camera, if the placement is known, the parallax is a function of the distance ΔD from the gazing point P ₁ to the target point P ₂ . Therefore, if the parallax δ can be detected from the left and right image information, the distance ΔD of the target point P _{2 from} the gazing point P ₁ can be obtained from the following equation. ΔD = δD ² / ab (2)

Conventionally, binocular parallax is detected by searching for corresponding points in two images. As a method of making this correspondence, a cross-correlation method and a feature point matching method are known. ing.

[0006] The cross-correlation method associates the left and right images with the similarity in local luminance change, that is, the position with the highest correlation value. On the other hand, in the feature point matching method, feature points corresponding to a physical phenomenon such as an edge are extracted from the left and right images in advance, and the feature points are associated with each other.

[0007]

However, since the cross-correlation method uses the similarity of the luminance change, it is weak against noise, and the distortion of the image due to the lens or the like, the difference in the illumination condition between the left and right cameras, and the like. Further, there is a problem that an error occurs due to a parallax gradient or the like in a small area for calculating a correlation function.

Further, in the feature point matching method, what kind of feature is used has a great influence on the calculation accuracy. The parallax can be calculated only at the positions where the feature points exist, so
In order to obtain parallax in many places, it is necessary to use features that appear frequently in the image, but in this case,
Since many correspondence candidates are generated, it is difficult to perform correct correspondence.

Further, in these methods, it is necessary to calculate some kind of evaluation function for all combinations of points that may correspond, which requires a substantial amount of calculation time.

The present invention has been made in view of such a situation, and it is an object of the present invention to suppress the occurrence of an error and to obtain distance information quickly and accurately.

[0011]

A distance calculation device of the present invention is a first image pickup device obtained by picking up an image of a target object by a first image pickup device and a second image pickup device which are horizontally separated from each other by a predetermined distance. In a distance calculation device that obtains distance information about a target object from a first image and a second image, the distance calculation device includes a plurality of spatial filters that perform a filtering process on the data of the first image and have equal amplitude characteristics and different phase characteristics. First
And filtering means, for filtering the data of the second image, equal amplitude characteristics and a second filtering phase characteristics are from different spatial filters
And an outer product operator for calculating an outer product of the output of the first filtering means and the output of the second filtering means.
Stage, the output of the first filtering means and the second filter
An inner product calculating means for calculating an inner product of the output of the tiling means,
From the output of the outer product calculating means and the output of the inner product calculating means,
Determine the phase difference and phase difference sign between the image and the second image
The judging means and the phase difference and position judged by the judging means.
Calculate the parallax between the first and second images from the phase difference code
And a conversion means for converting the parallax into a distance .

[0012] The distance calculating unit, the first filtering means and the second filtering means is divided into a plurality of channels of different frequency bands, channel
The position of the second image corresponding to the gazing point of the first image
The position coordinate is the wavelength and phase difference code corresponding to the channel.
By the corresponding position detection means calculated from
A recording means for recording the calculated position coordinates is further provided.
The conversion means converts the coordinates corresponding to the gazing point of the first image and
The parallax is calculated from the position coordinates of the second image, and the parallax is calculated.
It can be converted into separation .

The distance calculation method of the present invention is applied to a horizontal direction.
The first image pickup device and the second image pickup device which are arranged apart from each other by a fixed distance.
The first image and the first image obtained by imaging the target object with the imaging device.
Distance to obtain distance information about the target object from image 2
A distance calculation method of a calculation device, comprising: data of a first image
The amplitude characteristics are equal, and
A first filter including a plurality of spatial filters having different phase characteristics.
Fill the filtering step and the data of the second image
The same amplitude characteristic and phase characteristic
A second filter composed of a plurality of spatial filters with different
Processing of the filtering step and the first filtering step
Output and processing of the second filtering step
The outer product calculation step for calculating the outer product of the outputs
Output by the processing of the filtering step and the second fill
In calculating the inner product of the output by the processing of the tailing step
Output by processing of product operation step and outer product operation step
And the first image from the output of the inner product calculation step
And the determination for determining the phase difference and the phase difference sign of the second image
Step and the phase determined by the processing of the determination step
From the difference and phase difference codes, the first image and the second image are viewed.
A conversion step of calculating a difference and converting the parallax into a distance;
It is characterized by including.

[0014]

[0015]

[0016]

[0017]

[0018]

In the above structure, the left and right first and second images are filtered by the spatial filters having different phase characteristics, and the output of each filtering process is excluded.
The product and inner product are calculated to determine the phase difference and the phase difference sign.
Distance, the parallax is calculated based on the determination result, and the distance is calculated.
Is converted to. By combining the output of each filtering process, the sign of the phase difference between the left and right images is detected,
Distance information indicating whether the target object is in front of or behind the gazing point can be accurately and quickly obtained.

[0019]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS FIG. 1 shows the configuration of an embodiment of the distance calculating device of the present invention. Reference numerals 10L and 10R denote image input cameras, which are arranged at positions separated by a predetermined distance in the horizontal direction. 11L and 11R are A / D converters, and each camera 1
The signals from 0L and 10R are digitized. 12L and 1
3L, 12R and 13R have the same amplitude characteristics,
The spatial filters have different phase characteristics. As such a spatial filter, for example, formula (3) (spatial filter 12
L, 12R) and equation (4) (spatial filter 13L,
13R), a pair of Gabor filters can be used. g _c (x, y) = exp [-(x / σ) ² ] cos (2πu _s x) exp [-(y / σ) ² ] (3) g _s (x, y) = exp [ − (X / σ) ² ] sin (2πu _s x) exp [− (y / σ) ² ] ... (4)

Equations (3) and (4) are called two-dimensional Gabor filters, and the horizontal direction (x direction) is the product of the Gaussian function and the cosine or sine function, and the vertical direction (y direction). Is a Gaussian function. These are given a spatial locality according to an arbitrary constant σ representing the spread by a Gaussian function in both x and y directions. That is, by the Gaussian function, the predetermined coordinate point (x,
Only the data in the vicinity of y) can be substantially extracted.

Further, it has locality on the frequency plane as well, and is a band-pass filter centered on the frequency (u _s , 0) (in the x-axis direction, a band whose center frequency is the frequency u _s). It is a pass filter, but in the y-axis direction,
It is a low pass filter). Therefore, these filters extract components that vary horizontally at frequency u _s in the local region. However, the filters of formulas (3) and (4) differ in phase characteristics by π / 2.
Such a pair of Gabor filters is used for the right image (12R, 1
3R) and two sets for left image (12L, 13L) are prepared.

Here, the output values of the Gabor filters g _c (x, y) and g _s (x, y) at a certain position (x, y) are the local luminance centered at the position (x, y). It can be regarded as a projective component to the change functions cos (2πu _s x) and sin (2πu _s x). Therefore, the output of the Gabor filter pair for each image, two-dimensional vector P _L as shown in FIG. 2 (x, y) = ( o Lc (x, y), o Ls (x, y)) ··· (5 ) P _R (x, y) = (o _Rc (x, y), o _Rs (x, y)) (6), and these are called output vectors.

O _Lc and o _Ls are Gabor filters g _c (x, y) and g _s (x, y) (spatial filter 12) for the left image.
L, 13L) output values o _Rc , o _Rs are output values of the Gabor filters g _c (x, y), g _s (x, y) (spatial filters 12R, 13R) for the right image. The angles θ _L and θ _R that each output vector makes with the horizontal axis are considered to be the phases of the luminance waveform along the x-axis direction at the corresponding position (x, y) on the image.

The outer product calculator 14 calculates the outer product of the output vectors of the left and right images from the output values of the Gabor filters (spatial filters 12L, 12R, 13L, 13R) by the following equation. op (x, y) = _oLc (x, y) _oRs (x, y) _-oLs (x, y) _oRc (x, y) (7)

To be precise, the outer product is a vector, but here, it is the scalar quantity op (x, y) shown in the equation (7).
Will be referred to as the cross product.

The inner product calculator 15 calculates the inner product of the output vectors of the left and right images from the output values of the Gabor filters (spatial filters 12L, 12R, 13L, 13R) by the following equation. ip (x, y) = _oLc (x, y) _oRc (x, y) + _oLs (x, y) _oRs (x, y) (8)

Outer product op (x, y) and inner product ip (x, y)
Are two output vectors P _L (x, y),
It changes according to the angle formed by P _R (x, y), that is, the phase difference (θ _R −θ _L ) between the left and right images. Figure 3 shows the phase difference (θ _R
Outer product op (x, y) and the inner product ip (x against - [theta] _L),
The relationship of y) is shown.

In consideration of the magnitude relation between the outer product op (x, y) and the inner product ip (x, y) shown in FIG. 3, the phase difference code determiner 16 calculates the phase difference code psgn (x) as follows. , Y), and outputs this. if (ip (x, y)> | op (x, y) |) then psgn (x, y) ← 0 else if (op (x, y)> 0) then psgn (x, y) ← + else psgn (X, y) ← -... (9)

In principle, the outer product op expressed by equation (7)
From (x, y) only, the inner product ip (x,
(without y)), the sign (0, + or-) can be determined. However, the sign of 0 is unstable. Therefore, the inner product ip (x, y) is used, and when the value is larger than the absolute value of the outer product op (x, y), the code is determined to be 0.

Here, with respect to the selective wavelength λ (the reciprocal of the center frequency u _s ) of the Gabor filter used, the position (x,
parallax δ (x, y) which is the amount of displacement on the image in y)
However, when (−λ / 2) <δ (x, y) <(λ / 2) (10), the phase difference between the left and right images corresponds to the parallax. Therefore, the phase difference code determiner 16 outputs the code of the parallax, and this parallax code reflects the front-back relationship between the target point P ₂ and the gazing point P ₁ , as can be seen from FIG. 6. ing.

That is, the parallax code "0" is the target point P ₂
And the gazing point P ₁ are present at almost the same distance. For the parallax codes “+” and “−”, the target point P ₂ is the gazing point P ₁
On the other hand, it means to be located in front or rear (however, "+" or "-" represents front and which represents rear depends on how the coordinate axes on the imaging surface are taken). . Therefore, the information about the rough distance (depth) of the target point P ₂ can be obtained by the product-sum calculation of the equations (7) and (8) for the output value of the Gabor filter.

FIG. 4 shows the configuration of another embodiment of the present invention. In this embodiment, parts corresponding to those in the embodiment shown in FIG. 1 are designated by the same reference numerals. In this embodiment, the signals output by the cameras 10L and 10R are
After being digitized by the A / D converters 11L and 11R, they are stored in the corresponding frame memories 31L and 31R.

Spatial filters 12L1, 13L1 and 12R1, which have the same amplitude characteristic and different phase characteristics, respectively.
13R1 can use the pair of Gabor filters shown in the equations (3) and (4), as in the embodiment of FIG. These filters make up the first channel.

12L2, 13L2 and 12R2, 13
R2 is a similar filter pair that constitutes the second channel but uses a Gabor filter with a higher center frequency than the first channel. 12L3, 13L3 and 1
2R3 and 13R3 are filter pairs having a higher center frequency and form a third channel.

Spatial filter 12 in the i-th channel
Li, 13Li and spatial filters 12Ri, 13
The Ri sequentially reads the image data stored in the corresponding frame memories 31L and 31R, performs a filtering process, and outputs the image data to the outer product calculator 14 and the inner product calculator 15.
However, in the first channel, the frame memories 31L, 3
The filtering process starts as soon as the necessary data is stored in 1R, but in channels other than the first channel, the control signal cntl sent from the corresponding position detector 32 ends the process in the immediately preceding channel. The filtering process is started at the point of.

That is, the spatial filter 12 of the second channel
L2, 13L2, 12R2, 13R2 are spatial filters 12L1, 13L1, 12R1, 13 of the first channel.
After the processing of R1 is completed, the spatial filters 12L3, 13L3, 12R3, 13R3 of the third channel have the spatial filters 12L2, 13L2, 12R of the second channel.
After the processes of 2 and 13R2 are completed, the respective processes are started.

The outer product calculator 14 calculates the outer product of the output vectors of the Gabor filter pair for the left and right images in each channel by the equation (11). op (x, y) = _oLc (x, y) _oRs (x '(x, y), y) _-oLs (x, y) _oRc (x' (x, y), y) ...・ (11)

Further, the inner product calculator 15 calculates the inner product of the output vectors of the Gabor filter pair for the left and right images in each channel by the expression (12). ip (x, y) = _oLc (x, y) _oRc (x '(x, y), y) + _oLs (x, y) _oRs (x' (x, y), y) ... (12)

Here, x '(x, y) in the equations (11) and (12) is the x coordinate of the position on the right image corresponding to the position (x, y) on the left image (of the immediately preceding channel). The coordinates obtained by the data processing) are stored in the frame memory 33. This value in the frame memory 33 is updated each time the corresponding position detector 32 performs data processing on a newer channel. However, the initial value is x '(x, y) = x.

The phase difference code decision unit 16 calculates the results of the equations (11) and (12) (the outer product calculator 14 and the inner product calculator 1).
5 output), as in the case of the embodiment of FIG.
The phase difference code is determined by the method shown in equation (9). However, the phase difference code obtained here is different from the case of the embodiment of FIG. 1 at the position (x, y) in the left image,
Between the position (x ′ (x, y), y) in the right image (the embodiment of FIG. 1 is between position (x, y) and position (x, y)). . Corresponding position detector 32
Detects the position on the right image corresponding to each position on the left image from the phase difference code (0, + or-) obtained by the phase difference code detector 16.

Now, assuming that the selective wavelength of the Gabor filter of the nth channel is λ _n and the condition of the equation (10) is satisfied, when the obtained phase difference sign is positive, It is considered that the parallax δ (x, y) of is in the range of 0 <δ (x, y) <(λ _n / 2) (13). At this time, the x-coordinate x _R of the position of the right image corresponding to the position (x, y) of the left image is in the range of x <x _R <x + (λ _n / 2) (14) . Therefore, the position of the center of the range represented by the equation (14) is calculated as represented by the equation (15),
This is stored in the position (x, y) of the frame memory 33 as a correspondence candidate position. x '(x, y) = x + (λ _n / 4) (15)

Similarly, when the phase difference sign is negative (-) or zero (0), x '(x, y) = x- (λ _n / 4) (in the case of negative) ... -(16) x '(x, y) = x (when zero) ... (17) is stored. Further, when this processing is completed, the control signal cntl indicating that the processing of the nth channel is completed is output.

In the depth calculator 34, the corresponding position detector 3
When the control signal cntl output from 2 indicates that the process on the channel with the highest frequency is completed, the values stored in the frame memory 33 are sequentially read out, and the parallax δ (x, y) is set to (18 ) Calculate as δ (x, y) = x−x ′ (x, y) (18)

Further, by substituting this into the equation (19), the relative depth ΔD (x, y) with respect to the gazing point P ₁ is calculated, and as distance information for the position (x, y) on the left image, Output this. ΔD (x, y) = δ (x, y) D ² / ab (19) where a, b, and D are the lens-to-lens distances (a) of the two cameras, the lenses and the images, respectively. The surface distance (b) and the distance from the lens to the gazing point (D) are constants determined by the arrangement of the cameras.

By the way, as is apparent from the equation (10), the maximum parallax that can be handled by each channel is proportional to the selective wavelength λ of the filter used. Therefore, in order to detect a wide range of parallax, it is necessary to prepare a channel with a low selection frequency (long wavelength).

On the other hand, the parallax information obtained in each channel is given by an inequality as shown in equation (13) and has ambiguity. The degree of ambiguity increases as the selection frequency of the filter used is lower (the wavelength is longer). Therefore, in the embodiment of FIG. 4, as shown in Expressions (11) and (12), the parallax information is propagated from the low-frequency channel to the high-frequency channel, so that there is a trade-off between the detectable parallax range and the detection accuracy. Has been resolved.

FIG. 5 shows an example of how parallax is propagated between channels. In this figure, for simplicity, the image is represented by only one dimension in the horizontal direction. In the figure, the waveform on the left side represents the left image, and the waveform on the right side represents the right image. In the first channel CH ₁ , the phase difference code between the position x ₀ of the left image and the same position x ₀ on the right image is detected. If this is a positive (+), as shown in equation (14), the position on the right image corresponding to x ₀ of the left image is in the range (x ₀ to x ₀ + λ _1/2) indicated by the diagonal lines. Range). Therefore, the center of this range x ₁ (= x ₀
The + λ _1/4), sent to the second channel CH _2. This processing corresponds to substituting x ₁ into the position (x ₀ , y) of the frame memory 33 according to the equation (15).

On the second channel CH ₂ , x _{0 of} the left image
On the other hand, the phase difference code from x _{1 of} the right image is detected. FIG. 5 shows a case where this sign is negative (-),
The center of the hatched portion x ₂ a _{(x 0 + λ 1/8} ), and sends to the third channel CH _3. In the last channel CH ₃ , a phase difference code between x ₀ of the left image and x _{2 of the} right image is detected, and the center of the range indicated by the code is detected on the right image corresponding to x ₀ of the left image. Position. The example of FIG. 5 shows that the phase difference code 0 is detected in CH ₃ , and therefore x ₂ in the right image corresponds to x ₀ in the left image.

The present invention is not limited to these embodiments, and for example, in the embodiment of FIG. 4, three or more channels can be prepared. Further, the frame memories 31L and 31R are connected to the respective spatial filters (12Li,
13Li, 12Ri, 13Ri) may be placed after the output of the filtering process. In this case, 2 for each channel (1 for left and right)
Although a frame memory for each frame is required, all the filtering processes are performed in parallel, so the time until the final parallax is detected can be significantly shortened.

[0050]

As described above, according to the present invention,
The left and right first and second images are filtered with different phase characteristics and output from each filtering process.
Outer product and inner product of are calculated to obtain phase difference and phase difference code
Is determined and the parallax is calculated based on the determination result.
Converted to distance . By combining the outputs of the respective filtering processes, the sign of the phase difference between the left and right images is detected, and distance information (depth information) indicating whether the target object is in front of or behind the gazing point is obtained.

When divided into a plurality of channels of different frequency bands, the parallax between the left and right images is calculated by combining the phase difference codes obtained in each channel, and the distance (ΔD) corresponding to the parallax is calculated. Is calculated.

Therefore, distance information can be obtained accurately and quickly.

[Brief description of drawings]

FIG. 1 is a block diagram showing a configuration of a first embodiment of a distance calculation device of the present invention.

FIG. 2 is a diagram illustrating outputs of spatial filters 12L and 13L and spatial filters 12R and 13R of FIG.

FIG. 3 is a diagram illustrating a relationship between outputs of an outer product calculator 14 and an inner product calculator 15 of FIG.

FIG. 4 is a block diagram showing a configuration of a second embodiment of the distance calculation device of the present invention.

FIG. 5 is a diagram for explaining the operation of the spatial filter of three channels in the embodiment of FIG.

FIG. 6 is a diagram showing the principle of binocular stereoscopic vision.

[Explanation of symbols]

10L, 10R camera 11L, 11R A / D converter 12L, 12R, 13L, 13R Spatial filter 14 Outer product calculator 15 Inner product calculator 16 Phase difference code determiner 31L, 31R frame memory 32 Compatible position detector 33 frame memory 34 Depth calculator

─────────────────────────────────────────────────── ─── Continuation of front page (58) Fields surveyed (Int.Cl. ⁷ , DB name) G01C 3/00-3/32 G01B 11/00-11/30 102 G02B ⁷ /28-7/40 G03B 13 / 32-13/36 G06T 1/00-9/40

Claims (3)

(57) [Claims] 1. The object is obtained from a first image and a second image obtained by imaging a target object with a first imaging device and a second imaging device which are arranged apart from each other in a horizontal direction by a predetermined distance. In a distance calculation device that obtains distance information about an object, a first filtering unit configured to perform a filtering process on the data of the first image, the first filtering unit including a plurality of spatial filters having equal amplitude characteristics and different phase characteristics; A second filtering unit configured to perform a filtering process on the data of the image, the spatial filter having a same amplitude characteristic and different phase characteristics, and an output of the first filtering unit and an output of the second filtering unit . Outer product calculating means for calculating an outer product, an output of the first filtering means, and a second filter
From the inner product calculating means for calculating the inner product of the outputs of the ring means and the output of the outer product calculating means and the output of the inner product calculating means,
Note that the phase difference and phase difference code between the first and second images are
Determination means for determining, and the phase difference and the phase difference determined by the determination means
Calculate the parallax between the first image and the second image from the code
And a conversion unit that converts the parallax into a distance . 2. The first filtering means and the second filtering means are divided into a plurality of channels each having a different frequency band, and each channel corresponds to a gazing point of the first image.
Corresponding position coordinates of the second image to the channel
Corresponding position detection hand calculated from the wavelength and the phase difference code
And the position coordinates calculated by the position detecting means are recorded.
Further comprising a recording means for converting the position of the first image into the position corresponding to the gazing point of the first image.
The parallax is calculated from the position coordinates of the target and the second image,
The distance calculation device according to claim 1, wherein the parallax is converted into a distance. 3. Arranged horizontally at a predetermined distance.
The target object by the first image pickup device and the second image pickup device
From the first image and the second image obtained by imaging, the object
Distance calculation method of distance calculation device for obtaining distance information about body
Method, filtering the data of the first image,
Multiple space frames with equal width characteristics but different phase characteristics
A first filtering step comprising a filter, and a filtering process for filtering the data of the second image.
Multiple space frames with equal width characteristics but different phase characteristics
A second filtering step consisting of a filter and an output from the processing of the first filtering step.
Outer product of outputs from the processing of the second filtering step
And product computation step of computing an output by the processing of said first filtering step
Inner product of outputs from the processing of the second filtering step
And the output and inner product calculation step by the processing of the outer product calculation step.
From the output of the first processing and the second image.
Determination step for determining the phase difference and the phase difference sign of the image
And the phase difference determined by the processing in the determination step.
And the phase difference code, the first image and the second image are viewed.
A conversion step of calculating a difference and converting the parallax into a distance;
A method of calculating a distance, comprising: