JPH02268215A - Method and apparatus for detecting distance - Google Patents

Abstract

PURPOSE:To detect the accurate distance in the depth direction and to decrease the amounts of required operations by performing the operation of parallax by a correlation method only when the changing value of luminance is higher than a preset threshold value with respect to a blocks obtained by dividing a first image. CONSTITUTION:Operation for obtaining a parallax with a second image for every block obtained by dividing a first image is performed. The number of data of the changing values of the luminances of the divided blocks which are higher than preset threshold value is counted. The counted value is made to be an evaluating value for determining whether the parallax is to be operated. When the evaluated value is less than the threshold value, the operation is not performed. A code which indicates the unstable parallax is written into a parallax distribution memory 16. When the evaluating value is higher than the threshold value, the data in a parallax operating part 15 are read out. The correlation value between the data and a second image having the same size is operated. The distance to an object in the depth direction is detected by a triangulation method based on the obtained parallax. Thus, the accurate distance is detected, and the amounts of the required operations can be decreased.

Description

DETAILED DESCRIPTION OF THE INVENTION [Industrial Application Field] The present invention relates to a distance detection method and apparatus using image measurement. More specifically, the present invention provides a distance detection method that uses a pair of stereo images projected from different positions to determine the distance in the depth direction of an object with high precision and with fewer calculations. The aim is to provide the equipment.

[Prior art] Extracting only objects at a specific distance, or
BACKGROUND ART Various methods have been conventionally used to detect the distance in the depth direction of an object in a non-contact manner for the purpose of determining the positional relationship of objects in a dimensional space.

As such a detection method, there is a correlation method in which a required calculation is performed using a pair of two images projected from different positions.

FIG. 3 shows the principle of distance detection using this correlation method. In Figure 3, A1 is the center of the lens of the first camera, and A2 is the center of the lens of the second camera. Generally, the two cameras are on the same horizontal plane and their optical axes are parallel. are arranged so that Therefore, when a set of two images of the object are shot, the point P on the object is Xl
, Yl as the coordinate axis, a point P1 (Xl,
Yl), X2. In the second image with Y2 as the coordinate axis, it appears at point P2 (X2, Y2) on the line connecting point P and the center A2 of the lens of the second camera. Since the two cameras are placed on the same horizontal plane, the first
There is no vertical displacement between the image and the second image, and there is a horizontal displacement (parallax e) depending on the distance in the depth direction.

e-xl x2> is generated.

There is a relationship between the parallax e and the depth direction Z to the object as follows: Z = (fxh>/ (gxe> (1). Here, f is the focal length of the lens, and h is the focal length of the first camera. and the second camera, q is a conversion coefficient. Therefore, by calculating the distance in the depth direction from each parallax e for various points P in the target image, a distance distribution map of the image can be obtained. Obtainable.

Furthermore, if the parallax is determined for each block into which the image is divided, a distance distribution for each block can be obtained. That is, blocks of the same size as the blocks of a predetermined size divided in the first image are sequentially shifted in the horizontal direction, and the blocks of the first image and the second image are opened at the respective shifted positions. The correlation value between both blocks is calculated, and the position of the block having the highest correlation value among the obtained correlation values is determined. This corresponds to finding a matching correspondence relationship between blocks in the first image and blocks in the second image. Therefore, the difference in the horizontal direction between the obtained block position of the second image and the block position of the first image is regarded as the parallax, and from this parallax and the known jump position, the target is calculated using the method of triangulation. Find the distance to the object by calculating backwards.

The correlation method described above has been applied to a distance detection method that uses hierarchical images with a plurality of different resolution levels, making it possible to perform coarse and fine searches efficiently. In other words, the disparity is determined by the correlation method from a pair of images at the coarsest resolution level, and the obtained disparity is then used as the initial value for the disparity calculation using the next coarsest resolution level.The process of determining the disparity between images is repeated sequentially, and finally, the disparity between images is determined from the pair of images at the coarsest resolution level. The parallax between the images at the resolution level is determined, and the distance in the depth direction to the object is detected from this parallax using a triangulation method.

[Problems to be Solved by the Invention] According to the conventional distance detection method using the correlation method, in a part where the brightness value of the image is constant or gradually changes, the correlation value between the blocks of the first image and the second image is Since the change is extremely small, it is easy to mistakenly recognize blocks that are not in a position where the correlation is high as having a matching correspondence relationship.

Therefore, even if the distance distribution of the entire image is obtained, when looking at individual distance values, there are parts where the correlation method works effectively and accurate distance values are obtained, and parts where the correlation method does not work effectively and gives incorrect distance values. If you perform object extraction or positional relationship recognition processing using information that includes incorrect distance values, you may extract too many unnecessary objects or receive incorrect distance values. There was a problem to be solved in that the recognition results caused disturbances to the processing process using the distance detection results.

Such problems remain unresolved even when the correlation method is applied to a distance detection method using hierarchical images having a plurality of different resolution levels.

[Means for Solving the Problems] The present invention has been made in light of such problems to be solved, and is based on the best method of triangulation shown in FIG.
For each block into which the image is divided, if the luminance change evaluation value is greater than or equal to a preset threshold, parallax calculation is performed using the correlation method, and if it is less than the threshold, the correlation method is not applied and the neighboring neighboring blocks around the block are Interpolation is performed from the block parallax, and the distance to the object in the depth direction is detected using the triangulation method from the obtained parallax.

In addition, such a method is sequentially applied to each of hierarchical images having a plurality of different resolution levels to determine the parallax,
Using the finally obtained parallax, the distance to the object in the depth direction is detected by a triangulation method.

[Effects] By taking such measures, accurate distance values in the depth direction can be obtained even when performing distance detection using the correlation method using an image that includes blocks with different brightness changes. Moreover, it has become possible to reduce the overall amount of computation required.

[Embodiment] The present invention is based on the method of triangulation ω explained with reference to FIG. 3, and the circuit configuration of one embodiment thereof is shown in FIG. 1A and will be explained.

In FIG. 1A, each signal 4'lA, 41B indicating the first image and the second image taken by the first camera 31A and the second camera 31B is connected to each analog digital (
A/D> Each signal 42A, 4 is digitally converted by the converter 32A, 32B, and represents the obtained luminance data.
2B are respectively written into the first image memory 12A and the second image memory 12B of the parallax calculation interpolation section 11. The first image memory 12A and the second image memory 12B are two-dimensional arrays of N rows and M columns corresponding to the number of pixels N in the vertical direction and M in the horizontal direction of the digitally converted image, respectively.

The brightness change detection unit 13 that receives the signal 52C indicating the brightness data of the first image read out from the first image memory 12A having such a configuration detects each of M pixels in the horizontal direction and MXN pixels in the vertical direction. The brightness change value is calculated and detected. In this embodiment, the spatial differential value calculation result is used as the brightness change value, and the brightness change value V is as follows: Sx= u (x+1.y+1) +2 u (X+L
V) 10u (X+1.y-1) -u (X-Ly+1) -2u (x-1,y)-u
(x-1, y-1) Sy= u (x+Ly-1) +2 u (x, y-
1) + u (x-1, y-1) u (x+1.y+1) -2 days (x, y+1) u (
x-1, y+1). Here, U represents the luminance data of the pixel.

FIG. 1B is a flowchart showing the flow of control operations in the brightness change detection section 13 (FIG. 1A) for obtaining this brightness change value.

In FIG. 1B, the first image memory 12A (FIG. 1A)
Read out the signal 52C indicating the luminance data of the first image (3101), set the pixel at the position where the column number Execute (S103).

Sx= u (X+LV+1) +2 u (X+1.
V) + IJ (X+1.y-1) u (x-1, y+1) -2u (x-1, V)-u
(x-1, y-1) Sy= u (x+Ly-1) +2 u (x, y-
1) + u (x-1, V-1) -u (x+1.y+1) -2LJ (x, y+1
)u (x-1, y+1) Using the calculation results obtained in step 3103, calculations are performed to determine the luminance change value V according to calculation formula (2) (3104).

When the brightness change value ■ for one pixel is determined, it is asked whether the column number SX at the pixel position is smaller than the value obtained by subtracting 1 from the number of pixels M in the horizontal direction (3105). If so (S105Y), advance the value of X by 1 (31
06), repeat the work from step $103. Unless column number X is smaller than M minus 1 (310
5N), it is asked whether the row number y of the pixel position is smaller than the value obtained by subtracting 1 from the number of pixels in the vertical direction (N). <510
7) If the value is small (S107Y), the line number y is incremented by 1 (5108), and the operations from step 3103 are repeated. If the row number y is not smaller than the value obtained by subtracting 1 from N (3107N), the brightness change value V for the required pixel has been determined, and the process ends.

As described above, the signal 53 indicating the brightness change value for each pixel of the first image obtained by the brightness change detection unit 13 in FIG. 1A is arranged in a two-dimensional array of N rows and M columns corresponding to the number of pixels. The brightness change value memory 14 has a brightness change value.

In FIG. 1A, a signal @54 indicating a luminance change value from the luminance change value memory 14 and signals 52A and 52B indicating respective image data from the first image memory 12A and the second image memory 12B are received. The parallax calculation unit 15 sequentially scans each block of the first image divided so as not to overlap each other from the upper left block to the lower right block, and calculates the difference between the first image and the second image in each block. The parallax between the images is calculated and detected. Here, if the number of pixels per block is p pixels in the horizontal direction and 0 pixels in the vertical direction, the number of blocks in the horizontal direction of the image M is M/p, and the number of blocks in the vertical direction N1 is N/0. becomes.

To explain the operation of the parallax calculation unit 15, a signal 54 indicating a brightness change value from the brightness change value memory 14 is stored in a temporary storage area T1 with a capacity of N rows and M columns provided inside the first image memory. A signal 52A representing the first image from the second image memory 12A is stored in a temporary storage area T1 having a capacity of N rows and M columns, and a signal 52B4 . tN
Each is written to a temporary storage area T2 having N matrix and column capacity.

Therefore, for each divided block of the first image, calculations are performed to obtain the parallax between the first image and the second image for each block starting from the upper left block to the final lower right block. That is, among the data (+)
The number of pieces of data whose change value is equal to or greater than a preset change value threshold is counted, and the obtained count (direction) is used as an evaluation value for determining whether or not to calculate the parallax for the block. If this evaluation value is less than the preset evaluation fiffiflIf threshold, the disparity is assumed to be undefined and no calculation is performed to obtain it, and a special code representing the undefined disparity is written to the block position in the disparity distribution memory 16. As a special code, parallax "011", which corresponds to an infinite distance, is used.

If the evaluation value for that block is greater than or equal to the evaluation value threshold, data B1 for that block is read out from the temporary storage area T1 in the parallax calculation unit 15, and blocks of the second image having the same size as that block are sequentially horizontally read out. The position of the block in the second image where the correlation value is the minimum is determined by calculating the correlation value between both blocks, and the determined positional shift skew is used as the parallax for that block.

FIG. 1C shows how the range for searching for parallax is set by shifting the blocks of the second image in the horizontal direction. In FIG. 1C, the search range is the initial parallax value d. Set S pixels on the left and right of the center. Parallax initial value d. A signal 40 (FIG. 1A) indicative of , d3 are O. The parallax dt at the end of the search is d.+S5 pixels.

Therefore, first, the assumed parallax value d is set to d3 pixels, and the parallax d
The initial value of is set to d, and the initial value of the minimum error value Emin is set to +■. Once these values are determined, the assumed parallax value d is calculated from the position indicated by the block data B1 of the first image.
The data B2 of the block of the second image at the position horizontally shifted by pixels is successively outputted from the temporary storage area 10T2 included in the parallax calculation unit 15, and the correlation value between both blocks is determined. The correlation value is obtained by calculating the error evaluation value Err between both blocks according to the following calculation formula.

In addition to the calculation formula (3>), the correlation value may be calculated using the following calculation formula (4).

If the error evaluation value Err obtained in this manner is less than the minimum error value Emin, the parallax d is set as the assumed parallax value da, and the minimum error value EIIlin is set as the error evaluation value Err. Once the error evaluation 111Err is determined, the assumed parallax value da is advanced by one pixel, and the error evaluation value Err is determined in the same manner.

The above calculation is repeated until the assumed parallax value da reaches a preset maximum value dt or does not exceed the second image area, and the parallax d at the end of this repeated calculation is taken as the parallax in that block, A signal 55 (FIG. 1A) indicating this is sent to the parallax distribution memory 16.

FIG. 1D and FIG. 1E are flowcharts showing the flow of the control operation in the above-mentioned parallax calculating section 15 (FIG. 1A). In FIG. 1D, a signal 54 indicating the luminance change value V is sent from the luminance change value memory 14 (FIG. 1A), a signal 52A indicating the first double image data is sent from the first image memory 12A,
A message indicating the second image data from the second image memory 12B @5
2B respectively (5111), and set the column number i and row number j of the block position of the first image to 1 (S112>, the back of the data indicating the luminance change value of the block l+, j>, The number of pieces of data whose change value is equal to or greater than a preset change value threshold is determined, and this count value is used as the evaluation value for the block (i, j>) (S113).

It is asked whether the evaluation value, which is the obtained change value, is greater than or equal to the evaluation value threshold (3114), and if the value is less than the threshold (3114N), the disparity d (i,, J) is indefinite. 5115), the process moves to step 5117 without performing any calculation to obtain this. If the evaluation value, which is the change value, is greater than or equal to the threshold (S114Y), the block of the first image (
The position of the block in the second image with the minimum correlation value between the blocks (I, J) is determined, and the amount of positional shift is set as the parallax (i, j) (S116).

A signal 55 (FIG. 1A) indicating a special code indicating that the obtained block disparity d (i, j>, or that it is indeterminate) is written into the disparity distribution memory 16 (3117).

Therefore, it is asked whether the column number g+ of the block position of the first two images or the number M of blocks in the horizontal direction is a smaller value (
3118), if the value is small (S118Y), the column number i is incremented by 1 (S119), and the operations from step 5113 are repeated. If it is not a small value (3118N), it is asked whether the row number j is a value smaller than the number of vertical blocks Nq (S120>, and if it is a small value (Sl 2
0Y), the line number j is incremented by 1 (S121), and the operations from step 5113 are repeated.

If the value is not small (Sl 2ON>), the disparity calculation has been performed for all blocks, and the work is finished.

FIG. 1E shows the details of the work flow in step 3116 (FIG. 1D). When the data B1 of the block of the first image is read from the temporary storage area T1 of the parallax calculation unit 15 (S131), Horizontal search range d8 for blocks in the second image. 5132), set the assumed parallax value d to d88 pixels, set the initial value of the parallax d to d, and set the minimum error value Em.
Set the initial value of in to +■ (S133)
.

Once these values are determined, the data B2 of the block of the second image at a position horizontally shifted by da pixels from the position of the block of the first image is successively outputted from the temporary storage area T2 of the parallax calculation unit 15 (5134). According to formula (3),
An error evaluation value between data B1 and data B2 is calculated (S135>).

It is asked whether the obtained error evaluation value Err is smaller than the minimum error value E min (3136), and if it is smaller (S136Y), the parallax d is set to the assumed parallax value d,
The minimum error value E ・ is the error evaluation value “rr and a
mIn (S137). Once these values are determined, the assumed parallax value d is set to the maximum value d
It is asked whether the value is smaller than t (3138).

In step 5136, if the error evaluation value Err is not smaller than the minimum error value Emin (3136N>,
The process moves to step 8138. If the assumed disparity value da is smaller than the maximum value d1 (S138Y), it is asked whether a block of the second image at the position of the assumed disparity value da exists (3139), and if it exists (S139Y), The parallax assumption l11fda is advanced by one pixel (S140), and the operations from step 5134 are repeated. If the block of the second image does not exist (Sl 39N>), the parallax d at that point is set as the parallax d(+, j) of that block (S14
1) Finish the work. In step 8138, if the assumed parallax value d is less than the maximum value dt (31
38N>, the process moves to step 5141 and ends the work.

In FIG. 1A, as a result of calculation by the parallax calculation unit 15,
Since the evaluation value, which is data for determining whether or not to calculate the parallax, is less than the threshold value, the parallax is assumed to be indefinite, and the application of the correlation method to that block may be determined to be invalid.

Therefore, the parallax interpolation calculation unit 17 calculates and obtains the parallax of the block whose parallax is determined to be undefined by interpolating the parallax of the blocks around it.

Methods of interpolation include linear interpolation, least squares interpolation, etc., but in this embodiment, a method of interpolating using the average value of surrounding blocks is used. Specifically, the parallaxes of all blocks are read out from the parallax distribution memory 16, and the following calculations are performed for all blocks except those in the peripheral area of the image.

That is, the disparity of the block is read out, and if it is "'O" which is a special code indicating undefined disparity, the disparity of the four adjacent blocks above, below, left and right of the block is read out. If it is a code other than the special code, the required parallax is already 1q and is necessary, so the process moves on to the next block. Assuming that the disparity is 110 T1, disparity distribution memory 1
Among the four disparities of adjacent blocks read from 6, if the number of blocks with disparities other than "'O" is greater than or equal to a preset value, the disparity 1 (011
The average disparity value of the blocks other than the block is determined, and using this as the disparity of that block, the data at the block position in the disparity distribution memory 16 is corrected to the newly obtained disparity, and the process moves on to the next block. If the number of blocks with a parallax other than 11011 is less than the set value, interpolation from surrounding blocks is not performed and the process moves to the next block.

1F to 1H show the above parallax interpolation calculation unit 17.
1A is a flowchart illustrating the flow of control operations in FIG. 1A.

In FIG. 1F, the parallax distribution memory 16 (FIG. 11A)
The column number i and row number j of the block position from which the parallax is to be read are each set to 2 (S151), and the block (
The parallax d(i,,j> of i, j> is read out (3152).

Parallax d (i.

j) is a special code indicating undefined parallax (3153), and if it is a special code (3153Y),
The count number and integrated value are each set to O (S154
), the disparity d(i-1,j> of block (i-1,j>) is read from the disparity distribution memory 16 (3155).The read disparity d(i-1゜j) is a special It is asked whether it is a code (3156), and if it is not a special code (8156N), the count is advanced by 1 and a parallax d (i-1゜j) of -1,j> is added to that block as an integrated value. Add (3157). Next, block (i+1.j>
The parallax d (i+1.j> is read out (S158). In step 5156, if the parallax d (i-1, j> is a special code (S156Y), the process moves to step 8158. The read parallax d ( i+1.J) is a special code indicating undefined parallax (3159), and if it is not a special code <3159N), the count number is set to 1.
Then, the parallax d (i+1j> of that block (i+1.j>) is added as an integrated value (3160, Fig. 1G).

Next, the disparity d(j) of block (i, j-1) is read out (3161). In step 5159 (Fig. 1F), if the disparity d (i+1.J) is a special code, (S
l59Y), the process moves to step 8161. It is asked whether the read parallax d(i, Jl) is a special code indicating undefined parallax (3162), and if it is not a special code (8162N>), the count is advanced by 1 and the integrated value is the parallax d( i, j−H are added (S163).

Roughly speaking, the disparity d (i.

j+1) (3164). In step 5162, if the disparity d(i, j-1> is a special code, then (S
l 62Y), the process moves to step 8164. It is asked whether the read parallax d (i, j+1> is a special code indicating undefined parallax (S165), and if it is not a special code (3165N), the count number is advanced by 1 and the integrated value is set as the parallax d (+ , ”+1) are added (3166>.

It is asked whether the count number obtained in this way is larger than the set threshold value (3167, first
Figure H). In step 3165 (Figure 1G), if the parallax d (i, j+1> is a special code, then (3165Y
), the process moves to step 3167. If the count number is larger than the threshold (S167'Y), the average value obtained by dividing the obtained integrated value by the count number is calculated as the parallax d after interpolation.

j> and write it into the parallax distribution memory 16 (
8168). When the newly corrected parallax d(j) is 1q, it is asked whether the column number i of the block position is smaller than the value obtained by subtracting 1 from the number of blocks M1 in the horizontal direction (3169). In step 3153 (FIG. 1F), the disparity d(i.

j) is not a special code (8153N), and if the count obtained in step 8167 is not larger than the set threshold (3167N>, the process moves to step 5169.Step 8169
, if the block column number @i is smaller than the value obtained by subtracting 1 from Mp (S169Y), the column number i is set to 1.
Proceed (S170) and repeat the operations from step 3152 (FIG. 1F). If it is not a small value (3169N)
, it is asked whether the row number of the block position is smaller than the value obtained by subtracting 1 from the number of blocks Nq in the vertical direction (S1
71), if the value is small (S171Y), the line number j is incremented by 1 (S172), and the operations from step 3202 are repeated. If the value is not small (3171N> processing for parallax interpolation has been performed for all blocks, the work is finished.

Note that the above calculation for parallax interpolation is not performed unless all blocks of the first image show a special code with undefined parallax.

In FIG. 1A, the distance calculation unit 3 receives a signal 56 indicating the parallax for each block from the parallax distribution memory 16.
3 is obtained by calculating the distance of each block according to the arithmetic expression (1) used in the correlation method.

To explain its operation, the signal @56 indicating the disparity for all blocks is read out from the disparity distribution memory 16, and the disparity is 1.
No calculation is performed on the block 4011, and a signal 43 indicating infinity or the maximum value allowed in the distance distribution memory 34 is output. Parallax is OI? For other blocks, from the known focal length of the lens, the distance h between the first camera and the second camera, the conversion coefficient, and the obtained parallax,
Calculate the distance Z of the block according to formula (1),
A signal 43 indicating the calculation result is written into the distance distribution memory 34, and finally a distance detection output 44 for all blocks of the image is obtained.

FIG. 11 is a flowchart showing the flow of control operations in the distance calculation unit 33 (FIG. 1A), in which the column number @i and number @j of the block position are set to 1, respectively (
8181), block (i.

It is asked whether the disparity d(i, j> of j) is a special code indicating undefined disparity (3183), and if it is not a special code (183N>, the distance z (i, j) of the block is The calculation result is calculated according to the calculation formula % formula %) shown in (8184) and the calculation result is written into the distance distribution memory 34 (FIG. 1A).The distance z of the block is
(When i, j> is obtained, the column number i of the block position is the number of blocks in the horizontal direction M. It is asked whether it is a smaller value (3186).

In step 8183, if the parallax d(i, j) is a special code indicating undefined parallax (S183Y), the special code is written in the distance distribution memory 34 (3185), and the process moves to step 8186. In step 8186,
If the column number i is the block number M, a smaller value (31
86Y), the column number i is incremented by 1 (3187), and the operations from step 8182 are repeated. If it is not a small value (3
186N>, it is asked whether the row number j at the block position is a smaller value than the number of blocks Nq in the vertical direction (318
8), if the value is small (S188Y), set the line number j to 1.
Proceed (3189) and repeat the work from step 8182, and if the value is not small (3188N), the process ends because distance calculation processing has been performed for all blocks.

FIG. 1J is a flowchart showing the flow of control operations of the entire apparatus in the embodiment shown in FIG. 1A.

In FIG. 1J, the first camera 31A (FIG. 1A) takes a first image (5191), and the second camera 31B takes a second image (S192). The first photo taken
The brightness change detection unit 13, which has received the signal 52C indicating the brightness data of the image, performs calculations to detect a brightness change value for each pixel forming the image (3193). From the signal 54 indicating the obtained luminance change value and the multiplex @ 52A, 52B indicating the image data of the first image and the second image, the parallax calculation unit 15 calculates for each divided block of the first image. Calculate the parallax between the first image and the second image (
3194). As a result of the calculation by the parallax calculation unit 15, the application of the correlation method is determined to be invalid, and the parallax of a block whose parallax is undefined is determined to be undefined. This is done by (
S195). The calculation formula (
The distance is calculated according to 1) (3196), and the work is finished when the distances of all blocks are detected.

FIG. 2A shows a circuit configuration of an embodiment of the present invention, and components corresponding to those in FIG. 1A will be explained using the same symbols.

In FIG. 2A, the difference from the circuit configuration of the embodiment shown in FIG. 1A is that the third layer processes the input image at the coarsest resolution, the second layer processes it at an intermediate resolution, and the densest layer. This is established from the first layer, which processes at a high resolution. Here, in order to obtain three layers of images including the original image with the densest resolution, second layer image generation units 21A and 21B are used to obtain 1-set of intermediate resolution images.
and a third layer image generation unit 22A, 22B for obtaining a set of images with the coarsest resolution;
A third layer parallax calculation interpolation unit 23 calculates and interpolates the parallax from the hierarchical images, and a signal 63 indicating the parallax obtained by the third layer parallax calculation interpolation unit 23 calculates the parallax from the second image with an intermediate resolution. The parallax is calculated from the second layer initial value generation unit 24 for generating the initial value for interpolation, and the signals 61A and 61B representing the second layer images from the two second layer image generation units A and 21B, respectively. A second layer parallax calculation interpolation unit 25 for calculation interpolation, and a second layer parallax calculation interpolation unit 25
A first layer initial value generation unit 26 for generating an initial value when calculating and interpolating the parallax from the first layer image with the finest resolution from the signal 65 indicating the parallax obtained by
A first layer parallax calculation interpolation unit 27 is provided to calculate and interpolate parallax from each signal 42A, 42B indicating the input image from the /D conversion unit 32A, 32B. here,
The configurations of the third layer parallax calculation interpolation section 23, the second layer parallax calculation interpolation section 25, and the first layer parallax calculation interpolation section 27 are as follows.
The configuration is the same as that of the parallax calculation interpolation unit 11 in the embodiment shown in FIG.

Signals 41A and 41 representing the first and second images captured by the first camera 31A and the second camera 31B
B is digitally converted into image data for the first layer by each A/D converter 32A, 32B, and each signal 42A, 42B indicating the obtained luminance data is converted to a first layer difference calculation interpolator 27. The data are written into the first image memory and second image memory respectively, and are also input to the respective second layer image generation units 21A and 21B. Here, the number of pixels in the vertical direction of the first layer image is N1 pixels, and the number of pixels in the horizontal direction is M1 pixels.

Each second layer image generation unit 21A, 21B performs a spatial filter operation on each signal 42A, 42B indicating a first layer image, which is an original image, and each signal 61A, indicating a second layer image coarsened by one level, 61B is generated and written to the first image memory and second image memory in the second layer parallax calculation interpolation unit 25. The image of the second layer is the image of the first layer reduced by half vertically and horizontally by a spatial filter, and the number of pixels in the vertical direction N2 is N1/2 pixels, and the number of pixels M2 in the horizontal direction is M1/2. This means that data for one pixel of the second layer image is created from 2×2=4 pixels of the first layer image. Spatial filter calculation methods to be used include a low-pass method and a maximum value selection method, but in this example, the average value of a total of 4 pixels ('m2 pixels and 2 pixels horizontally) of the image on the first layer is used as the average value of the image on the second layer. An average value filter method is used that uses data for one pixel.

FIG. 2B is a flowchart showing the flow of control operations in each second layer image generation section 21A, 21B (FIG. 2A).

In FIG. 2B, multiplex @42A, 42B indicating the image data of the first layer is input from each A/D converter 32A, 32B (3201), column number 12 and row number of the pixel position of the image of the second layer. Set j2 to 1 (3202>
. Next, set the column number 11 of the pixel position of the first layer image to (
i2-1)X2+1, line number j as (,j2-1)X2
+1 (3203>, 4 pixels (1
1゜jl ), (il +1.jl >, (il, j1
+1>, (i1+1.j1+1>) each brightness value u1(11
゜jl)・ul(!1+1・jl>・ul(!1・J
l +1>, ul (!1 +”l, Jl +1>
are added to obtain a total value (S204). The obtained brightness value is divided by 4 to calculate the pixel of the second layer image (12゜j2)
The luminance value u2 (!2.J2) of (S205)
. When the luminance value u2 (!2.j2 > is obtained, the second
Column number 12 of the pixel position of the layer image is the number of pixels in the horizontal direction M
It is asked whether the value is smaller than 2 (3206), and if it is necessarily a smaller value (S206Y), the column number @12 is incremented by 1 (3207), and the operations from step 5203 are repeated.

Unless the column number 12 is smaller than the number of pixels M2 (32
06N), it is asked whether the row number j2 is a smaller value than the number of pixels in the vertical direction N2 (3208>, and if it is a smaller value (S208Y), the row number j2 is incremented by 1 (3209).
, repeat the operations from step 5203.

Unless the row number j2 is smaller than the number of pixels N2 (32
08N>, the brightness values of all the pixels that constitute the second layer image have been obtained, so the work is finished.

In FIG. 2A, each second layer image generation unit 21A, 21
From each signal 61A, 61B indicating the luminance value of each pixel constituting the second layer image obtained by B, the third layer image generation units 22A, 22B generate a second layer image using the average value filter method. Assuming that the average value of 4 pixels of the image is the data of 1 pixel of the 3rd layer image, the number of pixels in the vertical direction N5 = N2
/2 pixels, the number of pixels in the horizontal direction M3 = M2 /2 pixels. Write each to image memory.

The flow of control operations in each of the third layer image generation sections 22A and 22B is as shown in FIG. 2B.
Since the flow of control operations is the same as in A and 21B,
The explanation will be omitted.

The third layer receives each signal 62A, 62B indicating a third layer image obtained by each third layer image generation section 22A, 22B.
The hierarchical parallax calculation interpolation unit 23 has N3 pixels in the vertical direction and M pixels in the horizontal direction.
The first and second images of the third layer consisting of three pixels are
Each block is divided into blocks of 0 pixels in the vertical direction and 0 pixels in the horizontal direction, and the parallax of each block is calculated and determined using the same method as the parallax calculation method in the embodiment shown in FIG. 1A.

When the parallax of each block constituting the third-layer image is obtained, the second-layer initial value generation unit 24 sends a message @63 indicating the calculation result read from the parallax distribution memory in the third-layer parallax calculation interpolation unit 23. From this, an initial parallax d2o for parallax calculation in the second layer parallax calculation interpolation unit 25 is generated and stored. Here, the number of blocks N3q in the vertical direction of the third layer image is N
3/Ql, and the number of blocks N2° in the vertical direction of the second layer image is N2/q as described later, and N2-2
There is a relationship of ×N3. A similar relationship exists between the horizontal block number M3q of the third layer image and the horizontal block number M2 of the second layer. Therefore, the second layer initial value generation unit 24 uses the calculation result for one block of the third layer image as a parallax initial value that is enlarged and corresponds to the four blocks of the second layer image, and uses its internal Temporarily stored in the second layer parallax initial value array memory D2.

FIG. 2C is a flowchart showing the flow of control operations in the second layer initial value generation section 24 (FIG. 2A).

In FIG. 2C, the third layer parallax calculation interpolation unit 23 (second
Read out the signal 63 indicating the calculation result from Figure A) 3211
), the column number i3 and row number j3 of the block position of the image of the third layer are each set to 1 (3212>. Next, the column number 12 of the block position of the image of the second layer is set to (i
3 1) X2+1, numbered @j2 (J3 1>x2+1
(S213>, that block of the third layer image (
i3. The parallax d3 (13, J3> of j3) is set to four blocks (12, J2>) of the second layer image.

(i2+1.J2), (!2.J2+1), (2+1.
Each initial parallax value of J2+1>, d20('2,
j2), d20i2 +'l, J2 >, d2o(
12,j2+1), and d2o(12+1.j2+1) (3214>. When these initial parallax values are obtained, the column number 13 of the block position in the third layer image is the horizontal block @M3. The question is whether the value is smaller than (
8215>, if the value is small (S215Y), column number i
3 is incremented by 1 (<5216), and the operations from step 5213 are repeated. Column number i is the number of blocks in the horizontal direction M3. If it is not a smaller value (821, 5N), it is asked whether the row number j3 is smaller than the number of vertical blocks N3° (S217>, and if it is a smaller value (S217Y)
, advance the line number j3 by 1 (S218), and step 52
Repeat the steps from step 13. If the row number j3 is not smaller than the number of vertical blocks N3q (5217N), the second
Since the processing for obtaining the initial parallax value of the block of the hierarchical image has been performed, the work is finished.

In FIG. 2A, the second layer in the second layer initial value generation section 24
A signal 64 indicating the parallax initial value read out from the layer parallax initial value array memory D2, each second layer image generation unit 21A,
Multiple magnification showing second layer image from 21B @61A, 61
The second layer parallax calculation interpolation unit 25 that receives B calculates each block from the first and second images of the second layer using the same method as the parallax calculation method in the embodiment shown in FIG. 1A. Obtain by calculating the parallax.

When the parallax of each block constituting the second layer image is obtained, the first layer initial value generation unit 26 generates a signal 65 indicating the calculation result successively from the parallax distribution memory in the second layer parallax calculation interpolation unit 25. , generates and stores an initial parallax d1o for parallax calculation in the first layer parallax calculation interpolation unit 27. Here, the number of blocks in the vertical direction of the second layer image N2. is N
2/q, number of blocks in the vertical direction of the first layer image N1q
As will be described later, there are N 1 / Q numbers, and N1=2X
There is an N2 relationship. The same relationship exists even if there is one block number M2p in the horizontal direction of the second layer image and one block number M1 in the horizontal direction of the first layer. Therefore, the first layer initial value generation unit 26 uses the calculation result for one block of the second layer image as a parallax initial value that is enlarged to correspond to the four blocks of the first layer image, and uses the internal It is temporarily stored in the first layer parallax initial value array memory D1. The flow of control operations in the first layer initial value generation section 26 is similar to the flow of control operations in the second layer initial value generation section 24 shown in FIG. 2C, so a description thereof will be omitted.

A signal 66 indicating the parallax initial value read out from the first layer parallax initial value array memory D1 in the first layer initial value generation unit 26
The first layer parallax calculation interpolation unit 27, which receives the signals 42A and 42B indicating the input images from the A/D conversion units 32A and 32B, calculates the input image from the first image and the second image as shown in FIG. 1A. The parallax of each block is calculated and determined using the same method as the parallax calculation method in the embodiment.

The distance calculation unit 33 that receives the signal 67 indicating the calculation result from the first layer parallax calculation interpolation unit 27 and the distance distribution memory 34 that writes the signal 43 indicating the calculation result are configured in the first layer parallax calculation interpolation unit 27.
This is the same as the embodiment shown in the figure, and finally the distance detection output 44 for all blocks of the image is obtained by 1q.

FIG. 2D is a flowchart showing the control operation of the entire apparatus in the embodiment shown in FIG. 2A.

In FIG. 2D, the first camera 31A (FIG. 2A) takes a first image (3221), and the second camera 31B takes a second image (S222). The first photo taken
The second layer image generation unit 2 receives the signal 42A indicating the image.
1A generates the first image of the second layer that is one step coarser32
23>, the second layer image generation unit 21B that receives the signal 42B indicating the second image generates a second image of the second layer that is coarser by one level (3224>. The third layer image generation unit 22A that receives the signal 61A generates a first image of the third layer that is one step coarser.
25>, the third layer image generation unit 22B, which has received the signal 61B indicating the second image of the second layer, generates a second image of the third layer that is coarser by one level (3226). Each third layer image generation unit 22A.

Each signal 62 indicating the third layer image obtained by 22B
The third layer parallax calculation interpolation unit 23 that receives A and 62B calculates the parallax of each block obtained by dividing the image (3227>
. From the signal 63 indicating the obtained disparity, the second layer initial value generation section 24 generates a disparity initial value for the disparity calculation in the second layer disparity calculation interpolation section 25 (3228), and generates a signal 64 indicating this initial value. The second layer parallax calculation interpolation unit 25 that has read out calculates the parallax of each block obtained by dividing the second layer image (S229). From the signal 65 indicating the obtained disparity, the first layer initial value generation section 26 generates a disparity initial value for the disparity calculation in the first layer disparity calculation interpolation section 27 (3230). The first layer parallax calculation interpolation unit 27 that has read out 66 calculates the parallax of each block obtained by dividing the first layer image (3231). When the parallax of each block of the first layer image is obtained, the distance calculation unit 33 calculates the distance of each block from the signal 67 indicating the parallax (S232>, and ends the operation.

Although the above description has been made assuming that the number of image layers is three, the present invention can also be applied to hierarchical image structures having a plurality of different resolution levels.

[Effect of the invention] As is clear from the above explanation, according to the present invention,
A correlation method for distance detection is applied to each block into which the photographed image is divided, and it is determined whether or not the force ζ is valid.For blocks that are determined to be valid, calculations are performed using the correlation method. Since we decided to interpolate the distance values of surrounding adjacent blocks for blocks that have been detected, it is possible to accurately detect the distance in the depth direction.Moreover, since calculations using the correlation method are not performed for invalid parts,
As a result, the overall amount of required calculations can be reduced. Further, by applying such a method to each hierarchical image having a plurality of different resolution levels, it becomes possible to realize distance detection by efficient coarse-grained search. Therefore, the effects of the present invention are extremely large.

[Brief explanation of drawings]

FIG. 1A is a circuit configuration diagram of an embodiment of the present invention, FIG. 1B is a flowchart showing the operation flow of the brightness change detection section shown in FIG. 1A, and FIG. FIG. 1D is a flowchart showing the flow of operation of the parallax calculation section shown in FIG. 1A; 1E is a flow chart showing the details of one step in FIG. 1A is a flowchart showing the flow of operation of the distance calculation section shown in FIG. 1A. FIG. 1J is a flowchart showing the flow of operation of the entire circuit of the embodiment shown in FIG. 1A. A circuit configuration diagram of another embodiment of the present invention, FIG. 2B is a flowchart showing the operation flow of each second layer image generation section shown in FIG. 2A, and FIG. Flowchart showing the flow of the operation of the two-layer initial value generation section, 2nd
Figure D is a flowchart showing the operation flow of the entire circuit of the embodiment shown in Figure 2A, and Figure 3 is a principle for explaining the principle of distance detection by the correlation method using triangulation, which is the prior art. It is an explanatory diagram. 11... Parallax calculation interpolation unit 12A... First image memory 12B... Second image memory 13... Luminance change detection unit 14... Luminance change value memory 15... Parallax calculation unit 16... - Parallax distribution memory 17...Parallax interpolation calculation units 21A, 21B...Second layer image generation units 22A, 22
B... Third layer image generation section 23... Third layer parallax calculation interpolation section 24... Second layer initial value generation section 25... Second layer parallax calculation interpolation section 26... First layer Initial value generation unit 27...first layer parallax calculation interpolation unit 31A...first camera 31B...second camera 3
2A, 32B...A/D converter 33...Distance calculation unit 34...Distance distribution memory 44...Distance detection output 40.41A, 418.42A, 42B, 43°52A
, 528.52G, 53, 54, 55.56.61A,
61B, 62A, 62B, 63, 64.65, 66.6
7...Signal.

Claims (1)

[Claims] 1. Obtain a first image and a second image by photographing the object to be measured from different positions, and obtain the first image from each signal indicating the set of two images. For each divided block, search for the position of a block in the second image that has the highest correlation with the block obtained by dividing the first image, and perform triangulation based on the amount of positional shift between both blocks. In the distance detection method for obtaining the distance in the depth direction to the object using the method, the brightness change value of each pixel constituting the first image is determined, and the brightness of the pixel constituting the block obtained by dividing the first image is determined. A count value obtained by counting the number of data whose change value is equal to or greater than a preset change value threshold is used as an evaluation value for determining whether or not to calculate the positional deviation amount for the block; The positional shift amount is calculated for the blocks obtained by dividing the first image in which the evaluation value is equal to or greater than a preset evaluation value threshold, and A distance detection method in which the positional shift amount is not calculated for blocks into which an image is divided. 2. For blocks obtained by dividing the first image for which the amount of positional deviation is not calculated, the amount of positional deviation of the surrounding blocks is determined, and the amount of positional deviation is determined by interpolation from the obtained amount of positional deviation. The distance detection method according to claim 1, wherein the distance detection method is as follows. 3. Create images with a plurality of resolutions, including a first image and a second image with a coarse resolution, and a first image and a second image with a finer resolution, from each of the first original images from different positions. and dividing each of the coarse resolution first and second images into a plurality of blocks; and for each block of the coarse resolution first image, within the coarse resolution second image block; Searching for positions where blocks with a high degree of correlation exist, determining the amount of positional deviation between the blocks having the highest degree of correlation, and using the amount of positional deviation as an initial value, forming the first and second images with the fine resolution. In the distance detection method for obtaining the distance in the depth direction to the object by triangulation by determining the amount of positional deviation between the blocks with the highest correlation from the above, the amount of positional deviation at the coarse resolution and the fine resolution is determined. In this case, the brightness change value of each pixel constituting the first image is calculated, and the brightness change value of the pixel constituting the block obtained by dividing the first image is greater than or equal to a preset change value threshold. The count value obtained by counting the number of blocks is used as an evaluation value for determining whether or not to calculate the positional deviation amount for the block, and the evaluation value is greater than or equal to a preset evaluation value threshold. The positional deviation amount is calculated for blocks obtained by dividing the first image, and the positional deviation amount is not calculated for blocks obtained by dividing the first image whose evaluation value is less than the evaluation value threshold. distance detection method. 4. For blocks obtained by dividing the first image for which the amount of positional deviation is not calculated, the amount of positional deviation of the surrounding blocks is determined, and the amount of positional deviation is calculated by interpolation from the obtained amount of positional deviation. 4. The distance detecting method according to claim 3, wherein: 5. first and second image memory means (12A, 12B) for respectively storing a first image and a second image of the object to be measured taken from different positions; and the first image memory. a brightness change detection means (13) for detecting a brightness change value of each pixel constituting the first image in response to an output from the means; and the first image obtained by the brightness change detection means. a brightness change value memory means (14) for storing a brightness change value of each pixel constituting the image; and an output from the brightness change value memory means and the first and second images. In response to each output from the memory means, the first
The evaluation value, which is the count value obtained by counting the number of data in which the brightness change value of the pixels constituting the block obtained by dividing the image of For blocks obtained by dividing the first image, which is equal to or higher than the threshold value, the position of the block obtained by dividing the second image with the highest correlation is determined, and the parallax, which is the amount of positional shift between the two blocks, is determined. parallax calculation means (15) for calculating the parallax, parallax distribution memory means (16) for storing the parallax obtained by the parallax calculation means, and receiving the output from the parallax distribution memory means, Distance calculation means (33) for obtaining a distance value in the depth direction to a target object using surveying
and a distance detection device. 6. A parallax interpolation calculation means (17) is provided for determining the parallax of a block obtained by dividing the first image, the parallax of which has not been calculated by the parallax calculation means, by interpolating from the parallax of surrounding blocks. The distance detection device according to claim 5. 7. The first image obtained by photographing the object to be measured from different positions
a first set of image generation means (21A, 21B) for coarsening the resolution of the original image and the second original image, respectively;
and a second set of image generating means (22A, 22B) for further coarsening the resolution of each output from the first set of image generating means, and the second set of image generating means. a first parallax calculating means (23) for receiving the output of each image from the means, dividing each image into a plurality of blocks, and calculating the parallax for each block; a second parallax calculating means (25) for receiving the output of each image from the image generating means of the set, dividing each image into a plurality of blocks, and calculating the parallax for each block; A distance detection device including a distance calculation means (33) for receiving the output from the second parallax calculation means and obtaining a distance value in the depth direction to the target object using triangulation. 8. The first disparity calculation means divides the blocks into the plurality of blocks, calculates the disparity for each block, and for blocks for which no disparity can be obtained, calculates the disparity of blocks surrounding this block. 8. The distance detecting device according to claim 7, further comprising a parallax interpolation calculating means for determining the parallax by interpolation from . 9. The second parallax calculation means divides the blocks into the plurality of blocks, calculates the parallax for each block, and for blocks for which no parallax can be obtained, calculates the parallax of blocks surrounding this block. 8. The distance detecting device according to claim 7, further comprising a parallax interpolation calculating means for determining the parallax by interpolation from .