JP2001082927A - Method and apparatus for processing three-dimensional image and recording medium recording three-dimensional image processing program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To enable a three-dimensional image processing apparatus to determine an optimum block size without decreasing the accuracy of estimating parallax and reduce processing time. SOLUTION: An image input part 11 stores into an image storage part 12 picture element data about a right image and a left image taken by cameras 102, 101 and input to an apparatus. A brightness gradient calculating part 13 calculates a brightness gradient in an area near each picture element through a differential filter process or the like for all the picture elements. A spiral scan process part 15 performs spiral scans about each picture element and reads the brightness gradient value obtained for each picture element being scanned and performs a cumulating (adding) process. A block size calculating part 16 determines an optimum block size for each picture element in accordance with the number (or cycle) of scanned picture elements which is determined for each picture element. A corresponding point searching process part 18 performs a process for searching the corresponding points of the right and left images by use of the block size determined for each picture element.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

The present invention can be used for three-dimensional image processing such as visual control of a robot, shape recognition of an object, motion recognition of an object, obstacle recognition, position recognition in a space, and the like. Input multiple images taken,
The present invention relates to a method and an apparatus for obtaining a distance image from disparity information obtained by searching for corresponding points of pixels between images.

[0002]

2. Description of the Related Art Conventionally, as a method for obtaining a distance image (depth image) from stereo images taken from different viewpoints, a point corresponding to each pixel in a first image is determined in the other image. And a distance image is obtained from the difference (parallax) between the two pixel positions based on the principle of triangulation. As a method of searching for a corresponding point, a method of obtaining a correlation value between pixels of both images and associating the most correlated with each other is often used. As the correlation value, for example, the first
The sum of absolute differences obtained between the neighboring block including the pixel of interest in the image of interest and the neighboring block including the pixel to be searched for in the other image can be used. By searching for a pixel having the minimum value, the distance image can be obtained by obtaining the parallax information. At this time, as the block size is larger, a wider range of image information can be obtained, so that the accuracy of matching can be expected to be improved even in an area with a small change in luminance. And the accuracy drops. On the other hand, if the block size is reduced, it is effective for an area where the luminance change is large, but it becomes difficult to match an area where the luminance change is small.

As a first conventional method of three-dimensional image processing using such a block matching method, a method in which a block size is fixed (Tatenuma, Yuyama, "Optimization of left-right corresponding point search method of stereoscopic image", TV Journal of the John Society, Vol. 48, N
o.10, pp.1222-1229, 1994.). in this way,
As preprocessing, contour extraction is performed using a two-dimensional filter with a radius of three pixels, and the contour is input to perform block matching. When the luminance values in the blocks of the left image and the right image are PL (i, j) and PR (i, j), respectively, the correlation value is mainly

[0004]

(Equation 1) (Residual sum of squares, SSD) obtained by From the characteristics of the filter and the experimental results, the optimal block size is 7 × 7 (1 ≦ i ≦ 7, 1 ≦ j ≦ 7), and matching is performed with a fixed block size for all feature points.

[0005] As a second conventional method of three-dimensional image processing,
A method using multiple block sizes (Azuma, Uomori, Morimura, "Parallax estimation using edge information for generating intermediate viewpoint images of stereo images", Journal of the Institute of Image Information and Television Engineers, Vo
l.52, No.3, pp.322-330, 1998.). In this method, block matching is performed on a plurality of block sizes prepared in advance, and the parallax in each case is estimated. As the correlation value at the time of matching, an evaluation value E (X, Y) similar to the expression (1) of the first conventional method is used. Further, a reliability evaluation value J expressed by the following equation (2) is obtained, and parallax is determined by selecting from a plurality of obtained parallax estimation values based on the reliability evaluation value J.

[0006]

(Equation 2) Here, S is the number of pixels in the block, σ _n is the noise level of the image,

[0007]

[Outside 1] Is the horizontal and vertical average squared luminance gradient in each block,
The smaller the value of the expression (2), the higher the reliability of the association. Since this value J does not depend on the block size,
An estimated value that minimizes J may be selected from a plurality of estimated values obtained with different block sizes. The block sizes used for matching are 10 × 10, 20 × 20, 4
A plurality of large blocks such as 0 × 40 are used. In order to reduce the amount of computation, the types of block sizes are limited to a small number. However, in order to select the optimal block size, the parallax estimation value of Expression (1) and the disparity estimation value of Expression (2) are calculated for each block size. It is necessary to always obtain the reliability evaluation value. Also, since there are few types of block sizes to be selected, accuracy is low in selecting an optimal block size, and accuracy is increased by complicated post-processing.

[0008]

However, the above-mentioned conventional method has the following problems.

(1) In the first conventional method, the block size is fixed to the same size for all the feature point pixels to be associated. Therefore, when the block size is large, a block having a large luminance change is used. Fine luminance changes are averaged in the region, and the accuracy of parallax estimation is reduced. When the block size is small, matching of a block region with a small luminance change becomes difficult. This also gives
Since the relationship between the block size and the accuracy of parallax estimation differs depending on the image, it is difficult to determine an optimal block size.

(2) In the second conventional method, an optimal block size is selected for each pixel in order to solve the problem (1). At this time, the parallax of the formula (1) is used for all the pixels. The estimated value and the reliability evaluation value represented by the equation (2) are always obtained for each of a plurality of block sizes, which increases the amount of calculation.

(3) In the second conventional method, in consideration of the problem (2), the types of block sizes to be processed are thinned out in order to reduce the amount of calculation. Is selected, the number of block size candidates is reduced.

(4) In the second conventional method, when an optimum block size is obtained for each pixel, a process of obtaining a parallax estimation value and a process of obtaining a reliability evaluation are sequentially performed over all pixels. When the image size is large and the block size is large, the processing time becomes longer.

An object of the present invention is to reduce the accuracy of parallax estimation by fixing all block sizes to the same size as in the conventional method described above, and to determine an optimal block size in that case. It is an object of the present invention to provide a three-dimensional image processing method and apparatus which solve the difficult problem and the problem that the amount of calculation (processing time) increases when obtaining an optimal block size for each pixel.

[0014]

According to the three-dimensional image processing apparatus of the present invention, a plurality of images taken from different viewpoints are input, and each pixel of the first image is converted into an image other than the first image. The most corresponding pixel position is searched, the difference between those pixel positions is extracted as disparity information, and in a device for obtaining a distance image from the disparity information, a corresponding point search for the pixel is performed based on an evaluation value obtained from a luminance distribution. There is means for determining a block size to be used, and a corresponding point search is performed between images by block matching using the block size separately obtained for each pixel.

[0015] Since stereo matching can be performed by obtaining an optimum block size for each pixel, the block size is small for pixels having a nearby block region having a large change in luminance, and the block size is small for pixels having a nearby block region having a small change in luminance. By increasing the size, the accuracy of parallax estimation can be improved. In addition, since the optimum block size for each pixel is first obtained, and then the parallax estimation processing by stereo matching is performed using the determined block size, a plurality of times for one pixel in a case of a plurality of block sizes are performed. Does not need to be performed, and the processing time can be shortened.

In an embodiment of the present invention, the means for determining the block size includes means for calculating a luminance gradient for each pixel, and for each pixel, a pixel in the form of a spiral or concentric circle representing the cumulative value of the luminance gradient in the vicinity of the pixel. Means for scanning, means for normalizing the cumulative value based on the number of pixels scanned or the number of turns of spiral scanning, and means for obtaining a block size by threshold determination from the normalized value, Using the block size obtained separately for each pixel, a corresponding point search is performed between the images by block matching. Since the optimum block size is obtained while performing the spiral scanning on the cumulative value of the luminance gradient, it can be obtained by one spiral scanning process, and the processing time can be shortened. In addition, since the optimal block size is obtained only by the process of accumulating the luminance gradient of the neighboring pixels, there is no need to perform complicated calculations such as a plurality of parallax estimating processes and calculating a reliability evaluation value for a plurality of block sizes. Time can be shortened. Also,
Since the parallax estimation processing with a large amount of calculation is not performed when obtaining the block size, the optimal block size can be determined without a large amount of calculation without thinning out the types of block sizes to be processed.

In a still further embodiment of the present invention, the three-dimensional image processing device includes: a means for calculating a luminance gradient for each pixel; and a spiral or concentric cumulative value of the luminance gradient in the vicinity of the pixel for each pixel. Means for scanning a pixel, means for normalizing the accumulated value based on the number of scanned pixels or the number of times of spiral scanning, and means for calculating a block size by threshold determination from the normalized value. Having a processing block for each pixel or for a plurality of pixels of the first image;
A transfer path between the processing blocks capable of simultaneously transferring neighboring pixel values between all the processing blocks when calculating the luminance gradient and when calculating the cumulative value of the luminance gradient by spiral scanning is provided.

According to this embodiment, the processing block is the first
It has a transfer path that can transfer neighboring pixel values all at once for every pixel of the image or a plurality of pixels, so that it is not necessary to sequentially process all pixels. Processing can be executed simultaneously for all processing blocks, and processing time can be shortened.

[0019]

DESCRIPTION OF THE PREFERRED EMBODIMENTS First Embodiment FIG. 1 is a configuration diagram of a three-dimensional image processing apparatus according to the first embodiment of the present invention. The three-dimensional image processing apparatus of the present embodiment receives a left image and a right image, obtains an optimal block size for each pixel of one image, and performs block matching based on the obtained block size. A distance image (depth image) is output by associating pixels with each other and obtaining parallax information for each pixel. Here, two images (a left image and a right image) are described as inputs. However, in the case of a plurality of images taken from three or more viewpoints, any two of the images It is clear that can perform this process as well. Hereinafter, the operation of the present embodiment will be described.

First, a method of searching for corresponding points on the left and right images in the present apparatus will be described. The present apparatus uses a block matching method in which a correlation value between blocks near a pixel of interest on each image is obtained, and pixels having the highest degree of correlation are associated with each other. FIG. 2 shows an example of the association. When searching for a pixel corresponding to the pixel PR of the right image from the left image, the neighboring block B of each pixel is located between the pixel of interest PL of the left image.
For example, a correlation value E shown in the following equation (3) is obtained between R and BL.

[0021]

(Equation 3)This is the sum of absolute differences, but the correlation
Various values such as the residual sum of squares shown in equation (1)
It can be used arbitrarily. The block size is 3
The example of × 3 is shown, but it can be easily changed to any size and shape.
Can be extended. When searching, block BR of the right image
Is the search range (BL ₀ To BL
_max On the epipolar line (or
It is sufficient to search only the range (in the vicinity of the line). Where ma
x is the maximum value of the parallax between the left and right images, and
Narrows the range because there is no point corresponding to PR
As a result, the processing amount can be reduced, and erroneous responses can be reduced.

Next, a method for obtaining an optimum block size for each pixel, which is performed before the corresponding point search processing by the block matching method, will be described. The larger the block size, the wider the range of image information can be obtained, and the accuracy of matching can be expected to improve even in areas with little change in luminance. Falls. On the other hand, if the block size is reduced, it is effective for an area where the luminance change is large, but it becomes difficult to match an area where the luminance change is small. Therefore, first, the luminance gradient of the neighboring area is obtained for each pixel, and based on the obtained luminance gradient (luminance change), the block size is increased where the luminance gradient is gentle,
Where the luminance gradient is steep, it is possible to improve the accuracy of stereo matching by reducing the block size. For example, in the case of an image as shown in FIG. 3, a large block such as BA is preferably used for the pixel PA because the luminance gradient in the neighboring area is small, but the luminance gradient for the neighboring area is large in the pixel PC, and the pixel PA has a luminance gradient like BC. It is desirable to use a small block.

In this apparatus, the luminance gradient in all pixels is obtained in advance. For the specific calculation method, for example,
A differential filter such as a filter can be used, but if a value indicating the degree of luminance change is obtained,
A method using an arbitrary filter or the like can be easily executed. Further, as the luminance value, not only a monochrome gray-scale image but also each color component value of a color image may be used. In addition, by using a differential filter in a direction parallel to the epipolar line as a filter, it is possible to avoid a problem that an erroneous block size is obtained when a linear image parallel to the epipolar line is included in the neighboring area.

A specific method for obtaining the block size will be described below. As for each pixel on the right image, as shown in FIG. 4, neighboring pixels are helically scanned around the pixel of interest. While scanning, the luminance gradient value (meaning the absolute value of the luminance gradient; the same applies hereinafter) of the pixel being scanned is accumulated,
If the threshold value is exceeded, scanning is stopped at that pixel (or the number of rotations), and the number of scanned pixels (or number of rotations) up to that point is stored. The value of the number of scanning pixels (or number of rotations) obtained for each pixel in this manner increases when the luminance change in the neighboring area is small, and decreases when the luminance change in the neighboring area is large. The block size can be obtained as a value proportional to the value.

A specific operation for executing the above processing in the first embodiment shown in FIG. 1 will be described.

The image input unit 11 includes cameras 10 ₁ and 10 ₂
Then, the pixel data of the left image and the right image input to the apparatus are stored in the image storage unit 12.

The luminance gradient calculating section 13 obtains a luminance gradient of a neighboring region in each pixel for all pixels by the above-described differential filter processing or the like.

The helical scanning processing section 15 performs helical scanning on each pixel as shown in FIG. 4, centering on the pixel, reads out the obtained luminance gradient value from the pixel being scanned, and accumulates (adds) it. Perform processing. If the accumulated value exceeds a predetermined threshold value during scanning, scanning is stopped at that pixel (or its circumference), and the number of scanned pixels (or the number of rotations) up to that point is stored. Also, in this spiral scan,
A method of reducing the number of times of the threshold determination processing can be easily realized by performing the threshold determination after completing one round of scanning (one concentric circle) instead of performing the threshold determination processing for each pixel. These processes are executed for all pixels.

The block size calculation section 16 calculates an optimum block size for each pixel based on the number of scanning pixels (or the number of rotations) obtained for each pixel. As a method of obtaining the number of turns, there is a method of using the obtained number of turns and setting a block size of the same size corresponding to the number of turns. For example, if helical scanning has been completed three times around the pixel, 7 × 7
Block of size. That is, if n rounds, 1
The side has a size of (n + 1) × 2. In addition to using the value of the number of revolutions as it is, it is also easy to perform a calculation such as multiplying the obtained number of scanning pixels (or the number of revolutions) by a coefficient, and then obtain the block size based on the value. Can be considered.

If the block size is obtained immediately after the number of scanning pixels (or the number of revolutions) obtained by the spiral scanning processing unit 15 is obtained, the storage unit for the number of scanning pixels is not necessary and can be omitted.

The corresponding point search processing unit 18 performs a corresponding point search process on the left and right images using the obtained block size for each pixel. As a specific method, any method may be used as long as it performs a process of associating pixels between left and right images by block matching as described at the beginning of the present embodiment. A distance image (depth image) is output from the obtained parallax information for each pixel.

Second Embodiment FIG. 5 is a block diagram of a three-dimensional image processing apparatus according to a second embodiment of the present invention.

As in the first embodiment, the three-dimensional processing apparatus of the present embodiment receives a left image and a right image, obtains an optimal block size for each pixel of one of the images, and obtains the obtained block size. By block matching based on block size,
The distance image (depth image) is output by associating the pixels between the left and right images and obtaining parallax information for each pixel. Hereinafter, the operation of the present embodiment will be described.

This apparatus has a processing block (PE) described later.
5 are arranged in a NX × NY two-dimensional array as shown in FIG. The processing blocks 30 are connected by a transfer path 40. The connection form of the PE array unit 20 is configured by, for example, four adjacent mesh connections. The number of processing blocks 30 is the same as the number of pixels of an image, and one processing block 30
The information of one pixel of the left image and the information of one pixel of the right image are stored, and a luminance gradient calculation process, a spiral scan process, a block size calculation process, a corresponding point search process, and the like described below are performed. When the input image size is IX × IY, IX = NX, I
Y = NY, and a PE that performs a process related to a pixel at coordinates (X, Y) is PE (X, Y).

As shown in FIG. 5B, one processing block 30 includes a memory for storing the pixel data PL of the left image, a memory for storing the pixel data PR of the right image, and It has a luminance gradient value G of pixels, a cumulative value SG of luminance gradient values, a block size BS, a memory for storing disparity information MD, and, if necessary, a work memory required for calculation and transfer. Also, a luminance gradient calculation unit 33, a luminance gradient value accumulation processing unit 34, an accumulation value threshold determination processing unit 35, a block size calculation unit 36, a corresponding point search processing unit 37, a control command reception unit 31, and a data reception unit 3
2. It has a data transmission unit 38. In the operation described below, various calculations and data update processing using data stored in the data storage unit 39 and data received from the processing block 30 adjacent to the data storage unit 39 are performed.

The image input initializing section 21 inputs the pixel data of the left image and the right image input to the apparatus to all the processing blocks 30. At this time, the pixel data at the coordinates (X, Y) on the image is stored in the data storage unit PL (X, Y) of the PE (X, Y).
Y) and PR (X, Y).

The brightness gradient calculation control unit 22 sends a control signal for obtaining a brightness gradient of a neighboring area to the pixel value PR (X, Y) to the PE array 20 for all the processing blocks 30.
Send to A specific calculation method of the luminance gradient will be described below. FIG. 6 shows an example of a differential filter near 3 × 3.
The filter coefficients (weights) are shown in FIG.
Various types such as (b) and (c) can be easily applied. Also, an arbitrary filter having a larger size such as 5 × 5 can be applied. It is also possible to perform processing by combining a plurality of filters. Generally, the size is m × m, and each coefficient of the filter is W (i, j).
Then, the luminance gradient value G for the pixel PR (X, Y)
(X, Y) is calculated in PE (X, Y) based on the following equation.

[0038]

(Equation 4) Here,-(m-1) / 2≤i≤ (m-1) / 2,-
(M-1) / 2≤j≤ (m-1) / 2. Data transfer from adjacent processing blocks and arithmetic processing within the processing blocks are performed simultaneously for all processing blocks.

The helical scanning processing section 23 performs the helical scanning centering on the pixel as shown in FIG. 4 for all the processing blocks, as in the first embodiment, while accumulating (adding) the luminance gradient values. A) Send a control signal for performing the processing and the threshold determination processing to the PE array processing unit 20. In the scanning, while sequentially changing the target processing block corresponding to the neighboring pixel, the luminance gradient value G stored in the processing block is read, and the PE (X, Y) corresponding to the pixel of interest PR (X, Y) is read.
Y), and in PE (X, Y), this value is sequentially transferred to S
G is cumulatively added. The transfer and accumulation processing are performed while changing the processing block to be scanned in a predetermined order. When the accumulation value SG exceeds a predetermined threshold, the number of scanning pixels (or the number of rotations) up to that time is set to PE (X, Y). ). For all the processing blocks, data transfer from nearby processing blocks and arithmetic processing in the processing blocks are performed simultaneously.

The block size calculation control unit 24 sends a control signal for calculating an optimum block size for each pixel to the PE array unit 20 based on the number of scanning pixels (or the number of rounds) obtained for each pixel. . As a specific method of obtaining, as in the first embodiment, there is a method of using the obtained number of rotations to set the same block size corresponding to the number of rotations. For example, if helical scanning has been completed three times around the pixel, a 7 × 7 size block is set. That is, if it is n rounds, one side has a size of (n + 1) × 2. There is also a method of obtaining a block size by performing normalization such as multiplying the obtained number of scanning pixels (or the number of rotations) by a coefficient. Block size calculation processing is performed for all processing blocks at once.

The corresponding point search processing control unit 25 sends a control signal for performing a corresponding point search process between the left and right images to the PE array processing unit 20 using the obtained block size for each pixel. For all pixels on the right image, search for corresponding pixels on the left image. As a specific method, any method may be used as long as the process of associating pixels between left and right images by block matching as described at the beginning of the first embodiment. A distance image (depth image) is output from the obtained parallax information for each pixel.

The specific processing in each processing block will be described. The data receiving unit 32 stores data transmitted from another processing block in the data storage unit 39.
The data transmitting unit 38 transmits data that needs to be transferred in the processing block to one adjacent processing block.

The luminance gradient calculating section 22 reads out the pixel values in the neighboring area and obtains the luminance gradient in PE (X, Y). A specific calculation method of the luminance gradient will be described below. For example, if a 3 × 3 differential filter shown in FIG. 6 is used, the pixel value PR (X + i, Y) of each PE in the nearby area is used.
+ J) (-(m-1) / 2≤i≤ (m-1) / 2,-
(M-1) / 2≤j≤ (m-1) / 2) are sequentially read out by transfer, and the values obtained by multiplying each by the filter coefficient W (i, j) are added, and the luminance gradient value G is calculated. It is obtained and stored in the data storage unit 39. Since all the processing blocks can perform the same operation at the same time, it is possible to obtain the luminance gradient all at once.

The luminance gradient value accumulating section 34 sequentially reads out the luminance gradient values G for the neighboring PEs along the spiral with the corresponding pixel of the processing block as the center, and reads out the PEs.
(X, Y) and cumulatively add to SG in PE (X, Y). The order of spiral scanning can be determined in advance,
The external helical scanning control unit 23 may indicate the PE address to be scanned, or each processing block may remember the scanning order. If all the processing blocks always perform the same scanning order, the accumulation processing can be performed simultaneously for all the processing blocks.

In the cumulative value threshold determination processing unit 35, when the cumulative value SG exceeds a predetermined threshold value in the transfer and the cumulative processing executed for the processing block, the number of scanning pixels (or the number of rotations) up to that time is obtained. Is stored in the data storage unit 39. Since processing is independent for each processing block, all processing blocks can be processed simultaneously. After the SG exceeds the threshold, the processing may be continued with another processing block, or the processing may be terminated only for that processing block.

The block size calculation section 36 calculates the block size BS based on the number of scanning pixels (or the number of rotations) obtained for each pixel, and stores it in the data storage section 39. The specific calculation method is as follows.
As shown in the above, there is a method of using the obtained number of rounds to set the same block size corresponding to the number of rounds. Similar to the threshold determination processing unit 35, since the processing is independent for each processing block, all processing blocks can be processed simultaneously.

The corresponding point search processing section 37 performs a corresponding point search process between the left and right images by using the obtained block size for each pixel. As a specific method, any method may be used as long as the process of associating pixels between left and right images by block matching as described at the beginning of the first embodiment. That is, a method of calculating the correlation value between blocks between the left and right images for the neighboring area block of the pixel corresponding to the PE (X, Y) of interest and associating the pixels having the highest correlation with each other. For example, when calculating the correlation value E of the equation (3) with respect to FIG. 2, first, the absolute value of the difference between the pixel values PL and PR is calculated and stored in all the processing blocks, and then, in each processing block, The PE (X,
Y), the sum is obtained, and the correlation value E is obtained. The same processing and transfer in the same direction are performed for all processing blocks, so that all processing blocks can be executed simultaneously. From the positions of the corresponding pixels in the left and right images found by the corresponding point search process,
The parallax value MD can be obtained using the difference between the coordinate values. The obtained parallax value MD can be output as it is or as a distance image (depth image) obtained by performing normalization such as changing a coefficient.

In the above-described second embodiment, by using an apparatus capable of performing an external filtering process, if an image after obtaining a luminance gradient in advance is input as an input image instead of an image from a camera, PE It is not necessary to perform the brightness gradient calculation processing in the array processing unit 20. That is, the luminance gradient calculation control unit 22 and the luminance gradient calculation unit 33 in the processing block can be omitted. Also, for the corresponding point search processing, PE
Corresponding point search is possible without executing in the array processing unit 20,
It is also possible to input and use the obtained block size to an external corresponding point searching device capable of processing by changing the block size for each pixel.

Third Embodiment FIG. 7 is a block diagram of a three-dimensional image processing apparatus according to a third embodiment of the present invention.

In this embodiment, the PE array processing unit 20 shown in the second embodiment is constituted by a content addressable memory (CAM) 50 as shown in FIG. The CAM 50 has the same number of words V = NX × NY as the number of pixels of the image, and one word stores information relating to one pixel of the left image and one pixel of the right image.

The CAM 50 used here is a two-dimensional PE array type CAM (T. Ikenaga,
T. Ogura, “A Fully-Parallel 1Mb CAM LSI for Real-
TimePixel-Parallel Image Processing ”, ISSCC, 199
9., Ikenaga and Kokura, "Mass-Parallel Two-Dimensional Cellular Automata: CAM ² ", IEICE Technical Report, CPSY-ICDFTS, 1996.) The CAM 50 allocates one pixel to one word, is logically arranged and connected in a two-dimensional PE array, and can execute various parallel processes by external control. FIG. 9 shows a specific example of parallel processing. FIG.
(A) shows an example of processing in which, when reference data is input, a word having a value equal to only the bit portion of the search mask is searched for all words at once, and the answer is stored in a hit flag. FIG. 9B shows an example in which the input data masked by the write mask is written in parallel only to the word whose hit flag is 1. By combining such a parallel search and a parallel writing function, the CAM 50 can execute arbitrary arithmetic and logical operations such as addition processing in parallel with all words.
Since the operation is performed on the memory, there is no need to have an operation unit for the operation. Further, simultaneous transfer of all words between upper and lower words is possible via a hit flag. Here, the above-described two-dimensional PE array type CAM 50 is described as an example. However, if a normal one-dimensional array type CAM has a function of transferring data between upper and lower words, FIG.
By allocating pixels and words as described above, it can be easily realized by the same method as described above. Hereinafter, a specific operation of the embodiment using the CAM having such characteristics will be described.

The structure of one word of the CAM 50 stores the same information as in the second embodiment as shown in FIG. Regarding the operations described below, various operations using data stored in the CAM 50 and data received from adjacent processing blocks and data update processing can be executed on the memory of the CAM 50 by the parallel operation function of the CAM 50 and the like. Since the entire processing flow is almost the same as in the first embodiment, a specific processing in the CAM 50 will be described below.

The CAM 50 can transfer data between adjacent words, and can transfer word data to words corresponding to neighboring pixels. In addition, CAM50
The search processing, the writing processing, and the addition processing can be performed in parallel in any word among arbitrary data in a word, and the processing time can be reduced by using these parallel processing functions.

The image input initializing section 41, the luminance gradient calculation control section 42, the helical scanning processing control section 43, the block calculation control section 44, and the corresponding point search processing control section 45 correspond to the corresponding sections 21 to 25 in FIG. Performs the same operation as.

First, the pixel value of the neighboring area is transferred,
A luminance gradient value G in PE (X, Y) is obtained. The specific calculation method of the luminance gradient value is the same as that of the second embodiment.
For example, a differential filter near 3 × 3 in FIG. 6 can be used. The obtained luminance gradient value G is stored in the CAM 50.
In the CAM 50, all words (PE) can perform the same arithmetic processing at the same time, so that a luminance gradient can be obtained all at once.

Next, an accumulating process is performed. The luminance gradient value G is sequentially read out from the neighboring PEs along the spiral with the corresponding pixel at the center as the center, and transferred to the PE (X, Y). Cumulative addition to SG is performed within (X, Y). These processes can be simultaneously performed on all words of the CAM 50.

Further, the threshold value judgment processing of the accumulated value is also performed by the CAM.
All 50 words can be simultaneously executed by parallel comparison / update processing of all words.

In the process of calculating the block size in the block size calculation control unit 44, the block size BS is obtained based on the number of scanning pixels (or the number of rotations) obtained for each pixel and stored in the CAM 50. This can also be processed simultaneously for all words.

The corresponding point search processing section 45 performs a corresponding point search process between the left and right images by using the obtained block size of each pixel. As a specific method, any method may be used as long as the process of associating pixels between left and right images by block matching as described at the beginning of the first embodiment. In other words, this method is a method of obtaining a correlation value between blocks between left and right images for a neighboring area block of a pixel corresponding to a focused PE (X, Y), and associating pixels having the highest degree of correlation. Specifically, in a CAM having a structure equivalent to a two-dimensional PE array, an example similar to the first embodiment can be realized.

The processes (luminance gradient calculation process, spiral scanning process, block size calculation process, corresponding point search process) in the CAM 50 are all executed simultaneously for all words, that is, for all pixels. In addition, since all of these processes are performed on the memory of the CAM, there is no need to have a dedicated arithmetic unit for that, and the amount of hardware can be reduced.

Fourth Embodiment FIG. 10 is a block diagram of a three-dimensional image processing apparatus according to a fourth embodiment of the present invention.

In the first embodiment, the luminance gradient values of all the pixels are obtained in advance, and the accumulation processing is performed using the values. In this apparatus, however, the following accumulation processing and threshold determination are directly performed using the input image. Processing is performed.

The image input unit 51, the image storage unit 52, the block size calculation unit 56, the block size storage unit 57, and the corresponding point search processing unit 58 are the same as those of the first embodiment. Block size calculator 1
6, block size storage unit 17, corresponding point search processing unit 18
Performs the same operation as. No luminance gradient calculation unit is required, and the operation of the spiral scanning processing unit 55 is slightly different from that of the first embodiment. Hereinafter, the operation of the spiral scan processing unit 55 of the present apparatus will be described.

The helical scanning processing unit 55 calculates the absolute difference between the pixel value of interest PE (X, Y) and the target pixel value PR (X + i, Y + j) as a luminance gradient value, and helically scans this value. While performing the cumulative addition process. As for the threshold determination process, as in the first embodiment, scanning is stopped when the threshold value is exceeded, and the number of scanning pixels (or the number of rotations) up to that point is stored by comparison with a predetermined threshold value.
Since it is not necessary to obtain a luminance gradient value for each pixel in advance, the processing time can be reduced, and a memory for storing the luminance gradient value is not required. In this method, that is, a method in which the absolute value of the difference between the input pixel values is used instead of the luminance gradient value, when the luminance change is small in the vicinity of the target pixel, the value of the number of scanning pixels (or the number of rotations) increases, and When the luminance change of the area is large, the value becomes small, so that the block size can be obtained as a value proportional to this value.

In the above fourth embodiment, the second,
The processing method using the PE array as in the third embodiment can be easily realized.

Fifth Embodiment FIG. 11 is a block diagram of a three-dimensional image processing apparatus according to a fifth embodiment of the present invention.

In the second and third embodiments, the number of processing blocks (PE) is the same as the number of pixels of an image. However, an embodiment in which one processing block PE processes a plurality of neighboring pixels can be realized. It is. In this case, in the second embodiment,
In FIG. 5B illustrating the processing block PE, the number of words in the data storage unit 39 ′ may be set to have the number of words corresponding to the number of pixels handled by the processing block PE. In the third embodiment, for example, when four pixels are handled by one word (PE), the number of words V (= NX × N) is the same as the number of pixels.
Y) may be reduced to 1/4 (= V / 4 word), and the number of bits in one word may be extended by four times. At this time, the data of four pixels stored in one word may be sequentially processed. Good. Although the transfer of the neighboring pixel values is different, a part of the transfer processing from the adjacent processing block only relates to reading from the data storage unit in the processing block PE, and the overall configuration is the same as that of FIG. , Is easily feasible. Processing is performed sequentially for each pixel in the processing block PE (word), but all the processing blocks PE (word) can execute the processing at the same time. Although the processing time is longer than in the second and third embodiments, the amount of hardware can be reduced.

Sixth Embodiment In the embodiments described above, the helical scanning processing unit scans the neighboring pixels of the pixel of interest PE (X, Y) by using PE (X,
Y) While changing the scanning target PE in a spiral around the center,
Although the transfer processing is performed, the present embodiment employs a method of expanding the area (on a concentric circle) for each rotation, instead of expanding the range of the neighboring area for each pixel.

That is, as shown in FIG.
After performing a process of accumulating surrounding luminance gradient values for the × 3 region, a threshold determination process is performed. Next, the target area is 5 × 5
Then, the surrounding luminance gradient value is subjected to accumulation processing following the accumulation up to that point, and then threshold value determination processing is performed. In this manner, the scanning process is performed while enlarging the scanning target pixel region. As a result, the number of times of the threshold determination process can be reduced as compared with the above-described embodiments.

Seventh Embodiment In the case where the PE array as in the second and third embodiments is used, when the scanning processing of the sixth embodiment is executed, By performing the data transfer procedure from the neighboring processing block as shown by (1), a method that can reduce the number of processes of the accumulation process can be considered.

FIG. 13 shows that, in the accumulation processing at the time of 3 × 3, instead of scanning while rotating, upper and lower transfer and accumulation processing are performed first (FIGS. 13 (1) and (2)), and then Left and right transfer and accumulation processing using values (FIGS. 13 (3) and (4))
Is performed, the transfer of the center pixel of interest to the processing block PE and the accumulation processing can be performed for all the pixels on one round. At this time, the same processing is performed for all processing blocks PE in the vertical transfer and the horizontal transfer at the same time,
Neighboring scanning processing for all processing blocks PE (pixels) can be executed simultaneously, and the number of transfers and the number of cumulative processing at that time can be reduced. The procedure for the accumulation process for an arbitrary size of 5 × 5 or more can be easily realized.

The processing of each unit of the three-dimensional image processing apparatus of each embodiment described above is recorded as a three-dimensional image processing program on a recording medium such as a floppy disk, CD-ROM, or magneto-optical disk. It can also be executed on a computer such as a personal computer.

[0073]

As described above, the present invention has the following effects.

1) According to the third aspect of the invention, stereo matching can be performed by obtaining an optimum block size for each pixel.
The accuracy of parallax estimation can be improved by reducing the block size for pixels having a nearby block region with a large change in luminance and increasing the block size for pixels having a nearby block region with a small change in luminance. Also,
Since the optimal block size for each pixel is first obtained, and then the disparity estimation process is performed by stereo matching using the determined block size, multiple parallaxes for one pixel in a case of multiple block sizes are performed. There is no need to perform an estimation process, and the processing time can be shortened.

2) According to the first and fourth aspects of the present invention, in addition to the effect described in 1), since the optimum block size is determined while performing the spiral scanning of the cumulative value of the luminance gradient, it is determined by one spiral scanning process. And the processing time can be shortened. In addition, since the optimal block size is obtained only by the process of accumulating the luminance gradient of the neighboring pixels, there is no need to perform complicated calculations such as a plurality of parallax estimating processes and calculating a reliability evaluation value for a plurality of block sizes. Time can be shortened. Further, when calculating the block size, the parallax estimation processing with a large amount of calculation is not performed, so that the optimal block size can be determined without a large amount of calculation without thinning out the types of block sizes to be processed. .

3) The invention according to claim 5 is provided with a transfer path having a processing block for each pixel or a plurality of pixels of the first image, and capable of simultaneously transferring neighboring pixel values among all the processing blocks. Therefore, it is not necessary to perform the processing sequentially for all the pixels, the processing can be executed simultaneously for all the processing blocks, and the processing time can be shortened.

[Brief description of the drawings]

FIG. 1 is a configuration diagram of a three-dimensional image processing apparatus according to a first embodiment of the present invention.

FIG. 2 is a diagram illustrating a corresponding point search method according to the first embodiment of the present invention.

FIG. 3 is an explanatory diagram for changing a block size for each pixel according to the first embodiment of the present invention.

FIG. 4 is a diagram for explaining spiral scanning in the first embodiment of the present invention.

FIG. 5 is a configuration diagram of a three-dimensional image processing apparatus according to a second embodiment of the present invention.

FIG. 6 is a diagram illustrating a filter for calculating a luminance gradient according to a second embodiment of the present invention.

FIG. 7 is a configuration diagram of a three-dimensional image processing apparatus according to a third embodiment of the present invention.

FIG. 8 shows an image and a CAM according to a third embodiment of the present invention.
It is a figure explaining the relation of a word.

FIG. 9 is a diagram illustrating an example of parallel processing of a CAM according to a third embodiment of the present invention.

FIG. 10 is a configuration diagram of a three-dimensional image processing apparatus according to a fourth embodiment of the present invention.

FIG. 11 is a configuration diagram of a three-dimensional image processing apparatus according to a fifth embodiment of the present invention.

FIG. 12 is a diagram illustrating a sixth embodiment of the present invention.

FIG. 13 is a diagram illustrating a seventh embodiment of the present invention.

[Explanation of symbols]

10 ₁ , 10 ₂ Camera 11, 51 Image input unit 12, 52 Image storage unit 13 Brightness gradient calculation unit 14 Brightness gradient image storage unit 15, 55 Spiral scan processing unit 16, 56 Block size calculation unit 17, 57 Block size storage unit 18, 58 corresponding point search processing unit 20 PE processing array unit 21, 41 image input initialization unit 22, 42 luminance gradient calculation control unit 23, 43 spiral scanning processing control unit 24, 44 block size calculation control unit 25, 45 corresponding point Search processing control unit 30 Processing block 31 Control command receiving unit 32 Data receiving unit 33 Luminance gradient calculation unit 34 Luminance gradient value accumulation processing unit 35 Cumulative value threshold determination processing unit 36 Block size calculation unit 37 Corresponding point search processing unit 38 Data transmission unit 39, 39 'Data storage unit 40 Transfer path 50 Content addressable memory (CAM)

 ──────────────────────────────────────────────────続 き Continued on the front page F term (reference) 2F065 AA04 AA53 DD06 DD07 FF05 JJ03 JJ05 JJ26 MM00 QQ08 QQ13 QQ14 QQ24 QQ29 QQ31 QQ36 QQ38 QQ42 UU06 2F112 AC06 BA05 CA08 FA03 FA09 FA21 FA27 FA29 FA35 FA04 EA07 FA41 FA41 5B057 CH11 DA07 DB03 DC22 DC32

Claims (12)

[Claims] 1. A plurality of images taken from different viewpoints are input, and for each pixel of a first image, the most corresponding pixel position in an image excluding the first image is searched for, and the pixel positions are searched for. In the three-dimensional image processing method for extracting a difference between the two as parallax information and obtaining a distance image from the parallax information, a step of obtaining a luminance gradient for each pixel; and for each pixel, a cumulative value of the luminance gradient near the pixel is calculated by: Scanning the pixels in a spiral or concentric manner; obtaining the accumulated value; normalizing the accumulated value based on the number of pixels scanned or the number of turns of the spiral scan; and Determining a three-dimensional image processing method. 2. A method for calculating a sum of absolute differences between a pixel value of a pixel of interest and a pixel value of a target pixel being scanned, as the luminance gradient,
The method of claim 1. 3. A plurality of images taken from different viewpoints are input, and for each pixel of the first image, the most corresponding pixel position in the image excluding the first image is searched for, and the pixel positions are searched. A three-dimensional image processing apparatus that extracts a difference between the two as parallax information and obtains a distance image from the parallax information, wherein a means for determining a block size used for searching for a corresponding point of the pixel based on an evaluation value obtained from a luminance distribution; A three-dimensional image processing apparatus, comprising: means for performing a corresponding point search by block matching between images using a block size obtained separately for each image. 4. The means for determining the block size comprises:
Means for calculating a luminance gradient for each pixel, means for calculating the cumulative value of the luminance gradient in the vicinity of the pixel for each pixel by scanning pixels spirally or concentrically, and means for calculating the cumulative number of pixels or 4. The three-dimensional image processing apparatus according to claim 3, further comprising: means for normalizing based on the number of times of helical scanning, and means for obtaining a block size by threshold determination from the normalized value. 5. A means for calculating a luminance gradient for each pixel, a means for calculating a cumulative value of the luminance gradient in the vicinity of the pixel by scanning pixels in a spiral or concentric manner for each pixel, and calculating the cumulative value: A processing block including means for normalizing based on the number of scanned pixels or the number of revolutions of spiral scanning, and means for obtaining a block size by threshold determination from the normalized value, for each pixel of the first image or a plurality of processing blocks. Having for each pixel, when determining the luminance gradient, and when calculating the cumulative value of the luminance gradient by scanning spirally or concentrically, it is possible to simultaneously transfer neighboring pixel values between all processing blocks, The three-dimensional image processing apparatus according to claim 4, further comprising a transfer path between processing blocks. 6. The apparatus according to claim 5, comprising an associative memory as said processing block and transfer path. 7. A plurality of images taken from different viewpoints are inputted, and for each pixel of the first image, the most corresponding pixel position in the image excluding the first image is searched for, and the pixel positions are searched. A three-dimensional image processing apparatus that extracts a difference between the two images as parallax information and obtains a distance image from the parallax information, the unit that inputs the plurality of images, and the unit that obtains a luminance gradient for each of the plurality of pixels. Means for determining, for each pixel, the cumulative value of the luminance gradient in the vicinity of the pixel by scanning the pixels in a spiral or concentric manner; and determining the cumulative value based on the number of scanned pixels or the number of times of spiral scanning. Means for normalizing, means for obtaining a block size by threshold determination from the normalized value, and using the obtained block size for each pixel, perform a corresponding point search process between the first image and another image, Extract parallax information And a three-dimensional image processing apparatus having means for outputting a distance image. 8. A distance image is output by inputting a first image and another image, associating pixels between the first image and the other image, and obtaining parallax information for each pixel. A three-dimensional image processing apparatus, comprising: a memory that stores pixel data of a first image, a memory that stores pixel data of another image, a memory that stores a luminance gradient value of the pixel, and a memory that stores a cumulative value of the luminance gradient value. A data storage unit including a memory for storing, a memory for storing a block size, and a memory for storing disparity information; a data receiving unit for receiving data from an adjacent processing block and storing the data in the data storage unit; and a luminance gradient of each pixel. And a luminance gradient calculation unit that stores the luminance gradient value in the data storage unit, performs a luminance gradient value accumulation process, and stores a luminance gradient value accumulation processing unit that stores the luminance gradient value in the data storage unit. A cumulative value threshold determining unit for determining, a block size calculating unit that calculates an optimal block size for each pixel based on the number of scanning pixels or the number of rotations obtained for each pixel, and stores the block size in the data storage unit; Find the required block size,
A corresponding point search processing unit that performs a corresponding point search process between the first image and another image, a data transmission unit that transfers data that needs to be transferred within a processing block to one adjacent processing block, Receiving, in response to the command, the data receiving unit,
Processing including an instruction receiving unit that starts one of the luminance gradient calculation unit, the luminance gradient accumulation processing unit, the accumulated value threshold determination processing unit, the block size calculation unit, the corresponding point search processing unit, and the data transmission unit A PE array processing unit in which blocks are arranged corresponding to the respective pixels of the image data, and transmitting the input pixel data of the first image and the other image to the data receiving unit of the corresponding processing block; An image input initializing unit that transmits a command to instruct storage to a command receiving unit of a corresponding processing block; and a brightness gradient calculation control that issues a command to start the brightness gradient calculating unit to the command receiving units of all processing blocks. While scanning pixels spirally or concentrically with the unit, the luminance gradient value accumulating unit and the accumulated value threshold judging unit are activated for the command receiving units of all processing blocks. A spiral scan processing control unit that issues an instruction to execute the command, a block size calculation control unit that issues an instruction to activate the block size calculation unit to an instruction reception unit of all processing blocks, and a command reception unit of all processing blocks. A three-dimensional image processing apparatus having a corresponding point search processing control unit for issuing an instruction to start the corresponding point search processing unit. 9. A distance image is output by inputting a first image and another image, associating pixels between the first image and the other image, and obtaining parallax information for each pixel. A three-dimensional image processing apparatus, wherein each word has the same number of words as the number of pixels of an image, and each word includes pixel data of a first image, pixel data of another image, a luminance gradient value, a cumulative luminance gradient value, and a block. An associative memory for storing size and disparity information; an image input initialization unit for storing pixel data of the input first image and other images in the associative memory; a luminance gradient for each pixel; And a luminance gradient calculation control unit that stores the luminance gradient value sequentially for the neighboring processing blocks along the spiral with the corresponding pixel of the processing block as the center, and transfers the luminance gradient value to the associative memory. The brightness gradient A spiral scanning processing control unit for adding to the product value; a block size calculation control unit for obtaining a block size based on the number of scanning pixels or the number of rotations obtained for each pixel and storing the block size in the associative memory; Find the block size of
A three-dimensional image processing apparatus having a corresponding point search processing control unit that performs a corresponding point search process between an image of the image and another image. 10. A plurality of images taken from different viewpoints are input, and for each pixel of the first image, the most corresponding pixel position in the image excluding the first image is searched for, and the pixel positions are searched. A three-dimensional image processing program for extracting a difference between the two images as disparity information and obtaining distance information from the disparity information, wherein: a process of inputting the plurality of images; and a process of calculating a luminance gradient for each of the plurality of pixels A process of obtaining the cumulative value of the luminance gradient in the vicinity of the pixel for each pixel by scanning the pixels spirally or concentrically; and calculating the cumulative value based on the number of pixels scanned or the number of times of spiral scanning. Normalizing processing, obtaining a block size by threshold determination from the normalized value, and using the obtained block size of each pixel, perform a corresponding point search processing between the first image and another image, Parallax information Recording medium for recording a three-dimensional image processing program for causing a computer to execute a process of extracting a distance and outputting a distance image. 11. A first image and another image are input, and a first image is input.
A three-dimensional image processing program that outputs a distance image by associating pixels between an image of the image and another image and obtaining parallax information for each pixel, and stores pixel data of a first image A memory that stores pixel data of another image, a memory that stores a luminance gradient value of the pixel, a memory that stores a cumulative value of the luminance gradient value, a memory that stores a block size, and a memory that stores parallax information A data storage unit, a data receiving unit that receives data from an adjacent processing block and stores the data in the data storage unit, calculates a luminance gradient of each pixel, and stores the luminance gradient in the data storage unit; A luminance gradient value accumulating unit that performs a value accumulating process and stores the luminance gradient value in the data storage unit; a cumulative value threshold value determining unit that determines whether the luminance gradient value exceeds a threshold value; Calculate the optimal block size for each pixel based on the number of scan pixels or the number of rotations, and calculate the block size calculation unit to be stored in the data storage unit and the block size obtained for each pixel,
A corresponding point search processing unit that performs a corresponding point search process between the first image and another image, a data transmission unit that transfers data that needs to be transferred within a processing block to one adjacent processing block, Receiving, in response to the command, the data receiving unit,
The processing block including the luminance gradient calculation unit, the luminance gradient value accumulation processing unit, the cumulative value threshold determination processing unit, and a command receiving unit that activates one of the data transmission units corresponds to each pixel of the image data. An arrayed PE array processing unit, which transmits the input pixel data of the first image and the other image to the data receiving unit of the corresponding processing block, and issues a data storage instruction to the corresponding processing block instruction. An image input initialization process to be transmitted to the receiving unit; a luminance gradient calculating process for issuing a command to start the luminance gradient calculating unit to the command receiving units of all the processing blocks; and scanning pixels in a spiral or concentric manner. A spiral scanning process for issuing an instruction to activate the luminance gradient value accumulating unit and the accumulative value threshold value judging unit to the instruction receiving units of all the processing blocks; Block size calculation processing for issuing a command to start the block size calculation section to the command reception section of the block, and issuing a command to start the corresponding point search processing section to the command reception sections of all processing blocks. A recording medium that stores a three-dimensional image processing program for causing a computer to execute a point search process. 12. A first image and another image are input, and a first image is input.
This is a three-dimensional image processing program that outputs a distance image by associating pixels between an image and another image and obtaining disparity information for each pixel, and has the same number of words as the number of pixels of the image. The associative memory stores the pixel data of the first image, the pixel data of another image, the luminance gradient value, the cumulative value of the luminance gradient, the block size, and the disparity information in each word. Image input initialization processing for storing the pixel data of the image, luminance gradient calculation processing for obtaining a luminance gradient for each pixel, and storing the luminance gradient in the associative memory; and spiral processing centering on the corresponding pixel of the processing block. Helical scanning processing for sequentially reading out the luminance gradient values of the neighboring processing blocks, transferring the luminance gradient values to the associative memory, and adding the luminance gradient accumulated value in the associative memory; and the number of scanning pixels obtained for each pixel. Alternatively, a block size is calculated based on the number of revolutions, and a block size calculation process for storing the block size in the associative memory;
The recording medium which recorded the three-dimensional image processing program for making a computer perform the corresponding point search process which performs the corresponding point search process between the image of this and another image.