JP2948617B2 - Stereo image association device - Google Patents

Description

Description: FIELD OF THE INVENTION The present invention relates to a stereo image associating device for finding corresponding points in left and right images (stereo images) taken by two video cameras or the like. It is.

(Prior Art) Conventionally, in order to extract three-dimensional information of an object to be photographed from left and right images photographed by two television cameras or the like arranged in a horizontal direction at a predetermined interval, each image is analogized. / Digital conversion (hereinafter referred to as A / D conversion)
A stereo image associating apparatus for finding corresponding points corresponding to each scanning line has been studied, and the method for finding the corresponding points is based on finding points where the densities of both images are similar.

Conventionally, as an example of this kind of technology, a paper in the field of electronics and communication science, "Segment correspondence method of stereo images by dynamic programming" J6
8-D [4] (1985-4) As described in IEICE, pp. 554-561, similarity of density for each pair of scanning lines in left and right image data using dynamic programming. The corresponding point was determined with reference to the gender.

FIGS. 2 (a) and 2 (b) are diagrams showing the association by the dynamic programming method described in the above-mentioned document, and FIG. 2 (a) is a side view showing an example of an object to be photographed. (b) is a diagram showing a correspondence search plane. Note that a case where a section sandwiched between points on a scanning line at which a density change is equal to or more than a certain value (hereinafter, referred to as an edge point) is considered as a corresponding unit will be described.

When an object 1 as shown in FIG. 2A is photographed by a video camera or the like (not shown) from the left direction indicated by arrow 2 and the right direction indicated by arrow 3, the top surface is the brightest and the bottom surface is the darkest. Assuming that both sides have intermediate brightness, the brightness on one scanning line in the left and right images is the second brightness.
This shows changes as indicated by reference numerals 4 and 5 in FIG. However, in the brightness change curve 4, the horizontal axis is a scanning line in the left image (hereinafter, referred to as a left scanning line, N; the number of sections on the left scanning line).
The vertical axis indicates the brightness, and in the brightness change curve 5, the vertical axis indicates the scanning line in the right image (hereinafter, referred to as the scanning line in the right image).
The position on the right scanning line (M; number of sections on the right scanning line) is shown, and the horizontal axis shows brightness.

In FIG. 2B, reference numeral 6 denotes a corresponding search plane by dynamic programming corresponding to the change curves 4 and 5,
The search for the correspondence between the left and right scanning lines is solved as a problem of finding a path indicating the correspondence on the plane 6. In the plane 6, the horizontal axis corresponds to the position on the left scanning line, the vertical axis corresponds to the position on the right scanning line, the vertical line indicates the position of the edge point calculated on the left scanning line, and the horizontal line indicates It shows the positions of the edge points calculated on the right scanning line, and both ends of each scanning line are also treated as edge points for convenience. If the intersection of these straight lines is called a node, the node corresponds to a decision stage in the dynamic programming, and each node performs a scan to select a path having the minimum cost among paths that reach the node.

A rectangle separated by vertical and horizontal lines indicates the possibility of a pair of sections on the left and right scanning lines. Path indicating the correspondence between a pair of left and right sections (that is, a diagonal or side of a certain rectangle on the search plane)
Is called an atomic path.
It is represented by a path connecting the atomic paths from the upper left to the lower right on the search plane.

The atomic path cost as this similarity measure has been calculated by referring to the densities of both images.

(Problems to be Solved by the Invention) However, the above-mentioned atomic path cost is calculated with reference to only the image density, and there is a possibility that an accurate cost cannot be obtained.

For example, FIGS. 3A, 3B, and 3C are diagrams showing image density distributions when the distance between the area 10 and the camera is considerably smaller than that in the area 11. FIG. When an area corresponding to the section R1 shown in FIG. 11B is to be obtained from the right image, the sections R2 and R3 shown in FIG. However, it is difficult to determine which of the sections R2 and R3 corresponds to the section R1 based on the value of the atomic path cost calculated by referring to only the image density. for that reason,
There is a high possibility that erroneous correspondence will occur, and the reliability of the association accuracy is not sufficient.

An object of the present invention is to provide a stereo image associating apparatus which solves the problem of the prior art that the reliability of the associating accuracy is not sufficient.

(Means for Solving the Problems) In order to solve the above-mentioned problems, the present invention provides a method for solving the above-mentioned problem, based on left image data corresponding to each scanning line of a left input image obtained from a left image input device. A left density calculator for calculating a density; and a right input based on right image data corresponding to each scanning line of a right input image obtained from a right image input device arranged at a predetermined interval with respect to the left image input device. A right density calculator for calculating an image density of an image, and a correspondence for obtaining a similarity between the left image data and the right image data for each of the scans using pixel densities of the left input image and the right input image. The following means are taken in a stereo image associating device provided with a processing unit.

A right image density change calculation unit that calculates a right image density change amount that is a density change amount of a plurality of pixels located near each pixel in the left input image based on the pixel density of the left input image;
A right image density change calculating unit that calculates a right image density change amount that is a density change amount of a plurality of pixels located near each pixel in the right input image based on the pixel density of the right input image;
And a dividing unit that divides the right input image and the left input image from a left end to a right end on the scanning line into a plurality of sections; and a pixel density of the left input image, the right input A similarity calculating unit that obtains the similarity in each section with reference to the pixel density of the image, the left image density change, and the right image density change, and the similarity calculates each of the right input images. And an associating means for associating a section with each section of the left input image.

Further, the left image density change calculation unit calculates, for each pixel in the left input image, a density change amount of each pixel located in a horizontal direction, a vertical direction, a left diagonal direction, and a right diagonal direction. The left image density change calculation unit is configured to calculate an image density change amount, and the left image density change calculation unit calculates, for each pixel in the right input image, a pixel positioned in a horizontal direction, a vertical direction, a right diagonal direction, and a right diagonal direction. The right image density change amount may be calculated from the density change amount.

(Operation) According to the present invention, the stereo image associating device is configured as described above, and the left image density change calculation unit and the right image density change calculation unit correspond to the left image density change amount and the right image density change amount. The left and right image data are referred to the left image density change amount and the right image density change amount in addition to the pixel density of the left input image and the pixel density of the right input image. It works to find the similarity with the image data and improve the accuracy of the association process.

 Therefore, the above problem can be solved.

(Embodiment) FIG. 1 is a configuration block diagram of a stereo image associating apparatus showing an embodiment of the present invention.

This stereo image associating device performs A / D conversion on a left input image obtained from a left image input device such as a video camera installed at a predetermined distance from a shooting object (not shown) and outputs left image data D1. A left image input unit 50 and a right image input that A / D converts a right input image G2 obtained from a right image input device arranged at a predetermined interval with respect to the left image input device and outputs right image data D2. And a part 51.
The left image input unit 50 and the right image input unit 51 are configured by an A / D converter and the like.

On the output side of the left image input unit 50, a left density calculation unit 52 that calculates the pixel density of the left input image G1 based on the left image data D1,
A left image density change calculation unit 53 that obtains a left image density change amount that is a density change amount of a plurality of pixels located near each pixel in the left input image G1 is connected.

Further, the output side of the right image input unit 51 has the right image data
A right density calculating unit 54 for calculating a pixel density of the right input image G2 based on D2, and a right image for obtaining a right image density change amount which is a density change amount of a plurality of pixels located near each pixel in the right input image G2. The density change calculator 55 is connected. here,
Left image density change calculator 53 and right image density change calculator 55
Comprises a memory for storing image density, a multiplier, an adder, a register, and the like. The left image density change amount is calculated for each pixel in the left input image G1 by the density change amount of each pixel located in the horizontal direction, the vertical direction, the left diagonal direction, and the right diagonal direction. The amount is set to be calculated with respect to each pixel in the right input image G2 based on the density change amount of each pixel located in the horizontal direction, the vertical direction, the right oblique direction, and the right oblique direction.

On the output side of the left density calculation unit 52, the right density calculation unit 54, the left image density change calculation unit 53, and the right image density change calculation unit 55, the similarity between the left image data D1 and the right image data D2 is scanned. The associating processing unit 56 that is required every time is connected.

The associating processing unit 56 includes a dividing unit 56a that divides the right input image G2 and the left input image G1 from the left end to the right end on the scanning line into a plurality of sections, a pixel density of the left input image G1, and a right input image G2 pixel density, left image density change amount,
With reference to the amount of change in the density of the right image and the similarity calculating means 56b for calculating the similarity between the left image data D1 and the right image data D2 in each of the sections, and each section of the right input image G2 by the similarity. It is made up of an associating means 56c that performs association with each section of the left input image G1 and outputs an association result E.

Here, the left density calculation unit 52, the right density calculation unit 54, the left image density change calculation unit 53, the right image density change calculation unit 55, and the association processing unit 56 are configured as a part of the central processing unit.

The operation of the stereo image association device configured as described above will be described.

For example, the left input image G1 of the photographing object obtained from the left image input device is subjected to A / D conversion in the left image input unit 50 and output as left image data D1. Similarly, the right input image G2 of the photographing object obtained from the right image input device is
In the right image input unit 51, the right image data D
Output as 2. The left image data D1 is converted to the left density calculation unit 52.
, The pixel density of the left input image G1 is calculated by the left density calculator 52. The left image density change calculation unit 53 receives the pixel density and calculates the magnitude relationship of the density near the pixel for the pixel at the horizontal pixel number ii and the vertical pixel number jj. One example is shown in FIG.

The left image density change amount iv (ii, jj,) is given by the following equation.

The constant a (x, y,) is given as shown in FIGS. 5 (a) to 5 (d). FIG. 7A shows the horizontal change amount (=
0), FIG. 13B shows the vertical change amount (= 1), FIG. 14C shows the left oblique change amount (= 2), and FIG.
Is a given value for calculating the amount of change in the oblique right direction (= 3).

On the other hand, when the right image data D2 is input to the right density calculator 54, the right density calculator 54 calculates the pixel density of the right input image G2. The right image density change calculation unit 55 receives the pixel density, performs the same operation as the left image density change calculation unit 53,
The right image density change amount is calculated.

Subsequently, the association processing unit 56 divides the right input image G2 from the left end to the right end into a plurality of sections on a predetermined scanning line by using the dividing unit 56a, and similarly, the left input image G1
Is divided into a plurality of sections from the left end to the right end. further,
The pixel density of the left input image G1 by the similarity calculating means 56b,
The similarity between the left image data D1 and the right image data D2 in each section is obtained with reference to the pixel density, the left density change amount, and the right density change amount of the right input image G2. An example of the similarity (atomic path cost) is shown below.

However, d (m, k); the right input at node m; node k atoms path leading to the node m from cost s; node on pixel positions left input image G1 in k rs; right input image G2 on the pixel position in the node k e Pixel position on image G2 re; pixel position on right input image G2 at node m ii = 0 to XW-1 XW; prime number in horizontal direction jj; predetermined scanning line i (ii, jj); left image density in scanning line jj ir (ii, jj); right image density at scanning line jj f (ii) = rs + (ii−s) + (re−rs) /
(E−s) KN; number of density change directions; density change direction number (0 to KN−1) iv (ii, jj,); left image density change amount ivr (ii, jj,); right image density change amount The attaching means 56c uses, for example, the dynamic programming method shown in FIG. 2 to calculate each section of the right input image G2 and each section of the left input image G1 based on the similarity calculated by the similarity calculating section 56b. Perform association.

 This embodiment has the following advantages.

(1) FIG. 6 is a diagram showing an effect obtained by using a density change amount for similarity calculation. The portion W1 indicated by the similarity based on the density component shown in this figure is the first sum term of the equation (2). The portion W2 represented by the similarity based on the density change amount is the second sum term of the equation (2). Corresponding to the result. Conventional similarity is Equation (3) corresponds to the first sum term in Equation (2).

Thus, in the conventional similarity calculation using only the concentration,
By adding the amount of concentration change, R1 and R2 shown in FIG.
(The similarity shown in FIG. 6 becomes smaller as the value increases, the similarity decreases), and finally a stable association result E can be obtained.

Note that the present invention is not limited to the illustrated embodiment, and various modifications are possible. For example, in the above-described embodiment, the city distance is used as the similarity measure in the association processing unit 56, but the present invention is not limited to this. For example, another similarity measure such as a Euclidean distance or a cross-correlation function can be used. It is.

(Effect of the Invention) As described above in detail, according to the present invention, when calculating the similarity between the left input image and the right input image, not only the image density but also the right input image and the left input image are calculated. Since the amount of change in the density of a plurality of pixels located near each pixel in the image is also referred to, the association process becomes accurate, and the reliability of the association accuracy is expected to be improved.

[Brief description of the drawings]

FIG. 1 is a block diagram showing a configuration of a stereo image associating apparatus showing an embodiment of the present invention, FIG. 2 is an explanatory diagram of an associating process by a dynamic programming method, and FIGS. 3 (a) to (c) are images. FIG. 4 is a diagram showing a density distribution, FIG. 4 is a diagram showing a change amount of an image density, and FIG.
6A to 6D are diagrams showing constants given to neighboring pixels, and FIG. 6 is a diagram showing the effect of similarity. 50 left image input section, 51 right image input section, 52 left image density calculation section, 53 left image density change calculation section, 54 right image density calculation section, 55 right image density Change calculator, 56 ……
Correlation processing section, 56a ... dividing means, 56b ... similarity calculating means, 56c ... correlating means, G1 ... left input image, G2 ...
... Right input image, D1 ... Left image data, D2 ... Right image data.

Claims (2)

(57) [Claims] A left-density calculator for obtaining a pixel density of the left input image based on left image data corresponding to each scanning line of the left input image obtained from the left image input device; A right density calculator for calculating an image density of the right input image based on right image data corresponding to each scanning line of the right input image obtained from the right image input device arranged at a predetermined interval; A stereo image associating device, comprising: a pixel density of the right input image, and an associating processor that obtains a similarity between the left image data and the right image data for each of the scanning lines. A left image density change calculating unit that obtains a left image density change amount that is a density change amount of a plurality of pixels located near each pixel in the left input image based on a pixel density of the left input image; Pixel A right image density change calculating unit that calculates a right image density change amount that is a density change amount of a plurality of pixels located in the vicinity of each pixel in the right input image based on the degree. Dividing means for dividing the right input image and the left input image from the left end to the right end on the scanning line into a plurality of sections; pixel density of the left input image, pixel density of the right input image, and the left image A density change amount, and a similarity calculation unit that obtains the similarity in each of the sections by referring to the right image density change amount; and the similarity, each section of the right input image and each of the left input image. A stereo image associating device, comprising: associating means for associating with a section. 2. The stereo image associating device according to claim 1, wherein the left image density change calculating unit is configured to determine a horizontal direction, a vertical direction, a left oblique direction, and a right direction for each pixel in the left input image. The left image density change amount is calculated based on the density change amount of each pixel located in an oblique direction, and the left image density change calculation unit is configured to calculate a horizontal direction and a vertical direction for each pixel in the right input image. A stereo image associating device configured to calculate the right image density change amount based on the density change amount of each pixel located in the direction, the right oblique direction, and the right oblique direction.