JP3055721B2 - Method for searching corresponding points of images captured by left and right cameras - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method for picking up or recognizing an object in a scene by photographing a forward scene with two or more imaging cameras and performing stereo image processing. In particular, the present invention relates to matching the same point of an object in the images captured by the left and right cameras, that is, to searching for a corresponding point.

[0002]

2. Description of the Related Art For example, in a vehicle or a ship, a technique for automatically detecting a preceding vehicle, a ship, or an obstacle is desired. Japanese Patent Application Laid-Open No. 61-44312 responds to this.
Japanese Patent Application Laid-Open No. 61-44313 and Japanese Patent Application Laid-Open No. 61-44313 disclose stereoscopic processing of image information for one screen each taken by two television cameras on a vehicle, and perform object processing in a scene shown by the cameras. A distance measuring device for calculating a distance is presented.

[0003] In the former, regions having the same brightness are divided into blocks on images taken by two cameras arranged at a predetermined distance from each other. Next, each block on one image is moved in a direction to reduce the parallax, and is compared with the other block to find a position where the blocks match. Calculate the distance from the amount moved until they match.

In the latter, regions having the same brightness are divided into blocks on images taken by two cameras arranged at a predetermined distance from each other. Next, the feature amount of each of the blocks on one image and the block on the other image is calculated, and the feature amounts are compared between the blocks captured by different cameras to detect the best matching block ( Detection of corresponding block). Then, the distance is calculated from the position difference between the corresponding blocks.

Further, Japanese Patent Application Laid-Open No. 2-2 filed by the present applicant
No. 9878 and Japanese Patent Application Laid-Open No. 2-29879
Detect a region on the photographic film, that is, a texture region of a specific object in the scene, in particular, an object having a complex outer surface with a large number of surface irregularities, and detect a distance, that is, a three-dimensional shape, of each surface portion of the object. Presenting technology.

Further, Japanese Patent Application No. Hei.
No. 262269 discloses an image processing method in which a foreground is photographed by a left and right camera on a vehicle, and a road surface in a foreground image is cut out.

[0007]

Problems to be Solved by the Invention
In a distance measuring device as disclosed in Japanese Patent Application Laid-Open No. 312 and Japanese Patent Application Laid-Open No. 61-44313, when a parallax between two cameras is large, a corresponding search is performed for a very large area. If there are many, the probability of erroneous detection of the corresponding block increases.

The distance measuring device disclosed in the above-mentioned Japanese Patent Application Laid-Open Nos. 2-29878 and 2-29879 has many advantages in detecting an object having a three-dimensionally complicated shape. Objects that are relatively monotonous and relatively large in area, and are unlikely to appear as one independent entity (single body) on the screen, such as the top surface of a table, floor surface, road surface,
A water surface, etc., a plane or a pseudo plane (hereinafter referred to as a plane portion),
Is difficult to remove. This is, of course, given that these distance measuring devices originally attempt to accurately extract a three-dimensional individual and more accurately detect surface irregularities.

[0009] In any of the object detection and tracking by stereoscopic vision, an erroneous correspondence is taken in the corresponding processing for searching for the same point of the object in the image taken by the left camera and the image taken by the right camera. And the recognition of an object using the stereoscopic processing technology of the left and right cameras is confused. For example, in the recognition of a front object on a vehicle, the position and speed of the front object (relative speed with respect to the own vehicle) are calculated based on the object recognition. But these are incorrect.

SUMMARY OF THE INVENTION It is an object of the present invention to more accurately search corresponding points of images captured by left and right cameras.

[0011]

Means for Solving the Problems A scene in front of the left and right imaging cameras is taken to obtain video signals, and these video signals are digitally processed.
In searching for the same point of the same object in the right image, t
o time, at least t time and t of the left and right images at time t1 after Δt and time t2 after Δt.
The left and right images at one time are subjected to thinning processing for expressing the edges of the object in the image with thin lines, and these to, t1, and t
Based on the left and right images at two times, a point with a thin line in the left (right) image at time to is determined as the point of interest Poa1, and the thin line on the right (left) image at time to is perpendicular to the point of interest Poa1. Extract the points Pob1, Pob2, Po3b on the corresponding vertical coordinate i of the coordinate i and to
The corresponding candidate points Pob1, Pob2, and Po3b on the right (left) image at the time are set as the coordinates (j, j) of the target point Poa1 on the left (right) image at the time t1.
Each point on the thin line of the predetermined area centered on i) is defined as a corresponding candidate point P1a-d to P1a-e on the left (right) image at time t1, and for each of the corresponding candidate points Pob1, Pob2, Po3b , The three-dimensional position of the attention point Poa1 at the to time determined by the coordinates of the attention point Poa1, and the right of each time t1 when the attention point Poa1 moves to each of the corresponding candidate points P1a-d to P1a-e ( Left) intersection of positions on image and thin line on right (left) image at time t1
The three-dimensional position of the point of interest Poa1 at time t1 determined by the coordinates of P1b is calculated, and the three-dimensional position of the point of interest Poa1 at time t2 is calculated by extrapolation of both three-dimensional positions. Are converted to the coordinates at the left of time t2 of these coordinate positions,
The correlation value C of the image density on the right image is calculated, and the corresponding candidate point Pob is calculated.
The candidate point having the largest correlation value C among 1, Pob2, and Po3b is determined as a corresponding point on the right (left) image at the to time of the target point Poa1 on the left (right) image at the time.

[0012]

The left and right images at the to time and at the time t1 after Δt and at the time t2 after Δt, at least the left and right images at the time to and at the time t1 are formed by thin lines of the edges of the object in the image. By performing the thinning process represented by, the left and right images at the time to and at the time t1 represent the edges of the object in the captured image, and therefore, the left and right images have corresponding points in correspondence between the thin lines.

By arranging the left and right cameras substantially horizontally, the point corresponding to one point Poa1 on a certain scanning line (horizontal line) in the image of the left camera is defined as the thin line in the image of the right camera. It becomes an intersection with a scanning line (horizontal line) corresponding to the scanning line. This intersection is usually plural (many). One point Poa1 in the image of the left camera is the one point P in the right camera.
Since it appears on the left side of the horizontal position j of oa1, the vertical coordinate of the point of interest Poa1 of the thin line on the right (left) image at time to
The points Pob1, Pob2, Po3b on the corresponding vertical coordinate i of i are extracted and the corresponding candidate points Pob1, Pob2, Po3b on the right (left) image at the to time
By doing so, the number of correspondence candidate points is reduced.

Since the moving range of the object photographed by the camera is narrow within the short time Δt, the range (predetermined area) can be determined. Thus, on the left (right) image at the time t1, each point on the thin line of the predetermined area centered on the coordinates (j, i) of the target point Poa1 is converted to a corresponding candidate point on the left (right) image at the time t1.
By setting P1a-d to P1a-e, the point of interest Poa1 on the left (right) image at the time t is one of the corresponding candidate points P1a-d to P1a-e on the left (right) image at the time t1. It is. Therefore, each of the corresponding candidate points Pob1, Pob2, and Po3b on the right (left) image at the to time is assumed to be the corresponding point of the attention point Poa1, and for example, for one candidate point Pob1, the left (right) image at the to time Top attention point = Poa1, corresponding point on right (left) image of time to = Pob1, t
Assuming that the corresponding point on the left (right) image at one time = P1a-d to P1a-e, a point sequence corresponding to P1a-d to P1a-e on the right (left) image at time t1 is obtained. The intersection P1b between the thin line on the right (left) image at the time t1 and this point sequence is set as the corresponding point P1b on the right (left) image at the time t1, and the point of interest Poa1 at the to time from Poa1 and Pob1.
From the intersection P1b at time t1 and the corresponding point P1b on the left (right) image at time t1 corresponding to the intersection P1b, the motion of the attention point Poa1 at time t1 is determined. Since this time is extremely short during 2Δt, the extrapolation method is applied to the three-dimensional position at the time t and the three-dimensional position at the time t1 assuming that the motion is substantially constant-velocity linear motion.
The three-dimensional position of the corresponding point of a1 is calculated. This calculated 3
A left and right image correlation value C of the image density at a position on the left and right images at time t2 corresponding to the dimensional position is calculated.

The same processing is performed for the remaining candidate points. The candidate point (one of Pob1, Pob2, and Po3b) that obtained the maximum value of the correlation values obtained in this way is set to the right (left) of the to time corresponding to the target point Poa1 of the left (right) image of the to time. )
It is determined as a corresponding point on the image.

Since the moving range of the object photographed by the camera is narrow within a short time Δt, the predetermined area is defined so as to always include the moving range.
The corresponding point after (time t1) must be a corresponding candidate point P1a-d to P1a
Extracted as -e. That is, the corresponding candidate point P1 at the time t1
The extraction of ad to P1a-e becomes accurate. This (P1a-d to P1a-e)
And corresponding points Pob1, Pob2, Po on the right (left) image of the to time
3b, the corresponding point sequence at time t1 (Pob1, Pob
2, Po3b), the corresponding point P1b (Pob1, Pob2, Pb1) on the right (left) image at time t1, which is the intersection P1b between this point sequence and the thin line on the right (left) image at time t1. (One for each Po3b). Since the movement of the object can be regarded as a constant-velocity linear motion, the three-dimensional position of the target point Poa at the to time and t
Corresponding point P1b at one time (one for each of Pob1, Pob2, Po3b)
The corresponding point at time t2 (Pob
Since one three-dimensional position is obtained for each of 1, Pob2, and Po3b, the estimation accuracy of the corresponding point at time t2 is high. In addition, the densities of the same point of the object on the left and right images are substantially the same, and by paying attention to this, with respect to each corresponding point at t2 (one for each of Pob1, Pob2, and Po3b), at time t2 The correlation value C of the image densities on the left and right images is calculated, and the corresponding point supplementary point (one of Pob1, Pob2, Po3b) for which the maximum value of the correlation value is obtained is set to the left (right) of the to time.
Since the corresponding point on the right (left) image at the to time corresponding to the attention point Poa1 of the image is determined, the estimation of the corresponding point is accurate.

As described above, the corresponding points (X2a, Y2a), (X2b, Y2) obtained by extrapolation using the position change of the corresponding points due to the motion of the object.
By estimating b) and calculating and comparing the correlation values of the left and right image densities using the obvious matter that the same part of the same object has the same brightness, one point is obtained from a plurality of candidate points Pob1, Pob2, Po3b. Since the corresponding point is determined, the reliability of recognition of an object moving relatively quickly is high.

Other objects and features of the present invention will become apparent from the following description of embodiments with reference to the drawings.

[0019]

FIG. 2 shows the configuration of an image processing apparatus for implementing the present invention in one embodiment. This apparatus comprises a computer 1, an imaging camera 2a (left), 2b (right), an image memory 4, a display 5, a printer 6, a floppy disk 7, a keyboard terminal 8, and the like. It is mounted on passenger cars. The left and right cameras 2a and 2b are ITV cameras (CCD cameras) having the same specifications for photographing a front scene and outputting analog image signals in 512 × 512 pixel sections, respectively. A / D converter 2
2a and 22b convert digital data (image data) of 256 gradations per pixel. The imaging cameras 2a and 2b are at a certain height h from a plane (road surface), and image the front scene obliquely downward (downward angle) with two stereos. FIG.
As shown in the figure, the photographed movie surface FLa of the left camera 2a is referred to as a left image and the photographed movie surface FLb of the right camera 2b.

The image memory 4 shown in FIG. 2 is readable and writable, and stores various processing data including original image data of a left image and a right image. The display unit 5 and the printer 6 output the processing results of the computer 1 and the like, and the floppy disk 7 registers the processing results. The keyboard terminal 8 is operated by an operator to input various instructions. Computer 1
Is further connected to a host computer from which instructions or keyboard terminal 8 provided thereto are connected.
Each part is controlled in accordance with the instruction given by the controller and the corresponding processing is performed. Hereinafter, of the processing, "speed monitoring of a forward object" for recognizing an object in a forward scene and detecting its position (distance from the camera) and speed (relative speed with respect to the camera). explain.

In object recognition by this type of image processing apparatus, there are two points (objects or a part thereof) in front of them, and as shown in FIG.
B1 and A2 and B2 at a time t1 after Δt, and A3 and B3 at a time t2 after Δt, an erroneous correspondence occurs in object recognition, and as shown in FIG. Recognize a point as one point (corresponding point), PC1, PC2, PC
When the tracking point is tracked as 3, the recognition points PC1, PC2, and PC3 do not form a uniform linear motion (however, there are exceptions). Therefore, if the position PC3 ′ at time t2 is obtained by extrapolation from PC1 at time t1 and PC2 at time t1 assuming uniform linear motion, it is different from PC3 actually recognized, and
Of the brightness on the left image (PA3) and the brightness on the right image (PB3)
, And the correlation values of these brightnesses are low.
The displacement ΔX can be expressed by the following equation (1).

[0022]

(Equation 1)

Each parameter in the equation (1) is as follows.

[0024]

(Equation 2)

[0025]

(Equation 3)

[0026]

(Equation 4)

[0027]

(Equation 5)

[0028]

(Equation 6)

[0029]

(Equation 7)

In this embodiment, "speed monitoring of a forward object"
It is intended to quickly and accurately find a corresponding point by positively utilizing a correlation value between constant velocity linear motion and brightness. FIG.
In this embodiment, the host computer or the key
The outline of a "forward object speed monitoring" routine in which the computer 1 repeats at a substantially predetermined cycle until the instruction is released in response to the instruction given from the board terminal 8 is shown. 6 to 11 show processing contents of main processing items among them.

First, referring to FIG.
Writes the image data of the left image and the right image of the left and right cameras 2a and 2b into the image memory 4 at the beginning of the process of "monitoring the speed of the forward object" in one cycle (subroutine 1).
In the following, in the parentheses, the word "subroutine" or "step" is omitted, and only the numeral attached thereto is shown. Next, differential processing of the image data is performed (2). Figure 6
Shown in

In the "differential processing" (2), a forward raster scan is first set for the left image. The direction of the raster scan is the horizontal direction of the shooting scene (the left-right direction when looking forward from above the vehicle). In the forward raster scan, the image pickup area FLa (FLb on the right screen) shown in FIG.
The ua (ub) axis parallel to (Xb) is the main scanning axis, and
With the va (vb) axis parallel to (Yb) as a sub-scanning axis, scanning is performed on each pixel along a path from the upper left pixel to the lower right pixel. That is, scanning in the u direction (main scanning direction) and v (sub scanning direction) is performed from the upper left pixel (image origin 0, 0) to the lower right pixel (512, 512). Differential data is generated while performing a forward raster scan.

The differential data is obtained by converting the original image data of the original image data (the 3 × 3 image data matrix centered on the target pixel) of the target pixel and eight neighboring pixels into the target pixel. , The image density of the pixel on the right side is p1, the image density of the pixel above p1 is p2, the image density of the pixel above the pixel of interest is p3, and the pixel on the left side of p3 is p0. The image density of the pixel on the left of the pixel of interest is p5, and the image density of the pixel below p5 is p6.
And the image density of the pixel below the target pixel is p7, and p7
Assuming that the image density of the pixel on the right of the image is p8, (p1 + p
2 + p8)-(p4 + p5 + p6) and the main scanning direction differential data, and (p2 + p3 + p4)-(p6 + p
7 + p8), which is the sum with the differential data in the sub-scanning direction, and indicates the spatial variation of the original image data. The computer 1 writes these data into the image memory 4 in association with each pixel.

Next, the computer 1 converts the differential data into a threshold value TH1 by "threshold value processing" (3) shown in FIG.
In contrast to the above, the differential data larger than TH1 is left as it is for the pixel, and the differential data smaller than TH1 is replaced with 0 for the pixel. As a result of binarization in this manner, an area "edge area" where the change in the value of the original image data is large is detected, and only in this area, the differential data remains as a large value.
In other regions, the differential data is 0 (no change in brightness: continuous region).

Next, the computer 1 sets "1" as the switching point of the differential data from large to small in the vertical direction by "thinning" (4) shown in FIG. As “0”
"1" converts the differential data into a binary image representing the boundary (pixel of the boundary line). As a result, a thin line image in which the boundaries of objects in the image are represented by thin lines is obtained.

FIGS. 12 to 14 show examples of images taken by the left and right cameras 2a and 2b (original images: obtained by the image input 1) at times t, t1 and t2. (A) in these figures are images taken by the left camera 2a,
(B) is an image shot by the right camera 2b. Due to the above-described differentiation processing (2) to thinning (4), the position in the original image (FIGS. 12 to 14) where the brightness changes sharply, for example, the boundary between the road surface and the white line thereon, the road surface The boundary between the foreground and
The boundary between the road surface and the vehicle, the boundary between the foreground and the vehicle, the boundary between the shadow of the vehicle and the exposed road surface, the boundary between the body and the window of a single vehicle, etc.
A thin line image in which the edges of the object are substantially thin lines is obtained, and this is stored in the memory.

The computer 1 performs the above-described image input (1), differential processing (2), threshold processing (3) and thinning (4) at left and right cameras 2a and 2b with a period of Δt. Is executed for each of the captured images. When executing this image reading process, the data in the memory area for storing the thinned image at time t1 is moved to the memory area for storing the thinned image at time t, and the thinned image at time t2 is stored. The data in the memory area for storing is transferred to the memory area for storing the thinned image at time t1, and the above-described image input (1) to thinning (4) are executed to obtain the thin line image. Is written into the memory area for storing the fine line image at time t2 for each of the left and right images. As a result, from the time point when 2Δt has elapsed since the first image input (1) was executed, the above-described memory area contains the most recently read thin line image (the thin line image at time t2) and Δt before that. The read thin line image (the thin line image at time t1) and the thin line image read 2Δt ago (the thin line image at time to) are always present.
The latest (at time t2) original image data is held in the original image data memory area until the next image input (1) is executed.

The computer 1 operates at three times to, t1, t
From the time when the thin line image No. 3 is prepared, every time the thinning (4) is completed, “time-series binocular stereoscopic viewing” (6) is executed. This content is shown in FIG. 9, and the content of "process 1" (68) therein is shown in FIG. 10, and "process 2" in process (1) of FIG.
11 is shown in FIG. FIG. 1 shows a thin line image to be processed in the “time-series binocular stereopsis” (6) in a simplified manner for easy understanding. `` Time-series binocular stereovision ''
In (6), the computer 1 firstly displays the left image at the to time (thin line image: in the following description of the time-series binocular stereoscopic vision 6, the thin line image is simply expressed as an image). Is scanned to search for black ("1") information in the thin line portion (61-64). When one piece of black information Poa1 exists at the coordinates (j, i), this is set as a point of interest Poa1. Black information Pob1, Pob2, Pob3 on the left side (i.e., an area where i is smaller) than the horizontal position i of the black information Poa1 on the left image on the same scanning line as the horizontal scanning line Yi on the right image at the time to
The candidate points Pob1, Pob2, and Pob3 corresponding to the point of interest Poa1 in the left image.

-(1) When the first corresponding candidate point Pob1 is searched on the right image at the to-time (67), "processing 1" (6
Execute 8). First, in the left image at the time t1, a window area Win of (j ± 20, i ± 20) is set around the coordinates (j, i) of the target point Poa1, and the black information P1a-e to P
1a-d is tentatively defined as the corresponding candidate point group at time t1 (681-68).
Five).

-(2) Regarding the first point P1a-e of the first corresponding candidate point group at time t1, Poa1 (left image at time to), Poa1
ob1 (right image at time to) and P1a-e (left image at time t1)
Are assumed to be corresponding points, and corresponding points on the right image at time t1 are obtained (686). In this case, the horizontal position (X coordinate) X1b * of the corresponding point to be obtained is calculated by the following equation (8). The vertical position (Y coordinate) is the same as that of P1a-e.

[0041]

(Equation 8)

The meanings of the various symbols in the formulas and drawings are as follows. X1b *: Predicted corresponding point (P1a-d) in the right image at time t1
(Xca, Yca): Center of left image, (Xcb, Ycb): Center of right and left image, Sxa, Sya: Scale factor of left image, Sxb, Syb: of right image Scale factor, θ: downward angle of camera, Y1: Y coordinate of corresponding point P1b in right image at time t1, (X2a, Y2a): corresponding point on left image at time t2, (X2b, Y2b): time t2 F (Y, X): Original image data, G (Y, X): Differential data, H (Y, X): Differential data after threshold processing , P (Y, X): thin line data (“1”: thin line portion / “0”:
Not a thin line).

Next, t1 is added to the obtained corresponding point (X1b *).
It is checked whether black information exists in the right image of the time (689).
If there is no black information in the right image at time t1 at the position corresponding to the first corresponding point P1a-e, the second candidate of the corresponding candidate point group P1a-e to P1a-d at time t1 in the window area Win ,
Similarly, a corresponding point on the right image at time t1 is determined (686), and it is checked whether black information exists there. In this manner, the corresponding points on the right image at the time t1 of the corresponding candidate point groups P1a-e to P1a-d are sequentially calculated, and it is checked whether there is black information there.
This is repeated until there is black information. That is, a thin line (indicated by a dotted line in FIG. 1) that appears as a sequence of corresponding points on the right image at time t1 of the corresponding candidate point groups P1a-e to P1a-d, An intersection P1b with an existing thin line (shown by a solid line in FIG. 1) is searched, and this is determined as a corresponding point (685 to 689).
One of the corresponding candidate point groups P1a-e to P1a-d on the left image at time t1, which is the basis for calculating the intersection P1b (this is referred to as P1a
-P).

-(3) At this point, the point of interest Poa1 on the left image and its corresponding point Pob1 on the right image at the time to, and the point of interest P1a-p on the left image at the time t1. The corresponding point P1b on the right image has been determined. Therefore, this point (to
Poa1 at time = Pob1, P1a−p = P1b at time t1)
From the three-dimensional position at time to and the three-dimensional position at time t1,
From the three-dimensional position at time t1 on a straight line connecting them, t
o, t1 at a position separated by the distance between the three-dimensional positions at time t1
Assuming that there is a three-dimensional position at two times, the three-dimensional position at time t2 is obtained by extrapolation, and the three-dimensional position at time t2 is the position (X2a, Y2a) on the left image at time t2 and the position on the upper right image at time t2 (X2b, Y2b) is calculated (6901, 6902). The main calculation formulas are shown below. Accordingly, the corresponding points P1a-p, P1b on the left and right images at the time t1 of the corresponding points Poa1 and Pob1 on the left and right images at the time to, and the corresponding points (X2a, Y2a), (X2b, Y2b)
Has been estimated.

[0045]

(Equation 9)

[0046]

(Equation 10)

(4) In order to further ensure the accuracy of the estimation, the computer 1 outputs the left and right original image data at the time t2.
Corresponding points (X2a, Y2a), (X2a
b, Y2b), image data of 3 × 3 pixels is read out, the sum of image densities of 3 × 3 pixels is calculated by the following equation (11), and the sum of the left image and the right image is calculated. The difference between the sums is calculated, the reciprocal of the obtained difference is calculated, and this is set as the correlation value C (6903). That is, the brightness correlation value C of the corresponding points (X2a, Y2a) and (X2b, Y2b) is calculated.

[0048]

[Equation 11]

As described above, the correlation value C indicating the degree of possibility that the first corresponding candidate point Pob1 on the right image at the to time is a corresponding point has been obtained.

The computer 1 is located on the same scan line as the horizontal scan line Yi of the target point Poa1 on the right image at the time to, on the left side of the horizontal position i of the target point Poa1 (the area where i is smaller). Black information P for which calculation processing of correlation value C has been completed
Regarding the black information Pob2 following the ob1 as well, the above-(1)-
Similarly, the correlation value C calculation processing of (4) is executed to obtain the black information Pob2.
Is compared with the previously calculated correlation value of the black information Pob1, the larger correlation value is stored and retained, and the candidate point (one of Pob1 and Pob2) at which the larger correlation value was obtained is set at the time t. The corresponding point (X2
a, Y2a) and (X2b, Y2b) are stored as corresponding points at time t2 (6
903-6906).

Similarly, the computer 1 is located on the same scanning line as the horizontal scanning line Yi of the target point Poa1 on the right image at the time to, on the left side of the horizontal position i of the target point Poa1 (i
The above-described processing is similarly performed for the remaining black information Pob3 in an area where is small (6903 to 6906).

In the right image at the time to, on the same scanning line as the horizontal scanning line Yi of the target point Poa1, all the corresponding candidate points on the left side of the horizontal position i of the target point Poa1 (i. When processing such as the above-described calculation of the correlation value C is completed for the information Pob1, Pob2, and Pob3, at the end time point, the calculated maximum value of the correlation value C and the corresponding candidate point on the right image at the to time that resulted this, The corresponding candidate point at time t2 is stored in the computer 1. From the above,
The corresponding point on the right image at the to time and the corresponding point on the left and right images at the time t2 for one point Poa1 on the thin line on the left image at the to time are determined.

The computer 1 similarly performs the corresponding point search of the left and right thin line images at the to time for each of the points constituting the thin line in the image (61 to 71). When the corresponding point search of the left and right thin line images at the to time is completed, the three-dimensional position of each corresponding point can be calculated, and a diagram viewed from various directions based on the three-dimensional position data can be drawn. Can be. For example, if each point is drawn on the X and Z planes based on the three-dimensional position data, for example, a diagram shown in FIG. 15A is obtained, which is a diagram in front of the vehicle equipped with the cameras 2a and 2b. It is a top view of a road and shows the presence of the object (vehicle) on a road. When drawn on the X and Y planes, for example, a diagram shown in FIG. 15B is obtained, which shows the presence of an object in the left, right, and vertical directions in front of the vehicle equipped with the cameras 2a and 2b.
Drawing on a plane oblique to the X, Y, and Z axes, for example, FIG.
(C) is obtained, which is a perspective view of the road. Further, when drawn on the Y and Z planes, for example, a diagram shown in FIG. 15D is obtained, which is a side view of the road. In this way, the road and the objects thereon can be recognized three-dimensionally and can be three-dimensionally represented.

When the corresponding point search for one screen is completed, the computer 1 executes "distance calculation" (72) in this embodiment. In this case, the three-dimensional positions Xo, Yo and Zo are calculated by the following equation (12) based on the left and right corresponding points at the to time for each point for which corresponding point search has been performed.

[0055]

(Equation 12)

Xo is the position in the left and right directions viewed from the camera,
Yo is the position in the vertical direction, and Zo is the forward distance. After completing the "distance calculation" (72), the computer 1 executes "speed vector calculation" (73). In this case, as shown in the following equation (13), for each point for which a corresponding point has been searched, the difference between the position at the time to and the position at the time t2 is calculated.
(This corresponds to 2Δt) to calculate the lateral velocities (relative velocities with respect to the camera) Vx and Vy.

[0057]

(Equation 13)

By calculating the Z component of the difference between the three-dimensional position at the time t and the three-dimensional position at the time t2, the relative velocity in the Z (forward) direction can be calculated. When the speed in the Z direction is set on the horizontal axis and the number of points having the same speed is set on the vertical axis, a speed distribution is obtained, for example, as shown in FIG. FIG. 16 shows a vehicle having the same speed as the vehicle (own vehicle) equipped with the cameras 2a and 2b (the center position on the horizontal axis in FIG. 16: speed 0).
And the vehicle moving away at a faster speed than the own vehicle (-on the horizontal axis)
Signifies that there is a direction to go away). The same X,
Drawing a straight line connecting the corresponding points of the to time and the t2 time on the Y plane yields the diagram shown in FIG. These lines indicate speed vectors, and visually, the direction of the straight line indicates the relative movement direction with respect to the own vehicle, and the length of the straight line indicates the relative movement speed with respect to the own vehicle.

The computer 1 executes the above-described processing on one screen (time-series binocular stereovision 6) by “image input”.
It is executed every time (1) is executed, that is, in the Δt cycle.

[0060]

In the short time interval Δt, the movement of the object can be regarded as a uniform linear motion. In the present invention, attention point of to time
The three-dimensional position of Poa and the corresponding point P1b at time t1 (Pob1, Pob2,
From the three-dimensional position (one for each Po3b), t
3 of the corresponding points of two times (one for each of Pob1, Pob2, Po3b)
Since the dimensional position is obtained, the estimation accuracy of the corresponding point at the time t2 is high. In addition, the densities of the same point of the object on the left and right images are substantially the same, and by paying attention to this, with respect to each corresponding point at t2 (one for each of Pob1, Pob2, and Po3b), at time t2 Calculate the correlation value C of the image density on the left and right images, and calculate the corresponding point supplementary point (one of Pob1, Pob2, and Po3b) from which the maximum value of the correlation value is obtained,
to corresponding to the point of interest Poa1 of the left (right) image at the to time
Since the corresponding point is determined on the right (left) image of the time, the estimation of the corresponding point is accurate. Thus, the corresponding points (X2a, Y2a), (X2a) by extrapolation using the position change of the corresponding points due to the motion of the object
b, Y2b), by calculating and comparing the correlation values of the left and right image densities using the obvious matter that the same part of the same object has the same brightness, from a plurality of candidate points Pob1, Pob2, Po3b Since one point is determined as a corresponding point, the reliability of recognition of an object moving relatively quickly is high.

[Brief description of the drawings]

FIG. 1 is a plan view showing a simplified thin line image obtained by thinning a shooting screen in order to explain a corresponding point search according to the present invention.

FIG. 2 is a block diagram illustrating a schematic configuration of an image processing apparatus that implements the present invention in one aspect.

3 is a perspective view showing an optical distance relationship between images FLa and FLb captured by left and right cameras 2a and 2b shown in FIG. 2 and a point P on a plane in front of the cameras.

FIG. 4 is a plan view in which the optical distance relationship shown in FIG. 3 is projected on an X, Z plane shown in FIG.

FIG. 5 is a flowchart showing a processing content for performing “speed monitoring of a forward object” of the computer 1 shown in FIG. 2;

FIG. 6 is a flowchart showing processing contents of "image input" (1) shown in FIG.

FIG. 7 is a flowchart showing processing contents of “threshold processing” (3) shown in FIG. 5;

8 is a flowchart showing the processing content of "thinning" (4) shown in FIG.

FIG. 9 is a flowchart showing processing contents of “time-series binocular stereovision” (6) shown in FIG. 5;

FIG. 10 is a flowchart showing the processing contents of "processing 1" (68) shown in FIG.

FIG. 11 is a flowchart showing processing contents of “processing 2” (690) shown in FIG.

12 is a plan view showing an example of an image captured by the left and right cameras 2a and 2b shown in FIG. 2, where (a) shows an image captured by the left camera 2a, and (b) shows an image captured by the right camera 2b. 4 shows a captured image.

13 is a plan view showing an example of an image captured by the left and right cameras 2a and 2b shown in FIG. 2, and is an image Δt after the image shown in FIG. 12, (a) of the left camera 2a (B) shows the captured image of the right camera 2b.

14 is a plan view showing an example of an image captured by the left and right cameras 2a and 2b shown in FIG. 2, and is an image Δt after the image shown in FIG. 13, and (a) of the left camera 2a (B) shows the captured image of the right camera 2b.

15A is a plane projection thin line diagram of an object on a screen obtained by determining corresponding points by thinning and corresponding point search of the images of FIGS. 12 to 14, and FIG. 15B is a vertical plane projection thin line. (C) is a perspective thin line diagram, and (d) is a side thin line diagram.

FIG. 16 is a graph showing the velocity distribution of the forward object obtained by determining corresponding points by thinning and corresponding point search in the images of FIGS. 12 to 14, wherein the horizontal axis represents the velocity and the vertical axis represents the same. Indicates the accumulated value of the speed point.

FIG. 17 is a diagram showing a moving direction and a moving amount of the forward object for 2Δt time obtained by determining corresponding points by thinning and corresponding point search of the images of FIGS. 12 to 14;

[Explanation of symbols]

Poa1: a point on a thin line on the left image at the point of interest / to time corresponding to the corresponding point search target Pob1, Pob2, Pob3: corresponding candidate points on the right image at the to time P1a-d to P1a-e: left image at the time t1 Upper corresponding candidate point group P1b: Calculated corresponding point (X2a, Y2a) at time t1, calculated point (X2b, Y2b) on the left image at time t2, calculated on right image at time t2. Corresponding point

────────────────────────────────────────────────── ─── Continued on the front page Examiner Kazuo Shibata (56) References JP-A-3-122510 (JP, A) (58) Fields investigated (Int. Cl. ⁷ , DB name) G01B 11/00-11 / 30

Claims (2)

(57) [Claims] 1. A left and right imaging camera shoots scenes in front of them and obtains video signals, respectively, and digitally processes these video signals to obtain the same object in the left and right images. In searching for one point, at least the left and right images at the to time, the t1 time after Δt and further at the t2 time after Δt, the left and right images at the to time and the t1 time are the objects of the object in the image. Based on the left and right images at times to, t1 and t2, a point having a thin line in the left (right) image at time to is subjected to a thinning process of expressing edges as thin lines.
a1 and the point of interest of the thin line on the right (left) image at time to
Points Pob1, Pob2, on the corresponding vertical coordinate i of the vertical coordinate i of Poa1
Po3b is extracted and the corresponding candidate point on the right (left) image of the to time
Pob1, Pob2, Po3b, − (1) coordinates of the point of interest Poa1 on the left (right) image at time t1
Each point on the thin line of the predetermined area centered on (j, i) is set as a corresponding candidate point P1a-d to P1a-e on the left (right) image at time t1, and-(2) the target point Poa1 is at time t1 Corresponding to the corresponding candidate points P1a-d to P1a-e on the left (right) image, and corresponding corresponding points Pob1, Pob2, One of Po3b Calculates each movement position on the right (left) image at time t1 of Pob1 and coordinates of a point P1b on a thin line on the right (left) image at time t1 that overlaps with a series of these movement positions. (L, n) is extracted, and-(3) the attention point Poa1 at the to time and the one corresponding candidate point
Attention point Poa1 of to time when Pob1 is assumed to be the same point
And the overlapping point P1b on the right (left) image at time t1 is the corresponding point at time t1 of point of interest Poa1.
From the three-dimensional position of the attention point Poa1 at one time, t
The three-dimensional position of the attention point Poa1 at two times is calculated, and-(4) the center (X2a, Y2a) on the left image at time t2 corresponding to the three-dimensional position of the attention point Poa1 at time t2 The image density of a predetermined area on the left image at time t2 and the right of time t2 around the position (X2b, Y2b) on the right image at time t2 corresponding to the three-dimensional position of the target point Poa1 at time t2 A correlation value C with the image density of a predetermined area on the image is calculated, and corresponding candidate points Pob1, Pob2, Po3b on the right (left) image at time to
For each of the remaining Pob2 and Po3b, the above-(1) ~
The correlation value C is calculated in the same manner as in (4), and the corresponding candidate point Po on the right (left) image at the time to when the correlation value C becomes the maximum is obtained.
One of b1, Pob2, Po3b on the right (left) image of the to time,
It is defined as the corresponding point of the attention point Poa1,
A corresponding point search method for the image captured by the right camera. 2. The left and right imaging cameras photograph scenes in front of them and obtain video signals, respectively, and digitally process these video signals to obtain the same object in the left and right images. In searching for one point, at least the left and right images at the to time, the t1 time after Δt and further at the t2 time after Δt, the left and right images at the to time and the t1 time are the objects of the object in the image. A thinning process is performed to represent the edge as a thin line, and based on the left and right images at times to, t1 and t2, the left (right) of time to
A point with a thin line in the image is determined as the point of interest Poa1, and the time to
The points Pob1, Pob2, Po3b on the right (left) image of the thin line on the corresponding vertical coordinate i of the vertical coordinate i of the point of interest Poa1 are extracted and t
o Let the corresponding candidate points Pob1, Pob2, Po3b on the right (left) image at time t be the coordinates (j, j) of the target point Poa1 on the left (right) image at time t1.
Each point on the thin line of the predetermined area centered on i) is defined as a corresponding candidate point P1a-d to P1a-e on the left (right) image at time t1, and for each of the corresponding candidate points Pob1, Pob2, Po3b , And the point of interest Po
the three-dimensional position of the point of interest Poa1 at the to time determined by the coordinates of a1,
When the point of interest Poa1 moves to each of the corresponding candidate points P1a-d to P1a-e, a series of positions on the right (left) image at time t1 and a thin line on the right (left) image at time t1 Of the point of interest Poa1 at time t1 determined by the coordinates of the intersection P1b of
The three-dimensional position of the point of interest Poa1 at the time t2 is calculated by extrapolating the two-dimensional positions, and the three-dimensional position is converted into coordinates on the left and right images. The correlation value C of the image density on the left and right images at the time is calculated, and the candidate point having the largest correlation value C among the corresponding candidate points Pob1, Pob2, and Po3b is determined as the target point on the left (right) image at the time. A corresponding point search method for images captured by left and right cameras, which is defined as a corresponding point on the right (left) image of the to time of Poa1.