JPH09294280A - Stereoscopic image processing unit - Google Patents

Abstract

PROBLEM TO BE SOLVED: To attain high speed processing by receiving an image alternately from a first half and a latter half of a scanning line with respect to each scanning line of video signals from two cameras. SOLUTION: Cameras A, B are tilted in a direction so that optical axes are in crossing with a front side in the image pickup direction so that an image of an object is picked up to a first half of a scanning line of a pickup image by the camera A and an image of the object is picked up to a latter half of a scanning line of a pickup image by the camera B. The camera A is selected at the start of input for each scanning line and the camera B is selected when the input comes in the middle of the scanning line by a camera selector control section so as to receive an image while the image of the camera A is used for the first half of the scanning line and the image of the camera B is used for the latter half of the scanning line. Two images are entered for an image input time for one image by receiving the image with the arrangement of the cameras as above. Thus, the image input time is decreased, the capacity of the required image memory is reduced and stereo image processing to be conducted after image input is reduced.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a stereo image processing apparatus for performing stereo image processing, and more particularly to a stereo image input method.

[0002]

2. Description of the Related Art As a stereo image input method of a conventional stereo image processing apparatus, a video signal output from a camera is alternately input field by field based on a horizontal / vertical sync signal of a prescribed standard (for example, NTSC). There is a way to do it.

[0003]

However, such a conventional stereo image input method has the following problems. In a method of inputting a stereo image to one image processing apparatus having only one input system from two cameras, in the conventional stereo image processing apparatus, as shown in FIG. The image was input by switching the input alternately. In this case, a time difference for one field and a scanning position are generated between the two captured images, and the total image input time is long because it requires two fields.

Further, in order to eliminate the time difference between the two images, as shown in FIG. 34, one main scanning line (hereinafter, abbreviated as a scanning line unless otherwise specified) is alternated from two cameras. There is also a method of inputting a video signal for one scanning line from one camera, but at the same time, a video signal for one scanning line on the same scanning line is output from the other camera. In addition, the two images stored in the memory become images having a positional shift of one scanning line in the vertical direction of the scanning lines (sub-scanning line direction). Therefore, a larger deviation than the former input method occurs between the two images in the vertical direction of the scanning line on the imaging surface, that is, in the direction perpendicular to the direction in which parallax described below occurs. Since stereoscopic image processing that requires accuracy requires images captured at the same time with no difference in position other than the parallax direction, an accurate stereoscopic image is obtained in the image processing apparatus using the above-described conventional image input method. It is difficult to process.

In addition to these stereo image input methods, by using a camera with a memory for temporarily securing an image for one frame (or one field), a time difference and a position difference other than the parallax direction can be eliminated. There is also a method of inputting two images that do not exist, but with this method, it is necessary to prepare a camera with a memory and the input time is at least 1
There is a drawback that high-speed processing is difficult because it requires frames.

The present invention has been made in view of such conventional problems, and stereo image processing provided with an image input method capable of performing highly accurate stereo image processing while shortening the image input time. The purpose is to provide a device.

[0007]

In order to achieve the above object, in the invention described in claim 1, the first camera and the second camera are arranged side by side so that the optical axis is included in a specific horizontal plane. The object is imaged by the stereo camera, and the video signal of the area including the object obtained from the first camera is obtained with respect to the first half of the scanning line of the video signal imaged by the first and second cameras. On the other hand, to the latter half of the scanning line, the video signal of the area including the object corresponding to the area obtained from the second camera is input,
Stereo image processing is performed based on these input video signals.

According to a second aspect of the present invention, in a stereo image processing apparatus for inputting a stereo image, the first and second cameras are arranged in parallel so that the optical axis is included in a specific horizontal plane. Image pickup means for picking up an image of an object; and the first
And for the first half of the scanning line of the video signal from the second camera, the video signal of the region including the object obtained from the first camera is selected, while for the latter half of the scanning line, Video switching means for selecting and switching a video signal of an area including the object corresponding to the area obtained from the second camera, and image storage means for storing the video signal output from the video signal switching means. ,
Stereo image processing means for performing stereo image processing on the image stored in the image storage means.

According to a third aspect of the present invention, the stereo image processing means calculates distance information to the imaged object on the basis of the image stored in the image storage means. According to a fourth aspect of the present invention, the video switching means includes the video signals from the first and second cameras so that (1) both the first half and the second half of the scanning line include the object. , The case where the first half of the scanning line is switched to the video signal from one camera and the latter half is switched to the video signal from the other camera, and (2) the video signal from either camera over the entire scanning line. Either one of the case of fixing the image to the fixed state and the state of selecting the state to input the video signal from the camera to the image storage means, and performing the image processing on the monocular image in the case of (2). Included means.

According to a fifth aspect of the present invention, the monocular image processing means recognizes the content of the sign ahead of the imaging direction based on the image of only one of the cameras selected by the switching means. did. According to a sixth aspect of the present invention, the first and second cameras are arranged such that the optical axes of the cameras are tilted so as to intersect in front of the camera within the specific horizontal plane.

According to a seventh aspect of the present invention, the first and second cameras are arranged such that the optical axes of the cameras are substantially parallel within the specific horizontal plane, and the video signals of the cameras are An image is picked up so that the object is included in only one of the first half or the second half of the scanning line, and only the video signal of one of the first and second cameras is delayed by a half cycle of the scanning line. The input is made in the image storage means.

According to an eighth aspect of the present invention, the specific horizontal plane is replaced with a specific vertical plane, and the main scanning direction of each camera is made to coincide with the vertical plane. According to a ninth aspect of the present invention, there is provided A / D conversion means for converting the video signals from the first and second cameras into digital signals, and the sampling interval of the video signals by the A / D conversion means is The main scanning direction is set to be shorter than the sub scanning direction of the camera.

According to a tenth aspect of the present invention, the video signal input to the image storage means is restricted with respect to at least one of the horizontal direction and the vertical direction of the picked-up images picked up by the first and second cameras. I decided to do it. Claim 1
In the invention described in item 1, the video signals from the first and second cameras are input to the image storage means for each predetermined number of scanning lines.

According to a twelfth aspect of the present invention, the video signals from the first and second cameras are input to the image storage means by a predetermined number of scanning lines.

[0015]

According to the first aspect of the present invention, as shown in FIG. 1, for each scanning line of video signals from two cameras, images are alternately displayed in the first half and the second half of the scanning line. By capturing, one image (one field)
Since two images can be input from two cameras in a minute input time, the image input time is shortened, the required image memory capacity is reduced, and stereo image processing performed after the image input is performed. Therefore, the processing speed can be increased.

According to the second aspect of the present invention, a video signal from the image pickup means is switched by the video switching means to be recorded in the image recording means, and an image for stereo image processing is generated to generate one image. Since it is possible to input images for two images from two cameras in the input time for (one field), the image input time is shortened and the required image memory capacity is reduced, and the image input is performed after the image input. The stereo image processing can be reduced, and thus the processing speed can be increased.

According to the third aspect of the invention, the input stereo image is used to calculate the distance to the object imaged in the stereo image, whereby the distance is calculated at high speed and with high accuracy. be able to. According to the invention described in claim 4, as shown in FIG. 2, by switching the recording target portions of the images from both cameras, stereo image processing capable of high-speed input and monocular image having a wide-angle imaging range Processing can be selected and performed, and flexible image input can be performed according to the purpose of processing.

According to the fifth aspect of the invention, by recognizing the sign ahead of the imaging direction by using the image picked up from one camera, not only the sign ahead of the imaging direction but also the sign near the camera is recognized. Can also be included in the angle of view for imaging,
More detailed labeling patterns can be detected. Thus, the recognition accuracy of the sign content can be further improved.

According to the sixth aspect of the invention, by tilting the optical axes of the cameras so as to intersect in front of the cameras, it is possible to widen a substantial angle of view with respect to input images from both cameras. Since an object such as a road sign can be photographed at a closer position, the road sign can be identified more accurately. According to the invention described in claim 7, by arranging the optical axes of both cameras in parallel as shown in FIG. 3, the distance measurement error caused by the change of the inclination angle of the optical axes is almost ignored. Since it is reduced to the extent possible, it is possible to substantially eliminate the need to detect the vanishing point, reduce the distance measurement processing, and increase the speed.

According to the invention described in claim 8, by disposing both cameras on a vertical plane as shown in FIG. 3, in a stereo image obtained, a large angle of view in the direction in which the object largely moves. Therefore, when the vehicle ahead of the host vehicle is the target object, the distance to the target object can be accurately obtained. According to the invention described in claim 9, by making the input sampling interval of the video signal from each camera fine as shown in FIG. 4, more detailed image information can be obtained, for example, the distance to the object. It is possible to calculate the distance with high accuracy when measuring.

According to the tenth aspect of the present invention, by limiting the video signal input to the image recording means as shown in FIG. 5, the image memory can be saved and the calculation load of the image processing can be further reduced. Therefore, the processing speed can be increased. According to the eleventh aspect of the present invention, the image memory can be saved by performing the line input in the vertical direction alternately for each line, and
Since the matching process between the images can be performed at a higher speed and the timing of starting the image processing can be advanced, the processing time of the entire image processing including the image input time can be further shortened.

According to the twelfth aspect of the present invention, by not inputting the last several lines of one field, the image memory can be saved and the timing of starting the image processing can be advanced. Therefore, it is possible to further reduce the processing time of the entire image processing including the image input time.

[0023]

DESCRIPTION OF THE PREFERRED EMBODIMENTS Embodiments of the present invention will be described below with reference to the drawings. First, a distance calculation method using a stereo image common to each embodiment will be described. FIG.
Shows a distance calculation method using stereo images when the directions of the cameras are tilted inward so that the optical axes of the cameras intersect on the horizontal plane in the imaging direction of the two cameras. It is a figure. The equation for calculating the distance Z from the camera to the object in this configuration is shown in equation (1). Z = F × D / | (X _B −X _VB ) − (X _A −X _VA ) | (1) where F is the focal length, and D is the distance between the optical axes (eye distance) of each camera. It represents. Further, X _VA is the vanishing point position on the image A by the camera A, X _A is the position of the object on the image A, X _B is the position of the object on the image B by the camera B, and X _VB is the image. It is the vanishing point position on B.

When the optical axes of the two cameras are not parallel as shown in FIG. 6, the vanishing point on the two images and the edge position of the object corresponding to both images are obtained respectively, and on each image. Find the distance from the vanishing point to the corresponding edge position,
The distance Z from the camera to the object can be calculated by the equation (1) using the difference in the distance as the parallax. Next, a method of inputting a stereo image by these cameras will be specifically described. As shown in FIG. 1, first, a direction in which the optical axes intersect in the front of the imaging direction so that the object is imaged in the first half of the scanning line of the image captured by camera A and the second half of the scanning line of the image captured by camera B. Tilt each camera to. Then, with the two cameras arranged in such an inclined state, the camera A is selected at the start of input for each scanning line by the camera selector control unit described later as shown in FIG.
By selecting the camera B at the time when the input comes in the middle of the scanning line, the image is input while switching the image of the camera A in the first half of the scanning line and the image of the camera B in the second half of the scanning line.

By inputting images with such a camera arrangement, two images can be input in the image input time for one image. The output time for one scanning line from the camera at this time is always constant. Even if only the first half of one scan line is input, the input side must wait for the input of the next scan line until the camera finishes outputting the second half of one scan line. This input method makes it possible to reduce the image input time by half by effectively utilizing the waiting time and inputting an image while switching the cameras to be input in the first half and the second half of one scanning line.

However, in this input method, as shown in FIGS. 8 and 9, the angle of view in the horizontal direction of the image pickup surface, that is, the angle of view of an image which can be stereoscopically imaged commonly by two cameras is Although it is half the angle of view of a single camera, if the object is within that angle of view range, the angle of view outside that range is originally an angle of view unnecessary for distance recognition. It is possible to improve the efficiency of processing such as detection of a preceding vehicle.
That is, unnecessary image input to the left half of the camera A and the right half of the camera B is omitted, so that the image input time can be shortened and the image memory can be saved.

The right half of camera A and camera B in FIG.
The part other than the target imaged in the left half of can be used as an image for monocular processing. That is, by switching the input image area according to the processing content, it is possible to selectively perform both the stereo image processing and the monocular image processing by using one set of stereo cameras. For example, FIG.
When the input image area is switched as shown in, input method 1: recognition of the distance to the preceding vehicle in front by stereo image processing input method 2: road shoulder (for example, sign recognition is also possible) and object recognition of adjacent lanes input method 3: Recognition of road shoulder and front Input method 4: It becomes possible to recognize front of adjacent lane. At this time, the stereo cameras are arranged as shown in FIG.
As shown in, it is possible to input an image with a field angle about 1.5 times that of a single-lens camera that uses a lens with the same field angle. Normally, a wide-angle lens is distorted around its periphery, and thus it is difficult to perform image processing. However, in the stereo camera as shown in FIG. 1, since a lens having a standard angle of view such as a standard lens is used, distortion of a captured image is small. That is, by using this stereo camera, stereo image processing capable of high-speed image input can be performed, and monocular processing that requires a wide angle of view can be performed more reliably.

Thus, for example, while recognizing a distance to a preceding vehicle traveling in front of the host vehicle, a vehicle sign for recognizing a road sign diagonally ahead or a traveling vehicle on an adjacent lane as necessary is displayed. Image processing in the forward direction is possible. FIG. 10 is a diagram showing a state in which a video signal output from a camera as an analog signal is digitized. The video signal from the camera when the object is imaged is
An analog signal is output for each scanning line within a fixed time. On the image input side, this analog signal is A / D converted at a specific sampling interval and then input to the image memory.

As described above, the operation of digitizing the image signal for each scanning line and inputting it to the memory is repeated for the number of scanning lines to input one image. Usually, this sampling interval is determined from the aspect ratio of the image pickup surface of the image pickup device such as CCD (Charge Coupled device) of the camera. The aspect ratio of the image pickup surface of a normal CCD is 3: 4.

In the case of NTSC system, within one field
Since 240 scanning lines are input, the image signal for one scanning line is sampled 320 (= 240 × 4/3) times and digitized. Then, the operation of digitizing and inputting the image signal sent from the camera for each scanning line is performed by the number of scanning lines in the vertical direction (240 times in the case of the NTSC system) to input one image. You can For example, by narrowing the sampling interval,
When the number of samplings within one scanning line is doubled (320 times) from the above 320 times, the horizontal resolution is doubled without changing the vertical resolution or the imaging range, as shown in FIG. be able to.

FIG. 12 shows the relationship between the resolution on the image pickup surface and the size of one pixel in the image memory. 1 in image memory
How much a pixel corresponds to the actual image pickup surface is mainly determined by the sampling interval of the image signal for one scanning line, that is, the image signal for one scanning line on the image pickup surface in the horizontal direction. It depends on how many divisions are made into digital. The value of each of the divided image signals becomes the brightness value for each pixel on the image memory.

FIG. 13 is a diagram showing a distance calculation method by the most general stereo image processing. The distance Z from the camera to the object is obtained by the geometrical calculation shown in the equation (2). Z = F × D / | X B -X A | (2) (2) corresponds to the formula of the denominator X _A -X _B is disparity.
For the values of X _A and X _B , find the edge position of the corresponding object between the two images on the digital image stored in the image memory, and convert the position to the position on the imaging surface of the camera. Calculate by Conversion from the position on the image memory to the position on the imaging surface of the camera is performed as follows.

For example, as shown in FIG. 12, if the horizontal width of the image pickup element is 10 mm and the number of samplings is 100 times in one scanning line, the size of one pixel on the image memory corresponds to the image pickup surface. It corresponds to the upper 0.1 mm. From this,
If X _A -X _B is a 5 pixel on the image memory, X _A -X
The value of _B is converted to 0.5 mm. Further, in equation (2), if the focal length F of the camera and the inter-eye distance D (the distance between the optical axes of the two cameras) are fixed values, the accuracy of the distance Z is the parallax X.
It will be determined by the resolution of _A− X _B.

The reason for this will be described with reference to FIG. If the matching result between the two images performed in the image processing shifts by one pixel, the error is small if the size of one pixel on the imaging surface is small, but the error is large if the size of one pixel is large. . For example, the focal length F is 20 mm and the interocular distance D is 10
An image input by sampling the image signal in the horizontal direction 100 times when recognizing an object existing at a distance Z of 1000 mm using a stereo camera with a horizontal width of 0 mm and an image pickup surface of 10 mm. And the accuracy of measurement of the distance Z when the horizontal image signal is sampled 10 times and recognized using an input image.

When an object 1000 mm ahead is recognized, (2)
From the formula, the parallax on the imaging surface is 2 mm (X _A −X _B =
F × D / Z = 20 × 100/1000 = 2). In the case of an image that is sampled 100 times and input, the size of one pixel corresponds to 0.1 mm on the imaging surface, so the parallax on the digital image stored in the memory is 20 pixels. On the other hand, in the case of an image sampled 10 times and input, the size of one pixel corresponds to 1 mm on the imaging surface, so the parallax is two pixels.

Here, the parallax is 1 in each image.
Considering the case of pixel shift, in the case of 100 samplings, the parallax is 19 pixels, so the distance is 1052 mm {= 20 × 10
0 / (19 × 0.1)}, and in the case of sampling 10 times, the parallax is 1 pixel, so the distance is 2000 mm {= 20 × 100 /
(1 × 1)}. In this way, the former error is 52 mm
On the other hand, the latter error reaches 1000 mm.

Therefore, it is desirable to set the sampling interval in the horizontal direction in which the parallax, which has a great influence on the distance measurement accuracy, to be set more finely, and this affects other portions related to the input such as the image input time and the image pickup range. The accuracy of the measured distance can be improved without affecting. Generally, in stereo image processing, in order to improve the resolution in the parallax direction when higher accuracy is required, from the angle of view θ _{1 of the} lens shown in FIG. 15 (a) to a small lens as shown in FIG. 15 (b). there are countermeasures to set the angle of view theta _2, but there is a problem that the imaging range becomes narrower. Therefore, in the stereo camera having the arrangement shown in FIG. 1, since the direction in which parallax occurs is aligned with the horizontal direction of the image pickup surface, the video signal output from the camera for each scanning line is more finely sampled. By digitizing in this way, the resolution in the parallax direction can be improved without changing the resolution in the direction perpendicular to the parallax direction. In this method, it is not necessary to replace the lens of the camera and the imaging range does not change. Further, the input time of one scanning line is unchanged, and the input time of the entire image is also unchanged because the number of input scanning lines is constant. That is, it is possible to obtain a stereo image in which the distance can be accurately calculated without changing the imaging range or the input time.

Therefore, if the object to be recognized in the stereo image processing does not require a wide angle of view in the parallax direction,
In addition to the method of inputting while switching the inputs of the two cameras in the first half and the second half of the horizontal scanning line, by setting the sampling interval finely, a stereo image can be input at high speed, and the stereo image can be accurately distanced. Can be asked.

If the edge to be recognized has a sufficient length in the direction perpendicular to the parallax direction, the images are thinned out to the extent that the edges of the images from the cameras can be matched. However, it is possible to recognize the object and the distance. For example, by inputting an image for each scanning line and thinning out the image, and switching the input of two cameras in the former and latter half of the scanning line, the scanning line at the same position with respect to two captured images. Can be input on one scan line, which saves image memory and enables faster image input for stereo image processing. In addition, because the image data is thinned out and input,
Since the calculation amount of matching between two images for calculating parallax is also reduced, the time required for the entire image processing can be shortened.

FIG. 16 (a) is a timing chart showing the timing of image input and image processing start when input is made for each scanning line while switching the output from each camera. When all the scanning lines are input, the image data shown in FIG. 16 (b) will be input to the image memory. However, by inputting the image data for each scanning line and thinning out the image, the image data shown in FIG. As shown in, the amount of input image data is halved.

Normally, the image signal from the camera is output based on the horizontal / vertical synchronizing signals input to the camera according to a predetermined standard. Specifically, at the timing at which the camera starts image output, the camera waits for a vertical synchronization signal indicating the start of output, and outputs the image signals for the specified number of scanning lines from the time when the vertical synchronization signal is received. On the other hand, on the input side of the image output from the camera, it is possible to stop the input of the image signal from the camera on the way. In this case, since the image input side can perform processing other than input (for example, image processing) when the input is stopped, not only the image input time but also the entire image processing time can be shortened. .

Next, a first embodiment in which the method for inputting the stereo image is used for detecting a preceding vehicle, calculating a distance, and recognizing a sign will be described. FIG. 17 shows the configuration of the stereo image processing apparatus according to this embodiment. Here, the outline will be described first. The optical axes of the two cameras A and B that output the captured optical image as an electrical image signal are included in a plane parallel to the road surface so that parallax can be generated in the horizontal direction of the camera imaging plane. , Are arranged so as to be inclined so that the optical axes intersect each other in the front in the imaging direction. The video signals output from these cameras are input to the camera selector control unit 171, and the camera selector control unit 171
In 171 only the video signal from one of the cameras is selected and output to the A / D converter 172. The A / D converter 172 digitizes the input video signal at a predetermined sampling interval based on the reference clock signal generated by the reference clock generator 173.

Then, the HD signal generation unit 174 outputs the horizontal video signal output start timing to each of the two cameras every time the reference clock generated by the reference clock generation unit 173 is sent by the output of one scanning line. Output a horizontal sync signal. Also, every time the VD signal generator 175 sends the horizontal synchronizing signal for the number of scanning lines to be output within one field, a vertical synchronizing signal for notifying the output start timing of the image for one field is output to each of the two cameras. To do.

Based on these synchronization signals, a total of two digital images obtained by digitizing the video signals output from the cameras A and B for each reference clock are stored in two image memories.
The memory 176a, 176b temporarily stores the write permission signal to the image memories 176a, 176b and the designation of the address position on the memory by the image memory controller 176 for each reference clock.

Further, the object detecting section 177 obtains the edge position of the object from the image stored in either one of the two image memories, and finds the edge corresponding to the obtained edge of the object by the corresponding point searching section. 178 to search from the other image. Then, based on the coordinates of the range of highest similarity between the two images, the focal lengths of the two cameras, and the positional relationship between the two cameras, distance calculation unit 179 calculates the distance from the cameras based on the principle of triangulation. Find the distance to the object.

FIG. 18 is a flowchart showing a process of inputting a stereo image using the stereo image processing apparatus having the above-mentioned configuration, detecting the preceding vehicle, calculating the distance to the preceding vehicle, and recognizing the sign. In this embodiment, normally, the stereo image is input using the input method 1 shown in FIG. 2 (a), the distance to the preceding vehicle in front is recognized, and if necessary, the input shown in FIG. 2 (c) is performed. According to the method 3, the input image is switched to only one of the cameras and the image from the selected camera is used to recognize the road sign.

FIG. 19 shows the arrangement of the vehicle equipped with the camera, the preceding vehicle (object) and the camera, and the set absolute coordinate system. The two cameras A and B are installed such that the cameras are inclined so that the preceding vehicle appears in the first half of the image captured by the camera A in the horizontal direction and the second half of the image captured by the camera B in the horizontal direction. . Using the stereo camera having such a configuration, the distance to the preceding vehicle is calculated based on the distance recognition method using the stereo image described in FIG.

First, a distance recognition method by stereo image processing will be described. FIG. 20 is a diagram for explaining how to obtain the vanishing point which is the point at infinity. Here, a straight line expression on the picked-up image of the white lines existing on the left and right of the vehicle equipped with the stereo camera is obtained, and these intersections are set as vanishing points (step 1, hereinafter referred to as S1). This is because at the position close to the stereo camera on the captured image, the left and right white lines exist at positions distant from each other, but the farther the white line is, the narrower the white line interval becomes, and the two white lines intersect at infinity. Therefore, the position where the white lines intersect becomes the vanishing point.

Next, a method for detecting the left and right white lines will be described.
In general, create a window in the white line part in the captured image that can be calculated from the position and focal length of the attached stereo camera, and use the edge image obtained by subjecting this window to differential processing, for example, a straight line Targeted for detection
By the Hough transform, a straight line expression representing a white line in the window is obtained (S2).

A specific method of obtaining the window position will be described with reference to FIG. Since the window position may be an approximate position where a white line is likely to be detected, the minute tilt angle of each camera is ignored. Now, considering that a camera A having a lens with a focal length of F is arranged at a height h from the road surface and an image is taken by the camera A, a white line existing at a position Z'forward in the image pickup direction is considered. Can be calculated when imaged at a position of h · F / Z ′ in the X _C axis direction from the center of the imaging surface. Also, for the position in the Y _C axis direction, the road width is W,
Assuming that the camera A is located substantially in the center of the road, it can be calculated that a white line is imaged at a position of 2W · F / Z ′ from the center of the imaging surface. The conversion of this position into the position on the image memory can be obtained if the size for one pixel on the imaging surface is known.

The position of the window for detecting the white line is determined by such a method, the linear expression of the left and right white lines is obtained at that position, and the vanishing point (X _VA , Y) for the image A from the camera A is calculated from these intersections. _VA ). Similarly, with respect to the image B from the camera B, the vanishing point (X _VB , Y _VB ) is similarly obtained. Since these vanishing points are determined by the mounting position of the camera, they need only be obtained once at the time of initial setting.

Next, the presence or absence of a sign whose details will be described later is judged (S3), and when it is judged that there is no sign, the position of the preceding vehicle is detected (S4). Since the preceding vehicle can be determined to have a long horizontal edge at a position sandwiched between the two white lines, the horizontal edge is detected by using one of the images A and B. To detect the position of the preceding vehicle, first, a white line is detected by a method similar to the detection of the vanishing point, and the horizontal edges existing between the detected white lines are detected from the bottom on the screen. Search upwards. As shown in FIG. 22, when the horizontal edge is equal to or larger than a certain threshold and has a certain length, the horizontal edge is recognized as the horizontal edge of the preceding vehicle, and the preceding vehicle is detected.

Next, how to find the corresponding points between the two images A and B will be described with reference to FIG. This sets a window of a predetermined size at the position of the preceding vehicle in the detected image A, and performs matching processing using the image in this window as a template (S5). That is, the position of the image having the highest degree of similarity with this template is obtained from the image B. The calculation of the degree of similarity at this time may be performed using, for example, the normalized correlation method. The brightness value of each pixel of the template extracted from the image A is A _ij , and the brightness value of the image extracted from the image B is B _ij .
_{If ij} , the degree of similarity between the images by the normalized correlation method can be calculated by the equation (3).

[0054]

[Equation 1]

The higher the value of equation (3), the higher the degree of similarity. Therefore, the position of the image to be extracted from the image B is changed, and the similarity between the template and the image at each position of the image B is repeatedly calculated using the equation (3), and as shown in FIG. A position X _B having a high degree of similarity is obtained.
Then, the parallax required for distance calculation, the distance between the position X _A of the window set in the vanishing point X _VA and image A of the image A and (X _VA -X _A), a vanishing point X _VB of the image B the difference between the distance between the position X _B of the window position with the highest similarity as determined by the normalized correlation method _{(X VB -X B) (X} VB -X B) -
Determined by the absolute value of _{(X VA -X A) (S6} ).

Since this value is in pixel units, the length on the image pickup surface corresponding to one pixel is calculated based on the size of the image pickup surface of the camera used and the horizontal sampling interval, and based on this length. The parallax obtained in pixel units is converted into the length on the image pickup surface. After calculating this parallax, the distance to the preceding vehicle is calculated using the equation (1) (S7).

FIG. 25 is a time chart comparing the stereo image input method according to the present embodiment with the conventional input method. In the normal image input method, the image A is input in the first field of one frame 1 and the image B is input in the second field. Then, in the first field of frame 2, processing of the input images A and B is repeated.

On the other hand, in this embodiment, the images A and B are input in the first field of the frame 1 and the input images A and B are processed in the second field repeatedly. As a result, the image input time can be halved, and if the image processing time is one field or less, the processing on the input image can be completed within one frame. Also, by using the stereo camera with this arrangement, the horizontal direction of the image pickup surface is made substantially coincident with the parallax direction, and the sampling interval at the time of digitizing the image signal is narrowed, so that the object (for example, the preceding vehicle) The distance can be calculated accurately.

Next, the sign recognition will be described with reference to FIG. FIG. 26 illustrates a method of recognizing the presence of a sign from an image for stereo image processing. This sign recognition is performed, for example, only when the sign is detected, and normally distance recognition is performed. As a result, it is possible to perform sign recognition while suppressing the processing speed of distance recognition from decreasing as much as possible.

An input image in the normal state in which no marker is present is an image for stereo image processing as shown in FIG. 26 (a), for example. The image for distance recognition is as shown in Fig. 26 (b).
Since only half the angle of view of one camera is used, it is not possible to recognize a sign near the vehicle (oblique sideways), but it is possible to recognize the presence of a sign ahead. Also,
By using a wide-angle lens to widen the angle of view, you can image from a distant sign to a sign near your vehicle, but using a wide-angle lens causes distortion in the captured image due to the lens, so accuracy is increased at the image processing stage. It is not preferable because it is difficult to calculate the distance well.

This sign is picked up as if it were on the white line of the road surface in the picked-up image. Based on the white line detection result detected during the stereo image processing described above, FIG.
As shown in (c), a white line position (y coordinate) at the left end of the image is obtained, and a window for sign recognition having a predetermined size is set at the white line position. When it is confirmed that a long vertical edge exists in this window, it is judged that there is a sign. The image captured in the image memory at this time is shown in FIG.
Even if the presence of the label is detected, the imaged label is too small to judge the content of the label as shown in FIG. Then, after the presence of the sign is detected and the host vehicle travels a few meters, the input target is switched to only the camera B that inputs diagonally forward left,
An image is input once by the input method 3 shown in (c) (S8).
An example of the image input at this time is shown in FIG. 27 (a).
A large image of the sign is captured in this image, as shown in FIG. 26 (d).
Since it is possible to obtain sufficiently detailed information from the image shown in (4), it is suitable for recognizing the contents of the sign.

In order to recognize the contents of the sign from the captured image shown in FIG. 27 (a), first, a circle is detected from the captured image (S
9). This can be detected, for example, by using the Hough transform with a circle as a recognition target. The contents of the sign are the image inside the circle shown in Fig. 27 (b) and Fig. 27 (c).
Then, the degree of similarity with the template representing the previously prepared sign content is calculated, and the template with the highest degree of similarity is determined as the sign content (S10). The degree of similarity can be obtained by, for example, the above-described normalized correlation method.

As described above, since the stereo camera according to the present embodiment images the diagonally forward direction, it is possible to recognize a distant sign and a large image of a sign near the own vehicle, which is more reliable. Mark recognition can be performed. Next, a second embodiment will be described in which, when only the distance to the preceding vehicle is recognized by stereo image processing, the distance to the preceding vehicle is surely recognized at a higher speed.

FIG. 28 shows an own vehicle equipped with a stereo camera, a preceding vehicle (object), an arrangement of stereo cameras, and a world coordinate system set for explaining them.
This stereo camera arrangement is obtained by rotating the arrangement of the stereo cameras in the first embodiment by 90 degrees so that the horizontal direction of the imaging surface is perpendicular to the road surface.
Therefore, the two cameras A and B are respectively tilted so that the preceding vehicle appears in the horizontal first half of the image captured by the camera A and the horizontal second half of the image captured by the camera B. Then, the recognition of the preceding vehicle is obtained by the same method as described above using the stereo image rotated by 90 degrees.

Next, the effect of rotating the camera by 90 degrees will be described. In the world coordinate system shown in FIG. 28,
The preceding vehicle often has a long edge parallel to the direction parallel to the road surface (X-axis direction). Therefore, since it is easier to match the two parallaxes with the edge as long as possible, it is better to make the parallax in the direction perpendicular to the road surface (Y-axis direction). Further, although the preceding vehicle may move largely in the X-axis direction, but not in the Y-axis direction, the angle of view in the Y-axis direction may be narrow.

By the way, in the stereo image according to the first embodiment, the horizontal field angle of the image pickup surface is ⅔ of the vertical field angle. Therefore, even if the angle of view is narrow in the Y-axis direction, it is sufficient for distance recognition. Therefore, in the present embodiment, the horizontal direction of the imaging surface is aligned with the Y-axis direction. That is, as shown in FIG. 28, the stereo camera is arranged at a position rotated by 90 degrees. A feature of the movement of the preceding vehicle is that it tends to move greatly in the left-right direction with respect to the vehicle. With the arrangement as in this embodiment, the direction in which the preceding vehicle moves greatly (the left-right direction) and the direction with a wide angle of view Since and can be matched, it becomes possible to more reliably recognize a preceding vehicle moving in a wide range.

The edge of the vehicle is long in the left-right direction, and parallax calculation can be performed using the long edge, so that the accuracy and certainty of recognizing the preceding vehicle are improved. Further, since the preceding vehicle is considered to have a somewhat long edge, it is possible to sufficiently match the two images even if the captured images are thinned out. Therefore, using the method shown in FIG. 16, the scanning lines at the same position between the two cameras are input, for example, every other scanning line to thin out the images, and a part of the preceding vehicle remains in the imaging range. When such a portion is input, the input of the image is restricted by forcibly stopping the input of several scanning lines in the latter half of the field.

Since such an image can be input in the preceding vehicle recognition, the image memory can be saved and the image input time can be shortened. Further, when such an image is input, the image in the memory is compressed in the direction perpendicular to the parallax direction as shown in FIG. 29, and thus the size of the template used when matching two images is performed. Becomes smaller, the amount of calculation of the degree of similarity once decreases, and the matching processing can be performed at high speed. That is, not only the image input time but also the recognition processing time can be shortened.

FIG. 30 shows a time chart when the image input method according to the present embodiment is used. The conventional method is the same as the method shown in FIG.
And the images are input by the cameras A and B in the second field and the processing of the image input in the first field of the frame 2 is repeated. On the other hand, in the input method (1) in the present embodiment, the images from the cameras A and B are input in the first field of the frame 1 and the input of the unnecessary portion of the latter half of the image is forcibly stopped. This allows the input time to be less than one field. Then, in the second field, the first
Process the image entered in the field. This is repeated. By using such an input method (1), the image input time can be suppressed to less than half that of the conventional method.
Also, if the processing time of the input image is one field or less, processing within one frame is also possible.
As shown in (2), by stopping the input of the latter half of the image or inputting while thinning out the image, the input is aborted in the middle of the original image input time, and the image processing time is Real-time processing is possible if it can be done within the waiting time.

As described above, after narrowing the sampling interval of the image signal by using the stereo camera tilted by 90 degrees,
By inputting while switching the camera in the first and second halves of the horizontal scanning line, and by inputting a stereo image using a method of thinning out the image in the vertical direction of the imaging surface, distance recognition for the preceding vehicle is high speed and high. Can be done with precision.

Next, a third embodiment in which the optical axes of two cameras are arranged in parallel will be described. The configuration of this embodiment is shown in FIG. The configuration according to the present embodiment generates a horizontal synchronization signal that is delayed by a half cycle from the horizontal synchronization signal from the HD signal generation unit 174a having the same function as the HD signal generation unit 174 of the configuration according to the first embodiment. The delayed HD signal generation unit 174b is provided.

Specifically, as shown in FIG. 3, images are picked up so that the object whose distance is to be measured is projected in the first half of the scanning lines of the two cameras, and one of the cameras picks up the HD signal generator. A normal image is output by the horizontal synchronization signal generated by the 174a, and when the input of one scanning line of the camera comes to the middle, the image processing unit side sends a signal to the other camera to start the horizontal output. . As a result, even in a stereo camera in which the optical axes of the cameras are arranged in parallel, two images can be input in one image input time.

Even when the two cameras are arranged so as to be included in the horizontal plane, the same processing can be performed by capturing the object in the first half of the scanning line of the cameras. Here, when the image input methods of the first and second embodiments are used, since the optical axes of the cameras are inclined inward so as to intersect in the front in the imaging direction, the vanishing point is lost by the stereo image processing. Needs to be calculated.

In the preceding vehicle recognition described above, it is possible to find the position of the vanishing point based on the white line on the road surface, but it may be difficult to find the position of the vanishing point depending on the processing target. In such a case, the image input method according to the present embodiment is effective. Further, as the effect of installing the cameras in parallel, not only is it unnecessary to detect the vanishing point, but there are the following advantages.

That is, in the stereo image processing when the camera is tilted, when the camera is fixed, the vanishing point is obtained at the time of initial setting, so that distance recognition can be performed with the same accuracy as when the cameras are parallel. However, when the camera is mounted on a traveling car and the angle of the camera moves in the parallax direction, the accuracy of distance measurement may be slightly reduced. The principle of the decrease in accuracy will be described with reference to FIG. For the sake of simplicity, the optical axis 321 of one camera is
Is the same position as the line connecting the vanishing point 322 and the lens center 323. The position of the vanishing point changes when the optical axis 321 of the camera of this arrangement is inclined by α degrees with the lens center 323 as the center of rotation. For example, in the preceding vehicle recognition, vanishing points X _VA and X _VB are determined at the time of initial setting, and it is assumed that the position of these vanishing points is constant. However, when calculating the distance using the equation (2), if the difference X _VB −X _VA between the two vanishing points changes, the distance recognition result will be affected. That is, FIG. 32 (a)
In the case of FIG. 32, the vanishing point difference X _VB −X _VA becomes F · tan θ, and in the case of FIG. 32 (b), the vanishing point difference X _VB −X _VA is F · tan (θ−α) + F.・ Because tan α, the difference in vanishing points changes as shown in Eq. (4) when the camera tilts α degrees. F ・ tan θ- (F ・ tan (θ-α) + F ・ tan α) = F ・ tan (θ-α) ・ tan θ ・ tan α (4) In the case of preceding vehicle recognition, α is usually small (0.1 degree). Since the distance from the lens to the object is sufficiently larger than the interocular distance, θ is not a large value. For example, if the camera is set so that a preceding vehicle ahead of about 1 m is projected at an eye distance of 100 mm, the angle θ formed by the optical axes of the two cameras is about 6 degrees (tan 6 ° = 100 / 1000). Assuming that F is 10
In mm, the shift in parallax due to the movement of the α degree camera is about 0.00002 mm. Now, assuming that the size of the image pickup surface is 4.8 mm and one scanning line is sampled at 256 pixels, the size of one pixel corresponds to 0.018 mm. In other words, 0.000002mm which is the difference of this parallax
This is equivalent to 0.001 pixels.

From this fact, if the preceding vehicle is recognized with the above setting, the change in the difference of the vanishing points due to the change of the installation angle of the camera is a negligible value, and thus it is not necessary to obtain the vanishing point every time. Absent. However, it is necessary to obtain the vanishing point every time for the processing that the error of about 0.001 pixels has a great influence. As described above, according to this embodiment, even when the optical axes of two cameras are arranged in parallel, a stereo image having the advantages shown in the first and second embodiments is provided. In addition, it is not necessary to detect the vanishing point, and when the camera moves like a traveling vehicle, distance recognition can be performed with higher accuracy than when the camera is tilted.

[Brief description of drawings]

FIG. 1 is a diagram for explaining an outline of the invention according to claim 1.

FIG. 2 is a diagram showing four patterns of a method of switching a video signal and inputting it to an image recording means.

FIG. 3 is a diagram for explaining the outline of the invention according to claim 6;

FIG. 4 is a diagram for explaining an outline of the invention according to claim 7;

FIG. 5 is a diagram for explaining the outline of the invention according to claims 8 to 10.

FIG. 6 is a diagram illustrating a stereo image processing method for basic distance calculation.

7A and 7B are views for explaining switching between camera inputs in the first half and the second half of a scanning line for inputting.

FIG. 8 is a diagram showing an angle of view of one camera and an angle of view when the two cameras are tilted.

FIG. 9 is a diagram showing a stereo image area for an image from each camera.

FIG. 10 is a diagram illustrating a state of an image with respect to a sampling interval.

FIG. 11 is a diagram illustrating an image obtained by converting an analog image signal into a digital image signal.

FIG. 12 is a diagram showing the resolution on the imaging surface and the size of one pixel in the image memory.

FIG. 13 is a diagram showing a distance calculation method by general stereo image processing.

FIG. 14 is a diagram showing the relationship between the measurement accuracy of the distance Z and the resolution of parallax when the focal length F and the interocular distance D of the camera are constant.

FIG. 15 is a diagram showing a relationship between an imaging range and measurement accuracy of an object.

FIG. 16 is a diagram showing a relationship of image input timing from each camera and an image input at that time.

FIG. 17 is a block configuration diagram of a stereo image processing apparatus according to the first embodiment.

FIG. 18 is a flowchart of distance calculation and sign recognition processing to a preceding vehicle in the first embodiment.

FIG. 19 is a diagram showing an imaging method according to the first embodiment and an image obtained at that time.

FIG. 20 is a diagram illustrating a method of calculating a vanishing point.

FIG. 21 is a diagram showing a mounting position of a camera for explaining a method of setting a window position.

FIG. 22 is a diagram illustrating a method of detecting the position of a preceding vehicle.

FIG. 23 is a diagram illustrating how to find corresponding points between two captured images.

FIG. 24 is a diagram showing an image cutout position with the highest degree of similarity.

FIG. 25 is a timing chart showing the timing of image input and image processing according to the first embodiment.

FIG. 26 is a diagram illustrating a method of recognizing a sign.

FIG. 27 is a diagram illustrating a method of recognizing the content of a sign when recognizing the sign.

FIG. 28 is a diagram showing an imaging method according to the second embodiment.

[Fig. 29] Fig. 29 is a diagram illustrating processing in the case where many long edges are included in the direction in which parallax is detected.

FIG. 30 is a timing chart showing the timing of image input and image processing according to the second embodiment.

FIG. 31 is a block configuration diagram of a stereo image processing device according to a third embodiment.

[Fig. 32] Fig. 32 is a diagram for describing that the distance measurement accuracy decreases when the camera angle changes in the parallax direction.

FIG. 33 is a diagram illustrating a conventional image input method.

FIG. 34 is a diagram illustrating another conventional image input method.

[Explanation of symbols]

 171 Camera selector control unit 172 A / D conversion unit 174,174a, 174b HD signal generation unit 175 VD signal generation unit 176 Image memory 177 Target object detection unit 178 Corresponding point search unit 179 Distance calculation unit

[Procedure amendment]

[Submission date] June 21, 1996

[Procedure amendment 1]

[Document name to be amended] Statement

[Correction target item name] 0054

[Correction method] Change

[Correction contents]

[0054]

[Equation 1]

Claims (12)

[Claims] 1. An object is imaged by a stereo camera in which a first camera and a second camera are arranged side by side so that an optical axis is included in a specific horizontal plane, and the first and second cameras take images. The video signal of the area including the object obtained from the first camera is input to the first half of the scanning line of the video signal, while the second camera is input to the second half of the scanning line. A stereo image processing apparatus, wherein a video signal of an area including the object corresponding to the obtained area is input, and stereo image processing is performed based on the input video signal. 2. A stereo image processing apparatus for inputting a stereo image, wherein image pickup means for picking up an object by arranging first and second cameras in parallel so that an optical axis is included in a specific horizontal plane. With respect to the first half of the scanning line of the video signal from the first and second cameras, the second half of the scanning line is selected while selecting the video signal of the region including the object obtained from the first camera. With respect to, the video switching means for selecting and switching the video signal of the area including the object corresponding to the area obtained from the second camera, and storing the video signal output from the video signal switching means And a stereo image processing means for performing stereo image processing on the image stored in the image storage means.
The stereo image processing device according to. 3. The stereo image processing means calculates the distance information to the imaged object on the basis of the image stored in the image storage means. Stereo image processing device. 4. The video switching means outputs the video signals from the first and second cameras so that (1) the object is included in both the first half and the second half of the scan line. When switching the first half to the video signal from one camera and switching the second half to the video signal from the other camera, and (2) when fixing the video signal from either camera over the entire scanning line. One of the two states is selected so that the video signal from the camera is input to the image storage means, and a monocular image processing means for performing image processing on the monocular image in the case of (2) is included. The stereo image processing apparatus according to claim 1, wherein the stereo image processing apparatus is a stereo image processing apparatus. 5. The monocular image processing means is for recognizing the content of a sign ahead of the imaging direction based on the image of only one of the cameras selected by the switching means. Stereo image processing device. 6. The first and second cameras are arranged so as to be tilted so that the optical axes of the cameras intersect in the front of the camera within the specific horizontal plane. Stereo image processing device according to item 3. 7. The first and second cameras are arranged such that the optical axes of the cameras are substantially parallel in the specific horizontal plane, and the first half or the second half of the scanning line of the video signal of each camera. Of the first camera and the second camera, the video signal of one of the first and second cameras is delayed by a half cycle of the scanning line and input to the image storage means. The stereo image processing apparatus according to claim 1, wherein 8. A specific vertical plane is used instead of the specific horizontal plane, and the main scanning direction of each camera is made to coincide with the vertical plane. The stereo image processing device according to any one of the above. 9. An A / D conversion means for converting the video signals from the first and second cameras into a digital signal, wherein the sampling interval of the video signal by the A / D conversion means is
9. The stereo image processing apparatus according to claim 1, wherein the main scanning direction is set shorter than the sub scanning direction of the camera. 10. A video signal input to said image storage means is restricted in at least one of a horizontal direction and a vertical direction of a picked-up image picked up by said first and second cameras. ~ The stereo image processing device according to claim 9. 11. The stereo image processing apparatus according to claim 10, wherein the video signals from the first and second cameras are input to the image storage means for each predetermined number of scanning lines. 12. The stereo image processing apparatus according to claim 10, wherein the video signals from the first and second cameras are input to the image storage means by a predetermined number of scanning lines. .