JPH11325890A - Picture correcting apparatus for stereoscopic camera - Google Patents

Abstract

PROBLEM TO BE SOLVED: To correct positional deviation due to the secular change for optical positional deviations of pictures taken by a stereo camera. SOLUTION: An analog picture taken by a stereo camera 1 is A/D-converted, coordinate correction data of the positional deviation due to the secular change detected by reference markers in a microcomputer 50 for picture data stored in a picture memory 14 are added to horizontal and vertical addresses through a coordinate correction value computing circuit 19, and data of two upper and lower pixels in the vertical direction are accessed in two steps by a two-step access circuit 20 to complete mutual reference positions of a reference picture and comparison picture in the stereo matching by a data completion circuit 21, whereby the effect of the positional deviation due to the secular change is eliminated.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image correcting apparatus for a stereo camera for correcting an optical positional shift of an image picked up by a stereo camera.

[0002]

2. Description of the Related Art In general, as a three-dimensional measurement technique using images, a correlation between a pair of images obtained by imaging an object from different positions with two cameras (stereo cameras) is obtained, and a stereo camera is attached based on parallax for the same object. 2. Description of the Related Art There is known image processing by a so-called stereo method in which a distance is obtained by the principle of triangulation using camera parameters such as a position and a focal length.

In image processing by the stereo method, two image signals from a stereo camera are sequentially shifted and superimposed to obtain a position where the two image signals coincide with each other. It is desirable that only the displacement of the corresponding position resulting from the parallax exists,
If there is a misalignment due to other optical distortion or the like, mismatching occurs and the accuracy of the distance information decreases.

For this reason, when using a stereo camera, it is very important to adjust the optical position.
Japanese Patent No. 7557 discloses that a holding member for connecting and holding a pair of video cameras adjusts the arrangement of pixels of the image sensor of one video camera so as to be parallel to the arrangement of pixels of the image sensor of the other video camera. Adjusting means, an optical axis adjusting member for adjusting the optical axis of one video camera to be parallel to the optical axis of the other video camera, and mechanically adjusting and holding the correlation between the two cameras Is disclosed.

However, once the fixed stereo camera is displaced due to aging, it has conventionally been necessary to readjust the mechanical structure, which requires not only a complicated operation but also a readjustment for the readjustment. The time is also long, and there is a limit in ensuring accuracy in mechanical adjustment.

[0006] To address this, the applicant has first
Japanese Patent Application No. Hei 9-117268 proposes a technique for performing affine transformation on an image in accordance with a shift in the optical position of a stereo camera, thereby making electrical adjustment without mechanical adjustment. The optical position can be precisely adjusted to a level that is difficult to adjust mechanically, and a deviation due to aging after the adjustment can be easily readjusted.

[0007]

However, the correction of the image proposed by the present applicant is performed off-board, and when adjusting the shift due to the aging of the stereo camera, the vehicle must be serviced. There was the complication that they had to be brought to a factory or the like for inspection.

The present invention has been made in view of the above circumstances, and makes it possible to correct on-board a linear positional deviation due to aging with respect to an optical positional deviation of an image captured by a stereo camera, thereby improving the reliability of stereo processing. It is an object of the present invention to provide a stereo camera image correction device capable of improving the performance.

[0009]

According to the first aspect of the present invention,
An image correction device for a stereo camera, which corrects an optical positional shift of an image captured by a stereo camera having two cameras arranged at a set interval, wherein each of the two cameras has a field of view. Means for storing the initial position of the reference marker set in each captured image, and correcting the position shift of the entire image according to the shift amount when the position of the reference marker in the image is shifted from the initial position. It is characterized by having.

That is, according to the present invention, in order to cope with a positional shift due to aging of a pair of images taken by a stereo camera, each pair of images of a reference marker set in the field of view of each of the two cameras is used. Memorize the initial position of
When the position of the reference marker in the image deviates from the initial position,
The position shift of the entire image is corrected according to the shift amount.

[0011]

Embodiments of the present invention will be described below with reference to the drawings. 1 to 9 relate to an embodiment of the present invention, FIG. 1 is an overall configuration diagram of a stereo processing system, FIG. 2 is an explanatory diagram showing a vertical / horizontal correction pattern, and FIG. FIG. 4 is an explanatory diagram showing
Is an explanatory diagram showing a correction pattern for focal length correction, FIG. 5 is an explanatory diagram showing a lens distortion correction pattern, FIG. 6 is an explanatory diagram showing data complementation in a vertical direction, and FIG. 7 is an explanatory diagram showing a captured image of a horizontal lattice pattern. FIG. 8 is an explanatory view schematically showing an example of the data distribution of the coordinate correction table, and FIG. 9 is an explanatory view showing an image obtained by correcting the coordinates of the original image.

FIG. 1 shows a stereo image processing system for processing images picked up by a stereo camera 1 which is synchronized with each other and includes two cameras with variable shutter speeds, each of which incorporates an image sensor such as a charge-coupled device (CCD). Is mounted on a moving body such as an automobile or a helicopter, and is used as a part of a system for recognizing the surrounding environment and a self-position.

The stereo camera 1 includes a main camera that captures a reference image when one of the cameras performs stereo processing.
The other camera is provided with a predetermined base line length on a camera stay (not shown) as a sub camera that captures a comparative image in stereo processing. 2 captured by both cameras 2 and 3
The images are input to a stereo processing device 40 via an image correction device 10 which is an image input device, and the degree of coincidence between the two images is evaluated by stereo matching to obtain three-dimensional image information obtained by digitizing perspective information. A range image is generated.

In this case, the two cameras 2 and 3 are mechanically adjusted so that their optical axes are parallel to each other at the initial manufacturing stage and fixed to the camera stay.
The images captured in the above are optically affected by the limitations of mechanical assembly accuracy, the effects of lens focal length variations and lens distortion, and differences in the scale of the captured image due to the tilt of the light receiving surface of the image sensor. There is considerable non-linear positional deviation, and an image optimal for stereo processing is not always obtained.

For this reason, in the image correction apparatus 10, the optical position shift in the images picked up by the cameras 2 and 3 is initially corrected for each pixel so that image data optimal for stereo processing can be obtained. Further, in order to cope with deviation of the optical positions of the cameras 2 and 3 due to aging, that is, distortion of the camera stay and deviation of the frame and lens positions of the cameras 2 and 3, a microcomputer (microcomputer) 5 is used.
0 is used to correct the initial correction.

That is, the image correction device 10 converts analog images from the cameras 2 and 3 into A / D converters 11 and 12.
Data bus switching circuit 13
Is stored in the image memory 14 via the
Is corrected by the data complement circuit 21 for each pixel.

A reference / comparison image write address generation circuit 16 and a reference / comparison image read address generation circuit 17 are connected to the image memory 14 via an address switching circuit 15 to generate a reference / comparison image read address generation circuit. The circuit 17 accesses a reference / comparison image coordinate correction table 18 that stores coordinate correction data for each pixel as an initial value.

To the horizontal address output from the reference / comparison image read address generation circuit 17, coordinate correction data from a coordinate correction value calculation circuit 19 for calculating an image coordinate correction value according to an instruction from the microcomputer 50 is added. Are output to the address switching circuit 15. The reference / comparison image read address generation circuit 17
The coordinate correction data from the reference / comparison image read address generation circuit 17 and the coordinate correction data from the coordinate correction value calculation circuit 19 are added to the vertical address output from With 2
Address switching circuit 15 via the stage access circuit 20
Is output to

The output of the data complement circuit 21 is based on a reference
The output of the shading correction table 22 accessed from the comparative image read address generation circuit 17 is input to the luminance correction result table 23 together with the luminance correction, and the luminance corrected image data is output to the reference image memory 31 and the comparative image memory 32. You.

Reference image memory 31 and comparative image memory 3
The image data stored in 2 is a city block distance calculation circuit having a pipeline structure in which an absolute value calculator and an adder are connected in a pyramid shape, a minimum value for evaluating the minimum and maximum values of the city block distance, and the like. The image is read into the stereo processing device 40 including a pixel shift detection circuit and the like, and the degree of coincidence is evaluated for each small region (for example, a small region of 4 × 4 pixels) of each other's images, thereby generating a distance image. The generation processing of the distance image is described in Japanese Patent Application Laid-Open No. Hei 5-111409 by the present applicant.
No. 9 discloses this in detail.

Further, the distance image data output from the stereo processing device 40 and the corrected original image data output from the image correction device 10 are read into the microcomputer 50, and various recognition processes are performed. Then, a process of calculating data for correcting the storage contents of the above-described reference / comparison image coordinate correction table 18 in accordance with aging is performed.

In the above stereo processing system, the reference image picked up by the main camera 2 and the comparison image picked up by the sub camera 3 are processed in a time-division manner in the image correction device 10 as an image input device. Image data of a small area for stereo matching is stored in an image memory 31, and image data for referencing a corresponding position with respect to one small area is stored in a comparative image memory 32 by a predetermined pixel shift number in the horizontal direction. (E.g., 100 pixels offset by assuming that a small area is 4 × 4 pixels; data of 4 × 104 pixels).

That is, the image correction device 10 converts analog images taken by the cameras 2 and 3 into A / D converters 11 and
The digital image data is converted into a digital image of a predetermined luminance gradation (for example, 256 gray scales) at 12 and the bus is switched to the image memory 14 by the data bus switching circuit 13 to use the digital image data as a reference / comparison image writing address generation circuit. The image data is stored in the image memory 14 at the horizontal address and the vertical address designated by 16.

When predetermined image data is stored in the image memory 14, reading of the image data is started by the reference / comparison image read address generation circuit 17, and the reference / comparison image coordinate correction table 18 is accessed to obtain the coordinates. The correction data (address correction data) is added to the vertical address, the coordinate correction data from the coordinate correction value calculation circuit 19 is added to the horizontal address and the vertical address, and the image memory 14 is accessed via the address switching circuit 15. You.

In this embodiment, the coordinate correction data stored in the reference / comparison image coordinate correction table 18 is data for correcting a component perpendicular to the scanning direction (horizontal direction) of the stereo matching. This is initial correction data for correcting a vertical shift, a rotational shift, a lens distortion, and a variation in a focal length of a comparative image. For this coordinate correction data, for example, 8-bit data is used,
A plus / minus sign indicating the correction direction in bits, an integer part in the next four bits, and a decimal part in the lower three bits. The maximum coordinate correction amount is ± 16 pixels, and the minimum coordinate correction resolution is 0.125 pixel (１ ／ pixel). Make corrections.

Reference / comparison image coordinate correction table 18
Is created by previously capturing a specific correction amount measurement imaging pattern such as a square grid pattern with both cameras 2 and 3 at the same time, and calculating the coordinate correction amount for each pixel for the images captured by the cameras 2 and 3. Is done. That is, the coordinates of the center of each grid point are measured from the captured image with one pixel or less, and the deviation is calculated for each grid point by comparing with the corresponding grid point position of the reference square grid pattern stored in the measuring computer. From this deviation, a data format conversion is performed in consideration of the correction direction to obtain a correction amount for each grid point. Then, for each pixel, the correction amount of the four grid points surrounding the pixel is calculated as 4
The coordinate correction amount for each pixel is calculated by the point complement and stored in the table 18.

Reference / comparison image coordinate correction table 1
8 may store only vertical correction data or horizontal correction data as needed.
7 to 9 show examples in which the correction data in the vertical direction is stored in the table 18. FIG. In this example, both cameras 2 and 3 simultaneously image a horizontal grid pattern for measuring the correction amount in advance. FIG. 7 shows an example of the images of the cameras 2 and 3. A center line (centroid line) of the width of each horizontal grid line is obtained from the captured image, and each center line is compared with a reference horizontal grid stored in a measurement computer. Then, a vertical coordinate correction amount of each pixel located on the reference grid is obtained, and a coordinate correction amount of a pixel not on the reference grid is obtained by linear interpolation to obtain a coordinate correction table having a data distribution as shown in FIG. create.

FIG. 9 shows the result of correcting an image obtained by capturing the horizontal grid pattern of FIG. 7 using the coordinate correction table of FIG. It can be seen that distortions noticeable at the four corners of the captured image have been corrected, and all the grids have been corrected horizontally.

The correction data in the horizontal direction is stored in a table 18.
, The same processing is performed using a vertical lattice pattern. When the vertical or horizontal correction data is stored in the table 18, the other correction data may be calculated by a coordinate correction value calculation circuit 19 described later.

On the other hand, the coordinate correction data from the coordinate correction value calculation circuit 19 indicates the vertical displacement, the lateral displacement, the rotational displacement and the like of the cameras 2 and 3 due to the aging.
For correction based on the instruction from the microcomputer 50. The data is corrected by monitoring the reference original image of the main camera 2 and the comparison original image of the sub camera 3 with the microcomputer 50. In the horizontal direction, correction is performed in units of one pixel. In this case, the horizontal coordinate correction amount is set such that the corresponding positions of the images are shifted by a predetermined pixel (for example, 3 pixels) at infinity so that all objects up to infinity are within the scanning range of stereo matching. Set. The correction of the rotational deviation can be easily realized by increasing or decreasing the vertical correction amount with respect to the increase of the horizontal address according to the rotational deviation correction data set by the microcomputer.

In general, it can be considered that a positional shift due to a temporal change does not cause a change requiring a non-linear correction such as a lens distortion due to a mechanical position deviation of the stereo camera 1. Can respond. Therefore, in the microcomputer 50, a reference marker for monitoring a positional shift due to a temporal change is set in the field of view of the cameras 2 and 3 in the early stage of use of the system, and the reference original image and the comparison original image are set in the microcomputer 50. The initial position of the reference marker within is stored.

When the position of the reference marker in the original image deviates from the initial position, the amount of deviation is calculated from the amount of vertical deviation and the amount of horizontal deviation, and this deviation is used as the coordinate correction value calculation circuit 19. Output to the coordinate correction value calculation circuit 19
From the amount of displacement of the reference marker due to aging,
Calculate coordinate correction data for up to / 8 pixels. The coordinate correction data due to the change over time is added to the coordinate correction data from the reference / comparison image coordinate correction table 18, so that the initial correction value for each pixel can be corrected even when aging occurs.

For example, in the vertical / horizontal correction for the vertical and horizontal displacements as shown in FIG. 2, the vertical direction is corrected at a minimum 1/8 pixel pitch, and the horizontal direction is performed at a 1 pixel pitch. The correction for the rotational displacement as shown in FIG. 3, the correction for the variation in the focal length as shown in FIG. 4, and the correction for the lens distortion as shown in FIG. It is.

That is, when stereo matching is performed in a small area having a relatively small number of pixels as in the present embodiment, information about the luminance difference between adjacent pixels is somewhat lost due to data complementation, and processing time is required. For this reason, only the component perpendicular to the scanning direction of the stereo matching is complemented with the resolution of one pixel or less, and the read address is shifted in units of one pixel in the horizontal direction.

For the data obtained by adding the coordinate correction to the vertical address generated by the reference / comparison image read address generation circuit 17, the lower 3 bits representing decimals or less are output to the data complement circuit 21, and the upper 5 bits of data are output. Thus, the data of the upper and lower two pixels in the vertical direction are accessed in two stages by the two-stage access circuit 20. Then, from these data, the data complementing circuit 21 performs data complementation of the reference positions of the reference image and the comparative image in the stereo matching.

By the way, as the reference marker, for example, in a system mounted on a vehicle, a specific position of the tip of the vehicle hood on the images of the cameras 2 and 3 can be used. When one point at the tip of the vehicle hood on the image is set as a reference marker, a temporal change in the vertical direction, that is, a vertical shift can be corrected. Further, when two or more positions at the front end of the vehicle hood on the image are set as the reference markers, it is possible to correct not only the vertical displacement but also the rotational displacement. Furthermore, by using a convex three-dimensional object instead of the tip of the vehicle hood on the image as the reference marker, it is possible to correct not only vertical displacement and rotational displacement but also lateral displacement.

When a reference marker, such as a vehicle hood, whose absolute position accuracy with respect to the camera cannot be ensured, a vertical displacement, a left and right displacement based on the relative relationship between the marker position of the reference image and the marker position of the comparative image is used. Deviation and rotation deviation may be corrected. That is, in the early stage of using the system,
The relative initial positional relationship between the reference image and the comparative image is stored in advance, and one of the images is corrected so that the initial positional relationship is maintained when the relative positional deviation occurs with time.

In the data complement circuit 21, the image memory 14
As shown in FIG. 6, when the coordinates A (x, y) of a predetermined pixel deviate from the original position by m pixels in the original image of
From the coordinates b (x, y-1) and b (x, y + 1) of the upper and lower two pixels in the vertical direction obtained by accessing the stage access circuit 20, when there is a positive displacement, the following equation (1) is used. In the case of a shift in the negative direction, complement is performed by the following equation (2), and coordinates B (x, y) corrected for the position shift are obtained. This process is performed for all pixels of the original image.

B (x, y) = b (x, y) × (1-m) + b (x, y−1) × m (1) B (x, y) = b (x, y) × (1−m) + b (x, y + 1) × m (2) In this case, when both the vertical and horizontal corrections are performed with one pixel or less, the horizontal address and the vertical address are
Two-stage access is performed to obtain coordinate data around the reference position, and horizontal and vertical complementation are performed using the coordinate data and coordinate data from the horizontal address line and the vertical address line.

The image data corrected by the data complementing circuit 21 is stored in the brightness correction result table 23 according to the data from the shading correction table 22 accessed by the reference / comparison image read address generation circuit 17 for the optical system of each camera. The brightness reduction caused by the shading phenomenon occurring in the image is corrected and stored in the reference image memory 31 and the comparison image memory 32.

Reference image memory 31 and comparative image memory 3
2 is stored in the stereo processor 40
And stereo matching is performed. In this stereo matching, the processing of calculating the city block distance between one small area of the reference image memory 31 and the small area of the comparison image memory 32 is repeated while shifting one pixel at a time in the horizontal direction until a predetermined pixel is shifted. , The minimum value and the maximum value of the city block distance are evaluated to check whether the minimum value of the city block distance really indicates the coincidence of the small areas of the two images. Then, if the check condition is satisfied, the pixel shift amount that minimizes the city block distance is output as distance information corresponding to the small area of the reference image.

At the time of this stereo matching, the optical displacement between the images captured by the cameras 2 and 3 is
Not only the linear displacement that occurs uniformly in the entire image, but also the effect of the nonlinear displacement that is locally different in the correction amount is eliminated on-board, so that the matching of the corresponding position does not occur. It is possible to obtain accurate distance information and improve reliability in image processing.

[0043]

As described above, according to the present invention, 2
For a pair of images picked up by a stereo camera having a camera on a table, an initial position in each picked-up image of a reference marker set in each field of view of the two cameras is stored, and a position of the reference marker in the image is stored. Is shifted from the initial position,
In order to correct the position shift of the whole image according to the amount of the shift, the influence of the linear position shift uniformly occurring on the entire captured image due to the aging of the stereo camera can be eliminated on-board, and the corresponding position error can be eliminated. Excellent effects can be obtained such as accurate distance information can be obtained by preventing matching, and reliability in image processing can be improved.

[Brief description of the drawings]

FIG. 1 is an overall configuration diagram of a stereo processing system.

FIG. 2 is an explanatory diagram showing a correction pattern of vertical / horizontal correction.

FIG. 3 is an explanatory diagram showing a correction pattern of rotation correction.

FIG. 4 is an explanatory diagram showing a correction pattern of focal length correction.

FIG. 5 is an explanatory diagram showing a lens distortion correction pattern.

FIG. 6 is an explanatory diagram showing data complementation in the vertical direction.

FIG. 7 is an explanatory diagram showing a captured image of a horizontal lattice pattern.

FIG. 8 is an explanatory diagram schematically showing a data distribution example of a coordinate correction table.

FIG. 9 is an explanatory diagram showing an example of an image obtained by correcting the coordinates of an original image.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 1 ... Stereo camera 10 ... Image correction device 18 ... Reference / comparison image coordinate correction table 19 ... Coordinate correction value calculation circuit 21 ... Data complementation circuit 40 ... Stereo processing device 50 ... Microcomputer

Claims (1)

[Claims] 1. An image correction apparatus for a stereo camera, which corrects an optical displacement of an image taken by a stereo camera having two cameras arranged at a set interval, wherein the two cameras are provided. The initial position of the reference marker set in each field of view in each captured image is stored. When the position of the reference marker in the image deviates from the initial position, the position of the entire image is shifted according to the shift amount. And a means for correcting image distortion.