JP6122365B6 - Stereo camera adjustment system - Google Patents

Description

  The present invention relates to a stereo camera adjustment system that adjusts a positional deviation in the optical axis direction when a stereo camera is mounted on a vehicle body and a positional deviation in the vertical and rotational directions between left and right camera images.

  2. Description of the Related Art Conventionally, a stereo image processing technique is known in which the same object is imaged from different viewpoints using a stereo camera, and the object is three-dimensionally recognized from distance data (parallax) obtained. For example, in a vehicle such as an automobile, a stereo camera captures the driving environment in front and recognizes the driving environment three-dimensionally. A technology for detecting the vehicle and performing follow-up control and warning control on the preceding vehicle has been developed and put into practical use.

  Such a stereo camera is configured by mechanically fixing a pair of cameras, for example, with a predetermined base length (interval of optical axis) so that their optical axes are substantially parallel to each other. When this occurs, the imaging directions of the respective cameras are shifted, causing phenomena such as the cameras not being able to recognize the preceding vehicle and the accuracy of the distance data being deteriorated.

  The optical axis of the camera needs to face straight with respect to the traveling direction of the vehicle. If the camera is displaced with respect to the traveling direction when the camera is attached, it is difficult to recognize a front obstacle.

  In a stereo camera, there may be a shift in the vertical and rotational directions between the left and right camera images. These misregistrations appear as misalignment of the same horizontal line (epipolar line) in each image when stereo matching is performed. In stereo matching that assumes epipolar line matching, the reliability of distance data decreases. Connected.

  For this reason, in Patent Document 1, a chart for adjustment is installed in front of the stereo camera, and the captured image of the chart for adjustment is processed to adjust the positional deviation of the stereo camera. Techniques for improving are disclosed.

JP 2004-132870 A

  By the way, when a stereo image processing system is adopted, the resolution of the camera may be changed (usually the resolution is increased) due to a change in system specifications or the like. In particular, when correcting the positional deviation of the camera using the adjustment chart, a chart suitable for the resolution of the camera is necessary, and it is necessary to switch to a new adjustment chart.

  However, when moving from the old system to the new system, different types of adjustment charts will be mixed, and during adjustment work, there is a possibility that the operator will install the adjustment chart in error, which is the wrong adjustment chart. If the stereo camera is adjusted as it is, the distance accuracy may deteriorate despite the adjustment.

  The present invention has been made in view of the above circumstances, and autonomously determines whether or not an adjustment chart installed by an operator is compatible with the resolution of a stereo camera, thereby preventing human work errors. It is an object of the present invention to provide a stereo camera adjustment system capable of ensuring the reliability of distance data.

  An adjustment system for a stereo camera according to the present invention includes an adjustment pattern portion in which a pattern for correcting a positional shift between left and right camera images of a stereo camera and a mark for correcting a shift in the optical axis of the camera are arranged. A stereo camera adjustment system that corrects a positional shift of the stereo camera by processing a captured image of the adjustment chart having an adjustment mark portion, wherein the adjustment chart includes a difference in resolution of the stereo camera. In correspondence with the above, the pattern is made as a different pattern and is provided as a plurality of adjustment charts in which the complementary color relations of the marks are reversed to each other, and the mark cannot be recognized when the stereo camera is adjusted using the adjustment chart At this time, the adjustment chart is determined to be incompatible.

  According to the present invention, it is determined autonomously whether the adjustment chart installed by the worker is compatible with the resolution of the stereo camera, and it is possible to prevent human work mistakes and to improve the reliability of distance data. Can be secured.

Block diagram of an image processing apparatus having a stereo camera adjustment function Illustration of optical axis adjustment Explanation of coarse adjustment Explanatory drawing which shows the arrangement | positioning relationship between the chart for adjustment and a vehicle-mounted camera Explanatory drawing which shows the 1st chart for adjustment Explanatory drawing which shows the 2nd chart for adjustment Flow chart showing stereo camera adjustment process

Embodiments of the present invention will be described below with reference to the drawings.
In FIG. 1, reference numeral 1 denotes an image processing apparatus that is mounted on a vehicle such as an automobile and that recognizes a traveling environment outside the vehicle in a three-dimensional manner by processing a captured image of a stereo camera 2 that captures an object from different viewpoints. For example, it is used in a system that performs driving support for a driver, brake control for avoiding danger, steering control, and the like. The image processing apparatus 1 has an adjustment function that autonomously corrects and adjusts the positional deviation of the stereo camera 2 and can form a stereo camera adjustment system together with an adjustment chart AC described later. .

  In the following, correcting the displacement in the optical axis direction and the displacement between the left and right images mainly due to the displacement of the cameras constituting the stereo camera 2 will be referred to as “calibration”.

  Such an image processing apparatus 1 has an A / D conversion unit 3, an optical axis adjustment unit 4, a coarse adjustment unit 5, a distance correction unit 6, a stereo image processing unit 7, a recognition function as processing functions for a captured image of the stereo camera 2. The processing unit 9 is mainly provided with a distance data memory 8a and an image data memory 8b. The image processing apparatus 1 also has a positional deviation calibration unit 10 that calculates calibration parameters for correcting the deviation of image data caused by the positional deviation of the stereo camera 2 as an adjustment function for the stereo camera 2, before shipment of the product, or periodic inspection. An inspection / adjustment instruction unit 12 is sometimes provided that communicates with an external device and instructs the inspection / adjustment of the stereo camera 2 via the misalignment calibration unit 10 and the optical distortion correction unit 11 in accordance with a command input from the outside.

  In the present embodiment, the stereo camera 2 has two cameras 2a and 2b each having an image sensor such as a CCD or a CMOS with a predetermined baseline length (optical axis interval) so that their optical axes are substantially parallel to each other. The camera unit is configured as a mechanically fixed camera unit, and is installed, for example, in the vicinity of a room mirror inside the front window at the upper part of the vehicle interior. One camera 2a constituting the stereo camera 2 captures a reference image (right image) necessary for performing stereo image processing, and the other camera 2b captures a comparison image (left image). Hereinafter, the camera 2a that captures the reference image (right image) is appropriately described as a main camera 2a, and the camera 2b that captures a comparison image (left image) is appropriately described as a sub camera 2b.

  Each analog image output from each of the cameras 2a and 2b is converted into a digital image of a predetermined luminance gradation (for example, a gray scale of 256 gradations) by the A / D conversion unit 3, respectively. This digitized image data is expressed in an ij coordinate system with the lower left corner of the image as the origin, the horizontal direction as the i coordinate axis, and the vertical direction as the j coordinate axis, and reference image data is obtained from the main camera 2a. Comparative image data is obtained from the sub camera 2b.

  The optical axis adjustment unit 4 adjusts the position of the processing area used for recognition with respect to the imaging area of the camera. As for the mounting position of the stereo camera 2, it can be avoided that the direction of the optical axis with respect to the vehicle body (the direction of the image center) varies in the pitch direction and the yaw direction between the pair of cameras (left and right cameras) 2 a and 2 b due to variations among vehicles. Absent. For this reason, if the processing area targeted for recognition processing is uniquely set with respect to the imaging area of each camera, for example, based on the center of the image, the processing areas differ between the left and right cameras 2a and 2b.

  For this reason, the optical axis adjustment unit 4 moves the recognition area according to the calibration parameter calculated by the positional deviation calibration unit 10 using an adjustment chart AC described later, and corrects the deviation of the camera optical axis. As will be described later, the calibration parameters calculated by the positional deviation calibration unit 10 are the deviation in the optical axis direction of the recognition area of the image due to the positional deviation between the cameras 2a and 2b, and the translation / As shown in FIG. 2, the optical axis adjustment unit 4 moves the recognition area Rp having a preset size in the yaw direction within the imaging area Rm of each camera. Δy is translated by a movement amount Δp in the pitch direction.

  The coarse adjustment unit 5 corrects a shift in the translation / rotation direction of the image of the camera 2b with respect to the image of the camera 2a. Specifically, as shown in FIG. 3, the shift of the image of the camera 2b with respect to the image of the camera 2a is calculated, and the image of the camera 2b is geometrically moved by the translation Δd and the rotation angle Δθ.

  The calibration parameters for the reference image and the comparison image in the positional deviation calibration unit 10 are determined (or updated) during the adjustment process and fed back to the optical axis adjustment unit 4 and the coarse adjustment unit 5. The calculation of the calibration parameter is described in detail in Japanese Patent Application Laid-Open No. 2004-283553 proposed by the present applicant.

  The distance correction unit 6 corrects the reference image data and the comparison image data based on the correction parameters from the optical distortion correction unit 11. This correction parameter corrects image data distortion caused by optical distortion of the camera. The coordinates of the pixel points constituting the image data of the reference image and the comparison image are the same as those of the reference image and the comparison image. Based on each correction parameter, the coordinate position on the image plane is shifted. By this correction, basically, image distortion caused by optical distortion of the individual cameras 2a and 2b is corrected, and an error in distance data due to this distortion is corrected.

  The calculation of correction parameters for correcting image distortion caused by optical distortion of the camera is also described in detail in Japanese Patent Application Laid-Open No. 2004-283553 proposed by the present applicant.

  As described above, the calibration of the image data in the optical axis adjustment unit 4 and the coarse adjustment unit 5 and the correction of the image data in the distance correction unit 6 ensure the coincidence of the epipolar lines in the reference image and the comparison image. A pair of image data corresponding to one frame is output to the subsequent stereo image processing unit 7 and stored in the image data memory 8b.

  The stereo image processing unit 7 obtains a shift amount (parallax) between corresponding positions of the reference image and the comparison image by stereo matching processing. In stereo matching processing, a small area (also referred to as a block or window, hereinafter referred to as a block) is set around a certain point in the reference image, and the same size is set around a certain point in the comparison image. A well-known region search method in which a corresponding block is provided to search for corresponding points can be employed.

  In the corresponding point search processing by this region search method, the correlation calculation with the block of the reference image is performed while shifting the block on the comparison image, and the shift amount of the coordinates of the block having the largest correlation value, that is, the parallax d is calculated. A set of parallax d calculated for each block is stored in the distance data memory 8a as “distance data”.

  The recognition processing unit 9 obtains the distance in the real space using the distance data stored in the distance data memory 8a and performs various recognition processes. For example, a grouping process for grouping distance data within a predetermined threshold is performed, and compared with frames (windows) such as three-dimensional road shape data, side wall data, and three-dimensional object data stored in advance, a white line Data and side wall data such as guardrails and curbs that exist along the road are extracted, and three-dimensional objects are classified and extracted into two-dimensional objects such as two-wheeled vehicles, ordinary vehicles, large vehicles, pedestrians, and power poles.

  For each recognized data, the respective positions in the coordinate system with the own vehicle as the origin and the longitudinal direction and the width direction of the own vehicle as axes are calculated. In particular, in the vehicle data of two-wheeled vehicles, ordinary vehicles, and large vehicles, the length in the front-rear direction is estimated in advance, for example, 3 m, 4.5 m, 10 m, etc., and the width direction is the center position of the detected width. Is used to calculate the center position of the vehicle.

  Further, in the three-dimensional object data, the relative speed with respect to the own vehicle is calculated from the change in the axial direction of the distance from the own vehicle, and each three-dimensional object is calculated by calculating the relative speed in consideration of the speed of the own vehicle. The speed in each axis direction is calculated. Each information obtained in this way, that is, white line data, side walls data such as guardrails and curbs along the road, and three-dimensional object data (type, distance from own vehicle, center position coordinates, speed, etc.) Therefore, a moving object such as a pedestrian or a light vehicle around the own vehicle and other vehicles traveling on a road connected to the road on which the own vehicle travels is recognized.

  Such white line data from the recognition processing unit 9, guard rails existing along the road, side wall data such as curbs, three-dimensional object data, and the like are input to a vehicle control unit (not shown). The vehicle control unit recognizes the driving environment based on the information from the recognition processing unit 9 and vehicle information such as the vehicle speed and yaw rate of the host vehicle, and pre-crash brake control, cruise control with a tracking function, stagger and warning for lane departure Command driver assistance control such as control.

  Next, adjustment of the stereo camera 2 by the misalignment calibration unit 10 and the optical distortion correction unit 11 will be described.

  When adjusting the stereo camera 2, as shown in FIG. 4, the adjustment chart AC is arranged at a predetermined position in front of the vehicle C on which the stereo camera 2 to be inspected and adjusted is mounted. This adjustment chart AC is arranged so as to be substantially parallel to the base lines of the cameras 2a and 2b. In this state, when the image processing apparatus 1 is operated in the inspection / adjustment mode of the stereo camera from the outside of the vehicle via the inspection / adjustment instruction unit 12, the adjustment chart AC is generated by the stereo camera 2 (cameras 2a and 2b). Images are taken and adjustment parameters (calibration parameters and correction parameters) are calculated.

  Here, the adjustment chart AC is provided with different types of charts as shown in FIGS. 5 and 6 in accordance with the difference in resolution (number of pixels of the image sensor) of the cameras 2a and 2b. In this embodiment, the resolution of the camera is increased in accordance with the transition from the old system to the new system, and the first and second two systems correspond to the fact that the old and new systems are mixed in the transition period of the system transition. An adjustment chart AC # i (i = 1, 2) is provided. The first adjustment chart AC # 1 corresponds to a vehicle equipped with a camera having a relatively low resolution compared to the adjustment chart AC # 2, and the second adjustment chart AC # 2 corresponds to a vehicle equipped with a camera having a relatively high resolution.

  These adjustment charts AC # i basically have the same configuration, and an adjustment pattern portion ACrd # i in which a plurality of predetermined lattices for coarse adjustment / distance correction are arranged in random gradation patterns. (I = 1, 2) and an adjustment mark part ACmk # i (i = 1, 2) in which an optical axis adjustment mark is arranged below the adjustment pattern part ACrd # i. In the present embodiment, a cross-shaped mark is used as an optical axis adjustment mark.

  From the adjustment pattern portion ACrd # i, for example, the coordinates on the image plane of the feature points of the lattice pattern are detected, and the correction parameter is calculated from the difference between the coordinates of the detected feature points and the target coordinates corresponding to the feature points. Is determined. The target coordinates are coordinates detected when the adjustment chart is imaged with an ideal camera in which no optical distortion exists. For example, the adjustment chart pattern and the adjustment chart Can be theoretically calculated based on the distance between the camera and the stereo camera and the focal length of the camera.

  On the other hand, in the optical axis adjustment using the adjustment mark part ACmk # i, the calibration parameter is determined from the relationship between the coordinate position of the mark on the captured image and the mark position of the chart with respect to the vehicle. As described above, the calibration parameters calculated from the adjustment mark portion ACmk # i are calibration parameters for adjusting the optical axes of the left and right cameras that move the recognition area in the yaw direction and the pitch direction.

  Here, in the adjustment chart AC # i, the pattern of the adjustment pattern part ACrd # i differs depending on the resolution of the image sensor, and the adjustment pattern part ACrd # 2 of the adjustment chart AC # 2 has a relatively high resolution. The pattern is finer than the adjustment pattern portion ACrd # 1 of the adjustment chart AC # 1 so as to correspond to a high camera. When adjusting the camera, it is necessary for the operator to appropriately select which of these adjustment charts AC # i to use depending on the vehicle (camera) to be adjusted. It is difficult to completely eliminate the possibility of being lost.

  Therefore, although the adjustment mark portion ACmk # i has the same cross shape as the optical axis adjustment mark shape, the complementary color relationship of the marks is reversed with each other so that erroneous installation of the chart is detected on the vehicle side. ing. Specifically, a black cross mark mk # 1 is arranged on a white background in the adjustment mark portion ACmk # 1 corresponding to a camera with a relatively low resolution, whereas a camera with a relatively high resolution is provided. In the adjustment mark portion ACmk # 2 corresponding to, a white cross mark mk # 2 is arranged on a black background.

  When operating in the camera inspection / adjustment mode, the image processing apparatus 1 on the vehicle side first determines whether or not the adjustment chart AC # i arranged in front of the vehicle (front of the camera) is appropriate. Whether or not the chart is appropriate is determined by whether or not the cross mark mk # i of the adjustment mark portion ACmk # i can be recognized. In the unlikely event that the chart is selected incorrectly, an alarm is output to prompt the exchange of the chart.

  Hereinafter, a stereo camera adjustment process using the adjustment chart AC will be described with reference to a flowchart shown in FIG.

  In this adjustment process, first, as a preparation for adjustment, the vehicle is arranged at the adjustment position, and the adjustment chart AC # i corresponding to this vehicle is set at a specified position in front of the vehicle (step S1). Next, an external device such as a diagnostic device is connected to the vehicle, and communicates with the inspection / adjustment instruction unit 12 of the image processing apparatus 1 to set the inspection / adjustment mode of the stereo camera (step S2). When the image processing apparatus 1 is set to the inspection / adjustment mode, the image processing apparatus 1 first obtains the current adjustment chart AC # i from the captured image of the adjustment mark portion ACmk # i of the adjustment chart AC # i. It is determined whether or not the chart is appropriate for the stereo camera mounted on the host vehicle (step S3).

  When the image processing apparatus 1 on the vehicle side is an old system equipped with a stereo camera having a relatively low resolution, the old system uses a cross mark mk # i of the adjustment mark portion ACmk # i and a black mark shape on a white background. Is recognized by pattern matching. On the other hand, when the image processing apparatus 1 on the vehicle side is a new system equipped with a stereo camera having a relatively high resolution, in this new system, the cross mark mk # i of the adjustment mark portion ACmk # i is placed on a black background. Recognized by pattern matching as a white mark shape.

  Therefore, for example, when the adjustment chart AC # 2 is erroneously installed in front of the old system vehicle, the misalignment calibration unit 10 sets the white cross mark mk # 2 on a black background and the black cross mark mk on a white background. Because it tries to recognize as # 1, it fails to recognize the cross mark. Further, when the adjustment chart AC # 1 is erroneously installed in front of the vehicle of the new system, the misalignment calibration unit 10 sets the black cross mark mk # 1 on a white background and the white cross mark mk on a black background. Because it tries to recognize as # 2, it fails to recognize the cross mark.

  When the recognition of the cross mark mk # i fails, the misalignment calibration unit 10 determines that the adjustment chart AC # i actually installed for the stereo camera of the current system is incompatible. Then, an error signal is transmitted to the external device via the inspection / adjustment instruction unit 12 to warn the operator that the adjustment chart AC # i has been installed incorrectly (step S4). On the other hand, when the cross mark mk # i is correctly recognized, the optical axis adjustment is automatically executed as it is (step S5).

  After performing the optical axis adjustment, coarse adjustment is performed based on the captured image of the adjustment pattern portion ACrd # i (step S6), and then the average value of distance measurement by the adjustment pattern portion ACrd # i is within the predetermined range. Whether or not (step S7). If the inspection result of the distance measurement is OK, the adjustment parameter and the inspection result are written in the memory in the image processing apparatus 1 (step S8), and this adjustment process is terminated.

  On the other hand, when the test result of the distance measurement is NG, a process for newly calculating a distance correction parameter using the adjustment pattern portion ACrd # i is performed (step S9), and again using this correction parameter, A distance measurement inspection is performed (step S10). If the inspection result is OK, as described above, the adjustment parameter and the inspection result are written in the memory and this adjustment process is terminated. If the inspection result is still NG, the installation position of the vehicle and the adjustment chart The inspection environment around the camera is reviewed (step S11). Then, this adjustment process is restarted.

  As described above, in the present embodiment, when an adjustment chart is installed in front of a vehicle on which the image processing apparatus 1 having a stereo camera adjustment function is mounted and the stereo camera is adjusted, the adjustment chart is adapted to the camera to be adjusted. Since the vehicle side (image processing apparatus 1) autonomously determines whether or not the adjustment chart is the adjusted chart, it is possible to prevent the operator's human work error and to ensure the reliability of the distance data.

DESCRIPTION OF SYMBOLS 1 Image processing apparatus 2 Stereo camera 4 Optical axis adjustment part 5 Coarse adjustment part 6 Distance correction part 7 Stereo image processing part 9 Recognition processing part 10 Misalignment correction part 11 Optical distortion correction part 12 Inspection / adjustment instruction part AC # i For adjustment Chart ACmk # i Adjustment mark part ACrd # i Adjustment pattern part mk # i Cross mark

Claims (3)

An adjustment chart having an adjustment pattern portion in which a pattern for correcting a positional deviation between left and right camera images of a stereo camera is arranged, and an adjustment mark portion in which a mark for correcting a deviation of the optical axis of the camera is arranged A stereo camera adjustment system that corrects positional shift of the stereo camera by processing the captured image of
Corresponding to the difference in resolution of the stereo camera, the adjustment chart is provided as a plurality of adjustment charts in which the pattern is a different pattern and the complementary color relationship of the mark is inverted with respect to each other.
When adjusting the stereo camera using the adjustment chart, if the mark cannot be recognized, it is determined that the adjustment chart is incompatible.   The stereo camera adjustment system according to claim 1, wherein the mark is a cross-shaped mark.   The cross-shaped mark is a black mark on a white background for the relatively low-resolution stereo camera, and a white mark on a black background for the relatively high-resolution stereo camera. The stereo camera adjustment system according to claim 2.

Images