JP2016001378A - Calibration device of on-vehicle camera - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a calibration device capable of automatically calibrating deviation of a mount position of each camera in a vehicle imaging system equipped with a plurality of cameras.SOLUTION: During vehicle stop, it is determined whether or not a predetermined calibration image is captured at an overlapped area where one portion of imaging areas of a plurality of cameras overlaps (S150). When capturing of the calibration image is determined, operation, such as crossing, mutation, and copy, is performed to a plurality of seeds with displacement to a reference camera parameter as a variable, based on genetic algorithm. The degree of coincidence of the calibration image changed in accordance with the seeds after the operation is calculated as a matching score. A seed having the highest degree of coincidence is selected as an optimum solution corresponding to the actual deviation of cameras (S160-S200).

Description

  The present invention relates to an in-vehicle camera calibration apparatus suitable for calibrating a displacement of a mounting position of each camera in a vehicle imaging system including a plurality of cameras.

  In order to display a bird's-eye view around the vehicle on a display device in the passenger compartment, a plurality of cameras are arranged at different positions (for example, front, back, left, and right) of the vehicle, and the images captured by the cameras are converted by viewpoints and combined to display 2. Description of the Related Art A vehicle imaging system configured to generate an image (overhead image) is known.

  In this type of vehicle imaging system, if the mounting position of each camera is shifted, viewpoint conversion of the captured image cannot be performed satisfactorily, and therefore it is proposed to calibrate the mounting position shift for each camera ( For example, see Patent Document 1).

  In the proposed calibration apparatus, a square calibration pattern is installed in an imaging area (overlapping area) that overlaps between the camera to be calibrated and another camera, and an image is captured by each camera. Then, the conversion rule of the captured image with respect to the camera to be calibrated is adjusted so that the positional deviation of the coordinates (vertex etc.) of the calibration pattern is minimized in the display image obtained by performing viewpoint conversion of the captured image of each camera.

JP 2010-244326 A

  By the way, in the above-described conventional configuration apparatus, it is assumed that a predetermined calibration pattern is arranged in an overlapping area where the imaging areas of a plurality of cameras overlap at the time of calibration. Limited to specific facilities for sale or repair.

For this reason, when the owner of the vehicle feels an abnormality in the display image, he / she has to bring the vehicle into a facility capable of calibrating the in-vehicle camera, which is inconvenient.
The present invention has been made in view of these problems, and an object of the present invention is to provide a calibration device capable of automatically calibrating the displacement of the mounting positions of each camera in a vehicle imaging system including a plurality of cameras. .

  The in-vehicle camera calibration device of the present invention includes a plurality of cameras and display image generation means for generating a display image using conversion data set based on reference camera parameters representing the positions and orientations of the plurality of cameras. In the vehicular imaging system provided, the positional deviation of each camera from the reference position is calibrated.

In the on-vehicle camera calibration apparatus according to the present invention, the determination unit determines whether or not a predetermined calibration image has been captured in an overlapping region where a part of the imaging regions of a plurality of cameras overlaps.
When it is determined that the calibration image has been captured by the determination unit, the calibration data generation unit displaces the reference camera parameter according to a predetermined evolution algorithm, so that an overlapping region of a plurality of cameras that have captured the calibration image is detected. In the captured image, the displacement amount of the reference camera parameter that most closely matches the position and shape of the calibration image is searched.

Then, the calibration data generation means sets the searched displacement amount of the reference camera parameter as calibration data of a plurality of cameras.
As described above, according to the in-vehicle camera calibration device of the present invention, when a predetermined calibration image is captured in the overlapping region where the imaging regions overlap by a plurality of cameras where the imaging regions partially overlap. In addition, the calibration data generation means is automatically activated to generate calibration data.

  Therefore, according to the on-vehicle camera calibration device of the present invention, when the vehicle owner uses the vehicle, the calibration data generation means is configured to capture the calibration image in the overlapping region. Thus, the calibration data is frequently generated, and the display image can be prevented from being distorted or unclear due to the positional deviation of the camera.

In addition, since the user does not need to bring the vehicle to a predetermined facility such as a repair shop for camera calibration, the usability of the vehicle imaging system can be improved.
In the in-vehicle camera calibration apparatus according to the present invention, the calibration data generation means generates the calibration data by displacing the reference camera parameter according to an evolution algorithm such as a genetic algorithm. Can be calibrated properly.

  In other words, the mounting position (position and orientation) of the in-vehicle camera is caused by a temporal (in other words, continuous) factor caused by vibrations received from the vehicle, and a sudden ( In other words, there are two types of factors: discrete factors.

  On the other hand, in the evolution algorithm, a continuous search represented by crossover (local search) in a genetic algorithm and a discrete search represented by mutation (mutation) can be performed. .

  Therefore, if the optimal solution of the calibration data is searched using the evolution algorithm as in the present invention, it is possible to efficiently and optimally calibrate the displacement of the in-vehicle camera.

It is a block diagram showing the structure of the imaging system for vehicles of embodiment. It is explanatory drawing showing the mounting position of the camera to a vehicle, and the imaging area of a camera. It is a flowchart showing the calibration process performed in the calibration part of FIG. It is explanatory drawing explaining the calibration operation implemented after shipment of a vehicle. It is explanatory drawing explaining the seed used by the calibration operation | movement of FIG. 4, and its operation.

Embodiments of the present invention will be described below with reference to the drawings.
The present invention is not construed as being limited in any way by the following embodiments. An aspect in which a part of the configuration of the following embodiment is omitted as long as the problem can be solved is also an embodiment of the present invention. Moreover, all the aspects which can be considered in the limit which does not deviate from the essence of the invention specified only by the wording described in the claims are embodiments of the present invention. Further, the reference numerals used in the description of the following embodiments are also used in the claims as appropriate, but this is used for the purpose of facilitating the understanding of the present invention, and limits the technical scope of the present invention. Not intended.

  As shown in FIG. 1, the imaging system for vehicles of this embodiment produces | generates a bird's-eye view image from the four cameras 12, 14, 16, 18 for imaging the vehicle periphery, and the captured image by each camera 12-18. And an electronic control unit (ECU) 20 for image processing to be displayed on the display 30.

  As shown in FIG. 2, the four cameras 12 to 18 can be used as a front camera, a rear camera, a left camera, and a right camera so that the front, rear, left side, and right side of the vehicle 2 can be imaged. It is attached to the front, rear, left and right respectively.

  Each camera 12-18 is for imaging the surroundings of the vehicle 2, using the front, rear, left side, and right side of the vehicle 2 as imaging regions, as indicated by dotted lines in FIG. And the overlapping area 6 which overlaps with the imaging area of the camera which adjoins the left front of the vehicle 2, the left rear, the right front, and the right rear is set to the imaging area of each camera 12-18.

  In addition, the mounting positions (mounting positions and postures) of the cameras 12 to 18 on the vehicle 2 are set in advance so that the imaging areas of the cameras 12 to 18 are as described above. Then, the actual mounting positions and orientations (in other words, the imaging directions) of the cameras 12 to 18 are adjusted at a manufacturing factory, a repair factory, or the like based on the set mounting positions.

  On the other hand, the ECU 20 generates a bird's-eye view image by converting the viewpoints of the images captured by the cameras 12 to 18 and combining them using the attachment positions and orientations of the cameras 12 to 18 as reference camera parameters.

The reference camera parameters are obtained by quantifying, for example, the attachment positions of the cameras 12 to 18 in the vehicle 2 and the attachment angles in the three axial directions of the front, rear, left, and right of the vehicle.
And when ECU20 carries out viewpoint conversion of the captured image by each camera 12-18, the conversion data set based on the reference | standard camera parameter are utilized.

  For this reason, if the reference camera parameter is deviated from the actual mounting state of each of the cameras 12 to 18, the viewpoint conversion of the captured image cannot be performed accurately, and the finally obtained overhead image is distorted. It will be.

  Therefore, in a vehicle manufacturing factory, repair shop, etc., after adjusting the positions of the cameras 12 to 18, as shown in FIG. 3, a reference pattern 4 for calibration having a predetermined pattern (square shape in the figure) is arranged around the vehicle. Then, by performing mounting position calibration (so-called calibration) of each of the cameras 12 to 18, reference camera parameters are calculated, and conversion data is set.

  In this calibration, the calibration reference pattern 4 around the vehicle is imaged by each of the cameras 12 to 18, and the calibration reference pattern 4 in the converted image obtained by performing viewpoint conversion of the captured image and the actual calibration reference pattern 4 are converted. This is done by detecting a deviation from the arrangement position.

  For this reason, as shown in FIG. 1, the ECU 20 includes a camera calibration unit that calibrates the mounting positions of the cameras 12 to 18 in addition to the image conversion unit 22 that converts viewpoints of images captured by the cameras 12 to 18. 28 is provided.

  The camera calibration unit 28 is composed of a known microcomputer (microcomputer) centered on a CPU, ROM, RAM, etc., and performs the above-described mounting position calibration in accordance with a calibration command input from an external device at a factory or the like. .

  Further, the camera calibration unit 28 is adapted to perform the mounting position calibration of each of the cameras 12 to 18 even when the vehicle 2 is shipped from the factory and used by the user, and the execution condition is determined. In order to make a vehicle stop determination in this embodiment, a detection signal indicating a vehicle state such as a vehicle speed or a brake signal is input from another vehicle control device.

  When the camera calibration unit 28 calibrates the mounting positions of the cameras 12 to 18, the image conversion unit 22 performs the calibration based on the calibration result (in other words, the deviation from the reference mounting positions of the cameras 12 to 18). Conversion data used to convert viewpoints of images captured by the cameras 12 to 18 is updated.

  In the ECU 20, a camera calibration unit 28 includes a predetermined image (such as the above-described calibration reference pattern 4 described above) registered in the ROM or the nonvolatile memory from the input image (overhead image) from the image conversion unit 22. ) Is used to detect the image recognition engine 24.

The ECU 20 is also provided with a display control unit 26 for causing the display 30 to display an output image (such as a bird's-eye view image) from the camera calibration unit 28.
Next, calibration processing executed by the camera calibration unit 28 for calibration of the mounting positions of the cameras 12 to 18 will be described with reference to the flowchart shown in FIG.

  As shown in FIG. 1, the calibration process executed by the camera calibration unit 28 includes an initial calibration process of S <b> 110 to S <b> 130 (S represents a step) executed at the manufacturing factory or repair factory of the vehicle 2, and the factory. And the automatic calibration processing of S140 to S200 that is executed after the shipment.

  The initial calibration process of S110 to S130 is a process executed when a calibration command is input to the ECU 20 in a state where a large number of calibration reference patterns 4 shown in FIG. When the process is started, first, calibration of each camera 12 to 18 is performed in S110.

  In this process, for example, conversion data serving as an initial value is input to the image conversion unit 22 to convert the captured images from the cameras 12 to 18 into overhead images viewed from above the vehicle. The calibration reference pattern is recognized by using the image recognition engine 24, and the position and shape of the recognized calibration reference pattern and the actual position and shape deviation of the calibration reference pattern are detected. .

  In S110, reference camera parameters x = {x0, x1, x2,..., Xn} representing the mounting positions (arrangement positions and orientations) of the cameras 12 to 18 are calculated based on the calibration result, and the reference is calculated. The conversion data of the image conversion unit 22 is updated based on the camera parameter x.

  Thus, when the calibration of each camera 12 to 18 is performed, the process proceeds to S120, and the displacement Δx = {Δx0, Δx1, Δx2,... From the reference camera parameter x for the calibration process after the vehicle 2 is shipped. , Δxn} is initialized as a variable.

  This seed is for specifying the amount of displacement from the reference mounting position of each camera 12 to 18 according to the genetic algorithm. In S120, a plurality of seeds are set at random as illustrated in FIG. 5A. Is done. In addition, at least one of the plurality of seeds is set so that all the displacements are “0”.

  In subsequent S130, the images (for example, images of the calibration reference pattern 4) in the overlapping region 6 obtained by the respective cameras 12 to 18 obtained via the image conversion unit 22 are converted into the seeds (the respective cameras that are initially set in S120). 12 to 18), a matching score representing the degree of coincidence of the deformed image with the reference image is calculated and sorted.

  In this embodiment, the matching score is obtained as a weighted average of the variances of the pixels constituting the image with respect to the target region, and the matching score is the smallest (in the initial calibration process, Δx = {0, 0, 0,..., 0}) is the optimum solution corresponding to the current mounting position of each camera 12-18.

  In S130, as shown in FIG. 5 (b), the matching scores calculated for each seed are arranged (sorted) in order from the smallest matching score closest to the optimum solution, and stored in the nonvolatile memory.

  Next, in the automatic calibration process executed during normal use of the vehicle 2 after shipment from the factory, first, in S140, the vehicle 2 is stopped based on the vehicle speed, brake signal, etc. input from another vehicle control device. By determining whether or not, the vehicle 2 waits for the vehicle to stop.

  Then, when the vehicle 2 is stopped, the process proceeds to S150, and the calibration registered in the ROM or the non-volatile memory in the image in the overlapping region 6 by the cameras 12 to 18 obtained through the image conversion unit 22 is performed. By determining whether or not there is a use pattern, each camera 12 to 18 waits for the calibration pattern to be imaged.

  This calibration pattern includes, for example, a plurality of images such as an image of a pedestrian crossing or a stop line drawn on the road surface, an image of a road sign installed at a place where the vehicle stops, and an image around the storage location of the vehicle 2. This image is registered in advance as a calibration image.

  In S150, it is determined using the image recognition engine 24 whether or not at least one of the plurality of calibration patterns is present in the images in the overlapping area 6 captured by the cameras 12-18. To do.

When detecting a calibration pattern image from the overlapping region images, the calibration pattern can be easily detected by using a feature amount based on boosting.
If it is determined in S150 that a calibration pattern is present in the images in the overlapping area 6 captured by the cameras 12 to 18, the process proceeds to S160. In S160, as shown in FIG. 5C, operations in a genetic algorithm such as crossover, mutation, and copy are performed on each of the plurality of seeds stored in the nonvolatile memory.

  Next, in S170, as illustrated in FIG. 4, a predetermined area 9 including the image of the calibration pattern 8 is selected from the images of the overlapping area 6 obtained by the cameras 12 to 18 obtained via the image conversion unit 22. Extracted as a matching score calculation target region, the extracted image is deformed based on a plurality of seeds operated in S160, a matching score is calculated, and each seed is sorted based on the calculation result.

  The calculation of the matching score is performed by obtaining the degree of coincidence of images obtained from the two cameras that have captured the target overlapping region 6 as a weighted average of the variance of each pixel constituting the image. In step S170, the seeds are sorted in the order of seeds that minimizes the calculated matching score.

  As described above, when each seed operation, matching score calculation, and each seed sorting are performed based on the genetic algorithm in S160 and S170, the process proceeds to S180, and this series of processing (in the claims) It is determined whether or not the described search operation has been performed a plurality of times (N times) set in advance.

  If it is determined in S180 that the series of processes (search operations) has not been performed N times, the process proceeds to 160 again, and the processes in S160 and S170 are performed again. In S180, the series of processes is performed. If it is determined that the above process has been performed N times, the process proceeds to S190.

In S190, it is determined whether or not the minimum matching score among the matching scores obtained by the N processes is less than a preset threshold value.
Then, if the minimum matching score is not less than the threshold value, the process proceeds to S140 as it is, and if the minimum matching score is less than the threshold value, the seed corresponding to the minimum matching score is set as a reference for each of the current cameras 12-18. The optimum solution (displacement) representing the amount of displacement from the position (reference camera parameter) is selected, and the process proceeds to S140.

The seed selected in this way is used as calibration data for updating the conversion data of the image conversion unit 22.
As described above, in the vehicle imaging system according to the present embodiment, the camera calibration unit 28 in the ECU 20 performs calibration of the mounting positions (calibration) of the cameras 12 to 18 using the calibration reference pattern in a factory or the like. The reference camera parameters are obtained by executing the initial calibration process for performing the above.

In the initial calibration process, a plurality of seeds having the displacement with respect to the reference camera parameter as a variable are initially set for the mounting position calibration after shipment.
In addition, after the vehicle 2 is shipped, the camera calibration unit 28 in the ECU 20 executes an automatic calibration process, so that the calibration pattern registered in advance in the overlapping region 6 where the imaging regions of the cameras 12 to 18 overlap. Each seed is manipulated based on a genetic algorithm each time.

  In the automatic calibration process, the degree of coincidence between the images in the overlapping region 6 corresponding to each seed (displacement from the reference position of the camera) after the operation is calculated as a matching score, and the seed with the best match between the images is calculated. Set as calibration data.

  Therefore, according to the vehicle imaging system of the present embodiment, the calibration of the mounting positions of the cameras 12 to 18 is automatically and frequently performed during normal use of the vehicle 2. It is possible to prevent the display image from being distorted or unclear due to the displacement.

Further, since the user does not need to bring the vehicle to a repair shop or the like for calibration of the mounting positions of the cameras 12 to 18, the usability of the vehicle imaging system can be improved.
Further, in the automatic calibration process, each camera 12 is searched for a positional deviation amount from the reference position (reference camera parameter) of each camera 12 to 18 by operating each seed using a genetic algorithm. The amount of misalignment of ˜18 can be accurately obtained.

  That is, as described above, there are two types of factors that cause a shift in the mounting positions (positions and postures) of the cameras 12 to 18, continuous factors and discrete factors. Since a continuous search can be performed by crossing each seed and a discrete search can be performed by mutation of each seed, calibration of the mounting positions of the cameras 12 to 18 can be performed efficiently and optimally. It becomes possible.

As mentioned above, although one embodiment of the present invention was described, the present invention is not limited to the above-mentioned embodiment, and can take various modes without departing from the gist of the present invention.
For example, as a calibration pattern used in the automatic calibration process, for example, a feature pattern of a specific image captured in the overlapping region 6 when the vehicle is parked or stopped, such as an image of a place where the user frequently parks the vehicle 2 May be registered for each overlapping region 6.

  In the above embodiment, the automatic calibration process is described as calibrating the mounting positions of the cameras 12 to 18 using a genetic algorithm, but genetic programming, evolutionary strategy, evolutionary programming, etc. An evolutionary algorithm different from the genetic algorithm may be used.

  On the other hand, in the above embodiment, the vehicle imaging system that uses the four cameras 12 to 18 to generate a bird's-eye view image of the surroundings from above the vehicle 2 and displays the image on the display 30 has been described. However, the present invention is similar to the above embodiment as long as it is an imaging system that generates a display image using a plurality of in-vehicle cameras, such as an imaging system that generates a three-dimensional image using a pair of cameras. Can be applied.

  DESCRIPTION OF SYMBOLS 2 ... Vehicle, 4 ... Calibration reference pattern, 6 ... Overlapping area, 8 ... Calibration pattern, 9 ... Predetermined area, 12-18 ... Camera, 20 ... ECU, 22 ... Image converter, 24 ... Image recognition engine, 26 ... display control unit, 28 ... camera calibration unit, 30 ... display.

Claims (3)

A plurality of cameras (12, 14, 16, 18) mounted on the vehicle so that a part of the imaging region overlaps in order to image the vehicle periphery;
Display image generation for generating a display image obtained by viewing the surroundings of the vehicle from a predetermined viewpoint from the captured images of the plurality of cameras, using conversion data set based on reference camera parameters representing the positions and orientations of the plurality of cameras Means (22);
In a vehicle imaging system comprising: a vehicle-mounted camera calibration device (28) for calibrating positional deviations from a reference position of the plurality of cameras specified by the reference camera parameter,
A determination unit (28, S150) for determining whether or not a predetermined calibration image has been captured in an overlapping region where a part of the imaging regions of the plurality of cameras overlaps;
When the determination unit determines that the calibration image is captured, the captured image of the overlapping region is captured by a plurality of cameras that have captured the calibration image by displacing the reference camera parameter according to a predetermined evolution algorithm. The calibration data generation means (28, S160) searches for the displacement amount of the reference camera parameter that most closely matches the position and shape of the calibration image, and sets the displacement amount as calibration data of the plurality of cameras. To S200),
An on-vehicle camera calibration device characterized by comprising: The calibration data generation means includes
A plurality of seeds having a variable from a reference camera parameter of the plurality of cameras as a variable, and using a genetic algorithm as the evolution algorithm, any of crossing, mutation, and copying is performed on the plurality of seeds. For each seed after the operation, the position and shape of the calibration camera image in the overlap region obtained by the plurality of cameras when the reference camera parameter is displaced are most closely matched. A search operation for searching for a displacement amount is performed a plurality of times, and among the seeds obtained by the plurality of search operations, a seed that most closely matches the calibration image in the overlapping region is set as the calibration data. The in-vehicle camera calibration device according to claim 1.   The determination unit stores a plurality of types of calibration images set in advance, and one of the plurality of calibration images is duplicated by monitoring the display image generated by the display image generation unit. The in-vehicle camera according to claim 1 or 2, wherein it is determined whether or not a predetermined calibration image is captured in the overlapping region by determining whether or not the region exists in the region. Calibration equipment.

Images