JP7162489B2 - CALIBRATION DEVICE AND CALIBRATION METHOD - Google Patents

Description

The present invention relates to a calibration device and a calibration method.

A vehicle is equipped with a plurality of cameras that capture the surroundings of the vehicle. The images captured by each camera are converted into an overhead image, which is an image looking down from above the vehicle. A synthetic bird's-eye view image is generated by connecting a plurality of bird's-eye images based on a plurality of images obtained from the above. The composite bird's-eye view image formed in this way can express the scenery of the entire surroundings of the own vehicle with one image.

Here, when a bird's-eye view image is generated from images captured by each camera, camera parameters that specify the mounting posture (orientation) of each camera are required. The orientation of each camera is set in advance based on design values, but errors may occur in actual vehicles. When a synthetic bird's-eye view image is generated with camera parameters based on design values while such an error occurs, depending on the error, a gap may occur in the connecting part of the bird's-eye view images, or the part that originally extends in the front-back direction may occur. The power straight line becomes a slanted straight line, and an accurate synthetic bird's-eye view image cannot be generated.

In order to solve such problems, correction of camera parameters called calibration is performed. This calibration is performed, for example, on the production line of the factory where the vehicle was manufactured, when no one is on the vehicle or only one inspector of the finished vehicle is in the driver's seat and no luggage is loaded. , in a particular loading state.

However, when the user actually uses the vehicle, the loading condition of the vehicle changes depending on the number of passengers, the positions of the passengers, the presence or absence of luggage, and the like. The posture of the vehicle depends on the loading state, and the change in the posture of the vehicle changes the posture of the camera with respect to the road surface on which the synthetic bird's-eye view image is projected. That is, even if the vehicle attitude changes, the camera parameters for the vehicle do not change, but if the vehicle attitude differs from the vehicle attitude when calibration was performed, the above-described shift or the like may occur in the synthesized bird's-eye view image.

Therefore, a technique of performing calibration based on an image captured by a camera while the vehicle is running, which is the actual usage condition of the vehicle, has been disclosed (see, for example, Patent Document 1). According to this technology, since the calibration is performed with the vehicle attitude according to the actual use condition of the vehicle, it is possible to prevent the synthetic bird's-eye view image from being deviated while the vehicle is in use, and to produce an accurate synthetic bird's-eye view image. Obtainable.

JP 2016-070814 A

In the technique described in Patent Document 1 described above, lines extending linearly in the front-rear direction (white lines separating lanes, etc.) exist around the vehicle, and the imaging area of the front cameras provided at the front and rear of the vehicle. Two straight images are captured in the photographing area of the rear camera, and one straight line image is captured in the photographing area of the right side camera and the photographing area of the left side camera provided on each side of the vehicle. It is necessary.

Therefore, the calibration described in Patent Document 1 cannot be applied in situations where such a straight line does not exist in the imaging area.

SUMMARY OF THE INVENTION It is an object of the present invention to provide a calibration device and a calibration method capable of correcting camera parameters even in an environment where there is no straight line extending in the longitudinal direction of the vehicle. With the goal.

A first aspect of the present invention is an image conversion unit that converts images captured by a plurality of imaging devices each having an overlapping area in the imaging range into a bird's-eye view image by a plurality of imaging devices installed at different positions of a vehicle, and a parameter storage unit storing camera parameters set in advance corresponding to the attitude of the vehicle in the vehicle attitude of the initial state and a plurality of vehicle attitude parameters respectively corresponding to the plurality of attitudes of the vehicle, for each imaging device of generating a plurality of synthesized camera parameters that take into account the vehicle orientation by synthesizing each of the plurality of vehicle orientation parameters and the camera parameters; The similarity of each of the overlapped regions is obtained when the two images including the overlapped regions are converted into bird's-eye images by the image conversion unit, and a plurality of obtained corresponding to the plurality of synthesized camera parameters are calculated. One vehicle attitude parameter corresponding to the highest similarity among the similarities is identified, and the identified one vehicle attitude parameter and the camera parameters respectively corresponding to the plurality of imaging devices are synthesized. and a parameter setting unit that sets a synthesized camera parameter as a camera parameter for converting into the bird's-eye view image by the image conversion unit.

A second aspect of the present invention is for each of the plurality of imaging devices, wherein images captured by a plurality of imaging devices, each of which includes an overlapping area in its imaging range, are captured by a plurality of imaging devices installed at different positions of the vehicle, and are each converted into a bird's eye view image. When calibrating the camera parameters of , each of a plurality of vehicle attitude parameters respectively corresponding to a plurality of attitudes of the vehicle and each of the plurality of imaging devices are set in advance corresponding to the attitude of the vehicle in the initial state of the vehicle attitude generating a plurality of synthesized camera parameters that take into account the vehicle posture by synthesizing the obtained camera parameters, and generating two images including one of the overlapping regions based on the plurality of synthesized camera parameters; Each of the similarities of the one overlapped area is obtained when each of the overlapped areas is converted into a bird's-eye view image by the image conversion unit, and the highest similarity among the plurality of the similarities obtained corresponding to the plurality of the synthetic camera parameters. one vehicle attitude parameter corresponding to the angle of view is specified, and the synthesized camera parameters obtained by synthesizing the specified one vehicle attitude parameter and the camera parameters respectively corresponding to the plurality of imaging devices are displayed in the bird's-eye view image. It is a calibration method that is set as a camera parameter to be converted to .

According to the calibration device and calibration method of the present invention, camera parameters can be corrected even in an environment where there is no straight line extending in the longitudinal direction of the vehicle.

BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is a block diagram showing a vehicle surroundings monitoring system including a camera ECU, which is an example of implementing a calibration device and a calibration method according to the present invention; FIG. 2 is a plan view schematically showing the installation positions and shooting ranges of the cameras that make up the vehicle-mounted camera; It is a figure explaining the coordinate system used in the vehicle surroundings monitoring system of this embodiment, and is a figure which shows a world coordinate system and a vehicle coordinate system. It is a figure explaining the coordinate system used in the vehicle surroundings monitoring system of this embodiment, and is a figure which shows a camera coordinate system. 4 is a flowchart for explaining a specific processing flow of a camera ECU;

Hereinafter, specific embodiments of a calibration device and a calibration method according to the present invention will be described with reference to the drawings.

<Vehicle Surrounding Monitoring System>
FIG. 1 is a block diagram showing a vehicle surrounding monitoring system including a camera ECU 30, which is an example of implementing the calibration device and calibration method according to the present invention, and FIG. FIG. 3 is a plan view schematically showing positions and imaging ranges; The vehicle surroundings monitoring system shown in FIG. A vehicle speed sensor 20 for detecting the running speed of the own vehicle 200, a camera ECU 30 for generating a synthetic bird's-eye view image AP including the surroundings of the vehicle 200 based on the image captured by the vehicle-mounted camera 10, and the synthetic bird's-eye view image AP are displayed. and a video display device 90 for displaying images.

As shown in FIG. 2, the cameras 11 to 14 are installed at different positions on the vehicle 200 according to preset camera parameters corresponding to the attitudes (orientations) of the vehicle 200 . Each camera parameter is set in the initial loaded state of the vehicle. ) and the yaw angle γ cam1 (the rotation angle about the vertical axis orthogonal to the optical axis of the camera 10).

The initial loading state of the vehicle is, for example, the empty loading state of the vehicle 200 (the state in which there are no passengers and no cargo is loaded), and the camera parameters in the initial state are, for example, It is set by calibration performed in the final inspection process at the factory and stored in the parameter recording device 34, which will be described later.

The front camera 11 is specifically attached to the front end of the vehicle 200 and captures a forward image PO1 whose imaging range is the front region (F) of the vehicle 200 . The rear camera 12 is specifically attached to the rear end portion of the vehicle 200 and captures a rear image PO2 having a rear region (R) of the vehicle 200 as an imaging range. The right side camera 13 is specifically attached near the right door mirror of the vehicle 200 and captures a right side image PO3 whose imaging range is the right side region (SR) of the vehicle 200 . The left side camera 14 is specifically attached near the left door mirror of the vehicle 200 and captures a left side image PO4 having a left side area (SL) of the vehicle 200 as an imaging range.

Each of the cameras 11 to 14 has a wide-angle lens and has a wide angle of view of about 180 degrees. Then, as shown in FIG. 2, the front image PO1 and the right side image PO3 include an overlapping area in the right front area of the vehicle 200, and both the front image PO1 and the right side image PO3 are the overlapping areas. It includes a right front image PO13, which is an image of the area that has been moved. Similarly, the front image PO1 and the left side image PO4 include an overlapping area in the left front area of the vehicle 200, and both the front image PO1 and the left side image PO4 are images of the overlapping area. A left front image PO14 is included.

The same applies to the rear of vehicle 200. Rear image PO2 and right side image PO3 have an overlapping area in the area to the right rear of vehicle 200, and both rear image PO2 and right side image PO3 , has a right rear image PO23 that is an image of the overlapped area. The rearward image PO2 and the left side image PO4 have an overlapping area in the left rear area of the host vehicle 200, and both the rearward image PO2 and the left side image PO4 are images of the overlapping area. It has an image PO24.

Since FIG. 2 is a bird's-eye view image obtained as a plan view seen from above the own vehicle 200, in FIG. It is written in parentheses. Similarly, the rear image PO2, the right side image PO3, and the left side image PO4 are indicated in parentheses with respect to the corresponding rear bird's-eye view image P2, right side bird's-eye view image P3, and left side bird's-eye view image P4, respectively. The actual front image PO1, rear image PO2, right side image PO3, and left side image PO4 have distortion such that the image of the subject becomes smaller as the distance from the in-vehicle camera 10 increases.

<Camera ECU>
The camera ECU 30 includes an arithmetic processing device 31 , a parameter recording device 34 (parameter storage section), and I/O ports 37 , 38 and 39 . The I/O port 37 receives an image signal representing the front image PO1, an image signal representing the rear image PO2, an image signal representing the right side image PO3, and an image signal representing the left side image PO4 from each of the cameras 11 to 14 and performs arithmetic processing. A terminal for inputting to the device 31, an I/O port 38 is a terminal for inputting a vehicle speed signal representing the running speed of the own vehicle 200 from the vehicle speed sensor 20 to the processing unit 31, and an I/O port 39 is for inputting the processing unit 31 A terminal for outputting an image signal representing the synthesized bird's-eye view image AP generated by the arithmetic processing unit 31 to the image display device 90 .

The parameter recording device 34 stores camera parameters (four) corresponding to each of the cameras 11 to 14 and a plurality of preset vehicle attitude parameters respectively specifying a plurality of vehicle attitudes of the own vehicle 200. there is

3 and 4 are diagrams for explaining the coordinate system used in the vehicle surrounding monitoring system of this embodiment, FIG. 3 is a diagram showing the world coordinate system and the vehicle coordinate system, and FIG. 4 is a diagram showing the camera coordinate system. be.

The world coordinate system and the vehicle coordinate system are defined by left-handed coordinates and a left-handed screw rotation direction, and as shown in FIG. ), the z-axis is the vertical direction (upward is the positive direction), and the origin O is the center of the front wheel axle of the vehicle 200 . In the Euler angle definition of the vehicle, the order of rotation is the order of roll angle→pitch angle→yaw angle (zxy) of the vehicle 200 .

The camera coordinate system is defined by left-handed system coordinates and a left-handed screw rotation direction, and as shown in FIG. Let the axis be the vertical direction (upward direction is the positive direction), and the z-axis be the camera optical axis direction (the forward direction be the negative direction). In the Euler angle definition of the vehicle-mounted camera 10, the order of rotation is the order of roll angle→pitch angle→yaw angle of the vehicle 200 (zxy).

Here, the vehicle posture parameter is the vehicle width direction of the vehicle 200 with reference to the vehicle 200 being placed on a horizontal plane in the initial loaded state (no passengers and no cargo loaded). It is specified by a combination of a pitch angle α car representing a rotation angle about the x-axis extending in the vertical direction, a roll angle γ car representing a rotation angle about the y-axis extending in the longitudinal direction of the vehicle 200 , and a subsidence amount Zcar_sz of the vehicle 200 . be.

In other words, the vehicle posture parameters are such that the pitch angle αcar, the roll angle γcar, and the amount of subduction Zcar_sz are all 0 (zero) in the above-described reference loading state, and the pitch angle αcar , the roll angle γcar and the amount of subsidence Zcar_sz, the vehicle attitude parameters are set.

The vehicle posture changes depending on the number of passengers on the vehicle 200, the weight of the passengers (including whether they are adults or children), the seating position of each passenger, the weight of luggage to be loaded, the loading location, and the like. The plurality of vehicle attitude parameters stored in the parameter recording device 34 are pitch angle αcar, roll angle γcar, and amount of subsidence corresponding to a plurality of assumed loading conditions such as the number of passengers and the weight of luggage. It is set by a combination of Zcar_sz. Specifically, the parameter recording device 34 stores, for example, 50 vehicle attitude parameters.

The camera parameters stored in the parameter recording device 34 correspond to the respective cameras 11 to 14, for a total of four. Since each camera parameter is an attitude relative to the vehicle 200, it does not change even if the vehicle attitude changes.

The arithmetic processing device 31 includes an image conversion section 32 and a parameter setting section 33 . The image conversion unit 32 receives the front image PO1, the rear image PO2, the right side image PO3, and the left side image PO4 captured by the cameras 11 to 14, which are input from the cameras 11 to 14 via the I/O port 37. are converted into a front bird's-eye view image P1, a rear bird's-eye view image P2, a right side bird's-eye view image P3, and a left side bird's-eye view image P4 obtained by looking down from above the own vehicle 200 as a plan view.

Here, when generating the bird's-eye view images P1 to P4 corresponding to the images PO1 to PO4, the arithmetic processing unit 31 generates images based on the initially set camera parameters stored in the parameter recording device . However, when performing calibration in a vehicle posture corresponding to a loading state different from the initial state, the initially set camera parameters stored in the parameter recording device 34 and, for example, 50 New calibration according to the vehicle attitude is performed by performing image conversion processing based on the synthesized camera parameters obtained by synthesizing the vehicle attitude parameters different from each other.

Specifically, when performing new calibration, the parameter setting unit 33 synthesizes the camera parameters in the initial state stored in the parameter recording device 34 and 50 vehicle posture parameters, Generate 50 synthetic camera parameters every ~14. The 50 synthetic camera parameters are camera parameters for image conversion that take into account 50 vehicle orientations assumed according to the loading state of the vehicle 200 .

In this embodiment, as will be described later, similarity is evaluated only for the forward bird's-eye image P1 and the right side bird's-eye image P3. Therefore, the parameter setting unit 33 synthesizes the camera parameters of the front camera 11 that captures the front image PO1 and the 50 vehicle attitude parameters to generate 50 synthesized camera parameters for the front camera 11. Then, the camera parameters of the right side camera 13 that captures the right side image PO3 and the 50 vehicle attitude parameters are respectively synthesized to generate 50 synthesized camera parameters for the right side camera 13 .

Next, based on the 50 synthetic camera parameters generated by the parameter setting unit 33, the image conversion unit 32 converts four overlapping regions (the region of the right front image PO13, the region of the left front image PO14, the region of the right rear image PO23 area of the left rear image PO24), for example, two images (front image PO1 and right side image PO3) including the area of the right front image PO13 are converted into the front bird's-eye view image P1 and the right side bird's-eye view image P3, respectively. Convert.

Then, the parameter setting unit 33 sets the right front bird's-eye image P13 in the front bird's-eye image P1 obtained for each synthetic camera parameter generated for the front camera 11 and each synthetic camera parameter generated for the right side camera 13. The degree of similarity between the right side bird's-eye view image P3 obtained in 1 and the right front bird's-eye view image P13 is obtained. When calculating the degree of similarity, the front bird's-eye view image P1 and the right side bird's-eye view image P3 formed with synthesis parameters corresponding to the same vehicle posture parameter are targeted. The degree of similarity is the degree of similarity between images, and is the extent to which patterns formed by the distribution of pixel values (signal values) are similar between the two right front bird's-eye view images P13.

The degree of similarity may be obtained for the entire right front bird's-eye view image P13. can be obtained as In this way, obtaining the degree of similarity only for the partial region P13a can reduce the computational load required to calculate the degree of similarity. The partial area P13a is not limited to the rectangular area shown in FIG. 2, and may be a circular area, an elliptical area, or other polygonal areas.

Further, the partial area P13a is preferably a rectangular area whose dimension (the number of pixels) along the vehicle width direction of the vehicle 200 and the dimension (the number of pixels) along the longitudinal direction of the vehicle 200 are the same. but is not limited to such rectangular regions. Also, the partial region P13a may include at least eight pixels inside.

In addition, the parameter setting unit 33 calculates, for example, an SSD (Sum of Squared Difference: The degree of similarity is evaluated using the sum of squares of differences) or SAD (Sum of Absolute Differences) as an evaluation value S. Therefore, the parameter setting unit 33 determines that the degree of similarity increases as the evaluation value S decreases, and that the degree of similarity decreases as the evaluation value S increases.

Then, the parameter setting unit 33 sets one camera parameter corresponding to the synthesized camera parameter having the smallest similarity evaluation value S (highest similarity) obtained for each synthesized camera parameter corresponding to each of the 50 vehicle posture parameters. Identify vehicle attitude parameters. It can be said that the specified vehicle attitude parameter represents the attitude of vehicle 200 at that time.

The parameter setting unit 33 causes the parameter recording device 34 to store four synthesized camera parameters obtained by synthesizing the specified vehicle posture parameters with the camera parameters of the cameras 11 to 14, respectively. Until a new calibration is performed thereafter, the arithmetic processing unit 31 is used by the image conversion unit 32 to convert the images PO1 to PO4 into the bird's-eye images P1 to P4 to generate the composite bird's-eye view image AP. 34 based on the synthetic camera parameters stored in .

The camera ECU 30 of the present embodiment can generate a highly accurate synthetic bird's-eye view image AP through calibration appropriately corresponding to the vehicle attitude even when the vehicle attitude is different from the initial state.

In the example described above, when evaluating the degree of similarity, the region of the right front bird's-eye view image P13 is used. Other overlapping regions than the region of the image P13 (for example, the region of the left front overhead image P14, the region of the right rear overhead image P23, and the region of the left rear overhead image P24) may be applied.

FIG. 5 is a flowchart for explaining a specific processing flow of the camera ECU 30. As shown in FIG. The processing of the camera ECU 30 described above will be described more specifically with reference to the flowchart shown in FIG.

First, the cameras 11 to 14 are calibrated in the initial state by a pre-shipment inspection at the vehicle manufacturing factory or a pre-delivery inspection at a sales company that sells the vehicle (step 101 (S101)). This calibration in the initial state is performed in a loaded state in which the vehicle is empty and no cargo is loaded. The parameter setting unit 33 sets camera parameters for each of the cameras 11 to 14 by calibration based on the vehicle attitude in the initial state, and causes the parameter recording device 34 to store the set four camera parameters.

Here, as changes in the vehicle posture, the rotation matrix Rcar_pitch due to changes in the pitch angle αcar of the vehicle 200 is represented by the following formula (1), the rotation matrix Rcar_roll due to changes in the roll angle γcar of the vehicle 200 is represented by the following formula (2), A translational matrix Tcar_sz due to variations in the amount of subduction Zcar_sz is defined by the following equation (3).

From the equations (1) and (2), the vehicle attitude parameter rotation matrix Rcar_mat representing the variation of the vehicle attitude due to the variation of the pitch angle αcar and the roll angle γcar of the vehicle 200 is given by the following equation (4). The vehicle attitude parameter rotation matrix Rcar_mat and the translation matrix Tcar_sz of the subsidence amount Zcar_sz of the vehicle 200 are stored in the parameter recording device 34 .

Further, when the camera parameters of the cameras 11 to 14 set in the initial state calibration are pitch angle αcam1, roll angle βcam1, and yaw angle γcam1, the rotation matrix Rcam1 of the camera parameters is defined by the following equation (5). . The rotation matrix Rcam1 of this camera parameter is also stored in the parameter recording device 34. FIG.

Arithmetic processing unit 31 receives front image PO1, rear image PO2, right side image PO3, and left side image PO4 captured by cameras 11 to 14, which are input from cameras 11 to 14 via I/O port 37. are converted into a front bird's-eye view image P1, a rear bird's-eye view image P2, a right side bird's-eye view image P3, and a left side bird's-eye view image P4 obtained by looking down from above the own vehicle 200 as a plan view. Here, when generating the bird's-eye view images P1 to P4, the arithmetic processing unit 31 performs image conversion processing based on synthesized camera parameters obtained by synthesizing camera parameters and vehicle posture parameters stored in the parameter recording device 34. conduct.

Here, since the pitch angle αcar and the roll angle γcar are both 0 in the reference loading state (initial calibration state), which is the initial state, both equations (1) and (2) are unit matrices Thus, equation (4) also becomes a unit matrix. In addition, in the reference loading state (initial calibration state), which is the initial state, the subduction amount Zcar_sz of the vehicle 200 is also 0 (zero), so the parallel matrix Tcar_sz in Equation (3) is a zero matrix. . Therefore, in the calibration in the initial state, the synthesized camera parameters obtained by synthesizing the camera parameters and the vehicle posture parameters become the same as the camera parameters as a result. Generate a bird's-eye view image.

In the actual use state of the own vehicle 200, at least the driver sits in the driver's seat, so the vehicle posture changes from the initial state of the empty state (S102). In addition, in the actual use state of the own vehicle 200, luggage may be loaded, and this also changes the vehicle posture (S102). When converting an image captured by the on-vehicle camera 10 into a bird's-eye view image due to changes in the vehicle posture, it is necessary to perform bird's-eye conversion by correcting the camera parameters with new calibration that takes into account the change in the vehicle posture.

Here, as described above, the new calibration is to generate overhead images P1 and P3 using 50 synthetic camera parameters for each of the cameras 11 and 13 and compare the magnitude relationship of the similarity evaluation value S. process. This new calibration is performed while the vehicle 200 is stopped.

Therefore, the camera ECU 30 acquires the vehicle speed of the vehicle 200 detected by the vehicle speed sensor 20 (S103), and waits until the vehicle speed of the vehicle 200 becomes 0 (zero), that is, until the vehicle 200 stops (S103). S104).

When own vehicle 200 stops (YES at S104), camera ECU 30 acquires an image from vehicle-mounted camera 10 (S105). Camera ECU 30 does not acquire an image from vehicle-mounted camera 10 until host vehicle 200 stops (NO in S104).

When the vehicle 200 stops and the camera ECU 30 acquires an image from the vehicle-mounted camera 10, the camera ECU 30 stores the vehicle attitude parameters corresponding to, for example, 50 assumed vehicle attitudes stored in the parameter recording device 34. Synthetic camera parameters are generated by synthesizing each camera parameter. There are 50 synthesized camera parameters for each camera 11, 13 corresponding to the number of vehicle attitude parameters.

A rotation matrix Rcam2 of the combined camera parameters is represented by the following equation (6).

Then, the image conversion unit 32 of the camera ECU 30 generates bird's-eye view images P1 and P3 by performing image conversion on the image acquired in step 105 (S105) using each synthesized camera parameter (S106). The parameter setting unit 33 acquires the signal values of the pixels forming the partial region P13a in the front bird's-eye view image P1 and the signal values of the pixels forming the partial region P13a in the right side bird's-eye view image P3 (S107). An evaluation value S of the degree of similarity between the partial regions P13a, P13a is calculated (S108). For example, SAD is applied as the evaluation value S, but SSD may be applied as described above.

The parameter setting unit 33 calculates the similarity evaluation value S for each of the partial regions P13a obtained for each synthesized camera parameter corresponding to each of the 50 vehicle posture parameters, compares the evaluation values S, and A composite camera parameter with the smallest evaluation value S is identified (S109).

Then, the parameter setting unit 33 identifies the vehicle attitude parameters that make up the identified combined camera parameters, and synthesizes the identified vehicle attitude parameters and the camera parameters in the initial state of the cameras 11 to 14, respectively. The synthesized camera parameters thus obtained are stored in the parameter recording device 34 (S110), and the new calibration ends.

Here, an example of a method of specifying the vehicle posture parameter that minimizes the evaluation value S in step 109 (S109) will be described.

The arithmetic processing unit 31 decomposes the rotation matrix Rcam2 of the synthesized camera parameters expressed by Equation (6) to obtain the pitch angle of the synthesized camera parameters after the pitch angle αcar and the roll angle γcar of the vehicle attitude have changed, Angles such as roll angle and yaw angle can be calculated. Since this calculation method is publicly known, the explanation is omitted.

Next, a method of calculating the translational component of the synthesized camera parameters that have changed with changes in the vehicle attitude parameters will be described. Of the translational components x, y, and z of the synthesized camera parameters, the translational component x and the translational component y are set to 0 (zero) because the amount of variation is minute even if the vehicle posture changes. On the other hand, the translational component z corresponding to the variation in the height direction cannot be said to be very small, so it is used as a calculation target.

Since the coordinates of the true value of the rotation center when the vehicle attitude changes are unknown, the rotation center of the pitch angle αcar of the vehicle 200 is Ccar_pitch (equation (7) below), and the rotation center of the roll angle γcar is Ccar_roll (equation below (8)). These rotation center coordinates do not necessarily have to be true values, and the rotation center Ccar_roll may be a temporary value with the position of the front camera 11 or the center of the vehicle 200 in the vehicle width direction as the rotation center.

Further, Tcar_sz (formula (3) above) is the translation matrix of the variation in the vehicle attitude, Tcam1 (formula (9) below) is the translation matrix of the camera parameters before the variation in the vehicle attitude, and If the translation matrix is defined as Tcam2 (equation (10) below), the translation matrix Tcam2 of the synthesized camera parameters after the change in vehicle attitude is obtained by the following equation (11).

On the right side of equation (11), the unknown matrices are Rcar_pitch, Rcar_roll, Tcar_sz. Therefore, if these three matrices are known, it is possible to obtain the translation matrix Tcam2 of the synthesized camera parameters after the change in the vehicle attitude on the left side. These three matrices are known by obtaining the pitch angle αcar, the roll angle γcar, and the subduction amount Zcar_sz.

In summary, the "angle component of the synthesized camera parameters after the change in vehicle attitude" can be uniquely specified if the pitch angle αcar and the roll angle γcar are known. The translation component (only z) of the camera parameters can be uniquely specified if the pitch angle αcar, roll angle γcar, and sinking amount Zcar_sz are known.

Next, a method of specifying the pitch angle αcar, the roll angle γcar, and the amount of subduction Zcar_sz corresponding to the vehicle posture with the smallest similarity evaluation value S will be described. These three vehicle attitude parameters are solved using an optimization technique.

As described above, the evaluation function for optimization is set to the evaluation value S in order to obtain the vehicle posture parameter that minimizes the evaluation value S of the similarity. Since this evaluation function is a multi-peak function, any one of the following three methods can be applied as a method for obtaining the global optimum solution of the multi-peak function. For example, the case where the PSO technique of (1) is applied will be described.
(1) PSO (Particle Swarm Optimization)
(2) ABC (Artificial Bee Colony)
(3) FA (Firefly Algorithm)

PSO is a method of arranging particles for evaluation in a multi-dimensional evaluation space and repeating update a specified number of times to obtain an optimum solution. Particles decide how to update based on the evaluation results of themselves and other particles. The optimal number of particles and the number of updates differ depending on the evaluation target. The PSO update method is defined by the following equations (12) and (13).

Here, x indicates the current position of the particle. In this embodiment, the pitch angle αcar, the roll angle γcar, and the sinking amount Zcar_sz are defined at this current position x. v is a velocity vector, and the amount of change for one update is defined. i is the particle number (*If the number of particles is 100, i = 1 to 100). k is the number of iterations. w is the inertia term and is generally defined to be less than 1.0. c1 and c2 are weighting coefficients and are generally defined to have values around 1.0. r1 and r2 are random numbers between 0 and 1; pbest is the best position that the individual particle has evaluated so far, gbest is the best position that the whole particle has evaluated so far.

As an example, the optimum solution is obtained by paying attention only to the pitch angle αcar. Assume that the number of particles is 50, the number of updates is 100, and the evaluation space range of the pitch angle α car is −5.0 [deg] to +5.0 [deg]. This angle range is an angle range assumed as the vehicle attitude.

First, 50 particles (current position x) are randomly defined with mutually different pitch angles α car in the angle range of −5.0 [deg] to +5.0 [deg]. Also, 50 different velocity vectors v are randomly defined in the range of 0-1.

Similar to Equation (12), 50 velocity vectors v are added to 50 particles (current positions x). Evaluation values S corresponding to all 50 pitch angles αcar of the added particles are obtained, the evaluation values S are set as the best value pbest of each particle, and the particle with the smallest evaluation value S is set as gbest. Thus, the velocity vector v and the current position x1 when k=0 are obtained from equation (13).

After that, based on the velocity vector v when k=0 and the current position x1, the number of iterations k is sequentially increased from 1, 2, . all gather around a specific pitch angle α car , and the particle with the smallest evaluation value S among the particles gathered around the specific pitch angle α car is taken as the final gbest and adopted. The pitch angle αcar of the adopted particles becomes the pitch angle αcar that constitutes the vehicle attitude to be identified.

Incidentally, when the number of iterations k is sequentially increased and the value of x is repeatedly obtained each time, the situation in which all 50 particles gather near a specific pitch angle α is finally obtained, for example, Assuming that the optimum solution is 1.0 [deg], the pitch angle α car shows a random value at the beginning when the number of repetitions is small, but as the number of repetitions increases, the pitch angle α car becomes the optimum solution. It is in a state of convergence around 0 [deg] (for example, 0.997, 0.998, 0.9998, 1.0001, 1.0003 [deg]).

In the above description, only the pitch angle αcar is applied as the current position x. Each optimum solution for the loading amount Zcar_sz can be determined, thereby simultaneously determining the three parameters that specify the vehicle attitude. As described above, the arithmetic processing unit 31 can obtain the vehicle attitude parameter corresponding to the smallest evaluation value S (highest similarity).

Then, the parameter setting unit 33 synthesizes the calculated vehicle attitude parameter with the four camera parameters corresponding to the respective cameras 11 to 14 to obtain four synthesized camera parameters. At this time, for the angle components (pitch angle, roll angle, yaw angle) of the synthesized camera parameters, the values obtained by Equation (4) may be used. As for the translation components (x, y, z) of the synthetic camera parameters, x and y continue to adopt the design values or the values set in the initial state calibration at the vehicle 200 manufacturing factory or sales company. , z may adopt the amount of subsidence Zcam2 in the translational matrix Tcam2 of the synthesized camera parameters after the change in the vehicle attitude obtained by the equation (11).

As described above in detail, according to the camera ECU 30 of the present embodiment, camera parameters can be corrected even in an environment where there is no straight line extending in the front-rear direction of the vehicle 200 .

Further, according to the camera ECU 30 of the present embodiment, four areas (the area of the right front bird's-eye image P13, the area of the left front bird's-eye image P14, the area of the right rear bird's-eye image P23, Since the vehicle posture parameter can be specified based on the similarity of only one overlapping region (for example, the region of the right front bird's-eye image P13) of the left rear bird's-eye image P24), all four overlapping regions can be identified. , the computational load can be reduced compared to the case of evaluating the similarity of the images.

Further, according to the camera ECU 30 of the present embodiment, the parameter recording device 34 specifies combinations of the pitch angle αcar, the roll angle γcar, and the amount of subduction Zcar_sz corresponding to a plurality (for example, 50) of vehicle attitudes of the vehicle 200. Since the vehicle attitude parameters to be used are stored in advance, there is no need to calculate and set the plurality of vehicle attitude parameters each time a new calibration is performed, and the calculation load can be reduced. Note that the number of vehicle attitude parameters specified by the combination of the pitch angle αcar, the roll angle γcar, and the amount of subduction Zcar_sz corresponding to the vehicle attitude stored in the parameter recording device 34 is not limited to 50, and is less than 50. , or the number may exceed 50, and can be set according to the calculation load, the calculation cost, and the like.

10 vehicle-mounted camera 20 vehicle speed sensor 30 camera ECU
31 Arithmetic processing device 32 Image conversion unit 33 Parameter setting unit 34 Parameter recording device P1 Front bird's-eye view image P13 Right front bird's-eye view image P13a Partial region P3 Right side bird's-eye view image

Claims (4)

an image conversion unit that converts each of the images captured by a plurality of imaging devices installed at different positions of the vehicle into a bird's-eye view image including overlapping areas in the imaging range;
A parameter storage unit storing camera parameters preset corresponding to attitudes of the vehicle in an initial vehicle attitude and a plurality of vehicle attitude parameters respectively corresponding to the plurality of attitudes of the vehicle, for each of the plurality of imaging devices. When,
generating a plurality of synthesized camera parameters that take into account the vehicle orientation by synthesizing each of the plurality of vehicle orientation parameters and the camera parameters; The similarity of each of the overlapped regions is obtained when two images including the regions that are overlapped are converted into bird's-eye images by the image conversion unit, and a plurality of obtained corresponding to the plurality of synthesized camera parameters are calculated. One vehicle attitude parameter corresponding to the highest similarity among the similarities is identified, and the identified one vehicle attitude parameter and the camera parameters respectively corresponding to the plurality of imaging devices are synthesized. and a parameter setting unit that sets synthetic camera parameters as camera parameters for converting into the bird's-eye view image by the image conversion unit. 2. The calibration device according to claim 1, wherein the vehicle attitude parameter is specified by a combination of pitch angle, roll angle and subduction amount of the vehicle. The parameter storage unit stores the vehicle attitude parameter specified by the combination of the pitch angle, the roll angle, and the amount of sagging, wherein at least one of the pitch angle, the roll angle, and the amount of sagging is different. 3. The calibration device according to claim 2, wherein a plurality of combinations of said vehicle posture parameters are stored in advance. When calibrating the camera parameters for each of the plurality of imaging devices, each of which is captured by a plurality of imaging devices installed at different positions on the vehicle and whose imaging range includes areas that overlap each other and is converted into a bird's-eye view image. ,
Each of a plurality of vehicle attitude parameters respectively corresponding to a plurality of attitudes of the vehicle and a camera parameter set in advance corresponding to the attitude with respect to the vehicle in the initial state of the vehicle attitude for each of the plurality of imaging devices are synthesized. , generate multiple synthetic camera parameters that take into account the vehicle posture,
Based on the plurality of synthetic camera parameters, each similarity of the one overlapping area when two images including one overlapping area among the overlapping areas are converted into a bird's-eye view image respectively . ask,
identifying one vehicle posture parameter corresponding to the highest degree of similarity among the plurality of degrees of similarity obtained corresponding to the plurality of synthetic camera parameters;
A calibration method of setting the synthesized camera parameters obtained by synthesizing the one specified vehicle posture parameter and the camera parameters respectively corresponding to the plurality of imaging devices as camera parameters for converting into the bird's-eye view image.

Images