JP2018157496A - Calibration device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a calibration device capable calibrating a camera parameter under an environment where a liner feature part is not existed.SOLUTION: A calibration device 116 includes: an image generation part 118 that converts a plurality of frames obtained by imaging periphery by a plurality of cameras that is attached to a vehicle traveling on a road and images the periphery of the vehicle to overhead images, and generates one composite image by composing a plurality of overhead images; a locus line acquirement part 119 that acquires a locus line obtained by linearly connecting both coordinate points corresponded to a position of a specified feature material that had entered into one overlapping composite image obtained by overlapping an individual composite image generated at a different time in the image generation part 118; and a parameter calibration part 121 that calibrates a camera parameter in regards to an attachment state of the plurality of cameras so as to extend the locus line acquired in the locus line acquirement part 119 along a traveling direction of the vehicle.SELECTED DRAWING: Figure 7

Description

  The present invention relates to a calibration apparatus.

  Conventionally, each image captured by a plurality of cameras attached to the periphery of a vehicle is converted into a bird's-eye view image, and each bird's-eye view image is synthesized to generate one synthesized image. In order to generate a composite image from each bird's-eye view image, camera parameters relating to the mounting position and angle of each camera are required.

  By the way, each camera is attached to the vehicle at a position and an angle according to the design value, but an error may occur at that time. When the overhead image is generated with the camera parameter corresponding to the set value in spite of such an error, a composite image that is shifted by the error is generated. In order to solve such a problem, correction of camera parameters called calibration or calibration is performed.

  This calibration is a condition that reproduces a specific riding condition such as an empty vehicle state where no one is on the vehicle at the time of shipment of the vehicle, or a user is in the driver's seat. Done in

  However, when the user actually uses the vehicle, the riding state changes depending on the number of passengers, the place where the user sits, the loading position of the luggage, and the like. When the riding state changes, the posture of the vehicle also changes, and accordingly, the mounting state of the camera with respect to the ground also changes. That is, although the camera parameter does not change, there is a problem in that the composite image is deviated because the posture of the vehicle is not the posture of the vehicle when the calibration is performed.

  For such a problem, Patent Document 1 discloses a technique for correcting camera parameters while a vehicle is traveling.

  The technique disclosed in Patent Document 1 is a calibration method using an image obtained by capturing a linear feature amount (a feature amount having a linear shape) in a longitudinal direction such as a white line of a road with a camera. In this calibration method, camera parameters are calibrated by estimating parameters representing the attitude of the vehicle based on the captured image. The step of estimating the parameter representing the posture of the vehicle includes the step of estimating the pitch angle of the vehicle and the step of simultaneously estimating the roll angle and height of the vehicle.

JP 2016-70814 A

  However, the calibration method disclosed in Patent Document 1 is based on the premise that a linear feature such as a white line on a road is imaged with a camera. For this reason, in an environment where no linear feature exists, a parameter representing the attitude of the vehicle cannot be estimated, and the camera parameter cannot be calibrated.

  The present invention has been made in view of the above problems, and an object of the present invention is to provide a calibration device that can calibrate camera parameters even in an environment where no linear feature exists.

  A calibration device according to the present invention converts a plurality of frames obtained by imaging the periphery by a plurality of cameras attached to a vehicle traveling on a road surface and imaging the periphery of the vehicle into an overhead image. An image generation unit that generates a single composite image by combining the plurality of overhead images, and a position of a specific feature reflected in each composite image generated at different times by the image generation unit A trajectory line acquisition unit that acquires a trajectory line obtained by connecting the coordinate points corresponding to each other with a straight line between the individual composite images, and the trajectory line acquired by the trajectory line acquisition unit is the travel of the vehicle And a parameter calibration unit that calibrates camera parameters related to the mounting state of the plurality of cameras so as to extend along the direction.

  According to the calibration device according to the present invention configured as described above, it is possible to calibrate camera parameters even in an environment where no linear feature exists.

1 is a block diagram showing a schematic configuration of a calibration system equipped with a calibration apparatus according to an embodiment of the present invention. It is explanatory drawing which shows an example of the coordinate angle of a vehicle, and the rotation angle around each axis. It is the flowchart explaining the procedure until it implements the calibration at the time of product shipment and use. It is a figure which shows an example of the superimposition synthetic | combination image before implementation of a calibration in use. It is a figure which shows an example of the superimposition synthetic | combination image in the implementation process of a calibration at the time of use. It is a figure which shows an example of the superimposition synthetic | combination image after implementation of calibration at the time of use. 1 is a block diagram illustrating a schematic configuration of a calibration apparatus according to an embodiment of the present invention. FIG. 8 is a flowchart illustrating calibration processing by the calibration apparatus shown in FIG. 7.

  Hereinafter, embodiments of a calibration apparatus according to the present invention will be described with reference to the drawings.

<Schematic configuration of calibration system>
FIG. 1 is a block diagram showing a schematic configuration of a calibration system equipped with a calibration apparatus according to the present invention. The camera calibration system 100 mainly includes four cameras 111, 112, 113, 114, an arithmetic device 101, a RAM (Random Access Memory) 102, a ROM (Read Only Memory) 103, and a display device 104. The vehicle speed sensor 105, the steering angle sensor 106, the yaw rate sensor 107, the input device 108, and the communication device 109 are provided.

  Each of the cameras 111 to 114 is attached to the vehicle 1 traveling on the road surface and images the periphery of the vehicle 1. The cameras 111 to 114 are installed on the front, rear, left and right of the vehicle 1. For example, the front and rear cameras are attached to the vehicle body near the license plate, and the left and right cameras are attached to the lower part of the side mirror. Here, the camera 111 is attached to the front of the vehicle 1, the camera 112 is attached to the rear of the vehicle 1, the camera 113 is attached to the left of the vehicle 1, and the camera 114 is attached to the right of the vehicle 1 (see FIGS. 4 to 6). . Each of the cameras 111 to 114 is attached so that the optical axis faces obliquely downward from the horizontal direction.

  Each of the cameras 111 to 114 is attached according to known design information, but in reality, an attachment error has occurred, and this error is unknown. In addition, a wide-angle fisheye camera is employed for each of the cameras 111 to 114 so that an image of the entire periphery of the vehicle 1 can be acquired. Four images captured by the cameras 111 to 114 are output to the arithmetic device 101.

  The vehicle speed sensor 105 is a sensor that detects the vehicle speed. The steering angle sensor 106 is a sensor that detects the steering angle. The yaw rate sensor 107 is a sensor that detects the yaw rate. Information detected by each sensor is output to the arithmetic device 101 and used in arithmetic processing in the arithmetic device 101.

  The input device 108 is a device that accepts user operations such as switches and buttons, and is used for turning on / off the calibration function, initializing the calibration result, changing the calibration method, and the like. Various types of information input to the input device 108 through user operations are output to the arithmetic device 101.

  The communication device 109 is a device used for communication with an external device (not shown). The communication device 109 receives various types of information from an external device, and outputs the received various types of information to the arithmetic device 101, and outputs various types of information calculated by the arithmetic device 101 to the external device.

  In the RAM 102, numerical data required in the arithmetic processing process in the arithmetic device 101, program variables for processing results during the arithmetic processing, and the like are written. Also, the written data is read out as necessary in the arithmetic processing process of the arithmetic unit 101 and used for arithmetic processing. The RAM 102 also stores image data captured by the cameras 111 to 114.

  In the ROM 103, for example, a program for executing calibration and information used without rewriting among information necessary for the program are stored in advance. In the ROM 103, for example, external parameters that are camera parameters relating to the mounting state of each camera 111 to 114, such as the installation position and angle design value of each camera, the focal length, the pixel size, the optical axis center, and the distortion function of each camera And other internal parameters that are camera parameters are stored.

  The arithmetic device 101 receives various information transmitted from each of the cameras 111 to 114, the vehicle speed sensor 105, the rudder angle sensor 106, the yaw rate sensor 107, the input device 108, the communication device 109, and the like, and performs calculation processing based on a program or the like. It is an apparatus to be implemented. The arithmetic device 101 includes a calibration device 116. The configuration of the calibration device 116 will be described later.

  The display device 104 receives the processing result of the arithmetic device 101 and presents the processing result to the user using a display or the like. For example, a combined image generated by viewpoint-converting and combining four images of the cameras 111 to 114 is displayed to the user. The display device 104 can also switch display contents according to the output of the arithmetic device 101, for example, displaying only the image of the rear camera 112.

<Vehicle coordinate axes and rotation angles around each axis>
FIG. 2 is an explanatory diagram illustrating an example of a coordinate axis of a vehicle and a rotation angle around each axis. For convenience of explanation, in FIG. 2, an XYZ three-dimensional coordinate system is defined, and the center of gravity G of the vehicle 1 is defined as the origin O.

  In this example, a horizontal plane parallel to the ground EG including the origin O is defined as the XY plane, the traveling direction of the vehicle 1 is defined as the X-axis direction, and a straight line passing through the origin O and perpendicular to the X-axis is defined as the Y-axis. It is defined as A plane perpendicular to the ground EG including the origin O is defined as the XZ plane, and a straight line passing through the origin O and perpendicular to the X axis is defined as the Z axis.

  Rolling in the figure refers to rotation about the X axis, pitching refers to rotation about the Y axis, and yawing refers to rotation about the Z axis. The parameters of the rotation angles around the X, Y, and Z axes are referred to as the roll angle of the vehicle 1, the pitch angle of the vehicle 1, and the yaw angle of the vehicle 1, respectively. In addition, a height H in the figure is a height from the ground EG to the center of gravity G (origin O) of the vehicle 1 and indicates the height of the vehicle 1.

<Procedure until calibration at the time of product shipment and use>
FIG. 3 is a flowchart for explaining the procedure until the calibration at the time of product shipment and use. As shown in the figure, calibration S205 at the time of product shipment is performed after, for example, camera mounting S201, riding state reproduction S202, position adjustment S203, and calibration chart imaging S204.

  In camera attachment S201, each camera 111-114 is attached to a vehicle body, and the riding state reproduction S202 reproduces the state where the passenger is on the vehicle 1. For example, a boarding state that is assumed to actually occur is reproduced by, for example, an inspector having a weight of 60 kg actually getting on the driver's seat or mounting a corresponding baggage. The reason why the riding state reproduction S202 is performed is that the posture of the vehicle 1 changes depending on the riding state, and the angle of each of the cameras 111 to 114 with respect to the ground changes accordingly.

  When the angles and positions of the cameras 111 to 114 are different from the designed mounting state, the boundary between the overhead images captured by the cameras 111 to 114 and subjected to viewpoint conversion in the superimposed composite image (see FIGS. 4 to 6) ( Hereinafter, a shift occurs in the image boundary). Therefore, it is necessary to perform calibration in consideration of the changed angle and the like. By this riding state reproduction S202, the camera mounting state according to the corresponding vehicle posture is reproduced.

  In the position adjustment S203, the vehicle 1 and the calibration chart are adjusted so as to have a predetermined positional relationship. Here, calibration is performed after the positional relationship between a calibration chart (for example, a pattern printed on paper, a plate, etc.) and the camera is set to a predetermined positional relationship.

  In the calibration chart imaging S <b> 204, the calibration chart is imaged by the cameras 111 to 114 attached to the vehicle 1.

  In calibration S205 at the time of product shipment, calibration is performed using images captured by the cameras 111 to 114. At that time, for example, when each camera 111 to 114 is mounted as designed and the calibration chart is installed at a specified position, the position where the calibration chart should be displayed is calculated and actually captured The camera parameters used for image conversion in the program are adjusted so that the calibration chart in the image is displayed at the position to be reflected.

  For this calibration S205, for example, a technique capable of performing calibration even if the positional relationship between the calibration chart and the vehicle 1 is indeterminate may be used. In this case, the procedure of the position adjustment S203 is not necessary.

  By the way, when the calibration is performed in such a procedure, the calibration is performed on the assumption of a specific riding state as in the riding state reproduction S202. Therefore, if the actual boarding state is different from the boarding state at the time of calibration at the time of product shipment, there is a problem that the image boundary in the superimposed composite image (see FIGS. 4 to 6) is shifted. Therefore, it is not sufficient to perform calibration only in a specific boarding state. For example, it is necessary to perform calibration every time depending on the boarding state.

  Therefore, calibration according to the boarding state at that time is performed even during use, such as during travel or parking, after the product is shipped. That is, as shown in FIG. 3, after performing calibration S205 at the time of product shipment, calibration S206 at the time of use is performed.

  In use calibration S206, in order to deal with the above-described problem, the calibration device 116 (FIG. 1) allows the cameras 111 to 114 to capture images of the surrounding environment when the vehicle 1 is in use, such as running or parked. Each time calibration according to the boarding state is performed using the information.

<Outline of calibration S206 during use>
With reference to FIG. 4 to FIG. 6, the calibration S206 in use by the calibration device 116 (FIG. 1) will be outlined. 4 to 6, for convenience of explanation, calibration processing using a single superimposed composite image in which individual composite images generated corresponding to different times t1, t2,..., T7 are superimposed. Will be explained.

  However, the actual calibration process is performed using a trajectory line obtained as a result of tracking coordinate points corresponding to the position of a specific feature (hereinafter referred to as feature points as appropriate) across individual composite images. Just do it. As a specific feature, for example, a speed limit sign drawn on a road or a corner portion (point) of a road sign such as a traffic guide sign drawn on a road is used.

  FIG. 4 shows an example of the superimposed composite image before the calibration at the time of use. After the calibration at the time of product shipment in the factory by the processing of S201 to S205 in FIG. 6 shows an example of a superimposed composite image when the boarding state of the vehicle changes from the boarding state assumed in the calibration at the time of product shipment.

  As shown in FIG. 4, locus lines Fr 1, Fr 2, Lt 1, Rt 2, Rr 1, Rr 2 (trajectories of feature points in the illustrated example) are reflected on both sides of the vehicle 1.

  The two trajectory lines Fr1 and Fr2 are obtained by connecting the feature points at different times t1 and t2 arranged on the superimposed composite image with straight lines. That is, the trajectory line Fr1 is obtained by connecting the feature points Fr1_t1 at time t1 and Fr1_t2 at time t2 that are arranged on the superimposed composite image with an approximate straight line. The locus line Fr2 is obtained by connecting the feature point Fr2_t1 at the time t1 and the feature point Fr2_t2 at the time t2, which are arranged on the superimposed composite image, with an approximate straight line.

  Therefore, in order to draw the two trajectory lines Fr1 and Fr2 on the superimposed composite image, the feature point Fr1_t1 and the feature point Fr2_t1 are included in the bird's-eye view image captured by the front camera 111 at time t1 and subjected to viewpoint conversion. However, it is necessary that the feature point Fr1_t2 and the feature point Fr2_t2 are included in the bird's-eye view image captured by the front camera 111 and converted in viewpoint at time t2.

  For example, if the vehicle 1 is traveling at a speed of 100 km / h, and the total length of the vehicle is about 4.5 m to 5 m, the distance that the feature point moves between time t1 and time t2 is about 1.35 m, that is, forward If the imaging cycle of the camera 111 is set to about 0.05 seconds, both the feature points Fr2_t1 and Fr2_t1 at time t1 and the feature points Fr2_t2 and Fr2_t2 at time t2 may be located within the same overhead image range. It is considered possible. Thereby, it is considered that two trajectory lines Fr1 and Fr2 can be drawn on the superimposed composite image. The two trajectory lines Fr1 and Fr2 need to be parallel to each other, but the relative relationship between the vehicle 1 and the trajectory line, such as the relative angle between the vehicle 1 and the trajectory line, the distance to the trajectory line, or the like is not necessary. It is.

  The two trajectory lines Rr1 and Rr2 are obtained by connecting the feature points at different times t6 and t7 arranged on the superimposed composite image with straight lines. That is, the trajectory line Rr1 is obtained by connecting the feature points Rr1_t6 at time t6 and Rr1_t7 at time t7, which are arranged on the superimposed composite image, with an approximate straight line. The locus line Rr2 is obtained by connecting the feature point Rr2_t6 at the time t6 and the feature point Rr2_t7 at the time t7, which are arranged on the superimposed composite image, with an approximate straight line.

  Therefore, in order to draw the two trajectory lines Rr1 and Rr2 on the superimposed composite image, the feature point Rr1_t6 and the feature point Rr2_t6 are included in the bird's-eye view image captured by the rear camera 112 at time t6 and subjected to viewpoint conversion. However, it is necessary that the feature point Rr2_t6 and the feature point Rr2_t7 are included in the bird's-eye view image captured by the rear camera 112 and converted in viewpoint at time t7.

  For example, if the vehicle 1 is traveling at a speed of 100 km / h, and the total length of the vehicle is about 4.5 m to 5 m, the distance that the feature point moves between time t6 and time t7 is about 1.35 m, that is, If the imaging cycle of the camera 112 is set to about 0.05 seconds, both the feature points Rr1_t6 and Rr2_t6 at time t6 and the feature points Rr1_t7 and Rr2_t7 at time t7 may be located within the same overhead image range. It is considered possible. Thereby, it is considered that two trajectory lines Rr1 and Rr2 can be drawn on the superimposed composite image. The two trajectory lines Rr1 and Rr2 need to be parallel to each other, but the relative relationship between the vehicle 1 and the trajectory line, such as the relative angle between the vehicle 1 and the trajectory line, the distance to the trajectory line, or the like is not necessary. It is.

  One trajectory line Lt1 is obtained by connecting feature points at different times t3 to t5 arranged on the superimposed composite image with straight lines. That is, the trajectory line Lt1 is obtained by connecting the feature point Lt_t3 at time t3, the feature point Lt_t4 at time t4, and the feature point Lt_t5 at time t5, which are arranged on the superimposed composite image, with an approximate straight line.

  Therefore, in order to draw one trajectory line Lt1 on the superimposed composite image, it is not sufficient that the feature point Lt_t3 is included in the overhead image captured by the left camera 113 and converted in viewpoint at time t3. The feature point Lt_t4 is included in the overhead view image captured by the left camera 113 at time t4 and converted in viewpoint, or the feature point is included in the overhead image captured by the left camera 113 at time t5 and converted in viewpoint. The point Lt_t5 needs to be included.

  For example, if the vehicle 1 is traveling at a speed of 100 km / h, and the total length of the vehicle is about 4.5 m to 5 m, the distance that the feature point moves between time t3 and time t5 is about 1.35 m, that is, left If the imaging cycle of the one-way camera 113 is set to about 0.05 seconds, all of the feature point Lt_t3 at time t3, the feature point Lt_t4 at time t4, and the feature point Lt_t5 at time t5 are within the same overhead image range. It is thought that it can be located in.

  Further, if the distance that the feature point moves between time t3 and time t5 is set to about 2.7 m, that is, the imaging cycle of the left camera 113 is set to about 0.1 second, the feature point Lt_t3 at time t3. It is considered that both the feature point Lt_t5 at time t5 can be located within the same overhead view image. Thereby, it is considered that one trajectory line Lt1 can be drawn on the superimposed composite image.

  One locus line Rt2 is obtained by connecting feature points at different times t3 to t5 arranged on the superimposed composite image with straight lines. That is, the trajectory line Rt2 is obtained by connecting the feature point Rt_t3 at time t3, the feature point Rt_t4 at time t4, and the feature point Rt_t5 at time t5, which are arranged on the superimposed composite image, with an approximate straight line.

  Therefore, in order to draw one trajectory line Rt2 on the superimposed composite image, it is not sufficient that the feature point Rt_t3 is included in the overhead image captured by the right camera 114 at the time t3 and subjected to viewpoint conversion. The feature point Rt_t4 is included in the bird's-eye view image captured by the right camera 114 at time t4 and converted in viewpoint, or the feature point is included in the bird's-eye view image captured by the right camera 114 and converted in viewpoint at time t5. The point Rt_t5 needs to be included.

  For example, if the vehicle 1 is traveling at a speed of 100 km / h, and the total length of the vehicle is about 4.5 m to 5 m, the distance that the feature point moves between time t3 and time t5 is about 1.35 m, that is, right If the imaging cycle of the direction camera 114 is set to about 0.05 seconds, all of the feature point Rt_t3 at time t3, the feature point Rt_t4 at time t4, and the feature point Rt_t5 at time t5 are within the same range of the overhead view image. It is thought that it can be located in.

  Further, if the distance that the feature point moves between time t3 and time t5 is set to about 2.7 m, that is, the imaging cycle of the right camera 114 is set to about 0.1 second, the feature point Rt_t3 at time t3. It is considered that both the feature point Rt_t5 at time t5 can be located within the same overhead image range. Thereby, it is considered that one trajectory line Rt2 can be drawn on the superimposed composite image.

  In the superimposed composite image shown in FIG. 4, the trajectory lines Fr1, Fr2, Lt1, Rt2, Rr1, and Rr2 are displayed, but the boarding state of the vehicle 1 is different from the boarding state assumed in advance. For this reason, the postures (angles and positions) of the cameras 111 to 114 with respect to the ground are changed, and as shown in FIG. 4, the locus lines are displaced at the image boundaries in the superimposed composite image.

  Therefore, in use calibration, calibration is performed in such a situation so that the superimposed composite image is corrected so that there is no deviation between the trajectory lines. In this embodiment, calibration can be performed in as many scenes as possible, and in order to perform stable calibration, calibration is performed using each of the trajectory lines Fr1, Fr2, Lt1, Rt2, Rr1, and Rr2. To do.

  More specifically, in this in-use calibration, the parameters of the cameras 111 to 114 are not directly estimated, but the posture of the vehicle 1, that is, the parameter representing the posture of the vehicle 1 is estimated, and the parameters representing the posture of the vehicle 1 are estimated. The calibration is performed by estimating the camera parameters of each of the cameras 111 to 114.

  The step of estimating the attitude of the vehicle 1 is divided into a step of estimating the pitch angle of the vehicle 1 and a step of simultaneously estimating the roll angle and height of the vehicle 1. In the step of estimating the pitch angle of the vehicle 1, the pitch angle of the vehicle 1 is estimated based on the parallelism of the trajectory lines captured by the front camera 111 and the rear camera 112 installed in the front and rear of the vehicle 1.

  When the pitch angle of the vehicle 1 is estimated and the camera parameters are corrected, a superimposed composite image as shown in FIG. 5 is obtained. Here, the camera parameters are corrected so that the trajectory lines Fr1 and Fr2 are parallel on the superimposed composite image. Similarly, the camera parameters are corrected so that the trajectory lines Rr1 and Rr2 are parallel on the superimposed composite image.

  Thereafter, in the step of simultaneously estimating the roll angle and height of the vehicle 1, the roll angle and height of the vehicle 1 are estimated so that there is no deviation of the locus line at the image boundary between the overhead images in the superimposed composite image. To do.

  As a result of estimating the roll angle and height of the vehicle 1 in this way and correcting the camera parameters, estimating the camera parameters of the cameras 111 to 114 in accordance with changes in the vehicle posture, and performing calibration, FIG. As shown in FIG. 6, a superimposed composite image with no deviation of the trajectory line is obtained at the image boundary between the overhead images in the superimposed composite image.

<Embodiment of Calibration Apparatus>
FIG. 7 shows a schematic configuration of a calibration apparatus according to an embodiment of the present invention, and shows an internal configuration of a calibration apparatus that can embody the above-described calibration in use.

  FIG. 8 is a flowchart for explaining calibration processing by the calibration apparatus shown in FIG. Note that the calibration process shown in FIG. 7 is executed by loading a program stored in advance in the ROM 103.

  The calibration device 116 is mounted on the vehicle 1 and calibrates each of the cameras 111 to 114 that images the periphery of the vehicle. The calibration device 116 includes a calibration execution determination unit 117, an image generation unit 118, a locus line acquisition unit 119, a steering angle acquisition unit 120, and a parameter calibration unit 121.

  The calibration execution determination unit 117 determines whether it is necessary to perform camera calibration (S801 in FIG. 8). For example, with respect to a scene in which specific feature points are reflected in all four images captured by the cameras 111 to 114, a deviation occurs in the locus line at the image boundary between the overhead images in the superimposed composite image. It is automatically determined whether or not it is necessary to perform camera calibration. Here, whether or not there is a deviation in the trajectory line is determined by, for example, recognizing the trajectory of the feature point from the image using a known technique, calculating the position in the image, and specifying the image boundary position in advance. It is possible to determine whether or not the amount of deviation of the position at the point of time is measured and whether or not the amount of deviation exceeds a predetermined threshold.

  When the calibration execution determination unit 117 determines that camera calibration needs to be performed (YES in S801 in FIG. 8), the image generation unit 118 first has four cameras 111-1 attached to the vehicle 1. A plurality of images obtained by imaging the periphery of the vehicle 1 at 114 are acquired from the RAM 102 (S802 in FIG. 8). The RAM 102 stores images of several past frames. On the other hand, when it is determined that the calibration needs to be performed (NO in S801 in FIG. 8), the calibration process by the calibration apparatus ends.

  In addition, if there is a synchronization shift in the images captured by the cameras 111 to 114, this will appear as a calibration error, so the images captured by the cameras 111 to 114 are stored in the RAM 102 in complete synchronization. It is desirable that there is a mechanism for whether or not the image shift time can be acquired.

  Next, the image generation unit 118 converts the plurality of frames into a bird's-eye view image and synthesizes the plurality of bird's-eye view images to generate one composite image (S803 in FIG. 8). The RAM 102 stores individual composite images generated at different times by the image generation unit 118.

  The trajectory line acquisition unit 119 first extracts a specific feature reflected in a single superimposed composite image obtained by superimposing individual composite images acquired from the RAM 102 (S804 in FIG. 8). As a feature extraction method, a method such as Harris corner detection is known. Next, the trajectory line acquisition unit 119 acquires a trajectory line obtained by connecting the coordinate points corresponding to the positions of specific features on the superimposed composite image with straight lines (S805 in FIG. 8). The trajectory line acquisition unit 119 specifies a series of feature points by tracking coordinate points (feature points) corresponding to the positions of specific feature objects. As a feature point tracking method, a method such as the Lucas-Kanade method is known. The trajectory line acquisition unit 119 acquires a trajectory line by linearly approximating the feature point series.

  The steering angle acquisition unit 120 acquires information indicating the steering angle detected by the steering angle sensor 106 (S806 in FIG. 8).

  The parameter calibration unit 121 determines whether or not the vehicle 1 is traveling straight on the basis of the information indicating the steering angle acquired by the steering angle acquisition unit 120 (S807 in FIG. 8). When it is determined that the vehicle 1 is traveling straight (YES in S807 in FIG. 8), the parameter calibration unit 121 performs camera calibration using the trajectory line acquired by the trajectory line acquisition unit 119. . When it is not determined that the vehicle 1 is traveling straight ahead (NO in S807 of FIG. 8), the process of S807 is repeated.

  In this calibration process, the camera parameters of the cameras 111 to 114 are not directly estimated, but the attitude of the vehicle 1 is estimated, and all camera parameters are estimated via the attitude of the vehicle 1. Since the cameras 111 to 114 are attached to the rigid vehicle body, when the posture change of the vehicle body occurs, the angles and positions of the cameras 111 to 114 also change in conjunction with the change.

  For example, if an occupant gets on the front seat of the vehicle 1 and the front side of the vehicle 1 sinks, the angle of the front camera 111 will be downward and the rear camera 112 will be upward by the same amount. If an occupant gets on the right seat of the vehicle 1 and the right side of the vehicle 1 sinks, the angle of the right camera 114 will be downward and the angle of the left camera 113 will be upward by the same amount. At that time, the front camera 111 and the rear camera 112 rotate slightly to the right with respect to the optical axis.

  As described above, the variation of each of the cameras 111 to 114 is linked to the variation of the vehicle body posture. Further, the variation of each of the cameras 111 to 114 is uniquely determined with respect to the variation of the vehicle body posture. In order to estimate the camera parameters while considering the linkage of the cameras 111 to 114 to the vehicle body, the parameter calibration unit 121 does not estimate the camera parameters of each of the cameras 111 to 114 individually, but the attitude of the vehicle 1. The camera parameters are calibrated from the estimated posture of the vehicle 1.

  More specifically, when the calibration is performed using only the trajectory line, if the camera parameters are estimated individually for each of the cameras 111 to 114, there are parameters that cannot be estimated. For example, when calibration is performed using parallelism between the vehicle 1 and a straight line, the roll angle of the front camera 111 can be obtained by correcting the trajectory line in the superimposed composite image to be completely vertical. it can.

  Also, the pitch angle of the front camera 111 is parallel to the vehicle 1 if a plurality of trajectory lines are captured, and the trajectory lines are also parallel to each other. It can be estimated by correcting to satisfy.

  However, the yaw angle of the front camera 111 does not change the parallelism of the trajectory line and the angle of the trajectory line in the superimposed composite image even if the yaw angle is changed, and thus there is a lack of features for correction and is unique. Cannot be determined. Therefore, the camera parameters cannot be estimated, and a composite image that is not completely displaced cannot be generated.

  On the other hand, when the calibration is performed through the posture estimation of the vehicle 1 in use, such as when traveling, all camera parameters are obtained using only the trajectory line, even if there is no relative relationship between the trajectory line and the vehicle. Can be estimated. In order to estimate a certain camera parameter, some observable change must appear with respect to the change in the camera parameter. However, when the vehicle attitude parameter fluctuates, the observable trajectory line changes. Appears.

  Specifically, when the pitch angle of the vehicle 1 changes, the parallelism of the trajectory lines in the superimposed composite image changes. Further, when the roll angle of the vehicle 1 fluctuates, a deviation of the locus line occurs at the image boundary between the overhead images in the superimposed composite image. Further, when the height of the vehicle 1 fluctuates, a shift of the locus line different from the case where the roll angle of the vehicle 1 fluctuates occurs at the image boundary between the overhead images in the superimposed composite image. Accordingly, the parameters of the vehicle posture can be estimated only from the locus line in the superimposed composite image, and once the vehicle posture is determined, the positions and angles of the cameras 111 to 114 attached to the vehicle 1 can be calculated. All camera parameters can be estimated by using only trace lines.

  However, this calibration processing needs to be performed in advance at the time of factory shipment. When the calibration is completed at the time of factory shipment, the superimposed composite image is not displaced in the specific riding state assumed at the time of calibration.

  Since the deviation in the superimposed composite image is caused by the change in the vehicle posture accompanying the change in the riding state of the vehicle 1, if the change in the camera posture accompanying the change in the vehicle posture is cancelled, the original deviation is corrected. It is possible to return to a superimposed composite image.

  Therefore, it is possible to generate a superimposed composite image without deviation by estimating the vehicle posture and calculating the angle and position of the camera accompanying the change. In other words, if calibration has not been performed in advance, even if correction of the camera angle and position corresponding to the change in the vehicle attitude is performed, it will only return to the initial uncalibrated state, Since the misalignment of the image remains, a superimposed composite image without misalignment cannot be generated. Therefore, in this calibration process, calibration needs to be performed in advance.

  The parameter calibration unit 121 is configured so that the trajectory line acquired by the trajectory line acquisition unit 119 extends along the traveling direction of the vehicle 1 and no deviation occurs at the boundary between the overhead images in the superimposed composite image. The external parameters (camera parameters) relating to the attachment states of the cameras 111 to 114 are calibrated. Specifically, the parameter calibration unit 121 includes an attitude estimation unit 301, a translation correction unit 304, and a camera parameter calculation unit 305. The posture estimation unit 301 includes a pitch angle estimation unit 302 and a roll angle / height / rotation center estimation unit 303.

  The pitch angle estimation unit 302 performs pitch angle estimation of the vehicle 1 using the two trajectory lines Fr1 and Fr2 (FIGS. 4 to 6) and the two trajectory lines Rr1 and Rr2 (FIGS. 4 to 6). (S808 in FIG. 8). Two trajectory lines Fr1 and Fr2 (trajectory lines Rr1 and Rr2) when the vehicle 1 is traveling straight ahead are displayed in parallel with the direction extending along the traveling direction of the vehicle 1 in an ideal superimposed composite image. Should be done. However, when the pitch angle of the vehicle 1 varies, the two locus lines Fr1 and Fr2 (trajectory lines Rr1 and Rr2) are displayed in a shape in which the front side is widened and the rear side is narrowed in the superimposed composite image, and are not parallel. When the image captured by the front camera 111 (rear camera 112) is converted into a bird's-eye view viewpoint, the pitch angle of the vehicle 1 is set so that the two trajectory lines Fr1, Fr2 (trajectory lines Rr1, Rr2) are parallel. presume.

  Specifically, the pitch angle estimation unit 302 uses, for example, an inner product of straight line vectors as an evaluation function based on the parallelism of the straight line equations in the front camera 111 and the rear camera 112, and the inner product can be as much as possible for both the front camera 111 and the rear camera 112. The pitch angle of the vehicle 1 is optimized so as to be a value close to 1. The trajectory lines Fr1 and Fr2 (the trajectory lines Rr1 and Rr2) used here are obtained by the trajectory line acquisition unit 119 described above. The optimization can be realized by using a known technique such as the steepest descent method. That is, for example, an evaluation function related to the pitch angle of the vehicle 1 is obtained, and the process of minutely changing the pitch angle of the vehicle 1 may be repeatedly performed so that the evaluation function approaches the target value.

  The roll angle / height / rotation center estimation unit 303 causes the roll angle of the vehicle 1 and the height of the vehicle 1 so as to eliminate the shift of the trajectory line appearing at the image boundary between the overhead images in the superimposed composite image. Then, the center of rotation of the vehicle 1 is estimated (S809 in FIG. 8). Here, the pitch angle of the vehicle 1 is fixed to the pitch angle obtained by the pitch angle estimation unit 302.

  Specifically, the roll angle / height / rotation center estimation unit 303 designs an evaluation function that represents the deviation of the locus line that appears at the image boundary between the overhead images in the superimposed composite image, and the evaluation function is The parameters relating to the roll angle of the vehicle 1, the height of the vehicle 1, and the rotation center of the vehicle 1 are optimized so as to be minimized. Here, the evaluation function includes, for example, the shift of the locus line at the image boundary between the overhead images in the superimposed composite image, the image boundary between the front camera 111 and the left camera 113, and the relationship between the front camera 111 and the right camera 114. It is assumed that the image boundary, the image boundary between the rear camera 112 and the left camera 113, and the image boundary between the rear camera 112 and the right camera 114 are calculated, and the sum thereof is taken. The trajectory lines Fr1, Fr2, Lt1, Rt2, Rr1, and Rr2 used here are obtained by the trajectory line acquisition unit 119 described above.

  Although the above optimization is performed using a known technique, since there are a plurality of parameters to be estimated, it is not based on a gradient method such as the steepest descent method, but a global optimization method should be selected. Is desirable. The evaluation function that represents the deviation of the trajectory line at the image boundary between the overhead images in the superimposed composite image is minimized in the process of minutely changing the roll angle of the vehicle 1, the height of the vehicle 1, and the rotation center of the vehicle 1. Repeat as shown. Here, considering that the camera parameters differ depending on the rotation center of the vehicle 1, the optimization including the rotation center of the vehicle 1 is performed.

  The translation correction unit 304 is a parameter (translation parameter) corresponding to translation of the vehicle 1 with respect to the ground (movement in a direction parallel to the ground), that is, a camera, based on information obtained from calibration performed in the past. The position in the translation direction (the position in plan view) excluding the height of is corrected (S810 in FIG. 8).

  Here, since the information indicating the absolute distance between the ground and the camera is not measured, it is impossible to estimate the position of the camera with respect to the ground. Since the deviation of the direction is extremely small, for example, the value at the time of product calibration stored in the ROM 103 or the like is used.

  Since the relative positions of the cameras are obtained by the roll angle / height / rotation center estimating unit 303, one reference camera (reference camera) is selected (for example, the front camera 111), and the front camera 111 is selected. All the positions of the cameras 111 to 114 are corrected for translation so that the error in the translation direction between the calibration value at the time of product shipment and the current value is canceled.

  The camera parameter calculation unit 305 calculates camera parameters related to the postures of the cameras 111 to 114 corresponding to the posture of the vehicle 1 from the postures of the vehicle 1 obtained by the posture estimation unit 301 and the parameters obtained by the translation correction unit 304. Calculate (S811 in FIG. 8). Note that the camera parameters of the corresponding cameras 111 to 114 can be uniquely calculated by coordinate conversion.

  As described above, with such a configuration, the calibration process of the present embodiment is presumed in a state where calibration has been performed once in a factory or the like, but is proposed in Japanese Patent Laid-Open No. 2016-70814. All the camera parameters can be estimated using only the trajectory line without requiring the parallelism of the vehicle to the white line, that is, the relative relationship between the white line and the vehicle.

  Thus, according to the present embodiment, the camera parameters can be calibrated even in an environment where there is no linear feature such as a white line on the road. For this reason, the freedom degree of the environment where a calibration process is performed can be raised. Therefore, the range in which the calibration device 116 can be applied can be expanded.

  In the above embodiment, an example in which four cameras are used and a part of an image captured by two adjacent cameras is superimposed or adjacent is shown. However, it is not limited to this. For example, the number of cameras in the entire apparatus and the number of cameras that superimpose or image adjacent portions may be appropriately changed according to a request from a user or the like.

  In the above-described embodiment, the calibration execution determination unit 117 determines whether or not the locus line has shifted at the image boundary between the overhead images in the superimposed composite image, thereby calibrating the camera. An example is shown in which it is determined whether or not implementation is necessary. However, it is not limited to this. For example, the calibration execution determination unit 117 uses a sensor that directly estimates a change in the posture of the vehicle, such as a gyro sensor, to detect whether the loading state on the vehicle has changed, and whether or not calibration needs to be performed. It may be determined.

  In addition, for example, the calibration execution determination unit 117 can detect a specific feature extracted by the trajectory line acquisition unit 119 (for example, a road sign such as a speed limit sign drawn on a road or a traffic guide sign drawn on a road). Using the corner portion (point)), etc., the deviation of the linear feature amount at the image boundary between the overhead images in the superimposed composite image and parallel to each other on both sides (surroundings) of the vehicle in the superimposed composite image It may be determined whether or not the camera needs to be calibrated using the parallelism of the linear feature amount.

  In the above-described embodiment, the image generation unit 118 has obtained an example in which a plurality of frames obtained by capturing the periphery of the vehicle 1 with the cameras 111 to 114 are acquired from the RAM 102. However, it is not limited to this. For example, the image generation unit 118 may acquire images directly from the cameras 111 to 114.

  In the above embodiment, the parameter calibration unit 121 corrects the camera parameters by estimating the pitch angle of the vehicle 1 so that the trajectory lines Fr1 and Fr2 (the trajectory lines Rr1 and Rr2) are parallel on the superimposed composite image. The example which implements was shown. However, it is not limited to this. For example, the parameter calibration unit 121 corrects the camera parameters by estimating the pitch angle of the vehicle 1 so that one of the trajectory lines Fr1 and Fr2 (the trajectory lines Rr1 and Rr2) extends along the traveling direction of the vehicle 1. May be implemented.

  As mentioned above, although embodiment of this invention was explained in full detail with drawing, since embodiment is only an illustration of this invention, this invention is not limited only to the structure of embodiment, This invention is not limited. Of course, changes in design and the like within a range not departing from the gist are included in the present invention.

DESCRIPTION OF SYMBOLS 1 ... Vehicle 111 ... Front camera 112 ... Back camera 113 ... Left camera 114 ... Right camera 116 ... Calibration apparatus 118 ... Image generation part 119 ... Trajectory Line acquisition unit 120... Steering angle acquisition unit 121... Parameter calibration unit Fr1, Fr2, Lt1, Rt2, Rr1, Rr2. Lt_t5, Rt_t5, Rr1_t6, Rr2_t6, Rr1_t7, Rr2_t7... Feature points t1, t2, t3, t4, t5, t6, t7.

Claims (4)

A plurality of images obtained by capturing the periphery of the vehicle by being attached to a vehicle traveling on the road surface and capturing the periphery of the vehicle are converted into a bird's-eye view image, and the plurality of bird's-eye view images are combined. An image generation unit that generates one composite image,
It is obtained by connecting the coordinate points corresponding to the positions of specific features reflected in one superimposed composite image obtained by superimposing individual composite images generated at different times by the image generation unit with straight lines. A trajectory line acquisition unit for acquiring a trajectory line to be obtained;
The trajectory line acquired by the trajectory line acquisition unit extends along the traveling direction of the vehicle on the superimposed composite image so that no deviation occurs at the boundary between the overhead images in the superimposed composite image. A parameter calibration unit that calibrates camera parameters related to the mounting state of the plurality of cameras;
A calibration apparatus comprising: The calibration device according to claim 1,
A steering angle acquisition unit for acquiring the steering angle of the vehicle;
The parameter calibration unit is
A calibration apparatus, wherein the camera parameter is calibrated based on the superimposed composite image obtained when the vehicle is traveling straight on the basis of the steering angle acquired by the steering angle acquisition unit. In the calibration apparatus according to claim 1 or 2,
The superimposed composite image is
The periphery is imaged at a period such that at least two coordinate points corresponding to the position of the specific feature are included in a region corresponding to each overhead image in the superimposed composite image so that the locus line can be drawn. A calibration device obtained by In the calibration apparatus according to any one of claims 1 to 3,
The calibration apparatus according to claim 1, wherein the coordinate point corresponding to the position of the specific feature is a corner portion of a sign drawn on the road surface.