JP2020087210A - Calibration device and calibration method - Google Patents

Abstract

To provide a calibration device and a calibration method, capable of performing calibration in which the inclination of a road surface is taken into consideration, when performing the calibration of a camera parameter on the basis of the feature point of the road surface.SOLUTION: A calibration device 116 includes: a calibration operation part 205 which performs the calibration of a camera parameter on the basis of the feature point detected by a feature point detection part 201; and an inclination estimation part which estimates the inclination of a road surface in a direction crossing a traveling direction of a vehicle V on the basis of a calibration result based on an image signal output from a front camera 111. The calibration operation part 205 performs the calibration on the basis of the inclination of the road surface estimated by the inclination estimation part 206 as well.SELECTED DRAWING: Figure 3

Description

The present invention relates to a calibration device and a calibration method for calibrating camera parameters of an image pickup device based on an image signal output from the image pickup device.

In recent years, there has been proposed a system that assists a driver in driving by using an image captured by an image capturing device installed in a vehicle. Examples of such systems include a bird's-eye view image display system for checking the surroundings of a vehicle, an expected route display system when the vehicle is moving backward, and an automatic parking system.

The image pickup device is attached to the vehicle at a position/angle in accordance with design values, but at that time, an error may occur in the position/angle. In order to suppress such an error, a calibration process called calibration is performed on camera parameters such as the position and angle of the image pickup device.

Calibration at the time of shipment of a vehicle is generally performed in an empty state in which no one is on board. Therefore, when the state of the vehicle after the calibration is the empty state in which the calibration is actually performed, the image of the imaging device does not shift.

However, when the user actually uses the vehicle, the number of passengers, the places to sit, the loading state of the luggage, and the like change variously. When the loading state of the vehicle changes, the posture of the vehicle also changes, and the installation state of the imaging device on the ground changes accordingly. In addition, the posture of the imaging device may change due to deterioration over time. That is, the camera parameter changes and an error occurs.

With respect to such a problem, Patent Document 1 discloses a method of calibrating camera parameters while the vehicle is traveling. The method is to extract and track feature points from an image picked up by an image pickup device installed in a vehicle. The basic principle of the method is to correct camera parameters using a linear feature point series (calibration index) resulting from a two-dimensional movement of feature points between frames.

International Publication No. 2012/139636

By the way, in the method described in Patent Document 1 described above, it is premised that the feature points used are all at the same height, such as those obtained from road surface paint. However, not all feature points appearing in the image obtained from the image pickup device are at the same height. As an example, the road surface is often inclined downward from the center to the side from the viewpoint of drainage.

Therefore, for example, the characteristic points of the road surface obtained from the image of the right side camera that captures the right side of the vehicle actually become lower as the distance from the center of the road surface to the right side increases. Therefore, it is difficult to improve the accuracy of the calibration if the calibration operation is performed with all the characteristic points of the road surface obtained from the image of the right camera as being at the same height.

Therefore, it is an object of the present invention to provide a calibration device and a calibration method capable of performing calibration in consideration of the inclination of a road surface when calibrating camera parameters based on characteristic points of the road surface. I am trying.

In order to achieve the above object, a calibration device of the present invention is mounted in front of a vehicle to capture an image of the front of the vehicle including the road surface, and a vehicle mounted laterally of the vehicle including the road surface. Is a calibration device that calibrates camera parameters of an image pickup device based on an image signal output from an image pickup device that includes a side image pickup device that picks up side images of a plurality of image signals captured at different times. From at least a feature point detection unit that detects a plurality of feature points detected in the width direction of the road surface, a calibration calculation unit that calibrates camera parameters based on the feature points detected by the feature point detection unit, and A slope estimating unit that estimates a slope of a road surface in a direction intersecting with a traveling direction of the vehicle based on a calibration result based on an image signal output from the image pickup device; The calibration is performed based on the inclination of the road surface estimated by the section.

In the calibration device of the present invention configured as described above, the inclination estimation unit determines the inclination of the road surface in the direction intersecting with the traveling direction of the vehicle based on the calibration result based on the image signal output from the front imaging device. Then, the calibration calculation unit performs the calibration based on the inclination of the road surface estimated by the inclination estimation unit.

By doing so, when the camera parameters are calibrated based on the characteristic points of the road surface, it becomes possible to perform the calibration in consideration of the inclination of the road surface.

It is a block diagram showing a schematic structure of a calibration system to which a calibration device which is an embodiment of the invention is applied. It is a figure which shows an example of the arrangement position of the imaging device which is calibrated by the calibration device which is embodiment. It is a functional block diagram which shows the schematic structure of the calibration apparatus which is embodiment. 6 is a flowchart for explaining an example of the operation of the calibration device according to the embodiment. It is a figure for explaining an example of operation of a calibration device which is an embodiment. It is a figure for explaining an example of operation of a calibration device which is an embodiment. It is a figure for explaining an example of operation of a calibration device which is an embodiment. It is a figure for explaining an example of operation of a calibration device which is an embodiment. It is a figure for explaining an example of operation of a calibration device which is an embodiment. It is a figure for explaining an example of operation of a calibration device which is an embodiment. It is a figure which shows an example of the inclination provided in the road surface. It is a figure for explaining an example of operation of a calibration device which is an embodiment. It is a figure for explaining an example of operation of a calibration device which is an embodiment.

(Schematic configuration of image processing device)
Embodiments of the present invention will be described below with reference to the drawings. In the following, a case of using four image pickup devices will be described.

FIG. 1 is a block diagram showing a schematic configuration of a calibration system to which a calibration device according to an embodiment of the present invention is applied, and FIG. 2 shows an image pickup device in which calibration is performed by the calibration device according to the embodiment. It is a figure which shows an example of an arrangement position.

As shown in FIG. 1, the calibration system 100 is mounted on a vehicle V (see FIG. 2) and similarly performs a calibration operation on an imaging device mounted on the vehicle V.

As shown in FIG. 2, a plurality of small cameras (imaging devices) are provided on the front, rear, left and right of the vehicle V.

Specifically, the front bumper or front grill of the vehicle V is equipped with a front camera (front imaging device) 111 that images the front of the vehicle including the road surface toward the front of the vehicle V. A rear camera 112 is mounted on the rear bumper or rear garnish of the vehicle V so as to image the rear of the vehicle including the road surface toward the rear of the vehicle V. The left door mirror of the vehicle V is equipped with a left side camera (side image capturing device) 113 that images the left side of the vehicle including the road surface toward the left side of the vehicle V. The right side mirror of the vehicle V is equipped with a right side camera (side image capturing device) 114 that images the right side of the vehicle including the road surface toward the right side of the vehicle V.

The front camera 111, the rear camera 112, the left side camera 113, and the right side camera 114 are each equipped with a wide-angle lens or a fisheye lens capable of observing a wide range, and the four cameras 111 to 114 surround the vehicle V. The area including the road surface can be observed without omission. The cameras 111 to 114 form an image pickup device that picks up an image of the road surface around the vehicle V. The four images captured by the cameras 111 to 114 are output to the image processing apparatus 101 (see FIG. 1).

Returning to FIG. 1, the calibration system 100 mainly includes four cameras 111 to 114, an image processing device 101, a RAM (Random Access Memory) 102, a ROM (Read Only Memory) 103, and a display device. 104, a vehicle speed sensor 105, a steering angle sensor 106, a yaw rate sensor 107, an input device 108, and a communication device 109.

The vehicle speed sensor 105, the steering angle sensor 106, and the yaw rate sensor 107 are sensors that detect the vehicle speed, the steering angle, and the yaw rate, respectively, and the sensor information detected by each sensor is output to the image processing apparatus 101 and the image processing apparatus 101. It is used in the calculation processing in. The steering angle sensor 106 detects the tire angle of the front wheels as the steering angle.

The input device 108 is a device such as a switch or a button that receives a user operation. The input device 108 is used for turning on/off the calibration function, initializing the calibration result, changing the calibration method, and the like. Various information input to the input device 108 through a user operation is output to the image processing device 101.

The communication device 109 is a device used for communication with an external device (not shown). The image processing apparatus 101 receives various information from an external device via the communication device 109 and outputs various information calculated by the image processing apparatus 101 to the external device.

Numerical data required in the arithmetic processing process in the image processing apparatus 101, program variables for the processing result during the arithmetic processing, and the like are written in the RAM 102. The written data is appropriately read out and used in the arithmetic processing in the arithmetic processing process of the image processing apparatus 101. The RAM 102 also stores image data captured by the cameras 111 to 114.

The ROM 103 stores in advance, for example, a program that executes calibration and information that is used without being rewritten among information necessary for the program. For example, design values (external parameters) of installation positions and angles (pitch angle, yaw angle, roll angle, camera height) of the cameras 111 to 114, focal lengths of the cameras 111 to 114, pixel size, optical axis center, Camera parameters such as a distortion function (internal parameters) are stored. The calibration system 100 according to the present embodiment mainly performs calibration of external parameters.

The image processing apparatus 101 receives various information transmitted from the cameras 111 to 114, the vehicle speed sensor 105, the steering angle sensor 106, the yaw rate sensor 107, the input device 108, the communication device 109, and the like. The image processing apparatus 101 uses the received information to perform an image processing operation based on a program or the like.

The image processing apparatus 101, for example, converts the viewpoints of the images input from the cameras 111 to 114 to generate a bird's-eye view image that looks down on the ground from above. Specifically, with respect to the images captured by the cameras 111 to 114, which are fish-eye cameras, a known distortion function stored in advance in the ROM 103 or the like is used to generate an image from which image distortion has been removed.

Each image from which the distortion has been removed is subjected to viewpoint conversion into an image viewed from a bird's-eye view based on a known design value regarding camera attachment stored in advance in the ROM 103 or the like (image generation device 115). In such a viewpoint conversion process, a well-known camera geometric conversion formula is used to calculate a specific pixel of the overhead view image and a specific pixel of each of the cameras 111 to 114 corresponding to the specific pixel, and the brightness value of the pixel is calculated. It can be realized by assigning to the pixels of.

When the corresponding pixel includes a decimal point and there is no corresponding pixel, a process of assigning the intermediate brightness of peripheral pixels is performed by a known brightness interpolation process. In addition, the image processing apparatus 101 performs arithmetic processing using output results of the vehicle speed sensor 105, the steering angle sensor 106, the yaw rate sensor 107, and the communication apparatus 109, and performs operation program switching processing according to the input result of the input apparatus 108. To run.

Further, the image processing apparatus 101 is equipped with a calibration apparatus 116 that performs calibration of each of the cameras 111 to 114 so that the bird's-eye view image whose viewpoint has been converted is an image looking down on the vehicle V from directly above. The configuration of the calibration device 116 will be described later.

The display device 104 receives the processing result of the image processing device 101 and presents the processing result to the user using a display or the like. For example, a bird's-eye view image obtained by converting the viewpoints of the four images of the cameras 111 to 114 is displayed to the user. The display device 104 can also switch the display content according to the output of the image processing device 101, such as displaying only the image of the camera 112 that captures the rear of the vehicle V.

(Functional configuration of the calibration device)
FIG. 3 is a functional block diagram showing a schematic configuration of the calibration device 116 according to the present embodiment.

The calibration device 116 according to the present embodiment has a control unit 200 and a storage unit 210.

The control unit 200 controls the calibration device 116 as a whole. The control unit 200 has a CPU, a programmable logic device such as an FPGA, and an arithmetic element represented by an integrated circuit such as an ASIC.

A control program (not shown) is stored in the storage unit 210 of the calibration device 116, and this control program is executed by the control unit 200 when the calibration device 116 is activated, so that the calibration device 116 is shown in FIG. It has a functional configuration as shown. In particular, since the calibration device 116 of the present embodiment performs high-speed image processing as described later, it is preferable to have a computing element capable of high-speed computation, such as FPGA.

The control unit 200 includes a feature point detection unit 201, a feature point tracking unit 202, a feature point series creation unit 203, a coordinate conversion unit 204, a calibration calculation unit 205, and a tilt estimation unit 206.

The feature point detection unit 201 detects feature points from the image signals output from the cameras 111 to 114. In other words, the feature point detection unit 201 performs image processing on the images captured by the cameras 111 to 114 to extract feature points. The feature points mentioned here are, for example, intersection points of edges such as a corner of a broken line drawn on a road surface, a corner of a pillar, a corner of a road marking, or a corner feature point. The corner feature points can be extracted, for example, by applying a Harris operator, which is a known technique. Information on the feature points extracted by the feature point detection unit 201 is sent to the feature point tracking unit 202 and also stored in the storage unit 210.

The feature point tracking unit 202 tracks the same feature point from a plurality of time-sequential captured images and outputs the feature point tracking information to the feature point sequence creation unit 203. The characteristic point tracking information is information indicating the same characteristic point in different captured images, for example, information indicating the coordinates of the characteristic point in each captured image.

In order to track the characteristic points, information on the characteristic points in the latest captured image input from the characteristic point detection unit 201, information on the characteristic points in the past captured images stored in the storage unit 210, and output from the vehicle speed sensor 105. The motion information (speed history) of the vehicle V to be used is used.

For tracking feature points, known tracking methods such as SAD (Sum of Absolute Difference), SSD (Sum of Squared Difference), and LK (Lucas-Kanade) method can be used.

An example of the feature points detected by the feature point detection unit 201 and tracked by the feature point tracking unit 202 will be described with reference to FIG. The feature point detection unit 201 sets the image information output from the cameras 111 to 114 to an interval of two frames (66 ms if a frame is generated every 1/30 seconds) which is an image generation interval of the cameras 111 to 114 as an example. Detect with. 6A shows the feature point C1 detected by the feature point detecting unit 201 from the Nth frame (N is a natural number) captured image, and FIG. 6B shows the feature point detecting unit 201 from the N+2th frame captured image. The feature point C2 shown in FIG. 6C is a diagram showing the feature point C3 detected by the feature point detection unit 201 from the captured image of the Nth frame.

Next, the feature point tracking unit 202 tracks the same feature point among the feature points detected by the feature point detection unit 201, and outputs the feature point tracking information to the feature point sequence creation unit 203. In the example shown in FIG. 6, the feature points C1, C2, and C3 are all feature points detected for the same corner of the same road surface marker. The feature point tracking unit 202 tracks these feature points C1 to C3 as the same feature point and outputs the tracking information to the feature point sequence creation unit 203.

The feature point series creation unit 203 collects a series of tracked feature points as one feature point series for the feature points determined to be tracked by the feature point tracking unit 202, and outputs the series to the coordinate conversion unit 204. In addition, the feature point series creation unit 203 accumulates the feature point series and stores it in the storage unit 210.

Here, there is a possibility that an error may occur in the detection result of the feature point by the feature point detection unit 201, the tracking process of the feature point by the feature point tracking unit 202, and the information acquired from the vehicle speed sensor 105 or the steering angle sensor 106. is there. Therefore, it is desirable to accumulate a large amount of information indicating the characteristic points, the traveling speed, and the steering angle in order to reduce the error.

For example, when the feature point detection processing is performed by the feature point detection unit 201 while the vehicle V is traveling on a road that is not paved sufficiently and is not horizontal to the vehicle V, the pitch angle, yaw angle, and roll angle are detected. It is insufficient as a calibration index. Therefore, a large amount of error may occur in the camera parameters (pitch angle, yaw angle, roll angle, camera height) calculated by the calibration calculation unit 205. Even if the feature point sequence creating unit 203 calculates the length of the feature point sequence that is insufficient as a calibration index, it is insufficient as a calibration index if a large number of correct feature point sequences are accumulated as the calibration index. It is possible to mitigate errors that occur in camera parameters (pitch angle, yaw angle, roll angle, camera height) due to such a series of feature points.

The coordinate conversion unit 204 determines the coordinates of each feature point forming one feature point sequence output from the feature point sequence creation unit 203 from the two-dimensional coordinates in the captured image to the three-dimensional coordinates with the cameras 111 to 114 as a reference. , That is, the coordinates are converted into the real space. However, it is assumed that all the feature points exist on the road surface, and the coordinate transformation is performed with the constraint that the height of the feature points is zero. This coordinate transformation is realized by calculation using a rotation matrix.

The elements of the rotation matrix are determined using the internal parameters and external parameters of the cameras 111 to 114. The values stored in the ROM 103 are used as the internal parameters and the external parameters. The coordinates of the respective feature points forming one feature point series, which are coordinate-converted by the coordinate conversion unit 204, are output to the calibration calculation unit 205.

The calibration calculation unit 205 calibrates the camera parameters based on the feature points coordinate-converted by the coordinate conversion unit 204. Details of the calibration calculation by the calibration calculation unit 205 will be described later.

The inclination estimation unit 206 estimates the inclination of the road surface in the direction intersecting with the traveling direction of the vehicle V based on the calibration result based on the image signal output from the front camera 111. More specifically, the inclination estimation unit 206 estimates the inclination of the road surface based on the pitch angle and/or the yaw angle which are the camera parameters of the front imaging device calculated by the calibration calculation unit.

In particular, the inclination estimation unit 206 is based on the relationship between the difference in the moving speed of the two feature points detected in the width direction of the road surface by the feature point detection unit 201 and the distance in the width direction of the road surface between the feature points. , Estimate the slope of the road surface.

Then, the calibration calculation unit 205 calibrates the left side camera 113 and the right side camera 114 based on the inclination of the road surface estimated by the inclination estimation unit 206. In particular, the calibration calculation unit 205 calibrates the pitch angle, which is a camera parameter of the left side camera 113 and the right side camera 114, based on the inclination of the road surface estimated by the inclination estimation unit 206.

The specific operation of each part of the calibration device 116 shown in FIG. 3 will be described in detail later.

(Camera coordinate system)
FIG. 5 is an explanatory diagram showing an example of the camera coordinate system. An example of the camera coordinate system (X, Y, Z) will be described below with reference to FIG. In this camera coordinate system (X, Y, Z), the optical axis direction of the camera is defined as the Z axis direction. A surface IS indicated by hatching in the drawing is an imaging surface IS that is perpendicular to the optical axis (Z axis) of the camera. In this camera coordinate system (X, Y, Z), it is assumed that the X axis and the Y axis are parallel to the image pickup surface IS.

In the figure, pitching refers to rotation around the X axis, yawing refers to rotation around the Y axis, and rolling refers to rotation around the Z axis. The parameters of the rotation angle around the X, Y, and Z axes are called the pitch angle, the yaw angle, and the roll angle, respectively.

When the optical axis direction of the front camera 111 and the rear camera 112 coincides with the front-rear direction of the vehicle V, the front-rear direction is defined as the Z-axis direction, and the vehicle V perpendicular to the front-rear direction (Z-axis direction). The axis extending in the vertical direction of is defined as the Y-axis.

On the other hand, when the optical axis direction of the left side camera 113 and the right side camera 114 coincides with the vertical direction of the vehicle V, the vertical direction is defined as the Z axis direction, and the vertical direction (Z axis direction) is defined as the Z axis direction. The axis extending in the left-right direction of the vertical vehicle V is defined as the X axis.

(Operation of calibration device)
Next, an example of the operation of the calibration device 116 according to the present embodiment will be described with reference to the flowchart of FIG. 4 and FIGS.

FIG. 4 is a flow chart for explaining the operation of the calibration device 116. The operation shown in the flowchart of FIG. 4 is regularly activated and started during the image pickup operation of the cameras 111 to 114.

First, in step S1, the calibration device 116 acquires the image signals output from the cameras 111 to 114. In step S2, the feature point detection unit 201 extracts feature points in the images captured by the cameras 111 to 114 based on the image signal acquired in step S1. In step S3, one of the feature points extracted in step S2 is processed, and the feature point tracking unit 202 performs the tracking process. In step S4, the characteristic point series creation unit 203 accumulates the characteristic point series.

In step S5, the control unit 200 determines whether or not the feature point series accumulated in step S4 has reached a predetermined required number. If it is determined that the accumulated number of feature points has reached the required number (YES in step S5), the accumulation of the required number of feature point sequences has been completed, and the process proceeds to step S6. On the other hand, when it is determined that the number of characteristic point sequences has not reached the required number (NO in step S5), the accumulation of the required number of characteristic point sequences has not been completed, and thus the process returns to step S1.

In step S6, the coordinate conversion unit 204 converts the coordinates of the feature points forming the feature point series into the coordinates in the real space. Next, the calibration calculation unit 205 calibrates the camera parameters based on the feature points coordinate-converted by the coordinate conversion unit 204.

In the calibration device 116 according to the present embodiment, the calibration calculation unit 205 performs calibration of external parameters by performing pitch angle, yaw angle, roll angle, and height calibration that are external parameters of the cameras 111 to 114. Calculate. The outline of the pitch angle, yaw angle, roll angle, and height calibration taking the front camera 111 as an example will be described below with reference to FIGS.

First, the calibration calculation unit 205 calibrates the pitch angle (parameter of the rotation angle around the X axis shown in FIG. 5) using the design values of the camera mounting position and angle stored in the ROM 103. Various methods for calibrating the pitch angle have been introduced, and any of them may be used, but the calibration device 116 according to the present embodiment uses the feature point series accumulated in step S5.

A pitch angle calibration method using a feature point series is already known, for example, proposed in International Publication No. 2012/139636. Therefore, although detailed description is omitted, in the calibration calculation unit 205 according to the present embodiment, pitch angle calibration is performed so that two or more feature point sequences that have been bird's-eye view transformed by the coordinate transformation unit 204 are parallel. Is done.

When the pitch angle of the front camera 111 is deviated, as shown in FIG. 7A, two feature point sequences E ₁ and E ₂ extending in the traveling direction of the vehicle V on the bird's-eye view converted image. Is not projected in parallel. That is, the angles θ ₁ and θ _{2 formed} by the feature point series E ₁ and E ₂ and the virtual line L extending in the width direction of the vehicle V are not equal (θ ₁ ≠θ ₂ ). Therefore, the calibration calculation unit 205 calibrates the pitch angle so that the feature point series E ₁ and E ₂ are projected in parallel. That is, as shown in FIG. 7B, the pitch angle is calibrated so that the angles θ ₁ and θ ₂ are equal (θ ₁ =θ ₂ ).

Next, the calibration calculation unit 205 calibrates the yaw angle (parameter of the rotation angle around the Y-axis shown in FIG. 5) using the camera mounting position/angle design values stored in the ROM 103. Various methods for calibrating the yaw angle have been introduced, and any method may be used. However, in the calibration device 116 of the present embodiment, the method proposed in International Publication No. 2012/139636. The yaw angle is calibrated at. Specifically, the yaw angle when the angle of the feature point series accumulated in step S5 becomes parallel to the traveling direction of the vehicle V is calibrated as the yaw angle of the front camera 111.

When the yaw angle of the front camera 111 is deviated, as shown in FIG. 8A, two feature point series E ₁ and E ₂ extending in the traveling direction of the vehicle V on the bird's-eye view converted image. Does not match the traveling direction of the vehicle V. That is, the feature point series E ₁ and E ₂ do not intersect perpendicularly with the virtual line L extending in the width direction of the vehicle V (neither θ ₁ nor θ ₂ is 90°). Therefore, the calibration calculation unit 205 calibrates the pitch angle so that the feature point series E ₁ and E ₂ match the traveling direction of the vehicle V. That is, as shown in FIG. 8B, the yaw angle is calibrated so that the angles θ ₁ and θ ₂ are both 90°.

Furthermore, the calibration calculation unit 205 calibrates the roll angle (parameter of the rotation angle around the Z axis shown in FIG. 5).

First, the calibration calculation unit 205 converts the feature point series indicating the same point in the Nth frame FN and the N+2nd frame FN+2 from the image coordinate system to the world coordinate system. This conversion method is already known. Therefore, although detailed description is omitted, in the present embodiment, for example, the feature coordinates in the world coordinates of the frame FN are (P _X1 , P _Y1 ) and the feature coordinates in the world coordinates of the frame FN+2 are (P _X2 , P _{Y2 ).} ), the Euclidean distance D in the world coordinate system between the frame FN and the frame FN+2 is calculated using the following equation (1) as the length of the feature point tracking result, in other words, the length of the feature point sequence.

Next, the calibration calculation unit 205 calculates the traveling distance d [km] of the vehicle V between the frame FN and the frame FN+2 based on the traveling speed v _k of the vehicle V obtained from the vehicle speed sensor 105. .. The traveling distance d [km] between the frame FN and the frame FN+2 is expressed by the following equation (2), where v _k [km/h] is the traveling speed between the frames and f is the number of frames per second. Is calculated using.

As shown in the above formula (2), as the traveling speed of the vehicle V when moving from the frame FN to the frame FN+2, the arithmetic mean value of the traveling speeds acquired in the respective frames is adopted.

Next, the calibration calculation unit 205 divides the Euclidean distance D by the traveling distance d to obtain a normalized speed nv obtained by normalizing all accumulated feature point series, as shown in the following expression (3). calculate. As shown in Expression (3), the normalized speed nv has a value that depends on the traveling distance of the vehicle V.

Next, the calibration calculation unit 205 calibrates the roll angle so that the normalized speeds nv match in different feature point series. Such a roll angle calibration technique is already known, such as being proposed by the LM method (Levenberg-Marquardt Method). Therefore, detailed description is omitted.

When the pitch angle of the front camera 111 is deviated, as shown in FIG. 9A, two feature point series E ₁ and E ₂ extending in the traveling direction of the vehicle V on the bird's-eye view converted image. The normalized velocities nv ₁ , nv ₂ for do not match (nv ₁ ≠nv ₂ ). Therefore, the calibration calculation unit 205 rolls so that the normalized velocities nv ₁ and nv ₂ for the feature point series E ₁ and E ₂ match (nv ₁ =nv ₂ ) as shown in FIG. 9B. Calibrate the corner.

Then, the calibration calculation unit 205 calibrates the camera height of the front camera 111 using the design values of the camera mounting position and angle stored in the ROM 103. Various methods have been introduced for the camera height calibration method, and any of these methods may be used.However, in the present embodiment, the camera height is corrected by the method proposed in International Publication No. 2012/139636. Calibration is done. Specifically, the camera height when the running distance d of the vehicle V calculated during the roll angle calibration and the length L of the tracking result of the feature points at the same time are equal is calculated as the height of the front camera 111.

When the height of the front camera 111 is deviated from the design value of the camera mounting position/angle, as shown in FIG. 10A, the traveling distance d of the vehicle V and the feature points are tracked on the bird's-eye view converted image. An error Δ occurs between the length L of the result and (d≠L). Therefore, as shown in FIG. 10B, the calibration calculation unit 205 controls the front camera 111 so that the traveling distance d of the vehicle V and the length L of the tracking result of the feature points match (d=L). Calibrate height.

Here, the calibration calculation performed by the calibration calculation unit 205 in step S6 is based on the premise that all the feature points to be used are at the same height. In other words, it is assumed that all the feature points used are on the same plane.

However, the actual road surface is not flush. As an example, as shown in FIG. 11, the road surface has a slope for draining rainwater. FIG. 11 is a view of the vehicle V as viewed from the rear, and the vehicle V is proceeding from the front to the back of the drawing. A slope for draining rainwater is provided on the road surface from the center to the left and right, that is, in a direction intersecting with the traveling direction of the vehicle V.

The characteristic points existing in the area on the side of the opposite lane (shown by hatching in the figure) exist at a lower position than the characteristic points existing in the area on the side of the own lane. Therefore, if the calibration calculation of the right camera 114 is performed on the assumption that all the feature points to be used are on the same plane, the normalized speed nv calculated for the feature points on the opposite lane side in the pitch angle calibration is It becomes slower than the normalized speed nv calculated for the feature points on the own lane side. As a result, there is a possibility that pitch angle calibration cannot be performed with high accuracy.

As an example, FIG. 13 shows the result of obtaining the relationship between the normalized speed nv and the distance in the road surface width direction from the right side camera 114 to the feature point. In FIG. 13, the vertical axis represents the normalized speed nv, and the horizontal axis represents the distance to the feature point. In the figure, the solid line B is a straight line obtained by linearly approximating the point sequence. It is considered that the inclination of the solid line B is due to the road surface gradient.

Here, as the cameras 111 to 114, those having a wide imaging range such as a fisheye camera are used. Therefore, the images captured by the front camera 111 and the rear camera 112 include the lanes on the left and right of the vehicle in addition to the traveling lane.

Regarding the roll angle calibration and the height calibration of the calibration calculation in step S6 described above, it is preferable that the characteristic point sequence that is the premise of the calibration calculation is a characteristic point sequence traced on the road surface. Therefore, the front camera 111 and the rear camera 112 perform lane recognition to perform the calibration calculation using only the feature point series existing in the traveling lane area of the vehicle.

On the other hand, regarding pitch angle calibration and yaw angle calibration that are not limited in this way, the relationship between the normalized speed nv and the distance in the road surface width direction from the front camera 111 and the rear camera 112 to the feature point is calculated. As shown in FIG. Also in FIG. 12, the vertical axis represents the normalized speed nv, and the horizontal axis represents the distance to the feature point. In the figure, the solid line B is a straight line obtained by linearly approximating the point sequence. It is considered that the inclination of the solid line B is due to the road surface gradient.

Therefore, in step S7, the tilt estimating unit 206 draws a graph as shown in FIG. 12 for the front camera 111, further calculates the approximate straight line B, and obtains the tilt angle of the approximate straight line B on the graph. Then, in step S8, the calibration calculation unit 205 uses the tilt angle obtained by the tilt estimation unit 206 in step S7 as the pitch angle correction angle, and performs pitch angle calibration again.

Note that a graph as shown in FIG. 12 may be drawn for the rear camera 112 in step S7, but since the feature points that can be extracted from the image captured by the rear camera 112 move in a direction away from the vehicle V, the normalized speed is Blurring is likely to occur in the calculation of nv. Therefore, it is preferable to perform the calculation for the front camera 111.

(Effect of calibration device)
In the calibration device 116 according to the present embodiment configured as described above, the inclination estimation unit 206 intersects the traveling direction of the vehicle V based on the calibration result based on the image signal output from the front camera 111. The inclination of the road surface in the direction to be estimated is estimated, and the calibration calculation unit 205 performs the calibration based on the inclination of the road surface estimated by the inclination estimation unit 206.

Therefore, when the calibration calculation unit 205 calibrates the camera parameters based on the feature points of the road surface, it is possible to perform the calibration in consideration of the inclination of the road surface. As a result, the accuracy of the calibration calculation by the calibration calculation unit 205 can be further improved.

Although the embodiments of the present invention have been described in detail above with reference to the drawings, the specific configuration is not limited to the embodiments and examples, and design changes are made to the extent that they do not depart from the gist of the present invention. Are included in the present invention.

V vehicle 100 calibration system 101 image processing device 105 vehicle speed sensor 111 front camera (front imaging device)
112 rear camera 113 left camera (side imaging device)
114 right side camera (side imaging device)
115 image generation device 116 calibration device 200 control unit 201 feature point detection unit 202 feature point tracking unit 203 feature point series creation unit 204 coordinate conversion unit 205 calibration calculation unit 206 inclination estimation unit

Claims (5)

Imaging that includes a front imaging device that is mounted in front of the vehicle to image the front of the vehicle including the road surface, and a side imaging device that is mounted to the side of the vehicle to image the side of the vehicle including the road surface A calibration device for calibrating camera parameters of the image pickup device based on an image signal output from the device,
A feature point detection unit that detects a plurality of feature points detected at least in the width direction of the road surface from the plurality of image signals imaged at different times,
A calibration calculation unit that calibrates camera parameters based on the feature points detected by the feature point detection unit;
Based on a calibration result based on the image signal output from the front imaging device, an inclination estimation unit that estimates the inclination of the road surface in a direction intersecting the traveling direction of the vehicle,
The calibration device, wherein the calibration calculation unit performs calibration based on the slope of the road surface estimated by the slope estimating unit. The calibration device according to claim 1, wherein the calibration calculation unit calibrates the side imaging device based on the inclination of the road surface estimated by the inclination estimation unit. The inclination estimation unit estimates the inclination of the road surface based on a pitch angle and/or a yaw angle, which are camera parameters of the front imaging device calculated by the calibration calculation unit,
The said calibration calculation part calibrates the pitch angle which is a camera parameter of the said side imaging device based on the inclination of the said road surface which the said inclination estimation part estimated, The calibration of Claim 2 characterized by the above-mentioned. Calibration device. The inclination estimation unit is based on a relationship between a difference in moving speed of the plurality of feature points detected in the width direction of the road surface by the feature point detection unit and a distance in the width direction of the road surface between the feature points. The calibration apparatus according to claim 1, wherein the inclination of the road surface is estimated. Imaging that includes a front imaging device that is mounted in front of the vehicle to image the front of the vehicle including the road surface, and a side imaging device that is mounted to the side of the vehicle to image the side of the vehicle including the road surface A calibration method by a calibration device that calibrates camera parameters of the imaging device based on an image signal output from the device,
Detecting at least a plurality of feature points detected in the width direction of the road surface from the plurality of image signals imaged at different times,
Calibrate camera parameters based on the detected feature points,
Based on a calibration result based on the image signal output from the front imaging device, estimating the inclination of the road surface in a direction intersecting the traveling direction of the vehicle,
A calibration method, wherein calibration is performed based on the estimated inclination of the road surface.