KR102647929B1 - Apparatus and method for calibrating camera and lidar sensor of vehicle - Google Patents

Abstract

The present invention relates to a calibration device for a vehicle camera and lidar sensor, comprising: a camera unit, which is an image sensor that captures an image in front of the vehicle; LiDAR sensor unit, a sensor that uses light to measure distance and detect objects; a feature point detection unit that detects at least one feature point and its location information specified in the rigid body image of the vehicle viewed according to the field of view (FOV) of the camera unit and the lidar sensor unit; a feature point information storage unit that stores feature point information set as a standard at the time of final calibration for each sensor; and a control unit that detects sensors in which the feature point information of each detected sensor has changed compared to the feature points stored in the feature point information storage unit, and re-performs calibration for the corresponding sensor whose feature point information has been detected to have changed compared to the previous feature.

Description

Calibration device and method for vehicle cameras and lidar sensors {APPARATUS AND METHOD FOR CALIBRATING CAMERA AND LIDAR SENSOR OF VEHICLE}

The present invention relates to a calibration device and method for vehicle cameras and lidar sensors, and more specifically, to calibrate vehicle cameras and lidar sensors without using special markers or jigs, but using the rigid body of the vehicle. It relates to a calibration device and method for vehicle cameras and lidar sensors that enable automatic calibration.

Typically, various sensors are installed in vehicles to assist drivers, and representative examples include camera sensors (e.g., single camera sensor, stereo camera sensor).

In particular, the stereo camera sensor has the characteristic of operating in the same way as the principle of feeling three-dimensionality and distance when a person looks at an object. In other words, when the left and right cameras of a stereo camera look at a specific object, like both eyes of a person, parallax occurs due to spatial difference, and the parallax allows one to feel the distance to the object or the three-dimensional effect of the object.

In this way, the camera sensor mounted on the vehicle (or other sensor additionally mounted on the vehicle) must perform calibration and rectification. For example, in the stereo camera, calibration is the process of extracting camera parameters of the stereo camera, and rectification uses these extracted camera parameters to accurately align the left and right camera images, thereby reducing the amount of calculation and improving accuracy when calculating the stereo parallax. Elevation is a process.

Camera calibration as described above is a process that must be preceded for camera distortion correction and rectification of stereo images, and conventionally, a person performs the calibration using a checkboard (or marker).

However, in the conventional calibration method using a checkboard, the size of the checkboard is too large and space for calibration is required, and not only can the user not perform the calibration alone, but the calibration cannot be performed automatically at any time and at the desired time. There is a problem. In other words, the calibration method of the camera mounted on a conventional vehicle can be performed for the first time when the vehicle is shipped, and if the camera sensor is distorted due to vibration or impact of the vehicle, it must be delivered to an A/S center or factory to perform calibration. Therefore, there is a problem that the normal function of the camera sensor cannot be used until then.

Moreover, in recent vehicles, in addition to the camera sensor, other sensors (e.g., Lidar sensors) are additionally installed (attached) to complement the camera sensor, and thus, to complement the camera sensor and the camera sensor. Calibration must be performed in synchronization with other sensors (e.g. Lidar sensors) (i.e., calibration must be performed using the same standard).

In order to synchronize and calibrate a plurality of sensors (e.g., camera, lidar) in the conventional manner as described above, the plurality of sensors ( There is a problem in which the functions of (e.g. camera, lidar) cannot be used properly.

Therefore, there is a method to synchronize multiple sensors (e.g. cameras, lidar) and automatically perform calibration at any desired time (i.e., multiple sensors perform calibration using the same standard). It is a necessary situation.

The background technology of the present invention is disclosed in Republic of Korea Patent No. 10-1748398 (registered on June 12, 2017, Calibration device and method for a camera mounted on a vehicle).

According to one aspect of the present invention, the present invention was created to solve the above problems, and the rigid body of the vehicle is calibrated without using a special marker or jig for calibration of vehicle cameras and lidar sensors. The purpose is to provide a calibration device and method for vehicle cameras and lidar sensors that enable automatic calibration using the present invention.

A calibration device for a vehicle camera and lidar sensor according to an aspect of the present invention includes a camera unit, which is an image sensor that captures an image in front of the vehicle; Lidar sensor unit, a sensor that uses light to measure distance and detect objects; a feature point detection unit that detects at least one feature point and its location information specified in the rigid body image of the vehicle viewed according to the field of view (FOV) of the camera unit and the lidar sensor unit; a feature point information storage unit that stores feature point information set as a standard at the time of final calibration for each sensor; And a control unit that detects sensors in which the feature point information of each detected sensor has changed compared to the feature points stored in the feature point information storage unit, and re-performs calibration for the corresponding sensor whose feature point information has been detected to have changed compared to the previous one. It is characterized by

In the present invention, the feature point information includes type and position information of the feature point.

In the present invention, the feature point information is characterized in that it is information on feature points set as a standard at the time of final calibration.

In the present invention, when the calibration feature point information previously stored in the feature point information storage unit is changed, the feature point information correction unit modifies the feature point information with new feature point information detected by the feature point detection section.

In the present invention, when the feature point information is changed compared to the previous one, the feature point information further includes a parameter correction unit that calculates a correction parameter in response to the difference between the previously stored feature point information and the newly changed feature point information.

In the present invention, the correction parameter is characterized as a parameter for calibration to be re-performed according to a change in the feature point information in the previous calibration state.

A calibration method of a vehicle camera and LiDAR sensor according to another aspect of the present invention includes the steps of a control unit receiving a camera image and a LiDAR image; The control unit detecting feature points in the camera image and LiDAR image; the control unit checking for each sensor whether the location of the detected feature point is the same as the location of a previously stored feature point; When the position of the feature point is not the same as the position of a previously stored feature point, the control unit compares the detected feature point with previously stored feature point information to check whether a difference exceeds a specified error range; If the position of the detected feature point differs from the position of the previously stored feature point by more than an error range, the control unit correcting the previously stored feature point information with the new detected feature point information; The control unit calculating a correction parameter corresponding to a difference between the previously stored feature point information and the detected new feature point information; and re-performing, by the control unit, the calibration of the corresponding sensor whose feature point information has changed using the calculated correction parameters.

In the present invention, the camera image refers to an image captured by a camera unit, and the LIDAR image refers to an image sensed by a LiDAR sensor unit.

In the present invention, the feature point is characterized as being designated from the rigid body of the vehicle.

In the present invention, the type and location information of the feature point can be specified at the time of initial calibration.

In the present invention, the location information of the feature point is characterized in that it is coordinate information in each sensor image.

In the present invention, the location information of the feature point is used as a standard for calibration by each sensor, but the location information of the feature point stored for calibration in each sensor is set corresponding to the location where each sensor is installed. It is characterized by being

In the present invention, as a result of checking for each sensor whether the location of the detected feature point is the same as the location of the previously stored feature point, if the location of the feature point is the same as the location of the previously stored feature point, the control unit performs the previously performed function. The method further includes maintaining a calibration state for each sensor.

In the present invention, as a result of comparing the detected feature point with previously stored feature point information and checking whether a difference exceeds a specified error range, a difference occurs that exceeds the error range even if the position of the detected feature point is different from the position of the previously stored feature point. Otherwise, the control unit may further include maintaining the previously performed calibration state for each sensor.

In the present invention, the correction parameter refers to a parameter for calibration to be re-performed according to a change in the feature point information in the previous calibration state.

According to one aspect of the present invention, the present invention enables calibration of vehicle cameras and lidar sensors automatically using the rigid body of the vehicle without using a special marker or jig.

According to another aspect of the present invention, the present invention enables automatic calibration of vehicle cameras and lidar sensors at any desired time using the same standard.

1 is an exemplary diagram showing the schematic configuration of a calibration device for a vehicle camera and lidar sensor according to an embodiment of the present invention.
Figure 2 is a flowchart illustrating a calibration method of a vehicle camera and lidar sensor according to an embodiment of the present invention.
FIG. 3 is an example diagram showing an image captured by a vehicle camera and an image sensed using a LiDAR sensor in FIG. 1.
Figure 4 is an example diagram showing an emblem that can be used as a feature point in this embodiment in Figure 1.
Figure 5 is an example diagram shown to schematically explain a conventional method of calibrating cameras and lidar sensors and problems that may occur in this case.

Hereinafter, an embodiment of a calibration device and method for a vehicle camera and lidar sensor according to the present invention will be described with reference to the attached drawings.

In this process, the thickness of lines or sizes of components shown in the drawing may be exaggerated for clarity and convenience of explanation. In addition, the terms described below are terms defined in consideration of functions in the present invention, and may vary depending on the intention or custom of the user or operator. Therefore, definitions of these terms should be made based on the content throughout this specification.

Figure 1 is an exemplary diagram showing the schematic configuration of a calibration device for a vehicle camera and lidar sensor according to an embodiment of the present invention.

As shown in FIG. 1, the calibration device for a vehicle camera and LiDAR sensor according to this embodiment includes a camera unit 110, a LiDAR sensor unit 120, a feature point detection unit 130, a control unit 140, and a feature point. It includes an information storage unit 150, a feature point information modification unit 160, and a parameter modification unit 170.

The camera unit 110 captures and outputs an image of the front of the vehicle.

The camera unit 100 may include a monocular camera or a stereo camera.

The camera unit 100 may have its own standard parameters set. For example, if the camera unit 100 is a stereo camera, parameters may be set for each of the left and right cameras.

The LIDAR (Light Detection and Ranging) sensor unit 120 is a sensor that uses light to measure distance and detect objects, and LIDAR has a similar principle to radar.

However, radar emits electromagnetic waves to the outside and checks distance and direction using electromagnetic waves that are re-received, but the difference is that lidar emits pulse lasers. In other words, since a laser with a short wavelength is used, it has the advantage of high precision and resolution and even enables three-dimensional understanding of objects.

For example, the LIDAR sensor is mounted on the bumper or top of the vehicle and senses the front and rear of the vehicle to detect objects or structures. For reference, the upper part of Figure 3 (b) is an example showing an image captured by a vehicle camera, and the lower part of Figure 3 (b) shows the FOV (Field of View) of the lidar sensor mounted on the top of the vehicle. This is an example diagram.

In addition, Figure 3 (a) is an example diagram shown to illustratively explain the positions of the camera and lidar sensor mounted on the vehicle.

The feature point detection unit 130 is configured to include at least one or more rigid body images (e.g., camera images, LiDAR sensor images) of a vehicle visible according to the FOV of the camera unit 110 and the LiDAR sensor unit 120. Feature points (e.g., emblems in FIG. 4, etc.) and their location information (i.e., coordinate information in the image) are detected.

The feature point information is stored as a feature point set as a standard (i.e., a standard for calibration) at the time of the final calibration (e.g., calibration upon shipment from the factory).

The feature point information storage unit 150 stores feature point information (eg, type and location information of feature points) set as a standard during the final calibration.

When the previously stored calibration feature point information (e.g., type and position information of feature points) is changed, the feature point information correction unit 160 generates new feature point information (e.g., feature point information) detected by the feature point detection unit 130. type and location information).

When the feature point information is changed compared to the previous one, the parameter correction unit 170 calculates a corrected parameter corresponding to the difference between the previously stored feature point information and the newly changed feature point information. For example, the parameter is a parameter for a new calibration to be performed according to a change in the feature point information in the previous calibration state.

Here, since the feature point is set as a rigid body, changing the feature point information means that it is difficult for the rigid body itself to be deformed, so in reality, it means that the camera or lidar sensor is distorted due to vibration or impact of the vehicle, etc. it means.

Therefore, when a sensor (i.e., camera or LiDAR sensor) whose feature point information has changed compared to the existing one is detected, it means that the posture of the corresponding sensor (i.e., camera or LiDAR sensor) has changed, and the control unit 140 provides the feature point information. You only need to re-perform calibration for the corresponding sensor (i.e. camera or lidar sensor) that is detected to have changed.

As described above, the control unit 140 detects a sensor whose feature point information has changed and re-performs only the calibration for the corresponding sensor (e.g. camera), thereby detecting a plurality of sensors (e.g. camera, lidar sensor) calibrated using the same standard. ), calibration does not need to be performed on other sensors (e.g., lidar sensor) except the corresponding sensor (e.g., camera) that performs the above calibration.

For reference, in order to calculate the extrinsic parameter between the lidar and the camera, a method of fitting an arbitrary plane is used, as shown in FIG. 5. In other words, if both the camera and lidar sensors have the same data, calibration can be performed based on this.

Here, the same data mostly uses a plane, and as shown in Figure 5, the R/T (Rotation, This is a method of calculating the translation matrix. However, even if the R/T matrix is calculated using the existing method, the standard itself may change, which may cause some values to become incorrect in the future.

However, this embodiment prevents the standard itself from changing by using the rigid body of the vehicle included in the FOV (Field of View) of the camera and lidar as a feature point, and re-performs calibration accordingly. By accurately detecting the sensor to be used and calibrating only that sensor, calculating correction parameters corresponding to the difference in which the feature point has changed, and re-performing the calibration using these correction parameters, the calibration is automatically performed at the desired time. Allows calibration to be performed.

Figure 2 is a flowchart for explaining a calibration method of a vehicle camera and lidar sensor according to an embodiment of the present invention.

Referring to FIG. 2, the control unit 140 receives camera images and lidar images (S101). Here, the camera image refers to an image captured by the camera unit 110 (see the upper image in Figure 3 (b)), and the LiDAR image refers to an image sensed by the LiDAR sensor unit 120. (Refer to the lower image in Figure 3 (b)).

Additionally, the control unit 140 detects feature points in the camera image and LIDAR image (S102).

Here, the feature point is designated in the rigid body of the vehicle, and the type and location information (i.e., coordinate information in the image) of the feature point can be specified in advance during the first calibration. In addition, the feature point itself is used as the same reference by the plurality of sensors (e.g., camera and lidar sensor), but since the location where each sensor (e.g., camera and lidar sensor) is installed is different, calibration is required for each sensor. The location information of feature points stored for each feature may be different.

Accordingly, the control unit 140 checks for each sensor whether the location of the detected feature point is the same as the location of the previously stored feature point (S103).

As a result of the check (S103), if the position of the feature point is the same as the position of the previously stored feature point (example in S103), the control unit 140 maintains the previously performed calibration state for each sensor (S108).

However, as a result of the check (S103), if the position of the feature point is not the same as the position of the previously stored feature point (No in S103), the control unit 140 compares the detected feature point with the previously stored feature point information and sets the specified error range. Check whether an abnormal difference occurs (S104).

As a result of the check (S104), even if the position of the detected feature point is different from the position of the previously stored feature point, if the difference does not exceed the error range (No in S104), the control unit 140 detects the previously performed calibration status as a sensor. Each star is maintained (S108).

However, as a result of the check (S104), if the position of the detected feature point differs from the position of the previously stored feature point by more than the error range (example of S104), the control unit 140 previously stored the feature point information storage unit 150. The stored feature point information is modified (or updated) with the detected new feature point information (S105).

Additionally, the control unit 140 calculates a correction parameter corresponding to the difference between the feature point information stored in the feature point information storage unit 150 and the new detected feature point information (S106).

For example, the correction parameter refers to a parameter for a new calibration (i.e., re-performing calibration) to be performed according to a change in the feature point information in the previous calibration state.

Then, the control unit 140 performs calibration of the corresponding sensor (i.e., a sensor with changed feature point information, such as a camera or lidar sensor) using the calculated correction parameters (S107).

As described above, this embodiment does not use special markers or jigs to calibrate vehicle cameras and lidar sensors, but automatically performs calibration using the rigid body of the vehicle, which does not change unless the vehicle is damaged. This has the effect of determining the sensor that needs calibration and automatically performing calibration at a desired time using correction parameters corresponding to the changed feature point information.

The present invention has been described above with reference to the embodiments shown in the drawings, but these are merely illustrative, and various modifications and other equivalent embodiments can be made by those skilled in the art. You will understand the point. Therefore, the scope of technical protection of the present invention should be determined by the scope of the patent claims below. Implementations described herein may also be implemented as, for example, a method or process, device, software program, data stream, or signal. Although discussed only in the context of a single form of implementation (eg, only as a method), implementations of the features discussed may also be implemented in other forms (eg, devices or programs). The device may be implemented with appropriate hardware, software, firmware, etc. The method may be implemented in a device such as a processor, which generally refers to a processing device including a computer, microprocessor, integrated circuit, or programmable logic device. Processors also include communication devices such as computers, cell phones, portable/personal digital assistants (“PDAs”) and other devices that facilitate communication of information between end-users.

110: camera unit 120: lidar sensor unit
130: feature point detection unit 140: control unit
150: Feature point information storage unit 160: Feature point information correction unit
170: Parameter correction unit

Claims (15)

A camera unit, an image sensor that captures images of the front of the vehicle;
Lidar sensor unit, a sensor that uses light to measure distance and detect objects;
a feature point detection unit that detects at least one feature point and its location information specified in the rigid body image of the vehicle viewed according to the field of view (FOV) of the camera unit and the lidar sensor unit;
a feature point information storage unit that stores feature point information set as a standard at the time of final calibration for each sensor; and
A control unit that detects sensors in which the feature point information of each detected sensor has changed compared to the feature points stored in the feature point information storage unit, and re-performs calibration for the corresponding sensor whose feature point information has been detected to have changed compared to the previous one;
The above characteristic points are,
A calibration device for a vehicle camera and lidar sensor, characterized in that it includes an emblem designated as a rigid body of the vehicle.
The method of claim 1, wherein the feature point information is:
A calibration device for a vehicle camera and lidar sensor, comprising information on the type and location of feature points.
The method of claim 1, wherein the feature point information is:
A calibration device for a vehicle camera and lidar sensor, characterized in that it contains information on feature points set as a standard at the time of final calibration.
According to clause 1,
A vehicle camera and lidar sensor further comprising a feature point information correction unit that modifies the feature point information with new feature point information detected by the feature point detection unit when the calibration feature point information previously stored in the feature point information storage unit is changed. calibration device.
According to clause 1,
When the feature point information is changed compared to the previous one, a parameter correction unit that calculates a correction parameter in response to the difference between the previously stored feature point information and the newly changed feature point information; a vehicle camera and lidar sensor further comprising a. calibration device.
The method of claim 5, wherein the correction parameter is:
A calibration device for a vehicle camera and lidar sensor, characterized in that it is a parameter for calibration to be re-performed according to a change in the feature point information in the previous calibration state.
A control unit receiving camera images and lidar images;
The control unit detecting feature points in the camera image and LiDAR image;
the control unit checking for each sensor whether the location of the detected feature point is the same as the location of a previously stored feature point;
When the position of the feature point is not the same as the position of a previously stored feature point, the control unit compares the detected feature point with previously stored feature point information to check whether a difference exceeds a specified error range;
If the position of the detected feature point differs from the position of the previously stored feature point by more than an error range, the control unit correcting the previously stored feature point information with the new detected feature point information;
The control unit calculating a correction parameter corresponding to a difference between the previously stored feature point information and the detected new feature point information; and
Including, the control unit re-performing calibration of the corresponding sensor whose feature point information has changed using the calculated correction parameters,
The above characteristic points are,
A calibration method for a vehicle camera and lidar sensor, comprising an emblem designated as a rigid body of the vehicle.
According to clause 7,
The camera image refers to the image captured by the camera unit,
The LiDAR image is a calibration method of a vehicle camera and LiDAR sensor, characterized in that the image sensed by the LiDAR sensor unit.
delete According to clause 7,
A calibration method for a vehicle camera and lidar sensor, characterized in that the type and location information of the feature point can be specified at the time of initial calibration.
The method of claim 10, wherein the location information of the feature point is:
A calibration method for vehicle cameras and lidar sensors, characterized by coordinate information from each sensor image.
The method of claim 10, wherein the location information of the feature point is:
Each of the above sensors is used as a standard for calibration,
A calibration method for a vehicle camera and lidar sensor, characterized in that location information of feature points stored for calibration in each sensor is set in response to the location where each sensor is installed.
According to clause 7,
As a result of checking for each sensor whether the location of the detected feature point is the same as the location of the previously stored feature point,
If the position of the feature point is the same as the position of the previously stored feature point, the control unit maintains the previously performed calibration state for each sensor. Calibration method for a vehicle camera and a lidar sensor, further comprising:
According to clause 7,
As a result of comparing the detected feature point with the previously stored feature point information and checking whether a difference exceeds the specified error range,
Even if the position of the detected feature point is different from the position of the previously stored feature point, if the difference does not exceed the error range, the control unit maintains the previously performed calibration state for each sensor. How to calibrate cameras and lidar sensors.
The method of claim 7, wherein the correction parameter is:
A calibration method for a vehicle camera and lidar sensor, characterized in that it refers to parameters for calibration to be re-performed according to a change in the feature point information in the previous calibration state.

Images