KR102054455B1 - Apparatus and method for calibrating between heterogeneous sensors - Google Patents

Abstract

Disclosed are a device for calibration between heterogeneous sensors, and a method for the same. The purpose of the present invention is to provide a method for improving detection accuracy on an object in a long distance regardless of an installation position of the heterogeneous sensors by performing calibration between the heterogeneous sensors by a 3D marker board in which a hole exists. The method for calibration between heterogeneous sensors includes: a step of identifying image data in which the marker board is photographed by a camera sensor; a step of identifying point cloud data in which the marker board is sensed by a LiDAR sensor; a step of recognizing the hole existing in the marker board in each of the image data and the point cloud data identified; a step of determining a conversion vector for calibration of the camera sensor and the LiDAR sensor based on a radius of the hole recognized based on the point cloud data and the radius of the hole recognized based on the image data; and a step of performing the calibration between the camera sensor and the LiDAR sensor by projecting the point cloud data of the LiDAR sensor to the image data of the camera sensor, using the determined conversion vector.

Description

Calibration device and method between heterogeneous sensors {APPARATUS AND METHOD FOR CALIBRATING BETWEEN HETEROGENEOUS SENSORS}

The present invention relates to a calibration device and method between heterogeneous sensors, and more particularly, to photograph marker boards through heterogeneous sensors, and based on recognized holes after recognizing holes existing in marker boards from the data of respective heterogeneous sensors. The present invention relates to an apparatus and a method for performing calibration between heterogeneous sensors.

With the recent development of sensing technology, interest in multi-sensor data fusion and data integration technology using sensors such as cameras, lidars and radars has increased. In particular, robots and autonomous driving systems have emerged as an important issue for stable mobility through complementary complementary sensors. Recently developed sensor fusion technology allows the merits of each sensor to be fused to overcome the disadvantages of individual sensors and to fully solve the issue of stable movement.

In addition, by fusing heterogeneous sensors, various information such as the surrounding environment of the autonomous vehicle and its current location can be provided. For example, distance sensors such as high-speed LiDARs can be used with RGB cameras for various robot navigation tasks. Lidar sensors can provide 3D position and depth information about objects, while RGB cameras provide 2D position and color information. Therefore, by mapping the 3D location information to the 2D image data, it is possible to visualize more objects in the real world. To this end, the task of identifying the relative position and direction between heterogeneous sensors must be preceded. Therefore, as sensor fusion technology is recently applied to various fields, calibration issues between sensors have emerged as an important issue.

In particular, in order to develop cognitive technology using a camera and a lidar in an autonomous vehicle, calibration technology is very important because accurate information about a relative position (including posture and direction information) between the camera and the lidar is needed. In order to fuse camera and lidar sensor data, the task of considering the marker types that can detect objects quickly and accurately must be the first step for calibration. Most studies use a plane, such as a checker board or box, to perform calibration between the camera and lidar.

At this time, the calibration was performed by mapping the 3D lidar pointer to the camera 2D image and finding the internal and external parameters. These methods cause measurement errors or influence calibration results whenever there is a change in position of the plane or a change in position between the heterogeneous sensors. In particular, when the position change between the heterogeneous sensors becomes large, it is difficult to accurately find the edge of the plane, which makes it difficult to calibrate between the heterogeneous sensors. Far targets with different patterns and colors may also produce different calibration results depending on the sensor characteristics.

Therefore, conventional methods perform calibration by placing the mounting positions between heterogeneous sensors as close as possible to increase accuracy. Therefore, if the calibration is performed after closely disposing the heterogeneous sensors, only an object located at a close distance can be recognized, and thus, an autonomous driving system that needs to recognize a remote object has a disadvantage of using a conventional calibration method.

The present invention provides a method of improving the detection accuracy of a distant object regardless of the mounting position of the heterogeneous sensor by performing calibration between the heterogeneous sensors through a 3D marker board having holes.

The calibration method between heterogeneous sensors according to an embodiment of the present invention comprises the steps of identifying the image data photographed on the marker board through the camera sensor; Identifying point cloud data sensing the marker board through a LiDAR sensor; Recognizing holes present in the marker board in each of the identified image data and point cloud data; Determining a transformation vector for calibration of the camera sensor and the lidar sensor based on the radius of the hole recognized through the image data and the radius of the hole recognized through the point cloud data; And performing calibration between the camera sensor and the lidar sensor by projecting point cloud data of the lidar sensor into image data of the camera sensor using the determined transform vector.

Estimating a calibration parameter specific to the camera sensor corresponding to an internal characteristic including a focal length, distortion and center of the image using a checker board, identifying the image data; and The step of identifying the point cloud data may identify the point cloud data and the image data corresponding to the field of view (FOV) of the camera sensor extracted by using the estimated calibration parameter.

The recognizing of the hole may recognize a hole present in the marker board based on a change amount of a pixel value for pixels included in the identified image data.

The recognizing of the hole may recognize a hole present in the marker board by removing points constituting a line having a predetermined angle among points included in the identified point cloud data.

The determining of the transform vector may be determined based on a first distance from the camera sensor to the marker board and a second distance from the lidar sensor to the marker board.

The first distance and the second distance may be determined using a focal length of the camera sensor, a radius of a hole existing in the marker board, and a radius of a hole recognized through each of the camera sensor and a lidar sensor.

In performing the calibration, the point cloud data of the lidar sensor may be projected to the image data of the camera sensor by using the determined transformation vector in consideration of the movement transformation.

The method may further include detecting a target by using the calibrated camera sensor and the lidar sensor.

According to an embodiment of the present invention, a calibration method between heterogeneous sensors includes: recognizing a hole existing in a marker board through a first sensor; Recognizing a hole present in the marker board through a second sensor different from a first sensor; And determining a conversion vector for calibration of the first sensor and the second sensor based on the radius of the hole recognized by the first sensor and the radius of the hole recognized by the second sensor.

The transformation vector between the coordinate system of the first sensor and the coordinate system of the second sensor for the calibration may be determined based on a relative position difference between the first sensor and the second sensor.

According to an embodiment of the present invention, a calibration apparatus between heterogeneous sensors includes a processor that performs calibration between a camera sensor and a lidar sensor, and the processor identifies image data of photographing the marker board through the camera sensor. Identifying point cloud data sensing the marker board through a LiDAR sensor, recognizing a hole present in the marker board in each of the identified image data and the point cloud data, and through the image data A transformation vector for calibration of the camera sensor and a lidar sensor is determined based on a radius of the recognized hole and a radius of the hole recognized through the point cloud data, and the point cloud data of the lidar sensor is determined using the determined transformation vector. Of the camera sensor already By projecting the image data, it is possible to perform calibration between the camera sensor and the lidar sensor.

The processor estimates a calibration parameter specific to the camera sensor corresponding to an internal characteristic including a focal length, a distortion, and a center of the image of the camera sensor using a checker board, and extracts the extracted calibration parameter using the estimated calibration parameter. Image data and point cloud data corresponding to a field of view (FOV) of the camera sensor may be identified.

The processor may recognize a hole present in the marker board based on an amount of change of a pixel value with respect to pixels included in the identified image data.

The processor may recognize a hole present in the marker board by removing points constituting a line having a predetermined angle among points included in the identified point cloud data.

The processor may determine based on a first distance from the camera sensor to a marker board and a second distance from the lidar sensor to the marker board.

The first distance and the second distance may be determined using a focal length of the camera sensor, a radius of a hole existing in the marker board, and a radius of a hole recognized through each of the camera sensor and a lidar sensor.

The processor may project the point cloud data of the lidar sensor to the image data of the camera sensor by using the determined transform vector in consideration of a movement transformation.

The processor may detect a target using a camera sensor and a lidar sensor on which the calibration is performed.

According to an embodiment of the present invention, by performing calibration between heterogeneous sensors through a 3D marker board having holes, detection accuracy of a remote object may be improved regardless of a mounting position of the heterogeneous sensors.

1 is a diagram illustrating a calibration system between heterogeneous sensors according to an embodiment of the present invention.
2 is a diagram illustrating an example of a marker board according to an embodiment of the present invention.
3 is a diagram illustrating an example of extracting a hole from heterogeneous sensor data according to an exemplary embodiment of the present invention.
4 illustrates a method for determining a transform vector according to an embodiment of the present invention.
5 is a diagram illustrating a calibration method between heterogeneous sensors according to an embodiment of the present invention.

Hereinafter, exemplary embodiments will be described in detail with reference to the accompanying drawings. However, various changes may be made to the embodiments so that the scope of the patent application is not limited or limited by these embodiments. It is to be understood that all changes, equivalents, and substitutes for the embodiments are included in the scope of rights.

The terminology used herein is for the purpose of description and should not be construed as limiting. Singular expressions include plural expressions unless the context clearly indicates otherwise. In this specification, terms such as "comprise" or "have" are intended to indicate that there is a feature, number, step, operation, component, part, or combination thereof described on the specification, and one or more other features. It is to be understood that the present invention does not exclude the possibility of the presence or the addition of numbers, steps, operations, components, components, or a combination thereof.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art. Terms such as those defined in the commonly used dictionaries should be construed as having meanings consistent with the meanings in the context of the related art and shall not be construed in ideal or excessively formal meanings unless expressly defined in this application. Do not.

In addition, in the description with reference to the accompanying drawings, the same components regardless of reference numerals will be given the same reference numerals and duplicate description thereof will be omitted. In the following description of the embodiment, when it is determined that the detailed description of the related known technology may unnecessarily obscure the gist of the embodiment, the detailed description thereof will be omitted.

1 is a diagram illustrating a calibration system between heterogeneous sensors according to an embodiment of the present invention.

Referring to FIG. 1, the calibration system 100 between heterogeneous sensors of the present invention includes a first sensor 110, a second sensor 120 different from the first sensor 110, a marker board 130, and a processor 140. It can be configured as. For example, the first sensor 110 and the second sensor 120 may be one of various sensors such as a camera sensor, a lidar sensor, a laser sensor, and the like.

As described above, in order to fuse data collected through heterogeneous sensors, coordinate transformation, ie, calibration, between frames of each sensor must be performed. This requires identifying unique features in each of the data collected by the heterogeneous sensors, and establishing a correspondence relationship between the heterogeneous sensors based on the identified unique features.

The present invention provides a method for performing calibration between heterogeneous sensors using a marker board 130 having a plurality of holes as shown in FIG. 2 to provide a method for detecting a target using heterogeneous sensors. In this case, the holes 131 to 134 existing in the marker board 130 may be circular, but are not limited to circular and may have various shapes such as polygons. In addition, although four holes 131 ˜ 134 exist in the marker board 130 of FIG. 2, the number of such holes may be only one example and may exist in various numbers.

The processor 140 may identify data photographing the marker board 130 from the first sensor 110 and the second sensor 120, and recognize holes existing in the marker board 130 from the identified data. have. In addition, the processor 140 may calibrate the first sensor 110 and the second sensor 120 based on the radius of the hole recognized by the first sensor 110 and the radius of the hole recognized by the second sensor 120. Can determine the transform vector.

In other words, the coordinate system of the first sensor 110 and the coordinate system of the second sensor 120 may coincide with each other through the transformation vector. Such a transformation vector may correspond to the first sensor 110 and the second sensor 120. The relative position difference, that is, the distance between the first sensor 110 and the marker board 130 and the distance between the second sensor 120 and the marker board 130 may be determined. A more detailed transform vector determination method will be described in more detail later with reference to FIG. 4.

Thereafter, the processor 140 may perform calibration between the heterogeneous sensors by converting the coordinate system of the first sensor 110 and the coordinate system of the second sensor 120 to match each other using the determined transformation vector.

The existing calibration method between heterogeneous sensors may cause measurement errors whenever there is a change in the position of the marker board or a change in position between the sensors, or when the remote target is recognized, different results may be generated depending on the characteristics of the sensor itself. Therefore, in the conventional calibration method between heterogeneous sensors, calibration is performed by placing mounting positions between heterogeneous sensors as close as possible to increase the recognition accuracy of the target, and thus only targets located within relatively close distances can be recognized. In autonomous driving systems, there was a disadvantage.

However, in the calibration method between the heterogeneous sensors of the present invention, since the calibration is performed by using the marker board 130 in which the holes exist, the autonomous driving system or the like can be recognized with high accuracy regardless of the mounting position between the heterogeneous sensors. Can also be greatly utilized.

3 is a diagram illustrating an example of recognizing a hole from heterogeneous sensor data according to an exemplary embodiment of the present invention.

The present invention provides a calibration method between a camera sensor and a lidar sensor among calibration methods between various heterogeneous sensors. Therefore, in order to perform the calibration between the camera sensor and the lidar sensor, it is necessary to first recognize the holes existing in the marker board 130 from the data photographed by the camera sensor and the lidar sensor.

First, when one of the heterogeneous sensors is a camera sensor, the processor 140 may use a checker board to determine a calibration parameter unique to the camera sensor corresponding to internal characteristics of the camera sensor itself, such as a focal length, distortion, and center of the image of the camera sensor. It can be estimated. This may be an essential step for the convergence of point data and image data collected through heterogeneous sensors, that is, a camera sensor and a lidar sensor. As a result of the calibration performed using the calibration parameters unique to the camera sensor, a camera matrix, a distortion coefficient, a camera projection matrix, and the like may be obtained. In this case, the camera projection matrix may be obtained by combining the inner matrix and the outer matrix, as shown in Equation 1 below.

<Equation 1>

이고, 내부 행렬

는 하기의 식 2와 같이 2D 변환 행렬(2D translational matrix), 2D 스케일링 행렬(2D scaling matrix) 및 2D 전단 행렬(2D shear matrix)의 곱으로 분해될 수 있다. here,

, The inner matrix

May be decomposed into a product of a 2D translation matrix, a 2D scaling matrix, and a 2D shear matrix as shown in Equation 2 below.

<Equation 2>

와

는 이미지 데이터의 중심이고,

와

는 픽셀 유닛의 초점 거리이며, s 는 전단 계수이다. here

Wow

Is the center of the image data,

Wow

Is the focal length of the pixel unit and s is the shear coefficient.

FIG. 3A illustrates an example of recognizing a hole existing in the marker board 130 from the identified point cloud data by sensing the marker board 130 using a LiDAR sensor.

The processor 140 may segment the point cloud data collected through the lidar sensor based on a field of view (FOV) of the camera sensor. In this case, the field of view of the camera sensor may be extracted using calibration parameters inherent to the camera sensor. The divided point cloud data may include various objects around the marker board 130 as well as the sensed marker board 130.

The processor 140 calculates normal and curvature of all objects to extract only the marker board 130 from the point cloud data corresponding to the field of view of the camera sensor, and has a plane having the same normal and curvature. Can be identified. In this case, the identified plane may correspond to the marker board 130, and the processor 140 may extract only the identified plane and remove the remaining part.

The processor 140 may recognize a hole present in the marker board 130 by removing points constituting a line having a predetermined angle from the point cloud data included in the extracted plane. More specifically, the processor 140 finds a line having the same angle among the points mapped to the plane, extracts a line having the same angle through a line segmenting operation, and extracts the points mapped to the corresponding line. Can be removed Through this operation, the processor 140 may leave only points of a circle corresponding to a hole in a plane corresponding to the marker board 130.

Thereafter, the processor 140 may determine a threshold of a center point and a radius of a circle in order to recognize a hole existing in the marker board 130, and identify a hole having a predetermined radius among various detected polygons (including a circle). . For example, in the present invention, the radius was determined to be 20 to 22 cm in order to utilize the autonomous driving system, and four circles within the corresponding threshold range could be recognized as shown in FIG.

3B illustrates an example of recognizing a hole existing in the marker board 130 from the identified image data by sensing the marker board 130 using a camera sensor.

The processor 140 extracts the boundary of all objects including the entire marker board 130 from the collected image data corresponding to the field of view of the extracted camera sensor using calibration parameters inherent to the camera sensor, and includes a hole (among them). Only the area 130 may be extracted separately. In this case, the image data from which the edges of all objects are extracted may be in an RGB form, and the processor 140 may convert the image data extracted in the RGB form into image data having a gray scale form.

3 크기의 형렬을 필터로 사용하여 마커 보드(130) 및 마커 보드(130) 내에 존재하는 홀의 에지를 추출할 수 있다. 이때, 프로세서(140)는 그레이 스케일 형태의 이미지 데이터에 포함된 모든 픽셀에 대해 필터를 적용하는데 이미지 데이터 내의 어느 한 점을 기준으로 각 방향의 앞뒤 픽셀 값을 비교하여 그 변화량을 검출함으로써 가장자리 에지를 추출할 수 있다.The processor 140 may recognize a hole present in the marker board 130 based on an amount of change of pixel values for pixels included in the converted gray scale image data. In more detail, the processor 140 may execute 3 for image data in the form of gray scale.

An edge of the hole existing in the marker board 130 and the marker board 130 may be extracted using the three-sized matrix as a filter. In this case, the processor 140 applies a filter to all pixels included in the gray scale image data, and compares the front and rear pixel values in each direction based on a point in the image data to detect the change amount of the edge edge. Can be extracted.

The processor 140 finds all circular edges within the range of the center point and radius of the preset circle among the extracted edges, and determines the slope threshold of the boundary among the found circular edges to form a circle having strong boundary strength. Only bays can be extracted. The slope threshold of such a boundary may be determined by a user based on the found circular edge image, and the processor 140 may use only the sharper circle of the found circular edges by using the determined slope threshold of the boundary. Can be extracted. In one example, the processor 140 of the present invention could recognize four circles from the image data of the camera sensor as shown in FIG.

4 illustrates a method for determining a transform vector according to an embodiment of the present invention.

When the holes present in the marker board 130 are recognized from the data collected through the camera sensor and the lidar sensor as shown in FIG. 3, the processor 140 matches the coordinate system between the camera sensor and the lidar sensor using the recognized holes. Calibration can be performed. In the present invention, since the camera sensor and the lidar sensor of the autonomous vehicle are mounted in the same direction in three axes, the calibration between the heterogeneous sensors is performed by considering only the translation difference between the sensors without considering the rotation difference. By doing this, the amount of computation can be minimized.

That is, the processor 140 calculates the internal parameters of the two sensors, that is, the calibration parameters and the external parameters of the camera sensors, that is, the conversion vectors, and collects them through the Lidar sensor through Equation 3 below. Calibration between the two heterogeneous sensors may be performed by projecting the points of the collected point cloud data onto the image data collected through the camera sensor.

<Equation 3>

)의 구성요소인

를 계산하기 전

를 계산할 수 있다. 이때,

는 라이다 센서로부터 마커 보드(130)까지의 제1 거리(

)과 카메라 센서로부터 마커 보드(130)까지의 제2 거리(

)를 이용하여 하기의 식 4와 같이 계산될 수 있다.In this case, the conversion vector which is an external parameter may be determined based on a distance from the camera sensor and the lidar sensor to the marker boat 130. First, the processor 140 converts the transform vector (

) Is a component of

Before calculating

Can be calculated. At this time,

Is the first distance from the lidar sensor to the marker board 130 (

) And the second distance from the camera sensor to the marker board 130

) Can be calculated as in Equation 4 below.

<Equation 4>

는 카메라 센서의 초점 거리(

), 포인트 클라우드 데이터 및 이미지 데이터로부터 각각 인식된 홀의 반지름(

)을 이용하여 하기의 식 5와 같이 계산될 수 있다.At this time, the distance from each sensor to the marker board 130

Is the focal length of the camera sensor (

), The radius of the hole recognized from the point cloud data and the image data, respectively (

) Can be calculated as shown in Equation 5 below.

<Equation 5>

를 이용하여 나머지 변환 벡터인

를 하기의 식 6을 이용하여 계산할 수 있다.The processor 140 then calculates

Using the remainder of the transformation vector

It can be calculated using Equation 6 below.

<Equation 6>

)를 이용하여 라이다 센서의 포인트 클라우드 데이터를 카메라 센서의 이미지 데이터로 투영함으로써 프로세서(140)는 카메라 센서 및 라이다 센서 간의 캘리브레이션을 수행할 수 있다.The transform vector determined in this way (

The processor 140 may perform calibration between the camera sensor and the lidar sensor by projecting the point cloud data of the lidar sensor into the image data of the camera sensor.

5 is a diagram illustrating a calibration method between heterogeneous sensors according to an embodiment of the present invention.

In operation 510, the processor 140 of the calibration system 100 may estimate calibration parameters inherent to the camera sensor such as a focal length, distortion, and center of the image of the camera sensor. In this case, the processor 140 may estimate a calibration parameter inherent to the camera sensor using a checker board.

The processor 140 may obtain a camera matrix, a distortion coefficient, a camera projection matrix, etc., as a result of the calibration of the camera sensor performed by using the calibration parameters unique to the camera sensor, and through this, the camera sensor and the lidar sensor. The convergence of the collected image data and the point cloud data may be performed.

In operation 520, the processor 140 may identify image data of the marker board 140 according to a camera sensor, and in operation 530, the marker board may be sensed through a LiDAR sensor. One point cloud data can be identified. In this case, the holes present in the marker board 130 may be circular, but are not limited to circular and may have various shapes such as polygons and may exist in various numbers.

In operation 540, the processor 140 may recognize a hole present in the marker board 130 from the identified image data by sensing the marker board 130 using a camera sensor. The processor 140 extracts the boundary of all objects including the entire marker board 130 from the collected image data corresponding to the field of view of the extracted camera sensor using calibration parameters inherent to the camera sensor, and includes a hole (among them). Only the area 130 may be extracted separately. In this case, the image data from which the edges of all objects are extracted may be in an RGB form, and the processor 140 may convert the image data extracted in the RGB form into image data having a gray scale form.

3 크기의 형렬을 필터로 사용하여 마커 보드(130) 및 마커 보드(130) 내에 존재하는 홀의 에지를 추출할 수 있다. 이때, 프로세서(140)는 그레이 스케일 형태의 이미지 데이터에 포함된 모든 픽셀에 대해 필터를 적용하는데 이미지 데이터 내의 어느 한 점을 기준으로 각 방향의 앞뒤 픽셀 값을 비교하여 그 변화량을 검출함으로써 가장자리 에지를 추출할 수 있다.The processor 140 may recognize a hole present in the marker board 130 based on an amount of change of pixel values for pixels included in the converted gray scale image data. In more detail, the processor 140 may execute 3 for image data in the form of gray scale.

An edge of the hole existing in the marker board 130 and the marker board 130 may be extracted using the three-sized matrix as a filter. In this case, the processor 140 applies a filter to all pixels included in the gray scale image data, and compares the front and rear pixel values in each direction based on a point in the image data to detect the change amount of the edge edge. Can be extracted.

The processor 140 finds all circular edges within the range of the center point and radius of the preset circle among the extracted edges, and determines the slope threshold of the boundary among the found circular edges to form a circle having strong boundary strength. Only bays can be extracted. The slope threshold of such a boundary may be determined by a user based on the found circular edge image, and the processor 140 may use only the sharper circle of the found circular edges by using the determined slope threshold of the boundary. Can be extracted.

In operation 550, the processor 140 may recognize a hole present in the marker board 130 from the identified point cloud data by sensing the marker board 130 using a lidar sensor. The processor 140 may segment the point cloud data collected through the lidar sensor based on a field of view (FOV) of the camera sensor. In this case, the field of view of the camera sensor may be extracted using calibration parameters inherent to the camera sensor. The divided point cloud data may include various objects around the marker board 130 as well as the sensed marker board 130.

The processor 140 calculates normal and curvature of all objects to extract only the marker board 130 from the point cloud data corresponding to the field of view of the camera sensor, and has a plane having the same normal and curvature. Can be identified. In this case, the identified plane may correspond to the marker board 130, and the processor 140 may extract only the identified plane and remove the remaining part.

The processor 140 may recognize a hole present in the marker board 130 by removing points constituting a line having a predetermined angle from the point cloud data included in the extracted plane. More specifically, the processor 140 finds a line having the same angle among the points mapped to the plane, extracts a line having the same angle through a line segmenting operation, and extracts the points mapped to the corresponding line. Can be removed Through this operation, the processor 140 may leave only points of a circle corresponding to a hole in a plane corresponding to the marker board 130.

Thereafter, the processor 140 may determine a threshold of a center point and a radius of a circle in order to recognize a hole existing in the marker board 130, and identify a hole having a predetermined radius among various detected polygons (including a circle).

)과 카메라 센서로부터 마커 보드(130)까지의 제2 거리(

)는 카메라 센서의 초점 거리(

), 포인트 클라우드 데이터 및 이미지 데이터로부터 각각 인식된 홀의 반지름(

)을 이용하여 계산될 수 있다. In step 560, the processor 140 may determine the transform vector based on the distance from the camera sensor and the lidar sensor to the marker boat 130. At this time, the first distance from the lidar sensor to the marker board 130 (

) And the second distance from the camera sensor to the marker board 130

) Is the focal length of the camera sensor (

), The radius of the hole recognized from the point cloud data and the image data, respectively (

Can be calculated using

In operation 570, the processor 140 may perform the calibration between the camera sensor and the lidar sensor by projecting the point cloud data of the lidar sensor into the image data of the camera sensor using the determined transformation vector. have. The processor 140 may more accurately detect a target located at a far distance by using a calibrated camera sensor and a lidar sensor.

The method according to the embodiment may be embodied in the form of program instructions that can be executed by various computer means and recorded in a computer readable medium. The computer readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the media may be those specially designed and constructed for the purposes of the embodiments, or they may be of the kind well-known and available to those having skill in the computer software arts. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tape, optical media such as CD-ROMs, DVDs, and magnetic disks, such as floppy disks. Magneto-optical media, and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine code generated by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like. The hardware device described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

The software may include a computer program, code, instructions, or a combination of one or more of the above, and configure the processing device to operate as desired, or process it independently or collectively. You can command the device. Software and / or data may be any type of machine, component, physical device, virtual equipment, computer storage medium or device in order to be interpreted by or to provide instructions or data to the processing device. Or may be permanently or temporarily embodied in a signal wave to be transmitted. The software may be distributed over networked computer systems so that they may be stored or executed in a distributed manner. Software and data may be stored on one or more computer readable recording media.

Although the embodiments have been described with reference to the accompanying drawings, those skilled in the art may apply various technical modifications and variations based on the above. For example, the described techniques may be performed in a different order than the described method, and / or components of the described systems, structures, devices, circuits, etc. may be combined or combined in a different manner than the described method, or other components. Or even if replaced or replaced by equivalents, an appropriate result can be achieved.

Therefore, other implementations, other embodiments, and equivalents to the claims are within the scope of the following claims.

100: Calibration system between different sensors
110: first sensor
120: second sensor
130: marker board
140: processor

Claims (19)

delete In the calibration method between different sensors,
Using the checker board to estimate calibration parameters inherent to the camera sensor corresponding to internal characteristics including focal length, distortion and center of the image of the camera sensor;
Identifying image data of photographing the marker board through the camera sensor based on a field of view (FOV) of the camera sensor extracted using the estimated calibration parameter;
Identifying point cloud data sensing the marker board through a LiDAR sensor based on a field of view of the camera sensor;
Recognizing holes present in the marker board in each of the identified image data and point cloud data;
Determining a transformation vector for calibration of the camera sensor and the lidar sensor based on the radius of the hole recognized through the image data and the radius of the hole recognized through the point cloud data; And
Performing calibration between the camera sensor and the lidar sensor by projecting the point cloud data of the lidar sensor into the image data of the camera sensor using the determined transform vector.
Calibration method between heterogeneous sensors comprising a. The method of claim 2,
Recognizing the hole,
And a method for calibrating a hole existing in the marker board based on an amount of change of a pixel value for pixels included in the identified image data. In the calibration method between different sensors,
Identifying image data of the marker board according to the camera sensor;
Identifying point cloud data sensing the marker board through a LiDAR sensor;
Recognizing holes present in the marker board in each of the identified image data and point cloud data;
Determining a transformation vector for calibration of the camera sensor and the lidar sensor based on the radius of the hole recognized through the image data and the radius of the hole recognized through the point cloud data; And
Performing calibration between the camera sensor and the lidar sensor by projecting the point cloud data of the lidar sensor into the image data of the camera sensor using the determined transform vector.
Including,
Recognizing the hole,
The calibration method between heterogeneous sensors for recognizing a hole present in the marker board by removing points constituting a line having a predetermined angle among the points included in the identified point cloud data. delete In the calibration method between different sensors,
Identifying image data of the marker board according to the camera sensor;
Identifying point cloud data sensing the marker board through a LiDAR sensor;
Recognizing holes present in the marker board in each of the identified image data and point cloud data;
The first distance from the camera sensor to the marker board and the lidar sensor determined using a focal length of the camera sensor, a radius of a hole existing in the marker board, and a radius of a hole recognized through each of the camera sensor and a lidar sensor. Determining a transform vector for calibration of the camera sensor and a lidar sensor based on a second distance from the marker board to the marker board; And
Performing calibration between the camera sensor and the lidar sensor by projecting the point cloud data of the lidar sensor into the image data of the camera sensor using the determined transform vector.
Calibration method between heterogeneous sensors comprising a. The method of claim 2,
The performing of the calibration may include:
And calibrating the point cloud data of the lidar sensor to the image data of the camera sensor by using the determined transform vector. The method of claim 2,
Detecting a target using the calibrated camera sensor and a lidar sensor
Calibration method between heterogeneous sensors further comprising. delete delete A computer-readable recording medium having recorded thereon a program for executing the method of any one of claims 2 to 4 and 6 to 8. delete Processor performing calibration between camera sensor and lidar sensor
Including,
The processor,
Using a checker board to estimate calibration parameters inherent to the camera sensor corresponding to internal characteristics including focal length, distortion and center of the image of the camera sensor,
Identifying image data of the marker board through the camera sensor based on a field of view (FOV) of the camera sensor extracted using the estimated calibration parameter,
Identify the point cloud data sensing the marker board through a LiDAR sensor based on a field of view of the camera sensor,
Recognizes a hole present in the marker board in each of the identified image data and point cloud data,
A transformation vector for calibration of the camera sensor and the lidar sensor is determined based on the radius of the hole recognized through the image data and the radius of the hole recognized through the point cloud data.
And calibrating between the camera sensor and the lidar sensor by projecting the point cloud data of the lidar sensor into the image data of the camera sensor using the determined transform vector. The method of claim 13,
The processor,
And a calibration device for heterogeneous sensors that recognizes a hole present in the marker board based on a change amount of a pixel value for pixels included in the identified image data. Processor performing calibration between camera sensor and lidar sensor
Including,
The processor,
Identify the image data of the marker board according to the camera sensor, and identify the point cloud data sensing the marker board through a LiDAR sensor,
Recognizes a hole present in the marker board in the identified image data,
Recognizing a hole present in the marker board by removing points constituting a line having a predetermined angle among the points included in the identified point cloud data,
A transformation vector for calibration of the camera sensor and the lidar sensor is determined based on the radius of the hole recognized through the image data and the radius of the hole recognized through the point cloud data.
And calibrating between the camera sensor and the lidar sensor by projecting the point cloud data of the lidar sensor into the image data of the camera sensor using the determined transform vector. delete Processor performing calibration between camera sensor and lidar sensor
Including,
The processor,
Identify the image data of the marker board according to the camera sensor, and identify the point cloud data sensing the marker board through a LiDAR sensor,
Recognize a hole present in the marker board in each of the identified image data and point cloud data,
The first distance from the camera sensor to the marker board and the lidar sensor determined using the focal length of the camera sensor, the radius of the hole present in the marker board, and the radius of the hole recognized through each of the camera sensor and the lidar sensor. Determine a transform vector for calibration of the camera sensor and a lidar sensor based on a second distance from the marker board to
And calibrating between the camera sensor and the lidar sensor by projecting the point cloud data of the lidar sensor into the image data of the camera sensor using the determined transform vector. The method of claim 13,
The processor,
And a projection apparatus for projecting point cloud data of the LiDAR sensor into image data of the camera sensor using the determined transform vector, in consideration of movement transformation. The method of claim 13,
The processor,
The calibration device between the heterogeneous sensor for detecting a target using the camera sensor and the lidar sensor that the calibration is performed.

Images