KR20200023974A - Method and apparatus for synchronization of rotating lidar and multiple cameras - Google Patents

Abstract

According to the present invention, in the method for synchronizing a sensor between a rotating lidar and multiple cameras, the method is performed by a synchronization apparatus. The method comprises: an angle comparison step of comparing angle information obtained from the rotating lidar with a previously input target angle and searching for an angle closest to the target angle; a prediction time calculation step of calculating a camera shooting prediction time in the next frame in consideration of the offset between the detected angle and the actual angle according to the rotation of the rotating lidar and the rotation speed of the rotating lidar; and a process branch step of inputting a prediction time into a prediction time queue and branching the current process such that one of branched processes waits to receive information from the rotating lidar. According to the use of the present invention, errors between data between the rotating lidar and a multi-camera sensor can be reduced, thereby enhancing the results of all kinds of algorithms using visual sensors such as calibration, recognition, obstacle determination, autonomous action judgment and the like.

Description

 METHOD AND APPARATUS FOR SYNCHRONIZING SENSOR BETWEEN ROTARY LIDAER AND MULTI-CAMERA {METHOD AND APPARATUS FOR SYNCHRONIZATION OF ROTATING LIDAR AND MULTIPLE CAMERAS}

The present invention relates to a synchronization device for performing sensor synchronization between a rotary lidar and multiple cameras and a synchronization method for performing the same.

Unmanned autonomous systems of ground equipment, including autonomous vehicles, are widely used with various types of visual sensors such as rotary distance sensors (lidars) and cameras to recognize and determine the environment in real time. Basically, time synchronization between visual sensors works. In general, visual sensor synchronization between camera types can be relatively easily synchronized using a device that provides time information such as GPS. However, it is impossible to ensure precise synchronization between a rotary lidar that acquires the distance information of the surroundings by only time information provided from an external device and a camera that acquires a specific range of visual information at once.

  In order to solve such a problem, the present invention includes a technique for accurately synchronizing a method between a rotary lidar and a camera and a device, which are essentially used to secure visual information. In this regard, all lidars, except for solid-state lidars, are rotary lidars. As mentioned above, the conventional method of synchronizing between the rotary lidar and the camera uses the time information of an external device to store time information when the camera is photographed and time information of 3D data obtained by rotating the lidar. The most similar time value among the recorded time information is used for matching, or the camera is photographed at the point of time when data is obtained from the rotatable lidar at the software level, and the synchronization is used. This synchronized data is sufficient for general use, but is not suitable for use in algorithms used in unmanned autonomous systems that require sophisticated matching between Lidar and camera. To solve this problem, the Karlsruhe Institute of Technology in Germany solved the time synchronization problem by installing a hardware switch on a rotary lidar by physically providing a synchronization signal to the camera. However, this approach suffers from the durability issues associated with physical switches. In particular, in the unmanned autonomous system for defense that must travel in all-weather environment, it is difficult to use a physical synchronization signal device.

  Unsophisticated time synchronization between the rotating lidar and the camera introduces fundamental errors in the matching of data, which can be fatal in the operation of the visual sensor in an unmanned autonomous system, which must ensure stable results in a wide variety of outdoor environments. There is a problem that existing devices and methods have limitations to fundamentally overcome this disadvantage.

Accordingly, an object of the present invention is to provide a method and apparatus for synchronizing a sensor between a rotatable lidar and multiple cameras.

Another object of the present invention is to provide a method and an apparatus for providing the most accurate photographing moment of a camera through the angle information of the rotatable lidar in a system in which the rotatable lidar and the camera are used simultaneously.

In the sensor synchronization method between a rotary lidar and multiple cameras according to the present invention for solving the above problems, the method is performed by a synchronization device, and input in advance with the angle information obtained from the rotary lidar An angle comparison step of searching for the closest angle to the target angle by comparing the target angles; A prediction time calculating step of calculating a camera photographing prediction time in a next frame in consideration of an offset between the found angle and an actual angle according to the rotation of the rotary lidar and the rotational speed of the rotary lidar; And a process branching step of inputting the prediction time to a prediction time queue, branching a current process and waiting for receiving information from the rotatable lidar.

In one embodiment, in the process branching step, the other one of the branched processes may compare the current time with the prediction time input to the prediction time queue in real time.

According to an embodiment of the present disclosure, if the corresponding prediction time of the prediction times reaches the current time, after removing the corresponding prediction time from the queue, generating a photographing operation signal and transmitting the photographing operation signal to a camera sensor, further comprising: can do.

In an embodiment, in the photographing operation signal generating step, a camera corresponding to the current angle determined based on the found angle, the offset, the rotational speed, and the corresponding prediction time is activated by the photographing operation signal, Each of the multiple cameras may be triggered by the photographing operation signal at different times.

In an embodiment, in the photographing operation signal generating step, the angle of error of the rotatable lidar and the arrangement position and the coverage range of the multiple cameras may be synchronized based on different prediction times for each of the multiple cameras. Can be. In this case, when it is determined by the rotatable lidar that the object is disposed in a region overlapping between the first camera and the second camera among the multiple cameras, the first camera and the second camera may be simultaneously triggered.

According to an embodiment, the synchronization device may include a sensor information receiver, an angle information calculator, a prediction time waiter, and a camera signal generator. In this case, the sensor information receiver may receive both the distance information and the angle information from the rotatable lidar. On the other hand, if it is determined that the target angle does not converge within the threshold range based on the obtained angle information, the sensor information receiver may receive the distance information and the angle information from the rotatable lidar again. In this case, when it is determined that the target angle is converged within the threshold range based on the obtained angle information, the prediction time waiting unit and the sensor information receiving unit are the photographing operation signals according to the prediction time calculating step and the process branching step. Development steps can be performed.

The present invention provides an accurate method of synchronizing a rotational lidar and a camera sensor which are essentially used in an autonomous system.

Accordingly, by acquiring the most accurate data at the sensor information acquisition stage, the error between the heterogeneous sensor data is reduced and the results of all kinds of algorithms using the visual sensor such as calibration, recognition, obstacle determination, autonomous behavior determination, etc. are improved. There is an effect that can be done.

1 is a flow chart of a rotary lidar and a camera synchronization method according to the present invention.
2 is an apparatus diagram of a method for synchronizing a rotary lidar and a camera according to the present invention.

The above-described features and effects of the present invention will become more apparent from the following detailed description taken in conjunction with the accompanying drawings, and thus, those skilled in the art to which the present invention pertains may easily implement the technical idea of the present invention. Could be.

As the inventive concept allows for various changes and numerous embodiments, particular embodiments will be illustrated in the drawings and described in detail. However, this is not intended to limit the present invention to specific embodiments, it should be understood to include all changes, equivalents, and substitutes included in the spirit and scope of the present invention.

In describing each drawing, like reference numerals are used for like elements.

Terms such as first and second may be used to describe various components, but the components should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from another.

For example, without departing from the scope of the present invention, the first component may be referred to as the second component, and similarly, the second component may also be referred to as the first component. The term and / or includes a combination of a plurality of related items or any item of a plurality of related items.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art.

Terms such as those defined in the commonly used dictionaries should be construed as having meanings consistent with the meanings in the context of the related art, and shall not be construed in ideal or excessively formal meanings unless expressly defined in this application. Should not.

The suffix modules, blocks, and parts for components used in the following description are given or mixed in consideration of ease of specification, and do not have distinct meanings or roles from each other.

Hereinafter, with reference to the accompanying drawings, preferred embodiments of the present invention will be described to be easily carried out by those of ordinary skill in the art. In the following description of the embodiments of the present invention, when it is determined that a detailed description of a related known function or known configuration may unnecessarily obscure the subject matter of the present invention, the detailed description thereof will be omitted.

Hereinafter, a method of synchronizing a sensor between a rotatable lidar and multiple cameras, a synchronization device for performing the same, and a multiple camera system will be described.

In this regard, the present invention includes a method and an apparatus for providing the most accurate camera shooting moment through the angle information of the rotatable lidar in a system where the rotatable lidar and the camera are used simultaneously. In the case of a rotary lidar sensor, the laser rangefinder is rotated through an internal mirror or an external device to obtain peripheral distance information. Because the physical method of measuring the ambient is different from the camera sensor, a separate device is needed to physically minimize the gap between the shooting of the two sensors. When using the method and the apparatus included in the present invention, it is possible to minimize the physical problem to obtain the distance information that is the maximum match with the image information obtained from the camera. The present invention can also be expected to improve the results of calibration, recognition, autonomy, and the like caused by reducing errors between sensor information by seeking the most accurate information at the sensor information acquisition stage. The flowchart of the method included in this invention is the same as in FIG. 1, and FIG. 2 represents the device diagram of the present invention. The method of the invention expressed in flow chart is as follows.

1 and 2, the detailed configuration and operation of the present invention in the method and apparatus for synchronizing a rotary lidar and a camera according to the present invention are as follows.

(1) The method proposed in the present invention receives distance information and angle information from a rotary lidar sensor. The rotary lidar sensor referred to in the present invention is a sensor composed of a single or n distance measuring module having a specific resolution while rotating the distance measuring sensor.

(2) The rotating lidar-camera synchronizing apparatus extracts angle information from input information in real time, compares the target angle previously input with the angle of the rider continuously input, and finds the most matching angle. Serve

(3) Since the rotary lidar continues to rotate even while calculating the angle information in the above step (2), a difference (offset) from the actual current lidar's angle occurs. Because of this offset, if the camera signal (trigger) is generated by the camera camera after comparing the angles, the shooting gap between the two sensors cannot be minimized because the timing of the camera is already missed. However, the offset caused by this physical time can be calculated in advance because it shows a nearly uniform value each time. The synchronization device calculates the camera shooting prediction time in the next frame by considering the calculated offset and the rotational speed of the rotating lidar.

(4) The predicted time calculated in step (3) is entered into the predictive time queue and branches the current process. One of the forked processes waits to receive information from the rotating lidar again, and the other compares the current time with the predicted time entered in the predicted time queue in real time.

(5) If there is a predicted time that reaches the current time among the predicted times compared in the step (4), the corresponding predicted time is removed from the queue, and a photographing operation signal is generated by a camera sensor. The process continuously monitors the estimated time queue.

In this regard, referring to FIG. 1, the method for synchronizing a sensor between a rotatable lidar and multiple cameras according to the present invention may perform the following steps. At this time, each step is not limited to the order, the order may be changed or two or more steps may be performed simultaneously according to the application.

In the angle comparison step (S110), the angle information obtained from the rotatable lidar is compared with a previously input target angle to search for an angle closest to the target angle. On the other hand, in the prediction time calculation step (S120), the camera photographing prediction in the next frame in consideration of the offset between the searched angle and the actual angle according to the rotation of the rotary lidar and the rotational speed of the rotary lidar. Calculate the time

Meanwhile, in the process branching step S130, the prediction time is input to the prediction time queue, and the current process branches to wait for one of the branched processes to receive information from the rotatable lidar. In addition, in the process branching step S130, the other one of the branched processes may compare the prediction time input to the prediction time queue with the current time in real time.

On the other hand, in the shooting operation signal generation step (S140), if the corresponding prediction time of the prediction time reaches the current time, after removing the corresponding prediction time in the queue, it is possible to generate a shooting operation signal and transfer it to the camera sensor. .

In addition, in operation S140 of generating a photographing operation signal, a camera corresponding to the current angle determined based on the found angle, the offset, the rotation speed, and the corresponding prediction time may be activated by the photographing operation signal. . In this case, each of the multiple cameras may be triggered by the photographing operation signal at different times.

In addition, in the photographing operation signal generation step (S140), in consideration of the angular error of the rotatable lidar, the arrangement position and the coverage range of the multiple cameras, the multiple cameras may be synchronized based on different prediction times. have. In this case, when an object is disposed in a region overlapping between the first camera and the second camera, the first camera and the second camera may be simultaneously triggered.

2 is a block diagram of a device implemented by the method described above. Sensor information receiving unit 210 for receiving and processing the sensor information from the rotary lidar furnace 100, the angle information operation unit 220 for separating the angle information from the received sensor information and performing a comparison operation with the target angle, Next A prediction time waiting unit 230 for predicting and waiting for a camera trigger generation time in a frame, and a camera signal generator 240 for generating a camera photographing operation signal while comparing the prediction time with the current time in real time.

On the other hand, the synchronization device 200 for performing the sensor synchronization method between the rotary lidar and the multiple cameras according to the present invention can perform the following operations. The sensor information receiver 210 is configured to receive both distance information and the angle information from the rotatable lidar 100. Meanwhile, the angle information calculator 220 compares the angle information obtained from the rotatable lidar 100 with a target angle input in advance, and searches for an angle closest to the target angle. In addition, the prediction time waiting unit 230 considers the offset between the searched angle and the actual angle according to the rotation of the rotatable lidar and the rotational speed of the rotatable lidar. Calculate

In addition, the camera signal generator 240 inputs the prediction time to a prediction time queue, and branches a current process so that one of the branched processes waits to receive information from the rotatable lidar. Meanwhile, the prediction time waiter 230 may compare the current time with the prediction times input to the prediction time queue in real time in the other branched process. Accordingly, when the corresponding prediction time among the prediction times reaches the current time, the camera signal generator 240 may remove the corresponding prediction time from the queue, generate a photographing operation signal, and transmit it to the camera sensor.

The camera signal generator 240 may activate a camera corresponding to the current angle determined based on the found angle, the offset, the rotation speed, and the corresponding prediction time, by the photographing operation signal. In this case, each of the multiple cameras may be triggered by the photographing operation signal at different times.

Meanwhile, the camera signal generator 240 may perform synchronization based on different prediction times for each of the multiple cameras in consideration of an angle error of the rotatable lidar, an arrangement position and a coverage range of the multiple cameras. have. In this case, when it is determined by the rotatable lidar that the object is disposed in a region overlapping between the first camera and the second camera among the multiple cameras, the first camera and the second camera may be simultaneously triggered.

On the other hand, if it is determined that the target angle does not converge within the threshold range based on the obtained angle information, the sensor information receiver 210 receives the distance information and the angle information from the rotatable lidar again. Accordingly, when it is determined that the target angle is converged within the threshold range based on the obtained angle information, the prediction time waiting unit 230 and the sensor information receiving unit 240 calculate the prediction time step (S120) and The photographing operation signal generation step S140 according to the process branching step S130 may be performed.

In the above, a sensor synchronization method between a rotary lidar and multiple cameras according to the present invention and a synchronization device for performing the same have been described.

The present invention provides an accurate method of synchronizing a rotational lidar and a camera sensor which are essentially used in an autonomous system.

Accordingly, by acquiring the most accurate data from the sensor information acquisition stage, errors between the sensor data can be reduced, and the results of all kinds of algorithms using the visual sensor such as calibration, recognition, obstacle determination, and autonomous behavior determination can be improved. It has an effect.

According to the software implementation, the design and parameter optimization for each component as well as the procedures and functions described herein may be implemented as separate software modules. Software code may be implemented in software applications written in a suitable programming language. The software code may be stored in a memory and executed by a controller or a processor.

Claims (6)

In a sensor synchronization method between a rotary lidar and multiple cameras, the method is performed by a synchronization device.
An angle comparison step of searching for an angle closest to the target angle by comparing the angle information obtained from the rotatable lidar with a previously input target angle;
A prediction time calculating step of calculating a camera photographing prediction time in a next frame in consideration of an offset between the found angle and an actual angle according to the rotation of the rotary lidar and the rotational speed of the rotary lidar; And
And a process branching step of inputting the prediction time into a prediction time queue, branching a current process and waiting for one of the branched processes to receive information from the rotatable lidar. According to claim 1,
In the process branching step,
And the other one of the branched processes compares the current time with the predicted time input to the predicted time queue in real time. The method of claim 2,
If the corresponding prediction time of the prediction time reaches the current time, after removing the corresponding prediction time in the queue, further comprising the step of generating a shooting operation signal for generating a shooting operation signal to the camera sensor, the sensor synchronization method . The method of claim 3, wherein
In the photographing operation signal generation step,
Activate the camera corresponding to the current angle determined based on the found angle, the offset, the rotation speed, and the corresponding prediction time by the photographing operation signal,
And each of the multiple cameras is triggered by the photographing operation signal at different times. The method of claim 4, wherein
In the photographing operation signal generation step,
Synchronizing based on different prediction times for each of the multiple cameras, taking into account the angular error of the rotatable lidar and the placement position and coverage range of the multiple cameras,
And when an object is disposed in a region overlapping between a first camera and a second camera among the multiple cameras, simultaneously triggering the first camera and the second camera. The method of claim 1,
The synchronization device
A sensor information receiver, an angle information calculator, a prediction time waiter, and a camera signal generator;
The sensor information receiver receives both the distance information and the angle information from the rotatable lidar,
If it is determined that the target angle does not converge within the threshold range based on the obtained angle information, the sensor information receiver receives the distance information and the angle information from the rotatable lidar again.
When it is determined that the target angle converges within the threshold range based on the obtained angle information, the prediction time waiting unit and the sensor information receiving unit generate the photographing operation signal according to the prediction time calculating step and the process branching step. The sensor synchronization method, characterized in that for performing.

Images