KR20220130512A - Method and apparatus for rendering lidar data usning cyclegan - Google Patents

Abstract

The present specification relates to a method and device for rendering LiDAR data. According to one embodiment of the present specification, the method for rendering LiDAR data may include the steps of: acquiring LiDAR data including 3D location information from the LiDAR; converting the LiDAR data into a 2D spherical image based on an angle of a vertical channel which matches a preset effective angle among channels included in the acquired LiDAR data; and rendering a learning image, which is an image in which 3D point cloud data (PCD) appears, using a learning algorithm from the converted 2D spherical image.

Description

Rendering method and apparatus of lidar data using CycleGAN {METHOD AND APPARATUS FOR RENDERING LIDAR DATA USNING CYCLEGAN}

The present specification relates to a method and apparatus for rendering lidar data. More particularly, it relates to a method and apparatus for rendering lidar data for rendering lidar data using CycleGAN.

Recently, with the development of artificial intelligence, autonomous vehicles are attracting attention. In autonomous vehicles, LiDAR (Light Detection And Ranging, LiDAR) is the most important sensor for measuring range, road marking and surface data, and it is a sensor that uses laser signals to recognize objects around it.

However, such lidar is very difficult to reproduce in a simulation environment because it is affected by surface reflection characteristics and range, angle of incidence, and atmospheric transmittance. In addition, when the lidar data generated by the lidar is transformed into a two-dimensional spherical image, the angle of the vertical channel is not uniform, so it is difficult to render a precise learning image from the two-dimensional spherical image. In addition, there is a problem in that it is very difficult to restore the rendered training image back to a two-dimensional spherical image due to the angular imbalance of the vertical channel.

An object of the present specification is to provide a method and apparatus for rendering lidar data that can render a two-dimensional spherical image as a learning image similar to the actual three-dimensional lidar data by preprocessing the angle of the vertical channel included in the lidar data to be uniform will do

In addition, an object of the present specification is to provide a method and apparatus for rendering lidar data that can preserve original data and improve learning performance by additionally considering a self-normalization loss to the overall loss.

The objects of the present specification are not limited to the above-mentioned objects, and other objects and advantages of the present specification that are not mentioned may be understood by the following description, and will be more clearly understood by the examples of the present specification. Moreover, it will be readily apparent that the objects and advantages of the present specification may be realized by the means and combinations thereof indicated in the claims.

A method of rendering lidar data according to an embodiment of the present specification includes acquiring lidar data including 3D position information from a lidar (LiDAR), and a preset effective angle among channels included in the acquired lidar data. Transforming the lidar data into a two-dimensional spherical image based on the angle of the vertical channel matching with rendering the .

In an embodiment of the present specification, the step of converting the lidar data into a two-dimensional spherical image includes comparing the angle of the vertical channel with a plurality of effective angles stored in advance in a hash table, and the angle of the vertical channel is selected from among the plurality of effective angles. Converting the lidar data into a two-dimensional spherical image by using pixel coordinates (v) corresponding to one effective angle obtained when matching one and pixel coordinates (u) calculated from channels other than the vertical channel includes

Comparing the angle of the vertical channel with the preset effective angle stored in the hash table in an embodiment of the present specification comprises comparing the angle of the vertical channel with the preset effective angle using a hash function based on the hash table includes

In an embodiment of the present specification, the rendering of the training image includes an adversarial loss, a cyclic consistency loss, a self-regularization loss, and a total loss determined based on a weight parameter. and rendering a training image from the two-dimensional spherical image using a learning algorithm to minimize (total loss).

In an embodiment of the present specification, the self-normalization loss is calculated by Equation 5 below.

<Equation 5>

In one embodiment of the present specification, the learning algorithm is Cycle-Generative Adversarial Network (Cycle-Generative Adversarial Network).

The apparatus for rendering lidar data according to an embodiment of the present specification includes a data receiver that acquires lidar data including 3D position information from a lidar (LiDAR), and a preset validity among channels included in the acquired lidar data. An image in which three-dimensional PCD (Point Cloud Data) appears by using a preprocessing unit that converts the lidar data into a two-dimensional spherical image based on the angle of the vertical channel matching the angle and a learning algorithm from the converted two-dimensional spherical image and a rendering unit for rendering the training image.

In one embodiment of the present specification, the preprocessor compares the angle of the vertical channel with a preset effective angle stored in a hash table, and when the angle of the vertical channel matches the preset effective angle, pixel coordinates corresponding to the preset effective angle to convert the lidar data into a two-dimensional spherical image.

In an embodiment of the present specification, the preprocessor compares the angle of the vertical channel with a preset effective angle using a hash function included in the hash table.

In one embodiment of the present specification, the rendering unit is an adversarial loss, a cyclic consistency loss, a self-regularization loss, and a total loss determined based on a weight parameter w (total loss) A training image is rendered using a learning algorithm from the two-dimensional spherical image so that this is minimized.

In an embodiment of the present specification, the self-normalization loss is calculated by Equation 5 below.

<Equation 5>

In one embodiment of the present specification, the learning algorithm is Cycle-Generative Adversarial Network (Cycle-Generative Adversarial Network).

The method and apparatus for rendering lidar data according to an embodiment of the present specification preprocesses the angle of the vertical channel included in the lidar data to be uniform, and renders a two-dimensional spherical image as a learning image similar to the actual three-dimensional lidar data. can

In addition, the method and apparatus for rendering lidar data according to an embodiment of the present specification can preserve original data and improve learning performance by additionally considering a self-normalization loss to the total loss.

1 is a block diagram of an apparatus for rendering lidar data according to an embodiment of the present specification.
2 is a diagram illustrating a hash table including an effective angle and a pixel coordinate v.
3 is a diagram showing (a) a two-dimensional spherical image, (b) a learning image, and (c) actual three-dimensional lidar data.
4 is a flowchart of a method of rendering lidar data according to an embodiment of the present specification.

The above-described objects, features, and advantages will be described below in detail with reference to the accompanying drawings, and accordingly, those of ordinary skill in the art to which this specification belongs will be able to easily implement the technical idea of the present specification. In the description of the present specification, if it is determined that a detailed description of a known technology related to the present specification may unnecessarily obscure the gist of the present specification, the detailed description will be omitted. Hereinafter, preferred embodiments according to the present specification will be described in detail with reference to the accompanying drawings. In the drawings, the same reference numerals are used to indicate the same or similar components.

The above-described objects, features, and advantages will be described below in detail with reference to the accompanying drawings, and accordingly, those of ordinary skill in the art to which this specification belongs will be able to easily implement the technical idea of the present specification. In the description of the present specification, if it is determined that a detailed description of a known technology related to the present specification may unnecessarily obscure the gist of the present specification, the detailed description will be omitted. Hereinafter, preferred embodiments according to the present specification will be described in detail with reference to the accompanying drawings. In the drawings, the same reference numerals are used to refer to the same or similar components.

1 is a block diagram of an apparatus for rendering lidar data according to an embodiment of the present specification.

Referring to the drawings, the apparatus 10 for rendering lidar data includes a data receiving unit 100 , a preprocessing unit 200 , and a rendering unit 300 .

The data receiver 100 obtains lidar data from lidar. When the pulse laser signal generated from LiDAR comes back after colliding with a nearby object, the returned pulse laser can be analyzed to check the location, movement direction, and speed of the surrounding object. may include That is, the lidar data may be PCD (Point Cloud Data), which is a set of several points spread on a three-dimensional space.

The preprocessor 200 performs preprocessing of converting the lidar data into a two-dimensional spherical image based on an angle of a vertical channel that matches a preset effective angle among channels included in the acquired lidar data.

That is, since LiDAR data is a three-dimensional PCD, in order to convert it into a two-dimensional spherical image, the three-dimensional PCD must be changed to two-dimensional image coordinates. In addition, the lidar data can render a 3D PCD reconstructed learning image from the converted 2D spherical image, and the preprocessor 200 performs such a preprocessing process to restore similar to the actual 3D PCD. can

The number of channels is determined by how much the lidar can be divided into scans during one revolution. For example, in the case of a 16-channel lidar, it is divided into 16 channels during one rotation and surrounding objects are scanned. Therefore, the number of channels included in the acquired lidar data may be plural, and as the number of included channels increases, the ponint cloud of lidar data can be expressed in detail and precisely.

In addition, the number of channels may be different depending on the type of lidar, and vertical channel angles between channels may be different from each other. Accordingly, the preprocessor 200 may convert the lidar data into a two-dimensional spherical image by comparing the angle of the vertical channel with a preset effective angle using a hash function based on a hash table. A method of converting lidar data into two-dimensional spherical image coordinates based on the angle of the vertical channel will be described in detail with reference to FIG. 2 .

The rendering unit 300 renders a learning image, which is an image in which three-dimensional PCD (Point Cloud Data) appears, by using a learning algorithm from the converted two-dimensional spherical image. In this case, the learning algorithm used may be a Cycle-Generative Adversarial Network (Cycle-Generative Adversarial Network).

Accordingly, the rendering unit 300 is configured to minimize the adversarial loss, the cycle consistency loss, the self-regularization loss, and the total loss determined based on the weight parameter. A learning image may be rendered using a learning algorithm from the two-dimensional spherical image.

2 is a diagram illustrating a hash table including an effective angle and a pixel coordinate v.

First, 3D LiDAR data can be converted into (u,v) image coordinates in a 2D spherical image. Specifically, any one point P of LIDAR data, which is a three-dimensional PCD, may be expressed as P(x,y,z) in the Cartesian coordinate system, which is converted into two-dimensional image coordinates (u,v) through Equation 1 below. can be converted

<Equation 1>

In this case, the pixel coordinate v is calculated through a hash function based on a hash table. That is, h used in Equation 1 is a hash function, and when the angle of the vertical channel included in the lidar data matches one of a plurality of valid angles, the pixel coordinate value is returned from the hash table.

Specifically, referring to FIG. 2 , a plurality of effective angles and pixel coordinates v corresponding to the plurality of effective angles are previously stored in the hash table 210 . The preprocessor 200 compares the angle of the vertical channel among the channels included in the obtained lidar data with a plurality of effective angles stored in advance in the hash table 210 .

When the angle of the vertical channel coincides with one of the plurality of effective angles, the preprocessor 200 obtains pixel coordinates v corresponding to the matching effective angle. For example, if the angle of the vertical channel in the drawing is 34, it coincides with the effective angle 212 having a value of 34, so the preprocessor 200 obtains 14, which is a corresponding pixel coordinate v (214) value. The plurality of effective angles may be different depending on the type of lidar.

Also, there may be a plurality of vertical channel angles that coincide with the effective angle. As shown in FIG. 2 , when each of the plurality of vertical channel angles coincides with the effective angles 34 (212) and 88 (216), the preprocessor 200 converts the pixel coordinates v to 14 (214) and 24 (218). can be obtained individually. That is, since the preprocessor 200 acquires pixel coordinates v only for the vertical channel angle that matches the effective angle, the learning effect can be improved, so that the two-dimensional spherical image can be rendered as a learning image similar to the actual three-dimensional lidar data. .

Meanwhile, the pixel coordinate u may be calculated from the remaining channels except for the vertical channel among the channels included in the obtained lidar data. Specifically, the pixel coordinate u may be calculated from Equation 1.

Thereafter, the preprocessor 200 completes the preprocessing by converting the lidar data into a two-dimensional spherical image using the pixel coordinate u and the pixel coordinate v.

3 is a diagram showing (a) a two-dimensional spherical image, (b) a learning image, and (c) actual three-dimensional lidar data.

Referring to the drawings, the two-dimensional spherical image of (a) is obtained by converting the three-dimensional PCD of the lidar data into pixel coordinates u,v as described above. Thereafter, the rendering unit 300 renders a learning image that is an image in which the 3D PCD of (b) appears from the 2D spherical image through the learning algorithm. Learning performance is determined according to how similar the rendered training image is to the actual 3D lidar data shown in (c).

Meanwhile, the learning algorithm used by the rendering unit 300 for rendering may be CycleGAN (Cycle-Generative Adversarial Network). The learning algorithm used is not necessarily limited to CycleGAN, but for convenience of explanation, the following description is made on the premise that the learning algorithm is CycleGAN.

CycleGAN is a learning algorithm that applies a learning result to an unpaired input image, and is given a constraint that it can be restored back to the input image when a training image is generated from the input image.

That is, when the training image (b) is generated from the two-dimensional spherical image (a), it must be possible to restore the two-dimensional spherical image (a) again. At this time, a total loss occurs in the process of generating a training image and restoring it again. Learning performance can be improved only when this total loss is minimized.

On the other hand, the total loss is determined based on the adversarial loss, the cycle consistency loss, the self-regularization loss, and the weight parameter w, specifically, Equation 2 below can be calculated by

<Equation 2>

where L is the total loss, w is the weighting parameter, L _GAN is the adversarial loss, L _cycle is the coherence loss, and L _self is the self-normalization loss.

The weighting parameter is a weighted parameter for each loss, and the sum of all weighting parameters is 1. For example, w1, w2, and w3 may have values of 0.6, 0.3, and 0.1, respectively, according to the assigned weight.

_LGAN is a hostile loss, which is a loss caused by competition. The hostile loss can be calculated by Equation 3 below.

<Equation 3>

은 합성 도메인(Synthetic domain, S)의 2차 구형 이미지인 (a)로부터 실제 도메인(Real domain, R)의 학습 이미지(b)를 생성하는 생성자이고,

은 반대로 실제 도메인(Real domain)의 학습 이미지(b)로부터 합성 도메인(Synthetic domain)의 2차 구형 이미지를 생성하는 생성자이며, r 및 s 각각은 실제 도메인(R) 및 합성 도메인(S)에 포함된 원소를 의미한다.here,

is a constructor that generates a training image (b) of the real domain (Real domain, R) from (a), which is a secondary spherical image of the synthetic domain (S),

Conversely, is a generator that generates a secondary spherical image of the synthetic domain from the training image (b) of the real domain, and r and s are included in the real domain (R) and the synthetic domain (S), respectively. means an element.

및

은 각각 생성자가 생성한 학습 이미지를 진짜 이미지와 비교하여 진짜 이미지인지 또는 학습 이미지인지를 판별하는 판별자이다. In addition,

and

is a discriminator that compares the training image generated by the generator with the real image, respectively, and determines whether it is a real image or a training image.

The generator and the discriminator are competing, and the discriminator compares the image generated by the generator in the synthetic domain (S) and the real domain (R) with the real image (c) to determine whether the image is a generated image or an actual image.

The discriminator returns 0 or 1 depending on the generation result, and the hostile loss is determined by these return values. That is, the more similar the generator generates the synthetic image to the real image, the better the discriminator can judge whether it is a real image or not, the closer the adversarial loss is to zero.

L _cycle is a loss of cyclic coherence, and since it is difficult to ensure proper learning with adversarial loss alone, coherence loss is used. That is, when the synthetic domain and the real domain go through the process of generating and discriminating training images, the result value must be cycle-consistent, and loss occurs if it is not cycle-consistent. Specifically, the consistency loss may be calculated by Equation 4 below.

<Equation 4>

이고,

이면 생성자가 생성하기 이전의 이미지로 복원이 제대로 된 것으로 cyle-consistent하다. 따라서 이 경우 일관성 손실은 0이다. 그러나, 이미지 복원이 제대로 수행되지 않을 경우 그만큼 일관성 손실이 발생한다.here

ego,

If this is the case, it is cyle-consistent as it is properly restored to the image before the creator created it. Therefore, the loss of consistency in this case is zero. However, if image restoration is not performed properly, consistency loss occurs.

L _self is a self-normalization loss that occurs when the generator makes excessive changes to the input image. That is, if an excessive change is applied to the input image, it becomes difficult to restore the original input image from the training image. In addition, the training image to which this excessive change is applied has a large difference from the input image, resulting in meaningless data. Therefore, it is important to account for these self-normalization losses. The self-normalization loss can be calculated by Equation 5 below.

<Equation 5>

에 따라 실제 도메인(S)으로부터 생성된 학습 이미지(

)가 인풋 이미지인 실제 도메인의 이미지(r)와 유사 한지,

에 따라 합성 도메인(S)으로부터 생성된 학습 이미지(

)가 인풋 이미지인 합성 도메인의 이미지(r)와 유사 한지에 따라 자체 정규화 손실이 결정된다.in other words,

A training image generated from the real domain (S) according to

) is similar to the image (r) of the real domain, which is the input image,

A training image generated from the synthetic domain (S) according to

) is similar to the image (r) of the synthetic domain, which is the input image, the self-normalization loss is determined.

In the end, the smaller the difference between the input image and the training image, the more the loss is minimized. As such, the apparatus for rendering lidar data according to an embodiment of the present specification can preserve original data and improve learning performance by additionally considering a self-normalization loss to the total loss.

4 is a flowchart of a method of rendering lidar data according to an embodiment of the present specification.

Referring to the drawing, the apparatus 100 for rendering lidar data obtains lidar data including 3D position information from lidar (LiDAR) (S100). Thereafter, the lidar data rendering apparatus 100 converts the lidar data into a two-dimensional spherical image based on an angle of a vertical channel that matches a preset effective angle among channels included in the acquired lidar data (S200). ).

Specifically, the rendering apparatus 100 of the lidar data compares the angle of the vertical channel with a plurality of effective angles stored in advance in the hash table, and when the angle of the vertical channel matches one of the plurality of effective angles, one effective angle is obtained. The LiDAR data is converted into a two-dimensional spherical image by using pixel coordinates (v) corresponding to , and pixel coordinates (u) calculated from channels other than the vertical channel.

Thereafter, the rendering apparatus 100 of the lidar data uses a learning algorithm from the converted two-dimensional spherical image to render a learning image, which is an image in which three-dimensional PCD (Point Cloud Data) appears (S300).

Specifically, the rendering apparatus 100 of the lidar data includes an adversarial loss, a cycle consistency loss, a self-regularization loss, and a total loss determined based on a weight parameter. loss) is minimized, and a training image is rendered using a learning algorithm from the two-dimensional spherical image.

As such, the rendering apparatus 100 of the lidar data can preserve the original data and improve the learning performance by additionally considering the self-normalization loss to the total loss.

As described above, the present invention has been described with reference to the illustrated drawings, but the present invention is not limited by the embodiments and drawings disclosed in the present specification. It is obvious that variations can be made. In addition, although the effect of the configuration of the present invention has not been explicitly described and described while describing the embodiment of the present invention, it is natural that the effect predictable by the configuration should also be recognized.

Claims (12)

obtaining LiDAR data including 3D location information from LiDAR;
converting the lidar data into a two-dimensional spherical image based on an angle of a vertical channel that matches a preset effective angle among channels included in the acquired lidar data; and
Using a learning algorithm from the converted two-dimensional spherical image, comprising the step of rendering a learning image that is an image in which three-dimensional PCD (Point Cloud Data) appears
How to render lidar data.
The method of claim 1,
The step of converting the lidar data into a two-dimensional spherical image includes:
comparing the angle of the vertical channel with a plurality of valid angles previously stored in a hash table; and
When the angle of the vertical channel coincides with one of the plurality of effective angles, the pixel coordinates (v) corresponding to the one effective angle and pixel coordinates (u) calculated from the remaining channels except for the vertical channel are used. converting the lidar data into a two-dimensional spherical image
How to render lidar data.
3. The method of claim 2,
Comparing the angle of the vertical channel with a preset effective angle stored in a hash table comprises:
Comprising the step of comparing the angle of the vertical channel with the preset effective angle using a hash function based on the hash table
How to render lidar data.
According to claim 1,
The step of rendering the learning image is
Learning from the 2D spherical image so that adversarial loss, cyclic consistency loss, self-regularization loss, and total loss determined based on weight parameters are minimized Rendering the training image using the algorithm
How to render lidar data.
5. The method of claim 4,
The self-normalization loss is
A rendering method of lidar data calculated by Equation 5 below.
<Equation 5>

According to claim 1,
The learning algorithm is
CycleGAN (Cycle-Generative Adversarial Network)
How to render lidar data.
a data receiving unit for obtaining lidar data including 3D position information from a lidar (LiDAR);
a pre-processing unit converting the lidar data into a two-dimensional spherical image based on an angle of a vertical channel that matches a preset effective angle among channels included in the obtained lidar data; and
Using a learning algorithm from the converted two-dimensional spherical image, comprising a rendering unit for rendering a learning image that is an image in which three-dimensional PCD (Point Cloud Data) appears
A rendering device for lidar data.
8. The method of claim 7,
The preprocessor
The angle of the vertical channel is compared with a preset effective angle stored in a hash table, and when the angle of the vertical channel matches the preset effective angle, pixel coordinates corresponding to the preset effective angle are output to obtain the lidar data. Converting to a two-dimensional spherical image
A rendering device for lidar data.
9. The method of claim 8,
The preprocessor
Comparing the angle of the vertical channel with the preset effective angle using a hash function included in the hash table
A rendering device for lidar data.
8. The method of claim 7,
The rendering unit
Adversarial loss, cyclic consistency loss, self-regularization loss, and total loss determined based on the weight parameter w are minimized from the 2D spherical image. A rendering device for lidar data that renders a training image using a learning algorithm.
11. The method of claim 10,
The self-normalization loss is
A rendering device for lidar data calculated by the following Equation 5.
<Equation 5>

8. The method of claim 7,
The learning algorithm is
CycleGAN (Cycle-Generative Adversarial Network)
A rendering device for lidar data.

Images