KR20220009247A - Method for constructing input data for deep neural network (DNN) based on Light Detection and Ranging (LiDAR) and Radio Azimuth Direction and Ranging (RADAR) signal - Google Patents

Abstract

An objective of the present invention is to eliminate problems such as the divergence of a filter algorithm or the position error generation caused by the accumulation of sensor error information. According to an embodiment of the present invention, a method for constructing input data of a deep neural network based on LiDAR and radar signals comprises: a step in which, when scan data including coordinates representing the positions of a plurality of points making up an object and reflection intensities representing intensities at which a laser or a radio wave is reflected from the plurality of points are inputted through a three-dimensional orthogonal coordinate system from at least one among LiDAR and radar sensors, a plane classification unit classifies the reflection intensities included in the scan data in accordance with the coordinates to generate a plurality of pieces of two-dimensional plane information; and a step in which a three-dimensional formation unit generates three-dimensional information having a three-dimensional structure by stacking the plurality of pieces of plane information in a plurality of layers.

Description

Method for constructing input data for deep neural network (DNN) based on Light Detection and Ranging (LiDAR) and Radio Azimuth Direction and Ranging (RADAR) ) signal}

The present invention relates to a technique for constructing input data of a deep neural network (DNN), and more particularly, of LiDAR (Light Detection and Ranging: LiDAR) and Radar (Radio Azimuth Direction and Ranging: RADAR) It relates to a method of composing output information as input data of a deep neural network model.

The radar sensor detects an object using the intensity of the electromagnetic wave that emits electromagnetic waves and is reflected back to the object, and the lidar sensor emits light (laser) and uses the intensity of the light that is reflected back to the object to detect the surrounding area. image it.

Korean Patent No. 19987846 registered on June 04, 2019 (Title: Ship collision avoidance device and method through image analysis of radar device monitor)

An object of the present invention is to use a plurality of LiDAR (Light Detection and Ranging) sensors and RADAR (Radio Azimuth Direction and Ranging) sensors in indoor and outdoor environments to learn a deep neural network that estimates the posture information of an object in the environment. To provide a method for deriving input data.

In a method for constructing input data of a deep neural network model based on a lidar and a radar signal according to a preferred embodiment of the present invention for achieving the above object, the plane classification unit receives a laser beam from at least one of the lidar and the radar sensor. Alternatively, when a plurality of scan information indicating the coordinates on the three-dimensional Cartesian coordinate system of each of the plurality of points on the object surface that reflects the radio waves and the reflection intensity, which is the intensity of the laser or radio waves reflected from each of the plurality of points on the object surface, are input, the Generating a plurality of two-dimensional plane information by classifying a plurality of pieces of scan information according to respective coordinates, and a three-dimensional forming unit stacking the plurality of plane information in a plurality of layers to generate a plurality of three-dimensional information having a three-dimensional structure includes steps.

The generating of the plurality of plane information includes: classifying, by the plane classification unit, the plurality of scan information into a plurality of unit sections based on the Z-axis among the coordinates of the plurality of scan information, and each of the plurality of unit sections and generating a plurality of plane information by mapping the reflected intensity of the scanned information to a plane space corresponding to each of the plurality of classified unit sections.

에 따라 상기 스캔 데이터에 포함된 복수의 스캔 정보를 스캔 정보의 좌표의 Z축을 기준으로 소정의 분할 수의 단위 구간으로 구분하는 단계를 포함한다. The step of classifying into the plurality of unit sections includes the steps of: the plane classifying unit setting a maximum height (Zmax) on the Z-axis of the plurality of scan information; Replacing or erasing the value of the Z-axis of the information with the effective maximum height (Zmax-1);

and dividing the plurality of scan information included in the scan data into unit sections of a predetermined number of divisions based on the Z-axis of the coordinates of the scan information.

Here, M is the number of divisions, Zstart is a starting point on the Z-axis of a unit section, Zend is an end point on the Z-axis of a unit section, and Zmax is the maximum height of the Z-axis.

In the generating of the plurality of plane information, the plane classifying unit divides the reflection intensity of each of the plurality of scan information included in the scan data by a maximum value among the reflection intensities of the plurality of scan information included in the scan data, between 0 and 1 converting to a real value of , forming a plurality of planar spaces corresponding to a plurality of unit sections by the planar classification unit, and orthogonal to each of a plurality of scan information included in each of the plurality of unit sections by the planar classifying unit converting coordinates into spherical coordinates to derive distances and angles according to spherical coordinates; and mapping the reflection intensity of all scan information that meets the line segment on coordinates among the scan information of the unit section to the line segment.

,

에 따라 구면 좌표로 변환하여 구면 좌표에 따른 거리 및 각도를 도출하며, 상기 k는 스캔 정보의 인덱스이고, 상기 x, y, z는 스캔 정보의 직교좌표계의 좌표이고, 상기

은 구면좌표계에 따른 원점에서의 거리이고, 상기

는 구면좌표계에 따른 z축을 축으로 양의 방향의 x축과 이루는 각도를 나타내는 것을 특징으로 한다. In the step of deriving the distance and the angle according to the spherical coordinates, the plane classification unit calculates the orthogonal coordinates of each of the plurality of scan information included in each of the plurality of unit sections.

,

Converts to spherical coordinates according to , and derives distances and angles according to spherical coordinates, wherein k is an index of scan information, and x, y, and z are coordinates of a Cartesian coordinate system of scan information, and

is the distance from the origin according to the spherical coordinate system,

It is characterized in that it represents an angle formed with the x-axis in the positive direction with respect to the z-axis according to the spherical coordinate system.

The method includes the steps of: a grid forming unit arranging a plurality of three-dimensional information in a grid to form grid information; The method further includes the step of matching the heights of a plurality of stereoscopic information included in the grid information by erasing the plane information of , or adding at least one piece of padding information to the at least one piece of stereoscopic information.

The forming of the grid information is characterized in that the grid forming unit forms the grid information by arranging a plurality of three-dimensional information in a preset position according to a time sequence in which the three-dimensional information is generated and arranging the grid information.

The method includes the steps of: a learning unit collecting the posture information of an object as label data in response to the input data that is the three-dimensional information or the grid information; further comprising the step of

According to the present invention, three-dimensional input data, that is, three-dimensional information or grid information, can be generated from a plurality of lidar sensors or scan information of a plurality of radar sensors, and a deep neural network can be trained using this. The deep neural network learned through this can estimate the posture of an object without complicated filter calculations, and there is no problem such as the occurrence of position errors due to the accumulation of sensor error information or the divergence of filter algorithms.

1 is a conceptual diagram for explaining a method of deriving information of a component device according to an embodiment of the present invention.
2 is a diagram for explaining the configuration of an apparatus for configuring input data for a deep neural network model based on a lidar and a radar signal according to an embodiment of the present invention.
3 is a diagram for explaining a detailed configuration of a control unit of an apparatus for configuring input data for a deep neural network model based on a lidar and a radar signal.
4 is a diagram for explaining an example of a deep neural network model for estimating posture information based on input data based on lidar and radar signals according to an embodiment of the present invention.
5 is a flowchart illustrating a method for constructing input data for a deep neural network model based on lidar and radar signals.
6 is a flowchart illustrating a lidar and a method of generating stereoscopic information from a radar signal according to an embodiment of the present invention.
7 is a diagram for explaining a method of generating a grid image based on a lidar and a radar signal according to an embodiment of the present invention.
8 is a diagram for explaining a method of generating a lattice image based on a lidar and a radar signal according to another embodiment of the present invention.
9 is a flowchart illustrating a method of generating stereoscopic information (SI) according to an embodiment of the present invention.
10 is a flowchart illustrating a method of generating learning data according to an embodiment of the present invention.

Prior to the detailed description of the present invention, the terms or words used in the present specification and claims described below should not be construed as being limited to their ordinary or dictionary meanings, and the inventors should develop their own inventions in the best way. It should be interpreted as meaning and concept consistent with the technical idea of the present invention based on the principle that it can be appropriately defined as a concept of a term for explanation. Therefore, the embodiments described in the present specification and the configurations shown in the drawings are only the most preferred embodiments of the present invention, and do not represent all the technical spirit of the present invention, so various equivalents that can be substituted for them at the time of the present application It should be understood that there may be water and variations.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In this case, it should be noted that in the accompanying drawings, the same components are denoted by the same reference numerals as much as possible. In addition, detailed descriptions of well-known functions and configurations that may obscure the gist of the present invention will be omitted. For the same reason, some components are exaggerated, omitted, or schematically illustrated in the accompanying drawings, and the size of each component does not fully reflect the actual size.

First, an apparatus for configuring input data for a deep neural network model based on lidar and radar signals according to an embodiment of the present invention will be described. 1 is a conceptual diagram for explaining a method of deriving information of a component device according to an embodiment of the present invention. 2 is a diagram for explaining the configuration of an apparatus for configuring input data for a deep neural network model based on a lidar and a radar signal according to an embodiment of the present invention. 3 is a diagram for explaining a detailed configuration of a control unit of an apparatus for configuring input data for a deep neural network model based on a lidar and a radar signal. 4 is a diagram for explaining an example of a deep neural network model for estimating posture information based on input data based on lidar and radar signals according to an embodiment of the present invention.

First, referring to FIG. 1 , an apparatus (10: hereinafter, abbreviated as 'configuration device') for configuring input data for a deep neural network model based on lidar and radar signals according to an embodiment of the present invention is this A deep neural network according to an embodiment of the invention can be trained.

As shown, the component device 10 may be installed in the robot (R) or the computing component device (C) to learn a deep neural network. If it is installed in the computing device (C) and trained a deep neural network, the deep neural network can be ported to the robot (R). Such a deep neural network can be trained to estimate the posture information of the objects obj1, obj2, and obj3. To this end, the present invention uses scan data of an object measured by at least one of a lidar sensor and a radar sensor.

Referring to FIG. 2 , the configuration device 10 includes a scan unit 11 , an input unit 12 , a display unit 13 , a storage unit 14 , a communication unit 15 , a sensor unit 16 , and a location information unit 17 . and a control unit 18 .

The scan unit 11 is for scanning the objects obj1 , obj2 , and obj3 . To this end, the scan unit 11 includes a lidar (LiDAR) sensor 110 and a radar (RADAR) sensor 120 . The lidar sensor 110 and the radar sensor 120 scan the objects obj1 , obj2 , and obj3 to output scan data including a plurality of scan information.

The lidar sensor 110 and the radar sensor 120 radiate a laser or radio wave in an omni-Directional direction from the center of the sensors 110 and 120 in a vertical or horizontal direction in a three-dimensional space or a horizontal in a two-dimensional space. A plurality of scan data including a plurality of scan information including coordinates of an object for each angle of direction and a reflection intensity indicating a light reflected intensity is output. The scan information is the coordinates of the object on the three-dimensional Cartesian coordinate system consisting of the X and Y axes of a plane parallel to the ground and the Z axis in the height direction, and the reflected intensity of the laser or radio waves. to be. That is, the scan information output by the lidar sensor 110 and the radar sensor 120 by scanning the objects obj1, obj2, and obj3 is the position of any one of a plurality of points constituting the object surface through a three-dimensional Cartesian coordinate system. and a reflection intensity indicating the intensity at which the laser or radio wave is reflected from the point. In this way, the scan data output by the lidar sensor 110 and the radar sensor 120 is expressed by Equation 1 below.

In Equation 1, Sensor_output represents scan data. In addition, through the scan data, N represents the number of output scan information. That is, the number of scan information indicates the number of the plurality of points when there are a plurality of points on the surface of the object where the laser or radio waves are reflected. The number N of scan information may vary for each scan moment of the lidar sensor 110 and the radar sensor 120 in an indoor or outdoor environment. In addition, (x, y, z) is a coordinate according to the Cartesian coordinate system of each of a plurality of points on the surface of an object that reflects a laser or radio wave, and v_intensity represents the intensity of a laser or radio wave reflected from each of a plurality of points on the surface of the object. is the reflection intensity.

The input unit 12 may receive a user's operation for controlling the component device 10 , generate an input signal, and transmit it to the control unit 18 . The input unit 12 includes various types of buttons, keys, and the like for controlling the component device 10 . In the input unit 12, when the display unit 13 is formed of a touch screen, the functions of various keys can be performed on the display unit 13, and when all functions can be performed only with the touch screen, the input unit 12 may be omitted. may be

The display unit 13 is for screen display, and may visually provide a menu of the component device 10, input data, function setting information, and various other information to the user. The display unit 13 may be formed of a liquid crystal display (LCD), an organic light emitting diode (OLED), an active matrix organic light emitting diode (AMOLED), or the like. Meanwhile, the display unit 13 may be implemented as a touch screen. In this case, the display unit 13 includes a touch sensor. The touch sensor detects a user's touch input. The touch sensor may be composed of a touch sensing sensor such as a capacitive overlay, a pressure type, a resistive overlay, or an infrared beam, or may be composed of a pressure sensor. . In addition to the above sensors, all types of sensor devices capable of sensing contact or pressure of an object may be used as the touch sensor of the present invention. The touch sensor may detect a user's touch input, generate a detection signal including input coordinates indicating the touched position, and transmit it to the controller 18 . In particular, when the display unit 13 is formed of a touch screen, some or all of the functions of the input unit 12 may be performed through the display unit 13 .

The storage unit 14 serves to store programs and data necessary for the operation of the component device 10 . The storage unit 14 may store scan data obtained by scanning an object by the scanning unit 11 , inertia information detected by the sensor unit 12 , a GPS signal received by the location information unit 17 , and the like for a predetermined period of time. Various types of data stored in the storage unit 14 may be deleted, changed, or added according to a user's operation.

The communication unit 15 is for communicating with a communicable object obj1, such as a robot. The communication unit 15 may receive, from the object obj1 , a GPS signal received by the object obj1 , inertia information measured by the object obj1 , posture information measured by the object, and the like. The communication unit 15 includes an RF transmitter for up-converting and amplifying a frequency of a transmitted signal, and an RF receiver for low-noise amplifying and down-converting a received signal. In addition, the communication unit 15 includes a modem that modulates a transmitted signal and demodulates a received signal.

The sensor unit 16 is for measuring inertia. The sensor unit 16 includes a speed sensor, an acceleration sensor, an angular velocity sensor, a gyro sensor, an inertial measurement unit (IMU), a Doppler Velocity Log (DVL), and an attitude and heading reference. : AHRS). Such a sensor may be implemented as micro electro-mechanical systems (MEMS).

The location information unit 17 is for receiving a Global Positioning System (GPS) signal. For example, the location information unit 17 continuously receives a GPS signal from a GPS satellite or the like, and derives location information from the received GPS signal. The derived location information is transmitted to the control unit 18 . Such location information may be coordinates such as latitude, longitude, and altitude.

The control unit 18 may control the overall operation of the component device 10 and the signal flow between the internal blocks 11 to 18 of the component device 10 , and perform a data processing function of processing data. Also, the control unit 18 basically serves to control various functions of the component device 10 . The control unit 18 may be exemplified by a central processing unit (CPU), a digital signal processor (DSP), or the like.

Next, referring to FIG. 3 , the control unit 18 includes a configuration unit 200 , a learning unit 300 , a deep neural network (DNN) 400 , and an estimator 500 .

The configuration unit 200 is configured to generate input data input to the deep neural network 400 from a plurality of scan information (SCI: SCan Information) generated by the scan unit 11 scanning the objects obj1, obj2, obj3. will be. When scan data (SD) including a plurality of scan information (SCI) is input from the scan unit 11 , the configuration unit 200 is configured to perform the scan information (SCI) according to the coordinates of each of the plurality of scan information (SCI). ) to generate a plurality of two-dimensional flat information (FI), and a plurality of three-dimensional information (SI: Stereo Information) having a three-dimensional structure by stacking a plurality of flat information (FI) can create Then, the configuration unit 200 generates grid information (GI) by arranging a plurality of three-dimensional information SI in a grid. Such stereoscopic information (SI) or grid information (GI) may be used as input data input to the deep neural network 400 .

To this end, the constituent unit 200 includes a planar forming unit 210 , a three-dimensional forming unit 220 , and a grid forming unit 230 . When a plurality of scan information SCI is input, the plane forming unit 210 classifies the plurality of scan information SCIs based on the Z-axis among the scan information SCI to generate a plurality of plane information FI.

The three-dimensional forming unit 220 generates the three-dimensional information SI having a three-dimensional structure by stacking the plurality of planar information FI generated by the planar forming unit 210 in a plurality of layers. The grid forming unit 230 forms grid information GI by arranging a plurality of three-dimensional information SI in a grid. The operation of the component 200 including the planar forming unit 210 , the three-dimensional forming unit 220 , and the grid forming unit 230 will be described in more detail below.

The learning unit 300 generates learning data for the deep neural network 400 by performing labeling to give a label to the input data, and uses this to learn the deep neural network 400 .

The deep neural network 400 provides a probability for estimating posture information of an object included in an image. The deep neural network 400 may be a convolutional neural network, such as a Convolution Neural Network (CNN), or a recurrent neural network, such as a Recurrent Neural Network (RNN) or Long Short-Term Memory (LTSM). However, the deep neural network 400 is not limited thereto, and any type of artificial neural network in which the hidden layer is composed of a plurality of layers may be the deep neural network 400 according to an embodiment of the present invention.

According to an embodiment of the present invention, when the deep neural network 400 is a convolutional neural network, the deep neural network 400 is an input layer (IL), at least a pair of alternately repeated convolution layers (convolution layer: CL) ), a pooling layer (PL), and at least one fully-connected layer (FL) and an output layer (OL). As shown in FIG. 4 , the deep neural network 400 according to an embodiment of the present invention sequentially includes an input layer (IL), a convolution layer (CL), a pooling layer (PL), a fully connected layer (FL) and and an output layer OL.

The convolution layer CL and the pooling layer PL include at least one feature map (FM). The feature map FM is derived as a result of receiving a value to which a weight and a threshold are applied to the operation result of the previous layer, and performing an operation on the input value. These weights are applied through a filter or kernel W that is a weight matrix of a predetermined size. In the embodiment of the present invention, the first filter W1 is used for the convolution operation of the convolutional layer CL, and the second filter W2 is used for the pooling operation of the pooling layer PL.

When input data (a matrix or vector column of a predetermined size) is input to the input layer IL, the convolution layer CL performs convolution using the first filter W1 on the input data of the input layer IL. At least one first feature map FM1 is derived by performing an operation using an operation and an activation function. Next, the pooling layer PL performs a pooling or sub-sampling operation using the second filter W2 on at least one first feature map FM1 of the convolution layer CL to obtain at least one A second feature map FM2 is derived.

The final connection layer FL includes a plurality of operation nodes f1 to fx. The plurality of operation nodes f1 to fx of the final connection layer FL calculates a plurality of operation values through an operation using an activation function with respect to at least one second feature map FM2 of the pooling layer PL.

The output layer OL includes a plurality of output nodes g1 to gy. Each of the plurality of operation nodes f1 to fx of the final connection layer FL is connected to the output nodes g1 to gy of the output layer OL through a channel having a weight (W). In other words, a weight is applied to the plurality of operation values of the plurality of operation nodes f1 to fx and is input to each of the plurality of output nodes g1 to gy. Accordingly, the plurality of output nodes g1 to gy of the output layer OL calculates an output value through an activation function operation for a plurality of calculated values to which the weight of the final connection layer FL is applied.

The activation functions used in the convolutional layer (CL), final connection layer (FL) and output layer (OL) described above are Sigmoid, Hyperbolic tangent (tanh), Exponential Linear Unit (ELU), and ReLU. (Rectified Linear Unit), Leakly ReLU, Maxout, Minout, Softmax, etc. can be exemplified. Any one of these activation functions may be selected and applied to the convolutional layer CL, the finite connection layer FL, and the output layer OL.

In summary, as described above, the deep neural network 400 includes a plurality of layers. In addition, the plurality of layers of the deep neural network 400 includes a plurality of operations. The calculation result of each of the plurality of layers is transmitted to the next layer by applying parameters, ie, weights, thresholds, and the like. Accordingly, the deep neural network 400 may calculate an output value by performing a plurality of operations to which the weights of a plurality of layers are applied to the input data, and may output the calculated output value.

The estimator 500 estimates the posture information of the object included in the scan data by using the deep neural network 400 . The output value of the deep neural network 400 may be a probability. The estimator 500 estimates the posture information of the object included in the scan data based on the output probability of the deep neural network 400 .

Next, a method for constructing input data for a deep neural network model based on lidar and radar signals will be described. 5 is a flowchart illustrating a method for constructing input data for a deep neural network model based on lidar and radar signals. 6 is a flowchart illustrating a method of generating stereoscopic information from a lidar and a radar signal according to an embodiment of the present invention. 7 is a diagram for explaining a method of generating a grid image based on a lidar and a radar signal according to an embodiment of the present invention. 8 is a diagram for explaining a method of generating a grid image based on a lidar and a radar signal according to another embodiment of the present invention.

5, 6, 7 and 8, the scan unit 11 includes a plurality of lidar sensors 110 and a plurality of radar sensors 120, and the configuration unit 200 is configured in step S110. Scan data (SD: Scan Data) including a plurality of scan information (SCI: SCan Information) output from at least one of the plurality of lidar sensors 110 and the plurality of radar sensors 120 of the scan unit 11 can be input. As shown in Equation 1, each of the plurality of scan information (SCI) includes coordinates on the three-dimensional Cartesian coordinate system of each of a plurality of points on the surface of an object that reflects lasers or radio waves, and the laser or radio waves reflected from each of the plurality of points on the surface of the object. Includes reflection intensity indicating intensity.

The plane classifying unit 210 of the configuration unit 200 classifies the plurality of scan information (SCI) according to the coordinates (Z-axis) in step S120 and divides the classified plurality of scan information (SCI) into a plurality of scans belonging to each classification. A plurality of two-dimensional plane information FI is generated by using the information SCI, and the three-dimensional forming unit 220 of the configuration unit 200 includes a plurality of plane information (FI: Flat) generated by the plane classifying unit 210 . Information) is stacked in a plurality of layers to generate a plurality of stereo information (SI) having a three-dimensional structure. In particular, when generating the plurality of plane information, the plane classification unit 210 sets a plurality of unit sections for classifying the plurality of scan information SCI based on the Z-axis of the scan information SCI, and sets the plurality of units. A plurality of plane information may be generated by mapping the reflection intensity of one or more pieces of scan information (SCI) included in each section to a plane space corresponding to each of the plurality of classified unit sections. This step S120 will be described in more detail below.

When the plurality of three-dimensional information SI is generated, the grid forming unit 230 forms grid information (GI) by arranging the plurality of three-dimensional information SI in a grid in step S130 .

According to an embodiment, it is assumed that the scan unit 110 includes a first lidar sensor 111 and a second lidar sensor 112 , and a first radar sensor 121 and a second radar sensor 122 . do. In addition, the first lidar sensor 111 and the second lidar sensor 112, the first radar sensor 121, and the second radar sensor 122 scan the object at the same time point to the first lidar sensor ( 111) outputs the first scan data, the second lidar sensor 112 outputs the second scan data, the first radar sensor 121 outputs the third scan data, and the second radar sensor 121 It is assumed that ) outputs the fourth scan data. Accordingly, as shown in FIG. 7 , the configuration unit 200 generates the first stereoscopic information SI1 from the first scan data, the second stereoscopic information SI2 from the second scan data, and the third stereoscopic information SI1 from the second scan data. The fourth stereoscopic information SI4 may be generated from the stereoscopic information SI3 and the fourth scan data. Then, the grid forming unit 230 lattices the first three-dimensional information SI1, the second three-dimensional information SI2, the third three-dimensional information SI3, and the fourth three-dimensional information SI4 as shown in FIG. 7 . to form lattice information (GI).

According to another embodiment, it is assumed that the scan unit 110 includes one lidar sensor 110 . It is assumed that the lidar sensor 110 scans an object at times t-3, t-2, t-1, and t, respectively, and outputs first to fourth scan data. Accordingly, as shown in FIG. 8 , the configuration unit 200 generates t-3 th stereoscopic information SIt-3 from the first scan data and t-2 th stereoscopic information SIt-2 from the second scan data. , t-1 th stereoscopic information SIt-1 from the third scan data, and t-th stereoscopic information SIt from the fourth scan data may be generated.

Then, as shown in FIG. 8 , the grid forming unit 230 applies a predetermined rule according to the time sequence in which a plurality of three-dimensional information SIt-3, SIt-2, SIt-1, and SIt are generated in advance. The grid information GI is formed by arranging the grids at the set positions. In the example of FIG. 8 , they were arranged in a counterclockwise direction according to the time sequence.

Meanwhile, when at least one of the plurality of three-dimensional information SI1, SI2, SI3, and SI4 included in the grid information has a different height in step S130 , the grid forming unit 230 receives at least one of the three-dimensional information SI from the at least one stereoscopic information SI. A plurality of stereoscopic information SI1, SI2, included in the lattice information GI by erasing the plane information FI of The heights of SI3 and SI4) can be matched. Here, the padding information PI means plane information FI in which all values of the corresponding plane space are constants.

As described above, at least one of the generated stereoscopic information and grid information is used as input data for the deep neural network 400 . When such input data is prepared, the learning unit 300 generates learning data based on the input data in step S140. A method of generating such training data will be described in more detail below.

Then, the above-described step S120 will be described in more detail. That is, a method for generating stereoscopic information (SI) according to an embodiment of the present invention will be described. 9 is a flowchart illustrating a method of generating stereoscopic information (SI) according to an embodiment of the present invention.

6 and 9 , as described above, step S120 is started when scan data SD including a plurality of scan information SCI is input to the configuration unit 200 . When the plurality of scan information (SCI) is input, the plane classification unit 210 determines the maximum height on the Z-axis of the scan data SD including the plurality of scan information (SCI) in step S210 to determine the effective range on the Z-axis {T | Set 0 ≤ T < Zmax}.

Then, the plane classification unit 210 replaces or deletes the Z-axis value of the scan information SD equal to or greater than the maximum height Zmax on the Z-axis with the effective maximum height Zmax-1 in step S220. The effective maximum height (Zmax-1) is the effective range {T | As 0 ≤ T < Tmax}, since the end range is a closed section, the effective maximum height (Zmax-1) is subtracted by 1 from the maximum height (Zmax) on the Z axis of the effective range.

Then, in step S230 , the plane classifying unit 210 divides the plurality of scan information included in the scan data into unit sections of a predetermined number of divisions based on the Z-axis of the coordinates of the scan information according to Equation 2 below.

In Equation 2, M is the number of divisions, and is preset. In addition, Zstart is the starting point on the Z-axis of the unit section, Zend is the end point on the Z-axis of the unit section, and Zmax is the maximum height on the Z-axis of the scan data.

Next, the plane classification unit 210 normalizes the reflection intensity of the plurality of scan information included in the scan data in step S240. For example, the plane classifying unit 210 sets the maximum reflection intensity of each of the plurality of scan information SCI included in the scan data SD among the reflection intensities of the plurality of scan information SCI included in the scan data SD. It can be converted to a real value between 0 and 1 by dividing by a value.

Next, the plane classifying unit 210 forms a plurality (M pieces) of two-dimensional planar spaces corresponding to the plurality (M pieces) of unit sections in step S250 . Here, the plane space is a spherical coordinate system with Z=0. In this way, when a plurality (M pieces) of planar spaces are provided, each of the plurality of pieces of scan information SCI is mapped to a corresponding planar space. This will be described in detail as follows.

The plane classifying unit 210 converts the orthogonal coordinates of the scan information (SCI) included in the plurality of unit sections into spherical coordinates according to Equation 3 below in step S260 to derive distances and angles according to the spherical coordinates.

은 구면좌표계에 따른 원점에서의 거리이고,

는 구면좌표계에 따른 z축을 축으로 양의 방향의 x축과 이루는 각도를 나타낸다. 한편, 구면좌표계의 양의 방향의 z축과 이루는 각도 ??는 M개의 단위 구간 각각에 대응하는 평면 공간이 Z=0인 구면좌표계이기 때문에 고려하지 않는다. Here, k represents an index of scan information. In addition, x, y, and z represent coordinates of the Cartesian coordinate system of the scan information. Especially,

is the distance from the origin according to the spherical coordinate system,

represents the angle formed with the z-axis in the spherical coordinate system and the x-axis in the positive direction. On the other hand, the angle ?? formed with the z-axis in the positive direction of the spherical coordinate system is not considered because the plane space corresponding to each of the M unit sections is a spherical coordinate system in which Z=0.

및 각도

에 따른 선분을 형성한다. 그런 다음, 평면분류부(210)는 S280 단계에서 단위 구간의 스캔 정보 중 좌표 상 선분과 만나는 모든 스캔 정보의 반사 강도를 선분에 사상하여 평면 정보(FI)를 생성한다. Next, the plane classifying unit 210 determines the distance derived from the center point of a plurality of (M) two-dimensional planar spaces corresponding to the plurality of (M) unit sections formed earlier ( 250 ) in step S270 .

and angle

form a line segment according to Then, in step S280 , the plane classifying unit 210 maps the reflection intensity of all scan information that meets the line segment on the coordinates among the scan information of the unit section to the line segment to generate the plane information FI.

Next, the three-dimensional forming unit 220 generates the three-dimensional information SI having a three-dimensional structure by stacking the plurality (M pieces) of the previously generated planar information FI in step S290 in a plurality of layers, that is, in the depth direction. . Subsequently, the three-dimensional forming unit 220 may optionally add at least one piece of padding information PI to the three-dimensional information SI in step S300 . Here, the padding information PI means plane information FI in which all values of the plane space are constant C (here, C is a real number). The padding information PI is used to provide a reference value for the depth direction.

The three-dimensional information (SI) or lattice information (GI) described above is input data for the deep neural network 400 , and may be used as learning data for the deep neural network 400 by giving a label to the input data. A method of generating such training data will be described. 10 is a flowchart illustrating a method of generating learning data according to an embodiment of the present invention.

Referring to FIG. 10 , the scan unit 11 scans an object through at least one of the lidar sensor 110 and the radar sensor 120 to scan a plurality of scan data each including a plurality of scan information (SCI) ( SD) can be created. When a plurality of scan data is input from the scan unit 11 , the configuration unit 200 generates a plurality of stereoscopic images SI from the plurality of scan data SD in step S410 , and generates the plurality of stereoscopic images SI. The grid information GI may be formed by arranging them in a grid. The plurality of stereoscopic images (SI) or grid information (GI) is used as input data to the deep neural network 400 .

Then, the learning unit 300 collects label data corresponding to the input data in step S420 . Such label data may vary depending on the purpose for training the deep neural network 400 . The label data may include posture information of the object obj.

Then, the learning unit 300 generates training data by setting the previously collected label data as a label for the input data in step S430 . That is, the learning unit 300 labels with label data collected in input data that is three-dimensional information or grid information generated in response to the scan data.

According to an embodiment, the learning unit 300 may provide training data for the deep neural network 400 to estimate the postures of the robot R and the objects obj1, obj2, and obj3 in indoor and outdoor spaces. To this end, when the configuration unit 200 generates input data, the learning unit 300 generates the input data by the configuration unit 200, that is, in response to the time point t, the motion capture device, AHRS (attitude/heading) It is possible to obtain the posture information of the robot (R) or the object (obj1) as label data through the reference system) device. Next, the learning unit 300 labels the posture information obtained with respect to the input data at time t.

When the learning data is prepared as described above, the learning unit 300 may train the deep neural network 400 in step S440 . For example, the learning unit 300 inputs input data to the deep neural network 400 , and when the deep neural network 400 calculates an output value through a plurality of operations of a plurality of layers to which each weight is applied, the output value and the label data The deep neural network 400 can be trained through optimization of modifying the parameters of the deep neural network 400 so that the difference between .

As described above, when the learning is completed, the configuration unit 200 of the configuration device 10 generates a plurality of stereoscopic images SI from the scan data of the lidar sensor 110 or the object scanned by the radar sensor 110 . and arranging the plurality of stereoscopic images SI in a grid to generate grid information GI. Then, the configuration unit 200 may input the stereoscopic image (SI) or grid information (GI) as input data to the deep neural network 400 . Then, the deep neural network 400 will output the probability of the posture information as an output value through a plurality of calculations to which the weights of a plurality of layers are applied according to the learned parameters. Then, the estimator 500 may estimate the posture of the robot R or the objects obj1 , obj2 , and obj3 through the probability that is the output value.

Meanwhile, the method according to the embodiment of the present invention described above may be implemented in the form of a program readable by various computer means and recorded in a computer readable recording medium. Here, the recording medium may include a program command, a data file, a data structure, etc. alone or in combination. The program instructions recorded on the recording medium may be specially designed and configured for the present invention, or may be known and available to those skilled in the art of computer software. For example, the recording medium includes magnetic media such as hard disks, floppy disks and magnetic tapes, optical recording media such as CD-ROMs and DVDs, and magneto-optical media such as floppy disks ( magneto-optical media) and hardware devices specially configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of program instructions may include high-level languages that can be executed by a computer using an interpreter or the like as well as machine language such as generated by a compiler. Such hardware devices may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

Although the present invention has been described above using several preferred embodiments, these examples are illustrative and not restrictive. As such, those of ordinary skill in the art to which the present invention pertains will understand that various changes and modifications can be made in accordance with the doctrine of equivalents without departing from the spirit of the present invention and the scope of rights set forth in the appended claims.

10: component 11: scan unit
12: input unit 13: display unit
14: storage unit 15: communication unit
16: sensor unit 17: location information unit
18: control unit 110: lidar sensor
120: radar sensor 200: component
210: plane classification unit 220: three-dimensional forming unit
230: grid forming unit 300: learning unit
400: deep neural network 500: estimator

Claims (8)

A method for constructing input data of a deep neural network model based on lidar and radar signals,
Coordinates on the three-dimensional Cartesian coordinate system of each of the plurality of points on the object surface that the plane classification unit reflects the laser or radio wave from at least one of the lidar and the radar sensor, and the intensity of the laser or radio wave reflected from each of the plurality of points on the object surface generating a plurality of two-dimensional plane information by classifying a plurality of pieces of scan information indicating reflection intensity according to respective coordinates of the plurality of pieces of scan information; and
generating, by a three-dimensional forming unit, a plurality of three-dimensional information having a three-dimensional structure by stacking the plurality of plane information in a plurality of layers;
characterized in that it comprises
A method for organizing the input data. According to claim 1,
The step of generating the plurality of plane information is
The plane classification unit
classifying the plurality of scan information into a plurality of unit sections based on a Z-axis among coordinates of the plurality of scan information; and
generating a plurality of plane information by mapping the reflection intensity of the scan information included in each of the plurality of unit sections to a plane space corresponding to each of the classified plurality of unit sections;
characterized in that it comprises
A method for organizing the input data. 3. The method of claim 2,
The step of classifying into the plurality of unit sections is
setting, by the plane classification unit, a maximum height (Zmax) on the Z-axis of the plurality of scan information;
replacing or erasing a Z-axis value of scan information having a Z-axis value greater than or equal to the maximum height (Zmax) by the plane classification unit with an effective maximum height (Zmax-1);
The plane classification unit
formula

dividing the plurality of scan information included in the scan data into unit sections of a predetermined number of divisions based on the Z-axis of the coordinates of the scan information according to
wherein M is the number of divisions,
The Zstart is the starting point on the Z-axis of the unit section,
Zend is the end point on the Z-axis of the unit section,
The Zmax is the maximum height of the Z-axis
characterized by
A method for organizing the input data. 4. The method of claim 3,
The step of generating the plurality of plane information is
converting, by the plane classifying unit, the reflection intensity of each of the plurality of scan information included in the scan data by a maximum value among the reflection intensities of the plurality of scan information included in the scan data, into a real value between 0 and 1;
forming a plurality of planar spaces corresponding to a plurality of unit sections by the planar classification unit;
deriving a distance and an angle according to the spherical coordinates by the plane classifying unit converting the orthogonal coordinates of each of the plurality of scan information included in each of the plurality of unit sections into spherical coordinates;
forming, by the plane classification unit, a line segment according to the derived distance and angle from the center point of the plurality of planar spaces corresponding to the plurality of unit sections; and
mapping the reflection intensity of all scan information that meets the line segment on coordinates among the scan information of the unit section to the line segment;
characterized in that it comprises
A method for organizing the input data. 5. The method of claim 4,
The step of deriving the distance and angle according to the spherical coordinates
The plane classification unit
Cartesian coordinates of each of the plurality of scan information included in each of the plurality of unit sections
formula

,

Converts to spherical coordinates according to
where k is an index of scan information,
Wherein x, y, z are coordinates of the Cartesian coordinate system of the scan information,
remind

is the distance from the origin according to the spherical coordinate system,
remind

represents the angle formed with the z-axis in the spherical coordinate system and the x-axis in the positive direction.
characterized by
A method for organizing the input data. According to claim 1,
forming grid information by arranging a plurality of three-dimensional information in a grid by a grid forming unit; and
When at least one of the plurality of stereoscopic information included in the grid information has a different height, at least one piece of plane information is deleted from the at least one piece of stereoscopic information, or at least one piece of padding information is added to the at least one piece of stereoscopic information to add at least one piece of padding information to the grid. matching the heights of a plurality of stereoscopic information included in the information;
characterized in that it further comprises
A method for organizing the input data. 7. The method of claim 6,
The step of forming the grid information
wherein the grid forming unit forms grid information by arranging a plurality of three-dimensional information in a preset position according to a time sequence in which the three-dimensional information is generated and arranging them in a grid
A method for organizing the input data. 7. The method of claim 6,
collecting, by a learning unit, posture information of an object as label data in response to input data that is the three-dimensional information or the grid information; and
generating learning data by labeling, by a learning unit, posture information, which is the label data, on the input data;
characterized in that it further comprises
A method for organizing the input data.

Images