KR102538231B1 - Method for 3D analysis of semantic segmentation, and computer program recorded on record-medium for executing method thereof - Google Patents

Abstract

The present invention proposes a computer program recorded on a recording medium to execute the semantic segmentation method. The computer program may be combined with a computing device comprising a memory, a transceiver, and a processor that processes instructions resident in the memory. In addition, the computer program includes receiving, by the processor, point cloud data obtained from a lidar and an image captured through a camera at the same time, the processor, the received In order to execute the step of preprocessing the point cloud data and the image and the processor independently extracting features from each of the preprocessed point cloud data and the image and inferring semantic segmentation by fusing the extracted features, a recording medium It can be a computer program written in

Description

Method for 3D analysis of semantic segmentation, and computer program recorded on record-medium for executing method thereof}

The present invention relates to the processing of artificial intelligence (AI) learning data. More specifically, a 3D analysis method of semantic segmentation for inferring 2D semantic segmentation based on fusion data obtained through sensor fusion of a camera and lidar and reconstructing the inference result in 3D, and executing the same It relates to a computer program recorded on a recording medium for

Artificial intelligence (AI) refers to a technology that artificially implements some or all of human learning abilities, reasoning abilities, and perception abilities using computer programs. In relation to artificial intelligence (AI), machine learning refers to learning to optimize parameters with given data using a model composed of multiple parameters. Such machine learning is classified into supervised learning, unsupervised learning, and reinforcement learning according to the form of learning data.

In general, designing data for artificial intelligence (AI) learning proceeds in the steps of data structure design, data collection, data refinement, data processing, data expansion, and data verification.

To describe each step in more detail, data structure design is performed through ontology definition, classification system definition, and the like. Data collection is performed by collecting data through direct filming, web crawling, or associations/professional organizations. Data purification is performed by removing redundant data from collected data and de-identifying personal information. Data processing is performed by performing annotation and inputting metadata. Data extension is performed by performing ontology mapping and supplementing or extending the ontology as needed. In addition, data verification is performed by verifying validity according to the set target quality using various verification tools.

On the other hand, autonomous driving of a vehicle refers to a system that can judge and drive a vehicle by itself. Such autonomous driving may be classified into gradual stages from non-automation to complete automation according to the degree of involvement of the system in driving and the degree of control of the vehicle by the driver. In general, the level of autonomous driving is divided into six levels classified by the Society of Automotive Engineers (SAE) International. According to the six levels classified by the International Association of Automotive Engineers, level 0 is non-automation, level 1 is driver assistance, level 2 is partial automation, level 3 is conditional automation, level 4 is highly automated, and level 5 The steps are fully automated steps.

Autonomous driving of vehicles is performed through mechanisms of perception, localization, path planning, and control. Currently, several companies are developing to implement recognition and path planning among autonomous driving mechanisms using artificial intelligence (AI).

However, recently, as collision accidents of autonomous vehicles frequently occur, the demand for improving the safety of autonomous driving is increasing.

In particular, recent Advanced Driver Assistance Systems (ADAS) for autonomous driving centered on RGB camera sensors and 2D object detection technology can detect 3D relative distances or structures of surrounding objects. There was a problem that was hard to figure out.

In order to solve this problem, recently, a sensor fusion technique that fuses multi-modal data acquired from a 3-dimensional sensor such as lidar and a 2-dimensional sensor such as a camera and object recognition have been developed. Research on a semantic segmentation technique that extends a unit to a data configuration unit is being actively conducted.

However, most of the existing semantic segmentation studies using camera and lidar sensor fusion convert point cloud data into 2-dimensional plane data in order to mix 3-dimensional lidar point cloud data and 2-dimensional camera image data.

The planar inference result made using this has a problem in that it is difficult to provide three-dimensional information about the driving environment because the three-dimensional distance information, which is an advantage of LIDAR, is lost.

Republic of Korea Patent Registration No. 10-2073873, 'Semantic Segmentation Method and Apparatus', (2020.01.30. Registration)

One object of the present invention is to infer 2-dimensional semantic segmentation based on fusion data including 3-dimensional information obtained through sensor fusion of a camera and lidar, and to reconstruct the inference result in 3 dimensions. It is to provide a dimensional analysis method.

Another object of the present invention is to infer 2D semantic segmentation based on fusion data including 3D information obtained through sensor fusion of a camera and lidar, and to reconstruct the inference result in 3D. To provide a computer program recorded on a recording medium in order to execute a dimensional analysis method.

The technical problems of the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art from the following description.

In order to achieve the technical problem as described above, the present invention proposes a 3D analysis method of 2D semantic segmentation. The method includes receiving, by a learning data generating device, point cloud data obtained from a lidar and an image captured through a camera at the same time, the learning data generating device Generating a 2-dimensional semantic segmentation feature map from the point cloud data and the image based on machine learning artificial intelligence (AI), and the learning data generating device 3 from the generated feature map Interpreting dimensional information may be included.

Specifically, it is characterized in that it further comprises the step of pre-learning the artificial intelligence prior to the generating step.

The pre-learning step is the point cloud data obtained from the lidar included in the pre-stored data set, the image captured through the camera at the same time as the point cloud data, the calibration information between the lidar and the camera, and the 3-dimensional point cloud data It is characterized in that the artificial intelligence is pre-learned based on correct answer data in which class labels are specified in units of points.

In the pre-learning step, the point cloud data and the correct answer data are projected onto coordinates on a two-dimensional plane of the image, and pixels whose Manhattan distance is within a predetermined value from pixels to be projected in the image are assigned the same class. It is characterized by having.

The pre-learning step is characterized in that the artificial intelligence is pre-learned through at least one of a plurality of loss functions.

The pre-learning step is characterized in that the artificial intelligence is pre-learned through a loss function represented by Equation 1 below.

[Equation 1]

(Here, p _pred is the inference probability based on cross entropy, and p _true means the correct answer.)

The pre-learning step is characterized in that the artificial intelligence is pre-learned through a loss function represented by Equation 2 below.

[Equation 2]

(Where p _pred is the inference probability based on the Dice coefficient, and p _true means the correct answer.)

The pre-learning step is characterized in that pixels corresponding to the background class are excluded from the calculation of the loss value regardless of the type of the loss function.

The analyzing step is characterized in that a pair of 3D coordinates of the point cloud data and a 2D plane coordinate pair of an image projected with the point cloud data is prepared.

The interpreting step is characterized by applying a maximum pooling operation using a 3*3 kernel centered on the corresponding coordinate for each coordinate on a two-dimensional plane to the generated feature map.

The analyzing step is characterized in that a vector obtained by estimating a segment for each coordinate is obtained through the maximum pooling operation.

In the interpreting step, a class having the highest probability among the obtained vectors is predicted as a class inferred with respect to 3D coordinates of the point cloud data.

In order to achieve the technical problem as described above, the present invention proposes a computer program recorded on a recording medium to execute a 3D analysis method of 2D semantic segmentation. The computer program may be combined with a computing device comprising a memory, a transceiver, and a processor that processes instructions resident in the memory. In addition, the computer program includes the step of receiving, by the processor, point cloud data obtained from a lidar and an image captured through a camera at the same time, the processor, prior machine learning ( Generating a 2D semantic segmentation feature map from the point cloud data and the image based on machine learning (AI) and interpreting 3D information from the processor and the generated feature map To be executed, it may be a computer program recorded on a recording medium.

Details of other embodiments are included in the detailed description and drawings.

According to embodiments of the present invention, 2D semantic segmentation can be inferred based on fusion data including 3D information obtained through sensor fusion of a camera and lidar, and the inference result can be reconstructed into 3D.

The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description of the claims.

1 is a configuration diagram showing an artificial intelligence learning system according to an embodiment of the present invention.
2 is an exemplary diagram for explaining the configuration of a learning data collection device according to an embodiment of the present invention.
3 is a logical configuration diagram of an apparatus for generating learning data according to an embodiment of the present invention.
4 is a hardware configuration diagram of an apparatus for generating learning data according to an embodiment of the present invention.
5 is a flowchart illustrating a semantic segmentation and 3D analysis method according to an embodiment of the present invention.
6 is an exemplary view for explaining a semantic segmentation and 3D analysis method according to an embodiment of the present invention.
7 is an exemplary view for specifically explaining a semantic segmentation method according to an embodiment of the present invention.
8 is an exemplary diagram for explaining a 3D analysis method according to an embodiment of the present invention.
9 is an exemplary diagram for explaining performance of a 3D analysis method according to an embodiment of the present invention.
10 is a diagram showing a 3D analysis result of 2D semantic segmentation according to an embodiment of the present invention.

It should be noted that the technical terms used in this specification are only used to describe specific embodiments and are not intended to limit the present invention. In addition, technical terms used in this specification should be interpreted in terms commonly understood by those of ordinary skill in the art to which the present invention belongs, unless specifically defined otherwise in this specification, and are excessively inclusive. It should not be interpreted in a positive sense or in an excessively reduced sense. In addition, when the technical terms used in this specification are incorrect technical terms that do not accurately express the spirit of the present invention, they should be replaced with technical terms that those skilled in the art can correctly understand. In addition, general terms used in the present invention should be interpreted as defined in advance or according to context, and should not be interpreted in an excessively reduced sense.

Also, singular expressions used in this specification include plural expressions unless the context clearly indicates otherwise. In this application, terms such as "consisting of" or "having" should not be construed as necessarily including all of the various components or steps described in the specification, and some of the components or steps are included. It should be construed that it may not be, or may further include additional components or steps.

Also, terms including ordinal numbers such as first and second used in this specification may be used to describe various components, but the components should not be limited by the terms. These terms are only used for the purpose of distinguishing one component from another. For example, a first element may be termed a second element, and similarly, a second element may be termed a first element, without departing from the scope of the present invention.

When a component is referred to as being “connected” or “connected” to another component, it may be directly connected or connected to the other component, but other components may exist in the middle. On the other hand, when a component is referred to as “directly connected” or “directly connected” to another component, it should be understood that no other component exists in the middle.

Hereinafter, preferred embodiments according to the present invention will be described in detail with reference to the accompanying drawings, but the same or similar components are given the same reference numerals regardless of reference numerals, and redundant description thereof will be omitted. In addition, in describing the present invention, if it is determined that a detailed description of a related known technology may obscure the gist of the present invention, the detailed description will be omitted. In addition, it should be noted that the accompanying drawings are only for easily understanding the spirit of the present invention, and should not be construed as limiting the spirit of the present invention by the accompanying drawings. The spirit of the present invention should be construed as extending to all changes, equivalents or substitutes other than the accompanying drawings.

On the other hand, recently, sensor fusion, which fuses multi-modal data acquired from 3-dimensional sensors such as lidar and 2-dimensional sensors such as cameras, and object recognition units are used as data constituent units. Semantic segmentation that extends up to is being actively studied.

Sensor fusion is a technology that mutually fuses multi-modal data acquired from different types of sensors. Sensor fusion is classified into three types according to the point at which data or information formed from multiple sources in the field of deep learning is fused.

Data level or early fusion creates fusion data with a new phenotype by fusing the data itself acquired from the source. This convergence data is transmitted to the neural network and used to extract a feature map and create a prediction map. This method has the advantage that the overall structure of the neural network is configured relatively simply because the type of data received as input to the neural network is small.

Unlike this, deep feature level or mid-level fusion delivers multi-modal data obtained from the source to the neural network as input. The multi-modal features extracted from each input through the parallel structured neural network are fused into one inside the neural network to form a deep fusion feature, which is used to create the result inference map. This method has the advantage that it can be used in various ways, from each feature extracted from multi-modal data to their convergence features, in inferring the result of the neural network.

Inference level fusion (score level or late fusion) also transfers multi-modal data as input to the neural network, but unlike feature level fusion, independent neural networks are used for each source. The resulting inference maps independently created by each neural network are used to derive the final inference map by simple operations such as summation and average or by using a separately learned classifier. This method has the advantage that the inference result is robust to the influence of the characteristic difference or mutual interference between the sensors used for fusion.

Semantic segmentation is one of the key element technologies required for an autonomous vehicle to recognize a driving environment, and objects of a 2D RGB image acquired from a camera can be classified (dense prediction) in units of pixels.

In particular, recently, a transformer module that can extract deep features with higher expressiveness by finding out the relative importance between features based on self-attention, going further than the method using a deep neural network centered on the existing convolutional layer, has been developed. In addition, studies to improve the performance of 2D semantic segmentation are increasing.

Using the transformer module, it is possible to extract deep features with good expressiveness even if a relatively small number of convolutional layers are used in the neural network, and based on these deep features, it showed relatively better performance than existing methods.

However, as mentioned above, it is difficult to grasp the structure or relative distance of an object in 3D, which is the actual driving environment of a vehicle, with only the inference results in the form of a plane created by these 2D semantic segmentation methods. When using multiple cameras, there is a problem that the amount of calculation is excessively increased.

In addition, most of the existing semantic segmentation studies that utilize camera and lidar sensor fusion convert point cloud data into 2-dimensional plane data in order to mix 3-dimensional lidar point cloud data and 2-dimensional camera image data.

It is difficult to provide three-dimensional information about the driving environment because the planar inference result made using this is in a state where the three-dimensional distance information, which is an advantage of LIDAR, is lost.

In order to overcome these limitations, the present invention is a semantic segmentation technique and various 2D semantic segmentation results that can be reconstructed in 3D based on fusion data including 3D information obtained through sensor fusion of cameras and lidar. I would like to suggest means.

1 is a configuration diagram showing an artificial intelligence learning system according to an embodiment of the present invention.

As shown in FIG. 1, the artificial intelligence learning system according to an embodiment of the present invention may include a learning data collection device 100, a learning data generating device 200, and an artificial intelligence learning device 300. there is.

Since the components of the artificial intelligence learning system according to an embodiment are merely functionally distinct elements, two or more components are integrated and implemented in an actual physical environment, or one component is implemented in an actual physical environment. may be implemented separately from each other.

To describe each component, the learning data collection device 100 includes a lidar and a camera installed in a vehicle to collect data for machine learning of artificial intelligence (AI) that can be used for autonomous driving. It is a device that collects data in real time from a camera. However, it is not limited thereto, and the learning data collection device 100 may include a radar and an ultrasonic sensor. In addition, sensors that are controlled by the learning data collection device 100 and are installed in a vehicle to obtain, photograph, or detect machine learning data are not limited to being provided one by one for each type, and are provided in plural even if they are the same type of sensor. It can be.

The types of sensors that are controlled by the learning data collection device 100 and are installed in the vehicle to acquire, photograph, or sense machine learning data will be described in more detail later with reference to FIG. 2 .

With the following configuration, the learning data generating device 200 receives data collected in real time by each learning data collecting device 100 from each of the plurality of learning data collecting devices 100 using mobile communication. and annotate the received data.

The learning data generating device 200 is configured to preemptively use artificial intelligence (AI) learning before a request for artificial intelligence (AI) learning data is received from the artificial intelligence learning device 400. You can build big data that can generate data.

Characteristically, the learning data generating apparatus 200 receives point cloud data obtained from a lidar and an image captured through a camera at the same time, and transmits the received point cloud data and image Preprocessing, independently extracting features from each of the preprocessed point cloud data and images based on preprocessed artificial intelligence (AI), and fusing the extracted features to create a semantic segmentation feature map. can

In addition, the learning data generating device 200 receives point cloud data obtained from a lidar and an image captured through a camera at the same time, and performs prior machine learning. Based on artificial intelligence (AI), a 2D semantic segmentation feature map can be created from point cloud data and images, and 3D information can be analyzed from the created feature map.

The learning data generation device 200 having such characteristics is any device capable of transmitting and receiving data with the learning data collection device 100 and the artificial intelligence learning device 300 and performing calculations based on the transmitted and received data. Any device may be acceptable.

For example, the learning data generating device 200 may be any one of a fixed computing device such as a desktop, workstation, or server, but is not limited thereto.

Meanwhile, a detailed description of the learning data generating device 200 will be described later with reference to FIGS. 3 and 4 .

With the following configuration, the artificial intelligence learning device 300 is a device that can be used to develop artificial intelligence (AI).

Specifically, the artificial intelligence learning device 300 provides the learning data generating device 200 with a request value including requirements that AI learning data must satisfy in order for AI to achieve its development purpose. can transmit The artificial intelligence learning device 300 may receive artificial intelligence (AI) learning data from the learning data generating device 200 . And, the artificial intelligence learning device 300 may machine learn the artificial intelligence (AI) to be developed using the received artificial intelligence (AI) learning data.

As such, the artificial intelligence learning device 300 may be any device capable of transmitting and receiving data to and from the learning data generating device 200 and performing calculations using the transmitted and received data. For example, the artificial intelligence learning device 300 may be any one of a fixed computing device such as a desktop, workstation, or server, but is not limited thereto.

As described above, one or more learning data collection devices 100, learning data generating devices 200, and artificial intelligence learning devices 300 may include at least one of a security line, a common wired communication network, or a mobile communication network directly connected between devices. Data can be transmitted and received using the combined network.

For example, public wired communication networks may include Ethernet, x Digital Subscriber Line (xDSL), Hybrid Fiber Coax (HFC), and Fiber To The Home (FTTH). It may be, but is not limited thereto. In addition, in the mobile communication network, Code Division Multiple Access (CDMA), Wideband CDMA (WCDMA), High Speed Packet Access (HSPA), Long Term Evolution, LTE) and 5th generation mobile telecommunication may be included, but is not limited thereto.

2 is an exemplary view for explaining sensors according to an embodiment of the present invention.

As shown in FIG. 2, the learning data collection device 100 according to an embodiment of the present invention includes a radar 20, a lidar 30, a camera 40, and an ultrasonic sensor ( 50), it is possible to collect basic data for machine learning of artificial intelligence (AI).

Here, the vehicle 10 is a vehicle equipped with a radar 20, a lidar 30, a camera 40, and an ultrasonic sensor 50 for collecting basic data for machine learning of artificial intelligence (AI). It can be distinguished from vehicles that perform autonomous driving by intelligence (AI).

The radar 20 is fixedly installed in the vehicle 10 and emits electromagnetic waves toward the driving direction of the vehicle 10, and the electromagnetic waves reflected by an object located in front of the vehicle 10 and returned. By sensing, the vehicle 10 may generate sensing data corresponding to an image of the front side.

In other words, the sensing data is information on points at which electromagnetic waves emitted by the radar 20 fixedly installed in the vehicle 10 toward the driving direction of the vehicle are reflected. Accordingly, the coordinates of the points included in the sensing data may have values corresponding to the position and shape of an object located in front of the vehicle 10 . This sensed data may be 2D information, but is not limited thereto and may be 3D information.

The lidar 30 is fixedly installed on the vehicle 10, radiates a laser pulse around the vehicle 10, and detects light reflected back by an object located around the vehicle 10. , 3D point cloud data corresponding to a 3D image of the surroundings of the vehicle 10 may be generated.

In other words, the 3D point cloud data is three-dimensional information on points obtained by reflecting laser pulses emitted around the vehicle by the LIDAR 30 fixedly installed in the vehicle 10 . Accordingly, coordinates of points included in the 3D point cloud data may have values corresponding to the location and formation of objects located around the vehicle 10 .

The camera 40 is fixedly installed in the vehicle 10 and can capture a two-dimensional image of the surroundings of the vehicle 10 . As such, a plurality of cameras 40 may be installed horizontally or spaced apart from each other in a horizontal direction so as to photograph different directions. For example, FIG. 2 shows an example of a vehicle 10 in which six cameras 40 capable of capturing images in six different directions are fixedly installed, but the cameras 40 that can be installed in the vehicle 10 It will be apparent to those skilled in the art that it can be configured in various numbers.

In other words, the 2D image is an image captured by the camera 40 fixed to the vehicle 10 . Accordingly, the 2D image may include color information of an object located in a direction in which the camera 40 faces.

The ultrasonic sensor 50 is fixedly installed in the vehicle 50, emits ultrasonic waves around the vehicle 10, detects sound waves reflected by an object positioned adjacent to the vehicle 10 and returns, Distance information corresponding to the distance between the ultrasonic sensor 50 installed in (10) and the object may be generated. In general, a plurality of ultrasonic sensors 50 may be configured and fixedly installed at the front, rear, front and rear sides of the vehicle 10, which are easily contacted with objects.

In other words, the distance information is information about a distance from an object detected by the ultrasonic sensor 50 fixedly installed in the vehicle 10 .

Hereinafter, the configuration of the learning data generating device 200 as described above will be described in more detail.

3 is a logical configuration diagram of an apparatus for generating learning data according to an embodiment of the present invention.

Referring to FIG. 3 , the training data generating device 200 includes a communication unit 205, an input/output unit 210, a pre-learning unit 215, a data pre-processing unit 220, an inference unit 225, a 3D analysis unit ( 230) and a storage unit 235.

Since the components of the learning data generation device 200 are merely functionally distinct elements, two or more components are integrated and implemented in an actual physical environment, or one component is mutually exclusive in an actual physical environment. It could be implemented separately.

Specifically, the communication unit 205 may receive image and point cloud data for machine learning of artificial intelligence (AI) from the learning data collection device 100 .

Also, the communication unit 205 may receive a calibration matrix for synchronizing point cloud data and images from the learning data collection device 100 together.

In addition, the communication unit 205 may transmit the 3D analysis result of the semantic segmentation to the artificial intelligence learning device 300 .

With the following configuration, the input/output unit 210 may receive a signal from a user through a user interface (UI) or output a calculated result to the outside.

Specifically, the input/output unit 210 may generate a semantic segmentation feature map from a user or receive various setting values for 3D analysis of the generated semantic segmentation feature map, and output generated resultant values.

With the following configuration, the pre-learning unit 215 may pre-train artificial intelligence for inferring 2D semantic segmentation.

Specifically, the pre-learning unit 215 includes point cloud data obtained from lidar included in a pre-stored data set, an image taken through a camera at the same time as the point cloud data, calibration information between lidar and camera, and 3-dimensional point cloud data Artificial intelligence can be trained in advance based on correct answer data in which class labels are specified in units of points.

At this time, the pre-learning unit 215 projects the point cloud data and the correct answer data onto the coordinates of the 2D plane of the image, so that the pixel to be projected and the pixels whose Manhattan distance is within a preset value have the same class. can do.

Also, the pre-learning unit 215 may pre-learn artificial intelligence through at least one of a plurality of loss functions.

At this time, the pre-learning unit 215 may pre-learn artificial intelligence through a loss function represented by Equation 1 below.

[Equation 1]

(Here, p _pred is the inference probability based on cross entropy, and p _true means the correct answer.)

In addition, the pre-learning unit 215 may pre-learn the artificial intelligence through a loss function represented by Equation 2 below.

[Equation 2]

(Where p _pred is the inference probability based on the Dice coefficient, and p _true means the correct answer.)

In this case, the pre-learning unit 215 may exclude pixels corresponding to the background class from calculation of the loss value regardless of the type of the loss function.

With the following configuration, the data pre-processing unit 220 may pre-process the point cloud data and images received from the learning data collection device 100 .

Specifically, the data preprocessing unit 220 may project the point cloud data onto coordinates on a two-dimensional plane having the same size as the image based on the calibration matrix received together with the point cloud data and the image.

Here, the calibration matrix may be expressed by Equation 3 below.

[Equation 3]

(Where u and v are the 2-dimensional coordinates of the pixels in the image, x, y and z are the 3-dimensional coordinates of the point cloud data, f _u and f _v are the focal lengths in pixels, u ₀ and v ₀ are the point clouds on the image plane It means the x and y coordinates where the data is located.)

With the following configuration, the inference unit 225 independently extracts features from each of the preprocessed point cloud data and images based on pre-machine learning artificial intelligence (AI), fuses the extracted features, A semantic segmentation feature map may be created.

Specifically, the inference unit 225 may infer semantic segmentation through an encoder that extracts deep features from projected point cloud data and the image, and a decoder that generates a segmentation inference map by interpreting the deep features.

At this time, the inference unit 225 generates deep features through an encoder, accepts projected point cloud data and images in a multi-modal manner, and uses a plurality of convolution blocks arranged in a parallel structure to project a point cloud. Regional and structural features can be independently extracted from data and images.

That is, the inference unit 225 extracts a feature map by stepwise downsampling the projected point cloud data and the image to 1/2 and 1/4 of the input size through the first convolution block and the second convolution block, respectively. can

Next, the inference unit 225 connects the downsampled feature maps through a concatenation layer (concatenate), and generates a converged feature map having an area of 1/8 of the input size from the concatenated feature maps through a third convolution block. can

Next, the inference unit 225 divides the generated fusion feature map into patches of a preset size, constructs patch embeddings representing corresponding regions from each region, and position embeddings. ) to generate an embedding vector representing each patch.

Next, the inference unit 225 may convert the generated embedding vector into a fusion based deep feature through a plurality of consecutive transformer modules.

Next, the inference unit 225 may convert the deep feature vector converted through the reshape layer into a feature map having a size of 1/16 of the input size.

Next, the reasoning unit 225 expands the area of the converted feature map by a predetermined multiple through a plurality of up-sampling layers, and converts the feature maps in each of the plurality of up-sampling layers to the encoder through a joint layer. In each step, it can be joined with the corresponding feature map.

Here, the inference unit 225 may pass feature maps upsampled in each of the plurality of upsampling layers to a convolutional layer using a 3*3 kernel.

With the following configuration, the 3D analysis unit 230 may analyze 3D information from the generated feature map.

Specifically, the 3D analyzer 230 may prepare a pair of 3D coordinates of point cloud data and 2D plane coordinates of an image projected with the point cloud data.

In addition, the 3D analyzer 230 may apply a max pooling operation using a kernel having a size of 3*3 centered on the corresponding coordinate for each coordinate on the 2D plane to the generated feature map. .

Also, the 3D analyzer 230 may obtain a vector obtained by estimating segments for each coordinate through a maximum pooling operation.

Also, the 3D analyzer 230 may predict a class having the highest probability among the obtained vectors as a class inferred with respect to 3D coordinates of the point cloud data.

With the following configuration, the storage unit 235 may store data necessary for the operation of the learning data generating device 200 . The storage unit 235 may store data necessary for designing data for artificial intelligence (AI) learning.

Hereinafter, hardware for implementing the above-described logical components of the learning data generating device 200 will be described in more detail.

4 is a hardware configuration diagram of an apparatus for generating learning data according to an embodiment of the present invention.

Referring to FIG. 4, the learning data generation device 200 includes a processor (250), a memory (Memory, 255), a transceiver (260), an input/output device (265), and a data bus (Bus). , 270) and storage (Storage, 275).

The processor 250 may implement operations and functions of the learning data generating device 200 based on instructions according to the software 280a residing in the memory 255 . Software 280a implementing the method according to the present invention may be loaded in the memory 255 . The transceiver 260 may transmit and receive data with the learning data collection device 100 and the artificial intelligence learning device 300 .

The input/output device 265 may receive data necessary for the operation of the learning data design device 200 and output the generated result value. The data bus 270 is connected to the processor 250, the memory 255, the transceiver 260, the input/output device 265, and the storage 275, and is a movement path for transferring data between each component. role can be fulfilled.

The storage 275 may store an application programming interface (API), a library file, a resource file, and the like necessary for the execution of the software 280a implemented in another method according to the present invention. The storage 275 may store software 280b in which the method according to the present invention is implemented. In addition, the storage 275 may store information necessary for performing the semantic segmentation method and the 3D analysis method. In particular, the storage 275 may include a database 285 storing programs for performing a semantic segmentation method and a 3D analysis method.

According to one embodiment of the present invention, the software 280a, 280b resident in the memory 255 or stored in the storage 275 causes the processor 250 to obtain the point cloud data (point cloud) and At the same time, a step of receiving an image captured through a camera and preprocessing the received point cloud data and image, and the processor 250 independently extracts and extracts features from each of the preprocessed point cloud data and image In order to execute the step of inferring semantic segmentation by fusing the selected features, it may be a computer program recorded on a recording medium.

In addition, according to an embodiment of the present invention, the software 280a, 280b residing in the memory 255 or stored in the storage 275 allows the processor 250 to obtain point cloud data from lidar. ) and at the same time receiving an image taken through a camera, processor 250, based on artificial intelligence (AI) based on pre-machine learning, point cloud data and image 2 It may be a computer program recorded on a recording medium to execute the step of generating the dimensional semantic segmentation feature map and the step of interpreting the 3D information from the processor 250 and the generated feature map.

More specifically, the processor 250 may include an Application-Specific Integrated Circuit (ASIC), another chipset, a logic circuit, and/or a data processing device. The memory 255 may include read-only memory (ROM), random access memory (RAM), flash memory, a memory card, a storage medium, and/or other storage devices. The transceiver 260 may include a baseband circuit for processing wired/wireless signals. The input/output device 265 includes an input device such as a keyboard, a mouse, and/or a joystick, and a Liquid Crystal Display (LCD), an Organic LED (OLED), and/or a liquid crystal display (LCD). Alternatively, an image output device such as an active matrix OLED (AMOLED) may include a printing device such as a printer or a plotter.

When the embodiments included in this specification are implemented as software, the above-described method may be implemented as a module (process, function, etc.) that performs the above-described functions. A module may reside in memory 255 and be executed by processor 250 . The memory 255 may be internal or external to the processor 250 and may be connected to the processor 250 by various well-known means.

Each component shown in FIG. 4 may be implemented by various means, for example, hardware, firmware, software, or a combination thereof. In the case of hardware implementation, one embodiment of the present invention includes one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), FPGAs ( Field Programmable Gate Arrays), processors, controllers, microcontrollers, microprocessors, etc.

In addition, in the case of implementation by firmware or software, an embodiment of the present invention is implemented in the form of a module, procedure, function, etc. that performs the functions or operations described above, and is stored on a recording medium readable through various computer means. can be recorded. Here, the recording medium may include program commands, data files, data structures, etc. alone or in combination. Program instructions recorded on the recording medium may be those specially designed and configured for the present invention, or those known and usable to those skilled in computer software. For example, recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs (Compact Disk Read Only Memory) and DVDs (Digital Video Disks), floptical It includes hardware devices specially configured to store and execute program instructions, such as magneto-optical media, such as a floptical disk, and ROM, RAM, flash memory, and the like. Examples of program instructions may include high-level language codes that can be executed by a computer using an interpreter or the like as well as machine language codes generated by a compiler. These hardware devices may be configured to operate as one or more pieces of software to perform the operations of the present invention, and vice versa.

5 is a flowchart illustrating a semantic segmentation and 3D analysis method according to an embodiment of the present invention.

Referring to FIG. 5 , in step S100, the learning data generating device may receive point cloud data and images from the learning data collecting device. In this case, the learning data generating device may additionally receive a calibration sequence along with the point cloud data and the image.

Next, in step S200, the learning data generating device may pre-process the point cloud data and image received from the learning data collecting device.

Specifically, the learning data generating apparatus may project the point cloud data onto coordinates on a two-dimensional plane having the same size as the image based on the calibration matrix received together with the point cloud data and the image.

Next, in step S300, the learning data generating device independently extracts features from each of the preprocessed point cloud data and images based on machine learning artificial intelligence (AI), and fuses the extracted features. Thus, a semantic segmentation feature map can be generated.

Specifically, the learning data generation apparatus may infer semantic segmentation through an encoder that extracts deep features from the projected point cloud data and the image, and a decoder that analyzes the deep features and generates a segmentation inference map.

At this time, the learning data generation apparatus generates deep features through an encoder, accepts projected point cloud data and images in a multi-modal manner, and uses a plurality of convolutional blocks arranged in a parallel structure to projected point cloud data. and regional and structural features can be independently extracted from the image.

That is, the learning data generating apparatus extracts a feature map by stepwise downsampling the projected point cloud data and the image to 1/2 and 1/4 of the input size through the first convolution block and the second convolution block, respectively. there is.

In addition, the training data generating device may connect the downsampled feature maps through a concatenation layer, and generate a converged feature map having an area of 1/8 of the input size with a feature map connected through a third convolution block. .

The learning data generating device divides the generated fusion feature map into patches of a preset size, constructs patch embeddings representing corresponding regions from each region, and adds position embeddings to obtain An embedding vector representing each patch may be generated.

The learning data generating device may convert the generated embedding vector into a fusion based deep feature through a plurality of consecutive transformer modules.

The learning data generation apparatus may convert the deep feature vector converted through the reshape layer into a feature map having a size of 1/16 of the input size.

The learning data generating device expands the area of the converted feature map by a preset multiple through a plurality of up-sampling layers, and responds to the feature maps in each of the plurality of up-sampling layers in each step of the encoder through a joint layer. It can be joined with a feature map that

Here, the learning data generation apparatus may pass the feature map upsampled in each of the plurality of upsampling layers to a convolution layer using a 3*3 kernel.

In step S400, the learning data generating device may interpret 3D information from the generated feature map.

Specifically, the learning data generating device may prepare a pair of 3D coordinates of the point cloud data and 2D plane coordinates of an image projected with the point cloud data.

In addition, the learning data generating apparatus may apply a max pooling operation using a kernel having a size of 3*3 centered on the corresponding coordinate for each coordinate on a 2D plane to the generated feature map.

In addition, the learning data generating device may obtain a vector obtained by estimating a segment for each coordinate through a maximum pooling operation.

Also, the learning data generation apparatus may predict a class having the highest probability among the obtained vectors as a class inferred for 3D coordinates of the point cloud data.

With reference to the following drawings, a semantic segmentation and 3D analysis method according to an embodiment of the present invention will be described in more detail.

6 is an exemplary diagram for explaining a semantic segmentation and 3D analysis method according to an embodiment of the present invention, and FIG. 7 is an exemplary diagram for explaining a semantic segmentation method in detail according to an embodiment of the present invention, 8 is an exemplary diagram for explaining a 3D analysis method according to an embodiment of the present invention.

Referring to FIGS. 6 to 8 , in the data source step, a 2D RGB image of the front of the vehicle and 3D point cloud data of the entire area around the vehicle are obtained from a camera and lidar sensor, respectively. In addition, since the data acquisition areas of the two sensors are different from each other, a calibration matrix used to synchronize them is acquired together.

Then, in the data pre-processing step, a process of selecting data of the front area of the vehicle, that is, the same area as the area from which the camera acquires the image, is selected from among the data obtained from lidar.

The N point cloud coordinates (x, y, z) data (N, 3) of the front region identified through this are projected coordinates ( u,v). As a result, N three-dimensional coordinates (x, y, z), which are the point cloud data of the front area, are converted into a projection image of (H, W, 1) size, and this projection image and an RGB image of (H, W, 3) size are converted. It is passed on to the next step, the inference step.

In the prediction step, 2D semantic segmentation is performed using a deep neural network based on the two previously transmitted images. The deep neural network used at this time consists of an encoder-decoder structure including an encoder that extracts deep features from input and a decoder that creates a segmentation inference map by interpreting the deep features.

Here, the encoder extracts regional and structural features of the input data based on convolutional blocks of “ResNet50” and converts the extracted features into deep features with better expressiveness using a transformer module.

Instead, the neural network is configured so that it can simultaneously accept inputs for projection images (remission input) and inputs for RGB images (RGB input), and the modules in the neural network are arranged in a parallel structure according to the sensor fusion method, unlike the existing method. It enables processing of data.

In addition, the decoder consists of several neural network layers, including convolutional layers, so that the feature maps extracted at each stage of the encoder are connected to the deep features to construct a resulting inference map of the same (H, W, C) size as the input. do. Here, C is the total number of classes that can classify objects in units of pixels.

Accordingly, as shown in FIG. 7, the neural network accepts multi-modal inputs and independently extracts structural and regional features of each input from each input using convolution blocks arranged in a parallel structure.

Accordingly, feature maps of 1/2 and 1/4 area compared to the extracted input are extracted, and among them, feature maps of 1/4 area are connected to each other by concatenating them to be used for feature stage fusion. The connection feature map created in this way is made into a fusion feature map with an area 1/8 of the input through the third convolution block.

Then, in order to deliver the fusion feature map created previously to the transformer module, the fusion feature map is divided into (2, 2) sized patches, and then patch embeddings representing the corresponding regions are constructed from each region, and location embeddings ( position embedding) to create embedding vectors representing each patch. Subsequently, these embedding vectors are converted into a fusion based deep feature centered on important features included in the fusion feature map while passing through a transformer block composed of 9 consecutive transformer modules.

Next, in the decoder, the deep feature vectors first extracted from the encoder are converted into a feature map with a size of 1/16 of the input through a reshape layer, pass through a convolution layer using a 3*3 kernel, and The area is doubled through an up sampling layer.

Even if this extended feature map contains the main features previously included in the deep feature vector, the local and structural features lost due to vectorization are not easily recovered only by convolution and upsampling.

Therefore, by attaching the corresponding feature map and the feature map of the same size previously extracted from the encoder using the joint layer, the feature map can contain the main features of the deep feature vector and the regional and structural features extracted at the time of encoding at the same time. do.

Subsequently, the feature maps are mixed with each other using a convolution layer using a 3*3 kernel, extended through an upsampling layer, and the process of joining feature maps of the same size in the encoder is repeated twice more. Restores the size of up to 1/2 of the input.

Finally, the feature map is upsampled once again and a convolution layer using a 3*3 kernel is applied to create a feature map of the same size as the input, and then convolved using the softmax activation function and a 1*1 kernel. It is passed to the segmentation head, which is a solution layer, to create a segmentation inference map.

In the data post-processing step, which is the last step of the process, as shown in FIG. 8, a process of reinterpreting the 2D segmentation inference map created by the neural network into 3D information using point cloud data (reconstruction) Do it.

First, in the data pre-processing step, N 3-dimensional coordinates (x, y, z) used to generate the projected image and N 2-dimensional coordinates (u, v) pairs projected thereon are prepared. Next, for each 2-dimensional coordinate (u, v) for the (H, W, C) size segmentation inference map inferred by the neural network, maximum pooling using a 3*3 kernel centered on the corresponding coordinate (max pooling) operation is applied.

Through this, the present invention can obtain a vector obtained by estimating segments for each of the N coordinates, and among them, the class with the highest probability is regarded as the inferred class for the three-dimensional coordinates (x, y, z).

Therefore, it is possible to obtain semantic segmentation inference results for each point cloud data of the forward region separated in the data preprocessing step, and as a result, an output matrix of size (N, C) containing segment values of each coordinate can be obtained.

Hereinafter, performance of the semantic segmentation method and the 3D analysis method according to an embodiment of the present invention will be described.

9 is an exemplary diagram for explaining performance of a 3D analysis method according to an embodiment of the present invention.

Experiment

The experiment was conducted in a hardware environment equipped with an "AMD EPYC 7742 64-Core Processor" class CPU (Central Processing Unit), an "A100 80GB" class GPU (Graphics Processing Unit), and 2TB of memory.

The software environment for neural network implementation was used by setting "Tensorflow 2.80" and "Keras 2.80" to "Python 3.9" installed on "ubuntu 20.04" operating system.

The transformer module used in the encoder uses the GeLU (Gauusian Error Linear Unit) activation function, all convolution layers constituting the decoder use the ReLU activation function except for the segmentation head, and after each convolution layer A batch normalization layer and a dropout layer were placed to minimize overfitting of the neural network.

Adam (Adaptive Moment Estimation) optimizer was used to train the neural network, and the learning rate was started with 0.001 and reduced by a factor of 0.75 whenever learning did not progress more than 2 epochs.

Learning is set to proceed for up to 500 epochs, and at this time, if the loss value does not continuously decrease by more than 10 epochs, the learning is set to end early.

The data set used for learning and evaluation of the semantic segmentation method in an embodiment of the present invention is "semantic KITTI" and is widely used in research on 3D object recognition or semantic segmentation in the field of autonomous driving.

The data set used consists of a total of 21 sequences, and each sequence includes 2D RGB frame images, 3D point clouds, calibration information between the camera and lidar sensor that acquired each data, and class labels for each 3D point. contains the specified Ground Truth (GT) data.

In this experiment, a total of 10 sequences from 0 to 10, excluding the 8th sequence, and a total of 19,130 frames are used as training data, and the 8th sequence consisting of 4,071 frames is used as evaluation data.

However, answer data for 2D semantic segmentation does not exist in the used data set. Therefore, when correct answer data for 2D semantic segmentation is created by projecting the correct answer data assigned to the point cloud data in the same way as for projecting the point cloud data, most of the pixel values are 0 as shown in FIG. 9 (a). sparse data with

This acts as a factor that extremely hinders the learning of the neural network, so to compensate for this, pixels whose Manhattan distance is within 2 of the pixels to be projected in the correct answer data have the same class, and Likewise, the sparseness of the correct answer data was reduced and utilized.

In the course of the experiment, the "Focal loss" function and the "Dice loss" function represented by Equations 1 and 2 below were used as the loss function used for learning the neural network.

[Equation 1]

(Here, p _pred is the inference probability based on cross entropy, and p _true means the correct answer.)

[Equation 2]

(Where p _pred is the inference probability based on the Dice coefficient, and p _true means the correct answer.)

The focal loss function calculates the loss value by calculating the weighted sum between the inference probability (p _pred ) and the correct answer (p _true ) based on cross entropy as shown in Equation 1, and the inference accuracy (accuracy) for each pixel unit is aimed at promoting

On the other hand, the Dice loss function improves the similarity of overall inference by calculating the loss value by obtaining the similarity between the inference probability (p _pred ) and the correct answer (p _true ) based on the Dice coefficient as shown in Equation 2. aims to do

At this time, pixels corresponding to the background class are excluded from the calculation of the loss value regardless of the type of the loss function, thereby preventing the neural network from being biased in accurately inferring the background area.

To evaluate semantic segmentation performance according to an embodiment of the present invention, MIoU (Mean Intersection over Union) according to Equation 4, which is widely used in the field of semantic segmentation, was used.

[Equation 4]

As shown in Equation 4, MIoU represents the average of the IoU values obtained for each image in the image data set consisting of all N images. It means a value obtained by dividing the number of identical data (TP) by the total number of inferred data (TP+FN+FP).

This means the rate at which the inference value matches the correct answer (GT) in which the class label is specified, and may represent the accuracy of object inference in pixel units.

In this case, as in the case of the loss function, the class corresponding to the background pixel is excluded from the calculation of the evaluation value, thereby preventing the bias problem in which the performance is high even though the neural network cannot accurately infer the foreground area.

ablation studies

The results of ablation experiments performed to determine details based on the performance evaluation according to the structure of the neural network and the configuration of the loss function to be used in the semantic segmentation method according to an embodiment of the present invention are described.

First, in the first experiment, in order to determine the optimal sensor fusion for the proposed method, the fusion method applied to the neural network is divided into data stage fusion, feature stage fusion, and inference stage fusion, respectively. performance was compared.

At this time, the neural network to which the data-level fusion is applied is a fusion of (H, W, 4) size by connecting the projected image (X _PCD ) and the RGB image (X _RGB ) of the point group through the connection layer as shown in Equation 5 below. Data is passed to the input of the encoder, and the output is passed back to the input of the decoder to create a 2D segmentation inference map (Y _seg ).

[Equation 5]

On the other hand, the neural network to which the inference step fusion is applied applies independent neural networks having different weights ((θ _en1 , θ _de1 ), (θ _en2 , θ _de2 )) for the two inputs, respectively, as shown in Equation 6 below. Then, two 2D segmentation inference maps are created, and one 2D segmentation inference map is created by averaging them.

[Equation 6]

In addition, in the case of feature step fusion, starting from the initial block constituting the neural network, fusion is performed in deeper neural network blocks one by one so that the number of various cases can be compared, and the learning progress is based on the "Focal loss" function performed with

[Table 1]

Meanwhile, Table 1 is a table showing sensor fusion application test results for the semantic segmentation method according to an embodiment of the present invention.

As shown in Table 1, the single modal method using only RGB images as input without using sensor fusion recorded an MIoU of 0.1012 lower than any other case using sensor fusion.

When comparing the cases where sensor fusion was applied, the case where inference stage fusion was applied recorded the lowest MioU with 0.1844, the data stage fusion recorded the highest value with 0.2045 and the feature stage fusion averaged 0.2095.

Among the feature stage fusion, the highest MIoU of 0.2152 was recorded when the outputs of the second convolution block constituting the encoder were fused together, and after that, the fusion showed a gradual decrease in performance.

Accordingly, it can be seen that using sensor fusion can produce better results than not, and at the same time, optimal performance can be achieved when using feature stage fusion.

In the second experiment, in order to investigate the performance change according to the number of transformer modules used to construct the deep feature vector in the encoder, the number of modules was increased from 6 to 9, 12, 18, and 24 in order, for a total of 5 cases. and observed a series of performance changes.

At this time, the configuration to which the feature stage fusion was applied, which showed the best performance in the first experiment, was used for the rest of the encoder configuration, and the same “Focal loss” function was used as the loss function for learning.

On the other hand, Table 2 is a table showing the experimental results of encoder detailed configuration for deep feature vector formation.

As shown in Table 2, the performance gradually increased as the number of modules increased, showing the highest performance of 0.2240 based on MIoU when using 9 modules, and the performance gradually decreased as the number of modules increased. showed

In particular, the fact that the encoder using 9 modules showed about 1% better performance than the existing method using 12 modules despite having a shallower neural network indicates that sensor fusion can create highly expressive deep features relatively quickly. It can be seen to mean that

In the third experiment, two loss functions widely used in the field of semantic segmentation, the "Focal loss" function in Equation 1 and the "Dice loss" function in Equation 2, are applied to train the neural network and evaluate the result to obtain a loss function We wanted to observe the effect of changes in the performance.

In addition, as shown in Equation 7 below, one of the two loss functions is used as the main loss function and the other is added as a regularizer, so that the two loss functions have their own advantages, the accuracy in pixel units. An experiment was also conducted to simultaneously enhance the enhancement and overall structural similarity.

[Equation 7]

On the other hand, Table 3 is a table showing the experimental results according to the configuration of the loss function used for learning.

[Table 3]

At this time, the neural network was made with the semantic segmentation method according to an embodiment of the present invention, which showed the best performance in the second experiment, and as a result of the experiment, as described in Table 3 below, the "Dice loss" function was used as a control term, The best results were obtained when using the "Focal loss" function.

This is based on the "Focal loss" function and focuses on improving per-pixel accuracy, but uses the "Dice loss" function together as a regulation term to apply a strong regulation value to areas that hinder the overall similarity improvement, and vice versa. This seems to be the result of allowing the model to learn more evenly by assigning a weak regulation value to the part that helps to increase the overall similarity.

10 is a diagram showing a 3D analysis result of 2D semantic segmentation according to an embodiment of the present invention.

Referring to FIG. 10, through the semantic segmentation method according to an embodiment of the present invention, performance of 31.94% based on 2D segmentation MIoU was obtained, and based on this, as shown in FIG. 10, 3D semantic segmentation information was reconstructed. can be interpreted

[Table 4]

Meanwhile, Table 4 is a table showing semantic segmentation results according to Examples and Comparative Examples of the present invention.

Accordingly, the segmentation MIoU based on the three-dimensional coordinates, which was impossible to obtain by the existing method, can be obtained as shown in Table 4, and the figure was 17.12%.

As described above, although preferred embodiments of the present invention have been disclosed in the present specification and drawings, it is in the technical field to which the present invention belongs that other modified examples based on the technical spirit of the present invention can be implemented in addition to the embodiments disclosed herein. It is self-evident to those skilled in the art. In addition, although specific terms have been used in the present specification and drawings, they are only used in a general sense to easily explain the technical content of the present invention and help understanding of the present invention, but are not intended to limit the scope of the present invention. Accordingly, the foregoing detailed description should not be construed as limiting in all respects and should be considered illustrative. The scope of the present invention should be selected by reasonable interpretation of the appended claims, and all changes within the equivalent scope of the present invention are included in the scope of the present invention.

100: learning data collection device 200: learning data generating device
300: artificial intelligence learning device
205: communication unit 210: input/output unit
215: pre-learning unit 220: data pre-processing unit
225: reasoning unit 230: 3-dimensional analysis unit
235: storage unit

Claims (10)

Receiving, by an apparatus for generating learning data, point cloud data obtained from a lidar and an image captured through a camera at the same time;
generating, by the learning data generating device, a 2D semantic segmentation feature map from the point cloud data and the image based on AI (Artificial Intelligence) that has been machine learning; and
interpreting, by the learning data generating device, 3D information from the generated feature map; It is characterized by including,
prior to the creation of
pre-learning the artificial intelligence; It is characterized in that it further comprises,
The pre-learning step is
Point cloud data obtained from the lidar included in the pre-stored data set, an image taken through a camera at the same time as the point cloud data, calibration information between the lidar and the camera, and a class label for each 3D point of the point cloud data Characterized in that the artificial intelligence is pre-learned based on the specified correct answer data,
The pre-learning step is
The point cloud data and the correct answer data are projected onto coordinates on a two-dimensional plane of the image, and pixels having a Manhattan distance between a pixel to be projected and a Manhattan distance of the image within a preset value have the same class. , 3D interpretation method of semantic segmentation.
The method of claim 1, wherein the pre-learning step
The artificial intelligence is pre-learned through at least one of a plurality of loss functions, and one of the plurality of loss functions is added as a regularizer.
The method of claim 2, wherein the pre-learning step
A three-dimensional interpretation method of semantic segmentation, characterized in that the artificial intelligence is pre-learned through a loss function represented by Equation 1 below.
[Equation 1]

(Here, p _pred is the inference probability based on cross entropy, and p _true means the correct answer.)
The method of claim 2, wherein the pre-learning step
A three-dimensional interpretation method of semantic segmentation, characterized in that the artificial intelligence is pre-learned through a loss function represented by Equation 2 below.
[Equation 2]

(Where p _pred is the inference probability based on the Dice coefficient, and p _true means the correct answer.)
The method of claim 2, wherein the pre-learning step
A three-dimensional interpretation method of semantic segmentation, characterized in that pixels corresponding to the background class are excluded from the calculation of the loss value regardless of the type of the loss function.
Receiving, by an apparatus for generating learning data, point cloud data obtained from a lidar and an image captured through a camera at the same time;
generating, by the learning data generating device, a 2D semantic segmentation feature map from the point cloud data and the image based on AI (Artificial Intelligence) that has been machine learning; and
interpreting, by the learning data generating device, 3D information from the generated feature map; It is characterized by including,
The interpretation step is
A 3-dimensional analysis method of semantic segmentation, characterized in that preparing a pair of 3-dimensional coordinates of the point cloud data and a coordinate pair on a 2-dimensional plane of an image projected with the point cloud data.
The method of claim 6, wherein the interpreting step
3D analysis of semantic segmentation, characterized by applying a max pooling operation using a 3*3 kernel centered on the coordinates for each 2D plane coordinate for the generated feature map. method.
The method of claim 7, wherein the interpreting step
A three-dimensional interpretation method of semantic segmentation, characterized in that obtaining a vector obtained by estimating a segment for each coordinate through the maximum pooling operation.
memory;
transceiver; and
In combination with a computing device configured to include a processor for processing instructions resident in the memory,
Receiving, by the processor, point cloud data obtained from a lidar and an image captured through a camera at the same time;
generating a 2-dimensional semantic segmentation feature map from the point cloud data and the image based on artificial intelligence (AI) that has been machine-learned by the processor; and
interpreting 3D information from the generated feature map by the processor; Execute, including
prior to the creation of
pre-learning the artificial intelligence; It is characterized in that it further comprises,
The pre-learning step is
Point cloud data obtained from the lidar included in the pre-stored data set, an image taken through a camera at the same time as the point cloud data, calibration information between the lidar and the camera, and a class label for each 3D point of the point cloud data Characterized in that the artificial intelligence is pre-learned based on the specified correct answer data,
The pre-learning step is
The point cloud data and the correct answer data are projected onto coordinates on a two-dimensional plane of the image, and pixels having a Manhattan distance between a pixel to be projected and a Manhattan distance of the image within a preset value have the same class. , A computer program recorded on a recording medium.
memory;
transceiver; and
In combination with a computing device configured to include a processor for processing instructions resident in the memory,
Receiving, by the processor, point cloud data obtained from a lidar and an image captured through a camera at the same time;
generating a 2-dimensional semantic segmentation feature map from the point cloud data and the image based on artificial intelligence (AI) that has been machine-learned by the processor; and
interpreting 3D information from the generated feature map by the processor; Execute, including
The interpretation step is
A computer program recorded on a recording medium, characterized in that for preparing a three-dimensional coordinate pair of the point cloud data and a two-dimensional plane coordinate pair of an image projected with the point cloud data.

Images