KR102303727B1 - System for noise removal of point clouds based on ai - Google Patents

Abstract

The present invention relates to an artificial intelligence-based point cloud noise removing system. According to an embodiment of the present invention, the artificial intelligence-based point cloud noise removing system includes: an input unit for receiving a photographed image; a memory for storing a program which removes noise based on artificial intelligence for the photographed image; and a processor for executing the program. The processor distinguishes outliers and near-field noises based on the 3D point cloud, and removes outliers and near-field noises by using a plurality of noise filters.

Description

AI-based point cloud noise removal system {SYSTEM FOR NOISE REMOVAL OF POINT CLOUDS BASED ON AI}

The present invention relates to an artificial intelligence-based point cloud noise removal system.

Various techniques have been proposed to remove the noise of the 3D point cloud.

According to the prior art, even if the filter function for removing noise is used, the noise is not completely removed, so it is inconvenient for the user to manually remove the noise for a long time.

The present invention has been proposed to solve the above-mentioned problems, and it is an object of the present invention to provide an artificial intelligence-based point cloud noise removal system that can dramatically reduce manpower input time and secure a high noise removal rate through an automated process. .

An artificial intelligence-based point cloud noise removal system according to an embodiment of the present invention includes an input unit for receiving a captured image, a memory in which a program for removing noise based on artificial intelligence for a captured image is stored, and a processor for executing the program , the processor distinguishes outliers and near-field noises based on the 3D point cloud, and removes outliers and near-field noises using a plurality of noise filters.

According to an embodiment of the present invention, it is possible to simultaneously process outliers and C-noise (close noise) of large civil structures (roads, bridges, buildings) regardless of the size of image data, and It has the effect of improving accuracy and performance.

Effects of the present invention are not limited to those mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the following description.

1 shows an artificial intelligence-based point cloud noise removal system according to an embodiment of the present invention.
2 shows an outlier and C-noise according to an embodiment of the present invention.
3 shows a deep learning architecture structure of an artificial intelligence-based point cloud noise removal system according to an embodiment of the present invention.
4 shows a network and module configuration of an artificial intelligence-based point cloud noise removal system according to an embodiment of the present invention.
5 shows a 3D-based noise filter program configuration of an artificial intelligence-based point cloud noise removal system according to an embodiment of the present invention.
6 shows a UAV photographed dataset image according to an embodiment of the present invention.
7 shows a test application dataset image according to an embodiment of the present invention.
8A and 8B show images before/after noise removal using an outlier detector according to an embodiment of the present invention.
9A and 9B show noise-removed images using a C-noise detector and a BL filter according to an embodiment of the present invention.

The above and other objects, advantages and features of the present invention, and a method of achieving them will become apparent with reference to the embodiments described below in detail in conjunction with the accompanying drawings.

However, the present invention is not limited to the embodiments disclosed below, but can be implemented in various different forms, and only the following examples are provided to those of ordinary skill in the art to which the present invention pertains to the purpose of the invention, It is only provided to easily inform the configuration and effect, and the scope of the present invention is defined by the description of the claims.

On the other hand, the terms used herein are for the purpose of describing the embodiments and are not intended to limit the present invention. As used herein, the singular also includes the plural unless specifically stated otherwise in the phrase. As used herein, "comprises" and/or "comprising" means that a referenced element, step, operation and/or element is the presence of one or more other elements, steps, operations and/or elements. or added.

Hereinafter, in order to facilitate the understanding of those skilled in the art, the background to which the present invention is proposed will be first described, and embodiments of the present invention will be described.

Various techniques have been proposed to remove the noise of the 3D point cloud.

Due to the development of LiDAR technology, 3D imaging cost is reduced, and as stereo imaging technology advances, research for noise reduction is actively being conducted.

However, according to the prior art, the problem that parameter tuning is required for a specific 3D image, the problem of low accuracy, the problem of damaging the original surface due to the excessive degree of smoothing, the outlier There is a problem in that the removal ability of the sample) is lowered.

Regarding the types of noise filters, Outlier Removal (1995~2015) is an algorithm based on a classical and statistical method. It requires parameter tuning for a specific 3D image, and it takes a lot of time and effort for parameter tuning. However, there is a problem in that it is difficult to expect high accuracy for various 3D images in general.

Local Surface Fitting (2003~2011) is a method of moving the point cloud, which is judged as noise, to an object (surface, surface), and the moving least squares method (MLS) is commonly used. Robust Jet Fitting is also widely used.

Non Local Mean (2010~2018) is a method of finding parts with similar shapes and integrating them into a consistent and noise-free point cloud.

Denoising in Image (2005~2010) is a method of removing 3D noise generating factors included in 2D images in advance.

When using the filter function for noise removal provided by default in the general-purpose program for 3D registration of drone footage, the noise is not completely removed (removal rate is less than about 50%), so the user can manually reduce the noise for a long time Hours to days) There is an inconvenience that needs to be removed.

The present invention has been proposed to solve the above problems, and proposes a noise removal (filter) system based on image processing and artificial intelligence. It provides an artificial intelligence-based noise removal system that dramatically reduces input time.

According to an embodiment of the present invention, it is possible to remove noise from a point cloud output from a general-purpose program for 3D matching using a deep neural network, and to automate noise removal according to a user's setting input.

According to an embodiment of the present invention, by applying a deep learning-based algorithm dedicated to 3D point cloud, a noise removal system suitable for large civil structures such as roads, bridges, and buildings is proposed, and LAS files are processed simultaneously with outliers and C-noise. can be processed without conversion, it is possible to remove noise with high accuracy regardless of the size of image data, and it is possible to continuously improve performance.

1 shows an artificial intelligence-based point cloud noise removal system according to an embodiment of the present invention.

An artificial intelligence-based point cloud noise removal system according to an embodiment of the present invention includes an input unit 110 for receiving a captured image, a memory 120 storing a program for removing noise based on artificial intelligence for a captured image, and a program. It includes a processor 130 that executes, but the processor 130 classifies outliers and near-field noises based on the 3D point cloud, and removes outliers and near-field noises using a plurality of noise filters.

The processor 130 divides the captured image, which is the 3D model, and down-samples it.

The processor 130 removes outliers and near-field noise using a spatial transformation network, a fully connected neural network, and a point feature vector.

The processor 130 removes near-field noise by using a BL filter.

2 illustrates an outlier and noise according to an embodiment of the present invention.

Noise is anything other than points constituting a surface in the 3D point cloud, and noise to be removed is defined by dividing it into outlier and C-noise.

An outlier is defined as a point to be removed that is more than a certain distance away from the surface enough to be clearly distinguished with the naked eye, and the distance is generally 3 to 5 times or more of GSD (Ground Spatial Distance, reference pixel interval when shooting). The algorithm applies a method to remove when this threshold value (3~5 GSD) is exceeded.

C-noise is defined as a point to be removed that is located close to the surface, and a removal method that calculates a vector and moves it closer to the surface is applied.

3 shows a deep learning architecture structure of an artificial intelligence-based noise removal system according to an embodiment of the present invention.

As a spatial transform network (STN), it is a spatial transform network, a fully connected neural network (FNN) is a fully connected neural network, and C1 to Cn are point feature vectors.

STN rotates the point cloud so that the network performance is not affected by the rotation angle (direction) of the point cloud inside the spherical mask.

FNN consists of three parts: first, a part that extracts the features of the point cloud, second, a part that integrates each calculated feature into an invariant feature vector, and third, a regression part that evaluates the distance and direction from the feature vector do.

The point cloud is input to the outlier detection network which is a first network, and the first network is composed of a first STN, a first FNN, C1 to Cn, a second FNN, and a third FNN.

In order to accurately detect outliers, the L1 distance between GT (ground truth) and inference results was applied as a loss function for outliers that was experimentally confirmed to have superior performance compared to cross-entropy loss.

In addition, the outlier detection network takes into account the characteristics of each member of huge civil engineering structures (road pavement, bridge surface, building surface, unpaved surface, forest grass, steel structure, etc.) and the degree of light reflection by material to effectively remove noise in a specific size. Creates a spherical mask (1/10 to 1/50 of the total model size) of , and detects outliers by considering only the points inside the spherical mask, correlation between points, and vectors of points.

This spherical mask moves at regular intervals and evenly removes outliers throughout the model.

The point cloud from which the outlier is removed is input to the second network, the C-noise detection network, and the second network is composed of the second STN, the fourth FNN, C1 to Cn, the fifth FNN, and the sixth FNN. Output the removed point cloud.

In order to accurately detect c-noise, the L2 Distance between GT (ground truth) and the inferencing result was applied as a loss function dedicated to c-noise.

4 shows a network and module configuration of an artificial intelligence-based noise removal system according to an embodiment of the present invention.

According to an embodiment of the present invention, when designing a filter for removing C-Noise and outliers of 3D point cloud, the noise removal accuracy and noise removal speed must be considered first, and the scale is huge and the road, Considering the wide variety of types such as bridges, buildings, islands, and residential land, a plurality of noise filters are used at the same time.

The first noise filter is an outlier detector, which removes far-field noise, and takes a long time to learn and calculate based on a neural network.

The second noise filter is a C-noise detector, which moves the near-field noise toward the surface, and takes a long time to learn and calculate based on a neural network.

The third noise filter is a BL filter, which moves the near-field noise in the direction of the surface, does not require learning based on a rule base, and has a very fast calculation time. However, the BL filter has the disadvantage of moving excessively to the point corresponding to the surface of the surface in order to move the near-field noise of various types of civil structures with high accuracy, so the surface retention rate is relatively inferior to that of the C-noise detector.

According to an embodiment of the present invention, Training Tool (developer version), 3D Viewer, 3D Pre-Processor (segmentation of 3D model), 3D Edit (deletion of point cloud) functions for intuitive and efficient training of neural networks are provided. Included in program design.

5 shows a 3D-based noise filter program configuration of an artificial intelligence-based noise removal system according to an embodiment of the present invention.

A full point cloud (full point cloud) is input to the 3D pre-processor as input data.

The 3D pre-processor splits a massive 3D model and performs downsampling processing.

The reason for dividing the 3D model is that the scale is huge due to the nature of civil structures, and when tens of millions to hundreds of millions of point clouds are input into the network at once, it takes a lot of time to calculate the optimal spherical mask size, and a specific spherical mask size suitable for the huge 3D model. It is inefficient because it cannot be determined, and it takes a lot of time to move (scroll) the mask, so overall efficiency and performance are lowered compared to the split case.

On the other hand, if the size of the mask is determined and noise is removed after splitting to an appropriate size, the time to set the optimal mask size and mask movement time are reduced, and the noise removal performance is improved with the optimal mask size for each split area.

The first noise filter as an outlier detector removes far-field noise, the second and third noise filters remove near-field noise, and the 3D viewer supports the 3D view of the noise removal result.

According to an embodiment of the present invention, parameters and hyperparameters to be considered in training and detection of a neural network are defined.

The main parameters of training are epoch, learning rate, save interval, mini batch size, learning rate, momentum, point per mask, and the main parameters of inference (inferencing) are batch (dataset) size, mask size, worker (CPU vs. GPU thread).

According to an embodiment of the present invention, five datasets that are easy to create a correct answer dataset (Ground Truth: GT) and check performance are selected from among the six six point cloud datasets as shown in [Table 1], and test data three were formed.

[Table 1]

Since the point cloud dataset has 10 to 57 million point clouds and it takes more than several hours to calculate, the data set is divided into 1/10 to 1/20 for ease of testing, and after selecting a representative section, test data It was used as a set (at this time, trees are meaningless among various elements constituting the data set, so they are excluded from the calculation of accuracy).

Regarding the test criteria, the test was conducted in a way that can confirm the noise removal accuracy, and the accuracy was calculated by calculating Accuracy, which means general accuracy.

In addition, the precision score, which is standard used for object recognition in the field of artificial intelligence image processing, was measured as a reference.

(노이즈 제거 정확도(%))

(Noise Rejection Accuracy (%))

(노이즈 제거 성능 참고치(%))

(Reference value of noise removal performance (%))

6 shows a UAV photographing dataset image according to an embodiment of the present invention, and FIG. 7 shows a test application dataset image according to an embodiment of the present invention.

Hereinafter, an aggregate table of test application datasets is shown as shown in [Table 2], which is a result of the outlier detector and the above-described c-noise detector and BL filter are not passed.

[Table 2]

[Table 3] shows the factors for calculating accuracy and precision.

[Table 3]

8A and 8B show images before/after noise removal using an outlier detector according to an embodiment of the present invention. This is a result of the outlier detector, and the above-mentioned c-noise detector and BL filter are not passed through.

9A and 9B show noise-removed images using a C-noise detector and a BL filter according to an embodiment of the present invention.

Meanwhile, the AI-based noise removal method according to an embodiment of the present invention may be implemented in a computer system or recorded in a recording medium. The computer system may include at least one processor, a memory, a user input device, a data communication bus, a user output device, and storage. Each of the above-described components performs data communication through a data communication bus.

The computer system may further include a network interface coupled to the network. The processor may be a central processing unit (CPU) or a semiconductor device that processes instructions stored in a memory and/or storage.

The memory and storage may include various types of volatile or non-volatile storage media. For example, memory may include ROM and RAM.

Accordingly, the AI-based noise removal method according to an embodiment of the present invention may be implemented as a computer-executable method. When the AI-based noise removal method according to an embodiment of the present invention is performed in a computer device, computer readable instructions may perform the AI-based noise removal method according to the present invention.

On the other hand, the artificial intelligence-based noise removal method according to the present invention described above may be implemented as a computer-readable code on a computer-readable recording medium. The computer-readable recording medium includes any type of recording medium in which data that can be read by a computer system is stored. For example, there may be a read only memory (ROM), a random access memory (RAM), a magnetic tape, a magnetic disk, a flash memory, an optical data storage device, and the like. In addition, the computer-readable recording medium may be stored and executed as code readable in a distributed manner by being distributed in a computer system connected through a computer communication network.

So far, the embodiments of the present invention have been mainly looked at. Those of ordinary skill in the art to which the present invention pertains will understand that the present invention can be implemented in a modified form without departing from the essential characteristics of the present invention. Therefore, the disclosed embodiments are to be considered in an illustrative rather than a restrictive sense. The scope of the present invention is indicated in the claims rather than the foregoing description, and all differences within the scope equivalent thereto should be construed as being included in the present invention.

Claims (4)

an input unit for receiving a photographed image;
a memory in which a program for removing noise based on artificial intelligence for the captured image is stored; and
A processor for executing the program,
The processor separates outliers and near-field noises based on the 3D point cloud, and removes the outliers and near-field noises using a plurality of noise filters,
The processor rotates the 3D point cloud, detects the outlier, and integrates the feature of the 3D point cloud into an invariant feature vector, a spatial transformation network that detects the outlier and restores the rotation angle of the 3D point cloud, and determines the distance and direction from the feature vector. The outlier and near-field noise are removed by using the fully-connected neural network and point feature vector to be evaluated, and the characteristics of each member of the civil structure including building structures, bridges, roads, islands, and residential land, and the degree of light reflection by materials are considered. Detect the outlier using an outlier detection network that generates a spherical mask of a preset size;
The processor performs segmentation on the point cloud in consideration of the scale of the civil structure and the parameters of the deep learning network, and performs downsampling processing on the point cloud
A point cloud noise removal system based on artificial intelligence.
According to claim 1,
The processor divides the captured image, which is a 3D model, and down-samples
A point cloud noise removal system based on artificial intelligence.
delete According to claim 1,
The processor is to perform the removal of the near-field noise using a BL filter
A point cloud noise removal system based on artificial intelligence.

Images