KR102433632B1 - Apparatus And Method For Inpainting Depth of LiDAR - Google Patents

Abstract

The present invention is implemented with an input data acquisition unit that acquires input data by converting it into a two-dimensional image in a predetermined manner when a three-dimensional point cloud image obtained using lidar is applied, and a pre-learned artificial neural network, By repeating the process of extracting the feature map by encoding according to the learned method, the encoding feature map, which is the final encoded feature map, among a plurality of feature maps obtained is decoded according to the learned method to obtain a binary mask representing the dynamic object region, , according to the probabilistic structural similarity, an interest map is obtained by replacing a region corresponding to a binary mask in a corresponding feature map among a plurality of feature maps with an area of a previously obtained filled feature map, and the obtained interest map is pre-learned Including a background filling network that repeatedly acquires a filling feature map by decoding with a 3D map acquisition unit that receives the final acquired filling feature map as a filling image and converts it into a three-dimensional image in a predetermined way, the dynamic object area is natural It is possible to provide a lidar depth image filling apparatus and method that can be replaced with a background and obtain a high-quality 3D map usable for autonomous driving.

Description

Apparatus And Method For Inpainting Depth of LiDAR

The present invention relates to a lidar depth image filling apparatus and method, and to a lidar depth image filling apparatus and method.

Various sensors, such as visual cameras, radar, LiDAR, and GNSS, are attached to autonomous vehicles and are being added to vehicles. have.

Creating an accurate 3D map of the driving scene is essential for fully autonomous driving. However, it is difficult to create static and accurate 3D maps due to moving objects or changing environments. A typical method for acquiring a 3D high-density (HD) map is to acquire a 3D point cloud image composed of a 3D point cloud while moving in the same space several times using a high-performance 3D measuring device. Since the acquired 3D image often includes various dynamic objects (vehicles, pedestrians, etc.) together, in order to create a static 3D map, it is necessary to remove surrounding dynamic objects. In addition, the blank space must be filled by mapping information when the dynamic object does not exist to the area where the dynamic object is removed.

However, in the case of an area where dynamic objects are frequently located, even if a large number of repeated images are taken, dynamic objects are repeatedly photographed and thus dynamic objects cannot be removed normally, and even if removed, empty spaces cannot be filled. do.

1 shows an example of a 3D point cloud image obtained by a conventional method.

As indicated by a square box or an ellipse in FIGS. 1A to 1C , in the case of a 3D point cloud image obtained by the conventional method, even if it is repeatedly photographed several times, a road in the city center where a large number of vehicles move frequently In an area such as a dynamic object, different dynamic objects are continuously present at a corresponding position and are repeatedly and cumulatively detected, so that they are not recognized as a dynamic object and thus not removed. Therefore, even in the 3D map, an error occurs when a dynamic object is recognized as a background and included.

2 shows an example of a feature point extracted from a 3D point cloud image using an existing artificial neural network.

In FIG. 2(a), a 3D point cloud image is shown, and (b) and (c) are the feature points extracted from the 3D point cloud image of (a) using a conventional artificial neural network, respectively, and a manual based Intrinsic shape signatures (ISS) method. represents the extracted feature points. Referring to (b) and (c) of FIG. 2 , it can be seen that even when a feature point is extracted using an artificial neural network or manually, it is not easy to distinguish between a background and a dynamic object, so that the feature point is detected from the dynamic object. Therefore, when matching 3D point clouds using such a technique, an error occurs and there is a limitation in that the dynamic object cannot be accurately removed.

In addition, the method of removing a dynamic object based on manual operation and matching it with background information is a time-consuming task, and there is a hassle of acquiring an image again if there is a change in a building or format in the city center.

Korean Patent Publication No. 10-2018-0134563 (published on December 19, 2018)

An object of the present invention is to accurately detect a dynamic object region in a 3D point cloud image obtained using lidar, and fill the detected dynamic object region with a background to obtain a 3D map from which the dynamic object is removed. To provide an apparatus and method.

Another object of the present invention is to provide an apparatus and method for filling a LIDAR depth image that can obtain a natural 3D map by filling the detected dynamic object region with the background without any heterogeneity by filling the background by reflecting the structural characteristics of the surroundings. is to provide

In the lidar depth image filling apparatus according to an embodiment of the present invention for achieving the above object, when a three-dimensional point cloud image obtained using lidar is applied, it is converted into a two-dimensional image in a predetermined manner to obtain input data an input data acquisition unit; It is implemented as a pre-learned artificial neural network, and the encoding feature map, which is the final encoded feature map among a plurality of feature maps obtained by repeating the process of extracting the feature map by encoding the input data according to the learned method, is applied to the learned method. decoding to obtain a binary mask representing the dynamic object region, and replacing the region corresponding to the binary mask in the corresponding feature map among the plurality of feature maps with the region of the previously obtained filled feature map according to the probabilistic structural similarity a background filling network that acquires an interest map and repeatedly acquires a filling feature map by decoding the acquired interest map in a pre-learned manner; and a 3D map acquisition unit that receives the finally obtained filling feature map as a filling image and converts it into a three-dimensional image in a predetermined manner.

The background filling network may include an encoding network including a plurality of convolutional layers that receive input data or a feature map output from a previously arranged convolutional layer, and encode and output the feature map; an object detection network that receives the encoding feature map and decodes it according to a learned method to obtain the binary mask indicating a region in which a dynamic object designated by learning is located in the input data; a filling network for obtaining a filling feature map by decoding the encoding feature map or the previously acquired interest map in a learned manner; and a fusion feature map is obtained by replacing a region corresponding to the binary mask in a corresponding feature map among the plurality of feature maps with a region of a previously obtained filled feature map, and between the corresponding feature map and the acquired fusion feature map and a fusion block unit including a plurality of fusion blocks for calculating the probabilistic structural similarity and weighting the fusion feature map in a predetermined manner to obtain the interest map.

The object detection network may obtain a plurality of scale binary masks by scaling the binary mask to a size corresponding to each of a plurality of feature maps obtained from the encoding network.

Each of the plurality of fusion blocks receives a corresponding feature map from among the plurality of feature maps, a corresponding scaled binary mask from among a plurality of scaled binary masks, and a filling feature map previously obtained from the filling network, and in the corresponding feature map a fusion unit configured to obtain a fusion feature map by replacing an area designated by the corresponding scaled binary mask with an area of the previously obtained filling feature map; a structural similarity calculation unit that determines the structural similarity between a corresponding feature map and the fusion feature map according to a structural similarity index measurement (SSIM) structural similarity evaluation method, and calculates a stochastic structural similarity by additionally reflecting a stochastic distribution; and an interest map acquisition unit configured to acquire the interest map by weighting the calculated probabilistic structural similarity to the fusion feature map.

The fusion unit calculates the fusion feature map (F _f ⁿ ) by Equation

는 스케일 이진 마스크이며, F_i ⁿ 는 채움 특징맵이다.)에 따라 획득할 수 있다.(where F _e ⁿ is the feature map,

is a scaled binary mask, and F _i ⁿ is a filling feature map).

The structural similarity calculator calculates the structural similarity (S) by the equation

(where k is the feature map (F _e ⁿ ), q is the fusion feature map (F _f ⁿ ), σ is the variance, and c ₃ is a predetermined constant.)

The probabilistic structural similarity is calculated by the equation

can be calculated according to

The interest map obtaining unit calculates the interest map (C ⁿ ) by the equation

(Where λ _atn is the weight of interest.)

can be obtained according to

The input data acquisition unit acquires five two-dimensional distance images according to a depth value, an x-axis value, a y-axis value, a z-axis value, and an intensity value from a 3D point cloud image, and combines the obtained five two-dimensional distance images in the channel axis direction. to obtain the input data.

In the lidar depth image filling method according to another embodiment of the present invention for achieving the above object, when a three-dimensional point cloud image obtained using lidar is applied, it is converted into a two-dimensional image in a predetermined manner to obtain input data to do; obtaining a plurality of feature maps by repeating the process of extracting feature maps by encoding the input data according to the learned method using a pre-learned artificial neural network; using a pre-trained artificial neural network to decode an encoded feature map, which is a final encoded feature map among the plurality of feature maps, according to a learned method to obtain a binary mask representing a dynamic object region; Using a pre-trained artificial neural network, the region of interest is obtained by replacing the region corresponding to the binary mask in the corresponding feature map among the plurality of feature maps according to the degree of probabilistic structural similarity with the region of the previously obtained filled feature map, , decoding the obtained interest map in a pre-learned manner to obtain a filling feature map; and receiving the finally obtained filling feature map as a filling image, and converting it into a three-dimensional image in a predetermined manner.

Accordingly, the apparatus and method for filling a LIDAR depth image according to an embodiment of the present invention accurately detect a dynamic object area from a 3D point cloud image obtained using LIDAR, and the structural characteristics around the detected dynamic object area are probability distribution By filling in the reflected background based on , the dynamic object area is replaced with a natural background to obtain a high-quality 3D map usable for autonomous driving. In addition, since learning can be performed with a generative adversarial neural network technique, it is possible not only to learn without a learning data set, but also to provide a 3D map that can be used as a learning data set for learning other artificial neural networks.

1 shows an example of a 3D point cloud image obtained by a conventional method.
2 shows an example of a feature point extracted from a 3D point cloud image using an existing artificial neural network.
3 shows a schematic structure of a lidar depth image filling apparatus according to an embodiment of the present invention.
FIG. 4 shows an example of input data obtained by the input data obtaining unit of FIG. 3 .
FIG. 5 is a diagram for explaining an example of a detailed structure of the object reward network of FIG. 3 .
6 shows an example of the configuration of the fusion block of FIG. 5 .
7 shows an example of a 3D map image obtained by using the LIDAR depth image filling apparatus according to the present embodiment.
8 is a diagram illustrating a lidar depth image filling method according to an embodiment of the present invention.

In order to fully understand the present invention, the operational advantages of the present invention, and the objects achieved by the practice of the present invention, reference should be made to the accompanying drawings illustrating preferred embodiments of the present invention and the contents described in the accompanying drawings.

Hereinafter, the present invention will be described in detail by describing preferred embodiments of the present invention with reference to the accompanying drawings. However, the present invention may be embodied in several different forms, and is not limited to the described embodiments. In addition, in order to clearly explain the present invention, parts irrelevant to the description are omitted, and the same reference numerals in the drawings indicate the same members.

Throughout the specification, when a part "includes" a certain component, it does not exclude other components unless otherwise stated, meaning that other components may be further included. In addition, terms such as "... unit", "... group", "module", and "block" described in the specification mean a unit that processes at least one function or operation, which is hardware, software, or hardware. and a combination of software.

3 shows a schematic structure of an apparatus for filling a lidar depth image according to an embodiment of the present invention, and FIG. 4 shows an example of input data obtained by the input data acquisition unit of FIG. 3 .

Referring to FIG. 3 , the apparatus for filling a lidar depth image according to the present embodiment includes a point cloud image acquisition unit 100 , an input data acquisition unit 200 , a background filling network 300 , and a 3D map acquisition unit 400 . can do.

The point cloud image acquisition unit 100 acquires the acquired 3D point cloud image without considering the dynamic object. The point cloud image acquisition unit 100 may be implemented as a lidar to directly acquire a 3D point cloud image, a communication module that receives the 3D point cloud image obtained from the lidar, or a storage module in which a pre-acquired 3D point cloud image is stored. have. In addition, the lidar may acquire a 3D point cloud image by scanning the surroundings while rotating a predetermined rotation range in a predetermined height range. For example, if the lidar samples 2,048 times while rotating 360 degrees along each of 64 scan lines in a predetermined height range, the lidar may acquire a 3D point cloud image composed of 2,048 × 64 sampling pixels. In this case, the obtained 3D point cloud image may include position information, depth information, and intensity information for each point in x, y, and z-axis directions.

The input data acquisition unit 200 converts the 3D point cloud image acquired by the point cloud image acquisition unit 100 into a two-dimensional range image. The input data acquisition unit 100 converts each point of the 3D point cloud image into a vertical angle (θ) and a horizontal angle (φ) of the spherical coordinate system according to Equations 1 and 2 and projects it in a two-dimensional space, two-dimensional A street image can be obtained.

In this case, the input data acquisition unit 200 may obtain a normalized distance image corresponding to the lidar scan standard. That is, the two-dimensional distance image may be acquired with a size of 2,048 × 64 pixels from the lidar sensor that scans 360 degrees in all directions. However, the input data acquisition unit 200 may obtain the input data by extracting only a predetermined angular range (here, 90 degrees as an example) even if the 3D point cloud image is obtained from the lidar for 360 degrees omnidirectional. In this case, the input data may have, for example, 512×64 pixels.

In addition, as shown in FIG. 4 , the input data acquisition unit 200 divides the two-dimensional distance image into five according to a depth value, an x-axis value, a y-axis value, a z-axis value, and an intensity value. can be obtained as data. In FIG. 4, (a) to (e) represent two-dimensional distance images with respect to a depth value, an x-axis value, a y-axis value, a z-axis value, and an intensity value, respectively. In addition, the input data acquisition unit 200 may acquire input data by combining five distance images in a channel axis direction. Accordingly, the input data may be obtained, for example, with a size of 512×64×5. And the obtained input data is applied to the background filling network (300).

The background filling network 300 implemented as an artificial neural network receives input data, detects a dynamic object from the applied input data according to a pre-learned method, and learns the detected dynamic object region based on the structural similarity with the background. The background fill image is obtained by erasing it according to the specified method and filling it with the surrounding background.

The background filling network 300 may include an encoding network 310 , an object detection network 320 , a fusion block unit 330 , and a filling network 340 . Here, the encoding network 310 , the object detection network 320 , and the filling network 340 may be implemented as pre-trained artificial neural networks, respectively.

FIG. 5 is a diagram for explaining an example of a detailed structure of the object reward network of FIG. 3 .

When the background filling network 300 of FIG. 1 is described with reference to FIG. 5, first, the encoding network 310 receives input data and estimates the characteristics of the applied input data according to the learned method to obtain an encoding feature map. do. As shown in FIG. 5 , the encoding network 310 may be configured to include a plurality of sequentially connected convolutional layers. Although FIG. 5 illustrates an example in which the encoding network 310 includes five convolutional layers, the number and size of each convolutional layer may be variously adjusted. However, as shown in FIG. 5 , in the plurality of convolutional layers, the first convolution layer is composed of a convolution layer having a size of 5 × 5, while the remaining convolution layers are composed of a convolution layer having a size of 3 × 3. . That is, it is preferable that the remaining convolutional layers excluding the first convolutional layer among the plurality of convolutional layers of the encoding network 310 consist of convolutional layers having the same size.

Each of the five convolutional layers connected sequentially receives the input data or a feature map (F _e ^n-1 ) output from the previous convolution layer (where n is a natural number) and estimates feature points to obtain a feature map (F _e ⁿ ) Then, the acquired feature map F _e ⁿ is transferred to the next convolution layer, and finally, the feature map output from the convolution layer is called an encoding feature map F _e . Accordingly, the encoding network 310 accurately extracts the features of the input network through a plurality of convolutional layers to obtain the encoding feature map (F _e ).

The object detection network 320 receives the encoding feature map (F _e ) obtained from the encoding network 310 and estimates the dynamic object region from the applied feature map (F _e ) according to a pre-learned method to determine the dynamic object region. A binary mask (m) representing The object detection network 320 extracts a region estimated to contain the dynamic object from the applied feature map by learning the pattern for the dynamic object in advance. The extracted area is used to set an area for erasing a dynamic object from the feature map F _e ⁿ to which the fusion block unit 330 to be described later is applied.

The object detection network 320 may also be configured to include a plurality of sequentially connected convolutional layers, as shown in FIG. 5 , and the number of included convolutional layers is the number of convolutional layers included in the encoding network 310 . It can be variously adjusted according to the number and the number of types of dynamic objects to be detected. However, the two-dimensional size of the feature map F _e ⁿ output from each convolutional layer transmitted through a plurality of convolutional layers of the encoding network 310 and the object detection network 320 is gradually reduced. Accordingly, the object detection network 320 may further include a plurality of upsampling layers to increase the two-dimensional size of the feature map F _e ⁿ that is gradually reduced while passing through the plurality of convolutional layers. A plurality of upsampling layers are inserted after a predetermined number of convolutional layers in each of the plurality of convolutional layers, and the two-dimensional size of the feature map (F _e ⁿ ) applied from the previous layer according to a predetermined method or a learned method. expand and output. In this case, the plurality of upsampling layers may be expanded to a two-dimensional size by combining values in the channel axis direction of the applied feature map, for example, in a predetermined manner.

) 및 채움 네트워크(340)의 다수의 컨볼루션 레이어에서 출력되는 채움 특징맵(F_i ⁿ)을 인가받아, 기지정된 방식으로 결합하여 관심맵(attention map)(Cⁿ)을 획득한다.Meanwhile, the fusion block unit 330 includes a plurality of feature maps F e n output from a plurality of convolutional layers of the encoding network 310 and a binary mask (F _e ⁿ ) applied from the object detection network 320 .

) and a filling feature map (F _i ⁿ ) output from a plurality of convolutional layers of the filling network 340 are authorized, and combined in a predetermined manner to obtain an attention map (C ⁿ ).

As shown in FIG. 4 , the fusion block unit 330 may include a plurality of fusion blocks FB1 to FB3 , and the plurality of fusion blocks FB1 to FB3 are a plurality of convolutions of the filling network 340 , respectively. It can be placed between solution layers.

That is, although FIG. 3 shows the fusion block unit 330 and the filling network 340 separately for convenience of explanation, a plurality of fusion blocks FB1 to FB3 of the fusion block unit 330 are shown in FIG. 4 . As described above, it may be configured in combination with the filling network 340 .

Each of the plurality of fusion blocks FB1 to FB3 of the fusion block unit 330 is a binary mask (m ⁿ ) corresponding to a corresponding feature map (F _e ⁿ ) among the plurality of feature maps (F _e ¹ to F _e ³ ). and a plurality of filling feature maps F _i ¹ to F _i ³ , receiving and combining a corresponding filling feature map F _i ⁿ , to obtain a fusion feature map F _f ⁿ , and applying the applied feature map F _e ⁿ ) and the obtained fusion feature map (F _f ⁿ ) may obtain a corresponding interest map (C ⁿ ) according to the degree of structural similarity.

Here, the corresponding binary mask m ⁿ represents a scaled-down binary mask such that the binary mask m extracted from the object detection network 320 corresponds to the fusion block FBn. As described above, the binary mask m extracted from the object detection network 320 may be extracted to have a two-dimensional size extended compared to the encoding feature map F _e . Accordingly, the two-dimensional size does not match with the corresponding feature map F _e ⁿ applied to the fusion block FBn. This results in that, when the fusion block FBn masks the feature map F _e ⁿ using the binary mask m, an area other than the dynamic object area of the feature map F _e ⁿ is masked. may cause an error.

)를 획득할 수 있다. 그리고 퓨전 블록부(330)의 다수의 퓨전 블록(FBn) 각각은 다수의 스케일 이진 마스크(

) 중 대응하는 스케일 이진 마스크(mⁿ)를 인가받아 특징맵(F_e ⁿ)을 마스킹할 수 있다.Accordingly, the object detection network 320 scales down the finally obtained binary mask (m) to a size corresponding to each of the plurality of feature maps (F _e ⁿ ) to reduce the scale to a plurality of scale binary masks (

) can be obtained. In addition, each of the plurality of fusion blocks FBn of the fusion block unit 330 includes a plurality of scale binary masks (

) of the corresponding scale binary mask (m ⁿ ) may be applied to mask the feature map (F _e ⁿ ).

6 shows an example of the configuration of the fusion block of FIG. 5 .

Referring to FIG. 6 , each of the plurality of fusion blocks FB1 to FB3 may include a feature fusion unit 331 , a structural similarity calculator 332 , and an interest map obtainer 333 .

) 중 대응하는 스케일 이진 마스크(mⁿ)를 인가받으며, 채움 네트워크(340) 중 대응하는 컨볼루션 레이어로부터 채움 특징맵(F_i ⁿ)을 인가받는다. 그리고 인가된 특징맵(F_e ⁿ)과 스케일 이진 마스크(mⁿ) 및 채움 특징맵(F_i ⁿ)으로부터 수학식 3에 따라 퓨전 특징맵(F_f ⁿ)을 획득한다.In the n-th fusion block FBn, the feature fusion unit 331 receives the feature map F _e ⁿ from the corresponding convolutional layer of the encoding network 310 , and receives a plurality of scale binary masks from the object detection network 320 . (

), a corresponding scaled binary mask (m ⁿ ) is applied, and a filling feature map (F _i ⁿ ) is applied from a corresponding convolutional layer in the filling network 340 . Then, the fusion feature map F _f ⁿ is obtained according to Equation 3 from the applied feature map F _e ⁿ , the scale binary mask m ⁿ , and the filling feature map F _i ⁿ .

According to Equation 3, the feature fusion unit 331 maintains the applied feature map F _e ⁿ for a region that does not correspond to the scale binary mask m ⁿ , and fills the region corresponding to the scale binary mask. A fusion feature map F _f ⁿ is obtained by filling the corresponding region of the map F _i ⁿ . That is, the feature fusion unit 331 synthesizes the fusion feature map F _f ⁿ by replacing the area designated by the scale binary mask m ⁿ in the feature map F _e ⁿ with the filling feature map F _i ⁿ . do.

Then, the obtained fusion feature map F _f ⁿ is transmitted to the structure similarity calculating unit 332 and the interest map obtaining unit 333 .

The structural similarity calculator 332 receives the feature map F _e ⁿ applied to the feature fusion unit 331 and the fusion feature map F _f ⁿ obtained from the feature fusion unit 331 , and applies it from a structural point of view. The degree of similarity between the acquired feature map (F _e ⁿ ) and the fusion feature map (F _f ⁿ ) is determined.

The structural similarity calculator 332 may be implemented as a structural attention module (SAM), which is a structural component of structural similarity index measurement (SSIM) that is mainly used for image quality evaluation purposes. ) to determine the structural similarity between the feature map (F _e ⁿ ) and the fusion feature map (F _f ⁿ ).

SSIM is proposed to compare the similarity between two images (k, q) in terms of Luminance (l), Contrast (C), and Structure (S), as shown in Equation 4 It is a method that has been

(Where α, β, and γ are weights, respectively.)

In addition, the similarity of each of the brightness (l), contrast (C), and structure (S) in Equation 4 may be calculated according to Equation 5.

(Where k and q are images to be compared, μ is the mean, σ is the variance, and c ₁ , c ₂ and c ₃ are predefined constants.)

However, in this embodiment, the structural similarity calculator 332 evaluates the similarity between the two images by using only the structural element (s) among the three elements (l, c, s) of the SSIM of Equation 5 as described above. . Here, the two images in which the structural elements are compared are an input image and an output image of the feature fusion unit 331 , a feature map (k = F _e ⁿ ) and a fusion feature map (q = F _f ⁿ ). That is, the structural similarity calculator 332 determines the structural similarity between the input image and the output image of the feature fusion unit 331 .

However, in the present embodiment, the structural similarity calculator 332 additionally reflects a softmax function as in Equation 6 in order to additionally consider a stochastic distribution with respect to the structural similarity, so that the stochastic structural similarity (S ⁿ ) to determine

The interest map obtaining unit 333 is configured to obtain the interest map C ⁿ from the fusion feature map F _f ⁿ obtained by the feature fusion unit 331 and the probabilistic structure similarity S ⁿ obtained by the structure similarity calculating unit 332 . ) is obtained according to Equation 7.

(Where λ _atn is the weight of interest.)

According to Equation 7, the interest map acquisition unit 333 weights the stochastic structural similarity (S ⁿ ) proportional to the interest weight (λ _atn ) to the fusion feature map (F _f ⁿ ) to obtain the interest map (C ⁿ ). acquire

In this case, although not shown, the interest map acquisition unit 333 may expand and output the two-dimensional size of the interest map C ⁿ by further including a deconvolution layer.

That is, as the probabilistic structural similarity (S ⁿ ) is higher, the fusion feature map (F _f ⁿ ) obtained by fusion of the feature map (F _e ⁿ ) and the filling feature map (F _i ⁿ ) is emphasized and the interest map (C ⁿ ) to acquire

The filling network 340 is configured to include a plurality of convolutional layers, of which the first convolutional layer receives the encoding feature map (F _e ) and decodes it in a learned manner to obtain the initial filling feature map (F _i ¹ ). gain. In addition, each of the remaining convolution layers receives an interest map C ^n-1 from a previously disposed corresponding fusion block FBn-1 among a plurality of fusion blocks of the fusion block unit 330, and according to a pre-learned method By decoding the applied interest map (C ^n-1 ), the region in which the feature map (F _e ^n-1 ) and the filling feature map (F _i ^n-1 ) are fused in the interest map (C ^n-1 ) is smooth It transforms to change and outputs a filling feature map (F _i ⁿ ).

At this time, the filling network 340 is disposed between the first convolutional layer to which the encoding feature map F _e is applied among a plurality of convolution layers and the first fusion block FB1, and 2 of the filling feature map F _i ¹ ). By further including an upsampling layer that expands the dimension size to the size of the feature map F _e ¹ , the two-dimensional size of the initial filling feature map F _i ¹ may be adjusted and output.

)를 인가받아, 기지정된 방식으로 3차원으로 변환하여 3D 지도 이미지를 획득한다. 여기서 3D 지도 획득부(400)는 일 예로 입력 데이터 획득부(200)의 2차원 이미지 획득 방식에 대향하는 방식으로 3차원 변환하여 3D 지도 이미지를 획득할 수 있다.The 3D map acquisition unit 400 provides a filling image (F i n ) that is a filling feature map (F _i ⁿ ) that is finally output from the filling network 340 .

) to obtain a 3D map image by converting it into 3D in a predetermined manner. Here, the 3D map acquisition unit 400 may obtain a 3D map image by performing 3D transformation in a method opposite to the 2D image acquisition method of the input data acquisition unit 200, for example.

In order to use the LIDAR depth image filling apparatus described above, the background filling network 300 implemented as an artificial neural network must be learned in advance. Accordingly, the apparatus for filling the LIDAR depth image according to the present embodiment may further include a learning unit (not shown) coupled during learning of the background filling network 300 .

The learning unit may perform learning using learning data in which the dynamic object region is preset. The learner may first obtain a mask loss (L _m ) for the object detection network 320 of the background filling network 300 according to Equation (8).

(where m _gt represents the truth value for the dynamic object domain, |||||| ₁ is an L1-norm function, and λ _m ) is a mask loss weight.

In addition, the learning unit may obtain the filling loss (L _i ) for the fusion block unit 330 and the filling network 340 according to Equation (8).

는 채움 네트워크(340)에서 동적 객체가 배경으로 채워진 최종 채움 이미지이고, I_gt 는 배경 진리값을 나타낸다.)(here

is the final filling image in which the dynamic object is filled with the background in the filling network 340, and I _gt represents the background truth value.)

If the training data is prepared in large quantities, the learning unit calculates the total loss (L _total ) as the sum of the mask loss (L _m ) and the filling loss (L _i ) of Equations (8) and (9), and the calculated total loss (L _total ) ) can be backpropagated to train the background filling network 300 . However, in the case of training data in which a dynamic object is specified in a 3D image containing a dynamic object, it is relatively easy to collect, whereas it is realistic to acquire a large amount of training data set consisting of a 3D image filled with the background with the dynamic object removed. Not easy.

Accordingly, the learning unit of the present embodiment may apply a generative adversarial network (GAN) technique to enable learning without a training data set.

)이 리얼 이미지인지 페이크 이미지인지 판별하도록 학습되는 반면, 생성자인 배경 채움 네트워크(300)는 획득되는 채움 이미지(

)에 대해 판별자가 리얼 이미지로 인식하도록 학습된다.In order to apply the GAN technique, the learning unit uses the background filling network 300 as a generator and additionally includes a separate discriminator. The discriminator is a filling image (

) is learned to determine whether it is a real image or a fake image, while the background filling network 300, which is a generator, is

), the discriminator is trained to recognize it as a real image.

)에 대한 판별자의 판별 결과(D(

))에 기초하여 수학식 10으로 계산하는 한편, 적대적 손실(L_D)을 수학식 11에 따라 계산한다.Accordingly, the learning unit generates a loss (L _G ) of the background filling network 300, which is a generator, as a filling image (

) the discriminator's discriminant result (D(

)) based on Equation (10), while calculating the adversarial loss ( _LD ) according to Equation (11).

where E is a function that obtains the expected value for the data distribution.

As a result, the learning unit performs learning by acquiring the sum of the generation loss (L _G ) and the adversarial loss (L _D ) of Equations 10 and 11 as a total loss (L _total ) without training data, or the mask loss ( Learning may be performed by obtaining the sum of L _m ), the generation loss (L _G ), and the adversarial loss (L _D ) of Equations (10) and (11) as a total loss (L _total ).

And when learning is performed by acquiring the sum of the mask loss (L _m ) of Equation 8, the generation loss (L _G ) and the adversarial loss (L _D ) of Equation 10 and 11 as a total loss (L _total ), learning The completed lidar depth image filling apparatus according to the present embodiment may be used as a learning data acquisition apparatus for learning of other apparatuses.

7 shows an example of a 3D map image obtained by using the LIDAR depth image filling apparatus according to the present embodiment.

In FIG. 7, (a) is an input image of the LIDAR depth image filling device, which shows a 3D map image in which a dynamic object exists, (b) shows a 3D map image obtained from an input image according to this embodiment, ( c) is a 3D map image obtained when the fusion block FBn of the fusion block unit 330 is fused with the feature map F _e ⁿ and the filling feature map F _i ⁿ without considering the structural similarity. indicates. In (b) and (c), the purple line represents the truth value of the background substituted for the dynamic object, and the green line represents the background of the obtained 3D map image.

As shown in (a) of Figure 7, in the case of the input image, there is a large area where the background does not appear due to the dynamic object, but in the case of (b) and (c), it can be seen that the dynamic object area is filled. have. However, in the case of (c), it can be seen that the difference between the purple line, which is the truth value, and the obtained green line, occurs significantly, whereas in the present embodiment considering the probabilistic structural similarity, it can be seen that the truth value is obtained similarly. This means that you can replace dynamic objects with the background very accurately.

8 is a diagram illustrating a lidar depth image filling method according to an embodiment of the present invention.

Referring to FIGS. 3 to 6 , the LIDAR depth image filling method of FIG. 8 is described. First, a 3D point cloud image, which is a depth image acquired from LIDAR, is acquired without considering a dynamic object ( S11 ). Then, the obtained 3D point cloud image is converted into a two-dimensional distance image in a predetermined manner to obtain input data (S12). At this time, the 3D point cloud image is converted into five two-dimensional distance images for each depth value, x-axis value, y-axis value, z-axis value, and intensity value, and the five distance images are combined in the channel axis direction to obtain input data. can

And by inputting the input image into the encoding network 310 implemented as a pre-trained artificial neural network and encoding according to the learned method, the encoding feature map (F _e ) is obtained (S13). Here, the encoding network consists of a plurality of sequentially connected convolutional layers, each of the convolutional layers extracts a feature map (F _e ⁿ ) step by step from the applied input image, and the final convolution layer is the extracted feature map. Output as an encoding feature map (F _e ).

When the encoding feature map (F _e ) is obtained, the object detection network 320 implemented as a pre-trained artificial neural network receives the encoding feature map (F _e ) and decodes it in a learned manner, corresponding to a dynamic object in the input image A binary mask m representing the region to be obtained is obtained (S14). Here, the object detection network 320 also includes a plurality of convolutional layers, can perform sequential decoding, and is predetermined to expand the two-dimensional size of the feature map reduced by the convolution operation of the plurality of convolutional layers. It may further include a plurality of upsampling layers arranged at intervals.

)를 획득할 수 있다.Then, the obtained binary mask (m) is scaled down to a size corresponding to each of the plurality of feature maps (F _e ⁿ ), and a plurality of scaled binary masks (

) can be obtained.

)가 획득되면, 미리 학습된 인공 신경망으로 구현되는 채움 네트워크(340)의 다수의 컨볼루션 레이어 중 제1 컨볼루션 레이어가 학습된 방식에 따라 인코딩 특징맵(F_e)을 디코딩하여 초기 채움 특징맵(F_i ¹)을 획득한다(S15).Multiple scale binary masks (

) is obtained, the first convolutional layer among a plurality of convolutional layers of the filling network 340 implemented as a pre-trained artificial neural network is decoded according to the learned method of the encoding feature map (F _e ) to decode the initial filling feature map. (F _i ¹ ) is obtained (S15).

) 중 대응하는 스케일 이진 마스크(m¹)에 따라 동적 객체 영역을 마스킹하고, 마스킹된 영역을 초기 채움 특징맵(F_i ¹)의 대응하는 영역인 배경을 수학식 3에 따라 대입하여 채움으로써 퓨전 특징맵(F_f ¹)을 획득한다(S16).When the initial _filling feature map _{F i} ^{1 is} obtained, ^a plurality ^of scale binary masks ₍

) of the dynamic object region according to the corresponding scaled binary mask (m ¹ ), and filling the masked region with the background, which is the corresponding region of the initial filling feature map (F _i ¹ ), according to Equation 3, thereby filling the fusion A feature map F _f ¹ is obtained (S16).

And the stochastic structural similarity (S ¹ ) between the feature map (F _e ¹ ) and the fusion feature map (F _f ¹ ) is calculated according to Equation 6 (S17). Thereafter, the fusion feature map F _f ¹ is weighted with the stochastic structural similarity (S ⁿ ) calculated according to Equation (7) to obtain the interest map (C ¹ ).

When the interest map (C ¹ ) is obtained, the interest map (C ¹ ) is received and decoded according to the method in which the next convolutional layer among a plurality of convolutional layers of the filling network 340 is learned, and then the filling feature map (F _i ² ) ) is obtained (S19). When the filling feature map F _i ² is obtained, it is determined whether the number of background filling is smaller than a predetermined number (S20). Here, the predetermined number corresponds to the number of convolutional layers of the filling network 340 . If the number of background fills does not reach the predetermined number of times, the area corresponding to the scale binary mask (m ² ) of the corresponding feature map (F _e ² ) is refilled using the obtained filling feature map (F _i ² ). (S16).

)로서 기지정된 방식으로 3차원으로 변환하여 3D 지도 이미지를 획득한다(S21).However, if it is determined that the predetermined number of times has been reached, the obtained filling feature map (F _i ⁿ ) is filled with the filling image (

) to obtain a 3D map image by converting it into three dimensions in a predetermined manner (S21).

The method according to the present invention may be implemented as a computer program stored in a medium for execution by a computer. Here, the computer-readable medium may be any available medium that can be accessed by a computer, and may include all computer storage media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data, and read dedicated memory), RAM (Random Access Memory), CD (Compact Disk)-ROM, DVD (Digital Video Disk)-ROM, magnetic tape, floppy disk, optical data storage, and the like.

Although the present invention has been described with reference to the embodiment shown in the drawings, which is only exemplary, those skilled in the art will understand that various modifications and equivalent other embodiments are possible therefrom.

Accordingly, the true technical protection scope of the present invention should be defined by the technical spirit of the appended claims.

100: point cloud image acquisition unit 200: input data acquisition unit
300: background filling network 310: encoding network
320: object detection network 330: fusion block unit
FB: fusion block 331: features fusion unit
332: structure similarity calculation unit 333: interest map acquisition unit
340: filling network 400: 3D map acquisition unit

Claims (20)

an input data acquisition unit configured to obtain input data by converting a three-dimensional point cloud image obtained using lidar into a two-dimensional image in a predetermined manner when applied;
It is implemented as a pre-learned artificial neural network, and by repeating the process of extracting the feature map by encoding the input data according to the learned method, the encoded feature map, which is the final encoded feature map among the plurality of feature maps, is applied to the learned method. decoding to obtain a binary mask representing the dynamic object region, and replacing the region corresponding to the binary mask in the corresponding feature map among the plurality of feature maps with the region of the previously obtained filled feature map according to the probabilistic structural similarity a background filling network that acquires an interest map and repeatedly acquires a filling feature map by decoding the acquired interest map in a pre-learned manner; and
Including a 3D map acquisition unit that receives the finally obtained filling feature map as a filling image, and converts it into a three-dimensional image in a predetermined manner,
The background filling network is
an encoding network including a plurality of convolutional layers each of which receives input data or a feature map output from a previously arranged convolutional layer and encodes it to output a feature map;
an object detection network that receives the encoding feature map and decodes it according to a learned method to obtain the binary mask indicating a region in which a dynamic object designated by learning is located in the input data;
a filling network for obtaining a filling feature map by decoding the encoding feature map or the previously acquired interest map in a learned manner; and
A fusion feature map is obtained by replacing an area corresponding to the binary mask in a corresponding feature map among the plurality of feature maps with an area of a previously obtained filled feature map, and a probability between the corresponding feature map and the obtained fusion feature map and a fusion block unit including a plurality of fusion blocks for calculating the structural similarity of an enemy and weighting the fusion feature map in a predetermined manner to obtain the interest map,
The interest map (C ⁿ ) is expressed by the equation

(Where λ _atn is the weight of interest, F _f ⁿ is the fusion feature map, and S ⁿ is the probabilistic structural similarity.)
LiDAR depth image filling device acquired according to
delete The method of claim 1, wherein the object detection network is
A lidar depth image filling apparatus for obtaining a plurality of scaled binary masks by scaling the binary mask to a size corresponding to each of a plurality of feature maps obtained from the encoding network. The method of claim 3, wherein each of the plurality of fusion blocks comprises:
A corresponding feature map of the plurality of feature maps, a corresponding scaled binary mask of a plurality of scaled binary masks, and a filling feature map previously obtained from the filling network are applied, and the corresponding scaled binary mask in the corresponding feature map is used. a fusion unit configured to obtain a fusion feature map by replacing the designated area with an area of the previously obtained filling feature map;
a structural similarity calculation unit that determines the structural similarity between a corresponding feature map and the fusion feature map according to a structural similarity index measurement (SSIM) structural similarity evaluation method, and calculates a stochastic structural similarity by additionally reflecting a stochastic distribution; and
and an interest map acquisition unit configured to acquire the interest map by weighting the calculated probabilistic structural similarity to the fusion feature map. 5. The method of claim 4, wherein the fusion unit
The fusion feature map (F _f ⁿ ) is expressed by the equation

(where F _e ⁿ is the feature map,

is the scaled binary mask, and F _i ⁿ is the fill feature map.)
LiDAR depth image filling device acquired according to The method of claim 5, wherein the structural similarity calculator
The structural similarity (S) is expressed by the equation

(Where k is the feature map (F _e ⁿ ), q is the fusion feature map (F _f ⁿ ), σ is the variance, and c ₃ is a predetermined constant.)
calculated according to
The probabilistic structural similarity is calculated by the equation

A lidar depth image filling device that calculates according to delete According to claim 1, wherein the input data acquisition unit
Five two-dimensional distance images are obtained from the 3D point cloud image according to the depth value, the x-axis value, the y-axis value, the z-axis value, and the intensity value, and the obtained five two-dimensional distance images are combined in the channel axis direction to combine the input data. Acquiring lidar depth image filling device. According to claim 4, wherein the lidar depth image filling device is
Further comprising a learning unit for learning the background filling network implemented as an artificial neural network during learning,
the learning unit
By applying a training data set in which a dynamic object region is preset and filled with the background as input data to the background filling network, the mask loss and the background caused by the difference between the binary mask output from the object detection network and the set dynamic object region A LIDAR depth image filling device that calculates the total loss as the sum of the filling loss caused by the difference between the filling feature maps output from the filling network, and performs learning by backpropagating the calculated total loss. According to claim 4, wherein the lidar depth image filling device is
Further comprising a learning unit for learning the background filling network implemented as an artificial neural network during learning,
the learning unit
Lidar depth image filling further comprising a discriminator trained to determine whether the filling feature map output from the background filling network is a real image or a fake image, learning the generator and the discriminator as the background filling network in a generative adversarial network method Device. when the 3D point cloud image obtained using the lidar is applied, converting it into a 2D image in a predetermined manner to obtain input data;
obtaining a plurality of feature maps by repeating the process of extracting feature maps by encoding the input data according to the learned method using a pre-learned artificial neural network;
using a pre-trained artificial neural network to decode an encoded feature map, which is a final encoded feature map among the plurality of feature maps, according to a learned method to obtain a binary mask representing a dynamic object region;
Using a pre-trained artificial neural network, the region of interest is obtained by replacing the region corresponding to the binary mask in the corresponding feature map among the plurality of feature maps according to the degree of probabilistic structural similarity with the region of the previously obtained filled feature map, , decoding the obtained interest map in a pre-learned manner to obtain a filling feature map; and
Including the step of receiving the finally obtained filling feature map as a filling image, and converting it into a three-dimensional image in a predetermined manner,
The step of obtaining the filling feature map is
decoding an encoding feature map or a previously acquired interest map in a learned manner to obtain the filled feature map; and
obtaining a fusion feature map by replacing an area corresponding to the binary mask in a corresponding feature map among the plurality of feature maps with an area of a previously obtained filling feature map; and
calculating the probabilistic structural similarity between the corresponding feature map and the obtained fusion feature map and weighting the fusion feature map in a predetermined manner to obtain the interest map,
The interest map (C ⁿ ) is expressed by the equation

(Where λ _atn is the weight of interest, F _f ⁿ is the fusion feature map, and S ⁿ is the probabilistic structural similarity.)
LiDAR depth image fill method acquired according to
. The method of claim 11, wherein the obtaining of the plurality of feature maps comprises:
A LIDAR depth image filling method that receives input data or a feature map output from a previously placed convolutional layer using an encoding network, which is an artificial neural network including a plurality of convolutional layers, and encodes it to output a plurality of feature maps. 13. The method of claim 12, wherein obtaining the binary mask comprises:
decoding the encoding feature map according to a learned method to obtain the binary mask representing a region in which a dynamic object designated by learning is located in the input data; and
and obtaining a plurality of scaled binary masks by scaling the binary mask to a size corresponding to each of the plurality of feature maps. delete The method of claim 11, wherein the obtaining of the interest map comprises:
determining the structural similarity between the corresponding feature map and the fusion feature map according to the structural similarity index measurement (SSIM) structural similarity evaluation method, and calculating the stochastic structural similarity by additionally reflecting a stochastic distribution in the structural similarity ; and
and weighting the calculated probabilistic structural similarity to the fusion feature map to obtain the interest map. The method of claim 15, wherein the obtaining of the fusion feature map comprises:
The fusion feature map (F _f ⁿ ) is expressed by the equation

(where F _e ⁿ is the feature map,

is the scaled binary mask, and F _i ⁿ is the fill feature map.)
LiDAR depth image fill method acquired according to The method of claim 16, wherein calculating the probabilistic structural similarity comprises:
The structural similarity (S) is expressed by the equation

(Where k is the feature map (F _e ⁿ ), q is the fusion feature map (F _f ⁿ ), σ is the variance, and c ₃ is a predetermined constant.)
calculating according to; and
The probabilistic structural similarity is calculated by the equation

A lidar depth image filling method comprising the step of calculating according to delete 12. The method of claim 11, wherein the LIDAR depth image filling method is
Further comprising a learning step for training the artificial neural network,
The learning step is
Calculating the mask loss caused by the difference between the binary mask output in the step of obtaining the binary mask by applying a training data set in which the dynamic object region is preset and filled with the background as input data of the artificial neural network and the set dynamic object region to do;
calculating a filling loss caused by a difference between the filling feature maps output in the step of obtaining the filling feature map; and
and calculating a total loss as a sum of the mask loss and the fill loss, and backpropagating the calculated total loss. 12. The method of claim 11, wherein the LIDAR depth image filling method is
Further comprising a learning step for training the artificial neural network,
The learning step is
determining whether the filling feature map is a real image or a fake image;
calculating each of a generation loss and an adversarial loss in a generative adversarial network method based on the determination result of the filling feature map; and
A method of filling a lidar depth image comprising backpropagating the sum of the calculated generational loss and the adversarial loss to the total loss.

Images