KR102682069B1 - Apparatus and Method for Filling Depth Map - Google Patents

Abstract

The present invention is implemented with a pre-trained artificial neural network, receives image depth information that is a combination of image data and a sparse depth map containing depth information for a specified number of pixels, and provides information about each pixel of the image data from the image depth information. A target estimation unit that obtains a temporary depth map by estimating the depth value, an offset map consisting of an offset indicating the position of similar pixels for each pixel, and a weight map consisting of a weight indicating the similarity, depth information from the sparse depth map Select valid pixels containing , obtain the position and depth information of similar pixels corresponding to each selected effective pixel using an offset map and weight map, and add the similar pixels with acquired depth information to the sparse depth map as valid pixels. It obtains a combined depth map by combining the propagation module that acquires the added effective depth map and the temporary depth map and the effective depth map, and is implemented with a pre-trained artificial neural network to provide high-density depth according to the method learned from the combined depth map. Provided is a depth map filling device and method that can acquire a high-density depth map with high accuracy and high reliability, including a depth map acquisition unit that acquires the map.

Description

Apparatus and Method for Filling Depth Map}

The present invention relates to an apparatus and method for filling a depth map, and to an apparatus and method for filling a depth map.

Depth information can be acquired using various sensors and is used in various computer vision applications. It is typically used in fields such as autonomous vehicles, robot path navigation, and 3D reconstruction. The stereo matching technique using a stereo camera and epipolar geometry is widely used for depth information, but this method takes a considerable amount of time because a matching point must be searched for each point, and the method takes a considerable amount of time depending on the camera position. It has the limitation of not being able to cope with occlusion.

Accordingly, the number of cases of acquiring depth using LiDAR sensors is increasing recently. LiDAR sensors have a structure that acquires depth by rotating the laser to various heights and angles, but due to the nature of the sensor, depth is not obtained for all corresponding points and is in the form of a point cloud, at the level of 4 to 5% of the entire area. We obtain a sparse depth map with a low density of . In this way, the depth map obtained using LIDAR has high accuracy, but there is a problem that the available applications are limited due to low density.

To overcome this problem, a depth completion technique has been proposed that fuses the low density of depth information obtained from LIDAR with image information to obtain a high-density depth map. However, existing techniques simply concatenate images and depth maps and input them into an artificial neural network to obtain a high-density depth map. In this case, since the amount of depth information provided in the sparse depth map obtained from LIDAR is very insufficient, the surrounding area where depth information is not provided has a great influence, and the initially provided depth information is not utilized properly, resulting in There are limitations in obtaining reliable high-density depth maps.

Korea Registered Patent No. 10-2229861 (registered on March 15, 2021)

The purpose of the present invention is to provide a depth map filling device and method that can obtain a highly reliable, high-density depth map by effectively utilizing initial depth information obtained using a sensor such as lidar.

Another object of the present invention is to provide a depth map filling device and method that can obtain a high-density depth map with high accuracy by propagating depth information obtained from a sensor based on similarity and utilizing the propagated and expanded depth information. .

The depth map filling device according to an embodiment of the present invention for achieving the above object is implemented with a pre-trained artificial neural network, and the image depth is a combination of image data and a sparse depth map containing depth information for a specified number of pixels. Receive information, obtain a temporary depth map by estimating the depth value for each pixel of the image data from the image depth information, and obtain an offset map consisting of an offset indicating the position of a similar pixel for each pixel and indicating a degree of similarity. a target estimation unit that obtains a weight map composed of weights; Valid pixels containing depth information are selected from the sparse depth map, and the position and depth information of similar pixels corresponding to each selected valid pixel are obtained using the offset map and the weight map, and depth information is obtained. a propagation module for obtaining an effective depth map in which a pixel is added to the sparse depth map as a valid pixel; And obtaining a depth map by receiving and combining the temporary depth map and the effective depth map to obtain a combined depth map, implemented with a pre-trained artificial neural network to obtain a high-density depth map according to a method learned from the combined depth map. Includes wealth.

The target estimation unit includes an encoder that receives the image depth information and encodes it in a pre-learned manner to obtain an image depth feature map; a decoder that receives the image depth feature map and decodes the image depth feature map according to a learned method to obtain a feature decoding map; and a pixel estimator that receives the feature decoding map and additionally decodes the feature decoding map using various pre-learned methods to obtain the temporary depth map, the offset map, and the weight map.

The pixel estimator includes a depth estimator that receives the feature decoding map and estimates depth information of each pixel according to a learned method to obtain the temporary depth map; an offset estimation unit that obtains the offset map by receiving the feature decoding map and determining an offset indicating the position of each of the predetermined number of similar pixels that are most similar to each pixel according to a learned method; and a weight estimation unit that obtains the weight map by receiving the feature decoding map and determining a weight indicating the similarity of each pixel with a predetermined number of similar pixels according to a learned method.

The propagation module includes a valid pixel selection unit that receives the sparse depth map, selects a valid pixel containing depth information, and obtains effective pixel information according to the depth information and position of the selected valid pixel; an offset analysis unit that receives the effective pixel information and the offset map and determines the positions of a predetermined number of similar pixels corresponding to the positions of the effective pixels; a weight analysis unit that receives the effective pixel information and the weight map and obtains depth information of the similar pixel by adding a weight corresponding to the depth information of the effective pixel; and a depth value setting unit configured to additionally set the similar pixel from which depth information is obtained as a valid pixel and add it to the sparse depth map to obtain the effective depth map.

When the depth value setting unit determines that the position of the similar pixel overlaps with similar pixels for a plurality of different valid pixels, the depth value setting unit calculates the cumulative average of the weighted depth information for each valid pixel to provide depth information of the similar pixel. It can be obtained with

When the depth information for all previously set valid pixels is propagated to similar pixels and added as valid pixels, the depth value setting unit determines whether the number of repetitions is less than or equal to a predetermined standard number, and if it is less than the standard number, the location of the additionally set valid pixels. Effective pixel information including information and depth information may be transmitted to the offset analysis unit, and if the reference number is exceeded, the currently acquired effective depth map may be transmitted to the depth map acquisition unit.

The depth map filling device includes an image acquisition unit that acquires the image data; a depth information acquisition unit that acquires the sparse depth map containing depth information detected using lidar for an area corresponding to the image data; and an image depth combining unit configured to obtain the image depth information by combining the image data and the sparse depth map.

A depth map filling method according to another embodiment of the present invention to achieve the above object includes a depth map filling device that obtains a high-density depth map by receiving a sparse depth map containing image data and depth information for a specified number of pixels. A depth map filling method, comprising: obtaining image depth information combining the image data and the sparse depth map; Using a pre-trained artificial neural network, an offset map consisting of a temporary depth map in which the depth value for each pixel of the image data is estimated from the image depth information and an offset indicating the position of a similar pixel for each pixel, and each pixel Obtaining a weight map composed of weights representing the similarity of similar pixels to; Valid pixels containing depth information are selected from the sparse depth map, and the position and depth information of similar pixels corresponding to each selected valid pixel are obtained using the offset map and the weight map, and depth information is obtained. obtaining a valid depth map in which pixels are added to the sparse depth map as valid pixels; And obtaining a combined depth map by receiving and combining the temporary depth map and the effective depth map, and obtaining a high-density depth map according to a method learned from the combined depth map using a pre-trained artificial neural network. .

Therefore, the depth map filling device and method according to an embodiment of the present invention repeatedly propagates depth information acquired using a sensor such as lidar according to similarity based on the image acquired together, and depth information expanded through repeated propagation By filling the depth map using , a high-density depth map with high accuracy and high reliability can be obtained.

Figure 1 shows a schematic structure of a depth map filling device according to an embodiment of the present invention.
FIG. 2 shows an example of the detailed configuration of the radio wave module of FIG. 1.
Figures 3 and 4 are diagrams for explaining the operation of the radio wave module of Figure 1.
Figure 5 shows an example of the result of filling a depth map by the depth map filling device according to this embodiment.
Figure 6 shows a depth map filling method according to an embodiment of the present invention.
Figure 7 shows a detailed process of the depth information propagation step in Figure 6.

In order to fully understand the present invention, its operational advantages, and the objectives achieved by practicing the present invention, reference should be made to the accompanying drawings illustrating preferred embodiments of the present invention and the contents described in the accompanying drawings.

Hereinafter, the present invention will be described in detail by explaining preferred embodiments of the present invention with reference to the accompanying drawings. However, the present invention may be implemented in many different forms and is not limited to the described embodiments. In order to clearly explain the present invention, parts not relevant to the description are omitted, and like reference numerals in the drawings indicate like members.

Throughout the specification, when a part "includes" a certain element, this does not mean excluding other elements, unless specifically stated to the contrary, but rather means that it may further include other elements. In addition, terms such as "... unit", "... unit", "module", and "block" used in the specification refer to a unit that processes at least one function or operation, which is hardware, software, or hardware. and software.

FIG. 1 shows a schematic structure of a depth map filling device according to an embodiment of the present invention, FIG. 2 shows an example of the detailed configuration of the propagation module of FIG. 1, and FIGS. 3 and 4 show the propagation module of FIG. 1. This is a drawing to explain the operation.

Referring to FIG. 1, the depth map filling device according to this embodiment includes an input information acquisition unit 100, a target estimation unit 200, a propagation module 300, and a depth map acquisition unit 400.

The input information acquisition unit 100 acquires the sparse depth map S and the corresponding image data I as input information to be delivered to the target estimation unit 200 and combines them to obtain image depth information. The input information acquisition unit 100 may include an image acquisition unit 110, a depth information acquisition unit 120, and an image depth combining unit 130.

The image acquisition unit 110 acquires image data (I). Here, the image data (I) may be acquired using a camera, etc., and may be a single RGB image acquired from a monocular camera rather than a stereo camera. And the depth information acquisition unit 120 acquires depth information for each actually measured location using a sensor such as lidar. The depth information obtained here is usually obtained dispersedly in the form of a point cloud at the level of 4 to 5% of the total detection area, so it is called a sparse depth map (S). At this time, the image data acquired by the image acquisition unit 110 and the depth information acquired by the depth information acquisition unit 120 are acquired for the same area. The image depth combiner 130 acquires image depth information by concatenating the image data (I) acquired from the image acquisition unit 110 and the sparse depth map (S) acquired from the depth information acquisition unit 120. do.

The target estimation unit 200 is implemented with a pre-trained artificial neural network, and obtains a temporary depth map (R) by estimating the depth value for each pixel of the image data from the image depth information according to the learned method, and for each pixel. Obtain an offset map (O) indicating the location of similar pixels and a weight map (W) indicating the similarity.

The target estimator 200 may include an encoder 210, a decoder 220, and a pixel estimator 230. The encoder 210 receives image depth information from the input information acquisition unit 100, encodes the image depth information according to a learned method, and extracts an image depth feature map. The decoder 220 decodes the image depth features extracted from the encoder 210 according to a learned method to obtain a feature decoding map.

Meanwhile, the pixel estimation unit 230 receives the feature decoding map and additionally decodes the feature decoding map using various pre-learned methods to create a temporary depth map (R), an offset map (O), and a weight map (W). Acquire. The pixel estimator 230 may include a depth estimator 231, an offset estimator 232, and a weight estimator 233.

The depth estimation unit 231 receives the feature decoding map and decodes it according to a previously learned method, so that the low-density depth information of the sparse depth map (S) is converted into a high-density temporary depth map (R) with depth information corresponding to each pixel of the image. ) to obtain. Here, the temporary depth map (R) is similar to the existing sparse depth map (S) and obtaining a high-density depth map from an image. Therefore, due to the characteristics of the sparse depth map (S) in which the ratio of invalid pixels for which depth information is not provided is much higher than for valid pixels for which depth information is provided, the depth information provided from the valid pixels of the sparse depth map (S) is stored in the temporary depth map (R). ), the level of contribution is reflected at a low level, so the reliability of depth information is not high.

Meanwhile, the offset estimation unit 232 receives the feature decoding map and decodes it according to a previously learned method to obtain an offset map (O) consisting of offsets indicating the positions of similar pixels for each pixel of the image. At this time, the offset estimation unit 232 obtains an offset map (O) by estimating a predetermined number of pixel positions with the highest similarity to the pixel for each position (here, 2 to 4 as an example). That is, the offset map (O) is obtained by obtaining location information for a predetermined number of pixels with high similarity to each pixel and obtaining an offset. Here, the offset for each of the plurality of pixels may be obtained in the form of a vector value indicating the location of the corresponding specified number of pixels.

The weight estimation unit 233 receives the feature decoding map, decodes it according to a previously learned method, estimates a weight representing the similarity between each pixel of the image, and obtains a weight map (W). Here, the weight estimation unit 233 may be trained to estimate a weight corresponding to each offset of the offset map.

In Figure 1, for convenience of explanation, the offset estimator 232 and the weight estimator 233 are shown as separate components. However, the offset estimator 232 and the weight estimator 232 may be integrated into a similar pixel estimator. You can.

The propagation module 300 uses the depth information of the sparse depth map (S) obtained from the depth information acquisition unit 120 based on the offset map (O) and weight map applied from the similar pixel estimation unit to Propagate to other pixels.

Referring to FIG. 2 , the propagation module 300 may include a valid pixel selection unit 310, an offset analysis unit 320, a weight analysis unit 330, and a depth value setting unit 340.

The valid pixel selection unit 310 first receives the sparse depth map (S) obtained from the depth information acquisition unit 120, selects a valid pixel containing depth information from the sparse depth map (S), and positions the valid pixel. Effective pixel information (s) including information and depth information is transmitted to the offset analysis unit 320. In the sparse depth map (S), valid pixels are included in a sparse number, as shown in (a) of FIG. 3.

The offset analysis unit 320 receives the offset map O obtained from the offset estimation unit 232 and corresponds to each effective pixel based on the effective pixel information s applied from the effective pixel selection unit 310. The positions of a predetermined number of similar pixels are determined. As described above, since the offset map (O) indicates the location information of a predetermined number of similar pixels corresponding to each pixel, the offset analysis unit 320 can easily determine the effective pixel corresponding to the valid pixel using the offset map. Location information can be obtained. That is, the offset analysis unit 320 confirms the location of a similar pixel similar to the currently set valid pixel, as shown in (b) of FIG. 3. Figure 3 shows, as an example, a case where two similar pixels are obtained for each effective pixel.

In addition, the offset analysis unit 320 generates a new effective pixel added by propagating the depth information of the existing effective pixel in the depth value setting unit 340, and when new effective pixel information (s') is applied, the offset map ( O) is used to determine the location of a predetermined number of similar pixels corresponding to the new effective pixel.

The weight analysis unit 330 uses the weight map (W) obtained from the weight estimation unit 233 to determine the effective pixel information and the weight between two pixels according to the position information of similar pixels obtained from the offset analysis unit 320 ( w) is obtained. Here, the weight represents the similarity of the depth information of similar pixels based on the depth information of the effective pixel.

The depth value setting unit 340 acquires the depth value of a similar pixel by weighting the weight (w) obtained from the weight analysis unit 330 to the depth information of the effective pixel information (s), and performs offset analysis on the obtained depth value. It is set to the depth information of the similar pixel location determined by the unit 320. The depth value setting unit 340 may set a depth value for a predetermined number of similar pixels for each effective pixel, as shown in (d) of FIG. 3 .

However, as shown by the red circle in (c) of FIG. 3, the depth value setting unit 340 designates pixels at the same location as similar pixels for a plurality of different effective pixels, so that there is no conflict in the depth value of the corresponding pixel. When this occurs, the depth values weighted by the weight (w) are accumulated and divided by the number of effective pixels in which collisions were caused, so that the depth values of the effective pixels are normalized. That is, the average of the depth values specified for each valid pixel is set as the depth value of the corresponding pixel.

And, as shown in (e) of FIG. 3, the depth value setting unit 340 designates a similar pixel with a set depth value as a new effective pixel (s') and transmits it back to the offset analysis unit 320. Accordingly, the offset analysis unit 320 re-acquires similar pixels corresponding to the new effective pixel (s'), and the weight analysis unit 330 re-acquires the weight (w) for the similar pixels to obtain additional new effective pixels. You can obtain additional items by repeating them.

This is because the number of valid pixels (s) in the sparse depth map (S) initially acquired from the sensor is very sparse, so even if the depth information of the valid pixels is propagated once, the ratio of the valid pixels (s) to the entire area is still low. am. Accordingly, in the present invention, the depth information of the initial effective pixel(s) is propagated to similar pixels to add effective pixels with depth information, and the depth information of the added effective pixels is repeatedly propagated again, thereby ensuring reliable initial effective pixel(s). Ensure that the depth information of pixel(s) is utilized to the fullest extent when obtaining a high-density depth map.

In Figure 4, (a) shows a sparse depth map (S) obtained from LIDAR, and (b) to (d) show cases where the depth information of the effective pixel is propagated to similar pixels once, three times, and five times, respectively. represents. As shown in (a) of FIG. 4, the ratio of initial effective pixels (s) in the sparse depth map (S) is very low, while the number of effective pixels with depth information increases as the number of repetitions of depth information propagation increases. It can be seen that it increases.

If it is determined that the depth value propagation of the effective pixel has been performed a predetermined number of times (here, 5 times as an example), the depth value setting unit 340 creates an effective depth map (D) based on the effective pixel information (s) set to date. It is acquired and delivered to the depth map acquisition unit 400.

The depth map acquisition unit 400 receives and combines the temporary depth map (R) obtained from the depth estimation unit 231 and the effective depth map (D) obtained from the propagation module 300 to obtain a combined depth map, A high-density depth map is obtained from the combined depth map obtained according to a pre-learned method. Here, the depth map acquisition unit 400 is implemented as a pre-trained artificial neural network and can obtain a high-density depth map from the combined depth map.

As mentioned above, in the case of the temporary depth map (R), depth information for all pixels of the image data (I) is obtained, but the depth information of the sparse depth map (S), which is the actual depth value obtained using an actual sensor, is obtained. Because it is so insufficient, there is a limitation that the reliability of the depth information for each pixel is not high. On the other hand, in the case of the effective depth map (D), the number of effective pixels is increased by directly propagating the depth information of the sparse depth map (S) based on the similarity between pixels, so the number is much larger than that of the sparse depth map (S). Pixels can have high confidence, but we do not have depth information for every pixel. Therefore, the depth map acquisition unit 400 combines the temporary depth map (R) and the effective depth map (D) with an increased number of effective pixels to re-estimate the depth information of all pixels, thereby determining the initial sparse depth map (S). Depth information is reflected at a high rate to obtain a high-density depth map with high reliability.

In the above-described depth map filling device, the target estimation unit 200 and the depth map acquisition unit 400 implemented with an artificial neural network must be learned before actual use of the depth map filling device. At this time, learning is performed using image data, sparse depth maps, and each Depth information about pixels can be performed using a supervised learning method using multiple learning data sets composed of high-density depth maps with pre-annotated depth information. Also, during learning, the loss is calculated as the difference between the pre-annotated high-density depth map and the high-density depth map acquired by the depth map acquisition unit 400, and learning can be performed by back-propagating the calculated loss.

Figure 5 shows an example of the result of filling a depth map by the depth map filling device according to this embodiment.

In Figure 5, the top image represents RGB image data (I) acquired using a camera, etc., the middle image represents a sparse depth map (S) acquired using a lidar sensor, etc., and the image below represents the present embodiment. It represents the high-density depth map acquired by the depth map filling device. As shown in FIG. 5, the depth map filling device according to this embodiment can obtain a high-density depth map with very high accuracy by maximizing the depth information of the initially acquired sparse depth map (S).

Figure 6 shows a depth map filling method according to an embodiment of the present invention, and Figure 7 shows a detailed process of the depth information propagation step of Figure 6.

Referring to FIGS. 1 to 5, when describing the depth map filling method of FIG. 6, first, a sparse depth map (S) and image data (I) are obtained (S10). Here, the sparse depth map (S) can be acquired using a sensor such as LiDAR, and the image data (I) can be acquired using a camera, etc., and are acquired for the same area. Then, image depth information is obtained by concatenating the acquired image data (I) and the sparse depth map (S) (S20).

Afterwards, an image depth feature map is obtained by encoding image depth information and extracting features according to a previously learned method (S30). Then, the image depth feature map is decoded according to a previously learned method to obtain a feature decoding map (S40). Once the feature decoding map is obtained, the feature decoding map is decoded again according to a previously learned method to determine the similarity between each pixel and the offset map (O), which consists of offsets indicating the positions of each pixel of a predetermined number of similar pixels. A weight map (W) is obtained by estimating the weight represented (S50). At the same time, the low-density depth information of the sparse depth map (S) obtains a high-density temporary depth map (R) with depth information corresponding to each pixel of the image according to a method previously learned from the feature decoding map (S60). Here, the temporary depth map (R) can be obtained in a different way from the method of obtaining the offset map (O) and weight map (W) according to the learned method. In addition, the offset map (O) and the weight map (W) can also be acquired separately in different ways depending on the learned method, but in some cases, the offset map (O) and the weight map (W) are acquired together in the same way. It could be.

Once the offset map (O) and the weight map (W) are obtained, valid pixels containing depth information are selected from the sparse depth map (S), and the weight of the weight map (W) is weighted to the depth information of the selected valid pixels. It propagates to similar pixels designated according to the offset map (O) (S70).

Referring to FIG. 7, in the depth information propagation step (S70), valid pixels with depth information are first selected from the sparse depth map (S) (S71). Then, effective pixel information (s) including depth information and position information of the selected effective pixel is acquired, and a predetermined number of similar pixels corresponding to the effective pixels are obtained based on the offset map (O) and the obtained effective pixel information (s). Check the location of each pixel (S72). Once the position of the valid pixel and the position of the similar pixel are confirmed, the weight of the similar pixel for the valid pixel is obtained using the confirmed position and the weight map (W), and the obtained weight is calculated by weighting the depth information of the effective pixel. The depth value is propagated to similar pixels as depth information of similar pixels (S73).

Then, it is determined whether depth information propagated from another valid pixel exists in a similar pixel to which depth information is propagated and the depth values overlap (S74). If it is determined that the depth values overlap, the overlapping depth values are accumulated, an average is calculated, and the overlapped depth values are normalized to obtain depth information of the corresponding similar pixel (S75).

Afterwards, it is determined whether the depth information of all currently acquired valid pixels has been propagated to similar pixels (S76). If it is determined that there is a valid pixel for which depth information has not been propagated, a valid pixel for which depth information has not been propagated, that is, a previously unselected valid pixel, is selected so that the depth information is propagated (S71). However, if all currently set valid pixels are selected and it is determined that the depth information has been propagated, the similar pixels to which the depth information has been propagated are set as new valid pixels (S77).

Referring again to FIG. 6, when the depth information is propagated and the number of effective pixels is expanded, it is determined whether the number of repetitions is less than or equal to a predetermined reference number (S80). If it is determined that the number of repetitions is less than the standard number, the depth information propagation step is performed again (S70). At this time, depth information propagation may be performed only for new valid pixels, without being redundantly performed for existing idle pixels.

However, if it is determined that the number of repetitions exceeds the reference number, the effective depth map (D) consisting of valid pixels acquired so far is determined as the last updated effective depth map, and the current effective depth map (D) and temporary depth map (R) are determined as the last updated effective depth map. ) are combined to obtain a combined depth map, and a highly reliable, high-density depth map is obtained from the combined depth map obtained according to a pre-learned method.

And similar pixels with depth information propagated are set as new valid pixels and are repeatedly

The method according to the present invention can be implemented as a computer program stored on a medium for execution on a computer. Here, computer-readable media may be any available media that can be accessed by a computer, and may also include all computer storage media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules or other data, including read-only memory (ROM). It may include dedicated memory), RAM (random access memory), CD (compact disk)-ROM, DVD (digital video disk)-ROM, magnetic tape, floppy disk, optical data storage, etc.

The present invention has been described with reference to the embodiments shown in the drawings, but these are merely exemplary, and those skilled in the art will understand that various modifications and other equivalent embodiments are possible therefrom.

Therefore, the true technical protection scope of the present invention should be determined by the technical spirit of the attached claims.

100: input information acquisition unit 110: image acquisition unit
120: Depth information acquisition unit 130: Image depth combining unit
200: Target estimation unit 210: Encoder
220: Decoder 230: Pixel estimator
231: depth estimation unit 232: offset estimation unit
233: Weight estimation unit 300: Propagation module
310: Effective pixel selection unit 320: Offset analysis unit
330: Weight analysis unit 340: Depth value setting unit
400: Depth map acquisition unit

Claims (15)

It is implemented as a pre-trained artificial neural network, receives image depth information that is a combination of image data and a sparse depth map containing depth information for a specified number of pixels, and receives the depth for each pixel of the image data from the image depth information. a target estimation unit for estimating values to obtain a temporary depth map, an offset map consisting of an offset indicating the position of similar pixels for each pixel, and a weight map consisting of a weight indicating similarity;
Valid pixels containing depth information are selected from the sparse depth map, and the position and depth information of similar pixels corresponding to each selected valid pixel are obtained using the offset map and the weight map, and depth information is obtained. a propagation module for obtaining an effective depth map in which a pixel is added to the sparse depth map as a valid pixel; and
A depth map acquisition unit that receives and combines the temporary depth map and the effective depth map to obtain a combined depth map, and is implemented as a pre-trained artificial neural network to obtain a high-density depth map according to a method learned from the combined depth map. Including,
The radio wave module is
a valid pixel selection unit that receives the sparse depth map, selects a valid pixel containing depth information, and obtains effective pixel information according to the depth information and position of the selected valid pixel;
an offset analysis unit that receives the effective pixel information and the offset map and determines the positions of a predetermined number of similar pixels corresponding to the positions of the effective pixels;
a weight analysis unit that receives the effective pixel information and the weight map and obtains depth information of the similar pixel by adding a weight corresponding to the depth information of the effective pixel; and
A depth value setting unit configured to additionally set the pseudo-pixel from which depth information is obtained as a valid pixel and add it to the sparse depth map to obtain the effective depth map,
The depth value setting unit
If the position of the similar pixel is determined to overlap with similar pixels for a plurality of different effective pixels, a depth map is obtained as the depth information of the similar pixel by calculating the cumulative average of the weighted depth information in each valid pixel. Filling device.
The method of claim 1, wherein the target estimation unit
an encoder that receives the image depth information and encodes it in a pre-learned manner to obtain an image depth feature map;
a decoder that receives the image depth feature map and decodes the image depth feature map according to a learned method to obtain a feature decoding map; and
A depth map filling device comprising a pixel estimator that receives the feature decoding map and additionally decodes the feature decoding map using various pre-learned methods to obtain the temporary depth map, the offset map, and the weight map. The method of claim 2, wherein the pixel estimator
a depth estimation unit that obtains the temporary depth map by receiving the feature decoding map and estimating depth information of each pixel according to a learned method;
an offset estimation unit that obtains the offset map by receiving the feature decoding map and determining an offset indicating the position of each of the predetermined number of similar pixels that are most similar to each pixel according to a learned method; and
A depth map filling device including a weight estimation unit that obtains the weight map by receiving the feature decoding map and determining a weight representing the similarity with each of the predetermined number of similar pixels that is most similar to each pixel according to a learned method. . delete delete The method of claim 1, wherein the depth value setting unit
When the depth information for all previously set valid pixels is propagated to similar pixels and added as valid pixels, it is determined whether the number of repetitions is less than or equal to a predetermined standard number, and if it is less than the standard number, the location information and depth information of the additionally set valid pixels are A depth map filling device that transmits the included effective pixel information to the offset analysis unit and, when the reference number is exceeded, transmits the currently acquired effective depth map to the depth map acquisition unit. The method of claim 1, wherein the depth map filling device
an image acquisition unit that acquires the image data;
a depth information acquisition unit that acquires the sparse depth map containing depth information detected using lidar for an area corresponding to the image data; and
A depth map filling device further comprising an image depth combining unit that combines the image data and the sparse depth map to obtain the image depth information. In the depth map filling method of the depth map filling device, which obtains a high-density depth map by receiving a sparse depth map containing image data and depth information for a specified number of pixels,
Obtaining image depth information combining the image data and the sparse depth map;
Using a pre-trained artificial neural network, an offset map consisting of a temporary depth map in which the depth value for each pixel of the image data is estimated from the image depth information and an offset indicating the position of a similar pixel for each pixel, and each pixel Obtaining a weight map composed of weights representing the similarity of similar pixels to;
Valid pixels containing depth information are selected from the sparse depth map, and the position and depth information of similar pixels corresponding to each selected valid pixel are obtained using the offset map and the weight map, and depth information is obtained. obtaining a valid depth map in which pixels are added to the sparse depth map as valid pixels; and
Obtaining a combined depth map by receiving and combining the temporary depth map and the effective depth map, and obtaining a high-density depth map according to a method learned from the combined depth map using a pre-trained artificial neural network,
The step of obtaining the effective depth map is
receiving the sparse depth map, selecting a valid pixel containing depth information, and obtaining effective pixel information according to the depth information and location of the selected valid pixel;
receiving the effective pixel information and the offset map and determining the positions of a predetermined number of similar pixels corresponding to the positions of the effective pixels;
obtaining depth information of the similar pixel by receiving the effective pixel information and the weight map and weighting a weight corresponding to the depth information of the effective pixel; and
In order to obtain the effective depth map, additionally setting the similar pixel from which depth information is obtained as a valid pixel and adding it to the sparse depth map,
The step of adding to the sparse depth map is
determining whether the positions of the similar pixels overlap with similar pixels for a plurality of different effective pixels;
If it is determined that there is overlap, the depth map filling method further includes calculating the cumulative average of the weighted depth information in each valid pixel and obtaining it as the depth information of the corresponding similar pixel.
The method of claim 8, wherein the step of obtaining the weight map is
Obtaining an image depth feature map by receiving the image depth information and encoding it using a previously learned method;
Obtaining a feature decoding map by receiving authorization for the image depth feature map and decoding the image depth feature map according to a learned method; and
A depth map filling method comprising the step of receiving approval for the feature decoding map and further decoding the feature decoding map using several different pre-learned methods to obtain the temporary depth map, the offset map, and the weight map. The method of claim 9, wherein the additional decoding step
obtaining the temporary depth map by receiving the feature decoding map and estimating depth information of each pixel according to a learned method;
Obtaining the offset map by receiving the feature decoding map and determining an offset indicating the position of each of the predetermined number of similar pixels that are most similar to each pixel according to a learned method; and
A depth map filling method comprising the step of obtaining the weight map by receiving the feature decoding map and determining a weight indicating the similarity of each pixel with a predetermined number of similar pixels according to a learned method. delete delete The method of claim 8, wherein adding to the sparse depth map
When depth information for all previously set valid pixels is propagated to similar pixels and added as valid pixels, determining whether the number of repetitions is less than or equal to a predetermined reference number;
If the number is less than or equal to the reference number, additionally setting a similar pixel as a valid pixel using valid pixel information including position information and depth information of the additionally set effective pixel, an offset map, and a weight map; and
When the reference number of times is exceeded, setting the currently obtained effective depth map as the last updated effective depth map. The method of claim 8, wherein the step of obtaining the image depth information
acquiring the image data;
Obtaining the sparse depth map containing depth information detected using LIDAR for an area corresponding to the image data; and
A depth map filling method comprising obtaining the image depth information by combining the image data and the sparse depth map. A recording medium on which program instructions readable by a computing device for performing the depth map filling method according to claim 8 are recorded.