KR20220043830A - Method and device for performing plane detection - Google Patents

Abstract

A method and a device for performing plane detection are provided. A method of extracting a plane includes the steps of: acquiring an input image; extracting features of the input image using a deep neural network and estimating a depth map of the input image based on the extracted features; and performing region segmentation using the depth map to detect plane regions in the input image. Thus, planes can be detected in a texture-free region.

Description

METHOD AND DEVICE FOR PERFORMING PLANE DETECTION

The present disclosure relates to the field of human-computer interaction, and more particularly, to a method and apparatus for performing plane detection.

In the field of human-computer interaction, Augmented Reality (AR) is one of the important branches. Augmented reality is an interactive experience of a real world environment. In augmented reality, objects present in the real world are enhanced by computer-generated perceptual information, sometimes spanning multiple sensory modes including sight, hearing, touch, somatosensory, and/or smell. can be improved An augmented reality system can be defined as a system that meets three basic characteristics: combination of the real world and virtual world, real-time interaction, and accurate three-dimensional registration of virtual and real objects. The superimposed sensory information may be augmented with a real environment, or concealment of the real environment.

Due to the popularity of mobile smart devices and the tremendous increase in computing power, augmented reality technology has made significant progress in the past few years. Augmented reality, a new human-computer interaction technology, can more intuitively display physical objects and data information in real scenes. Many studies are being conducted to better combine virtual objects with the real environment to provide a better immersive sensory experience.

Planar detection technology is one of the core technologies of augmented reality. The plane detection technology can detect the position and size of various planes (such as the ground, desktop, wall, etc.) in a real environment. In augmented reality, virtual items may be placed in detected planes. Accurate plane detection results are one of the key factors for augmented reality applications to provide a good user experience.

Existing plane detection methods not only fail to detect planes in texture-less areas (such as solid-color walls, desktops, etc.) may not be aligned with In consideration of this, there is a need for a method and an apparatus capable of accurately detecting a plane.

According to one aspect of the present disclosure, there is provided a method for performing plane detection, the method comprising: acquiring an input image; extracting features of the input image and estimating a depth map of the input image based on the extracted features using a deep neural network; and performing region segmentation using the depth map to detect planar regions in the input image.

According to various embodiments, the deep neural network includes a feature extractor for extracting features of the input image, a depth estimation branch for estimating depth information of the input image, and a normal ( normal) for estimating information, and when estimating the depth map of the input image, the depth information estimated by the depth estimation branch uses normal information estimated by the normal estimation branch to be optimized

According to various embodiments, when estimating the depth map of the input image, the feature map of a predetermined resolution obtained by feature extraction of the input image using the feature extractor is a depth estimation using the depth estimation branch. The fused depth feature map and the fused normal feature map are respectively fused with a depth feature map of the same resolution generated during normal estimation and a normal feature map of the same resolution generated during normal estimation using the normal estimation branch. You can get a depth map.

According to various embodiments, the process of optimizing the depth information estimated by the depth estimation branch using the normal information estimated by the normal estimation branch exceeds a predetermined degree of normal feature change in the normal feature map. extracting information related to a region, and optimizing the depth feature map using the information to obtain the optimized depth feature map.

According to various embodiments, the process of extracting information related to a region in which the normal feature change exceeds the predetermined degree from the normal feature map and optimizing the depth feature map using the information includes: performing horizontal depth convolution and vertical depth convolution on the normal feature map, respectively, and using an activation function to obtain a horizontal attention map and a vertical attention map for the information; and obtaining the optimized depth feature map based on the horizontal attention map, the vertical attention map, and the depth feature map.

According to various embodiments, the process of obtaining the optimized depth feature map based on the horizontal attention map, the vertical attention map, and the depth feature map includes: a process of assigning weights to the horizontal attention map and the vertical attention map ; and fusing the weighted horizontal attention map and the weighted vertical attention map with the depth feature map to obtain the optimized depth feature map.

According to various embodiments, the process of detecting planar regions in the input image by performing region segmentation using the depth map may include: 3D points in the input image for plane estimation using the depth map. and depth-continuous regions, and detecting the planar regions in the input image by performing the region segmentation using the calculated 3D points and information on the depth continuous regions. may include

According to various embodiments, the process of detecting the planar regions in the input image by performing the region segmentation using the calculated 3D points and information on the depth continuous regions includes: using the calculated 3D points calculating a normal map of the input image, and fusing the calculated normal map with a normal map estimated by the deep neural network; and clustering to segment the planar regions using the fused normal map and information on the depth continuous regions.

According to various embodiments, the process of detecting the planar regions in the input image by performing the region segmentation using the calculated 3D points and information on the depth continuous regions includes: using the calculated 3D points calculating a normal map of the input image; and clustering to segment the planar regions using the calculated normal map and information on the depth continuous regions.

According to various embodiments, the deep neural network includes an extractor for extracting features of the input image and a depth estimation branch for extracting depth information of the input image, When estimating a depth map, a feature map of a predetermined resolution obtained by feature extraction of the input image using the feature extractor is a depth feature map of the same resolution generated during depth estimation using the depth estimation branch The depth map may be generated using the fused depth feature map by fusion with .

According to various embodiments of the present disclosure, the method may further include fine-tuning the boundaries of the detected planar areas so that the boundaries of the planar areas are aligned with the boundaries of real objects in the input image. .

According to various embodiments of the present disclosure, the process of fine-tuning the boundaries of the detected planar regions may include: acquiring discrete label values corresponding to each of the detected planar regions; converting each detected planar region into a three-dimensional volume based on the individual label value; The method may include a process of fine-tuning the planar areas based on the converted 3D volume and the input image, so that the boundaries of the planar areas are aligned with the boundaries of real objects in the input image.

According to various embodiments of the present disclosure, the process of fine-tuning the boundaries of the detected plane regions may include: acquiring region information corresponding to each pixel in the input image based on each detected plane region; Based on the shortest distance on the two-dimensional single-channel image between each pixel and the detected boundaries of each planar region, each obtaining plane weight information of pixels; The fine-tuned planar area is determined by determining similarity between pixels based on a pixel value, the area information, and the plane weight information corresponding to each pixel, and performing image segmentation based on the similarity between the respective pixels It may include the process of obtaining boundaries.

According to another aspect of the present disclosure, there is provided an apparatus for performing plane detection, the apparatus comprising: an image acquiring unit configured to acquire an input image; an estimation unit, configured to extract features of the input image and, using a deep neural network, estimate a depth map of the input image based on the extracted features; and a region segmentation unit configured to perform region segmentation using the depth map to detect planar regions in the input image.

According to various embodiments, the deep neural network includes a feature extractor for extracting features of the input image, a depth estimation branch for estimating depth information of the input image, and a normal of the input image and a normal estimation branch for estimating (normal) information, wherein when estimating the depth map of the input image, the estimation unit is configured to access the depth estimation branch by using normal information estimated by the normal estimation branch. Optimize the depth information estimated by

According to various embodiments, when estimating the depth map of the input image, the estimation unit generates a feature map of a predetermined resolution obtained by feature extraction of the input image using the feature extractor to the depth estimation branch. The fused depth feature map and the fused normal feature map are respectively fused with a depth feature map of the same resolution generated during depth estimation using The depth map using the map may be obtained.

According to various embodiments, the estimation unit for optimizing the depth information estimated by the depth estimation branch using the normal information estimated by the normal estimation branch may include: a normal feature change in the normal feature map is predetermined extracting information related to a region exceeding the degree, and optimizing the depth feature map using the information to obtain the optimized depth feature map.

According to various embodiments, extracting information related to a region in which the normal feature change exceeds the predetermined degree from the normal feature map and optimizing the depth feature map using the information may include: performing horizontal depth convolution and vertical depth convolution on the feature map, respectively, and using an activation function to obtain a horizontal attention map and a vertical attention map for the information; The method may include acquiring the optimized depth feature map based on the horizontal attention map, the vertical attention map, and the depth feature map.

According to various embodiments, obtaining the optimized depth feature map based on the horizontal attention map, the vertical attention map, and the depth feature map may include: assigning weights to the horizontal attention map and the vertical attention map; and fusing the weighted horizontal attention map and the weighted vertical attention map with the depth feature map to obtain the optimized depth feature map.

According to various embodiments, detecting planar regions in the input image by performing region segmentation using the depth map includes: 3D points in the input image for plane estimation using the depth map and detecting the planar regions in the input image by calculating depth-continuous regions and performing the region segmentation using the calculated three-dimensional points and information on the depth continuous regions can

According to various embodiments, detecting the planar regions in the input image by performing the region segmentation using the calculated 3D points and information on the depth continuous regions includes: using the calculated 3D points calculating a normal map of the input image, and fusing the calculated normal map with a normal map estimated by the deep neural network; and clustering to segment the planar regions using the fused normal map and information on the depth continuous regions.

According to various embodiments, detecting the planar regions in the input image by performing the region segmentation using the calculated 3D points and information on the depth continuous regions includes: using the calculated 3D points calculating a normal map of the input image; and clustering to segment the planar regions using the calculated normal map and information on the depth continuous regions.

According to various embodiments, the deep neural network may include an extractor for extracting features of the input image and a depth estimation branch for extracting depth information of the input image, When estimating a depth map of an image, the estimating unit generates a feature map of a predetermined resolution obtained by feature extraction of the input image using the feature extractor in depth estimation using the depth estimation branch By fusion with a depth feature map of the same resolution, the depth map may be generated using the fused depth feature map.

According to various embodiments, the device is a plane boundary fine-tuning unit, configured to fine-tune the boundaries of the detected planar regions so that the boundaries of the plane regions are aligned with the boundaries of real objects in the input image may further include.

According to various embodiments, fine-tuning the boundaries of the detected planar regions may include: acquiring discrete label values corresponding to each of the detected planar regions, respectively; converting each detected planar area into a three-dimensional volume based on the individual label value; The method may include fine-tuning the planar areas based on the transformed 3D volume and the input image, so that the boundaries of the planar areas are aligned with the boundaries of real objects in the input image.

According to various embodiments, fine-tuning the boundaries of the detected planar regions may include: acquiring region information corresponding to each pixel in the input image based on each detected plane region; Based on the shortest distance on the two-dimensional single-channel image between each pixel and the detected boundaries of each planar region, each obtaining plane weight information of the pixel; The fine-tuned planar area is determined by determining similarity between pixels based on a pixel value, the area information, and the plane weight information corresponding to each pixel, and performing image segmentation based on the similarity between the respective pixels may include obtaining boundaries.

According to various embodiments, there is provided an apparatus including a processor and a memory for storing program instructions, the instructions, when executed by the processor, cause the processor to perform the method as described above.

According to various embodiments, there is provided a computer-readable recording medium storing program instructions recorded thereon, and when the instructions are executed by the processor, the processor performs the method as described above to do it

According to the method for detecting a plane and an apparatus for detecting a plane of the present disclosure, since planes are detected based on a depth map of an entire input image, the planes may be detected in a texture-free region.

Additionally, based on this, the accuracy of plane detection can be efficiently improved through feature map fusion, and/or by optimizing depth information using normal information, and the boundaries of the detected plane regions are subjected to a plane fine-tuning operation. may be aligned to the boundaries of real objects in the input image by further performing .

Aspects, other aspects, and/or advantages of the present application may be clearly understood from the following detailed description of embodiments of the present application in conjunction with the accompanying drawings.
1 is a schematic diagram of plane detection in an existing augmented reality framework.
2 is a schematic diagram of plane detection in an augmented reality framework according to the present disclosure;
3 is a flowchart of a method for performing plane detection according to the present disclosure;
4 is a schematic diagram of a plane detection method according to an exemplary embodiment of the present disclosure.
5 is a schematic diagram of operations of a deep neural network according to an exemplary embodiment of the present disclosure;
6 is a schematic diagram illustrating operations performed by a feature fusion module according to an exemplary embodiment of the present disclosure.
7 is a schematic diagram of a region in which normal information is obviously changed but depth information is slightly changed.
8 is a schematic diagram of optimizing depth information using normal information according to an exemplary embodiment of the present disclosure.
9 is a schematic diagram illustrating operations performed by a normal-guided attention module according to an exemplary embodiment of the present disclosure.
10 is a schematic diagram illustrating an example of a planar area segmentation operation according to an exemplary embodiment of the present disclosure.
11 is a schematic diagram illustrating an example of a plane refinement operation according to an exemplary embodiment of the present disclosure.
12 is a schematic diagram illustrating a plane detection method according to another exemplary embodiment of the present disclosure.
13 is a schematic diagram illustrating operations of the deep neural network according to another exemplary embodiment of the present disclosure.
14 is a schematic diagram illustrating a planar region segmentation operation according to another exemplary embodiment of the present disclosure.
15 is a block diagram illustrating an apparatus for performing plane detection according to the present disclosure.
16 is a schematic diagram illustrating a semantic-aware plane detection system suitable for augmented reality according to the present disclosure.

Before describing the concept of the present disclosure and exemplary embodiments of the present disclosure, in order to help a better understanding of the present disclosure, a plane detection method in an existing augmented reality framework is first briefly described. will be explained

1 is a schematic diagram of plane detection in an existing augmented reality framework.

As shown in FIG. 1 , a plane detection method in an existing augmented reality framework largely relies on a three-dimensional point cloud from a Simultaneously Localization and Mapping (SLAM) system. In particular, the existing AR framework acquires a three-dimensional point cloud from the SLAM system as an input, and performs plane parameter fitting using RANSAC (Random Sample Consensus) to complete the plane detection. After that, the AR content is displayed along with the camera pose. It can be seen that the conventional plane detection method detects planes based on a sparse 3D point cloud generated by the SLAM system.

The SLAM system may first extract feature points from an input image, match the extracted two-dimensional feature points, and calculate three-dimensional points using the matched two-dimensional feature points. Since the two-dimensional feature points cannot be detected in a texture-less region, the SLAM system fails to acquire the three-dimensional points corresponding to the texture-less region, which is why the existing method fails to obtain the texture-less region. It may cause a problem in that the plane detection cannot be performed in an area where there is no .

Second, as described above, the conventional plane detection method performs plane detection based on a sparse 3D point cloud output by the SLAM system. However, the 3D point cloud cannot provide sufficient information to accurately estimate plane boundaries, and the detected plane boundaries cannot be aligned with the boundaries of a real object. .

The present disclosure proposes a new plane detection method and apparatus capable of solving the above problems. Hereinafter, a concept and exemplary embodiments for performing plane detection of the present disclosure will be described in detail with reference to FIGS. 2 to 15 .

2 is a schematic diagram of plane detection in an augmented reality framework according to the present disclosure; As shown in FIG. 2 , the present disclosure directly determines planes based on an input image (which may be a color image), instead of performing plane detection according to a sparse three-dimensional point cloud output by the SLAM system. can be detected. Since information in the entire image is used for plane detection, the present disclosure can perform plane detection even in a texture-free region.

In order to solve the problem that plane detection cannot be performed in a texture-free region, the present disclosure may design a deep neural network for acquiring scene information. The deep neural network may provide information about the entire scene, including areas without texture, to estimate planes in the scene.

In order to solve the problem that the detected planes cannot be aligned with the boundaries of real objects, the present disclosure adopts a depth region segmentation technique to obtain an initial plane, and divides the boundaries of the detected initial planes into real objects. Adopt a plane boundary refinement technique to align to

Hereinafter, a method for performing plane detection according to an exemplary embodiment of the present disclosure will be described in detail with reference to FIGS. 3 to 14 .

3 is a flowchart of a method for performing plane detection according to the present disclosure (hereinafter, for convenience of description, it will be referred to as a “plane detection method”).

Referring to FIG. 3 , in step S310, an input image may be obtained. For example, the input image may be acquired through a camera in response to a user request, and the input image may be acquired in real time. Embodiments of the present disclosure do not have any limitation on a method of acquiring the input image.

In step S320 , the deep neural network may extract features of the input image and estimate a depth map of the input image based on the extracted features. Here, a deep neural network may be used to obtain scene information. For example, the features of the input image may include a feature map of 1/2 resolution, a feature map of 1/4 resolution, a feature map of 1/8 resolution, a feature map of 1/16 resolution, and 1/32 resolution of the input image. It can be used to create a feature map of The depth map of the input image includes various deconvolution and convolution operations of layers used for information estimation in the deep neural network based on the generated feature map (using the feature map) It can be finally estimated using a larger size depth map and a normal map (used to predict a normal map) by magnification.

In step S330, the depth map may be used to perform region segmentation to detect planar regions in the input image. According to various embodiments,

In step S330, the depth map may be used to calculate 3D points and depth-continuous regions in the input image for plane estimation. The calculated 3D points and information of the depth continuous regions may be used to perform region segmentation for detecting planar regions in the input image.

For example, a depth map can be used to calculate three-dimensional points in the following way. According to a camera imaging model (eg, a pinhole camera model), the following equation (1) can be used to calculate three-dimensional points:

[Equation 1]

Here, u and v are image pixel coordinates, fx and fy are focal lengths, and cx and cy are image principal points. When traversing each pixel in the image, Z is the depth in the depth map corresponding to the pixel, and substituting u, v, and Z into the above equation, the X and Y coordinates of the spatial point can be obtained, and thus all three-dimensional points corresponding to the entire image can be obtained.

In the present disclosure, the above method for calculating three-dimensional points using a depth map is described as an example, but it is apparent to those skilled in the art that other methods of calculating three-dimensional points may also be used depending on the design method, The present disclosure does not limit specific calculation methods of the three-dimensional points.

According to various embodiments, depth continuous regions may be calculated using the depth map in the following manner. For example, when using a relatively simple region growth method, the normal vector of each point is first calculated using three-dimensional points in the neighborhood of the point (such as 9x9). can Set a certain threshold (eg 10 degrees), compare the normal vectors of adjacent points one by one, and if the result is within the certain threshold, two points are considered continuous, and the average of the two points The value can be used as a normal vector of the contiguous region after merging. Thereafter, the adjacent points or other contiguous regions may be continuously compared.

In the present disclosure, a method of calculating the depth contiguous region using the depth map is described as an example, but it will be appreciated by those skilled in the art that other methods may also be used to calculate the depth contiguous region according to design specifications. self-evident

According to various embodiments, after calculating the 3D points and the depth contiguous regions, the region segmentation is performed using the information of the calculated 3D points and the depth contiguous regions in the following manner to input the input may be performed to detect the planar regions in the image. According to various embodiments, a normal map of the input image may be calculated using the 3D points. However, the calculated normal map may be fused with a normal map estimated by the deep neural network. The fused normal map and information on the depth continuation regions may be used, and the planar regions may be clustered to segment the plane regions using the fused normal map and information on the depth continuation regions.

According to various embodiments, after calculating the 3D points and the depth contiguous regions, the region segmentation is performed using the information of the calculated 3D points and the depth contiguous regions in the following manner to input the input may be performed to detect the planar regions in the image. According to various embodiments, a normal map of the input image is calculated using the calculated 3D points, and clustering is performed to segment the planar regions using the calculated normal map and information on the depth continuous regions. )can do. According to various embodiments, the method of performing the region segmentation to detect the planar regions in the input image using the information of the calculated three-dimensional points and the depth continuous regions is not limited to the example as above. The point is obvious to those skilled in the art.

As described above, the plane detection method of the present disclosure uses the entire input image to estimate the depth map of the input image, instead of performing the plane detection using the sparse feature points generated by the SLAM system. Since using the information of and using the depth map to detect the plane regions in the input image, it may be possible to realize the plane detection even in a texture-free region.

According to various embodiments, the plane detection method may further include the following steps (not shown in FIG. 3 ). According to various embodiments, the boundaries of the detected plane regions may be finely adjusted so that the boundaries of the plane regions are aligned with the boundaries of real objects in the input image.

Hereinafter, content related to a plane detection method according to an exemplary embodiment of the present disclosure will be described in more detail with reference to FIGS. 4 to 14 .

According to various embodiments, the deep neural network described in step S320 includes a feature extractor for extracting features of an input image, a depth estimation branch for estimating depth information of the input image, and the input It may include a normal estimation branch for estimating normal information of the image.

However, the structure of the deep neural network of the present disclosure is not limited to the above examples. According to various embodiments, the deep neural network may not include a normal estimation branch, and to estimate a depth estimation branch used for estimating features of the input image and depth information of the input image It may include a feature estimator for The present disclosure does not limit the specific structure of the deep neural network as long as the features of the input image can be extracted and at least a depth map of the input image can be estimated. According to various embodiments, the deep neural network may estimate a normal map of the input image in addition to the depth map of the input image.

4 is a schematic diagram of a plane detection method according to an exemplary embodiment of the present disclosure.

As shown in FIG. 4 , the input image may be input to a deep neural network, and the detected planes may be output through plane region segmentation and plane region fine-tuning. According to various embodiments, when considering that depth information of an image and surface normal information are closely related, a deep neural network includes a feature extractor for extracting features of an input image, and depth estimation for estimating depth information of the input image. It may be configured to include a depth estimation branch and a normal estimation branch for estimating normal information of the input image.

According to various embodiments, the deep neural network may simultaneously estimate depth information and normal information from a single input image in a multi-task manner, and a dense three-dimensional point cloud for plane estimation ) can be obtained. According to various embodiments, the input image may be input to a feature extractor of a deep neural network for feature extraction, and the extracted features may be input to a depth estimation branch and a normal estimation branch, respectively.

According to various embodiments, the depth estimation branch may perform depth estimation to output the depth map, the normal estimation branch may perform normal estimation to output the normal map, and the depth map and the normal map may include: It may be used during the flat area segmentation. 4 , when estimating the depth map of the input image, depth information estimated by the depth estimation branch may be optimized using normal information estimated by the normal estimation branch. In Fig. 4, this approach is referred to as "the normal-guided attention mechanism". With a normal-guided attention mechanism, by using the normal information estimated by the normal estimation branch to optimize the depth information estimated by the depth estimation branch, the estimated depth map can be made more accurate, and the accuracy of the plane detection is improved can do it

Hereinafter, the deep neural network in the plane detection method shown in FIG. 4 will be described with reference to FIG. 5 .

5 is a schematic diagram of operations of a deep neural network according to an exemplary embodiment of the present disclosure. In the example of the deep neural network in FIG. 5 , a feature fusion module and a normal guide attention module may be included in the depth estimation branch and the normal estimation branch.

According to various embodiments, when estimating the depth map of the input image, the feature map of a predetermined resolution obtained by feature extraction of the input image using the feature extractor is generated during depth estimation using the depth estimation branch. The depth feature map of the same resolution and the normal feature map of the same resolution generated during normal estimation using the normal estimation branch are respectively fused to obtain the fused depth feature map and the depth map using the fused normal feature map can be obtained The above operations are operations performed by the feature fusion module.

Here, the predetermined resolution may be, for example, 1/8 resolution and 1/16 resolution, but is not limited thereto. Selecting partial resolution feature maps (such as the 1/8 and 1/16 resolution feature maps above) for fusion improves the accuracy and specific information of the final depth map and the normal map, while improving the training speed of the neural network ( training speed) can be accelerated.

According to various embodiments, the feature fusion module in the depth estimation branch uses a feature extractor to obtain a feature map of a predetermined resolution obtained by feature extraction of an input image and a depth feature map of the same resolution generated during depth estimation. can be fused. According to various embodiments, the feature fusion module in the normal estimation branch uses a feature extractor to extract a feature map of a predetermined resolution obtained by feature extraction of an input image and a normal feature of the same resolution generated during normal estimation. Maps can be fused (refer to the normal-guided attention module in the drawings).

Through the fusion of the feature maps, the accuracy and specific information of the final depth map and the normal map may be improved. For example, spatial details of the final result of the depth estimation and normal estimation may be reconstructed, and thus it may be possible to provide an accurate plane detection result. The fusion of the feature maps may also contribute to accelerating the training speed of the deep neural network.

Hereinafter, operations performed by the feature fusion module may be described with reference to FIG. 6 . First, a feature map of a predetermined resolution obtained using an input feature extractor (e.g., DenseNet encoder) and a depth feature map (hereinafter, for convenience of explanation, is referred to as “feature extractor feature map”). abbreviated) and normal feature maps of the same resolution may be input to the feature fusion module, respectively.

Second, the input feature extractor feature map is divided into three groups (a certain number of processing units) consisting of two-dimensional convolution, batch normalization (BN) and activation function Relu. is not limited, and the three groups can be processed using only an example). wherein the purpose of the first group of processing units is to reduce the number of channels of the input feature map, and the function of the second group of processing units relates to the feature map to be fused from the feature extractor feature map. extracting features, and moving the features from the feature domain of the feature extraction to the feature domain of the features to be fused, a function of the last group of processing units is that the number of output channels is the input depth feature map (or the normal feature map) is adjusted to be equal to the number of channels.

The processed feature map output in the previous step may be added with the corresponding elements of the input depth feature map (or the feature map of the normal branch). Finally, the feature map output from the previous step is processed using another processing unit consisting of 3x3 convolution, batch normalization and activation function Relu to obtain the fused depth feature map (or the normal feature map). can

Such operations performed on a feature map are not limited to being processed by such processing units (eg, the activation function used may not be limited to Relu), and the fusion of the feature maps It should be noted that the specific method is not limited to the example shown in FIG. 6 , and the method of fusion of feature maps in the published information may be used.

Next, operations performed by the normal guide attention module in FIG. 5 will be described. As described above, when estimating the depth map of the input image, the depth information estimated by the depth estimation branch may be optimized using the normal information estimated by the normal estimation branch. For example, as shown in FIG. 8 , in each stage of the depth estimation branch and the normal estimation branch, the output of the depth estimation branch may be optimized using the output of the normal estimation branch. For example, high-frequency information in the normal feature map that reflects the normal information in the normal estimation branch (eg, in the normal feature map in which the normal feature change exceeds a predefined degree) region-related information) may be used to optimize the depth feature map in the depth estimation branch. The depth feature map in the depth estimation branch allows the normal information to change obviously but remain sensitive in regions where the depth information changes slightly, so that the final depth map in these regions It has sharp edges and can provide more accurate three-dimensional points for the plane detection.

7 is a schematic diagram of a region in which normal information is definitely changed but depth information is slightly changed. As shown in FIG. 7 , the area marked with a circle is an area in which normal information is definitely changed but depth information is not changed much, and the high-frequency information in the normal feature map is used to optimize the depth feature map in the depth estimation branch. Therefore, the normal information is definitely changed but the depth information can be kept sensitive in the slightly changed regions, and thus the finally obtained depth map can have sharp edges in these regions.

Returning again to FIG. 5 , optimizing the depth information estimated by the depth estimation branch using the normal information estimated by the normal estimation branch is an area in the normal feature map in which the normal feature change exceeds a predetermined degree. extracting information related to , and optimizing the depth feature map using the information to obtain an optimized depth feature map.

According to various embodiments, the process of extracting information related to a region in which the normal feature change exceeds a predetermined degree from the normal feature map and optimizing the depth feature map using the information includes the following process can do. First, horizontal depth convolution and vertical depth convolution are performed on the normal feature map, respectively, and a horizontal attention map and a vertical attention map for the information are generated using an activation function. can be obtained Second, the optimized depth feature map may be obtained based on the horizontal attention map, the vertical attention map, and the depth feature map. For example, the process of obtaining the optimized depth feature map based on the horizontal attention map, the vertical attention map, and the depth feature map is a process of assigning weights to the horizontal attention map and the vertical attention map; and fusing the horizontal attention map and the vertical attention map to which has been assigned with the depth feature map to obtain the optimized depth feature map.

According to various embodiments, the above optimization operation may be performed by the normal guide attention module illustrated in FIG. 5 .

9 is a schematic diagram illustrating operations performed by a normal-guide attention module according to an exemplary embodiment of the present disclosure. The optimization operation as described above will be exemplified below with reference to FIG. 9 .

및 상기 융합된 노말 특징 지도

이 각각 상기 노말 가이드 어텐션 모듈로 입력된다.Referring to FIG. 9 , first, the fused depth feature map

and the fused normal feature map.

Each of these is input to the normal guide attention module.

는 (-1, 2, -1)의 컨볼루션 커널(convolution kernel)을 가지는 수평 깊이 컨볼루션

와 (-1, 2, -1)^T 의 컨볼루션 커널을 가지는 수직 깊이 컨볼루션

를 각각 사용하여 연산되고, 상기 Tanh 활성화 함수는 수평 어텐션 지도와 수직 어텐션 지도를 각각 획득하는 데 사용된다. 여기에서 상기 수평 어텐션 지도와 수직 어텐션 지도는 상기 노말 특징 지도에서 상기 고주파 정보에 대한 어텐션 지도들이다. 도 9에서 그들은 각각 "수평 고주파 어텐션 지도(a horizontal high-frequency attention map)"와 "수직 고주파 어텐션 지도(a vertical high-frequency attention map)"라고도 칭해질 수 있다. 그 후, 상기 수평 및 수직 어텐션 지도들은 각각 수평 어텐션 결과 및 수직 어텐션 결과를 획득하기 위해 상기

의 입력 깊이 특징 지도를 곱하는데 사용된다 (융합의 방법, 다른 융합 방법들 역시 사용될 수 있다). the normal feature map

is a horizontal depth convolution with a convolution kernel of (-1, 2, -1)

A vertical depth convolution with a convolution kernel of (-1, 2, -1) ^T

, respectively, and the Tanh activation function is used to obtain a horizontal attention map and a vertical attention map, respectively. Here, the horizontal attention map and the vertical attention map are attention maps for the high frequency information in the normal feature map. In FIG. 9 , they may also be referred to as “a horizontal high-frequency attention map” and “a vertical high-frequency attention map”, respectively. Then, the horizontal and vertical attention maps are used to obtain a horizontal attention result and a vertical attention result, respectively.

is used to multiply the input depth feature map of (method of fusion, other fusion methods can also be used).

의 상응하는 엘리먼트들과 더해진다. 요약하면, 이 모듈에 의해 출력되는 상기 어텐션 가이드 깊이 특징 지도는 다음 수학식 2로 설명될 수 있다.Finally, the horizontal attention result and the vertical attention result are the input depth branch feature map using the weight coefficients α and β respectively to obtain the optimized depth feature map.

is added with the corresponding elements of In summary, the attention guide depth feature map output by this module can be described by the following Equation (2).

[Equation 2]

와

은 각각 상기 입력 깊이 특징 지도와 입력 노말 특징 지도이고,

와

는 각각 상기 수평 깊이 컨볼루션과 수직 깊이 컨볼루션이고, tanh는 상기 활성화 함수고, α와 β는 상기 가중치 계수들이다. here,

Wow

are the input depth feature map and the input normal feature map, respectively,

Wow

are the horizontal depth convolution and the vertical depth convolution, respectively, tanh is the activation function, and α and β are the weighting coefficients.

According to various embodiments, high-frequency information in the normal feature map (that is, information related to a region in the normal feature map in which the normal feature exceeds the predetermined degree) is extracted from the example of FIG. 9 and , although the residual sum method is used to optimize the depth feature map, the specific method of using the high frequency information in the normal feature map to optimize the depth feature map is as an example in FIG. It is not limited, and other methods may be used as long as the high frequency information in the normal feature map can be used efficiently.

Sizes (-1, 2, -1) and (-1, 2, -1) ^T of the vertical depth convolution kernel and the horizontal depth convolution kernel in FIG. 9 are only examples, and the vertical depth according to design specifications The size of the convolution kernel and the horizontal depth convolution kernel may be implemented in various ways.

The deep neural network shown in the example of FIG. 5 may include both a feature map fusion module and a normal guide attention module. However, according to various embodiments of the present disclosure, the deep neural network may include only one of the feature map fusion module and the normal guide attention module.

According to various embodiments, in the case where the deep neural network includes both the feature map fusion module and the normal guide attention module (as shown in FIG. 5 ), when estimating the depth map of the input image, the depth estimation branch The normal feature map and the depth feature map used when the depth information estimated by ? is optimized using the normal information estimated by the normal estimation branch include the fused normal feature map obtained by fusion of the feature maps and It may be a fused depth feature map.

According to various embodiments, in a case in which the deep neural network does not include a feature map fusion module, when estimating a depth map of an input image, the depth information estimated by the depth estimation branch is calculated using the normal information estimated by the normal estimation branch. The normal feature map and the depth feature map used when optimized by using the normal feature map and the depth feature map are the normal feature map and the depth feature map without the fusion of the feature maps.

Referring back to FIG. 4 , in the exemplary embodiment of FIG. 4 , after obtaining a depth map and a normal map, the depth map and the normal map may be used to perform plane region segmentation for detecting an initial plane. As described above with reference to step S330 of FIG. 3 , after the depth map is estimated, the depth map can be used to calculate three-dimensional points and depth continuous regions in the input image for the plane estimation, and , the calculated 3D points and information on the depth continuous regions may be used to detect the planar regions in the input image by performing the region segmentation.

According to various embodiments, in a case in which the deep neural network of FIG. 4 includes a normal estimation branch, the region segmentation is performed using the calculated 3D points and information on the depth continuous regions in the input image. The detecting of the planar regions includes calculating a normal map of the input image using the calculated 3D points, and fusing the calculated normal map with a normal map obtained using the normal estimation branch, and and clustering the planar regions by using the fused normal map and information on the depth continuous regions. Such an operation may be referred to as "a plane region segmentation operation".

10 is a schematic diagram illustrating an example of a planar area segmentation operation according to an exemplary embodiment of the present disclosure. Hereinafter, an example of the planar area segmentation operation will be described with reference to FIG. 10 . Referring to FIG. 10 , the result of the plane region segmentation is, first, the 3D points and the depth continuous region in the input image for plane estimation using the depth map output by the deep neural network. , and secondly, a normal map of the input image is calculated using the calculated 3D points, and the calculated normal map is used as a normal map estimated by the deep neural network (eg, the normal map It can be obtained by fusion with the normal map estimated by the estimated branch).

For example, as shown in FIG. 10 , the following fusion method may be employed. In the fusion method, an average normal map is obtained using the calculated normal map and a normal map estimated by a deep estimation network, and information of a normal-consistent region is calculated. and clustering to segment the planar regions using the fused normal map, information on the normal-consistent region, and information on the depth-continuous region.

The fusion of the calculated normal map and the normal map estimated by the deep neural network is not limited to the example method as described above, and other fusion methods such as calculating a weighted average normal map may be used. can

Referring again to FIG. 4 , after the planar areas are detected by the planar area segmentation, alternatively the boundaries of the planar areas are continuous to align the boundaries of the plane areas with the boundaries of real objects of the input image. can be fine-tuned. According to various embodiments, the process of fine-tuning the boundaries of the detected planar regions may include acquiring discrete label values corresponding to each detected planar regions, respectively, based on the individual label values. The method may include converting each detected planar area into a 3D volume, and finely adjusting the planar area based on the converted 3D volume and the input image. According to various embodiments, boundaries of the planar regions may be aligned with boundaries of real objects in the input image.

11 is a schematic diagram illustrating an example of a plane fine-tuning operation according to an exemplary embodiment of the present disclosure. 11 , first, each planar area obtained by the planar area segmentation can be numbered using an individual label value, and then the labeled planar areas are one-hot encoded. encoding) can be used to transform the three-dimensional volume. Finally, the boundaries of the planar regions apply the edge-preserving optimization algorithm (such as a bilateral solver algorithm) using the transformed three-dimensional volume and the input color image. to be fine-tuned, so that they can be aligned with the boundaries of the real-world objects of the scene.

For example, the edge-preserving optimization of the three-dimensional volume may be performed for each layer, and after optimization, the weight of each pixel in different planes is obtained, and then the label with the greatest weight is the fineness of the pixel for each pixel. It is chosen as the adjusted plane label, so it is possible to determine which plane the pixel belongs to.

The plane detection method according to an exemplary embodiment of the present disclosure has been described above with reference to FIGS. 4 to 11 . However, the plane detection method of the present disclosure is not limited to the exemplary embodiment described above.

12 is a schematic diagram illustrating a plane detection method according to another exemplary embodiment of the present disclosure.

Referring to FIG. 12 , a deep neural network (which may also be referred to as a depth estimation network above) includes a feature extractor for extracting features of an input image, and a depth estimation branch for estimating depth information of the input image (the depth). It may include a branch (also referred to as a depth branch). According to various embodiments, the deep neural network includes a normal estimation branch (the normal branch) for estimating normal information of an input image as shown in FIG. 5 . (also referred to as a normal branch) may not be included.

Since the deep neural network used in FIG. 12 includes only the depth estimation branch and the feature extractor and does not include the normal estimation branch, the amount of computation is less, and the depth information can be obtained in real time, so that the mobile devices It is advantageous for applications in

13 is a schematic diagram illustrating operations of the deep neural network according to an exemplary embodiment of the present disclosure.

13 , the feature fusion module may be included in the depth estimation branch. According to various embodiments, when estimating a depth map of an input image, a feature map of a predetermined resolution (eg, a feature map of 1/16 resolution) obtained by feature extraction of the input image using the feature extractor ) may be fused with a depth feature map of the same resolution generated during depth estimation using the depth estimation branch, and the depth map may be generated using the fused depth feature map. Through the fusion of the feature maps, the accuracy and specific information of the final depth map may be improved, and a more accurate plane detection result may be provided.

For example, the feature map generated by the feature extractor is fused with the depth feature map using the method shown in FIG. 7 to obtain the fused depth feature map, and the final depth map is the fused depth map. Generated using depth maps.

The depth map can then be used to detect the planes through the plane area segmentation.

14 is a schematic diagram illustrating a planar region segmentation operation according to another exemplary embodiment of the present disclosure.

Referring to FIG. 14 , the depth map may be used to calculate 3D points and depth-continuous regions in an input image for plane estimation, and the information of the calculated 3D points and the depth-continuous regions is an area It can be used to perform segmentation to detect planar regions of the input image.

For example, as shown in FIG. 14 , the process of detecting planar regions of the input image by performing the region segmentation using the calculated three-dimensional points and information on the depth-continuous regions is the calculation. calculating a normal map of the input image using the obtained three-dimensional points, and clustering the planar regions to segment the planar regions using the calculated normal map and information on the depth-continuous regions. there is.

Referring back to FIG. 13 , the plane detection method according to another exemplary embodiment of the present disclosure may continuously fine-tune the boundaries of the detected plane regions after the plane regions are detected through plane region segmentation, and the plane region Boundaries of the s may be aligned with the boundaries of real objects in the input image. According to various embodiments, a planar fine adjustment technique based on image segmentation may be used, but is not limited thereto.

For example, the planar fine-tuning method as described above with reference to FIG. 11 may also be used. The following fine-tuning method based on the image segmentation may be applied to the plane detection method shown in FIG. 4 to fine-tune the boundaries of the planar regions. For example, the process of fine-tuning the boundaries of the detected planar regions includes: acquiring region information corresponding to each pixel in the input image based on the detected plane regions; Based on the shortest distance on the two-dimensional single-channel image between the detected boundaries of each planar region, the plane weight of each pixel in the 4-channel image composed of the input image and the two-dimensional single-channel image composed of the region information A process of obtaining information, determining similarity between the pixels based on pixel values, the region information, and the plane weight information corresponding to each pixel, and performing the image segmentation based on the similarity between the respective pixels performing to obtain the fine-tuned planar region boundaries.

According to various embodiments, a digital label may be added to each planar area as a number, and the image composed of the planar area number is the same as in the fourth channel of the color image. The four-channel image is segmented, and the result of the segmentation is used to fine-tune the boundaries of the planar regions. Through the plane fine-tuning operation as described above, the detected planes may be aligned with the boundaries of the real objects.

The plane fine-tuning based on the image segmentation will be specifically described below.

이며, 여기서 R은 적색 채널, G는 녹색 채널, B는 청색 채널, P는 평면 영역 레이블 번호로 구성된 채널이다. P 값은 각 픽셀에 상응하는 상기 영역 정보를 반영할 수 있다. 평면 가중치 지도(plane weight map)

는 상기 평면 영역 번호로 구성되는 상기 2차원 단일-채널 이미지를 사용하여 계산된다. First, the individual value labels can be used to number each planar area output by the planar segmentation module, and a two-dimensional single-channel image composed of the planar area number and a scene color image are combined to form the four-channel form an image. At this time, the value of each pixel of the 4-channel is

where R is a red channel, G is a green channel, B is a blue channel, and P is a channel consisting of a flat area label number. The P value may reflect the region information corresponding to each pixel. plane weight map

is calculated using the two-dimensional single-channel image composed of the planar area number.

The weight value of each pixel in the weight map is proportional to the shortest distance in the two-dimensional single-channel image consisting of the plane region number between the pixel and the boundaries of the plane regions. In this way, the plane weight information of each pixel in such a four-channel image is the shortest distance in a two-dimensional single-channel image consisting of the plane region number between each pixel and the boundaries of each detected plane region. can be obtained based on

Finally, the image segmentation is performed based on a pixel similarity function using an image segmentation algorithm (such as an Efficient Graph-based image segmentation algorithm) to perform the fine-tuned Obtain the boundaries of the planar regions. The pixel similarity function is defined as Equation 3 below:

[Equation 3]

는 그 유사성이 계산될 2개의 픽셀들이고,

,

,

,

은 각각 상기 4-채널 이미지에서의

의 픽셀 값들이고,

,

,

,

는 각각 상기 4-채널 이미지에서의

의 픽셀 값들이고,

,

는 각각 상기 가중치 지도

에서

의 가중치 값들이고, f(p1,p2)는 평면 거리 함수 혹은 평면 거리 메트릭(metric) 함수다. here,

is the two pixels whose similarity is to be calculated,

,

,

,

are each in the 4-channel image.

are the pixel values of

,

,

,

are each in the 4-channel image.

are the pixel values of

,

are each of the weighted maps

at

are weight values of , and f(p1,p2) is a plane distance function or a plane distance metric function.

Some exemplary embodiments of the plane detection method of the present disclosure have been described above. Since the plane detection method proposed in the present disclosure detects the planes based on the depth map of the entire input image, the planes may be detected in a texture-less region. In addition, based on this, the accuracy of the planar region can be efficiently improved by the feature map fusion module and/or the normal guide attention module described above, and by additionally performing the plane fine-tuning operation, the detection Boundaries of the obtained planar regions may be aligned with boundaries of real objects in the input image.

Specifically, in AR applications, it is often necessary to place virtual objects in a real scene, but there are many non-textured areas (such as solid color walls, desktops, etc.) in the real scene. By using the plane detection method according to the present disclosure, it may be possible to support detecting planes in the non-textured regions, so that a user can place the virtual objects in the non-textured regions, which Satisfy the user's needs and improve the user experience.

Also, for example in AR games, it is often necessary for virtual objects to interact with real objects. By using the plane detection method according to the present disclosure, a plane detection result aligned with the boundaries of the real objects can be provided, thus improving the accuracy of the interaction of the virtual objects with the real objects, and a game can improve the experience.

The plane detection method of the present disclosure may be applied to AR glasses, smart phones or other AR terminals. In addition, it can be applied to applications such as navigations, exhibitions, trainings, games, and the like.

15 is a block diagram illustrating an apparatus for performing plane detection according to the present disclosure (hereinafter, for convenience of description, it will be referred to as a “plane detection apparatus”).

Referring to FIG. 15 , the plane detection apparatus 1500 may include an image acquisition unit 1510 , an estimation unit 1520 , and a region segmentation unit 1530 . Specifically, the image acquisition unit 1510 may be configured to acquire an input image. The estimation unit 1520 may be configured to extract features of the input image using a deep neural network, and estimate a depth map of the input image based on the extracted features. The region segmentation unit 1530 may be configured to perform region segmentation to detect planar regions in the input image using the depth map. Specifically, for example, the region segmentation unit 1530 calculates three-dimensional points and depth-continuous regions in the input image for plane estimation using the depth map, and the calculated three-dimensional points and The region segmentation is performed to detect planar regions in the input image using information of depth-continuous regions.

According to various embodiments, the plane detection apparatus 1500 may further include a plane boundary fine-tuning unit (not shown), wherein the plane boundary fine-tuning unit determines whether boundaries of the plane regions are determined in the input image. Boundaries of the detected planar regions may be finely adjusted to be aligned with the boundaries of real objects.

Since content and details related to the plane detection operations according to the present disclosure have been described above, a detailed description thereof will be omitted. The corresponding content or details may refer to the description of FIGS. 3 to 14 .

A plane detection method and a plane detection apparatus according to embodiments of the present disclosure have been described above with reference to FIGS. 1 to 15 . However, each unit of the apparatus shown in FIG. 15 may be configured with software, hardware, firmware, or a combination of the above items that perform a specific function, respectively. For example, these units may correspond to a dedicated integrated circuit, may also correspond to pure software code, and may also correspond to a module combining the software and hardware. For example, the device described with reference to FIG. 15 may be a PC computer, a tablet device, a personal digital assistant, a smart phone, a web application, or other devices capable of executing program instructions, but is not limited thereto. does not

In addition, when the plane detection device 1500 is described above, the plane detection device 1500 is divided into units for performing corresponding processing, but the processing performed by each unit is performed by the plane detection device. It is apparent to those skilled in the art that the specific units may be performed in a case where no division is performed or there is no clear boundary between the units.

The apparatus described with reference to FIG. 15 is not limited to including the units described above, but some other units (eg, storage unit, data processing unit, etc.) may be added as needed; Alternatively, such units may be combined.

As shown in FIG. 16 , a semantic-aware plane detection system suitable for augmented reality proposed by the present disclosure may include three parts.

The first part is a deep neural network for acquiring long life (or new) information, the second part is a planar region segmentation module, and the third part is a plane boundary fine-tuning module.

In a first implementation manner, the deep neural network is used to estimate the depth information and the normal information. A normal-guided attention module is designed in the network structure of this module, which can use the high-frequency information in the normal feature map to sharpen the estimated depth map at the boundaries more, thus it It is possible to provide a more accurate dense three-dimensional point cloud for plane detection, and the plane region segmentation module performs clustering and segmentation on the dense three-dimensional point cloud including the normal information, and thus a more accurate and robust plane area can be obtained. The plane boundary fine-tuning module may use the edge-preserving optimization algorithm to fine-tune the obtained plane regions to align with the boundaries of the real objects, and thus obtain a semantic-aware plane.

In a second implementation manner, first, the input image is input into the estimation network to obtain depth information of the entire image, and then the obtained depth map is subjected to the plane region segmentation to obtain a result of the plane region segmentation. input into a module, and finally, the boundaries of the planar regions can be better aligned with the boundaries of the real objects using the plane fine-tuning method based on the image segmentation. Here, the deep estimation network can provide sufficient information for the next planar area segmentation module, and thus can calculate planar areas on the entire image including the non-textured area. Here, the plane region segmentation module operates together with the next plane fine-tuning module based on image segmentation to better align the boundaries of the detected plane regions with the boundaries of the real objects of the scene, and the plane fine-tuning algorithm This is more efficient.

In addition, the plane detection method according to the present disclosure may be recorded in a computer-readable recording medium. Specifically, according to the present disclosure, it is possible to provide the computer-readable recording medium for recording program instructions, which, when executed by a processor, enables the processor to execute the plane detection method as described above. can Examples of the computer-readable recording media include magnetic media (eg, hard disk, floppy disk, magnetic tape), optical media (eg, CD-ROM and DVD), magneto-optical ) media (eg, an optical disk) and a hardware device specially configured to store and execute the program instructions (eg, read only memory (ROM), random access memory (RAM) ), flash memory, etc.). Further, according to the present disclosure, there may be provided an electronic device including a processor and a memory for storing program instructions, wherein the program instructions, when executed by the processor, cause the processor to perform the above-described plane detection method. make it run Examples of the program instructions include, for example, machine code generated by a compiler and a file containing high-level code that can be executed by a computer using an interpreter. include

In addition, some operations of the plane detection method according to an exemplary embodiment of the present application may be implemented in a software manner, and some operations may be implemented in a hardware manner, and these operations may be performed by a combination of the software and hardware. may be implemented.

In addition, the present disclosure also provides an electronic device including a processor and a memory for storing program instructions, wherein the program instructions, when executed by the processor, cause the processor to execute the plane detection method of the present disclosure. For example, the electronic device may be a PC computer, a tablet device, a personal digital assistant, a smart phone, or other devices capable of executing the set of instructions. Here, the electronic device is not necessarily a single electronic device, and may be an aggregation of devices or circuits capable of individually or jointly executing the above instructions (or a set of instructions). . The electronic device may also be part of an integrated control system or system administrator, or it may consist of a portable electronic device interconnected locally or remotely (eg, via wireless transmission) by an interface.

While the present disclosure has been particularly shown and described with reference to exemplary embodiments of the present disclosure, those skilled in the art will take form and form without departing from the spirit and scope of the disclosure as defined by the appended claims. It should be understood that various changes in details may be made.

Claims (16)

A method for performing plane detection, comprising:
acquiring an input image;
extracting features of the input image using a deep neural network, and estimating a depth map of the input image based on the extracted features; and
and detecting planar regions in the input image by performing region segmentation using the depth map. The method of claim 1,
The deep neural network includes a feature extractor for extracting the features of the input image, a depth estimation branch for estimating depth information of the input image, and normal information of the input image. a normal estimation branch for estimating;
When estimating the depth map of the input image, the depth information estimated by the depth estimation branch is optimized using the normal information estimated by the normal estimation branch. 3. The method of claim 2,
When estimating the depth map of the input image, a feature map of a predetermined resolution obtained by feature extraction of the input image using the feature extractor is the same resolution generated when estimating the depth using the depth estimation branch to obtain the depth map using the fused depth feature map and the fused normal feature map by fusion with a depth feature map of and a normal feature map of the same resolution generated during normal estimation using the normal estimation branch Way. 3. The method of claim 2,
The process of optimizing the depth information estimated by the depth estimation branch using the normal information estimated by the normal estimation branch includes:
extracting information related to a region in which normal feature change exceeds a predetermined degree from the normal feature map, optimizing the depth feature map using the extracted information, and obtaining an optimized depth feature map; How to include. 5. The method of claim 4,
The process of extracting information related to a region in which the normal feature change exceeds the predetermined degree from the normal feature map, and optimizing the depth feature map using the extracted information,
A process of performing horizontal depth convolution and vertical depth convolution on the normal feature map, respectively, and obtaining a horizontal attention map and a vertical attention map for the information using an activation function ; and
and obtaining an optimized depth feature map based on the horizontal attention map, the vertical attention map, and the depth feature map. 6. The method of claim 5,
The process of obtaining an optimized depth feature map based on the horizontal attention map, the vertical attention map, and the depth feature map,
assigning weights to the horizontal attention map and the vertical attention map; and
and fusing the weighted horizontal attention map and the weighted vertical attention map with the depth feature map to obtain an optimized depth feature map. The method of claim 1,
The process of detecting the planar regions in the input image by performing the region segmentation using the depth map,
3D points and depth-continuous regions are calculated from the input image for plane estimation using the depth map, and information on the calculated 3D points and the depth continuous regions and detecting the planar regions in the input image by performing the region segmentation using 8. The method of claim 7,
The process of detecting the planar regions in the input image by performing the region segmentation using the calculated 3D points and information on the depth continuous regions includes:
calculating a normal map of the input image using the calculated three-dimensional points, and fusing the calculated normal map with a normal map estimated by the deep neural network; and
and clustering to segment the planar regions using the fused normal map and information on the depth continuous regions. 8. The method of claim 7,
The process of detecting the planar regions in the input image by performing the region segmentation using the calculated 3D points and information on the depth continuous regions includes:
calculating a normal map of the input image using the calculated 3D points; and
and clustering to segment the planar regions using the calculated normal map and information on the depth continuous regions. 10. The method of claim 9,
The deep neural network includes an extractor for extracting features of the input image and a depth estimation branch for extracting depth information of the input image,
When estimating the depth map of the input image, a feature map of a predetermined resolution obtained by feature extraction of the input image using the feature extractor is a feature map of the same resolution generated when depth estimation using the depth estimation branch. A method of generating the depth map using the fused depth feature map by fusion with a depth feature map. The method of claim 1,
and fine-tuning the boundaries of the detected planar regions so that the boundaries of the planar regions are aligned with the boundaries of real objects in the input image. 12. The method of claim 11,
The process of fine-tuning the boundaries of the detected planar regions includes:
acquiring discrete label values corresponding to each of the detected planar regions, respectively;
converting each detected planar region into a three-dimensional volume based on the individual label value; and
A method of fine-tuning the planar regions based on the transformed three-dimensional volume and the input image so that the boundaries of the planar regions are aligned with the boundaries of real objects in the input image. 12. The method of claim 11,
The process of fine-tuning the boundaries of the detected planar regions includes:
obtaining area information corresponding to each pixel in the input image based on each detected planar area;
Based on the shortest distance on the two-dimensional single-channel image between each pixel and the detected boundaries of each planar region, each obtaining plane weight information of pixels; and
The fine-tuned planar area is determined by determining similarity between pixels based on a pixel value, the area information, and the plane weight information corresponding to each pixel, and performing image segmentation based on the similarity between the respective pixels A method comprising obtaining boundaries. An apparatus for performing plane detection, comprising:
an image acquiring unit, configured to acquire an input image;
an estimation unit, configured to extract features of the input image and, using a deep neural network, estimate a depth map of the input image based on the extracted features; and
and a region segmentation unit configured to detect planar regions in the input image by performing region segmentation using the depth map. An electronic device comprising a processor and a memory for storing program instructions, the electronic device comprising:
The electronic device of claim 1 , wherein the program instructions, when executed by the processor, cause the processor to perform the method of claim 1 . A computer-readable recording medium storing program instructions,
The recording medium of claim 1 , wherein the program instructions, when executed by a processor, cause the processor to perform the method of claim 1 .

Images