KR102169243B1 - Semantic segmentation method of 3D reconstructed model using incremental fusion of 2D semantic predictions - Google Patents

Abstract

The present invention is a three-dimensional restoration model through gradual mixing of two-dimensional semantic segmentation information that performs three-dimensional restoration from a continuous color and depth image stream from a diffusion depth image camera and gradual semantic segmentation of the reconstructed model. In the semantic segmentation method of the three-dimensional reconstruction model according to the present invention, the semantic segmentation method of (a) a depth image corresponding to the color image (RGB) of each input image (Depth ) To perform deep learning-based semantic segmentation by pixel to obtain probability information according to an object class for each pixel; (b) updating the probability information obtained for each pixel in the voxel grid by raycasting; (c) extracting a mesh model from a voxel grid by a marching cube algorithm; And (d) performing semantic division of the 3D reconstructed model by selecting a class having the highest probability for each vertex in the mesh model.

Description

Semantic segmentation method of 3D reconstructed model using incremental fusion of 2D semantic predictions}

The present invention relates to a method of mixing semantic segmentation information of multiple 2D images when performing 3D semantic segmentation of a 3D model reconstructed from a color and depth image stream of an RGBD camera.

Three-dimensional restoration means acquiring three-dimensional position and color information on an object of interest or environment by using various scanning devices such as a laser scanner and a structured light-based depth camera.

With the advent of low-end depth cameras such as Kinect cameras and the development of various algorithms, real-time restoration is possible even without expensive scanning equipment at the technical level that only small-sized objects (eg, people) could be restored in non-real time. It has become possible to show the level of technological maturity. Starting with the popular KinectFusion technology as a real-time 3D restoration technology, large-scale restoration technologies such as Voxel Hashing and BundleFusion, which solved the limitation of the restoration space size, appeared one after another.

The general process performed to perform 3D restoration using a depth camera is as follows. First, the pose of the camera (or the pose of a rotating object) is calculated every frame. As a method of obtaining a pose, a variant of the ICP (Iterative Closest Point), which considers the plane-to-point error, is mainly used, and the input of this algorithm is the current depth image information and the raycasting depth information of the model. Is used.

Unlike expensive scanners, popular depth cameras contain a lot of noise in the depth image value, and to solve this, the average technique in the expression of TSDF (Truncated Signed Distance Function) is used. TSDF values are calculated from images continuously coming from the depth camera and stored in a pre-configured voxel grid. If there is already stored TSDF values, a new TSDF value is calculated using the sum of the existing values and weights. After the scan is completed, the final model mesh is extracted by applying the Marching cubes algorithm to the voxel grid where the TSDF values are stored.

In the past, several studies have been conducted to classify objects in 3D restoration models.

Liangliang Nan, Ke Xie, and Andrei Sharf's "A search-classify approach for cluttered indoor scene understanding," [Reference 1], is a fragmented patch by performing over-segmentation on a point cloud type model. (patch) is created, and the classification likelihood is calculated for the configured area by accumulating it again. Using this probability, we perform replacement of the current partial model with the most similar neat model. However, since it uses features on the point cloud, it is less accurate and can only be applied to a fairly limited class.

Dai A., Chang AX, Savva M., Halber M., Funkhouser T., Niessner M's "Scannet: Richly-annotated 3D reconstructions of indoor scenes," [Reference 2], is a voxel of the entire scene unlike [Reference 1]. Using the converted data as an input of a 3D CNN (Convolution Neural Network), a result of labeling each voxel as an output is obtained.

"An interactive approach to semantic modeling of indoor scenes with an RGBD camera," by Tianjia Shao, Weiwei Xu, Kun Zhou, Jingdong Wang, Dongping Li, and Baining Guo, describes semantic segmentation using CRF in RGBD images. Apply and classify the divided objects into a random regression forest. Also, in scene registration, SIFT (Scale-Invariant Feature Transform), RANSAC (RANdom SAmple Consensus) and semantic segmentation label information are used. However, this technique is difficult to fusion with the large-scale 3D reconstruction algorithm of the KinectFusion method.

Liangliang Nan, Ke Xie, and Andrei Sharf. "A search-classify approach for cluttered indoor scene understanding," ACM Trans. on Graph., 31(6):137:1-137:10, 2012. Dai A., Chang A. X., Savva M., Halber M., Funkhouser T., Niessner M. "Scannet: Richly-annotated 3D reconstructions of indoor scenes," IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017. Tianjia Shao, Weiwei Xu, Kun Zhou, Jingdong Wang, Dongping Li, and Baining Guo. "An interactive approach to semantic modeling of indoor scenes with an RGBD camera," ACM Trans. on Graph., 31(6):136:1-11, 2012. Seong-Jin Park, Ki-Sang Hong, Seungyong Lee. "RDFNet: RGB-D Multi-Level Residual Feature Fusion for Indoor Semantic Segmentation," The IEEE International Conference on Computer Vision (ICCV), 2017, pp. 4980-4989. Dai A., Niessner M., Zollhofer M., Izadi S., Theobalt C.: Bundlefusion: Real-time globally consistent 3D reconstruction using on-the-fly surface reintegration. ACM Transactions on Graphics (TOG) 36, 3 (2017), 24 Lorensen W. E., Cline H. E.: Marching cubes: A high resolution 3D surface construction algorithm. In ACM Transactions on Graphics (TOG) (1987), vol. 21, ACM, pp. 163-169

The present invention provides a method for gradual mixing of two-dimensional semantic segmentation information for performing a gradual semantic segmentation of a model reconstructed simultaneously with three-dimensional restoration from a continuous color and depth image stream from a depth image camera for distribution. For that purpose.

In order to achieve the above object, the semantic segmentation method of a three-dimensional reconstruction model through gradual mixing of two-dimensional semantic segmentation information according to the present invention is (a) for multiple two-dimensional input images, the color image of each input image ( Obtaining probability information according to an object class for each pixel by performing deep learning-based semantic segmentation by pixel using a depth image corresponding to RGB); (b) updating the probability information obtained for each pixel in the voxel grid by raycasting; (c) extracting a mesh model from the voxel grid by a marching cube algorithm; And (d) performing semantic division of the 3D reconstructed model by selecting a class having the highest probability for each vertex in the mesh model.

In the semantic segmentation method of the three-dimensional reconstruction model through gradual mixing of the two-dimensional semantic segmentation information, step (b) includes the distance from the camera of the object and the front/background boundary in probability information according to the object class of each pixel. By adding a weight determined according to a weighted average, probability information according to an object class stored in a voxel corresponding to a corresponding pixel is weighted and updated.

In the semantic segmentation method of the three-dimensional reconstruction model through the gradual mixing of the two-dimensional semantic segmentation information, the 20 probabilities of each vertex of the mesh model in step (c) are bilinear interpolation. It is characterized by being determined through.

In the semantic segmentation method of the three-dimensional reconstruction model through the gradual mixing of the two-dimensional semantic segmentation information, the probability information of pixels, the weights, and the probability of vertices are each a vector having a dimension equal to the number of object classes.

의 t번째 프레임까지 통합된 부류 확률은 수학식In the semantic segmentation method of a three-dimensional restoration model through gradual mixing of the two-dimensional semantic segmentation information, each voxel in step (b)

The class probability integrated up to the t-th frame of

와

는 각각 복셀

의 t-1번째 프레임까지 통합된 부류 확률과 신뢰도 가중치이고,

와

는 각각 t번째 프레임에서 픽셀 p의 부류 확률과 신뢰도 가중치이다)에 의해 산출되고,(here

Wow

Is each voxel

Is the class probability and reliability weight integrated up to the t-1th frame of,

Wow

Is calculated by the class probability and reliability weight of pixel p in each t-th frame),

는 수학식Confidence weight

Is the equation

는 깊이 기반 정확도 가중치이고,

는 전·배경 경계 오정렬 가중치이다)에 의해 산출되는 것을 특징으로 한다.(here

Is the depth-based accuracy weight,

Is the weight of the misalignment of the front/background boundary).

는 수학식In the semantic segmentation method of the three-dimensional restoration model through the gradual mixing of the two-dimensional semantic segmentation information, the pre-background boundary misalignment weight

Is the equation

,

은 각각 현재 픽셀

의 깊이 값, 픽셀

위치를 중심으로 한 윈도우 안에서 최소 깊이 값 및 최대 깊이 값이고,

,

는 양의 상수이다)에 의해 산출되는 것을 특징으로 한다.(here

,

Each is the current pixel

Depth value in pixels

The minimum and maximum depth values within the window centered on the location,

,

Is a positive constant).

In order to achieve the above object, the computer-readable recording medium according to the present invention records a program for executing the semantic segmentation method of the three-dimensional restoration model through the gradual mixing of the two-dimensional semantic segmentation information. .

In order to achieve the above object, the computer program according to the present invention is stored in a medium in order to execute the semantic segmentation method of the three-dimensional reconstructed model through the gradual mixing of the two-dimensional semantic segmentation information.

According to the present invention, the three-dimensional semantic segmentation information for the three-dimensional model is finally obtained by using semantic segmentation probability information in a two-dimensional image, just as the geometry is gradually completed in three-dimensional restoration using an RGBD camera. If it supports sufficient computational power, it can be easily transferred to the real-time method, and it also solves the problems encountered by other methodologies (resolution decreases in large-scale restoration models due to memory limitations) and expresses delicate geometry even in large-scale restoration models. It is possible to maintain semantic division results while maintaining.

This semantic segmentation result can be used when interaction with various users is required, such as augmented reality and virtual reality. As a simple application example, in the relocation of the indoor interior structure, the user can freely move the desired object (for example, chair, desk) by separating it from the original mesh using the result of semantic division.

1 is a diagram for explaining the overall flow of a semantic segmentation method of a three-dimensional reconstruction model through gradual mixing of two-dimensional semantic segmentation information according to the present invention.
2 shows a weight map used for mixing semantic segmentation information.
3 shows prediction accuracy according to distance of the latest deep learning-based semantic segmentation technology.
4 shows the results of probabilistic visualization and final semantic division of major classes.
5 shows a result of semantic division according to the present invention in the restoration of an indoor environment.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

The overall flow of a semantic segmentation method of a 3D restoration model through gradual mixing of two-dimensional semantic segmentation information according to the present invention is shown in FIG.

First, by performing deep learning-based semantic segmentation using color and depth input images, probability information according to object class is obtained for each pixel. Assuming that 20 classes are classified, each pixel contains the probability of occurrence of 20 classes. This information is stored and mixed in voxels corresponding to each pixel according to raycasting. When mixed, the weight of the two-dimensional semantic segmentation result is adaptively determined according to the distance of the object from the camera and the front/background boundary.

When the above processing for all image stream data is completed, a mesh model is extracted using a Marching cubes (refer to Non-Patent Document 6) algorithm. At this time, the 20 probabilities of each vertex of the mesh in the marching cubes process are determined through bilinear interpolation.

All of the previously determined probabilities are results obtained by performing semantic segmentation by looking at only some of the shape and color of the object in the 2D image.

To classify each vertex in the final mesh, simply select the class with the highest probability.

In the mixing process, an update as shown in Equation 1 is performed for each class information of each voxel.

와

는 각각 복셀

의 t-1번째 프레임까지 통합된 부류 확률과 신뢰도 가중치를 의미한다.

와

는 각각 t번째 프레임에서 픽셀 p의 부류 확률과 신뢰도 가중치를 나타낸다. 만약 분류할 부류가 20개라면

는 20차원 벡터가 된다. here

Wow

Is each voxel

It means the integrated class probability and reliability weights up to the t-1th frame of.

Wow

Represents the class probability and reliability weight of the pixel p in each t-th frame. If there are 20 categories to be classified

Becomes a 20-dimensional vector.

로 처리하게 되면 여전히 부정확한 결과를 얻을 수 있다. 이를 경감하기 위해 본 발명은 수학식 2와 같은 적응적 가중치를 활용한다. The class probability of the pixels in each frame is equally weighted (

If you do it, you can still get inaccurate results. To alleviate this, the present invention utilizes an adaptive weight as shown in Equation 2.

는 깊이 기반 정확도 가중치이고,

는 전·배경 경계 오정렬 가중치이다. 관련 가중치 맵은 도 2에서 확인할 수 있다.here

Is the depth-based accuracy weight,

Is the weight of the misalignment of the front/background boundary. The related weight map can be found in FIG. 2.

는 이를 반영하기 위한 가중치이다.In general, a convolution neural network (CNN) has a fixed receptive field size, and the accuracy of semantic division varies according to the size of an object in an image. Depth-based accuracy weighting

Is the weight to reflect this.

는 이와 같이 피팅된 다항식 함수를 통해 결정된다. In order to calculate the base accuracy weight of the depth image, the performance of the two-dimensional semantic segmentation result of RDFNet (see Non-Patent Document 4), which is a deep learning-based methodology used in the present invention, must be calculated. First, the depth and color images of the training set are received as inputs from RDFNet and the semantic segmentation results are estimated. Class information (one of 20 classes) estimated for each pixel is stored in the semantic segmentation image, and the average performance of the semantic segmentation result for the depth value can be calculated by comparing it with the ground truth. have. Drawing this as a graph is like the blue solid line in FIG. 3. This is a graph of discrete depth values, and therefore, fitting is performed with a fourth-order polynomial to obtain a weight graph with noise-removed weights for continuous depth values. The result is indicated by a solid yellow line in FIG. 2. Depth-based accuracy weighting

Is determined through the polynomial function fitted in this way.

The semantic segmentation result mainly depends on the color image, and the depth image is used as a supplement. However, even if the RGBD camera is well calibrated, there is still a misalignment between the color and depth images. In particular, this misalignment deepens between the foreground (object) and the background (mainly the wall and floor). This misalignment results in incorrect labeling in voxels around the boundary. In order to alleviate this problem, the present invention detects an edge of a depth image and generates a weight based on it.

The method of determining the edge and determining the weight is as follows. If the deviation is larger than a predetermined constant value by covering a 7x7 window in the pixel, it is detected as an edge, and a weight is determined through Equation 3.

,

은 각각 현재 픽셀

의 깊이 값, 픽셀

위치를 중심으로 한 윈도우 안에서 최소 깊이 값 및 최대 깊이 값을 의미한다.

,

는 양의 상수를 의미한다. here

,

Each is the current pixel

Depth value in pixels

It means the minimum depth value and the maximum depth value in the window centered on the position.

,

Means positive constant.

When the mixing process is finished, there are 20 probabilities mixed through Equation 1 for each voxel. Since the current scene is expressed in the form of a voxel grid, a marching cubes algorithm is required to convert it into a mesh form expressed in normal graphics. In a general case, the voxel contains a TSDF value or a color value.In the present invention, since there are 20 probabilities, the probability values are bilinearly interpolated in the marching cubes process and finally 20 probabilities of each vertex of the mesh Determine the value.

When a mesh including 20 class probabilities is extracted through the preceding process, a probability map for the main classes can be obtained as shown on the left side of FIG. 5, and the final semantic division result selects the class with the highest probability for each vertex. It can be obtained by doing (see the right side of Fig. 5).

5 shows a result of semantic division according to the present invention in the restoration of an indoor environment.

Meanwhile, the above-described embodiments of the present invention may be recorded on a medium used in general-purpose computers including personal computers. The medium is a magnetic recording medium (e.g., ROM, floppy disk, hard disk, etc.), optical reading medium (e.g., CD-ROM, DVD, etc.), and electrical recording medium (e.g., flash memory, memory stick, etc.) Includes recording media such as.

So far, the present invention has been looked at around its preferred embodiments. Those of ordinary skill in the art to which the present invention pertains will be able to understand that the present invention can be implemented in a modified form without departing from the essential characteristics of the present invention. Therefore, the disclosed embodiments should be considered from an illustrative point of view rather than a limiting point of view. The scope of the present invention is shown in the claims rather than the foregoing description, and all differences within the scope equivalent thereto should be construed as being included in the present invention.

Claims (8)

(a) For multiple 2D input images, by using the color image (RGB) of each input image and the corresponding depth image (Depth), deep learning-based semantic segmentation is performed for each pixel according to the object class. Obtaining probability information;
(b) updating the probability information obtained for each pixel in the voxel grid by raycasting;
(c) extracting a mesh model from the voxel grid by a marching cube algorithm; And
(d) performing semantic division of the three-dimensional reconstructed model by selecting a class having the highest probability for each vertex in the mesh model; and
The semantic segmentation method of a three-dimensional restoration model through gradual mixing of two-dimensional semantic segmentation information, characterized in that the probability of each vertex of the mesh model in step (c) is determined through bilinear interpolation. The method of claim 1, wherein step (b)
A weighted average of probability information according to the object class stored in the voxel corresponding to the corresponding pixel is updated by adding a weight determined according to the distance from the camera and the front/background boundary to the probability information according to the object class of each pixel. Semantic segmentation method of 3D reconstruction model through gradual mixing of 2D semantic segmentation information. delete The method of claim 1, wherein the probability information of pixels, the weights, and the probability of vertices are vectors having dimensions equal to the number of object classes, respectively, by gradual mixing of two-dimensional semantic segmentation information. . The method of claim 2,
Each voxel in step (b)

The class probability integrated up to the t-th frame of

(here

Wow

Is each voxel

Is the class probability and reliability weight integrated up to the t-1th frame of,

Wow

Is the class probability and reliability weight of pixel p in each tth frame)
Is calculated by,
Confidence weight

Is the equation

(here

Is the depth-based accuracy weight,

Is the weight of the misalignment of the front/background boundary)
Semantic segmentation method of a three-dimensional restoration model through gradual mixing of two-dimensional semantic segmentation information, characterized in that calculated by The weight of claim 5, wherein the misalignment weight of the front/background boundary

Is the equation

(here

,

Each is the current pixel

Depth value in pixels

The minimum and maximum depth values within the window centered on the location,

,

Is a positive constant)
Semantic segmentation method of a three-dimensional restoration model through gradual mixing of two-dimensional semantic segmentation information, characterized in that calculated by Any one of claims 1, 2, and 4 to 6 can be read by a computer that records a program for executing the semantic segmentation method of a 3D restoration model through gradual mixing of the 2D semantic segmentation information. Recording medium. A computer program stored in a medium to execute the semantic segmentation method of a three-dimensional reconstructed model through gradual mixing of the two-dimensional semantic segmentation information of any one of claims 1, 2, and 4 to 6.

Images