KR101970488B1 - RGB-D Multi-layer Residual Feature Fusion Network for Indoor Semantic Segmentation - Google Patents

Abstract

The present invention relates to a network apparatus using one indoor color image and a depth image thereof to perform deep learning-based pixel unit semantic segmentation. According to the present invention, the network apparatus comprises: a first residual network module including a plurality of steps, which are connected from an upper step to a lower step and formed with a convolutional neural network (CNN) for each step, to extract gradual feature information from a color image; a second residual network module including a plurality of steps, which are connected from an upper step to a lower step and formed with a CNN for each step, to extract gradual feature information from a depth image; and a multimodal feature fusion network (MMFNet) module fusing the feature information extracted from each step of the first and second residual network modules. The MMFNet module is formed by sequentially connecting: a convolution block reducing dimensions of the feature information with respect to color and depth feature information extracted from a step corresponding to the first and second residual network modules for each step in order to smoothen a rapid increase of a parameter; two residual convolution units performing nonlinear modification for shape combination; and a convolution block adaptively combining feature information of different types and adjusting a scale of a feature value for addition. Moreover, scaled color feature information and scaled depth feature information are combined by addition. The present invention effectively extracts and combines various dimensions of feature information at the same time such that the feature information can be efficiently learned.

Description

Technical Field [0001] The present invention relates to a color-depth image, and more particularly, to an RGB-D multi-layer Residual Feature Fusion Network for Indoor Semantic Segmentation

The present invention relates to a network apparatus that performs deep learning based pixel unit semantic segmentation utilizing a single indoor color image (RGB) and a depth image thereof.

The technique of recognizing the image in semantically divided pixels is a technology that can be the basis of many image processing techniques and can provide particularly important image information in fields such as autonomous driving.

RefineNet is the latest technology for semantic segmentation of general color images (see Figure 1). Based on Kaiming He's "Deep Residual Larning for Image Recognition (Residual Network)", which was used in existing image classification technology, RefineNet extracts characteristics of various dimensions from images and combines them step by step to create clearer and more accurate semantic Design a network module that can obtain segmentation.

In FIG. 1, (a) is a network structure for semantic division, (b) is a detailed configuration diagram of a refineNet block, and (c) to (e) are detailed configuration diagrams of sub-modules of RefineNet. In FIG. 1, Conv performs convolution, ReLU performs a rectified linear unit activation function, and Pool performs a pooling operation.

However, it is difficult to obtain good recognition performance due to high image complexity in indoor image. On the other hand, it is easy to acquire depth image from equipment such as Microsoft Kinect, and it can be used with color image to improve accuracy.

Referring to FIG. 2, existing depth learning based technologies using a color image and a depth image together are as follows. 2 (a) shows a method of simply connecting a color image and a depth image as an input of a network module, FIG. 2 (b) shows a method of extracting characteristics from each image, (D) describes the structure of C. Hazirbas's "Fusenet: Incorporating depth into semantic segmentation via fusionbased cnn architecture", J. Wang's "Learning common and specific features for rgb-d semantic segmentation with deconvolutional networks" will be. Here, 'C', 'T' and '+' mean concatenation, transformation, and element-wise summation, respectively.

J.Wang's network module first extracts characteristics from color and depth images, respectively. After dividing the image into common characteristics and specific characteristics of each image in a specific layer, the result is passed through the network module, and the final segmentation result is obtained by combining the characteristics. C. Hazirbas extracts the characteristics of each image from the encoder part, which is the front part of the network module, and sequentially adds the characteristics. The obtained characteristic map is passed through the deconvolution network module, Obtain the result.

These networks are difficult to utilize the characteristic information of various layers that can be extracted from each image through the deep learning and have a disadvantage in that it is also necessary to directly determine a part to combine each characteristic information. Therefore, it may be difficult to learn a network module capable of extracting each image characteristic effectively.

 Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.

In the present invention, when recognizing an image through semantic division from an indoor color image and a depth image, it is possible to efficiently learn and effectively obtain a semantic division result by extracting and combining characteristic information of various dimensions effectively Depth image by using a stepwise residual characteristic combining network device and a stepwise residual characteristic combining network device for a color depth image for an indoor semantic partitioning capable of achieving an indoor color depth image.

In order to achieve the above object, a step-and-repeat residual characteristic combining network apparatus for color depth image for indoor semantic partitioning according to the present invention utilizes a single indoor color image (RGB) and a depth image A network apparatus for performing a deep learning-based pixel-by-pixel semantic division, the network apparatus comprising a plurality of steps of a CNN (Convolutional Neural Network) A first residual network module; A second residual network module which is composed of a plurality of steps of a CNN (Convolutional Neural Network), which leads from a higher stage to a lower stage, and extracts the stepwise characteristic information from the depth image; And a multi-characteristic combination network module (MMFNet) module for combining extracted characteristic information for each step of the first and second residual network modules, A convolution for alleviating the sudden increase in the parameter by reducing the dimension of the characteristic information for each of the color characteristic information and the depth characteristic information extracted at the corresponding step of the first residual network module and the second residual network module, block; Two residual call polarization units for performing nonlinear transformation for shape combining; And a convolution block for adaptively combining other types of characteristic information and adjusting a scale of a characteristic value for addition are connected in this order and the scaled color characteristic information and depth characteristic information are additionally combined .
A stepwise residual characteristic combining network device for color depth images for interior semantic segmentation, the device further comprising a residual pooling block for combining contextual information with characteristic information in which color characteristic information and depth characteristic information are combined, do.

delete

delete

delete

A stepwise residual characteristic combining network device for color depth images for indoor semantic segmentation, the method comprising the steps of: inputting, at each step, refined characteristic information of a lower level and characteristic information combined at a corresponding step of the multi- And RefineNet for purifying the purified water.

In order to achieve the above object, the semantic separation method of the indoor color depth image using the stepwise residual characteristic combining network device according to the present invention uses a single indoor color image (RGB) and a depth image thereof (A) a first residual network module consisting of a plurality of steps of a CNN (Convolutional Neural Network) that leads from a higher level to a lower level, Extracting stepwise characteristic information by using an image as an input; (b) extracting stepwise characteristic information by inputting a depth image by a second residual network module, which is composed of a plurality of steps of a CNN (Convolutional Neural Network), from an upper step to a lower step and each step; And (c) combining the extracted characteristic information for each step of the multi-characteristic coupling network module in each of the first and second residual network modules, wherein the step (c) comprises the steps of: Wherein the multi-characteristic combining network module reduces the dimension of the characteristic information for each of the color characteristic information and the depth characteristic information extracted at the corresponding steps of the first and second residual network modules in each step Mitigating a surge in parameters; Performing a non-linear transformation for shape combining; And adjusting the scale of the characteristic value for the addition, wherein the scaled color characteristic information and the depth characteristic information are additionally combined.

delete

delete

The method of claim 1, wherein the step (c) comprises the step of combining the information on the context with the characteristic information in which the color characteristic information and the depth characteristic information are combined .

The method of claim 1, further comprising the steps of: (d) refining the color depth image by using the stepwise residual characteristic combining network device, And refining the inputted characteristic information as an input.

In order to achieve the above object, a computer-readable recording medium according to the present invention records a program for causing a computer to execute a method of semantic division of an indoor color depth image using the stepwise residual characteristic combining network device .

In order to achieve the above object, a computer program according to the present invention is stored in a medium for executing a method of semantic division of indoor color depth images using the stepwise residual characteristic combining network device in a computer.

According to the present invention, not only is the function of extracting, combining, and improving characteristics of a color-depth image sequentially in order to achieve an indoor semantic division, but also a function To be automatically learned.

1 is a structural diagram showing the structure of RefineNet of semantic division on a color image.
FIG. 2 shows the structure of existing deep-learning-based techniques using a color image and a depth image together.
FIG. 3 shows the overall structure of a step-by-step residual characteristic combining network device for color depth images for interior semantic segmentation.
4 is a diagram showing a general CNN structure (a) having two layers and a residual network structure (b) having a skip connection structure added to two layers.
FIG. 5 is a structural diagram illustrating a portion corresponding to the MMFNet of FIG. 3 in detail.
Figure 6 shows the qualitative results of the semantic partitioning.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

The present invention relates to a method and apparatus for extracting characteristic information of a step (step 1) from a color (RGB) image and a depth image, a part for effectively combining the extracted characteristic information for each step (step 2) (3 levels) that improve the characteristics information more clearly.

From the final property information map of the network thus constructed, the result of the semantic segmentation of the indoor image can be obtained.

The overall structure of the stepped residual characteristic combining network device (called RDFNet) of the color depth image for the interior semantic partitioning according to the present invention is shown in FIG.

The main issue in the semantic segmentation of color - depth images is the efficient extraction of depth characteristics according to color characteristics and the use of such characteristics in the desired work of semantic segmentation. RefineNet has suggested a means of combining different levels of properties for color images only, but RDFNet according to the present invention introduces a simple structure for combining multi-type CNN features that maintains the benefits of skip connection for color depth images .

RDFNet allows to handle multi-type CNN feature combinations and has RefineNet blocks for combined feature refinement. Unlike existing networks that utilize depth information, RDFNet is designed to take full advantage of multi-level depth characterization information through MMF blocks with additional deep depth feature paths based on ResNet.

<Step 1 (ResNet)>

In FIG. 3, the network module displayed as CNN (Convolutional Neural Network) is a network module for extracting stepwise characteristic information from a color image and a depth image. The present invention utilizes the structure of a residual network (ResNet) module of Kaiming He, which has recently exhibited the best performance in the image classification, and modifies the parameters previously learned in the image classification data set according to the purpose of semantic image segmentation The network module for extracting the characteristic information according to the present invention comprises a first residual network module for extracting the stepwise characteristic information from the color image and a second residual network module for extracting the step characteristic information from the depth image .

Conventional CNN structures are learning using backpropagation. As the CNN structure becomes deeper, the gradation value of the next step may be exponentially decreased according to the gradient of the previous step. Therefore, the gradients near the output stage have a value, but at the input stage, the value approaches zero, causing a problem that the learning does not proceed any more (gradient vanishing problem).

The residual network module solves this gradient vanishing problem and enables effective learning of the deep network structure. In this network module, the result of calculating the sum of the output value (F (x) in FIG. 4) and the input value (identity in FIG. 4) obtained by passing the input value through several stages using a skip connection structure is obtained .

로 표현할 수 있다. 도 4의 (b)는 레지듀얼 네트워크(residual network) 모듈 구조인데, 2개의 단계를 건너 뛴 skip connection을 활용한 것이 특징이다. 여기서

는

로 표현할 수 있다. 이러한 구성은 그라디언트를 계산하는데 있어 그 값이 너무 작아지지 않게 해주기 때문에 기존 CNN에서 발생되던 gradient vanishing 문제를 해결할 수 있다. 따라서 보다 많은 단계의 모델을 잘 학습시킬 수 있고 적은 수의 단계를 갖는 모델로도 높은 성능을 얻을 수 있다. 4 (a) shows a conventional network structure. In this structure, after two steps, the mapped result is

. Figure 4 (b) shows the structure of a residual network module, which uses a skip connection that skips two steps. here

The

. This configuration can solve the gradient vanishing problem that occurred in existing CNN because it does not become too small in calculating the gradient. Therefore, it is possible to learn the model with more steps and obtain the high performance even with the model with fewer steps.

<Step 2 (MMFNet)>

In FIG. 3, the portion indicated by the MMFNet module combines extracted characteristic information for each step of the first and second residual network modules. Simply concatenating or adding the features extracted from the color and depth images can not utilize the two characteristics sufficiently effectively. Therefore, in this network module, the structure that has been proven to combine the characteristics of various stages in the color image is modified to suit the combination of characteristics extracted from different images. Also, unlike the existing technologies, the characteristics extracted from the respective images are combined at a specific stage of the network. In the present invention, the network module is designed to utilize most of the intermediate characteristics obtained in the first stage. The structure thus designed makes it possible to automatically extract and combine features important to semantic segmentation from each image naturally in the process of learning it.

The detailed configuration of MMFNet is shown in Fig.

Given the characteristic information of the first and second residual network modules for color (RGB) and depth, MMFNet first reduces the dimension of each characteristic information through a single columbration block, And facilitates efficient training.

Then, each property information passes through two residual convolution units (RCU) and one convolution block as in RefineNet. There are differences in the purpose of RCU in MMFNet and RefineNet. The RCU in MMFNet is specifically intended to perform nonlinear transformations for shape combining. The two characteristics in different modalities are compensated to be mutually improved through the operation of the RCU. In MMFNet, high-level features are refined by introducing high-resolution into low-level features such as RefineNet.

The following additional convolution block in MMFNet is important in adaptively combining other types of characteristics and adjusting the scale of the property values appropriately for addition.

The additive combination in MMFNet works to learn supplemental or residual depth characteristics, since the color properties in semantic splitting generally have better discrimination than depth properties. The depth properties here will improve the RGB characteristics to identify confusing patterns. The importance of each shape characteristic can be controlled by the learnable parameters in the convolution block after the RCU.

Finally, MMFNet performs an additional residual pooling operation to integrate contextual information into the combined property information. One residual pooling in the MMFNet at each step is sufficient. Stronger contextual information can be further combined in a subsequent multilevel combination through the RefineNet block.

MMFNet enables efficient multi-level color (RGB), depth feature extraction and efficient end-to-end training by introducing residual learning with skip connection through all steps.

<Step 3 (RefineNet)>

In FIG. 3, the portion denoted by RefineNet is a part that is refined to obtain a high-resolution semantic division result by inputting the refined characteristic information of the lower level and the characteristic information combined at the corresponding step of MMFNet. It is important to combine them step by step because the characteristics obtained in stages have different resolution and dimensional information.

RefineNet of FIG. 3 can utilize the structure of RefineNet according to the prior art as it is. However, it differs from the input of the network module in that it extracts and combines the characteristics extracted from the color image and the depth image, instead of extracting the characteristics extracted from the color image.

RefineNet of FIG. 3 receives extracted and combined characteristic information and previously refined characteristic information from color and depth images of multi-level MMFNet as input. Such characteristics are refined and combined by a series of subcomponents (residual-call-convolution unit, multi-resolution combine block, chain-residual pooling) shown in Figure 1 (b).

The Residual Convolution Unit (RCU) shown in FIG. 1 (c) is an adaptive convolution set that fine-tunes the weight of ResNet pre-trained for semantic partitioning.

The Multiresolution Fusion Block shown in FIG. 1 (d) combines multipath inputs into a high resolution feature map. One convolution in the block is for input adaptation, adjusting the number of characteristic channels and adjusting the scale of the characteristic value to add.

The purpose of the Chained Residual Pooling (CRP) shown in FIG. 1 (e) is to encode information in a context from a large area. The block consists of a chain of multiple pooling blocks. Each pooling block consists of one max-pooling layer and one convolution layer. The pulling operation has the effect of propagating a large activation value accessed from a neighborhood to a contextual characteristic. The additional convolution layer learns the importance of the pooled property, which is combined with the original property through a residual connection.

At the end of RefineNet there is an additional RCU for introducing nonlinear behavior on the combined property map.

In the present invention, the learning of the network module is made of a back-propagation algorithm using a stochastic gradient descent.

&Lt; Evaluation result >

In the present invention, the network module is learned for two types of indoor image data set of NYUDv2 and SUN-RGBD, and the semantic image division result obtained through the learning is evaluated by three evaluation criteria.

First, we used pixel accuracy to measure pixel accuracy, mean accuracy, which is the average value of the accuracy of each object, and mean IoU, which is the average value of overlap with actual labels for each object type.

This figure is also compared with the results of the other major study groups in Tables 1 and 2.

<Comparison of semantic image segmentation results for NYUDv2 dataset>

Table 1 shows the results for the NYUDv2 dataset. As shown in Table 1, the performance of the present invention (RDF-152 (ours)) is superior to that of data (RGB-D) using color image and depth image. In addition, it can be seen that the present invention using the color image and the depth image at the same time has much better performance than RefineNet [26] which used only color image.

<Comparison of semantic image segmentation results for SUN-RGBD dataset>

Table 2 shows a comparison of the results of the present invention with the results of other study groups tested on the SUN-RGBD dataset. Table 2 shows that the present invention is superior to the existing research using FuseNet using color image and depth image, and shows better performance than RefineNet using only color image.

The documents used in Tables 1 and 2 are as follows.

[Document 11] S. Gupta, R. Girshick, P. Arbel'aez, and J. Malik. Learning rich features from rgb-d images for object detection and segmentation. In Proc. ECCV, pages 345-360. Springer, 2014

[Literature 7] D. Eigen and R. Fergus. Predicting depth, surface normal and semantic labels with a common multi-scale convolutional architecture. In Proc. ICCV, pages 2650-2658, 2015

[Literature 30] J. Long, E. Shelhamer, and T. Darrell. Fully convolutional networks for semantic segmentation. In Proc. CVPR, pages 3431-3440, 2015

[Literature 39] J. Wang, Z. Wang, D. Tao, S. See, and G. Wang. Learning common and specific features for rgb-d semantic segmentation with deconvolutional networks. In Proc. ECCV, pages 664-679. Springer, 2016.

[Literature 27] G. Lin, C. Shen, A. v. d. Hengel, and I. Reid. Exploring context with deep structured models for semantic segmentation. arXiv preprint arXiv: 1603.03183, 2016

[26] G. Lin, A. Milan, C. Shen, and I. Reid. RefineNet: Multipath refinement networks for high-resolution semantic segmentation. In CVPR, July 2017

[32] X. Ren, L. Bo, and D. Fox. Rgb- (d) scene labeling: Features and algorithms. In Proc. CVPR, pages 2759-2766. IEEE, 2012.

[Literature 19] A. Kendall, V. Badrinarayanan, and R. Cipolla. Bayesian segnet: Model uncertainty in deep convolutional encoderdecoder architectures for scene understanding. arXiv preprint arXiv: 1511.02680, 2015.

[25] Z. Li, Y. Gan, X. Liang, Y. Yu, H. Cheng, and L. Lin. Lstmcf: Unifying context modeling and fusion with lstms for rgbd scene labeling. In Proc. ECCV, pages 541-557. Springer, 2016.

[Literature 13] C. Hazirbas, L. Ma, C. Domokos, and D. Cremers. Fusenet: Incorporating depth into semantic segmentation via fusionbased cnn architecture. In Proc. ACCV, volume 2, 2016.

[Literature 28] G. Lin, C. Shen, A. van den Hengel, and I. Reid. Efficient piecewise training of deep structured models for semantic segmentation. In Proc. CVPR, pages 3194-3203, 2016

Figure 6 shows the qualitative results of the semantic partitioning.

The results of the present invention are the results of RefineNet using only the input image, the actual label, and the color image in order from left to right for each image. The colors are expressed differently according to the classification of objects. It can be confirmed that the result of semantic division is much more accurate than that of using only color image.

Meanwhile, the above-described embodiments of the present invention can be recorded in a medium used in a general-purpose computer including a personal computer. The medium may be a magnetic recording medium (e.g., ROM, floppy disk, hard disk, etc.), an optical reading medium (e.g. CD ROM, And the like.

The present invention has been described with reference to the preferred embodiments. It will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims. The disclosed embodiments should, therefore, be considered in an illustrative rather than a restrictive sense. The scope of the present invention is defined by the appended claims rather than by the foregoing description, and all differences within the scope of equivalents thereof should be construed as being included in the present invention.

Claims (12)

delete delete 1. A network apparatus for performing deep learning based pixel unit semantic division using a single indoor color image (RGB) and a depth image (Depth)
A first residual network module, which is composed of a plurality of steps of a Convolutional Neural Network (CNN) for each step, from an upper step to a lower step, and extracts stepwise characteristic information from a color image;
A second residual network module which is composed of a plurality of steps of a CNN (Convolutional Neural Network), which leads from a higher stage to a lower stage, and extracts the stepwise characteristic information from the depth image; And
(MMFNet) module that combines the extracted characteristic information for each step of the first and second residual network modules,
Wherein the multi-characteristic combining network module is configured to perform, for each step, the color characteristic information and the depth characteristic information extracted at corresponding steps of the first and second residual network modules,
A convolution block for reducing the dimension of the characteristic information to alleviate the surge of the parameter;
Two residual call polarization units for performing nonlinear transformation for shape combining; And
A convolution block for adaptively combining other types of characteristic information and adjusting a scale of characteristic values for addition,
Wherein the scaled color characteristic information and the depth characteristic information are additionally combined. &Lt; RTI ID = 0.0 &gt; 10. &lt; / RTI &gt; The method of claim 3,
And a residual pulling block for combining context information with characteristic information in which color characteristic information and depth characteristic information are combined. 4. The method according to claim 3,
The method of claim 1, further comprising refining the refinement by inputting the refinement characteristic information of the lower level and the characteristic information combined at the corresponding stage of the multi-characteristic combining network module. Residual characteristic combining network device. delete delete A method of performing deep-learning-based pixel-by-pixel semantic division using a single indoor color image (RGB) and a depth image thereof,
(a) extracting stepwise characteristic information by inputting a color image as a first residual network module, which is composed of a plurality of steps of a CNN (Convolutional Neural Network), which leads from a higher level to a lower level and has a CNN (Convolutional Neural Network)
(b) extracting stepwise characteristic information by inputting a depth image by a second residual network module, which is composed of a plurality of steps of a CNN (Convolutional Neural Network), from an upper step to a lower step and each step; And
(c) combining the extracted characteristic information for each step of the first residual network module and the second residual network module,
The step (c)
Wherein the multi-characteristic combining network module is configured to perform, for each step, the color characteristic information and the depth characteristic information extracted at corresponding steps of the first and second residual network modules,
Reducing the dimension of the characteristic information to alleviate the surge of the parameter;
Performing a non-linear transformation for shape combining; And
Adaptively combining other types of characteristic information and adjusting a scale of the characteristic value for addition,
Wherein the scaled color feature information and the depth feature information are additionally combined. &Lt; RTI ID = 0.0 &gt; 10. &lt; / RTI &gt; 9. The method of claim 8, wherein step (c)
Further comprising the step of incorporating contextual information into the characteristic information in which the color characteristic information and the depth characteristic information are combined, in the context of the indoor color depth image. 9. The method of claim 8,
(d) refining the refined network by inputting the refined characteristic information of the lower level and the characteristic information combined at the corresponding stage of the multi-characteristic combining network module for each step, A Semantic Split Method for Indoor Color - Depth Imagery Using Feature Combining Network Devices. A computer-readable recording medium having recorded thereon a program for causing a computer to execute a method of semantic division of indoor color depth images using a stepped residual characteristic combining network apparatus according to any one of claims 8 to 10. A computer program stored on a medium for executing in a computer a method for semantic segmentation of indoor color depth images using a stepped residual characteristic combining network device according to any one of claims 8 to 10.

Images