KR20190119261A - Apparatus and method for segmenting of semantic image using fully convolutional neural network based on multi scale image and multi scale dilated convolution - Google Patents

Abstract

The present invention relates to an apparatus for segmenting a semantic image using a fully convolutional neural network based on a multi scale image and multi scale extended convolution and a method thereof capable of quickly and accurately segmenting meaningful part from a specific image through a fully convolutional neural network having a cascade architecture of extended convolution and a multi scale image as an input.

Description

FIELD AND METHOD FOR SEGMENTING OF SEMANTIC IMAGE USING FULLY CONVOLUTIONAL NEURAL NETWORK BASED ON MULTI SCALE IMAGE AND MULTI SCALE DILATED CONVOLUTION }

The present invention relates to a semantic image segmentation apparatus using a multi-scale image and a multi-scale extended convolution-based full convolutional neural network, and a method thereof. More particularly, the present invention relates to a cascade architecture of an extended convolution and a multi-scale image. Semantic Image Segmentation Apparatus and Method Using Full-Convolution Neural Network Based on Multi-Scale Image and Multi-Scale Extended Convolution Based on Full-Convolutional Neural Network It is about.

Recently, due to rapid development of image processing technology, semantic image segmentation technology is rapidly developing.

Semantic image segmentation technology is used to classify meaningful parts from images. It is applied to various fields such as autonomous vehicles and medical fields, and contributes to notable scene understanding and object recognition.

Before the development of deep learning algorithms, most semantic image segmentation techniques rely heavily on conditional random fields. The conditional random field method is used to label any of several predefined object classes on image pixels.

As a result, the conditional random field method makes it possible to simultaneously recognize and segment various objects. Most techniques that use conditional randomfields deal with semantic image segmentation issues by developing a contextual information model for maximizing label aggregation between neighboring pixels and classifying various object classes.

In general, a typical model of a conditional random field is calculated by the unitary potential of each pixel and the pair wise potential for neighboring pixels of each pixel.

In addition, the development of deep learning algorithms, deep convolutional neural networks, has led to breakthroughs in object detection and image recognition by enabling the automatic generation of meaningful features.

In particular, the fully convolutional neural network, one of the deep convolutional neural networks, is under great effort and research to improve the performance of semantic image segmentation due to the advantages of computational efficiency for dense prediction. have.

Accordingly, many conventional methods for performing semantic image segmentation have recently been developed based on a fully convolutional neural network including a neural network.

Conventional methods for semantic image segmentation also include a feature selected from a pooling layer and a sub sampling layer in the convolutional neural network, covering the receptive field in the original image. Can be extended to

However, the conventional semantic image segmentation method using the convolutional neural network reduces the resolution of the original image and loses detail and local features in the image, so that the semantic image based on the original image is performed. There is a problem in that the accuracy of segmentation is very low.

Accordingly, the present invention proposes a fully convolutional neural network having a cascade architecture of extended convolution and a deep network architecture of multi-scale image input, thereby overcoming the problems of the conventional semantic image segmentation method of reducing the resolution of the original image. A fully convolutional neural network based on multi-scale image and multi-scale extended convolution, which efficiently extracts features for multi-scale images and suppresses the increase of network learning parameters, enabling fast and accurate semantic image segmentation. The present invention provides a semantic image segmentation apparatus and a method thereof.

Next, the prior art existing in the technical field of the present invention will be briefly described, and then the technical matters to be made differently from the prior art will be described.

First, the non-patent literature, Deconvolution Network Learning Method for Semantic Segmentation (see arXiv: 1505.04366), uses an up-convolutional layer to overcome the problems with the conventional semantic segmentation method. In the pooling layer of the convolutional network, the downsampling process recovers lost information so that relatively semantic segmentation can be performed.

However, the prior art has a problem that the accuracy of semantic image segmentation is remarkably inferior because it recovers only part of the lost information, rather than recovering all the lost information.

On the other hand, the present invention combines an extended convolution cascade architecture and a deep network architecture for multi-scale image input to reduce the resolution of the original image and the original image in the process of learning the multi-scale image for semantic image segmentation. In order to efficiently recover lost features, so that semantic image segmentation can be performed accurately, the prior art does not describe or suggest such technical features of the present invention.

In addition, non-patent literature, Semantic Image Segmentation (see ICLR) using conditional random fields fully connected to deep convolution networks, uses downsampling operations at several layers of the convolutional network to prevent the resolution of the original image from decreasing. By extracting a more dense feature map, the semantic image segmentation can be performed.

However, the prior art has a problem in that the learning process for performing semantic image segmentation requires a very high computational cost due to the huge amount of input parameters.

On the other hand, the present invention can efficiently and quickly perform semantic image segmentation on the original image by efficiently extracting the features of the multi-scale image without increasing the number of learning parameters through the extended convolution cascade architecture. As such, the prior art does not present any configuration of the technical features of the present invention, and does not imply any indication thereof.

The present invention was created to solve the above problems, and learns a meaningful portion from the original image by learning the multi-scale image through a full convolutional neural network composed of at least one sub-network that takes a multi-scale image as an input. It is an object of the present invention to provide a semantic image segmentation apparatus and a method using a full scale convolution neural network based on multi-scale image and multi-scale extended convolution to enable semantic image segmentation to be classified quickly and accurately.

In addition, the present invention applies extended convolution to a full convolutional neural network to prevent an increase in learning parameters and to effectively recover a reduced spatial resolution in the process of learning a multi-scale image, thereby improving the accuracy of the semantic image segmentation. Another object of the present invention is to provide a semantic image segmentation device and a method using a multi-scale image and a multi-scale extended convolution-based fully convolutional neural network.

In addition, the present invention by applying a conditional random field model fully connected to a full convolutional neural network, it is possible to effectively identify the object identification and pixel level object position for the multi-scale image, thereby significantly improving the performance of the semantic image segmentation Another object of the present invention is to provide a semantic image segmentation apparatus using the multi-scale image and the multi-scale extended convolution-based full convolutional neural network, and a method thereof.

The semantic image segmentation apparatus according to an embodiment of the present invention is a preprocessor for pre-processing the training data to generate a multi-scale image of the original image of the training data, learning model for the semantic image segmentation by learning the generated multi-scale image A learning model generator for semantic image segmentation for generating a multi-scale convolutional cascade architecture includes a plurality of sub-networks and a multi-scale extended convolution cascade architecture. It is characterized in that performed through the neural network.

In addition, the semantic image segmentation device restores the resolution of the feature map output from each of the sub-networks through the multi-scale extended convolution cascade architecture to generate a high-resolution final feature map, and each of the generated final features And readjust the map to the same resolution.

The multi-scale extended convolution cascade architecture also includes a plurality of extended convolution layers connected in cascade to perform extended convolution, each extended convolution layer having a different rate. It is characterized by having a higher rate than the previous extended convolution layer.

In addition, the fully convolutional neural network is configured to integrate all feature maps output from the multi-scale extended convolution cascade architecture to extract a boundary for an object and a fully connected layer for restoring the boundary for the extracted object. And further comprising a conditional random field model.

The semantic image segmentation apparatus, when a specific image for semantic image segmentation is input, controls the preprocessor to generate a multi-scale image of the specific image, and learns the generated multi-scale image for the semantic image segmentation. The method may further include a semantic image segmentation unit configured to perform semantic image segmentation on the specific image.

In addition, the semantic image segmentation method according to an embodiment of the present invention is a preprocessing step of pre-processing the training data to generate a multi-scale image of the original image of the training data, learning the semantic image segmentation by learning the generated multi-scale image Generating a training model for semantic image segmentation that generates a model, the training comprising a plurality of sub-networks consisting of a convolutional neural network with multi-scale images as input and a multi-scale extended convolution cascade architecture. Characterized in that it is performed via a convolutional neural network.

In addition, the semantic image segmentation method recovers the resolution of the feature map output from each of the sub-networks through the multi-scale extended convolution cascade architecture to generate a high-resolution final feature map, and each of the generated final features And readjust the map to the same resolution.

In addition, the semantic image segmentation method, when a specific image for semantic image segmentation is input, by controlling the pre-processing unit to generate a multi-scale image for the specific image, learning the semantic image segmentation for the generated multi-scale image The method may further include a semantic image segmentation step of performing semantic image segmentation on the specific image.

As described above, the semantic image segmentation apparatus using the multi-scale image and the multi-scale extended convolution-based full convolutional neural network, and the method according to the multi-scale extended convolution cascade architecture and multi-scale image input deep Deep convolutional neural networks, combined with a network architecture, speed semantic image segmentation by efficiently extracting multiscale features from multiscale images, restoring reduced resolution of multiscale images, and suppressing increases in learning parameters. It has the effect of performing correctly.

In addition, the present invention applies a conditional random field model that is fully connected to the deep convolutional neural network, thereby effectively removing isolated false positives, thereby improving segmentation prediction along the boundary of an object included in a multiscale image, thereby improving semantic image segmentation. There is an effect that can significantly improve the accuracy for.

1 is a diagram illustrating a semantic image segmentation method using a deep convolution neural network according to the related art.
FIG. 2 is a conceptual diagram schematically illustrating a semantic image segmentation apparatus and a method using a multi-scale image and a multi-scale extended convolution based full convolution neural network according to an embodiment of the present invention.
3 is a diagram illustrating feature extraction using a multi-scale extended convolution cascade having different rates according to an embodiment of the present invention.
4 illustrates a deep convolutional neural network having a multiscale extended convolution according to an embodiment of the present invention.
5 illustrates a full convolutional neural network according to an embodiment of the present invention.
6 is a diagram illustrating a max out layer included in a full convolutional neural network according to an embodiment of the present invention.
7 is a block diagram illustrating a configuration of a semantic image segmentation apparatus according to an embodiment of the present invention.
8A is a diagram illustrating the structure of a ResNet-101 based fully convolutional neural network according to an embodiment of the present invention.
8B is a diagram showing the structure of a VGG-16 based fully convolutional neural network according to an embodiment of the present invention.
FIG. 9 is a diagram comparing accuracy of semantic image segmentation through a full convolutional neural network and accuracy of semantic image segmentation using another method using the PASCAL VOC 2012 dataset according to an embodiment of the present invention.
FIG. 10 is a diagram comparing processing speeds of a VGG-16 based fully convolutional neural network and a ResNet-101 based fully convolutional neural network according to an embodiment of the present invention.
FIG. 11 is a diagram comparing the performance of a ResNet-101 based fully convolutional neural network and a VGG-16 based fully convolutional neural network by applying a fully connected conditional random field model according to an embodiment of the present invention.
FIG. 12 is a diagram illustrating an example of segmentation of a semantic image emphasizing an activity including a person-object interaction and a person-person interaction according to an embodiment of the present invention.
FIG. 13 is a diagram illustrating an example of segmentation of a semantic image for an animal according to an embodiment of the present invention. FIG.
14 illustrates an example of segmentation of a semantic image of an object according to an embodiment of the present invention.
FIG. 15A is a diagram comparing the performance of a complete convolutional neural network with other methods using a skin lesion data set for melanoma detection according to an embodiment of the present invention.
FIG. 15B is a diagram showing semantic image segmentation results of a skin lesion data set for melanoma detection through a full convolutional neural network according to an embodiment of the present invention. FIG.
16 is a flowchart illustrating a procedure of performing semantic image segmentation according to an embodiment of the present invention.

Hereinafter, with reference to the accompanying drawings will be described in detail a preferred embodiment of the semantic image segmentation apparatus and method using a full scale convolution neural network based on the multi-scale image and multi-scale extended convolution. Like reference numerals in the drawings denote like elements. In addition, specific structural to functional descriptions of the embodiments of the present invention are only illustrated for the purpose of describing the embodiments according to the present invention, and unless otherwise defined, all terms used herein including technical or scientific terms These have the same meaning as commonly understood by one of ordinary skill in the art to which the present invention belongs. Terms such as those defined in the commonly used dictionaries should be construed as having meanings consistent with the meanings in the context of the related art, and are not construed in ideal or excessively formal meanings unless expressly defined herein. It is preferable not to.

1 is a diagram illustrating a semantic image segmentation method using a deep convolution neural network according to the related art.

As shown in FIG. 1, the semantic image segmentation method using the deep convolutional neural network according to the related art includes a plurality of convolution layers and a max-pooling layer for performing subsampling. Learning the input image to generate a learning model for semantic image segmentation.

In addition, the deep convolutional neural network according to the prior art uses the max pooling layer and the stride operator repeatedly to downsample the input image and reduce the dimension of the downsampled image, thereby reducing the number of learning parameters. Reduce the cost of computation.

However, the conventional technique using the deep convolutional neural network has a problem that the accuracy of semantic image segmentation is remarkably degraded because it greatly reduces the spatial resolution of the original image.

Therefore, the present invention proposes a full convolutional neural network including an extended convolutional layer, thereby accurately and quickly by not reducing the resolution of the input matrix input to each convolutional layer without increasing the number of learning parameters. The present invention provides a semantic image segmentation device and a method using a full scale convolution neural network based on multi-scale image and multi-scale extended convolution to perform semantic image segmentation.

FIG. 2 is a conceptual diagram schematically illustrating a semantic image segmentation apparatus using the full scale convolution neural network based on a multi-scale image and a multi-scale extended convolution according to an embodiment of the present invention, and a method thereof.

As shown in FIG. 2, a semantic image segmentation apparatus (hereinafter, referred to as a semantic image segmentation apparatus) using a multi-scale image and a full-convolution neural network based on extended convolution according to an embodiment of the present invention may be described. The training data stored in the training data database 310 of the database 300 is learned to generate a training model for semantic image segmentation.

In addition, the learning data includes medical images used in the medical field, such as skin images including skin lesions, as well as images including various objects such as people, bottles, cars, and trains.

That is, the semantic image segmentation apparatus 100 may learn a plurality of images constituting the learning data, and may segment meaningful parts such as a person, a bottle, a car, a train, a damaged spot, and the like from a specific image.

The learning is also performed on a fully convolutional neural network. In addition, the full convolutional neural network is composed of at least one sub-network (sub-network) for receiving a multi-scale image of the original image of the training data, respectively, an extended convolution cascade (end) at the end of each sub-network Architecture is combined.

Meanwhile, the multi-scale image refers to a plurality of images that are rescaled to different scales through the preprocessing process, and the exchange convolutional neural network receives the rescaled images through the respective subnetworks. In addition, the learning data is performed.

The fully convolutional neural network is also combined with a fully connected conditional random field model, which effectively eliminates isolated false positives, improving the accuracy of semantic segmentation.

Meanwhile, the structure of the full convolutional neural network according to the embodiment of the present invention described above will be described in detail with reference to FIG. 5.

In addition, the semantic image segmentation apparatus 100 stores the learning model generated by learning the training data in the learning model database 320.

In addition, when a specific image for semantic image segmentation is input, the semantic image segmentation apparatus 100 performs the preprocessing to generate a multi-scale image of the specific image.

Thereafter, the semantic image segmentation apparatus 100 loads the stored learning model from the learning model database 320 and then applies the multi-scale image of the specific image to the loaded learning model, thereby obtaining semantics of the corresponding image. Perform image segmentation.

In addition, the specific image may be input from a user terminal 200 or a camera (not shown) associated with a specific system such as a medical system or an autonomous vehicle.

The user terminal 200 refers to a wireless communication terminal provided by the user, such as a smart phone, PDA, notebook PC, and the like.

Meanwhile, as described above, the user terminal 200 may receive a semantic image segmentation result from the semantic image segmentation device 100 by transmitting a specific image to the semantic image segmentation device 100 through a network. The learning model may be downloaded from the semantic image segmentation apparatus 100 to perform semantic image segmentation on the user terminal 200 itself. In this case, the user terminal 200 becomes a device for semantic image segmentation.

In addition, the semantic image segmentation apparatus 100 may interwork with a medical system (not shown), receive an image of a patient's affected part from the medical system, and segment the affected part from the corresponding image to provide it to a medical associate such as a doctor. It can be implemented to perform an accurate diagnosis on the basis of the patient. In this case, the semantic image segmentation apparatus 100 may be linked with the medical system through a network, and may be integrated with the medical system to perform semantic image segmentation on the affected image locally.

In addition, the semantic image segmentation apparatus 100 receives a driving-related image from a camera provided in the autonomous vehicle in cooperation with the autonomous vehicle, and segments the plurality of objects included in the image, thereby surrounding the autonomous vehicle. By recognizing a plurality of objects located in the can be implemented to perform a stable driving.

That is, the semantic image segmentation apparatus 100 according to an embodiment of the present invention may be applied to various fields for image recognition to segment meaningful parts from a specific image, thereby quickly and easily detecting object recognition and scene recognition based on the specific image. It can be implemented to perform effectively.

In addition, the database 300 includes a learning data database 310 for storing learning data and a learning model database 320 for storing a learning model generated by the face recognition apparatus 100.

Hereinafter, a cascade structure of a multi-scale extended convolution according to an embodiment of the present invention will be described with reference to FIGS. 3 and 4.

3 is a diagram for explaining dense feature extraction using a multi-scale extended convolution cascade having different rates according to an embodiment of the present invention, and FIG. 4 is a diagram illustrating multi-scale extension according to an embodiment of the present invention. A diagram showing a deep convolution neural network having a convolution.

As shown in FIG. 3, the cascade structure of the multi-scale extended convolution according to an embodiment of the present invention is different from the max pooling layer and the stride operator according to the related art. It is also known as a convolutional layer with filters). The extended convolutional layer is generally applied to the input matrix of the extended convolutional layer by extending the filter before calculating the convolution.

The size of the filter is expanded and the empty position is completely filled with zeros. As a result, the weight is consistent with the distance elements of the input matrix. The distance component is determined by the rate r.

If the kernel center is aligned at any position of the image, the kernel element will coincide with the input element as shown in FIG.

As a simple example of extended convolution, one-dimensional extended convolution may be applied to a one-dimensional signal. The output y [i] of the expanded convolution of the input x [i] with the filter w [k] is calculated by Equation 1 below.

[Equation 1]

Where m is the length of filter w [k].

Also in the present invention, the main advantage of the extended convolution is that it extends the accepting field of the filter in the convolutional layer while not reducing the resolution of the input matrix. By applying extended convolution with rate r, a filter with kernel size k x k can be extended up to k '. At this time, k 'has a value of k + (k-1) (r-1).

The extension offers several advantages, but also introduces some disadvantages. On the other hand, the present invention applies extended convolution at a large rate r to capture a larger context for multi-scale images without increasing the computational cost of a full convolutional neural network.

However, extended convolution with a large rate r introduces more zeros between the filter values and can cause more local context information than in smaller regions.

Extended convolution with a small rate r can also be used to improve localization performance. Nevertheless, each extended convolutional layer produces a feature map with a narrow acceptance range. Therefore, in order to extract a more dense feature map, a combination of extended convolutions having different rates r is required.

Accordingly, the present invention proposes a cascade architecture of a multiscale extended convolution layer for extracting context information from a multiscale image.

The cascade architecture consists of successive convolution layers, each of which uses only an extended convolution kernel with the same rate r to produce a dense feature map with the same acceptance field.

In addition, the previous convolutional layer uses a convolution with a lower rate than the current convolutional layer, so that local features can be extracted and localization accuracy is improved.

In contrast, the current convolutional layer uses extended convolution with a higher rate to increase context assist.

Therefore, the feature maps of the current extended convolution layer are aggregated and can be more dense than the previous feature maps.

Meanwhile, the multi-scale extended convolution cascade architecture is added to the deep convolutional neural network for calculating the final feature map at high resolution, as shown in FIG. 4.

FIG. 5 illustrates a full convolutional neural network according to an embodiment of the present invention, and FIG. 6 illustrates a max out layer included in the full convolutional neural network according to an embodiment of the present invention.

As shown in FIG. 5, a full convolutional neural network according to an embodiment of the present invention is configured to learn a feature map for a multi-scale image with respect to an original image.

Meanwhile, as described above, the multi-scale image means that the image is adjusted to a plurality of different scales from the original image of the training data.

The full convolutional neural network also comprises a plurality of sub-networks as inputs to each image adjusted at the different scales. The subnetwork consists of a deep convolutional neural network.

Context information, on the other hand, is effectively captured by combining features from extended convolution with different rates r. However, increasing the size of the acceptance field using a large rate r increases the chance of losing contextual information at locations where extended convolution introduces zeros between successive filter values. The larger the rate r, the more zeros are introduced.

It is also reasonable to search for multi-scale features in a multi-scale image input because the same object may have different sizes in different images.

Thus, a multi-scale image can be applied to the sub-network, with each adjusted scale passing through a branch of the full convolutional neural network, and the features are fused from all scales.

In general, inputting a multi-scale image into a deep convolutional neural network requires a large number of learning parameters, which is very expensive in the learning process.

Therefore, the fully convolutional neural network proposed by the present invention uses a multi-scale image as an input matrix, and integrates an extended convolution cascade architecture at the end of each sub-network, thereby preventing an increase in learning parameters, thereby increasing semantic segmentation. Learning can be done quickly and accurately.

In addition, the original image is scaled to multiple scales through a bilinear interpolation method, with each scaled image being a specific branch of the full convolutional neural network (i.e. multi-scale extended convolution of each sub-network). Pass an extended convolution cascade architecture).

The output of each branch is a feature map readjusted to the same resolution as the other feature maps.

Finally, all the feature maps are merged into a shared feature map. Since each object in the image is prominent on different feature maps, the full convolutional neural network employs a maxout layer and obtains the maxout layer to obtain competitive and key features from all feature maps. Through the fusion of all the feature map to the shared feature map.

Also, the max out layer is regarded as a maximum feature map, as shown in FIG. 6.

In particular, the extended convolution layer includes groups of feature maps, compares feature values at the same coordinates of these groups, selects a maximum value, and then assigns the feature value of the same coordinates of the maxout layer to the selected maximum value. .

The fully convolutional neural network also comprises a fully connected conditional random field, as shown in FIG.

A full convolutional neural network may identify an object and provide a reliable final score map of the approximate location of that object, but the boundary may not be extracted accurately and sharply. This is due to the fact that two difficult tasks must be completed: object identification balancing pixel accuracy and locating object at pixel level.

Accordingly, the present invention solves an object localization problem for locating an object at the pixel level by applying a conditional random field model that is fully connected to the end of a full convolutional neural network, and accurately recovers the boundary of the object. It can significantly improve the accuracy of semantic image segmentation.

The conditional random field model is an energy function that uses two sets of potential functions consisting of the potential potential and the phasewise potential according to Equation 2 to determine the position of the object and recover the boundary of the object. Minimize).

[Equation 2]

Wherein α _i (x _i) is an oil Nourishing potential for measuring station likelihood of the label (label) x _i from pixel i, α _ij (x _i, x _j) is the pixel i and a pixel having the label x _i and x _j In j, the pairwise potentials for estimating label allocation costs are shown.

In addition, the binary potential is the output of the pixel classifier calculated by Equation 3 below.

[Equation 3]

Where P (x _i ) is the probability of assigning the label i to pixel i. The P (x _i ) is also calculated by the fully convolutional neural network of the present invention, and the pairwise potential is calculated based on an image gradient between a pixel and its neighboring pixels.

In particular, pixels and neighboring pixels of the pixel are classified under the same label when the calculated gradient is small. Thus, Pairwise Potential brings consistency to the shape of the segmented object and improves the description of the object.

In addition, the pairwise potential is calculated by Equation 4 below.

[Equation 4]

Where f ^m is a Gaussian kernel weighted by the parameter ω _m . The Gaussian kernel f ^m is calculated based on the features τ _i and τ _j collected for pixels i and j, respectively.

The Gaussian kernel f ^m is calculated by the following Equation 5.

[Equation 5]

Here, the features τ _i and τ _j are represented by the intensity of the pixel color represented by I, and the pixel position is represented by p.

Finally, the energy function E (x) is minimized to find the best label assignment for the input image. However, this minimization problem is inherently difficult to deal with. Therefore, the present invention applies mean-field approximation to the distribution of the conditional random field model in order to approximate the stochastic inference efficiently.

That is, the distribution P (x) of the conditional random field model is approximated by the distribution Q (x), and Q (x) may be expressed as ∏Q _i (x _i ).

7 is a block diagram illustrating a configuration of a semantic image segmentation apparatus according to an embodiment of the present invention.

As shown in FIG. 7, the semantic image segmentation apparatus 100 according to an embodiment of the present invention comprises an image collection unit 110 for collecting at least one image for performing semantic image segmentation, and constituting learning data. Pre-processing unit 120 for generating a multi-scale image for a plurality of images, Semantic image segmentation learning model generation unit 130 for learning the generated multi-scale image to generate a learning model for the semantic image segmentation, the learning model A semantic image segmentation unit 140 that performs semantic image segmentation on a specific image collected by the image collecting unit 110 by using a semantic image segmentation result providing unit providing a result of the semantic image segmentation performed. 150, database interface 160 and is configured to include a control unit 170 for overall control of the semantic image segmentation unit (100).

In addition, the image collector 110 performs a function of collecting a specific image for performing semantic image segmentation at the user's request.

The specific image may be collected from the user terminal 200 through a network or from a specific system (eg, a medical system).

That is, the specific image may be collected through a network or directly input locally. However, the image collector 110 collects at least one image for semantic image segmentation, and the method of collecting the image is not limited thereto.

In addition, the image collector 110 may periodically collect the image that is the basis of the training data to update the semantic segmentation learning model generated by the semantic image segmentation learning model generation unit 130. That is, the image collecting unit 110 periodically collects an image from an institution that provides learning data for performing semantic segmentation, and reflects the collected image in the learning data database 310, thereby providing the semantic image segmentation. Keep your learning model up to date.

In addition, the preprocessor 120 generates a multi-scale image by preprocessing the original image of the training data so as to learn the training data through the learning model generator 130 for semantic image segmentation.

The preprocessor 120 generates a plurality of scaled images by adjusting the original image at a preset ratio and adjusting the original image at a preset scale ratio (eg, 1.0, 0.75, 0.5).

That is, the preprocessor 120 generates a multi-scale image of the original image by adjusting the scale of the original image at a preset ratio.

In addition, even when a specific image for semantic image segmentation is input from the user, the preprocessor 120 generates a multi-scale image of the corresponding image through the above process.

In addition, the learning model generation unit 130 for semantic image segmentation generates a learning model for semantic image segmentation by learning a plurality of scaled images of the generated learning data through the preprocessor 120.

The learning model generation unit 130 for semantic image segmentation learns the plurality of scaled images based on the complete convolutional neural network proposed by the present invention.

The full convolutional neural network includes a plurality of sub-networks, and the plurality of scaled images are respectively input to the sub-networks.

Each sub-network also includes a plurality of convolution layers, a fully connected layer and a multi-scale extended convolution architecture.

In the multi-scale extended convolution architecture, a plurality of multi-scale extended convolution layers are formed in a cascade form, and the resolution of the feature map output through the fully connected layer of each sub-network is restored, resulting in a high resolution final feature. Create a map.

In other words, the multi-scale extended convolution architecture restores the resolution of the feature map of each sub-network so as to a plurality of reduced resolutions through the sub-network.

In addition, the full convolutional neural network readjusts each final feature map output through each of the multiscale extended convolution architectures to have the same resolution, and outputs the adjusted final feature map.

The fully convolutional neural network also includes a max out layer for extracting feature values for an object by incorporating the readjusted feature map and a fully connected conditional random field model connected with the max out layer.

That is, the max out layer integrates the readjusted final feature map to extract a boundary of an object, and the fully connected conditional random field model performs a function of restoring the boundary of the extracted object.

That is, the fully convolutional neural network learns the training data quickly and accurately based on a multi-scale image input, an extended convolution, and a fully connected conditional random field model, as described with reference to FIG. 5, for semantic image segmentation. You will create a learning model.

In addition, when the semantic image segmentation unit 140 receives a specific image for semantic image segmentation from the user, the semantic image segmentation unit 140 controls the preprocessor 120 to generate a multi-scale image from the specific image.

In addition, the semantic image segmentation unit 140 loads the learning model for semantic image segmentation from the learning model database 320 and applies the generated multi-scale image to the loaded semantic image segmentation learning model, thereby providing the specific image. You will perform semantic image segmentation on.

In addition, the semantic image segmentation result providing unit 150 may provide the user with a result of the semantic image segmentation performed through the semantic image segmentation unit 140.

In addition, the database interface unit 160 is connected to the database 300 to perform a function of storing or loading data generated or needed by the semantic image segmentation apparatus 100 from the database 300.

In addition, the controller 170 performs a function of controlling data movement and overall configuration of each component of the semantic image segmentation apparatus 100.

Hereinafter, we will compare the performance of the semantic image segmentation between the full convolutional neural network of the present invention.

For the comparison see the complete convolutional neural networks of the present invention see ResNet-101 (deep residual learning for image recognition.arXiv: 1512.03385) and VGG-16 (very deep convolutional neural networks for large-scale image recognition.arXiv: 1409.1566 Based on

8A is a diagram illustrating the structure of a ResNet-101 based fully convolutional neural network according to an embodiment of the present invention.

On the other hand, network depth is very important for improving the accuracy of neural networks. But learning deep networks is a very difficult task.

The deep residual network is a deep convolutional neural network, and can perform learning at a deeper level than a conventional deep neural network. This is because it adopts a residual learning framework that makes the learning process easier, and increases the depth of the network to improve better performance.

Since the deep residual network has reached the state-of-the-art performance in image classification, we implemented the fully convolutional neural network of the present invention using a ResNet-based model with multi-scale input images.

As shown in FIG. 8A, the structure of a ResNet-101 based fully convolutional neural network according to an embodiment of the present invention uses 5 blocks and 100 or more layers for each scaled image, which is a large number of networks. It consists of a number of parameters that easily reach the maximum RAM capacity of the GPU device in the learning process. Therefore, the computational cost to increase the accuracy is very high.

For this reason, the present invention uses only two scaled image inputs, and each scale is set to 1.0 and 0.75 so that each scaled image is input to each residual network of ResNet-101.

Each residual network may summarize the features of the scaled image input in the feature map taken from the output of the last block. In addition, a ResNet-101 based fully convolutional neural network passes through the cascaded architecture of the extended convolutional layer to generate multi-scale contextual information from the feature map to generate a score map for it.

In addition, the cascade architecture applies a bilinear interpolation method to increase the resolution of the score map so that the feature maps have the same resolution.

The ResNet-101 based fully convolutional neural network then plays an important role in merging the score map through the max out layer into a final score map that can maintain competitive features.

Finally, ResNet-101-based fully convolutional neural networks encode pixel-level pairwise similarities and sharpen object boundaries with fully-connected conditional random field models that are integrated into the output of their ResNet-101-based fully convolutional neural networks. .

8B is a diagram showing the structure of a VGG-16 based fully convolutional neural network according to an embodiment of the present invention.

As shown in FIG. 8B, the structure of the VGG-16-based full convolutional neural network according to an embodiment of the present invention receives three images scaled at a ratio of 1.0, 0.75, and 0.5, and each scaled It is configured to include 16 weighted layers for the image and is implemented to learn the scaled image.

In addition, the structure of the VGG-16-based full convolutional neural network includes a multi-scale extended convolution architecture and a fully connected conditional random field model proposed by the present invention.

FIG. 9 is a diagram comparing the accuracy of semantic segmentation through a completely convolutional neural network with the accuracy of semantic segmentation through other methods using the PASCAL VOC 2012 dataset according to an embodiment of the present invention.

As shown in FIG. 9, the accuracy of semantic segmentation through a full convolutional neural network according to an embodiment of the present invention is compared with the accuracy of semantic segmentation through another method.

PASCAL VOC 2012 database was used for the comparison, and the complete convolutional neural network, FCN-8s model, DeepLab network, BoxSup network, Zoom-out network, CRF of the present invention We evaluated recently developed state-of-the-art algorithms such as -RNN network, DPN network and DeconvNet network.

In addition, the complete convolutional neural network of the present invention was previously trained using the MS-COCO data set, and 20 classes including people, bottles, cars, trains, and one background class were used using the PASCAL VOC 2012 database. To learn.

Since the original training data consisted of 1464 images, it is not sufficient to train the complete convolutional neural network of the present invention and is further added by Hariharan et al. (Semantic contours inverse detectors IN: International coference on computer vision (ICCV)) We used reinforced learning data. In total, 10,582 learning images were used in the learning process.

We also used a data set of Skin Lesion Analysis Toward Melanoma Detection for melanoma detection, the largest collection of skin lesion images, to assess the accuracy of the semantic image segmentation. Recognizing melanoma in skin microscopic images is a very difficult problem because of the low contrast of skin lesions.

The data set of skin lesion analysis consists of images for 900 training and images for 350 tests.

In addition, since the image and the test image of all the training data is a high resolution image of 1024 x 768, the learning sample for learning was increased by randomly cutting the same size from the image for the training data.

Also during the test, the entire image is segmented globally by combining the prediction results of the sub-images that overlap the entire image.

The evaluation is also performed by applying pixel intersection over union (IoU) scores for a set of tangentially measured ground-truth bounding boxes and predicted bounding boxes.

The IoU is a method for measuring the accuracy of an object segmentation algorithm. If the object segmentation algorithm can provide a predicted bounding box, the IoU score can be measured to assess the accuracy of the segmentation.

The measurement of the IoU score is defined as a value obtained by dividing the size of the intersection point for the bounding box set by the size of the union of the sample sets as shown in Equation 6 below.

[Equation 6]

Here, P∩G is an overlapped area between the predicted bounding box P and the actually measured bounding box G, and P∪G means an area in which both P and G are included.

The ResNet-101 based fully convolutional neural network may also replace the ResNet-101 complete convolution by replacing the last fully connected layer for the original ResNet-101 network with the extended convolution cascade of the present invention, as mentioned above. The correction neural network has been modified to be fully convolutional.

Each cascade consists of an extended convolutional layer having rates r corresponding to 6, 12, 18 and 25. In addition, the layer of the fully connected conditional random field is separated from the ResNet-101 based fully convolutional neural network in the learning phase.

In addition, by changing the number of learning object classes in the last layer and applying a loss function that is the sum of cross-entropy terms for each spatial position in the final dense feature map, the ResNet- The 101-based full convolutional neural network was fine tuned.

In addition, the initial learning rate of the ResNet-101 based fully convolutional neural network is set to 0.001, the momentum (momemtum) is set to 0.9, the weight reduction is set to 0.005, the learning rate is multiplied by 0.1 after 20,000 iterations.

The VGG-16 based fully convolutional neural network is also similar to the ResNet-101 based fully convolutional neural network except for the number of multi-scale inputs. The VGG-16 based fully convolutional neural network accepts images scaled at three ratios of 1, 0.75 and 0.5 instead of the two inputs of the ResNet-101 based fully convolutional neural network.

To train the VGG-16 based fully convolutional neural network, we adopted stochastic gradient descent (SDG) to minimize the loss function, which is the sum of the cross-entropy terms for each spatial location.

The initial learning rate of the VGG-16 based fully convolutional neural network is set to 0.001, and the mini-batch size is 20 images with an initial learning rate of 0.001. The learning rate was multiplied by 0.1 for the 2000 iterations, the momentum was set to 0.9, and the weight reduction was set to 0.0005.

In addition, as shown in Figure 9, it can be seen that the ResNet-101 based fully convolutional neural network according to an embodiment of the present invention achieves the best result IoU score 78.5 among all methods.

In addition, the VGG-16 based fully convolutional neural network according to an embodiment of the present invention achieved an IoU score of 74.8, which is still higher than other methods except DPN and BoxSup networks.

This is evidenced by the fact that multi-scale feature extraction plays an important role in recognizing objects in different contexts and scales.

Therefore, it can be seen that the semantic image segmentation through the multi-scale image input and the multi-scale extended convolution architecture proposed by the present invention is superior to other methods using only the single-scale image input.

We also see that semantic image segmentation over ResNet-101-based fully convolutional neural networks achieves better performance than semantic image segmentation over VGG-16-based fully convolutional neural networks.

FIG. 10 is a diagram comparing processing speeds of a VGG-16 based fully convolutional neural network and a ResNet-101 based fully convolutional neural network according to an embodiment of the present invention.

As shown in FIG. 10, semantic image segmentation over a ResNet-101 based fully convolutional neural network according to an embodiment of the present invention provides better performance than semantic image segmentation over a VGG-16 based fully convolutional neural network. We can see that the VGG-16-based fully convolutional neural network is fast, but achieves this.

This is because the learning parameters of the VGG-16 based fully convolutional neural network are much less than the learning parameters of the ResNet-101 based fully convolutional neural network.

Thus, the VGG-16 based fully convolutional neural network shows that it can be used for real-time applications in semantic image segmentation tasks.

FIG. 11 is a diagram comparing the performance of a ResNet-101 based fully convolutional neural network and a VGG-16 based fully convolutional neural network by applying a fully connected conditional random field model according to an embodiment of the present invention.

   As shown in FIG. 11, it can be seen that the fully connected conditional random field model according to the embodiment of the present invention effectively improves the performance of the fully convolutional neural network.

In particular, the fully connected conditional random field model improves the performance of the ResNet-101 based fully convolutional neural network by 0.7% and improves the performance of the VGG-16 based fully convolutional neural network by 0.8%.

This result clearly demonstrates the benefits of a fully connected conditional random field model in solving pixel-level object localization and recovery of object boundaries.

Thus, the fully convolutional neural network of the present invention is combined with the fully connected conditional random field model, thereby making it possible to perform semantic image segmentation more accurately and quickly.

Hereinafter, the results of performing semantic image segmentation through the full convolutional neural network of the present invention will be described with reference to FIGS. 12 to 14.

12 is a diagram illustrating an example of segmentation of a semantic image emphasizing an activity including a person-object interaction and a person-person interaction according to an embodiment of the present invention, and FIG. 13 is an embodiment of the present invention. FIG. 14 is a diagram illustrating an example of segmentation of a semantic image of an animal according to an example, and FIG. 14 is a diagram illustrating an example of segmentation of a semantic image of an object according to an embodiment of the present invention.

Images for semantic image segmentation were collected from the PASCAL VOC 2012 data set.

12 to 14, the result of the semantic image segmentation for a specific image through the fully convolutional neural network of the present invention based on multi-scale image input and multi-scale extended convolution, the specific image By accurately predicting the meaningful part (ie the object) from the above, it is shown that the semantic image segmentation can be performed accurately.

FIG. 15A is a diagram comparing the performance of a complete convolutional neural network with other methods using a skin lesion data set for melanoma detection according to an embodiment of the present invention.

The data set of skin lesion analysis for melanoma detection according to an embodiment of the present invention was used to evaluate the accuracy of the semantic segmentation between the complete convolutional neural network and other methods proposed in the present invention.

As shown in FIG. 15A, it can be seen that the ResNet-101 based full convolutional neural network has much higher accuracy than the VGG-16 based full convolutional neural network.

That is, if the ResNet-101 based fully convolutional neural network has obtained an IoU score of 83.5, this is the best result among other methods.

It can also be seen that the fully convolutional neural network of the present invention is effective in detecting melanoma of varying size and shape.

This is because the full convolutional neural network can efficiently utilize multi-scale features from multi-scale image input and multi-scale extension convolution.

The fully connected conditional random field model also shows an additional 0.4% performance improvement for ResNet-101 based fully convolutional neural networks.

FIG. 15B is a diagram illustrating semantic image segmentation results of a skin lesion data set for melanoma detection through a full convolutional neural network according to an embodiment of the present invention. FIG.

As shown in FIG. 15B, it can be seen that the fully convolutional neural network according to an embodiment of the present invention shows high performance on semantic image segmentation from skin lesion images including complex changes of melanoma and the presence of artifacts. .

16 is a flowchart illustrating a procedure of performing semantic image segmentation according to an embodiment of the present invention.

As shown in FIG. 16, in the procedure of performing semantic image segmentation according to an embodiment of the present invention, the semantic image segmentation apparatus 100 first loads training data from the training data database 310 and then performs corresponding training. Performs a preprocessing process for the data (S110).

That is, the semantic image segmentation apparatus 100 generates a multi-scale image of the original image constituting the training data according to a scaling ratio preset through a preprocessing process.

Next, the semantic image segmentation apparatus 100 learns the generated multi-scale image and generates a learning model for semantic image segmentation (S120).

As described above, the learning is performed through a full convolutional neural network based on multi-scale image input and extended convolution. In addition, the full convolutional neural network is a key technical feature of the present invention. Since the structure of the full convolutional neural network has been described with reference to FIG. 5, a detailed description thereof will be omitted.

Next, when a specific image for semantic image segmentation is input from the user (S130), the semantic image segmentation apparatus 100 performs the same preprocessing process as in step S110 to generate a multi-scale image of the input specific image. (S140).

Next, the semantic image segmentation apparatus 100 loads a learning model for semantic image segmentation from the training model database 320, and applies the multi-scale image of the generated specific image to the semantic image segmentation learning model, thereby specifying the specific model. The semantic image segmentation on the image is performed (S150), and the result is provided to the user (S160).

As described above, the present invention relates to a semantic image segmentation apparatus and a method using a full-convolution neural network based on multi-scale image and multi-scale extended convolution. The present invention relates to a cascade architecture of multi-scale image input and extended convolution. The complete convolutional neural network on which it is based has the effect of enabling accurate and fast semantic segmentation on specific images.

In addition, the present invention combines the full convolutional neural network with a conditional random field model, thereby effectively improving the accuracy of semantic image segmentation.

In addition, in the above, the present invention has been described above with reference to a preferred embodiment, but the technical idea of the present invention is not limited thereto, and each component of the present invention may be changed or modified within the scope of the present invention to achieve the same object and effect. Could be.

In addition, while the above has been shown and described with respect to the preferred embodiment of the present invention, the present invention is not limited to the specific embodiments described above, in the technical field to which the invention belongs without departing from the spirit of the invention claimed in the claims Various modifications can be made by those skilled in the art, and these modifications should not be individually understood from the technical spirit or the prospect of the present invention.

100: semantic image segmentation device 110: image collection unit
120: pre-processing unit 130: semantic image segmentation learning model generation unit
140: semantic image segmentation unit
150: semantic image segmentation result providing unit
160: database interface unit 170: control unit
200: user terminal 300: database
310: training data database 320: training model database

Claims (10)

A preprocessor for preprocessing learning data to generate a multi-scale image of the original image of the learning data;
And a learning model generator for semantic image segmentation that generates the learning model for semantic image segmentation by learning the generated multi-scale image.
The learning is performed through a full convolutional neural network comprising a plurality of sub-networks consisting of a convolutional neural network with multi-scale images as input and a multi-scale extended convolutional cascade architecture. . The method according to claim 1,
The semantic image segmentation device
Recovering the resolution of the feature map output from each of the sub-networks through the multi-scale extended convolution cascade architecture to generate a high-resolution final feature map,
And re-adjust the generated final feature maps to the same resolution. The method according to claim 2,
The multi-scale extended convolution cascade architecture,
It includes a plurality of extended convolution layers connected in cascade form to perform extended convolution,
Wherein each extended convolutional layer has a different rate, but has a higher rate than a previous extended convolutional layer. The method according to claim 1,
The full convolutional neural network,
A max out layer integrating all feature maps output from the multi-scale extended convolution cascade architecture to extract a boundary for an object; And
And a fully connected conditional random field model for restoring the boundary for the extracted object. The method according to claim 1,
The semantic image segmentation device,
When a specific image for semantic image segmentation is input, the preprocessor controls the pre-processing unit to generate a multi-scale image of the specific image, and applies the generated multi-scale image to the semantic image segmentation learning model. And a semantic image segmentation unit for performing semantic image segmentation for the semantic image segmentation apparatus. A preprocessing step of preprocessing the training data to generate a multi-scale image of the original image of the training data;
And generating a learning model for semantic image segmentation by learning the generated multi-scale image to generate a learning model for semantic image segmentation.
The learning is performed through a full convolutional neural network comprising a plurality of sub-networks consisting of a convolutional neural network with multi-scale images as input and a multi-scale extended convolutional cascade architecture. . The method according to claim 6,
The semantic image segmentation method is
Recovering the resolution of the feature map output from each of the sub-networks through the multi-scale extended convolution cascade architecture to generate a high-resolution final feature map,
And re-adjust the generated final feature maps to the same resolution. The method according to claim 7,
The multi-scale extended convolution cascade architecture,
It includes a plurality of extended convolution layers connected in cascade form to perform extended convolution,
Wherein each extended convolutional layer has a different rate, but has a higher rate than a previous extended convolutional layer. The method according to claim 6,
The full convolutional neural network,
A max out layer integrating all feature maps output from the multi-scale extended convolution cascade architecture to extract a boundary for an object; And
And a fully connected conditional random field model for restoring the boundary for the extracted object. The method according to claim 6,
The semantic image segmentation method,
When a specific image for semantic image segmentation is input, the preprocessor controls the pre-processing unit to generate a multi-scale image of the specific image, and applies the generated multi-scale image to the semantic image segmentation learning model. And a semantic image segmentation step of performing semantic image segmentation on the semantic image segmentation method.

Images