JP6830742B2 - A program for pixel-based image segmentation - Google Patents

Description

The present invention relates to a pixel-based image segmentation technique using deep learning.

There is a technique of image segmentation to recognize an object reflected in an image. This technology detects what object each pixel of an image belongs to, and has been conventionally used by random forest, support vector machine, adaboost, etc. Has been done.

In recent years, a convolutional neural network in deep learning has been applied to image segmentation. By inputting an image into the convolutional neural network, it is possible to extract features and detect the position, contour, and region where the features appear.

"Neural network" refers to a mathematical model that aims to express the characteristics of a living body in the brain by computer simulation. It refers to a general model in which artificial neurons (units) that form a network by synaptic connection change the synaptic connection strength by learning and have problem-solving ability.
Further, in a narrow sense, the "convolutional neural network" means that layers having a plurality of units are connected in one direction from the input stage to the output stage, and the units on the output layer side are on the adjacent input layer side. A forward-propagating network with a "convolutional layer" coupled to a particular unit.

The parameter of the function that connects the unit in the front layer to the unit in the rear layer is called "weight". Learning is to calculate an appropriate "weight" as a parameter of this function. The weight of each layer is optimally updated using the error between the output data from the output layer for the input data of the teacher data and the correct label of the teacher data. The error is propagated one after another from the output layer side to the input layer side by the "error backpropagation method", and the weight of each layer is updated little by little. Finally, a convergence calculation is performed that adjusts the weights of each layer to the appropriate values so that the error is small. The "convolutional neural network" in the present invention includes a single-layer inner circulation type convolutional neural network represented by RNN (Recurrent Neural Network) and LSTM (Long short-term memory), and a multi-layer circulation type convolutional neural network. The network etc. shall be included.

Compared to image recognition, image segmentation must recognize not only "what" is reflected in the image, but also "where" and what the "outline" is. It doesn't become.
Neural networks need to perform image downsampling in order to recognize "what". On the other hand, downsampling reduces the size of the image and makes the contour unclear.

Conventionally, there is a technique using an encoder-decoder structure as a convolutional neural network for recognizing an image object (see, for example, Non-Patent Document 1). According to this technique, the encoder performs a position detection process in object detection, and the decoder maps its features to the position of the object. It integrates the position information of the shallow layer and the feature information of the deep layer of the neural network by a skip structure.

FIG. 1 is a schematic diagram of an encoder-decoder convolutional neural network applied to image segmentation.

According to FIG. 1, the encoder-decoder convolutional neural network inputs a photographic image showing a plurality of people and outputs an object recognition image having the same size as the input image. According to FIG. 1, an object such as a person, a table, or a chair is detected from the output object recognition image, and the position of the object is specified.

Further, as another conventional technique, there is also a technique using a completely convolution structure (see, for example, Non-Patent Document 2). According to this technique, the image is encoded and some layers are integrated by a skip structure to infer the position. It divides the neural network into a Pooling stream and a Residual stream, one that integrates location information and the other that discriminates.

Further, there is also a technique of connecting after the skip structure (see, for example, Non-Patent Document 3). It uses the DenseNet structure to downsample the image and then upsample it to restore it to its original resolution.

Fully Convolutional Networks for Semantic Segmentation, [online], [Search on November 25, 2017], Internet <URL: https://arxiv.org/pdf/1605.06211.pdf> “Full-Resolution Residual Networks for Semantic Segmentation in Street Scenes”, [online], [Searched November 25, 2017], Internet <http://openaccess.thecvf.com/content_cvpr_2017/papers/Pohlen_Full-Resolution_residual_networks_CVPR_2017_paper.pdf ＞ “The One Hundred Layers Tiramisu: Fully Convolutional DenseNets for Semantic Segmentation”, [online], [Search November 25, 2017], Internet <http://openaccess.thecvf.com/content_cvpr_2017_workshops/w13/papers/Jegou_The_One_Hundred_CVPR_2017_paper. pdf ＞

According to the techniques described in Non-Patent Documents 1 and 3, it is necessary to synthesize (for example, resize) images in order to simultaneously learn the estimation of object features and the estimation of pixel position features in one network. There is. Therefore, the pixel class information (class to be recognized) may be lost.
Further, according to the technique described in Non-Patent Document 2, the Residual stream part can also have Full-Resolution, but the pixel class information is not learned and integrated by simple addition.

On the other hand, the inventors of the present application realize high-precision image segmentation without resizing the image in both the pixel estimation stream and the feature estimation stream while maintaining the full-resolution. I wondered if I could do it.

Therefore, an object of the present invention is to provide a program that realizes highly accurate image segmentation by using both pixel estimation and feature estimation.

According to the present invention, in a program that causes a computer to function as pixel-based image segmentation.
A pixel estimation stream that executes a plurality of pixel estimation means for extracting pixel classes in series for an input image,
For the same input image, a feature estimation stream that executes a plurality of feature estimation means for extracting object features in series is executed in parallel, and at the same time.
In the pixel estimation stream, the pixel map output from the pixel estimation means in the previous stage and the feature map output from the feature estimation means in the previous stage are input, and the feature map is uploaded according to the size of the pixel map. After downsampling, the feature map is connected or added to the pixel map, and the pixel map is output to the pixel estimation means in the next stage.
In the feature estimation stream, the pixel map output from the pixel estimation means in the previous stage and the feature map output from the feature estimation means in the previous stage are input, and the pixel map is uploaded / uploaded according to the size of the feature map. After downsampling, the pixel map is connected or added to the feature map, and the feature map is output to the feature estimation means in the next stage.
It is characterized in that a computer functions as a stream integration means for connecting or adding the pixel estimation stream and the feature estimation stream in the final stage.

According to other embodiments in the program of the present invention
The pixel estimation stream contains at least one or more pixel integration means.
It is also preferred to have the computer function so that the feature estimation stream contains at least one feature integration means.

According to other embodiments in the program of the present invention
It is also preferable that the pixel estimation means and the feature estimation means each function the computer so as to be composed of a convolutional neural network.

According to other embodiments in the program of the present invention
It is also preferable that the stream integration means causes the computer to function to further execute the convolution layer after connecting or adding the pixel estimation stream and the feature estimation stream in the final stage.

According to the program of the present invention, high-precision image segmentation can be realized by using both pixel estimation and feature estimation. In particular, in both the flow of the pixel estimation stream and the feature estimation stream, it is possible to realize high-precision image segmentation without resizing the image and maintaining full-resolution.

It is a schematic diagram of an encoder-decoder convolutional neural network applied to image segmentation. It is a functional block diagram of the neural network in this invention. It is a structural drawing which shows the internal structure of each function in this invention. It is explanatory drawing which shows the 1st flow of the pixel map in a pixel integration part. It is explanatory drawing which shows the 2nd flow of the pixel map in a pixel integration part. It is a structural diagram which shows the internal structure of the stream integration part in this invention. It is an image diagram which shows the estimation result in this invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

FIG. 2 is a functional configuration diagram of the neural network in the present invention.

According to FIG. 2, the neural network has a pixel estimation unit 11, a pixel integration unit 12, a feature estimation unit 21, a feature integration unit 22, and a stream integration unit 3. These functional components are realized as programs that make the computer function.

For example, an image (video) taken by a camera is input to the neural network. Of course, this image may be a CG (Computer Graphics) image. On the other hand, the estimation result of the image segmentation in which the pixel class and its position, contour, and region are specified is output.

According to FIG. 2, <pixel estimation stream 1> and <feature estimation stream 2> are executed in parallel for image segmentation based on pixels.
<Pixel estimation stream 1>
The pixel estimation stream 1 executes a plurality of "pixel estimation units 11" for extracting pixel classes in series with respect to the input image.
<Characteristic estimation stream 2>
The feature estimation stream 2 executes a plurality of "feature estimation units 21" for extracting object features in series for the same input image.

According to FIG. 2A, the “pixel integration unit 12” is included between the pixel estimation units 11 in the pixel estimation stream 1, and between the feature estimation units 21 in the feature estimation stream 2. Includes "feature integration unit 22".
Pixel integration section 12, within the pixel estimation streams 1 inputs the pixel map output from the previous stage pixel estimating unit 11n, and a feature map output from the preceding characteristic estimating unit 21 n, a pixel map treated Is output to the pixel estimation unit 11n + 1 in the next stage.
Further, the feature integration unit 22 inputs and processes the feature map output from the feature estimation unit 21n in the previous stage and the pixel map output from the pixel estimation unit 11n in the previous stage in the feature estimation stream 2. The map is output to the feature estimation unit 21n + 1 in the next stage.
Then, the pixel estimation stream 1 and the feature estimation stream 2 are integrated by the stream integration unit 3 of the output stage.

By forming a structure having a complementary relationship between the pixel estimation stream 1 and the feature estimation stream 2 in this way, it is possible to achieve both high-precision pixel class information estimation and high-resolution position information estimation.

According to FIG. 2B, the pixel integration unit 12 is included only between the pixel estimation units 11 in the final stage in the pixel estimation stream 1, and the feature estimation unit 21 in the final stage in the feature estimation stream 2. The feature integration unit 22 is included only between.
As described above, the number of the pixel integration unit 12 and the number of the feature integration units 22 may be at least one, and can be increased or decreased based on the amount of memory used and the computational cost.

FIG. 3 is a structural diagram showing the internal structure of each function in the present invention.

[Pixel estimation unit 11]
The pixel estimation unit 11 is configured by a convolutional neural network in order to calculate the correct answer class of each pixel in the input image.
According to FIG. 3, the pixel estimation unit 11 has a U-shaped shortcut structure (nested structure). According to the U-shaped network, a pixel map is created while reducing the number of elements (number of pixels) by the convolution layer, the normalization layer, and the activation layer. Here, the pixel estimation unit 11 has a structure similar to, for example, ResNet (Residual Network) or U-Net.

The convolutional layer applies a weight filter to the input data, and the sum of the products of the elements is set as one element value of the pixel map. Then, while sliding the weight filter with respect to the input data, a pixel map with enhanced local features is generated. The size of the pixel map output from the convolution layer is S × S, and the number of pixels is N. The number N of pixel maps coincides with the number N of weight filters.
Then, the same weight filter is moved with respect to the input data to generate one pixel map. Here, the number of elements to be moved (movement amount) is referred to as "stride".

The normalization layer executes subtraction normalization in which the average of the shading values in the region is 0, and division normalization in which the variance is normalized.

The activation layer uses, for example, ReLU (Rectified Linear Unit) to enhance neurons with strong signals and suppress weak neurons. The data output from the activation layer is image data in which an object is mapped to each pixel.

Finally, the streams of different layers are merged by linear concatenate rather than by add. That is, the size of the pixel map is S × S × 2N. This allows the difference between different layers to be mixed to prevent network overfitting.
By deepening the layer of the U-shaped network, the amount of calculation increases, but the position for a feature with high expressive power can be detected.

[Pixel integration unit 12]
In the pixel estimation stream 1, the pixel integration unit 12 inputs a pixel map output from the pixel estimation unit 11n in the previous stage and a feature map output from the feature estimation unit 21n in the previous stage. Then, the feature map is connected or added to the pixel map, and the pixel map is output to the pixel estimation unit 11n + 1 in the next stage.

As a result, the pixel estimation unit 11n + 1 in the next stage reflects not only the estimation result (pixel class information) of the pixel estimation unit 11n in the previous stage but also the estimation result (object feature) of the feature estimation unit 21n in the previous stage. Is entered. As a result, the pixel estimation unit 11n + 1 in the next stage is expected to estimate the pixel class more accurately. On the other hand, since the pixel map (pixel class information) output from the pixel estimation unit 11n in the previous stage is directly input, the computational cost required for the learning process can be reduced.

Here, as another embodiment, when the size of the feature map output from the feature estimation unit 21n is different from the size of the pixel map output from the pixel estimation unit 11n, the size of the feature map is set to the size of the pixel map. After resizing, integrate both feature maps. Therefore, the pixel integration unit 12 up / down samples the feature map according to the size of the pixel map. After that, the pixel map and the feature map are integrated by concatenation or addition.

FIG. 4 is an explanatory diagram showing a first flow of the pixel map in the pixel integration unit.

The pixel integration unit 12 inputs a pixel map having a size of S × S × N from the pixel estimation stream 1 (pixel estimation unit 11n in the previous stage). Here, a feature map having a size of, for example, S / 2 × S / 2 × N from the feature estimation stream 2 (feature estimation unit 21n in the previous stage) is input. In this case, the feature map is upsampled to a size of S × S × N that is twice as long and twice as wide. As a result, the feature map output from the upsampling layer and the pixel map of the pixel estimation stream 1 have the same size and can be linearly connected. Then, a pixel map having a size of S × S × 2N can be output to the pixel estimation unit 11n + 1 in the next stage.

On the contrary, when the size of the pixel map of the pixel estimation stream 1 is smaller than the size of the feature map of the feature estimation stream 2, the feature map is downsampled. As a result, the feature map output from the downsampling layer and the pixel map of the pixel estimation stream 1 have the same size and can be linearly connected.

FIG. 5 is an explanatory diagram showing a second flow of the pixel map in the pixel integration unit.

According to FIG. 5, as compared with FIG. 4, the integrated portion is not “consolidated” but “added”. In this, pixel maps and feature maps of the same size are added for each element to obtain N. A pixel map having a size of S × S × N can be output to the pixel estimation unit 11n + 1 in the next stage.

By repeating the pixel estimation unit 11 and the pixel integration unit 12 at least once in order to perform abstract pixel estimation, the estimation result of the feature estimation stream 2 can be reflected in the pixel estimation stream 1. Since the up / down sampling process is not inserted in the flow of the pixel estimation stream 1, Full-Resolution can be maintained and all pixels of the input image can be estimated.

[Characteristic estimation unit 21]
The feature estimation unit 21 is also configured by a convolutional neural network in order to encode the features of the object.
According to FIG. 3, the feature estimation unit 21 creates a feature map while reducing the number of elements by the convolution layer, the pooling layer (pooling layer and average pooling), and the activation layer.
The convolution layer is the same as the pixel number estimation unit 11 described above.
The pooling layer generates a feature map reduced from the input feature map to only the important feature elements, and conversely excludes non-important information.
The activation layer is also the same as the pixel number estimation unit 11 described above, and may be based on, for example, the ReLU, softmax function, or sigmoid function.
By repeating these three layers in order to extract highly abstract features, it can be expected that more expressive features will be extracted.

[Feature integration unit 22]
In the feature estimation stream 2, the feature integration unit 22 inputs a pixel map output from the pixel estimation unit 11n in the previous stage and a feature map output from the feature estimation unit 21n in the previous stage. Then, the pixel map is connected or added to the feature map, and the feature map is output to the feature estimation unit 21n + 1 in the next stage.
Further, the feature integration unit 22 also up / down samples the pixel map according to the size of the feature map, and then connects or adds the pixel map to the feature map in exactly the same manner as the pixel integration unit 12 described above.

As a result, the feature estimation unit 21n + 1 in the next stage reflects not only the estimation result (object feature) of the feature estimation unit 21n in the previous stage but also the estimation result (pixel class information) of the pixel estimation unit 11n in the previous stage. Is entered. As a result, the feature estimation unit 21n + 1 in the next stage is expected to estimate the object recognition more accurately. On the other hand, since the feature map (object feature) output from the feature estimation unit 21n in the previous stage is directly input, the computational cost required for the learning process can be reduced.

[Stream integration unit 3]
The stream integration unit 3 connects or adds the pixel estimation stream and the feature estimation stream at the final stage.

FIG. 6 is a structural diagram showing the internal structure of the stream integration unit in the present invention.

According to FIG. 6, the stream integration unit 3 integrates the pixel map and the feature map having the same size by concatenation or addition. In this case, it is also preferable to insert a convolution layer in the subsequent stage.

FIG. 7 is an image diagram showing an estimation result in the present invention.

According to FIG. 7, the image segmentation result of the prior art and the image segmentation result of the present invention are shown with respect to the input image and the correct answer data.
It can be understood that the present invention has less overall variation in segmentation as compared with the prior art. In particular, it can be understood that the segmentation of the parapet portion on the left side is also relatively clear.

As described in detail above, according to the program of the present invention, highly accurate image segmentation can be realized by using both pixel estimation and feature estimation. In particular, in both the flow of the pixel estimation stream and the feature estimation stream, it is possible to realize high-precision image segmentation without resizing the image and maintaining full-resolution.

According to the present invention, high-precision image segmentation and object detection can be expected while repeating mutual use of the pixel map and the feature map between the pixel estimation stream and the feature estimation stream.
For example, for an input image in which a person is reflected, more accurate contours and regions of the person can be obtained by giving the approximate position information of the person detected in the feature estimation stream to the pixel estimation stream. On the other hand, by giving the approximate outline or region of the person detected in the pixel estimation stream to the feature estimation stream, the position of the person can be obtained more accurately.
In addition, for each of the pixel estimation stream and feature estimation stream, the pixel map and feature map are not resized, and by maintaining full-resolution, the pixel class, position, contour, and pixel class, position, contour, and resolution generated by the change in resolution due to map integration No need to learn the area.

With respect to the various embodiments of the present invention described above, various changes, modifications and omissions within the scope of the technical idea and viewpoint of the present invention can be easily made by those skilled in the art. The above explanation is just an example and does not attempt to restrict anything. The present invention is limited only to the scope of claims and their equivalents.

1 Pixel estimation stream 11 Pixel estimation unit 12 Pixel integration unit 2 Feature estimation stream 21 Feature estimation unit 22 Feature integration unit 3 Stream integration unit

Claims (4)

In a program that makes a computer function as pixel-based image segmentation
A pixel estimation stream that executes a plurality of pixel estimation means for extracting pixel classes in series for an input image,
For the same input image, a feature estimation stream that executes a plurality of feature estimation means for extracting object features in series is executed in parallel, and at the same time.
In the pixel estimation stream, the pixel map output from the pixel estimation means in the previous stage and the feature map output from the feature estimation means in the previous stage are input, and the feature map is uploaded according to the size of the pixel map. After downsampling, the feature map is connected or added to the pixel map, and the pixel map is output to the pixel estimation means in the next stage.
In the feature estimation stream, the pixel map output from the pixel estimation means in the previous stage and the feature map output from the feature estimation means in the previous stage are input, and the pixel map is uploaded according to the size of the feature map. After downsampling, the pixel map is connected or added to the feature map, and the feature map is output to the feature estimation means in the next stage.
A program characterized in that a computer functions as a stream integration means for connecting or adding the pixel estimation stream and the feature estimation stream in the final stage. The pixel estimation stream includes at least one or more pixel integration means.
The program according to claim 1 , wherein the computer is made to function so that the feature estimation stream includes at least one feature integration means. The program according to claim 1 or 2 , wherein each of the pixel estimation means and the feature estimation means causes a computer to function so as to be configured by a convolutional neural network. The program according to claim 3 , wherein the stream integrating means causes a computer to further execute a convolution layer after connecting or adding the pixel estimation stream and the feature estimation stream in the final stage.