AU2020103901A4 - Image Semantic Segmentation Method Based on Deep Full Convolutional Network and Conditional Random Field - Google Patents

Abstract

The invention provides an image semantic segmentation method based on deep full convolutional network and conditional random field (CRF), which comprises the following steps: construction deep full convolutional semantic segmentation network model; structured prediction of pixel label based on fully connected CRF; model training, parametric learning, and image semantic segmentation. In the omvention, dilated convolution and spatial pyramid pooling modules are introduced in deep full convolutional networks, and CRF is used to further modify the label prediction map produced by the deep full convolution network. The dilated convolution expands the receptive field, while ensuring that the resolution of the feature image remains the same. In the spatial pyramid pooling module, the contextual features of different scale regions are extracted from the convolutional local feature map, and the correlation between different objects and the relationship between object and their different scale region features are provided for label prediction. Finally, the fully connected CRF further optimizes the pixel label according to the feature similarity of pixel intensity and position to generate a semantic segmentation map with the features of high resolution, accurate boundary and good spatial continuity. -1/3 Deep full convolutional semantic segmentation Fully connected CRF Regional--------- - feature atII Itrio Feature Space pyramid different PIeai I scales reasoning of extraction P pooling module Feature concatenation classification approximate module prediction M probability of UT module CD mean field Convolution local feature Figure. 1

Description

-1/3

Deep full convolutional semantic segmentation Fully connected CRF

Regional--------- - feature atII Itrio Feature Space pyramid different PIeai I scales reasoning of extraction P pooling module Feature concatenation classification approximate module prediction M probability of UT module CD mean field

Convolution local feature

Figure. 1

Image Semantic Segmentation Method Based on Deep Full Convolutional

Network and Conditional Random Field

TECHNICAL FIELD

The invention relates to the technical field of image understanding, in particular to an

image semantic segmentation method based on deep full convolutional network and

conditional random field (CRF).

BACKGROUND

Image semantic segmentation is to label image pixels according to their semantics to

form different segmentation areas. Semantic segmentation is a crucial technology of

image understanding, which plays a pivotal role in street scene recognition and

understanding of the automatic driving systems, the landing site judgment of

Unmanned Aerial Vehicle (UAV), and lesion recognition and position in medical

images.

The emergence of deep learning technology has significantly improved the

performance of the semantic segmentation of image compared with traditional

methods. Using deep convolutional neural networks to perform supervised learning on

large data sets is the current mainstream method of semantic segmentation of image.

In this method, the image to be segmented is input, the continuous convolution and

down-sampling operations are used to extract image features step by step, and then

the final features is used to classify image pixels. However, the continuous

convolution and down-sampling operations in the method of semantic segmentation

of image based on deep learning technology will continuously reduce the feature map,

and the position detail information will be continuously lost, resulting in low

resolution of the segmentation map, difficulty in positioning the segmentation

boundary, and rough segmentation map. In addition, in the pixel classification method based on the deep convolutional network, the prediction of each pixel label is performed independently. When there is a lack of the prior knowledge and structural constraint, pixels with similar features are not encouraged to generate the same classification label, and isolated misclassification regions may be generated easily. Beyond that, when there are many categories of semantic segmentation, due to the lack of context relations between the different objects and between the object and the background, objects with similar appearances may be confused easily during classification. So the smaller objects are difficult to discover, and the larger objects result in discontinuous prediction when they are beyond the receptive field.

SUMMARY

In view of the problems of the existing methods, the invention provides a semantic segmentation method of image based on deep full convolutional networks and CRF. In the omvention, dilated convolution and spatial pyramid pooling modules are introduced in deep full convolutional networks, and CRF is used to further modify the label prediction map produced by the deep full convolution network.The dilated convolution expands the receptive field, while ensuring that the resolution of the feature image remains the same.. In the spatial pyramid pooling module, the contextual features of different scale regions are extracted from the convolutional local feature map, and the correlation between different objects and the relationship between object and their different scale region features are provided for label prediction. The fully connected CRF further optimizes the pixel label according to the feature similarity of the pixel intensity and position to generate a semantic segmentation map with the characteristics of high resolution, accurate boundary and good spatial continuity.

The invention adopts the following technical scheme to solve the above technical problems.

The invention relates to an image semantic segmentation method based on deep full convolutional network and conditional random field (CRF), which comprises the following steps:

Si. Construction of deep full convolutional semantic segmentation network

model.

S11. The deep full convolutional semantic segmentation network model consists

of a feature extraction module, a pyramid pooling module and a pixel label prediction

module. The recited feature extraction module extracts local features of the image by

performing convolution, maximum pooling and dilated convolution operations on the

input image; the recited pyramid pooling module performs spatial pooling of different

scales on the convolutional local features to extract the context features at different

scales; and the recited pixel label prediction module utilizes the convolutional local

features and combines the regional context features at different scale to predict the

pixel categories;

S12. The recited feature extraction module comprises the first to the fifth

convolutional layer groups, the first to the third maximum pooling layers, the first

dilated convolutional layer and the second dilated convolutional layer. The recited

first maximum pooling layer is located after the first convolutional layer group. The

recited second maximum pooling layer is located after the second convolutional layer

group, and the recited third maximum pooling layer is located after the third

convolutional layer group. The recited first dilated convolutional layer is located after

the fourth convolutional layer group, and the recited second dilated convolutional

layer is located after the fifth convolutional layer group. The recited pyramid pooling

module first use N types of different container sizes to perform N-level average

pooling on the convolutional local features output by the second dilated convolutional

layer to obtain N types of low-resolution regional context features at different scales.

And then, N types of regional context features at different scales are convoluted

respectively, and the output channel number is 1/N of the number of original feature

channel. After that, N types of low-resolution regional context features at different

scales are up sampled to the size of the original feature image. The recited pixel label prediction module comprises a first feature projection layer, a second feature projection layer, a category prediction layer and a Softmax probability conversion layer which are sequentially arranged. The recited pixel label prediction module first connects and fuses the convolutional local features and N types of up-sampled regional context features at different scales, uses the features after fusion to predict the pixel classification label, and then uses the Softmax probability conversion layer to convert the pixel classification label prediction score into pixel classification label prediction probability distribution;

S2. Structured prediction of pixel label based on a fully connected CRF: the fully

connected CRF is used to postprocess pixel classification label which is output by a

deep full convolutional semantic segmentation network to remove misclassified

isolated pixels or regions, optimize pixel labels near the boundary of a complex object,

and allow output segmentation map to have good spatial consistency and precise

boundary, which specifically comprises:

S21. The mutual relations between the variable probabilities of any two pixel

labels are modeled by using the fully connected CRF;

S22. The following Gibbs energy function is used for the fully connected CRF

model:

E(x)= Vy (xi) + I ,P(xi,,xj) i i,i,i<i

Where, x is a pixel classification label variable, xi and Xj are the labels

corresponding to the i t pixel and the j pixel respectively, V, is a univariate

potential function and V, is a pairwise potential function;

S23. The pixel classification label probability is modeled by using an iterative

reasoning algorithm of mean approximate probability of field, and an optimized pixel

classification label prediction probability distribution map is outputted.

S3. Model training and parameter learning;

S31. The parameters of the segmentation network model is initialized by using a

Xavier method;

S32. After expanding the training data, the training data will be divided into

training set, validation set and test set according to the ratio of 5:1:1, and the six-fold

cross-validation method is used to train the segmentation network model;

S33. The RGB image to be segmented is inputted as three channels into a deep

full convolutional semantic segmentation network, and a pixel classification label

prediction probability distribution is generated. The prediction loss is calculated by

using the label prediction probability and the segmentation label, and specifically a

classified cross entropy loss function is used as the objective function. The definition

is as follows:

1B S C L()= x jl log() 1I S B 1

Where, Y is the probability vector of segmentation label, Y is the label

prediction probability vector, C is the number of pixel classifications, S is the

number of image pixels, log(.) is the natural logarithm and B is the batch size;

S34. The stochastic gradient descent algorithm is used to optimize the objective

function, and the reverse propagation algorithm of error is used to update the

parameters of the deep full convolutional semantic segmentation network model. The

specific optimization process is as follows:

g, =V ,_ 1 L(O,_,)

v, = p*v,-r g,

Ot =0,1 +V,

Where, the subscript t is the number of iterations, 0 is a network model

parameter, L(0O,_) is a loss function when using 6,_I as a network parameter, g, v, and P are a gradient, a momentum and a momentum coefficient respectively, and t1 is a learning rate;

S4. Semantic segmentation of image:

S41. The RGB image to be segmented is inputted as three channels into a deep

full convolutional semantic segmentation network to carry out forward calculation;

S42.The convolutional local feature image of the image is outted through

convolution, maximum pooling and dilated convolution operations by the feature

extraction module;

S43. The convolutional local feature image is inputted into the pyramid pooling

module to generate regional context feature images at different scales;

S44. The convolutional local feature image to the regional context feature images

at different scales are connected, and the pixel label prediction module is inputted;

S45. The pixel label prediction module first performs convolution fusion on the

convolutional local features and the regional context features at different scales, and

then predicts the pixel classification by using the fusion features. After that, a pixel

classification label prediction probability distribution map is output;

S46. The label prediction probability distribution map of pixel classification

output by the deep full convolutional semantic segmentation network is inputted into

a fully connected CRF, the pixel classification label prediction probability distribution

of according to the intensity between pixels and the similarity of position features is

optimized, and the structured pixel classification label prediction probability

distribution map is outputted;

S47. The subscript of the maximum probability component in each pixel

probability distribution vector is taken as a pixel classification label to obtain a final

semantic segmentation image.

Furthermore, in Step S12, each convolutional layer group is composed of two convolutional layers, the size of convolution kernel of each convolution layer is 3x3, and the step length is 1. The number of convolution kernels of the first to fifth convolutional layer groups is 64, 128, 256, 512 and 1,024 in sequence; the pool kernel size of each maximum pool layer is 2x2, and the step length is 2; and the convolution kernel size of each dilated convolutional layer is 3x3, and the step size is 1. The dilated factors of the first and the second dilated convolutional layer are 2 and 4 respectively; the number of pyramid pooling stages in the pyramid pooling module is

4, the average size of 4- levels of average pooling containers is 1x1, 2x2, 4x4 and 8x8,

respectively, the size of convolution kernel in each stage is 1x1, the step length is 1,

and the number of convolution kernels in each stage is 256; and the size of

convolution kernel in each feature projection layer is 1x1, and the step length is 1.

The number of convolution kernels in the first and the second feature projection layer

are 1,024 and 512 respectively; and the size of convolution kernel of the category

prediction layer is 1x1, the step length is 1, and the number of convolution kernels is

32.

Furthermore, in Step S12, the dilated convolution of the first and the second

dilated convolution layer are calculated by using the following formula:

M N

Z(i,j) Y X(i+ r x m, j+ r x n)@ W(m,n) m=1 n=1

Where, (i, j) is theith lineand "thcolumn, W is the convolution kernel,

X is the input channel, Z is the convolution output image, (M, N) is the number

of convolution kernel dimensions, @ is the convolution operation, and r is the

dilated factor.

Furthermore, the output feature image Zr corresponding to any dilated

convolution kernel in the dilated convolution is calculated by using the following

formula:

Z,(i, j)= X,(i + r x m,j + r x n)@ W,,n) k=1 m=1 n=1

Where, 1 is the dilated convolution kernel number, K is the number of input

channels, and (M, N) is the number of dimensions of the convolution kernel.

Furthermore, Step S12 also comprises the batch standardization operation of

output feature image generated by the convolution layer, dilated convolution layer and

feature projection layer.

Furthermore, the LReLU function is used as an activation function in the recited

deep full convolutional semantic segmentation network, which is used for carrying

out non-linear transformation on each value in the feature image after batch

standardization, and the recited LReLU function is defined as follows:

f(z)= max(O, z)+a min(O, z)

Where, f(z) is a nonlinear excitation unit function, max (.) function is to find a

maximum value, min(.) function is to find a minimum value, z is an input value,

and a is a Leaky parameter.

Furthermore, in Step S12, the Softmax function is defined as follows:

Y = soft max(0,)= cexp(O) exp(o')

Where, 0, is a prediction score of a certain pixel in the i* classification,}Y

is the prediction probability of a pixel on the i th classification, C is the number of

pixel classifications, and exp(.) is an exponential function with natural constant e as

base.

Furthermore, in Step S22, the univariate potential function J is defined as

follows:

V.(x,) = -log P(x1

) Where, P(xi) is the label prediction probability of the classification label of the

ith pixel output by the deep full convolutional semantic segmentation network, and

log(.) is the natural logarithm;

The pairwise potential function V, is defined as follows:

2 2 2

x_)=M _xXmexp(-)+m 2 exp(-

) 20 2a 20,

2 2

Where, exp(- - ) is an external Gauss kernel, 2c, 2

exp(- ) is a smooth Gauss kernel, P(x,xj) is a label compatibility 20>

function, P(xi,xj)=[x,#x 1 ]. pi and P are the corresponding positions of the

i th and i tf pixel, I, and 17 are the corresponding intensities of the i th and i th

pixel, a, Up and 0, are Gauss kernel parameters, and Ci and 02 are relative

intensity of the two Gauss kernels.

Furthermore, in Step S33, the final objective function obtained is as follows by

adding Li and L2 regularization term into loss function:

L(6) = 1B x B SCQQ lI og(Y,,) + -1 6+2 S x Bk1 i1 j1 Q =1 Q i=1

Where, 21 and A2 are the regularization factors of Li and L 2 , O6 is the

parameter of the segmented network, and Q is the number of parameters of O6.

Furthermore, in Step 34, the linear attenuation of learning rate is introduced, and

the learning rate is attenuated according to the following rule:

77, =1-)x 77o +-x 77,

Where, q, is the learning rate used by the th iteration, 70 is the starting

learning rate, 7, is the final learning rate and r is the total number of iterations.

Compared with the existing technique, the image semantic segmentation method

based on deep full convolutional network and CRF provided by the invention have the

following advantages:

1. By using dilated convolution, the number of dimensions of the feature image

will not be reduced while the neuron receptive field is enlarged, and the resolution of

the feature image is improved, so that the final segmentation image has a high

resolution;

2. The pyramid pooling module extracts regional context features at different

scales from the convolutional local feature image. These features are taken as priori

knowledge to predict the pixel classification with the local features generated by the

deep full convolutional network, which is equivalent to fully considering the

relationship between different objects and the interrelationship between the objects

and the background during the pixel prediction, so that the prediction error rate of the

pixel classification can be obviously reduced;

3. The fully connected CRF utilizes pixel intensity and pixel position features,

and the same label shall be allocated to pixels with similar positions and similar

features, which can remove isolated segmentation areas, so that the segmentation

image has good appearance and spatial consistency;

4. By the combination of the multi-level pyramid pooling technology and the

fully connected CRF, the fine grain boundary of the complex object can be segmented,

so that the regional boundary of semantic segmentation image could be more

accurate;

5. The segmentation of the small-size object can be realized, and continuous label prediction can be generated when the large-size object exceeds the receptive field.

BRIEF DESCRIPTION OF THE FIGURES

Figure 1 is a schematic flow diagram of an image semantic segmentation method based on deep full convolutional network and CRF provided by the present invention.

Figure 2 is a schematic diagram of the feature extraction network structure provided by the present invention.

Figure 3 is a schematic diagram of the multi-scale regional feature extraction module based on multi-level pyramid pooling provided by the present invention.

DESCRIPTION OF THE INVENTION

In order to make the technical means, creative features, achieved objectives and efficacy of the invention easy to understand, the present invention will be further described with reference to the specific drawings and preferred embodiments.

Please refer to Figure 1 to 3. The invention provides an image semantic segmentation method based on deep full convolutional network and CRF, which comprises the following steps:

Si. Construction of deep full convolutional semantic segmentation network model.

S11. The deep full convolutional semantic segmentation network model comprises a feature extraction module, a pyramid pooling module and a pixel label prediction module. The recited feature extraction module extracts local features of the image by performing convolution, maximum pooling and dilated convolution operations on the input image; the recited pyramid pooling module pools the convolution local features at different scales to extract the regional context features at different scales; and the recited pixel label prediction module uses the convolutional local features and combines the regional context features at different scales to predict pixel classifications.

S12. The recited feature extraction module comprises the first to the fifth convolutional layer groups, the first to the third maximum pooling layers, the first dilated convolutional layer and the second dilated convolutional layer. the recited first maximum pooling layer is located after the first convolutional layer group, the recited second maximum pooling layer is located after the second convolutional layer group, the recited third maximum pooling layer is located after the third convolutional layer group, the recited first dilated convolutional layer is located after the fourth convolutional layer group, and the recited second dilated convolutional layer is located after the fifth convolutional layer group. That is, each convolutional layer group is followed by a maximum pooling layer or dilated convolutional layer. In order to ensure that the size of the feature image after convolution is the same as it before Convolution, set Padding as 1 in the convolution process. That is, fill the surrounding image with a value of 0 during convolution. The recited pyramid pooling module first uses N types of different container sizes (bin size) to perform N-level average pooling on the convolutional local features output by the second dilated convolutional layer to obtain N types of low-resolution regional context features at different scales. After that, N types of regional context features at different scales are convolved separately, the number of output channels is 1/N of the number of original feature channel, and then N types of regional context features at different scales are up-sampled to the size of the original feature image. The recited pixel label prediction module comprises a first feature projection layer, a second feature projection layer, a classification prediction layer and a Softmax probability conversion layer which are sequentially arranged. The recited pixel label prediction module first connects and fuses the convolutional local features and N types of up-sampled regional context features at different scales, predicts a pixel classification label by using the features after fusion, and then converts a pixel classification label prediction score value into a pixel classification label prediction probability distribution by using the Softmax probability conversion layer;

As a specific embodiment, the detailed structure of the recited deep full

convolutional semantic segmentation network model is shown as Table 1 below,

wherein the 480x480 input image is taken as an example for illustration in Table 1. Of

course, the size of the input image can be any other size:

Table 1 Parameter Table of Deep Full Convolutional Semantic Segmentation

Network Model (Padding=1) Convolution Container Convolution Step Dilated No. Layer name kernel Input Output size kernel size length factor number Convolution 1 Layer - 3x3 1 64 480x480x3 480x480x64 1_l+BN+LReLU Convolution 2 Layer - 3x3 1 64 480x480x64 480x480x64 1_2+BN+LReLU Maximum 3 - 2x2 2 - 480x480x64 240x240x64 Pooling Layer 1 Convolution 4 Layer - 3x3 1 128 240x240x64 240x240x128 2_1+BN+LReLU Convolution Layer - 3x3 1 128 240x240x128 240x240x128 2_2+BN+LReLU Maximum 6 - 2x2 2 - 240x240x128 120x120x128 Pooling Layer 2 Convolution 7 Layer - 3x3 1 256 120x120x128 120x120x256 3_1+BN+LReLU Convolution 8 Layer - 3x3 1 256 120x120x256 120x120x256 3_2+BN+LReLU Maximum 9 - 2x2 2 - 120x120x256 60x60x256 Pooling Layer 3 Convolution Layer - 3x3 1 512 60x60x256 60x60x512 4_1+BN+LReLU 11 Convolutiol - 3x3 1 512 60x60x512 60x60x512 -

Layer 4_2+BN+LReLU Dilated 12 Convolution 1 - 3x3 1 512 60x60x512 60x60x512 2 +BN+LReLU

Convolution 13 Layer - 3x3 1 1024 60x60x512 60x60x1024 5_1+BN+LReLU

Convolution 14 Layer - 3x3 1 1024 60x60x1024 60x60x1024 5_2+BN+LReLU Dilated Convolution 2 - 3x3 1 1024 60x60x1024 60x60x1024 4 +BN+LReLU

1x1 lx1 1 256

16 Pyramid 4-scale 2x2 lx1 1 256 60x60x1024 60x60x2048 pooling 4x4 lx1 1 256 8x8 1xi 1 256

Feature

17 Projection Layer - lx1 1 1024 60x60x2048 60x60x1024 1+BN+LReLU

Feature 18 Projection Layer - lxl 1 512 60x60x1024 60x60x512 2+BN+LReLU

Classification 19 Prediction 1x1 1 32 60x60x512 60x60x32 Layer+Softmax

As can be drawn from Table 1, in Step S12, each convolution layer group

consists of two convolutional layers, the size of each convolution kernel of each

convolution layer is 3x3, and the step size is 1. The number of convolution kernels of

the first to the fifth convolution layer groups is 64, 128, 256, 512 and 1,024 in

sequence; the pool kernel size of each maximum pooling layer is 2x2, and the step

length is 2; and the convolution kernel size of each dilated convolution layer is 3x3,

and the step length is 1. The dilated factors of the first dilated convolution layer and

the second dilated convolution layer are 2 and 4 respectively; the number of pyramid

pooling stages in the pyramid pooling module is 4 and the average size of 4 levels of

average pooling containers are 1x1, 2x2, 4x4 and 8x8, respectively. Through 4-level

average pooling, the original feature map can be averagely divided into 1, 4, 16 and

64 parts, the average value is obtained in each part to replace the original feature

value, and 4 types of regional context features are obtained. For each level,

convolution kernels with a size of lx 1, a step length of 1, and a number of 256 are used for convolution. Then, the size of the original feature image is obtained by

up-sampling; finally, through a pixel label prediction module, the convolutional local

features and the 4 types of up-sampled regional context features at different scales are

connected (Concatenation) and fused; the size of the convolution kernel of each

feature projection layer is 1 X1 and the step length is 1; the number of convolution

kernels of the recited first feature projection layer and the second feature projection

layer are 1,024 and 512 respectively; and the size of the convolution kernel of the

recited class prediction layer is 1 X 1, the step length is 1, the number of the

convolution kernels is 32, and 32 represents the number of classifications of the

semantic label output of pixel. Of course, the number of pyramid pooling stages, the

size of each level of containers, and the number of classifications of semantic label

output of pixel are not limited to the aforementioned parameter settings, and can also

be determined according to actual conditions.

As a specific embodiment, the computing operation of the deep full

convolutional semantic segmentation network model comprises:

(1) Dilated convolution:

The dilated convolution is to up-sample (dilation) the convolution kernel. The

weight of the original position of the convolution kernel remains unchanged, while the

middle position is supplemented with 0. The dilated convolution can improve the

receptive field by using different dilated factors to obtain the regional context features

at different scales, but does not increase the network parameters and the calculation

amount. Compared with the maximum pooling operation, it does not lead to the

reduction of the resolution of the feature image. Specifically, in Step S12, the dilated

convolution of the first dilated convolution layer and the second dilated convolution

layer are calculated by using the following formula:

Z(ij)= YX(i+rx m,j+rx n)@ W(m,n) (1)

Where, (i, j) is the ith line and J th column, W is the convolution kernel,

X is the input channel, Z is the convolution output image, (M, N) is the number

of convolution kernel dimensions, 0 is the convolution operation, and r is the

dilated factor. When r =1, it is equivalent to ordinary convolution.

Where, the output feature image Z, corresponding to any dilated convolution

kernel in the recited dilated convolution is calculated by using the following formula:

Z,(i,j)= EX,(i+rxm,j+rxn)0W(m,n) (2) k=1 m=1 n=1

Where, / is the number of dilated convolution kernel, K is the number of input

channels, and (M, N) is the number of convolution kernel dimensions.

(2) Batch standardization:

In order to make the input of each layer have a stable distribution and the

activation function to be distributed in a linear interval, thereby resulting in a greater

gradient to accelerate convergence. in Step S12, batch normalization (BN) operation

on the output feature image generated by the convolutional layer, the dilated

convolutional layer, and the signature projection layer is performed. namly, normalizing the output image generated by the convolution and the dilated

convolution, subtracting the average value, and dividing by the standard deviation.

(3) Nonlinear excitation LReLU:

The recited deep full convolutional semantic segmentation network uses an

LReLU (Leaky Rectifier Linear Units) function as an activation function for carrying

out non-linear transformation on each value in the feature image after batch

standardization, and the recited LReLU function is defined as follows: f(z)= max(0, z)+a min(0, z) (3)

Where, f(z) is a nonlinear excitation unit function, max (.) function is to find a

maximum value, min(.) function is to find a minimum value, z is an input value, a

is a Leaky parameter, and a = 0.3.

(4) Classification function Softmax:

The Softmax function is used to convert the prediction score of pixel

classification label output by the split network into a prediction probability

distribution of pixel classification label, and the Softmax function used is defined as

follows:

),=soft max(0,)= exp(Oi) C(4

exp(o,)

Where, 0, is a prediction score of a pixel on the i th classification, g is the

prediction probability of a pixel on the i th classification, C is the number of pixel

classifications, C = 32 , and exp(.) is an exponential function with natural

constant e as base.

S2. The structured prediction of pixel label based on a fully connected CRF: the

fully connected CRF is used to postprocess pixel classification label which is output

by a deep full convolutional semantic segmentation network to remove misclassified

isolated pixels or regions, optimize pixel labels near the boundary of a complex object,

and allow output segmentation map to have good spatial consistency and precise

boundary, which specifically comprises:

S21. The mutual relations between the variable probabilities of any two pixel

labels are modeled by using the fully connected CRF, and specifically, using

undirected graphical model of probability well known in this field to model the

prediction probability of the pixel classification label;

S22. The following Gibbs energy function is used for the fully connected CRF

model

E(x)= V/ (x,) + IV, (x,, xj) (5) i i,j,i<j

Where, x is a pixel classification label variable, xi and X; are the labels

corresponding to the i ' pixel and the jth pixel respectively, q, is a univariate

potential function and V, is a pairwise potential function;

In Gibbs energy function, q, is a unitary potential function, which is defined as

follows:

V"((x) = -log P(xi) (6)

Where, P(xi) is the prediction probability of classification label of the ith

pixel output by the deep full convolutional semantic segmentation network, and

log(.) is the natural logarithm;

In the Gibbs energy function, V, is a pairwise potential function, which is

defined as follows:

2 2 2

V, i, xj)=#P , xj)[oi exp(- - _2 exp(- )] (7) 20 2o-, 20

2 2 pi-pi I-I Where, exp(- - is an external Gauss kernel, 20T 20 |||2

exp(- ) is a smooth Gauss kernel, P(xi,x;) is a label compatibility

function, p(xi,x)=[x # x], p, and Pj are the corresponding positions of the i

th and i * pixel, Iand I, are the corresponding intensities of the i th pixel and

the j th pixel (or RGB color values), o7 , 0-p and 0-, are Gauss kernel parameters, and 0i and 02 are relative intensity of the two Gauss kernels; the appearance

Gaussian kernel is related to the pixel position and intensity, forcing the same label to

be assigned to pixels with similar positions and intensities; the smooth Gaussian

kernel is only related to pixel positions, smoothing local pixel boundaries and

removing the abnormal classification points or regions; the function of the label

compatibility function is to punish only when the i th and the j Ith pixel have

different labels; and specifically, the grid search method well known in this field can

be used to obtain 1 ,o and op. m2 =1 and O,=1;

S23. The pixel classification label probability is calculated by using an iterative

reasoning algorithm of mean approximate probability of field well-known in this field,

and an optimized pixel classification label prediction probability distribution map is

outputted ;

S3. Model training and parameter learning:

S31. The parameters of segmentation network model initialized by using a

Xavier method;

S32. The training data samples acquired , and the training data samples is

augmented by using a data augmentation technology of horizontal flip, vertical flip,

cutting after being zoomed out, rotation of 45, rotation of 90, rotation of 135°,

rotation of 180, rotation of 225, rotation of 270° and rotation of 315° to increase the

number of training data samples to be 10 times of the original number of data samples;

and then the training data classified into a training set, a validation set and a test set

according to the ratio of 5:1:1, and the segmentation network model is trained by

using a six-fold cross-validation method;

S33. The RGB image to be segmented is inputted as three channels into a deep

full convolutional semantic segmentation network, a pixel classification prediction

probability distribution is generated , a prediction loss is calculated by using the label

prediction probability and the segmentation label, and specifically using a classified cross entropy loss function as an objective function. The definition is as follows:

B S C 1 L()= x B log( ) (8) S k=1 1 1=

' Where, Y is the probability vector of segmentation label, Y is the label

prediction probability vector, C is the number of pixel classifications, S is the

number of image pixels, log(.) is the natural logarithm and B is the batch size, i.e.,

the number of samples used in each iteration during the random gradient descent

iteration. Assume that C = 32, S = 480x 480 = 230400 and B = 16;

To prevent overfitting, a regularization term including L, and L 2 is added to

the loss function shown in Equation (8), resulting in the final target function as

follows:

1 B SC Q Q L(O)= x B log(y.)+_J0 i+ 0i 21 9 Sx k1 i1 1= Q =1 Q =1

Where, A and A2 are the regularization factors of Li and L 2 , 4 and A2

are both set as 0.1, O, is the parameter of the segmented network, and Q is the

number of parameters of 0,.

S34. The stochastic gradient descent algorithm is used to optimize the objective

function and use the reverse propagation algorithm to update the parameters of the deep full convolutional semantic segmentation network model. The specific optimization process is as follows:

g, = V,_1L(0,_) (10)

v,=P*v, 1 -r ,g, (11)

t,= 0,_I+ v, (12)

Where, the subscript t is the number of iterations, 0 is a network model

parameter, L(O,-) is a loss function when using 0,_, as a network parameter,

g, v, and P are a gradient, a momentum and a momentum coefficient respectively,

q is a learning rate; and setting p = 0.9, and setting the initial learning rate as 10-;

In order to suppress the gradient noise caused by the random gradient descent and ensure the model convergence, the linear attenuation of learning rate 1 is introduced in Step S34, and the learning rate is attenuated according to the following rule:

77, = - )x 77o+ -x 77, (13)

Where, 7, is the learning rate used by the t * iteration, 7o is the starting

learning rate, q, is the final learning rate and r is the total number of iterations.

Assumethat 7,=770/1000 and r=100000.

S4. Semantic segmentation of image:

S41. The RGB image to be segmented is inputted as three channels into a deep full convolutional semantic segmentation network to carry out forward calculation;

S42. The convolutional local feature image of the image is outted through convolution, maximum pooling and dilated convolution operations by the feature extraction module;

S43. The convolutional local feature image is inputted into the pyramid pooling module to generate regional context feature images at different scales;;

S44. The convolutional local feature image to the regional context feature images at different scales are connected, and the pixel label prediction module is inputted;

S45. The pixel label prediction module first performs convolution fusion on the convolutional local features and the regional context features at different scales, and then predicts the pixel classification by utilizing the fusion features. After that, a pixel classification label prediction probability distribution map is output;

S46. The label prediction probability distribution map of pixel classification output by the deep full convolutional semantic segmentation network is inputted into a fully connected CRF, the pixel classification label prediction probability distribution of according to the intensity between pixels and the similarity of position features is optimized, and the structured pixel classification label prediction probability distribution map is outputted;

S47. The subscript of the maximum probability component in each pixel probability distribution vector is taken as a pixel classification label to obtain a final semantic segmentation image..

Compared with the existing technique, the image semantic segmentation method based on deep full convolutional network and CRF provided by the invention have the following advantages:

1. By using dilated convolution, the number of dimensions of the feature image will not be reduced while the neuron receptive field is enlarged, and the resolution of the feature image is improved, so that the final segmentation image has a high resolution;

2. The pyramid pooling module extracts regional context features at different scales from the convolutional local feature image, and the features are taken as priori knowledge to predict the pixel classification with the local features generated by the deep full convolutional network, which is equivalent to fully considering the relationship between different objects and the interrelationship between the objects and the background during the pixel prediction, so that the prediction error rate of the pixel classification can be obviously reduced;

3. The fully connected CRF utilizes pixel intensity and pixel position features, and the same label shall be allocated to pixels with similar positions and similar features, which can remove isolated segmentation areas, so that the segmentation image has good appearance and spatial consistency;

4. By the combination of the multi-level pyramid pooling technology and the

fully connected condition random field, the fine grain boundary of the complex object

can be segmented, so that the regional boundary of semantic segmentation image is

more accurate;

5. The segmentation of the small-size object can be realized, and continuous

label prediction can be generated when the large-size object exceeds the receptive

field.

Finally, it is to be noted that the above-mentioned embodiments are only used to

illustrate rather than limit the technical solutions of the invention. Although the

invention has been described in detail with reference to preferred embodiments

thereof, those skilled in the field should understand that the technical solution of the

present invention can be modified or equivalently replaced without departing from the

purpose and scope of the technical solution of the invention, and all of them shall be

covered by the scope of the claims of the present invention..

Claims (10)

Claims

1. An image semantic segmentation method based on deep full convolutional

network and CRF, which is characterized by comprising the following steps:

S. Construction of deep full convolutional semantic segmentation network

model.

S11. The deep full convolutional semantic segmentation network model

comprises a feature extraction module, a pyramid pooling module and a pixel label

prediction module. The recited feature extraction module extracts local features of the

image by performing convolution, maximum pooling and dilated convolution

operations on the input image; the recited pyramid pooling module pools the

convolution local features at different scales to extract the regional context features at

different scales; and the recited pixel label prediction module uses the convolutional

local features and combines the regional context features at different scales to predict

pixel categories.

S12. The aid feature extraction module comprises the first to the fifth

convolutional layer groups, the first to the third maximum pooling layers, the first

dilated convolutional layer and the second dilated convolutional layer. the recited first

maximum pooling layer is located after the first convolutional layer group, the recited

second maximum pooling layer is located after the second convolutional layer group,

the recited third maximum pooling layer is located after the third convolutional layer

group, the recited first dilated convolutional layer is located after the fourth

convolutional layer group, and the recited second dilated convolutional layer is

located after the fifth convolutional layer group. The recited pyramid pooling module

first uses different container sizes (bin size) to perform N-level average pooling on the

convolutional local features output by the second dilated convolutional layer to obtain

N types of low-resolution regional context features at different scales. And then, N

types of regional context features at different scales are convoluted respectively, and

the output channel number is 1/N of the number of original feature channel. After that,

N types of regional context features at different scales are up-sampled to the size of

the original feature image. The recited pixel label prediction module comprises a first

feature projection layer, a second feature projection layer, a classification prediction

layer and a Softmax probability conversion layer which are sequentially arranged. The

recited pixel label prediction module first connects and fuses the convolutional local

features and the N types of up-sampled regional context features at different scales,

uses the features after fusion to predict the pixel classification label, and then uses the

Softmax probability conversion layer to convert the pixel classification label

prediction score into pixel classification label prediction probability distribution;

S2. Structured prediction of pixel label based on a fully connected CRF: the fully

connected CRF is used to postprocess pixel classification label which is output by a

deep full convolutional semantic segmentation network to remove misclassified

isolated pixels or regions, optimize pixel labels near the boundary of a complex object,

and allow output segmentation map to have good spatial consistency and precise

boundary, which specifically comprises:

S21. The mutual relations between the variable probabilities of any two pixel

labels are modeled by using the fully connected CRF;

S22. The following Gibbs energy function is used for the fully connected CRF

model:

E(x)= Vf,(x,) + Vf ,(x,, xj) i,i,i<j

Where, x is a pixel classification label variable, x and x. are the labels

corresponding to the ith pixel and the jth pixel respectively, V, is a univariate

potential function and /, is a pairwise potential function;

S23. The pixel classification label probability is modeled by using an iterative

reasoning algorithm of mean approximate probability of field, and an optimized pixel classification label prediction probability distribution map is outputted;

S3. Model training and parameter learning:

S31. The parameters of the segmentation network model is initialized by using a

Xavier method;

S32. After expanding the training data, the training data will be divided into training set,

validation set and test set according to the ratio of 5:1:1, and the six-fold cross-validation method

is used to train the segmentation network model;

S33. The RGB image to be segmented is inputted as three channels into a deep

full convolutional semantic segmentation network, and a pixel classification label

prediction probability distribution is generated. The prediction loss is calculated by

using the label prediction probability and the segmentation label, and specifically a

classified cross entropy loss function is used as the objective function.. The definition is

as follows:

1B S C 1 ~~' log(Yl) Sx Bk-i i-I j

Where, Y is the probability vector of segmentation label, Y is the label

prediction probability vector, C is the number of pixel classification, S is the

number of image pixels, log(.) is the natural logarithm and B is the batch size;

S34. The stochastic gradient descent algorithm is used to optimize the objective

function, and the reverse propagation algorithm of error is used to update the

parameters of the deep full convolutional semantic segmentation network model. The

specific optimization process is as follows:

gt = V,_1L(0,,)

v, = P * v,_, - qtg,

O, = 0,_1 + v,

Where, the subscript t is the number of iterations, 0 is a network model

parameter, L(O,_,) is a loss function when using 0,1 as a network parameter,

g, v, and ' are a gradient, a momentum and a momentum coefficient respectively,

and 'l is a learning rate;

S4. Semantic segmentation of image:

The RGB image to be segmented is inputted as three channels into a deep full

convolutional semantic segmentation network to carry out forward calculation;

S42.The convolutional local feature image of the image is outted through

convolution, maximum pooling and dilated convolution operations by the feature

extraction module;

S43. The convolutional local feature image is inputted into the pyramid pooling

module to generate regional context feature images at different scales;

S44. The convolutional local feature image to the regional context feature images

at different scales are connected, and the pixel label prediction module is inputted;

S45. The pixel label prediction module first performs convolution fusion on the

convolution local features and the regional context features at different scales, and

then predicts the pixel classification by using the fusion features. After that, a pixel

classification label prediction probability distribution map is output;

S46. The label prediction probability distribution map of pixel classification

output by the deep full convolutional semantic segmentation network is inputted into

a fully connected CRF, the pixel classification label prediction probability distribution

of according to the intensity between pixels and the similarity of position features is

optimized, and the structured pixel classification label prediction probability

distribution map is outputted;

S47. The subscript of the maximum probability component in each pixel

probability distribution vector is taken as a pixel classification label to obtain a final semantic segmentation image.

2. An image semantic segmentation method based on deep full convolutional

network and CRF according to Claim 1, which is characterized in that in Step S12

each convolution layer group consists of two convolutional layers, the size of each

convolution kernel of each convolution layer is 3x3, and the step size is 1. The

number of convolution kernels of the first to the fifth convolution layer groups is 64,

128, 256, 512 and 1,024 in sequence; the pool kernel size of each maximum pooling

layer is 2x2, and the step length is 2; and the convolution kernel size of each dilated

convolution layer is 3x3, and the step length is 1. The dilated factors of the first

dilated convolution layer and the second dilated convolution layer are 2 and 4

respectively; the number of pyramid pooling stages in the pyramid pooling module is

4 and the average size of 4 levels of average pooling containers are 1x1, 2x2, 4x4 and

8x8, respectively. For each level, convolution kernels with a size of 1 X 1, a step

length of 1, and a number of 256 are used for convolution. The size of the convolution

kernel of each feature projection layer is 1 X 1 and the step length is 1; the number of

convolution kernels of the recited first feature projection layer and the second feature

projection layer are 1,024 and 512 respectively; and the size of the convolution kernel

of the recited class prediction layer is 1 X 1, the step length is 1, the number of the

convolution kernels is 32.

3. An image semantic segmentation method based on deep full convolutional

network and CRF according to Claim 1, which is characterized in that in Step S12 the

dilated convolution of the first dilated convolution layer and the second dilated

convolution layer is calculated by the following formula:

Z(i,j)= YX(i+ r x m, j+ r x n) W(m,n)

Where, (i, j) is the i* line and1 th j column, W is the convolution kernel, X is

the input channel, Z is the convolution output image, (M,N) is the convolution kernel dimension order, @ is the convolution operation, and r is the dilated factor.

4. An image semantic segmentation method based on deep full convolutional

network and CRF according to Claim 3, which is characterized in that the output

feature image Z, corresponding to any dilated convolution kernel in the dilated

convolution is calculated by using the following formula:

K M N Z,(i, j)= X,(i+ r x m, + r x n)0 Wk(m,n) k=1 m=1 n=1

Where, 1 is the dilated convolution kernel number, K is the number of input

channels, and (M, N) is the number of dimensions convolution kernel.

5. An image semantic segmentation method based on deep full convolutional

network and CRF according to Claim 1, which is characterized in that in Step S12

also comprises the batch standardization operation of output feature image generated

by the convolution layer, dilated convolution layer and feature projection layer.

6. An image semantic segmentation method based on deep full convolutional

network and CRF according to Claim 5, which is characterized in that the recited deep

full convolutional semantic segmentation network adopts an LReLU function as an

activation function, which is used for carrying out non-linear transfonnation on each

value in the feature image after batch standardization, and the recited LReLU function

is defined as follows:

f(z)= max(O, z)+ a min(O, z)

Where, f(z) is a nonlinear excitation unit function, max (.) function is to find a

maximum value, min(.) function is to find a minimum value, z is an input value,

and a is a Leaky parameter.

7. An image semantic segmentation method based on deep full convolutional

network and CRF according to Claim 1, which is characterized in that in Step S12 the

Softmax function is defined as follows:

=softmax(O,)= cexp(O) Iexp(o') c=1

Where, Oi is a prediction score of a pixel on the i th classification, Y is the

prediction probability of a pixel on the i th classification, C is the number of pixel

classifications, and exp(.) is an exponential function with natural constant e as base.

8. An image semantic segmentation method based on deep full convolutional

network and CRF according to Claim 1, which is characterized in that in Step S22 the

univariate potential function V/ is defined as follows:

VxO)= -log P(x)

Wherein P(xi) is the label prediction probability of the classification label of

the first pixel output by the deep full convolutional semantic segmentation network,

and log(.) is the natural logarithm;

The pairwise potential function is defined as follows:

2 2 2

W x i~;Xm )M xp(-)+w 2 exp(- )

2 2

Where, exp(- - ' ) is an external Gauss kernel, 2a 2aO 2

exp(- ) is a smooth Gauss kernel, i(xi,x1 ) is a label compatibility 2ar7

function,p(xi,x 1 )=[xi x1 ], p and P are the corresponding positionofthe ith

and j th pixel,I, and IJ are the corresponding intensity of the i th pixel and the

j th pixel, a, a# nd a are Gauss kernel parameters, and o1 and m2 are

relative intensity of the two Gauss kernels.

9. An image semantic segmentation method based on deep full convolutional

network and CRF according to Claim 1, which is characterized in that in Step S33 the

final objection obtained is as follows by adding Li and L2 regularization term into

loss function:

1 B SC Q + Q L (0)= xB , log(,, )+ E + Y, Sx Bk1 1j I Q 1 Q i

Where, A, and 2 are the regularization factors of Li and L2 , O is the

parameter of the segmented network, and Q is the number of parameters of O,.

10. An image semantic segmentation method based on deep full convolutional

network and CRF according to Claim 1, which is characterized in that in Step S34 the

linear attenuation of learning rate is introduced and the learning rate is attenuated

according to the following rule:

t t 77, =(1--)x 7h +-x 7,

Where, 7, is the learning rate used by the th iteration, 70 is the starting

learning rate, 7, is the final learning rate and T is the total number of iterations.

-1/3-

Deep full convolutional semantic segmentation Fully connected CRF

Semantic segmentation image Regional Original image

feature at different Iteration Feature Space pyramid Pixel scales reasoning of extraction pooling module Feature concatenation classification approximate 2020103901

module prediction probability of module mean field

Convolution local feature

Figure. 1

-2/3- Convolution Convolution layer 1_1 layer 1_2 Convolution layer 2_1 Maximum Convolution pooling layer layer 2_2 1 Maximum Convolution Maximum pooling layer Convolution layer 3_2 layer 3_1 pooling layer Convolution Convolution Dilated Convolution Convolution Dilated

Local feature 2 3 layer 4_1 layer 4_2 convolution 1 layer 5_1 layer 5_2 convolution 2 Original image 2020103901

Figure. 2

-3/3-