CN110458841A - A method to improve the running rate of image segmentation - Google Patents

Abstract

A method for improving the running rate of image segmentation, comprising: step 1, designing a multi-scale hollow convolution kernel; step 2, designing a channel convolution network; step 3, designing a full convolution connection and a deconvolution network; The network of convolution and full convolution operation can be applied to any image size and size, and can perform semantic analysis on each pixel of the image, achieve the purpose of fast image segmentation, and can quickly and accurately locate image features.

Description

A method to improve the running rate of image segmentation

technical field

The present invention relates to a method for changing the image segmentation rate.

technical background

In recent years, with the rapid development of computer science and technology, image processing and image target detection based on computer technology have also achieved unprecedented rapid development. Among them, deep learning extracts key target features by learning massive digital image features. It has surpassed human beings, bringing one surprise after another to the industry. With the re-emergence of neural network, the video image method based on convolutional neural network has become the mainstream technology of image segmentation and recognition. It adopts template matching, edge feature extraction, gradient histogram and other means to achieve accurate image recognition. Although image feature detection based on neural network can perform effective feature recognition for targets in complex scenes, and its effect is much better than traditional methods, it also has shortcomings: (1) It is weak against noise; (2) ) The over-fitting problem is solved by using the Dropout method, and the convolutional neural network model and parameters are improved, but the accuracy is slightly decreased; (3) The variable convolution and separable convolution structures are introduced to improve the generalization of the model. (4) A newer image segmentation method, namely End-to-End, directly predicts the image pixel classification information, and achieves Pixel localization of the target object, but the model has problems such as large amount of parameters, slow efficiency, and rough segmentation. In short, traditional detection methods and video image methods have problems such as cumbersome operation, low recognition accuracy, slow recognition efficiency and rough segmentation.

SUMMARY OF THE INVENTION

In order to overcome the above-mentioned shortcomings of the prior art, the present invention provides a fully convolutional method for improving the running rate of image segmentation aiming at the sample problem. The invention adopts a deep learning framework, and optimizes and improves the convolutional neural network; adopts the method of channel convolution to reduce the parameter quantity of the model; adopts multi-scale hole convolution to increase the characteristics of the image, and solves the problem of small receptive field of the traditional network .

To achieve the above object, the present invention adopts the following technical solutions:

A method for improving the running rate of image segmentation, comprising the following steps:

Step 1, design a multi-scale hole convolution kernel;

In order to solve the problem of increasing the receptive field by using traditional convolution and maximum pooling methods, the present invention adopts a hole convolution kernel, increases the sampling rate rate based on the traditional convolution kernel, and changes the original convolution kernel into "fluffy".

In this way, while maintaining the original calculation amount, the receptive field is increased, so that the information of image segmentation is accurate enough. The calculation formula of the receptive field size based on the hole convolution kernel is:

In the formula: F is the size of the receptive field of the current layer; rate is the sampling rate of the atrous convolution kernel, that is, the number of spacings, the rate of the traditional convolution kernel can be regarded as 1, and the sampling rate of the atrous convolution is regarded as 2. The traditional convolution receptive field calculation formula is

In the formula: F _i-1 is the size of the receptive field of the previous layer; ki is the size of the convolution kernel or pooling kernel of the _i -th layer; n is the total number of convolution layers; s _i is the convolution kernel of the i-th layer. Convolution stride Stride.

Drawing on the idea of multi-scale image changes to design multi-scale hole convolution, the sampling rate and convolution kernel size are diversified, so that it can adapt to the feature extraction process of different size targets. The multi-scale atrous convolution is calculated as

In the formula: y[i] is the convolution summation result corresponding to the i-th step position; K is the convolution kernel; k is the parameter coordinate position in the convolution kernel, k∈K; w[k] is the convolution kernel Weight; rate is the sampling rate, which can take the corresponding value of 1, 2, and 3.

Step 2, design a channel convolutional network;

Since the traditional convolution method is a dimensional increase operation, it can be considered to use the channel convolution method at the beginning to achieve the function of feature convolution dimension reduction. First, the traditional convolution is changed to a two-layer convolution, similar to the group operation in ResNet. This new structure shortens the calculation time by about 1/8 of the original without affecting the accuracy, and reduces the amount of parameters by about The original 1/9, and can be well applied to the mobile terminal to achieve real-time detection of targets, and the model compression effect is obvious.

For traditional convolution, it is assumed that the number of input feature channels is M; the width or height of the convolution kernel is D _k or D _k , respectively; the number of convolution kernels is N. Then, each time the convolution slides a certain position, there are N parameters of _M · _Dk ·Dk, and the sliding step size is set to s. Then the formula for calculating the size of the image after sliding is:

In the formula: h', w' are the height and width after convolution; pad is the border of width and height padding. Therefore, after the convolution of h'·w', a certain point of the size corresponds to N parameters of _M ·Dk· _Dk , and the total parameter size can be obtained as N· _M ·Dk· _Dk ·h'·w '(6)

With the improved channel convolution method, the convolution step is divided into two steps:

1) Convolve the M channels respectively by using the convolution of D _k · D _k · M. The same step size s is used for sliding, and the size after convolution is h', w', then the amount of parameters generated in this step is D _k · D _k · M · h' · w' (7)

2) Set a convolution kernel of 1·1·N for feature extraction. At this time, the feature map obtained from the above results is extracted again with a step size of 1. There are original M channel features, each of which uses N convolution kernels for feature extraction, and the total calculated parameter size is M· N·h'·w'·1·1 (8)

Combining the convolution structure of these two steps, the final parameter size of channel convolution is D _k · D _k · M · h' · w'+M · N · h' · w' (9)

As mentioned above, the parameter quantity of the traditional convolution kernel and the improved channel convolution parameter quantity are as follows:

From the analysis of formula (10), it can be seen that if the size of the convolution kernel is 3×3, the channel convolution operation can reduce the amount of parameters to 1/9 of the original.

Step 3: Design a fully convolutional connection and a deconvolutional network;

The last layer of the traditional network structure adopts a fixed size, so that the input image needs to be converted into a fixed size in advance, which is not conducive to the acquisition of the coordinates of the length of the logistics vehicle; and the traditional fully connected layer network has a certain loss of digit space coordinates, resulting in The spatial information of the image is distorted, which cannot effectively locate the target. In order to solve the problem of information loss, the present invention adopts a fully convolutional connection method to precisely locate the position coordinates of the features in the picture.

The full connection of the traditional network converts the convolutional network [b,c,h,w] of the previous part to [b,c·h·w], that is, [b,4096], and then to [b,cls], where b represents the batch size, and cls represents the number of categories. The use of a fully convolutional network is relative to a 1×1 convolutional network followed by a fully connected layer. Hence, it is called a fully convolutional network. The calculation method of full convolution is

In the formula: 1≤n≤N; y _n [i][j] is the value after convolution at the (i, j)th position of the nth convolution kernel; s _i is the horizontal convolution step size; s _j is the vertical convolution step size; k _n is the nth convolution kernel; D _k is the width and height of the convolution kernel, and the size of the convolution kernel corresponds to D _k · D _k in step 2; δ _i , δ _j are the The position in the convolution kernel, this layer has a total of N different types of convolution kernels, 0≤δ _i ,δ _j ≤D _k , and the sliding convolution operation of the convolution kernel can be converted into two matrix multiplication operations. The result of convolution between the pixels of the corresponding image can be expressed as

Among them: the dimension of the matrix on the left is [N, _M · _Dk · _Dk ]; the dimension of the matrix on the right is [M·Dk· _Dk ,w′·h′]; the dimension after convolution is [N,w '·h']. In the matrix on the right, I is img, and its subscripts are image width and image height in turn, that is, I _wh .

Finally, through the deconvolution operation, [N,w′·h′] is converted into the image size at the time of input, so that the specific semantic information represented by each pixel can be accurately identified, and the loss of spatial information can be avoided. The specific operation of deconvolution is equivalent to the inverse operation of convolution, namely

In the formula: k ₁ ,...,k _N corresponding weights of convolution kernels are changed from the original change to The weight is the weight size adjusted by training, and has the weight of the image semantic information feature.

Therefore, the network through deconvolution and full convolution operations can be applied to any image size and size, and can perform semantic analysis on each pixel of the image to achieve the purpose of fast image segmentation, and can quickly and accurately perform image features. position.

The advantages of the present invention are:

The present invention adopts a full convolution method to improve the running rate of image segmentation for the problem of samples. The number of parameters of the model is reduced by channel convolution; a multi-scale hollow convolution kernel is set up, which reasonably and simply improves the receptive field of the model and enhances the generalization of the model. The algorithm can be widely used in the field of image positioning and recognition, such as vehicle identification in logistics parks.

Description of drawings

Figure 1 is a schematic diagram of the existing traditional convolution kernel convolution operation;

Fig. 2 is the convolution operation schematic diagram of the improved hollow convolution kernel of the present invention;

Figures 3a to 3c are multi-scale hole convolution kernels of the present invention, Figure 3a is a hole convolution kernel with a sampling rate of 1, Figure 3b is a hole convolution kernel with a sampling rate of 2, and Figure 3c is a hole convolution kernel with a sampling rate of 3 Atrous convolution kernel;

Figure 4 is the existing convolution method;

Fig. 5 is the channel convolution mode of the present invention;

Fig. 6 is the channel convolution structure of the present invention;

Fig. 7 is the fully convolutional network design structure of the present invention;

FIG. 8 is a schematic diagram of the calculation process of the full convolution matrix of the present invention.

Note: In Figure 6, DW is a channel convolution group, which represents a fixed combination of channel convolution kernels; BN is a batch normalization operation, which solves the problem that the data distribution of the middle layer changes during the training process; Conv is a convolution Layer operation; RelU is a modified linear unit, which is an activation function.

Note: In Figure 8: k ₁ , ..., k _N is the number of convolution kernels; is the position weight of the nth convolution kernel.

Detailed ways

In order to overcome the above-mentioned shortcomings of the prior art, the present invention provides a fully convolutional image segmentation method for the sample problem, adopts a deep learning framework, and optimizes and improves the convolutional neural network; adopts the channel convolution method to reduce the model The amount of parameters; multi-scale atrous convolution is used to increase the characteristics of the image and solve the problem of small receptive field of the traditional network.

To achieve the above object, the present invention adopts the following technical solutions:

A method for improving the running rate of image segmentation, comprising the following steps:

Step 1, design a multi-scale hole convolution kernel;

In order to solve the problem of increasing the receptive field by using traditional convolution and maximum pooling methods, the present invention adopts a hole convolution kernel, increases the sampling rate rate based on the traditional convolution kernel, and changes the original convolution kernel into "fluffy".

In this way, while maintaining the original calculation amount, the receptive field is increased, so that the information of image segmentation is accurate enough. The calculation formula of the receptive field size based on the hole convolution kernel is:

In the formula: F is the size of the receptive field of the current layer; rate is the sampling rate of the atrous convolution kernel, that is, the number of spacings, the rate of the traditional convolution kernel can be regarded as 1, and the sampling rate of the atrous convolution is regarded as 2. The traditional convolution receptive field calculation formula is

In the formula: F _i-1 is the size of the receptive field of the previous layer; ki is the size of the convolution kernel or pooling kernel of the _i -th layer; n is the total number of convolution layers; s _i is the convolution kernel of the i-th layer. Convolution stride Stride.

Drawing on the idea of multi-scale image changes to design multi-scale hole convolution, the sampling rate and convolution kernel size are diversified, so that it can adapt to the feature extraction process of different size targets. The multi-scale atrous convolution is calculated as

In the formula: y[i] is the convolution summation result corresponding to the i-th step position; K is the convolution kernel; k is the parameter coordinate position in the convolution kernel, k∈K; w[k] is the convolution kernel Weight; rate is the sampling rate, which can take the corresponding value of 1, 2, and 3.

Step 2, design a channel convolutional network;

Since the traditional convolution method is a dimensional increase operation, it can be considered to use the channel convolution method at the beginning to achieve the function of feature convolution dimension reduction. First, the traditional convolution is changed to a two-layer convolution, similar to the group operation in ResNet. This new structure shortens the calculation time by about 1/8 of the original without affecting the accuracy, and reduces the amount of parameters by about The original 1/9, and can be well applied to the mobile terminal to achieve real-time detection of targets, and the model compression effect is obvious.

For traditional convolution, it is assumed that the number of input feature channels is M; the width or height of the convolution kernel is D _k or D _k , respectively; the number of convolution kernels is N. Then, each time the convolution slides a certain position, there are N parameters of _M · _Dk ·Dk, and the sliding step size is set to s. Then the formula for calculating the size of the image after sliding is:

In the formula: h', w' are the height and width after convolution; pad is the border of width and height padding. Therefore, after the convolution of h'·w', a certain point of the size corresponds to N parameters of _M ·Dk· _Dk , and the total parameter size can be obtained as N· _M ·Dk· _Dk ·h'·w '(6)

With the improved channel convolution method, the convolution step is divided into two steps:

1) Convolve the M channels respectively by using the convolution of D _k · D _k · M. The same step size s is used for sliding, and the size after convolution is h', w', then the amount of parameters generated in this step is D _k · D _k · M · h' · w' (7)

2) Set a convolution kernel of 1·1·N for feature extraction. At this time, the feature map obtained from the above results is extracted again with a step size of 1. There are original M channel features, each of which uses N convolution kernels for feature extraction, and the total calculated parameter size is M· N·h'·w'·1·1 (8)

Combining the convolution structure of these two steps, the final parameter size of channel convolution is D _k · D _k · M · h' · w'+M · N · h' · w' (9)

As mentioned above, the parameter quantity of the traditional convolution kernel and the improved channel convolution parameter quantity are as follows:

From the analysis of formula (10), it can be seen that if the size of the convolution kernel is 3×3, the channel convolution operation can reduce the amount of parameters to 1/9 of the original.

Step 3: Design a fully convolutional connection and a deconvolutional network;

The last layer of the traditional network structure adopts a fixed size, so that the input image needs to be converted into a fixed size in advance, which is not conducive to the acquisition of the coordinates of the length of the logistics vehicle; and the traditional fully connected layer network has a certain loss of digit space coordinates, resulting in The spatial information of the image is distorted, which cannot effectively locate the target. In order to solve the problem of information loss, the present invention adopts a fully convolutional connection method to precisely locate the position coordinates of the features in the picture.

The full connection of the traditional network converts the convolutional network [b,c,h,w] of the previous part to [b,c·h·w], that is, [b,4096], and then to [b,cls], where b represents the batch size, and cls represents the number of categories. The use of a fully convolutional network is relative to a 1×1 convolutional network followed by a fully connected layer. Hence, it is called a fully convolutional network. The calculation method of full convolution is

In the formula: 1≤n≤N; y _n [i][j] is the value after convolution at the (i, j)th position of the nth convolution kernel; s _i is the horizontal convolution step size; s _j is the vertical convolution step size; k _n is the nth convolution kernel; D _k is the width and height of the convolution kernel, and the size of the convolution kernel corresponds to D _k · D _k in step 2; δ _i , δ _j are the The position in the convolution kernel, this layer has a total of N different types of convolution kernels, 0≤δ _i ,δ _j ≤D _k , and the sliding convolution operation of the convolution kernel can be converted into two matrix multiplication operations. The result of convolution between the pixels of the corresponding image can be expressed as

Among them: the dimension of the matrix on the left is [N, _M · _Dk · _Dk ]; the dimension of the matrix on the right is [M·Dk· _Dk ,w′·h′]; the dimension after convolution is [N,w '·h']. In the matrix on the right, I is img, and its subscripts are image width and image height in turn, that is, I _wh .

Finally, through the deconvolution operation, [N,w′·h′] is converted into the image size at the time of input, so that the specific semantic information represented by each pixel can be accurately identified, and the loss of spatial information can be avoided. The specific operation of deconvolution is equivalent to the inverse operation of convolution, namely

In the formula: k ₁ ,...,k _N corresponding weights of convolution kernels are changed from the original change to The weight is the weight size adjusted by training, and has the weight of the image semantic information feature.

Therefore, the network through deconvolution and full convolution operations can be applied to any image size and size, and can perform semantic analysis on each pixel of the image to achieve the purpose of fast image segmentation, and can quickly and accurately perform image features. position.

In order to verify the superiority of the invention, taking the logistics park vehicle as an example, the following network model is constructed and a control experiment is carried out:

First, construct the network: collect four types of logistics vehicles from the logistics park, including vans, traction trucks, dump trucks, and tank trucks, and divide them into 8,000 training sets, 2,000 for each category, and test them. Set of 4,000 sheets, 1,000 sheets per category. The parameter configuration of the network model structure built is shown in Table 1 below.

In Table 1: k is the size of the convolution kernel; s is the step size; p is the size of the padding; DW is the channel convolution group, which represents a fixed combination of channel convolution kernels; the use of residual summation is beneficial to large networks. Gradient transfer; activation of each layer and Batch Normalization (BN) are beneficial to speed up the training of the network; ReLU is a modified linear unit and an activation function.

Table 1 Design of parameters of network model structure

The computer used in this example is configured as a Gigabyte NVIDIA GTX1080Ti graphics card with 11 G of memory and 1 607 MHz.

Finally, the model test performance of this example network and the traditional network is compared, and the results are shown in Table 2.

Table 2 Performance comparison of lightweight segmentation models

The evaluation index MPA in Table 2 represents the mean pixel accuracy; MA represents the ratio of the foreground area to the label area (Mean accuracy); and MIOU represents the mean intersection over union ratio (Mean intersection over union), That is, the ratio of the correctly predicted area to the union of the predicted area and the label area; the unit M pic-1 represents the memory occupied by training a picture, and the unit of memory is mega (M); the unit ms iter-1 represents the need for each iteration. time in milliseconds (ms); after channel convolution, the occupied video memory is reduced by 51%, the training speed is increased by 78%, the test speed is increased by 79%, and the evaluation indicators of segmentation and positioning have been greatly improved. , among which MIOU has the largest increase.

Through this example, it is verified that the improved method can indeed improve the performance of model testing, that is, improve the running rate of image segmentation.

The advantages of this scheme are:

The present invention adopts a full convolution method to improve the running rate of image segmentation for the problem of samples. The number of parameters of the model is reduced by channel convolution; a multi-scale hollow convolution kernel is set up, which reasonably and simply improves the receptive field of the model and enhances the generalization of the model. The algorithm can be widely used in the field of image positioning and recognition, such as vehicle identification in logistics parks.

The content described in the embodiments of the present specification is only an enumeration of the realization forms of the inventive concept, and the protection scope of the present invention should not be regarded as limited to the specific forms stated in the embodiments, and the protection scope of the present invention also extends to those skilled in the art. Equivalent technical means that can be conceived by a person based on the inventive concept.

Claims (1)

1. a method for improving image segmentation running rate, comprising the steps: Step 1, design a multi-scale hole convolution kernel; In order to solve the problem of increasing the receptive field by using traditional convolution and maximum pooling methods, a hole convolution kernel is used, and the sampling rate rate is increased based on the traditional convolution kernel to make the original convolution kernel "fluffy". "; In this way, while maintaining the original calculation amount, the receptive field is increased, so that the information of image segmentation is accurate enough. The calculation formula of the receptive field size based on the hole convolution kernel is: In the formula: F is the size of the receptive field of the current layer; rate is the sampling rate of the hole convolution kernel, that is, the spacing number, the rate of the traditional convolution kernel can be regarded as 1, and the sampling rate rate of the hole convolution is regarded as 2; The calculation formula of the convolution receptive field is In the formula: F _i-1 is the size of the receptive field of the previous layer; ki is the size of the convolution kernel or pooling kernel of the _i -th layer; n is the total number of convolution layers; s _i is the convolution kernel of the i-th layer. Convolution stride Stride; Drawing on the idea of multi-scale image changes, we design multi-scale atrous convolution, and diversify the sampling rate and convolution kernel size, so as to adapt to the feature extraction process of objects of different sizes; the calculation of multi-scale atrous convolution is as follows: In the formula: y[i] is the convolution summation result corresponding to the i-th step position; K is the convolution kernel; k is the parameter coordinate position in the convolution kernel, k∈K; w[k] is the convolution kernel Weight; rate is the sampling rate, which can take the corresponding value of 1, 2, and 3; Step 2, design a channel convolutional network; Since the traditional convolution method is a dimensional increase operation, the channel convolution method is used at the beginning to achieve the function of feature convolution dimension reduction; first, the traditional convolution is changed to a two-layer convolution, similar to ResNet In the group operation, this new structure shortens the calculation time by about 1/8 of the original and reduces the amount of parameters by about 1/9 of the original without affecting the accuracy rate, and can be well applied to the mobile terminal. The real-time detection of the target, the model compression effect is obvious; For traditional convolution, it is assumed that the number of input feature channels is M; the width or height of the convolution kernel is D _k or D _k respectively; the number of convolution kernels is N; There are N parameters of _M · _Dk ·Dk, and the sliding step is set to s; then the calculation formula of the size of the image after sliding is: In the formula: h', w' are the height and width after convolution; pad is the border of width and height padding; therefore, a certain point of the size of h'·w' after convolution corresponds to N parameters of _M · _Dk ·Dk quantity, the total parameter size can be obtained as N· _M ·Dk· _Dk ·h'·w' (6) With the improved channel convolution method, the convolution step is divided into two steps: 1) Use the convolution of D _k · D _k · M to convolve the M channels respectively; use the same step size s to slide, and the size of the convolution is h', w', then the parameters generated by this step The amount is D _k · D _k · M · h' · w' (7) 2) Set the convolution kernel of 1·1·N to extract the dimension-raising feature; at this time, the feature map obtained from the above results is extracted again with a step size of 1. The original M channel features, each of which uses N convolution kernels are used for feature extraction, and the total calculated parameter size is M·N·h'·w'·1·1 (8) Combining the convolution structure of these two steps, the final parameter size of the channel convolution is obtained as D _k · D _k · M · h' · w' + M · N · h' · w' (9) As mentioned above, the parameter quantity of the traditional convolution kernel and the improved channel convolution parameter quantity are as follows: From the analysis of formula (10), it can be obtained that the channel convolution operation reduces the amount of parameters; Step 3: Design a fully convolutional connection and a deconvolutional network; The last layer of the traditional network structure adopts a fixed size, so that the input image needs to be converted into a fixed size in advance, which is not conducive to the acquisition of the coordinates of the length of the logistics vehicle; and the traditional fully connected layer network has a certain loss of digit space coordinates, resulting in The spatial information of the image is distorted, and the target cannot be accurately located; in order to solve the problem of information loss, the full convolution connection method is used to accurately locate the position coordinates of the features in the image; The full connection of the traditional network converts the convolutional network [b,c,h,w] of the previous part to [b,c·h·w], that is, [b,4096], and then to [b,cls], where b represents the batch size, and cls represents the number of categories; the use of a fully convolutional network is relative to a 1×1 convolutional network followed by a fully connected layer; therefore, it is called a fully convolutional network; a fully convolutional network The calculation method is In the formula: 1≤n≤N; y _n [i][j] is the value after convolution at the (i, j)th position of the nth convolution kernel; s _i is the horizontal convolution step size; s _j is the vertical convolution step size; k _n is the nth convolution kernel; D _k is the width and height of the convolution kernel, and the size of the convolution kernel corresponds to D _k · D _k in step 2; δ _i , δ _j are the The position in the convolution kernel, this layer has a total of N different types of convolution kernels, 0≤δ _i ,δ _j ≤D _k , and the sliding convolution operation of the convolution kernel can be converted into two matrix multiplication operations; The result of convolution between the pixels of the corresponding image can be expressed as Among them: the dimension of the matrix on the left is [N, _M · _Dk · _Dk ]; the dimension of the matrix on the right is [M·Dk· _Dk ,w′·h′]; the dimension after convolution is [N,w '·h']; in the matrix on the right, I is img, and its subscripts are image width and image height in turn, i.e. I _wh ; Finally, through the deconvolution operation, [N,w′·h′] is converted into the image size at the time of input, so that the specific semantic information represented by each pixel can be accurately identified, and the loss of spatial information is avoided; The specific operation is equivalent to the inverse operation of convolution, that is, In the formula: k ₁ ,...,k _N corresponding weights of convolution kernels are changed from the original change to The weight is the weight size adjusted by training, and has the weight of the image semantic information feature.