CN114565792B - Image classification method and device based on lightweight convolutional neural network - Google Patents

Abstract

The invention discloses an image classification method and device based on a lightweight convolutional neural network, and belongs to the field of image classification in computer deep learning. The invention comprises the following steps: s1: constructing a lightweight convolutional neural network model, wherein the lightweight neural network model comprises the following components connected in sequence: the system comprises a standard convolution layer, a plurality of sampling splicing units, a global pooling layer and a full connection layer, wherein the sampling splicing units comprise a plurality of components which are sequentially connected: the system comprises a downsampling layer, a plurality of universal layers and a splicing layer; s2: and inputting the pictures to be classified into the lightweight convolutional neural network model to obtain a classification result. According to the invention, the lightweight convolutional neural network model with low parameter quantity, low calculation quantity and high reasoning speed is constructed, and then the lightweight convolutional neural network model is utilized to classify images, so that compared with the prior lightweight convolutional neural network model, the classification speed of the model is greatly improved while the parameter quantity is greatly reduced under the similar classification accuracy.

Description

A method and device for image classification based on lightweight convolutional neural network

Technical Field

The present invention belongs to the field of deep learning image classification, and more specifically, relates to an image classification method and device based on a lightweight convolutional neural network.

Background technique

In recent years, Convolutional Neural Networks (CNN) have been widely used in computer vision, such as image classification. In order to improve the accuracy of classification, the depth and width of CNN models have increased rapidly, resulting in a rapid increase in the number of model parameters and the amount of computation, which has hindered the application of CNN on devices with weaker computing power.

In order to apply it on mobile or embedded devices, lightweighting of the model is an important approach. At present, the method of building a model based on depthwise separable convolution (DSCs) consisting of a layer of deep convolution and a layer of point-by-point convolution and using the Neural Architecture Search (NAS) algorithm to search for the best architecture has achieved remarkable success, such as the MobileNets series and the EfficientNets series. These models all use depthwise separable convolution instead of standard convolution. Compared with standard convolution, the number of parameters and the amount of calculation of depthwise separable convolution are reduced by several times; after determining the backbone of the network architecture, the NAS algorithm is used to search for the optimal model. For example, the width of each layer, etc., ensures the classification accuracy of the model while reducing the number of model parameters and the amount of calculation.

This type of method greatly reduces the number of parameters and computation, but the inference speed on the GPU has not improved and is even lower than that of classic networks such as ResNet. The reason is that the depthwise separable convolution cannot make good use of GPU resources, and these networks use more network layers than classic networks with the same accuracy, including more nonlinear activation layers and batch normalization layers, which have an impact on the inference speed of the model.

Summary of the invention

In view of the above defects or improvement needs of the prior art, the present invention provides an image classification method and device based on a lightweight convolutional neural network, the purpose of which is to design the lightweight neural network model to include: a standard convolutional layer, a plurality of sampling splicing units, a global pooling layer and a fully connected layer connected in sequence, and the sampling splicing unit includes: a downsampling layer, a plurality of general layers and a splicing layer connected in sequence; the image to be classified is input into the lightweight convolutional neural network model to obtain a classification result; thereby reducing the number of parameters of the convolutional neural network model while improving the inference speed of the model.

To achieve the above object, according to one aspect of the present invention, a method for designing a lightweight convolutional neural network architecture is provided, comprising:

S1: construct a lightweight convolutional neural network model, wherein the lightweight neural network model includes: a standard convolution layer, a plurality of sampling splicing units, a global pooling layer, and a fully connected layer connected in sequence, and the sampling splicing unit includes: a downsampling layer, a plurality of general layers, and a splicing layer connected in sequence;

S2: Inputting the image to be classified into the lightweight convolutional neural network model to obtain a classification result, including:

S21: using the standard convolutional layer to expand the channels of the image to be classified to a specified number of channels, thereby obtaining an original feature map;

S22: using the downsampling layer in the first sampling splicing unit to downsample the original feature map to obtain two groups of first feature maps, then using multiple general layers therein to respectively extract features from the two groups of the first feature maps to obtain respective corresponding second feature maps, and then using the splicing layer therein to splice the two groups of the second feature maps to obtain a first target feature map; inputting the first target feature map to the adjacent second sampling splicing unit so that it downsamples, extracts features from and splices the first target feature map to obtain a second target feature map; then inputting the second target feature map to the adjacent third sampling splicing unit, and so on, until the last sampling splicing unit outputs the final target feature map;

S23: The final target feature map output by the last sampling splicing unit is input into the global pooling layer to reduce the dimension and then input into the fully connected layer, so that the fully connected layer outputs the classification result corresponding to the classified image.

In one embodiment, the downsampling layer is used to include: a Gaussian downsampling layer, a point-by-point convolution layer, a nonlinear activation layer and a batch normalization layer connected in sequence; the downsampling layer outputs two sets of output feature maps;

The input feature map is passed through the Gaussian downsampling layer to obtain a set of output feature maps;

The input feature map is sequentially input into the Gaussian downsampling layer, the point-by-point convolution layer, the nonlinear activation layer and the batch normalization layer in the downsampling layer to obtain another set of output feature maps.

In one of the embodiments, the Gaussian downsampling layer in the downsampling layer performs a convolution operation on the feature map output by the previous layer, and the resolution of the output feature map will be half of that of the input.

In one of the embodiments, the point-by-point convolution layer in the downsampling layer expands or contracts the input feature map according to the number of input and output feature channels.

In one of the embodiments, ReLU is used as an activation function in the nonlinear activation layer in the downsampling layer.

In one embodiment, the general layer includes: a depth convolution layer, a concatenation layer, a point-by-point convolution layer, a nonlinear activation layer, and a batch normalization layer connected in sequence;

Each of the universal layers is inputted with a group of feature maps and a group of feature maps; the group of feature maps b are inputted into the depth convolution layer for convolution to obtain a group of feature maps c; the group of feature maps a and the group of feature maps c are inputted into the concatenation layer, the point-by-point convolution layer, the nonlinear activation layer and the batch normalization layer in the universal layer to obtain a new group of feature maps b;

The b group feature maps serve as new a group feature maps and the new b group feature maps serve as inputs of the next adjacent general layer.

According to another aspect of the present invention, there is provided an image classification device based on a lightweight convolutional neural network, comprising:

A construction module is used to construct a lightweight convolutional neural network model, wherein the lightweight neural network model includes: a standard convolution layer, a plurality of sampling splicing units, a global pooling layer, and a fully connected layer connected in sequence, and the sampling splicing unit includes: a downsampling layer, a plurality of general layers, and a splicing layer connected in sequence;

The classification module is used to input the image to be classified into the lightweight convolutional neural network model to obtain the classification result, which specifically includes:

Expanding the channels of the image to be classified to a specified number of channels using the standard convolutional layer, thereby obtaining an original feature map;

The original feature map is downsampled by using the downsampling layer in the first sampling splicing unit to obtain two groups of first feature maps, and then the two groups of the first feature maps are respectively extracted by using the multiple general layers therein to obtain the corresponding second feature maps, and then the two groups of the second feature maps are spliced by using the splicing layer therein to obtain the first target feature map; the first target feature map is input into the adjacent second sampling splicing unit so that it downsamples, extracts features and splices the first target feature map to obtain the second target feature map; the second target feature map is then input into the adjacent third sampling splicing unit, and so on, until the last sampling splicing unit outputs the final target feature map;

The final target feature map output by the last sampling splicing unit is input into the global pooling layer, and the low dimension is then input into the fully connected layer, so that the fully connected layer outputs the classification result corresponding to the classified image.

In general, the above technical solution conceived by the present invention has the following beneficial effects compared with the prior art:

The present invention modifies the network unit structure to construct a lightweight convolutional neural network model with low parameter count, low computational complexity and fast inference speed, and then uses the lightweight convolutional neural network model to classify images. Since the model of the present invention has low parameter count and computational complexity and has a faster inference speed on a GPU, using the model for image classification can improve the classification efficiency of images.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a schematic diagram of a portion of convolution kernels of a downsampling layer of MobileNetV2 visualized in one embodiment of the present invention, where the convolution kernel size is 1××3×3;

FIG2 is a schematic diagram of a portion of convolution kernels of a non-downsampling layer of EfficientNet-B0 visualized in one embodiment of the present invention, where the convolution kernel size is 1×5×5;

FIG3 is a partial similar feature map of adjacent layers 5 and 7 of RegNetX-400MF visualized in one embodiment of the present invention;

FIG4 is a schematic diagram of a downsampling layer constructed in one embodiment of the present invention;

FIG5 is a schematic diagram of a universal layer constructed in one embodiment of the present invention;

FIG6 is a flowchart of an image classification method based on a lightweight convolutional neural network in one embodiment of the present invention.

Detailed ways

In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

The present invention proposes a design basis for a lightweight convolutional neural network architecture by visualizing feature maps and convolution kernels. It should be noted that the visualized network can be any depth-separable convolutional neural network with trained parameters; the feature maps and convolution kernels are normalized during the visualization process, and if the center value of the convolution kernel is a negative number, the entire convolution kernel is multiplied by -1.

According to 1, as shown in Figure 1, most of the convolution kernels in the downsampling layer (deep convolution layer with a stride of 2) are similar to Gaussian blur convolution kernels, and the blur operation before downsampling conforms to the sampling theorem. Therefore, in the network we constructed, the convolution kernel in the downsampling layer will be replaced by Gaussian blur kernel. It should be noted that Figure 1 is composed of multiple 3x3 convolution kernels (each smallest square represents a pixel, 9 pixels), and the darker the pixel value, the larger it is. Many of these 3x3 convolution kernels are similar to Gaussian convolution kernels, that is, two Gaussian kernels with different variances. Therefore, you can try to use Gaussian convolution kernel instead.

According to 2, as shown in Figure 2, a large part of the convolution kernels in other deep convolution layers except the downsampling layer are similar to the identity kernel (that is, the convolution kernel with only the central value). Therefore, in the network we constructed, the identity kernel will be used to replace some deep convolution kernels, which is equivalent to directly removing the convolution kernels of some deep convolution layers. Note that in order to improve accuracy and computational efficiency, nonlinear activation layers and batch normalization layers are no longer used after the deep convolution layer. It should be noted that Figure 2 is also composed of multiple 3x3 convolution kernels. Many of these convolution kernels are similar to the 3x3 identity kernel (Identity Kernel, the middle value is 1, and the other values are 0), and the Identity Kernl can be directly removed to reduce the amount of calculation.

According to 3, as shown in Figure 3, a large part of the feature maps output by adjacent layers are divided into similar repeated feature maps. Therefore, we will reuse some feature maps in similar layers through identity mapping, and make the reused feature maps not participate in the next deep convolution operation. This ensures that the network width remains unchanged while reducing the number of parameters and calculations by half.

The present invention provides an image classification method based on a lightweight convolutional neural network, comprising:

S1: construct a lightweight convolutional neural network model, wherein the lightweight neural network model includes: a standard convolution layer, a plurality of sampling splicing units, a global pooling layer, and a fully connected layer connected in sequence, and the sampling splicing unit includes: a downsampling layer, a plurality of general layers, and a splicing layer connected in sequence;

S2: Inputting the image to be classified into the lightweight convolutional neural network model to obtain a classification result, including:

S21: using the standard convolutional layer to expand the channels of the image to be classified to a specified number of channels, thereby obtaining an original feature map;

S22: using the downsampling layer in the first sampling splicing unit to downsample the original feature map to obtain two groups of first feature maps, then using multiple general layers therein to respectively extract features from the two groups of the first feature maps to obtain respective corresponding second feature maps, and then using the splicing layer therein to splice the two groups of the second feature maps to obtain a first target feature map; inputting the first target feature map to the adjacent second sampling splicing unit so that it downsamples, extracts features from and splices the first target feature map to obtain a second target feature map; then inputting the second target feature map to the adjacent third sampling splicing unit, and so on, until the last sampling splicing unit outputs the final target feature map;

S23: The final target feature map output by the last sampling splicing unit is input into the global pooling layer to reduce the dimension and then input into the fully connected layer, so that the fully connected layer outputs the classification result corresponding to the classified image.

In one embodiment, the downsampling layer is used to include: a Gaussian downsampling layer, a point-by-point convolution layer, a nonlinear activation layer and a batch normalization layer connected in sequence; the downsampling layer outputs two sets of output feature maps;

The input feature map is passed through the Gaussian downsampling layer to obtain a set of output feature maps;

The input feature map is sequentially input into the Gaussian downsampling layer, the point-by-point convolution layer, the nonlinear activation layer and the batch normalization layer in the downsampling layer to obtain another set of output feature maps.

As shown in Figure 4, the downsampling layer provided by the present application consists of 4 layers: a Gaussian downsampling layer (a deep convolution layer with a step size of 2 after being replaced by a Gaussian convolution kernel), a point-by-point convolution layer, a nonlinear activation layer, and a batch normalization layer. The Gaussian downsampling layer performs a convolution operation on the feature map output by the previous layer, and the resolution of the output feature map will be half of that of the input; the point-by-point convolution layer expands or contracts the input feature map according to the number of feature channels of the input and output; the nonlinear activation layer and the batch normalization layer perform nonlinear activation and normalization operations on the output features of the previous layer, respectively; finally, the output feature map of the Gaussian downsampling layer and the output feature map of the batch normalization layer are used together as the output of this unit. In particular, ReLU is used as the activation function in the nonlinear activation layer.

In one of the embodiments, the Gaussian downsampling layer in the downsampling layer performs a convolution operation on the feature map output by the previous layer, and the resolution of the output feature map will be half of that of the input.

In one of the embodiments, the point-by-point convolution layer in the downsampling layer expands or contracts the channels of the input features according to the number of input and output feature channels.

In one of the embodiments, ReLU is used as an activation function in the nonlinear activation layer in the downsampling layer.

In one embodiment, the general layer includes: a depth convolution layer, a concatenation layer, a point-by-point convolution layer, a nonlinear activation layer, and a batch normalization layer connected in sequence;

Each of the universal layers is inputted with a group of feature maps and a group of feature maps; the group of feature maps b are inputted into the depth convolution layer for convolution to obtain a group of feature maps c; the group of feature maps a and the group of feature maps c are inputted into the concatenation layer, the point-by-point convolution layer, the nonlinear activation layer and the batch normalization layer in the universal layer to obtain a new group of feature maps b;

The new b-group feature map is used as the a-group feature map of the new round, and the b-group feature map input in the new round is used as the input of the next adjacent general layer.

As shown in Figure 5, the general layer consists of 5 layers: depth convolution layer, concatenation layer, point-by-point convolution layer, nonlinear activation layer, and batch normalization layer. According to the basis 2 in step 1, we removed half of the convolution kernels in the depth convolution layer, that is, the depth convolution layer only performs convolution operation on the second set of input feature maps; the concatenation layer concatenates the first set of feature maps input to the unit and the feature maps output by the depth convolution layer into one set as output; according to the basis 3 in step 1, in order to reuse feature maps and reduce the number of parameters, the point-by-point convolution layer processes the output feature maps of the concatenation layer and reduces the number of channels by half, and then processes them through the nonlinear activation layer and batch normalization layer; finally, the unit uses the second set of input feature maps as the first set of output feature maps, and uses the output of the point-by-point convolution layer as the second set of output feature maps of the unit.

It should be noted that the present invention uses a downsampling layer (DownBlock) and a general layer (HalfConvBlock) to construct a lightweight neural network, as shown in FIG6 :

1. Use standard convolution to expand the input image channels to the specified number of channels;

2. Use DownBlock to downsample and expand the number of channels, and output two sets of feature maps;

3. Repeatedly use HalfConvBlock to extract features, with both input and output being 2 sets of feature maps;

4. Combine two sets of feature maps into one set;

4. Repeat steps 2, 3 and 4 3 more times;

5. Use the fully connected layer to output the final classification result.

It should be noted that the final model can set the number of layers of different layers according to specific requirements (such as the number of parameters or the amount of calculation). For example, the specific details of building a network with a parameter amount of 1.5M are shown in Table 1 (the concatenation layer after each general layer is omitted in the table).

Table 1

When the number of parameters is limited to 1.5M or 2.5M and the accuracy of other models is similar, the model of the present invention is significantly ahead in both the number of parameters and the inference speed, as shown in Table 2 (the accuracy of the model of the present invention on ImageNet when using 1.5M parameters. The GPU inference speed is measured on RTX 6000) and Table 3. In addition, the embodiments constructed according to the number of parameters of 1.5M and 2.5M are only used to illustrate the technical solution of the present invention. Those skilled in the art should understand that only modifying or equivalently replacing the technical solution of the present invention, especially only modifying the number of network layers and the number of channels, does not deviate from the purpose and scope of the technical solution.

Table 2

table 3

According to another aspect of the present invention, there is provided an image classification device based on a lightweight convolutional neural network, comprising:

A construction module is used to construct a lightweight convolutional neural network model, wherein the lightweight neural network model includes: a standard convolution layer, a plurality of sampling splicing units, and a fully connected layer connected in sequence, and the sampling splicing unit includes: a downsampling layer, a plurality of general layers, and a splicing layer connected in sequence;

The classification module is used to input the image to be classified into the lightweight convolutional neural network model to obtain the classification result, which specifically includes:

Expanding the channels of the image to be classified to a specified number of channels using the standard convolutional layer, thereby obtaining an original feature map;

The original feature map is downsampled by using the downsampling layer in the first sampling splicing unit to obtain two groups of first feature maps, and then the two groups of the first feature maps are respectively extracted by using the multiple general layers therein to obtain the corresponding second feature maps, and then the two groups of the second feature maps are spliced by using the splicing layer therein to obtain the first target feature map; the first target feature map is input into the adjacent second sampling splicing unit so that it downsamples, extracts features and splices the first target feature map to obtain the second target feature map; the second target feature map is then input into the adjacent third sampling splicing unit, and so on, until the last sampling splicing unit outputs the final target feature map;

The final target feature map output by the last sampling splicing unit is input into the global pooling layer to reduce the dimension and then input into the fully connected layer, so that the fully connected layer outputs the classification result corresponding to the classified image.

It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims (6)

1. An image classification method based on a lightweight convolutional neural network is characterized by comprising the following steps:

s1: constructing a lightweight convolutional neural network model, wherein the lightweight neural network model comprises the following components connected in sequence: the system comprises a standard convolution layer, a plurality of sampling splicing units and a full connection layer, wherein the sampling splicing units comprise sequentially connected: the system comprises a downsampling layer, a plurality of universal layers and a splicing layer;

s2: inputting the picture to be classified into the lightweight convolutional neural network model to obtain a classification result, which comprises the following steps:

S21: expanding the channels of the pictures to be classified into the number of specified channels by using the standard convolution layer, so as to obtain an original feature map;

S22: downsampling the original feature images by using a downsampling layer in a first sampling and stitching unit to obtain two groups of first feature images, then respectively extracting features of the two groups of first feature images by using a plurality of universal layers in the downsampling layer to obtain respective corresponding second feature images, and stitching the two groups of second feature images by using a stitching layer in the downsampling layer to obtain a first target feature image; inputting the first target feature map into a second adjacent sampling and splicing unit so as to enable the first target feature map to be subjected to downsampling, feature extraction and splicing to obtain a second target feature map; inputting the second target feature image into the adjacent third sampling and splicing unit, and so on until the last sampling and splicing unit outputs a final target feature image;

S23: inputting the final target feature image output by the last sampling and splicing unit into a global pooling layer to reduce the dimension and then inputting the final target feature image into the full-connection layer so that the full-connection layer outputs a classification result corresponding to the classification picture;

The universal layer comprises the following components connected in sequence: the system comprises a depth convolution layer, a splicing layer, a point-by-point convolution layer, a nonlinear activation layer and a batch standardization layer; each universal layer input is an a-group characteristic diagram and a b-group characteristic diagram; b group of feature graphs are input into the depth convolution layer to be convolved to obtain c group of feature graphs; inputting the a group of feature images and the c group of feature images into a splicing layer, a point-by-point convolution layer, a nonlinear activation layer and a batch standardization layer in the universal layer to obtain a b group of feature images; the new b-group feature map is used as a new a-group feature map and the new b-group feature map is used as the input of the next adjacent general layer.

2. The method for classifying images based on a lightweight convolutional neural network of claim 1,

The downsampling layer is used for comprising the following steps of: a Gaussian downsampling layer, a point-by-point convolution layer, a nonlinear activation layer and a batch standardization layer; the downsampling layer outputs two groups of output feature graphs;

the input feature map is subjected to Gaussian downsampling layer to obtain a group of output feature maps;

The input feature images are sequentially input into a Gaussian downsampling layer, a point-by-point convolution layer, a nonlinear activation layer and a batch standardization layer in the downsampling layer to obtain another group of output feature images.

3. The method for classifying images based on a lightweight convolutional neural network according to claim 2,

The Gaussian downsampling layer in the downsampling layers carries out convolution operation on the feature map output by the upper layer, and the resolution of the output feature map is half of that of the input feature map.

4. The method for classifying images based on a lightweight convolutional neural network according to claim 2,

And the point-by-point convolution layer in the downsampling layer expands or contracts channels of input features according to the number of the input and output feature channels.

5. The method for classifying images based on a lightweight convolutional neural network according to claim 2,

ReLU is used as an activation function in a nonlinear activation layer in the downsampling layer.

6. An image classification apparatus based on a lightweight convolutional neural network, for performing the method of any one of claims 1-5, comprising:

The building module is used for building a lightweight convolutional neural network model, and the lightweight neural network model comprises the following components connected in sequence: the system comprises a standard convolution layer, a plurality of sampling splicing units and a full connection layer, wherein the sampling splicing units comprise sequentially connected: the system comprises a downsampling layer, a plurality of universal layers and a splicing layer;

The classification module is used for inputting the pictures to be classified into the lightweight convolutional neural network model to obtain classification results, and specifically comprises the following steps:

Expanding the channels of the pictures to be classified into the number of specified channels by using the standard convolution layer, so as to obtain an original feature map;

Downsampling the original feature images by using a downsampling layer in a first sampling and stitching unit to obtain two groups of first feature images, then respectively extracting features of the two groups of first feature images by using a plurality of universal layers in the downsampling layer to obtain respective corresponding second feature images, and stitching the two groups of second feature images by using a stitching layer in the downsampling layer to obtain a first target feature image; inputting the first target feature map into a second adjacent sampling and splicing unit so as to enable the first target feature map to be subjected to downsampling, feature extraction and splicing to obtain a second target feature map; inputting the second target feature image into the adjacent third sampling and splicing unit, and so on until the last sampling and splicing unit outputs a final target feature image;

And inputting the final target feature image output by the last sampling and splicing unit into the global pooling layer, reducing the dimension, and then inputting the final target feature image into the full-connection layer, so that the full-connection layer outputs a classification result corresponding to the classification image.