CN115457464B - Crowd counting method based on transformer and CNN - Google Patents

Abstract

The invention discloses a crowd counting method based on transformer and CNN, which includes: obtaining training samples and performing preprocessing enhancement; inputting the enhanced RGB image into the model backbone network to obtain global feature maps of different resolutions; After sampling on the map, channel superposition is performed to obtain an aggregate feature map; the aggregate feature map is input into a multi-branch convolutional neural network to obtain a multi-scale feature map, and then added in the channel dimension to obtain a multi-scale aggregate feature map; the multi-scale aggregate feature map is obtained The graph input density map regression layer performs smooth dimensionality reduction and outputs the density map; uses the optimal transmission loss for training, and finally makes predictions. The present invention combines a pyramid transformer with a multi-branch convolutional neural network, which increases the receptive field of the model, effectively reduces the impact of variable scales, and improves prediction accuracy.

Description

Crowd counting method based on transformer and CNN

Technical field

The invention relates to a crowd counting method based on transformer and CNN, and belongs to the field of image processing.

Background technique

With the rapid development of society and the continuous improvement of people's living standards, there are more and more crowd gathering scenes, especially in large venues, transportation hubs, shopping malls and other public places. The demand for safety in these scenes continues to increase, so master the above scenes. Conducting safety management and emergency evacuation based on the distribution status of medium-sized crowds has attracted widespread attention from researchers. The crowd density estimation method is to estimate the density of the crowd in a given scene, generate a density map with distribution information, and give the total number of crowds.

There are currently three main methods for crowd density estimation: detection-based methods, regression-based methods and density map-based methods. Detection-based methods count pedestrian targets through manually designed window detectors, but have poor detection results in densely populated areas. Regression-based methods directly regress the total number of people and lack crowd spatial information. Compared with the first two methods, the method based on density map can output the crowd density map while providing the total number of people, providing key information on the spatial distribution of the crowd, and has certain adaptability to dense areas, increasing the accuracy of counting. It reduces the difficulty of method design. Therefore, the current crowd density estimation methods are mainly based on density maps.

Most of the existing crowd density estimation methods based on density maps use the convolutional neural network model in deep learning for density map regression. However, the crowd target has serious scale variability problems, and the convolutional neural network model has a limited receptive field and cannot be effective. Capturing multi-scale features ultimately results in reduced counting model accuracy. Therefore, how to increase the receptive field of the model and reduce the impact of scale changes on counting accuracy has become an urgent problem that needs to be solved.

Contents of the invention

In view of the problem of variable scale in crowd scenes and the problem of limited receptive field in the convolutional neural network model, the present invention proposes a crowd counting method based on transformer and CNN: combining the pyramid transformer with the convolutional neural network, through the pyramid transformer Learn the global features of the image so that the model has a global receptive field and reduce the impact of variable scale problems on accuracy; at the same time, multi-scale feature learning and enhancement is carried out through multi-branch convolutional neural networks to enrich the feature representation of the model and provide local properties and inductive bias, and finally perform density map regression accurately.

In order to solve the above technical problems, the technical solution adopted by the present invention is:

The crowd counting method based on transformer and CNN includes the following steps:

(1) Obtain training samples and obtain a large number of crowd RGB images in multiple scenes, then obtain crowd annotations, and label pixels at each head position. The number of pixels represents the total number of people in the scene. The images are then enhanced, randomly flipped horizontally or vertically, and standardized. During training, the training images are cropped to a size of 256×256 for training.

(2) Calculate the backbone network of the enhanced crowd RGB image input model to obtain global feature maps of different resolutions. The backbone network is a pyramid transformer composed of four stages. Each stage includes an overlapping image block embedding layer. and an encoder.

Further, in the overlapping image block embedding layer, the input image is divided into overlapping image blocks in a convolution layer, and then a convolution operation is performed to output a two-dimensional feature map. The output two-dimensional feature map is expanded into a one-dimensional vector. And regularized, this vector will be used as the encoder input. The convolution kernel size of the convolution layer in the overlapping image block embedding layer in the first stage is 7 × 7, and the step size is 4; the convolution kernel size of the convolution layer in the overlapping image block embedding layer in the remaining three stages is 3 × 3, and the step size is 4. The length is 2; the output dimensions of the four convolutional layers are: 64, 128, 320, 512. By controlling the step size of the convolution layer, pyramid-shaped feature maps of different resolutions can be output.

Furthermore, in the encoder, the input vector undergoes self-attention calculation through multiple blocks. Each block includes a self-attention calculation layer and a forward propagation layer, and each layer is connected by skip connection. The number of blocks in each stage is: 3, 8, 27, 3; the number of heads in the multi-head self-attention layer in each stage is 1, 2, 5, 8 respectively. The vector calculated by the encoder is reshaped into a two-dimensional feature map and used as the input of the next stage. The final four stages output four sets of global feature maps with different resolutions, and the resolutions are 1/4, 1/8, 1/16, and 1/32 of the input enhanced crowd RGB image resolution.

(3) First, the four sets of global feature maps of different resolutions extracted in step (2) are upsampled to the same resolution while keeping the number of channels unchanged, and the feature maps of the last three stages are combined using bilinear interpolation. Upsample to the resolution of the first-stage feature map, which is 1/4 the size of the input enhanced crowd RGB image.

Then the feature maps of the four stages are aggregated. The aggregation method is to superimpose all the feature maps of the four stages in the channel dimension. The total number of channels is the sum of the number of channels of the feature maps of the four stages, that is, 64+128+320+ 512=1024, and finally an aggregated feature map with a channel number of 1024 is obtained.

(4) Input the aggregated feature map into the multi-branch convolutional neural network module of the network model to obtain a multi-scale feature map. The multi-branch convolutional neural network module includes three branches, each branch includes a convolution layer, the first branch convolution kernel size is 3×3, the second branch convolution kernel size is 5×5, and the third branch convolution kernel size is 5×5. The branch convolution kernel size is 7×7. The number of output channels of each branch is 256, and the convolutional layer of each branch is followed by a batch regularization layer and a ReLU activation function layer.

After three branch calculations, three sets of multi-scale feature maps with the same resolution and number of channels are obtained. Then the multi-scale feature maps are added pixel by pixel on the corresponding channels, specifically the corresponding position pixels of the three feature maps on each corresponding channel are added, and the number of channels of the finally obtained multi-scale aggregated feature map is 256.

(5) Input the multi-scale aggregated feature map output from step (4) into the density map regression layer for smooth dimensionality reduction and output a density map. The density map regression layer includes two convolution layers. The convolution kernel size of the first convolution layer is 3×3, the stride is 1, and the number of output channels is 64. The convolution kernel size of the second convolution layer is 1. ×1, the step size is 1, the number of output channels is 1, each convolutional layer is followed by a layer of batch regularization layer and ReLU activation function, and finally the crowd density estimation map and crowd counting results are output.

(6) Use the optimal transmission loss for training, regress the crowd density estimation map and the total number of people, optimize the model parameters, and then save the model parameters with the smallest loss; when predicting, load the saved model parameters with the smallest loss and directly obtain the crowd density The estimated map and crowd count results are used as prediction results.

Due to the adoption of the above technical solutions, the technical progress achieved by the present invention is:

1. The pyramid-type backbone network can output feature maps of different resolutions, so that crowd targets at different scales can have rich feature representation information. The high-resolution feature maps have rich details, which is beneficial to small-scale crowd prediction. There is rich semantic information in the resolution feature map, which is beneficial to large-scale crowd prediction. Aggregating feature maps of different resolutions can improve the accuracy of crowd density map estimation.

2. The transformer used has a global receptive field and can use all input pixels for modeling. By using the transformer as the backbone network of the model, the receptive field of the model can be increased, making up for the shortcomings of traditional convolutional neural network models that only have local receptive fields. It can effectively reduce the impact of scale changes in crowd scenes on the model and improve accuracy. .

3. Adopt a multi-branch convolutional neural network and use convolution kernels of different sizes to learn the global feature map, enrich the detailed representation of features of different scales in the feature map, further reduce the impact of scale-variable issues, and ultimately complete the density map. return.

4. Combine the transformer with the convolutional neural network so that the model has a global receptive field and is also local, avoiding the problem that a single transformer needs a large amount of data for training due to the lack of visual task induction bias and poor generalization. , while improving the problem of the limited receptive field of the convolutional neural network, and has strong adaptability to scenes with varying scales.

Description of the drawings

Figure 1 is a flow chart of the present invention;

Figure 2 is a schematic diagram of the overall network structure of the present invention;

Figure 3 is a schematic diagram of training according to the present invention;

Figure 4 is a schematic diagram of crowd density estimation using the present invention.

Detailed ways

The present invention will be further described in detail below in conjunction with the examples:

Figure 1 is a flow chart of the crowd density estimation method of the present invention. As shown in Figure 1, the crowd density estimation method in the present invention includes the following steps:

(1) Obtain training samples of crowd RGB images and obtain a large number of crowd RGB images from multiple scenes. At the same time, the annotation data of the RGB image of the corresponding crowd is obtained, and pixel points are annotated on the heads of the target group. One annotated pixel represents a pedestrian, and the sum of the pixel points is the total number of people. Finally, the horizontal and vertical coordinates of the annotated points are obtained. The training samples are then augmented with random horizontal and vertical flipping and normalized. During training, images are randomly cropped into 256×256 image block size for training.

The crowd RGB image consists of three channels: R (red), G (green), and B (blue). The crowd RGB image is standardized in the channel dimension, and the mean and standard deviation of the three channels are constrained to obtain the enhanced crowd. RGB image. The standardization method is:

Among them, C is the channel data, C’ is the channel data after normalization, mean is the channel mean, and std is the channel standard deviation. The enhanced crowd RGB image is then input into the pyramid transformer backbone network of the model as the prediction input.

(2) Figure 2 is a schematic diagram of the overall network structure of the present invention, which is used to build the network. The pyramid transformer backbone network consists of four stages. After the input is completed, the enhanced crowd RGB image passes through the four stages of the backbone network in sequence to obtain feature maps of different resolutions. The resolution of the output feature map at each stage is in turn after enhancement. 1/4, 1/8, 1/16, 1/32 of the crowd RGB image resolution.

Each stage of the backbone network consists of an overlapping patch embedding layer and an encoder. In each stage, the input feature map first passes through the image block embedding layer, and is mapped from the two-dimensional feature map to a one-dimensional vector. The one-dimensional vector is calculated by self-attention in the encoder and reshaped into a two-dimensional feature map to obtain the global The feature map is used as input to the next stage.

The structure and steps of each stage are explained in detail below:

The overlapping image patch embedding layer in the first stage includes a convolution layer and a regularization layer. Specifically, the convolution kernel size of the convolution layer is 7×7, the stride is 4, the number of input channels is 3, and the output The number of channels is 64. The step size is greater than 1, so that the resolution of the enhanced crowd RGB image is reduced to 1/4 of the input image after passing through the convolution layer, and the step size is smaller than the size of the convolution kernel, so that the image blocks covered by the convolution kernel can overlap, increasing Local information exchange. Then the output two-dimensional feature map is flattened into a one-dimensional vector, which can meet the calculation requirements while retaining all pixel information. The flattened vector is input to the regularization layer for regularization processing in the channel dimension to facilitate model convergence. Finally, the regularized result is fed into the encoder. The first-stage encoder includes 3 blocks. Each block consists of a multi-head self-attention layer and a forward propagation layer. Each layer is connected using skip connections. The self-attention calculation layer includes a regularization layer and a multi-head self-attention layer, where the regularization layer regularizes the input vector in the channel dimension. In the first stage, the number of heads of the multi-head self-attention layer is 1, the input vector dimension is 64, and the calculated vector is input to the forward propagation layer.

The self-attention calculation method is:

Among them, Q, K, and V represent the query matrix, key matrix, and value matrix, which are obtained by multiplying the input vector and the weight matrix W _Q , W _K , and W _V. d _k is the dimension of the input vector during multi-head self-attention calculation.

The softmax calculation method is:

The forward propagation layer consists of two fully connected layers and a deep convolutional layer. The output dimension of the first fully connected layer is 8 times the input dimension, the output dimension of the second fully connected layer is the dimension of the input vector of the forward propagation layer, and there is a depth convolution layer between the two fully connected layers. The convolutional layer first reshapes the vector into a two-dimensional feature map, then divides the two-dimensional feature map into groups with the same channel dimension, and performs a separate convolution on each group as a position encoding of the feature map. The kernel size is 3×3, the step size is 1, the input channel and the output channel are the same, both are the number of channels of the input vector. After the calculation is completed, the two-dimensional feature map is flattened into a vector as the output of the second fully connected layer. After the first fully connected layer and the deep convolution layer, they are activated through the GELU activation function. The activation function is calculated as:

GELU(x)＝xP(X≤x) (4)

where x is the input vector and P is the cumulative distribution of Gaussian distribution. The vectors passing through the forward propagation layer are then reshaped into two-dimensional feature maps as input to the second stage.

Furthermore, the composition structures of the second, third and fourth stages are similar to those of the first stage, with only some differences in the number of layers. In the second, third, and fourth stages, the convolution kernel of the convolutional layer in the overlapping image block embedding layer is 3×3, the stride is 2, and the output dimensions are 128, 320, and 512 respectively; the number of blocks in the encoder is 4 , 18, 3; the number of heads of the multi-head self-attention layer is 2, 5, and 8 respectively; the number of output channels of the first fully connected layer in the forward propagation layer is 8 times, 4 times, and 4 times the number of input channels. . The outputs of the first, second, and third stages are used as inputs to the next stage. Finally, the four stages output a total of four sets of two-dimensional global feature maps with different resolutions.

(3) Upsample the global feature map of the four stages. Using bilinear interpolation, upsample the output feature map of the second stage twice, upsample the output feature map of the third stage four times, and upsample the output feature map of the third stage four times. The output feature map of the four stages is upsampled eight times, while the output feature map of the first stage remains unchanged in size.

Furthermore, the four-stage feature maps are upsampled to the same size and then superimposed in the channel dimension to obtain an aggregate feature map. The output channel number of the aggregate feature map is the sum of the channel numbers of the four stages, and the resolution is 1 of the enhanced crowd RGB image. /4.

(4) Input the aggregated feature map into the multi-branch convolutional neural network module to obtain a multi-scale feature map. The multi-branch convolutional neural network module consists of three branches, each branch includes a convolution layer, a regularization layer and a ReLU activation function. Specifically, the convolution kernel sizes of the convolutional layers in the three branches are 3×3, 5×5, and 7×7, and the number of output channels is 256. The calculation method of the ReLU activation function is:

Further, after the aggregated feature map passes through three branches, three sets of multi-scale feature maps with the same resolution and number of channels are output, and then the multi-scale feature maps are added by pixel position in the channel dimension. Specifically, in the three sets of features On the corresponding channel of the image, pixels at the same position are added, the output resolution is 1/4 of the enhanced crowd RGB image, the output channel is 256, and a multi-scale aggregated feature map is obtained.

(5) Smooth and reduce the dimensionality of the multi-scale aggregated feature map, and finally output the crowd density estimation map and the number of people statistics results. Two convolutional layers are used to perform smoothing and dimensionality reduction operations. Each convolutional layer is followed by a regularization layer and a ReLU activation function. The convolution kernel size of the first convolution layer is 3×3 and the output channel is 64. The convolution kernel size of the second convolution layer is 1×1 and the number of output channels is 1. The feature map after dimensionality reduction and smoothing is the final crowd density estimation map, and all pixels in the density map are added to obtain the final people counting result.

(6) Figure 3 is a schematic diagram of the crowd density estimation method of the present invention for training. First, the network is trained. The loss function used in the present invention is the optimal transmission loss, as shown below:

L＝L ₁ +L _OT (6)

L ₁ =||S|| ₁ -||S'|| ₁ (7)

Among them, L is the overall loss function of the network model, L ₁ is the absolute error of the total number of people, which is used to optimize the number of people statistics results, and L _OT is the optimal transmission loss, which is used to optimize the distribution of the crowd density estimation map. S is the crowd label point map, ||S|| ₁ is the norm of S; S' is the crowd density estimation map, ||S'|| ₁ is the norm of S'. Φ is calculated by Wasserstein distance.

During training, the model parameters are optimized through the backpropagation gradient descent method, the target threshold is set, and the model parameters with the smallest loss function are selected and saved to complete the model training. Figure 4 is a schematic diagram of crowd density estimation using the crowd density estimation method of the present invention. During prediction, the image to be predicted is not enhanced and is directly input into the network to obtain the crowd density estimation map and the total number of people as the final prediction result.

The above-described embodiments are only descriptions of preferred embodiments of the present invention and do not limit the scope of the present invention. Without departing from the design spirit of the present invention, those of ordinary skill in the art may make various modifications to the technical solutions of the present invention. All modifications and improvements shall fall within the protection scope determined by the claims of the present invention.

Claims (6)

1. The crowd counting method based on the transformer and the CNN is characterized by comprising the following steps of: the method comprises the following steps:

(1) Obtaining training samples, obtaining crowd RGB images of multiple scenes, and preprocessing and enhancing the crowd RGB images;

(2) Inputting the enhanced crowd RGB images into a backbone network of a model for calculation, wherein the backbone network comprises a pyramid transducer formed by four stages, and the enhanced crowd RGB images sequentially pass through the four stages of the backbone network to obtain global feature images with different resolutions; wherein each stage includes an overlapping tile embedding layer and an encoder;

(3) Carrying out channel superposition after upsampling on global feature graphs with different resolutions to obtain an aggregate feature graph;

the step (3) specifically comprises:

firstly, up-sampling four groups of global feature images with different resolutions extracted in the step (2) to the same resolution, keeping the number of channels unchanged, and up-sampling the feature images of the last three stages to the resolution of the feature images of the first stage by a bilinear interpolation method, namely enhancing the 1/4 size of the RGB image of the crowd;

then, the feature images of the four stages are aggregated, wherein the aggregation method is to superimpose all the feature images of the four stages in the channel dimension, the total channel number is the sum of the channel numbers of the feature images of the four stages, namely 64+128+320+512=1024, and finally, the aggregation feature image with the channel number of 1024 is obtained;

(4) Inputting the aggregation feature map into a multi-branch convolutional neural network to obtain a multi-scale feature map, and adding the multi-scale feature map in the channel dimension to obtain the multi-scale aggregation feature map;

the step (4) specifically comprises:

the multi-branch convolutional neural network module comprises three branches, each branch comprises a convolutional layer, the first branch has a convolutional kernel size of 3 multiplied by 3, the second branch has a convolutional kernel size of 5 multiplied by 5, and the third branch has a convolutional kernel size of 7 multiplied by 7; the number of output channels of each branch is 256, and a batch regularization layer and a ReLU activation function layer are arranged behind the convolution layer of each branch;

after three branch calculation, three groups of multi-scale feature images with the same resolution and channel number are obtained; then adding the multi-scale feature images pixel by pixel on the corresponding channels, specifically adding pixels at positions corresponding to the three feature images on each corresponding channel, and finally obtaining 256 channels of the multi-scale aggregate feature images;

(5) Inputting the multi-scale aggregation feature map into a density map regression layer to carry out smooth dimension reduction and output a density map;

(6) Training is carried out by using the optimal transmission loss, and finally prediction is carried out.

2. The transformer and CNN based population counting method of claim 1, wherein: the step (1) is to acquire labeling data of the RGB images of the crowd before preprocessing and enhancing, pixel point labeling is carried out at each head position, and the number of the pixel points represents the total number of people in the scene.

3. The transformer and CNN based population counting method of claim 1, wherein: the preprocessing enhancement in the step (1) specifically comprises random horizontal or vertical overturn and standardization, and training is carried out by cutting a training image into 256×256 image blocks during training.

4. The transformer and CNN based population counting method of claim 1, wherein: the step (2) specifically comprises:

in the overlapped image block embedding layer, an input image is divided into mutually overlapped image blocks in one convolution layer, then a two-dimensional feature map is output through convolution operation, and the output two-dimensional feature map is unfolded into one-dimensional vectors and regularized to be used as encoder input; the convolution kernel size of the convolution layer of the overlapped image block embedded layer in the first stage is 7 multiplied by 7, and the step length is 4; the convolution kernel size of the convolution layer in the other three-stage overlapped image block embedded layer is 3 multiplied by 3, and the step length is 2; the output dimensions of the four layers of convolution layers are as follows: 64 128, 320, 512; outputting pyramid-type feature graphs with different resolutions by controlling the step length of the convolution layer;

in the encoder, the input vector carries out self-attention calculation through a plurality of blocks, each block comprises a self-attention calculation layer and a forward propagation layer, and each layer is connected in a jump connection mode; the number of blocks in each stage is as follows: 3,8, 27,3; the number of the multi-head self-attention layers in each stage is 1,2,5 and 8 respectively; the vector after being calculated by the encoder is remodeled into a two-dimensional feature map and is used as input of the next stage; and finally, outputting four groups of global feature maps with different resolutions at four stages, wherein the resolutions are sequentially 1/4,1/8,1/16 and 1/32 of the input enhanced crowd RGB image resolution.

5. The transformer and CNN based population counting method of claim 1, wherein: the step (5) specifically comprises:

the density map regression layer comprises two layers of convolution layers, wherein the convolution kernel size of the first layer of convolution layer is 3 multiplied by 3, the step length is 1, the number of output channels is 64, the convolution kernel size of the second layer of convolution layer is 1 multiplied by 1, the step length is 1, the number of output channels is 1, a batch of regularization layers and a ReLU activation function are arranged behind each layer of convolution layer, and finally the crowd density estimation map and crowd counting result are output.

6. The transformer and CNN based population counting method of claim 1, wherein: the step (6) specifically comprises:

training by using optimal transmission loss, carrying out regression on a crowd density estimation graph and the total number of people, optimizing model parameters, and then storing the model parameters with minimum loss; and loading the stored minimum loss model parameters during prediction, and directly acquiring a crowd density estimation graph and a crowd counting result as prediction results.