CN110414301A - A method for estimating crowd density in train cars based on dual cameras - Google Patents

Abstract

The present invention discloses a kind of based on double compartment crowd density estimation methods for taking the photograph head, it include: to propose multi-angle of view crowd density estimation network, the network consists of two parts, a part is the convolutional neural networks of parameter sharing, another part is full articulamentum, which can distinguish the crowd density grade in current train compartment.Model training stage is iterated optimization using the sample with 5 class density ratings；The model application stage, according to the regular Sampling Estimation of subway practical operation situation.The present invention is based on deep learning methods to estimate crowd density, replaces the feature of previous hand-designed, using the automatic learning characteristic of convolutional neural networks to improve the accuracy rate and robustness of crowd density estimation.

Description

A method for estimating crowd density in train cars based on dual cameras

technical field

The invention relates to the technical field of crowd density estimation, in particular to a method for estimating crowd density in a train compartment based on dual cameras.

Background technique

There are still many deficiencies in the existing crowd density estimation techniques. Pixel-based methods are simple and easy to implement, but only suitable for scenes with low crowd density. Although the method based on texture analysis has a good effect, the calculation is complicated, and the real-time performance is often not achieved in practical applications. The method based on object detection can obtain reliable results in relatively crowded situations, but it loses its application ability in scenes with high crowd overlap.

The existing crowd density estimation techniques mainly fall into the following categories:

1) Methods based on pixel statistics [1]. The total area of the crowd and the pixels at the edge of the crowd are counted, and the crowd density is estimated according to the linear relationship between the obtained pixels and the total number of people. This method obtains the number of foreground, background and edge pixels in the image through background subtraction and edge detection technology. This method is mainly used in scenes with sparse crowd distribution.

2) Method based on texture analysis [2]. Image texture features are extracted by gray co-occurrence matrix and wavelet packet decomposition, and then these features are learned and trained using support vector machine, adaboost and neural network as classification models. This method is mainly used in scenes with dense crowd distribution.

3) The method based on object detection [3]. Through the head detector based on haar-like and haar wavelet transform, the SVM classifier is used to judge whether it is a head, and finally the density of the whole crowd is estimated.

SUMMARY OF THE INVENTION

The main purpose of the present invention is to propose a method for estimating crowd density in train compartments based on dual cameras, aiming at overcoming the above problems.

To achieve the above purpose, a method for estimating crowd density in train compartments based on dual cameras includes the following steps:

S10 Prepare training samples: build a neural network with 4 parameter-sharing convolutional layers and 5 fully connected layers, input video frames from two different perspectives at the same time in the same car, and train samples with density-level labels, where The convolutional layer is used to extract the feature vector of the video, and the fully connected layer is used to classify the feature vector extracted by the convolutional layer according to the density level;

S20 neural network training: several iterative optimization training neural network;

S30 application stage: intercept the video frames of the current train car captured by the dual cameras and input them to the optimized neural network to obtain the image classification result of the current train car.

Preferably, the four convolution layers in S10 include a first Conv layer, a second Conv layer, a third Conv layer and a fourth Conv layer, and the size of the convolution kernel of the first Conv layer is 9×9 , the step size is 1, the number of convolution kernels is 16, and the image is input to the first Conv layer to generate 16 feature maps. After connecting the linear rectification function Relu layer and the maximum pooling Max-pooling layer, the output size is 288×464 × 16 feature maps.

Preferably, the size of the convolution kernel of the second Conv layer is 7×7, the stride is 1, the number of convolution kernels is 32, and the image is input into the second Conv layer to generate 32 feature maps, which are connected to the Relu layer and Max -pooling layer that outputs feature maps of size 144×232×32.

Preferably, the size of the convolution kernel of the third Conv layer is 7×7, the stride is 1, the number of convolution kernels is 16, and the image is input into the third Conv layer to generate 16 feature maps, and then the Relu layer and The Max-pooling layer outputs feature maps of size 72×116×16.

Preferably, the size of the convolution kernel of the fourth Conv layer is 7×7, the stride is 1, the number of convolution kernels is 8, and the image is input to the fourth Conv layer to generate 8 feature maps, which are connected to the Relu layer and The Max-pooling layer outputs feature maps of size 36×58×8.

Preferably, the five fully connected layers include FC5, FC6, FC7, FC8 and Softmax layers, and the fifth Conv layer outputs two 36×58×8 feature maps of dual cameras, which are respectively input to the fully connected layer FC5_0 and FC5_1 to obtain two sets of 1024-dimensional feature vectors; the two sets of vectors are input to the FC6_0 layer and the FC6_1 layer to obtain two sets of 512-dimensional feature vectors, and then the two sets of 512-dimensional feature vectors are added to obtain a new set of 512 dimensional feature vector; the new set of 512-dimensional feature vector is input to the FC7 layer to obtain a 256-dimensional feature vector; then the 256-dimensional feature vector is input to the FC8 layer to obtain a 128-dimensional feature vector; finally the 128-dimensional feature vector The vector is input to the Softmax layer to obtain a set of 5-dimensional probability vectors.

Preferably, the density levels include ex-low, low, medium, high, and ex-high, the ex-low sample label is [1, 0, 0, 0, 0], and the low sample label is [0, 1, 0, 0, 0], the sample label of the medium is [0, 0, 1, 0, 0], the sample label of the high is [0, 0, 0, 1, 0], The sample label of the ex-high is [0, 0, 0, 0, 1], and the crowd density level of the image is determined according to the output value of the last layer.

Preferably, the 20 specifically includes:

S201 sets each batch of the neural network as a predetermined value, and each iteration inputs the samples of the predetermined value;

S202 uses the gaussian to initialize the parameters of the convolutional layer of the neural network, and uses the ^xavier method to ^initialize the parameters of the fully connected layer. The xavier is to initialize the parameters in a uniformly distributed manner, specifically where n represents the input dimension of the layer where the parameter is located, and m represents the output dimension;

S203 uses the Adam algorithm for optimization training, where the formula of the Adam algorithm is as follows:

Parameter update rules:

In the above formula, Indicates the gradient of the t-th iteration, β ₁ , β ₂ are hyperparameters, generally set to 0.9 and 0.999, ∈ is a small value in case the denominator is zero, generally set to 10 ^-8 , m _t is approximately regarded as right the expectation of , v _t is approximately regarded as right expectations, while and are unbiased estimates of m _t and v _t , respectively;

S204 uses the loss function Softmax to iterate the neural network several times until it reaches the optimum. The formula of the loss function Softmax is as follows:

Among them, the left term is the cross-entropy cost function, [f ₁ , f ₂ ,..., f _K ] is the output vector of the network, in this task K=5, y _i represents the ith sample in this iteration corresponding to The rank density of , the right term R(W) is the regular term, W represents the network parameter, λ is a hyperparameter and is set to 0.0002.

Preferably, the weighted fusion of the two video frames is performed in the step 30, and the calculation formula for obtaining the image classification result of the current train car is as follows:

class=argmax{[F(X ₁ ; θ)+F(X ₂ ; θ)]/2}

Among them, F(X _i ; θ) is the output of the network model, X ₁ and X ₂ are the images input by the two cameras respectively, and θ is the parameter of the convergent model.

The present invention is a multi-view crowd density estimation method based on deep learning, adopts the convolutional neural network to automatically learn features to replace the previous hand-designed features, and obtains a more robust model, and proposes a dual-level model for the special environment of subway cars. The input of the end camera can be used to deal with the problem of severe occlusion in extreme cases and improve the accuracy of crowd density estimation.

Description of drawings

In order to explain the embodiments of the present invention or the technical solutions in the prior art more clearly, the following briefly introduces the accompanying drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only These are some embodiments of the present invention, and for those of ordinary skill in the art, other drawings can also be obtained according to the structures shown in these drawings without creative efforts.

Fig. 1 is a method flow chart of a method for estimating crowd density in train cars based on dual cameras of the present invention;

The realization, functional characteristics and advantages of the present invention will be further described with reference to the accompanying drawings in conjunction with the embodiments.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

It should be noted that if there are directional indications (such as up, down, left, right, front, back, etc.) involved in the embodiments of the present invention, the directional indications are only used to explain a certain posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication also changes accordingly.

In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present invention, the descriptions of "first", "second", etc. are only used for the purpose of description, and should not be construed as indicating or implying Its relative importance or implicitly indicates the number of technical features indicated. Thus, a feature delimited with "first", "second" may expressly or implicitly include at least one of that feature. In addition, the technical solutions between the various embodiments can be combined with each other, but must be based on the realization by those of ordinary skill in the art. When the combination of technical solutions is contradictory or cannot be realized, it should be considered that the combination of such technical solutions does not exist. , is not within the protection scope of the present invention.

As shown in Figure 1, Figure 1 is a flow chart of the method for estimating crowd density in train cars based on dual cameras of the present invention, (a) is the training stage, (b) is the application stage, and Back Propagation , referred to as BP) algorithm iteratively optimizes the model parameters; in the test phase, the method of probability vector fusion is used to improve the classification accuracy.

A method for estimating crowd density in train compartments based on dual cameras, comprising the following steps:

S10 Prepare training samples: build a neural network with 4 parameter-sharing convolutional layers and 5 fully connected layers, input video frames from two different perspectives at the same time in the same car, and train samples with density-level labels, where The convolutional layer is used to extract the feature vector of the video, and the fully connected layer is used to classify the feature vector extracted by the convolutional layer according to the density level;

S20 neural network training: several iterative optimization training neural network;

S30 application stage: intercept the video frames of the current train car captured by the dual cameras and input them to the optimized neural network to obtain the image classification result of the current train car.

Preferably, the five convolution layers in S10 include a first Conv layer, a second Conv layer, a third Conv layer, a fourth Conv layer and a fifth Conv layer, and the convolution kernel of the first Conv layer The size is 9 × 9, the stride is 1, the number of convolution kernels is 16, the image is input to the first Conv layer to generate 16 feature maps, and after connecting the linear rectification function Relu layer and the maximum pooling Max-pooling layer, the output Feature maps of size 288×464×16.

Preferably, the size of the convolution kernel of the second Conv layer is 7×7, the stride is 1, the number of convolution kernels is 32, and the image is input into the second Conv layer to generate 32 feature maps, which are connected to the Relu layer and Max -pooling layer that outputs feature maps of size 144×232×32.

Preferably, the size of the convolution kernel of the third Conv layer is 7×7, the stride is 1, the number of convolution kernels is 16, and the image is input into the third Conv layer to generate 16 feature maps, and then the Relu layer and The Max-pooling layer outputs feature maps of size 72×116×16.

Preferably, the size of the convolution kernel of the fourth Conv layer is 7×7, the stride is 1, the number of convolution kernels is 8, and the image is input to the fourth Conv layer to generate 8 feature maps, which are connected to the Relu layer and The Max-pooling layer outputs feature maps of size 36×58×8.

Preferably, the five fully connected layers include FC5, FC6, FC7, FC8 and Softmax layers, and the fifth Conv layer outputs two 36×58×8 feature maps of dual cameras, which are respectively input to the fully connected layer FC5_0 and FC5_1 to obtain two sets of 1024-dimensional feature vectors; the two sets of vectors are input to the FC6_0 layer and the FC6_1 layer to obtain two sets of 512-dimensional feature vectors, and then the two sets of 512-dimensional feature vectors are added to obtain a new set of 512 dimensional feature vector; the new set of 512-dimensional feature vector is input to the FC7 layer to obtain a 256-dimensional feature vector; then the 256-dimensional feature vector is input to the FC8 layer to obtain a 128-dimensional feature vector; finally the 128-dimensional feature vector The vector is input to the Softmax layer to obtain a set of 5-dimensional probability vectors.

Preferably, the density levels include ex-low, low, medium, high, and ex-high, the ex-low sample label is [1, 0, 0, 0, 0], and the low sample label is [0, 1, 0, 0, 0], the sample label of the medium is [0, 0, 1, 0, 0], the sample label of the high is [0, 0, 0, 1, 0], The sample label of the ex-high is [0, 0, 0, 0, 1], and the crowd density level of the image is determined according to the output value of the last layer.

Preferably, the 20 specifically includes:

S201 sets each batch of the neural network as a predetermined value, and each iteration inputs the samples of the predetermined value;

S202 uses the gaussian to initialize the parameters of the convolutional layer of the neural network, and uses the ^xavier method to ^initialize the parameters of the fully connected layer. The xavier is to initialize the parameters in a uniformly distributed manner, specifically where n represents the input dimension of the layer where the parameter is located, and m represents the output dimension;

S203 uses the Adam algorithm for optimization training, where the formula of the Adam algorithm is as follows:

Parameter update rules:

In the above formula, Indicates the gradient of the t-th iteration, β ₁ , β ₂ are hyperparameters, generally set to 0.9 and 0.999, ∈ is a small value in case the denominator is zero, generally set to 10, mt is approximately regarded as correct the expectation of , v _t is approximately regarded as right expectations, while and are unbiased estimates of m _t and v _t , respectively;

S204 uses the loss function Softmax to iterate the neural network several times until it reaches the optimum. The formula of the loss function Softmax is as follows:

Among them, the left term is the cross-entropy cost function, [f ₁ , f ₂ ,..., f _K ] is the output vector of the network, in this task K=5, y _i represents the ith sample in this iteration corresponding to The rank density of , the right term R(W) is the regular term, W represents the network parameter, λ is a hyperparameter and is set to 0.0002.

Preferably, the weighted fusion of the two video frames is performed in the step 30, and the calculation formula for obtaining the image classification result of the current train car is as follows:

class=argmax{[F(X ₁ ; θ)+F(X ₂ ; θ)]/2}

Among them, F(X _i ; θ) is the output of the network model, X ₁ and X ₂ are the images input by the two cameras respectively, and θ is the parameter of the convergent model.

In the embodiment of the present invention, the present invention proposes a classification neural network including 5 convolution layers. The subway crowd pictures and their classification labels are input into the network, and the loss function Softmax is used to continuously train and optimize the network parameters; the invention also proposes multi-camera input to solve the occlusion problem, and the final prediction result is the fusion of each input result, which improves the classification accuracy.

Compared with the previous crowd density estimation technology, the present invention has the following advantages:

1. It can achieve satisfactory results in the environment from sparse to extremely crowded subway cars, and has better robustness;

2. The training is completed end-to-end. Compared with the traditional method, there is no complicated operation process, and it can achieve real-time performance in practical applications.

The basis of the invention is that it can accurately estimate the crowd density level in dense places such as subway cars, and has the effects of robustness and real-time performance. Therefore, any application technology of crowd density classification based on the present invention is included in the present invention, such as video intelligent monitoring and the like.

Claims (9)

1. a method for estimating crowd density in train compartments based on dual cameras, is characterized in that, comprises the steps: S10 Prepare training samples: build a neural network with 4 parameter-sharing convolutional layers and 5 fully connected layers, input video frames from two different perspectives at the same time in the same car, and train samples with density-level labels, where The convolutional layer is used to extract the feature vector of the video, and the fully connected layer is used to classify the feature vector extracted by the convolutional layer according to the density level; S20 neural network training: several iterative optimization training neural network; S30 application stage: intercept the video frames of the current train car captured by the dual cameras and input them to the optimized neural network to obtain the image classification result of the current train car. 2 . The method for estimating crowd density in train compartments based on dual cameras according to claim 1 , wherein the four convolution layers in S10 include a first Conv layer, a second Conv layer, and a third Conv layer. 3 . layer and the fourth Conv layer, the size of the convolution kernel of the first Conv layer is 9 × 9, the stride is 1, the number of convolution kernels is 16, and the image is input to the first Conv layer to generate 16 feature maps, which are connected to the line. After the rectification function Relu layer and the maximum pooling Max-pooling layer, the output size is 288 × 464 × 16 feature map. 3 . The method for estimating crowd density in train cars based on dual cameras according to claim 1 , wherein the convolution kernel size of the second Conv layer is 7×7, the step size is 1, and the number of convolution kernels is 1. 4 . It is 32, the image is input to the second Conv layer to generate 32 feature maps, connected to the Relu layer and the Max-pooling layer, and the output size is 144 × 232 × 32 feature maps. 4 . The method for estimating crowd density in train cars based on dual cameras according to claim 1 , wherein the convolution kernel size of the third Conv layer is 7×7, the step size is 1, and the number of convolution kernels is 1. 5 . It is 16, the image is input to the third Conv layer to generate 16 feature maps, and then connected to the Relu layer and the Max-pooling layer, and the output size is 72 × 116 × 16 feature maps. 5 . The method for estimating crowd density in train cars based on dual cameras according to claim 1 , wherein the convolution kernel size of the fourth Conv layer is 7×7, the step size is 1, and the convolution kernel is 1. 6 . The number is 8, and the image is input to the fourth Conv layer to generate 8 feature maps, which are connected to the Relu layer and the Max-pooling layer, and the output size is 36 × 58 × 8 feature maps. 6. The method for estimating crowd density in train compartments based on dual cameras according to claim 1, wherein the five fully connected layers include FC5, FC6, FC7, FC8 and Softmax layers, and the fifth Conv layer outputs The two 36×58×8 feature maps of the dual camera are input to the fully connected layers FC5_0 and FC5_1 respectively, and two sets of 1024-dimensional feature vectors are obtained; the two sets of vectors are input to the FC6_0 layer and the FC6_1 layer respectively to obtain two sets of 512-dimensional feature vectors. Then two sets of 512-dimensional feature vectors are added to obtain a new set of 512-dimensional feature vectors; the new set of 512-dimensional feature vectors is input to the FC7 layer to obtain a 256-dimensional feature vector; then the 256-dimensional feature vector is obtained. The dimensional feature vector is input to the FC8 layer to obtain a 128-dimensional feature vector; finally, the 128-dimensional feature vector is input to the Softmax layer to obtain a set of 5-dimensional probability vectors. 7 . The method for estimating crowd density in train compartments based on dual cameras according to claim 1 , wherein the density levels include ex-low, low, medium, high, and ex-high, and the ex-low The sample label is [1,0,0,0,0], the low sample label is [0,1,0,0,0], the medium sample label is [0,0,1,0, 0], the sample label of the high is [0,0,0,1,0], the sample label of the ex-high is [0,0,0,0,1], according to the size of the output value of the last layer Determine the crowd density level of the image. 8. The method for estimating crowd density in train compartments based on dual cameras according to claim 1, wherein the method 20 specifically comprises: S201 sets each batch of the neural network as a predetermined value, and each iteration inputs the samples of the predetermined value; S202 uses gaussian to initialize the parameters of the convolutional layer of the neural network, and uses the xavier method to initialize the parameters of the fully connected layer. The gaussian" is a Gaussian distribution initialized as W~N(μ,σ ² ), where μ=0,σ ² =0.01, The xavier is to initialize the parameters in a uniformly distributed manner, specifically where n represents the input dimension of the layer where the parameter is located, and m represents the output dimension; S203 uses the Adam algorithm for optimization training, where the formula of the Adam algorithm is as follows: Parameter update rules: In the above formula, Indicates the gradient of the t-th iteration, β ₁ , β ₂ are hyperparameters, generally set to 0.9 and 0.999, ∈ is a small value in case the denominator is zero, generally set to 10 ^-8 , m _t is approximately regarded as right the expectation of , v _t is approximately regarded as right expectations, while and are unbiased estimates of m _t and v _t , respectively; S204 uses the loss function Softmax to iterate the neural network several times until it reaches the optimum. The formula of the loss function Softmax is as follows: Among them, the left term is the cross-entropy cost function, [f ₁ , f ₂ ,...,f _K ] is the output vector of the network, in this task K=5, y _i represents the ith sample in this iteration corresponding to The rank density of , the right term R(W) is the regular term, W represents the network parameter, λ is a hyperparameter and is set to 0.0002. 9. The method for estimating crowd density in train carriages based on dual cameras as claimed in claim 1, wherein the weighted fusion of two video frames is carried out in the described 30, and the calculation formula for obtaining the image classification result of the current train carriage is as follows: class=argmax{[F(X ₁ ; θ)tF(X ₂ ; θ)]/2} Among them, F(X _i ; θ) is the output of the network model, X ₁ and X ₂ are the images input by the two cameras respectively, and θ is the parameter of the convergent model.