CN110889360A - Crowd counting method and system based on switching convolutional network - Google Patents
Crowd counting method and system based on switching convolutional network Download PDFInfo
- Publication number
- CN110889360A CN110889360A CN201911142695.6A CN201911142695A CN110889360A CN 110889360 A CN110889360 A CN 110889360A CN 201911142695 A CN201911142695 A CN 201911142695A CN 110889360 A CN110889360 A CN 110889360A
- Authority
- CN
- China
- Prior art keywords
- features
- map
- density
- image
- training
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/52—Surveillance or monitoring of activities, e.g. for recognising suspicious objects
- G06V20/53—Recognition of crowd images, e.g. recognition of crowd congestion
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
Abstract
The utility model discloses a crowd counting method and system based on a switching convolution network, which comprises the steps of partitioning a target image, inputting the image block into the switching convolution neural network, and classifying the image block according to density through a classifier; performing feature extraction on the classified image blocks through a regressor, and performing feature splicing on the obtained density map features to obtain a feature map combined with global density map features; and processing the feature map combined with the global density features through a mean pooling layer and a deconvolution layer to obtain a target estimated density map, and obtaining the number of people in the target image through integration. The switching convolution network uses CNNs with different convolution kernel sizes as regressors of density map prediction, selects the optimal CNN regressor for each input image by the trained selection classifier, and uses the result as a final result, so that the accuracy and robustness of the predicted crowd number are improved.
Description
Technical Field
The present disclosure relates to the field of image processing technologies, and in particular, to a crowd counting method and system based on a switched convolutional network.
Background
The statements in this section merely provide background information related to the present disclosure and may not necessarily constitute prior art.
The crowd counting is a branch of crowd analysis, and is essential for video monitoring, public area planning, traffic flow monitoring and the like. Its purpose is to be able to accurately predict the number of targets. Furthermore, the method for cluster technology can be extended to other fields of counting tasks, such as cell microscopy, vehicle counting, environmental investigations, etc. However, due to various complexities such as occlusion, high clutter, uneven distribution of people, uneven illumination, variations in appearance, scale, and perspective within and outside the scene, the accuracy of the results is far from optimal. Existing population counting methods fall into three main categories: detection-based methods, regression-based methods, and density estimation-based methods.
Based on the detection method, a sliding window detector is used for detecting the crowd in the scene, and the corresponding number of people is counted. Detection-based methods are mainly divided into two broad categories, one is ensemble-based detection and the other is partial-body-based detection. The detection method based on the whole body mainly trains a classifier, and detects the pedestrian by using the characteristics of wavelets, HOG, edges and the like extracted from the whole body of the pedestrian. The learning algorithm mainly comprises methods such as SVM, boosting and random forest. The method based on the overall detection is mainly suitable for sparse population counting, and with the increase of population density, the shielding between people becomes more and more serious. Methods based on partial body detection are used to deal with the people counting problem. The number of people is counted based on detecting the partial structure of the body, such as the head, shoulders, etc. This method is slightly more efficient than the overall-based detection.
Regression-based methods. Whatever the detection-based method, it is difficult to deal with the problem of severe occlusion between people. Therefore, regression-based methods are increasingly being used to solve the problem of population counts. The main idea is to use a regression-based approach by learning a feature-to-population mapping. The method mainly comprises two steps, wherein in the first step, low-level features such as foreground features, edge features, textures and gradient features are extracted; the second step is to learn a regression model, such as linear regression, piecewise linear regression, ridge regression, and Gaussian process regression, to learn a mapping relationship of low-level features to population.
In recent years, deep learning has been widely used in various research fields such as computer vision, natural language processing, and the like. Deep learning has achieved great success in many ways by virtue of its excellent feature learning capabilities. Researchers have also begun to apply convolutional neural networks to population counting methods based on density estimation, which networks can better learn the non-linear mapping between images and density maps.
Disclosure of Invention
In order to solve the problems, the disclosure provides a crowd counting method and a crowd counting system based on a switched convolutional network, wherein the switched convolutional network uses CNNs with different convolutional kernel sizes as regressors for density map prediction, then uses a trained selection classifier to select an optimal CNN regressor for each input image, and uses the result as a final result.
In order to achieve the purpose, the following technical scheme is adopted in the disclosure:
in a first aspect, the present disclosure provides a crowd counting method based on a switched convolutional network, including:
partitioning a target image, inputting image blocks into a switching convolutional neural network, and classifying the image blocks according to density through a classifier;
performing feature extraction on the classified image blocks through a regressor, and performing feature splicing on the obtained density map features to obtain a feature map combined with global density map features;
and processing the feature map combined with the global density features through a mean pooling layer and a deconvolution layer to obtain a target estimated density map, and integrating the target estimated density map to obtain the number of people in the target image.
As some possible implementation manners, the switching convolutional neural network comprises a plurality of CNN regressors with different structures and a classifier for selecting an optimal regressor;
the classifier is a VGG 16-based three-class classifier, a full connection layer in a VGG16 is removed, a global average pool on a Conv 5 feature is used for removing spatial information and aggregation discriminant features, the full connection layer and the three-level Softmax classifier behind the global average pool correspond to a CNN regression, the classified image blocks with different densities are transmitted to the corresponding regressors, and feature maps with different scales are obtained.
As some possible implementation manners, the feature stitching specifically includes that the tensor of the target image block passes through a regressor to obtain a corresponding input tensor, the row dimension is kept unchanged, and the column dimensions are stitched to obtain a feature map combining features of the global density map.
As some possible implementation manners, the processing of the mean-value pooling and deconvolution layers includes enhancing the feature map combined with the global density features by two convolution layers to generate features of the feature map, reducing the resolution of the feature map by using the two deconvolution layers to obtain a target estimation feature map, and mapping the target estimation feature map to obtain a target estimation density map.
In a second aspect, the present disclosure provides a crowd counting system based on a switched convolutional network, comprising:
the classification module is used for partitioning a target image, inputting the image block into the switching convolutional neural network, and classifying the image block according to the density through the classifier;
the feature splicing module is used for extracting the features of the density map of the classified image blocks through the regressor, and obtaining a feature map combined with the features of the global density map through feature splicing of the obtained features of the density map;
and the calculation module is used for processing the feature map combined with the global density features through a mean pooling layer and a deconvolution layer to obtain a target estimated density map, and integrating the target estimated density map to obtain the number of people in the target image.
In a third aspect, the present disclosure provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the switched convolutional network based crowd counting method when executing the program.
In a fourth aspect, the present disclosure provides a computer-readable storage medium having a computer program stored thereon, where the program is to be executed by a processor to perform the steps of a switched convolutional network based crowd counting method.
Compared with the prior art, the beneficial effect of this disclosure is:
(1) according to the method, the estimated number of people is obtained by integrating the crowd density map, so that the influence of crowd shielding and crowd uneven distribution in the crowd image on counting can be avoided;
(2) the method and the device generate the characteristic graph by using the Switch-CNN network, generate the density graph with higher resolution and accuracy by deconvolution, can be suitable for the uneven crowd distribution scene, generate the density graph containing more comprehensive information, and have higher accuracy and robustness of the people number estimation result.
Drawings
The accompanying drawings, which are included to provide a further understanding of the disclosure, illustrate embodiments of the disclosure and together with the description serve to explain the disclosure and are not to limit the disclosure.
Fig. 1 is a schematic flow diagram of the disclosed method.
The specific implementation mode is as follows:
the present disclosure is further described with reference to the following drawings and examples.
It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the disclosure. Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs.
It is noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of example embodiments according to the present disclosure. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, and it should be understood that when the terms "comprises" and/or "comprising" are used in this specification, they specify the presence of stated features, steps, operations, devices, components, and/or combinations thereof, unless the context clearly indicates otherwise.
Example 1
As shown in fig. 1, the present disclosure provides a crowd counting method based on a switched convolutional network, including:
s1, the input image is divided into 9 pieces of original image with length 1/3 and width not overlapping each other. The purpose is to make the input image block be regarded as having single density, scale and visual angle information as a minimum unit for selecting a regressor;
s2, distributing labels to the input image blocks through the Switch-CNN classifier to classify the image blocks, and selecting a proper CNN regressor to perform density estimation;
the Switch-CNN classifier is a VGG 16-based three-class classifier, removes the full connection layer in the VGG16, and removes the spatial information and the aggregation discriminant features by using a Global Average Pool (GAP) on the Conv 5 features. The GAP is followed by a smaller full-link layer and a three-level Softmax classifier, which corresponds to three regressor networks in Switch-CNN, and training is performed according to a classification result obtained in a differential training phase, so that the classifier can distribute images with different densities to corresponding regressors.
S3, respectively extracting the features of the classified image blocks with different densities through three different regressors R1, R2 and R3;
three CNN regressors were introduced in Switch-CNN to predict population density. These CNN regressors have different receptive fields and can capture different people. The structure of each shallow CNN regressor is similar: four convolutional layers and two pond layers. R1 has a larger initial filter size of 9 x 9 that can capture the more dense regions of the scene. R2 and R3 with initial filter sizes of 7 x 7 and 5 x 5, respectively, capture the population at a lower scale. And respectively entering the image blocks with different densities distributed by the Switch classifier into CNN regressors with different sizes to obtain feature maps with different scales.
S4, fusing the density map features of each image block extracted by the R1, R2 and R3 regressors with the features extracted by each regressor in a feature splicing mode to obtain a feature map combined with the features of the global density map;
the specific operation of fusing the obtained feature maps of the regressors is as follows: and (3) enabling the tensor of the input data to pass through a regressor to obtain a corresponding input tensor, keeping the row dimension unchanged, and splicing the column dimensions to obtain a global characteristic diagram.
And S5, processing the feature map combined with the global density features by using the mean pooling and deconvolution layers to obtain a final estimated density map.
S51, using the mean pooling as a pooling layer of the network, reserving more image feature information, and expressing the mean pooling by a formula:
wherein v ismRepresenting the mth pixel of the T pixels in the sliding window extracted from the image, m representing the spatial orientation of the element in the sliding window, and pooling step using a defined spatial pooling operator F to pool vmMapping to corresponding statistical values;
s52, enhancing the feature map combined with the global density features through the two convolution layers to generate features of the feature map, and restoring the resolution of the feature map and partial detail features lost in the down-sampling process by using the two deconvolution layers to obtain a final estimated feature map.
And S6, performing convolution processing with the final estimated feature map and the convolution kernel size of 1 x 1, mapping the high-resolution feature map into a density map, and integrating the density map to obtain the number of people in the target image.
The Switch-CNN network completes the training of the Switch-CNN through pre-training, differential training, Switch training and integrated training. Pre-training gives the loss function definition and parameter updates with SGD gradient descent. The difference divides the training image block into three classes, corresponding to the training data needed by the three CNN networks, and then finely adjusts each CNN network by using the corresponding data set. The Switch training trains image blocks under different labels to generate classifiers. The integrated training is used for fine tuning the parameters of the image block classifier and the CNN regressor through the switching training of switch and regressors.
The pre-training loss function is defined as:
where N is the number of training samples,
representing an image xiThe ground truth density map of (a), theta represents a parameter of the input image,
representing the output of the CNN regressor with the parameters Θ of the input image xi. L is2The loss function is used as the count error between the regression estimate count and the true count.
Example 2
The present disclosure provides a crowd counting system based on switching convolutional network, including:
the classification module is used for partitioning a target image, inputting the image block into the switching convolutional neural network, and classifying the image block according to the density through the classifier;
the feature splicing module is used for extracting the features of the density map of the classified image blocks through the regressor, and obtaining a feature map combined with the features of the global density map through feature splicing of the obtained features of the density map;
and the calculation module is used for processing the feature map combined with the global density features through a mean pooling layer and a deconvolution layer to obtain a target estimated density map, and integrating the target estimated density map to obtain the number of people in the target image.
Example 3
The present disclosure provides an electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the steps of the switched convolutional network based crowd counting method when executing the program.
Example 4
The present disclosure provides a computer readable storage medium having stored thereon a computer program, which when executed by a processor, performs the steps of a switched convolutional network based crowd counting method as described.
The above is merely a preferred embodiment of the present disclosure and is not intended to limit the present disclosure, which may be variously modified and varied by those skilled in the art. Any modification, equivalent replacement, improvement and the like made within the spirit and principle of the present disclosure should be included in the protection scope of the present disclosure.
Although the present disclosure has been described with reference to specific embodiments, it should be understood that the scope of the present disclosure is not limited thereto, and those skilled in the art will appreciate that various modifications and changes can be made without departing from the spirit and scope of the present disclosure.
Claims (10)
1. A crowd counting method based on a switched convolutional network is characterized by comprising the following steps:
partitioning a target image, inputting image blocks into a switching convolutional neural network, and classifying the image blocks according to density through a classifier;
performing feature extraction on the classified image blocks through a regressor, and performing feature splicing on the obtained density map features to obtain a feature map combined with global density map features;
and processing the feature map combined with the global density features through a mean pooling layer and a deconvolution layer to obtain a target estimated density map, and integrating the target estimated density map to obtain the number of people in the target image.
2. The population counting method based on switched convolutional network of claim 1,
the switching convolutional neural network comprises a plurality of CNN regressors with different structures and a classifier for selecting an optimal regressor;
the classifier is a VGG 16-based three-class classifier, a full connection layer in a VGG16 is removed, a global average pool on a Conv 5 feature is used for removing spatial information and aggregation discriminant features, the full connection layer and the three-level Softmax classifier behind the global average pool correspond to a CNN regression, the classified image blocks with different densities are transmitted to the corresponding regressors, and feature maps with different scales are obtained.
3. The population counting method based on switched convolutional network of claim 2,
the classifier of the Switch-CNN completes training through pre-training, differential training, Switch training and integrated training; in the pre-training, a loss function and SGD gradient descent are adopted for parameter updating;
the pre-training loss function is defined as:
where N is the number of training samples,
representing an image xiGround truthPhase density map, Θ represents the parameters of the input image,
output of the CNN regressor with the parameters theta of the input image xi, L2The loss function is used as the count error between the regression estimate count and the true count.
4. The population counting method based on switched convolutional network of claim 3,
the differential training classifies the training image blocks, each class corresponds to a CNN (convolutional neural network) regressor, and each CNN regressor is subjected to fine tuning by using a corresponding data set;
the Switch training trains image blocks under different labels to generate classifiers;
the integrated training is used for finely adjusting parameters of the image block classifier and the CNN regressor through switching training of switch and regressors.
5. The population counting method based on switched convolutional network of claim 1,
the feature splicing specifically includes that the tensor of the target image block passes through a regressor to obtain a corresponding input tensor, the row dimension is kept unchanged, and the column dimension is spliced to obtain a feature map combining the features of the global density map.
6. The population counting method based on switched convolutional network of claim 1,
the mean pooling equation represents:
wherein v ismRepresenting the mth pixel of the T pixels in the sliding window extracted from the image, m representing the spatial orientation of the element in the sliding window, and pooling step using a defined spatial pooling operator FvmMapped to corresponding statistical values.
7. The switched convolutional network-based crowd counting method of claim 6,
and enhancing the feature map combined with the global density features through the two convolution layers to generate the features of the feature map, restoring the resolution of the feature map by using the two deconvolution layers to obtain a target estimation feature map, and mapping the target estimation feature map to obtain a target estimation density map.
8. A switched convolutional network based crowd counting system, comprising:
the classification module is used for partitioning a target image, inputting the image block into the switching convolutional neural network, and classifying the image block according to the density through the classifier;
the feature splicing module is used for extracting the features of the density map of the classified image blocks through the regressor, and obtaining a feature map combined with the features of the global density map through feature splicing of the obtained features of the density map;
and the calculation module is used for processing the feature map combined with the global density features through a mean pooling layer and a deconvolution layer to obtain a target estimated density map, and integrating the target estimated density map to obtain the number of people in the target image.
9. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the steps of the method according to any of the claims 1-7 are implemented when the processor executes the program.
10. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the steps of the method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911142695.6A CN110889360A (en) | 2019-11-20 | 2019-11-20 | Crowd counting method and system based on switching convolutional network |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911142695.6A CN110889360A (en) | 2019-11-20 | 2019-11-20 | Crowd counting method and system based on switching convolutional network |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110889360A true CN110889360A (en) | 2020-03-17 |
Family
ID=69748059
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911142695.6A Pending CN110889360A (en) | 2019-11-20 | 2019-11-20 | Crowd counting method and system based on switching convolutional network |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110889360A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112541891A (en) * | 2020-12-08 | 2021-03-23 | 山东师范大学 | Crowd counting method and system based on void convolution high-resolution network |
| CN113409246A (en) * | 2021-04-14 | 2021-09-17 | 宁波海棠信息技术有限公司 | Method and system for counting and positioning reinforcing steel bar heads |
| CN113553921A (en) * | 2021-07-02 | 2021-10-26 | 兰州交通大学 | Convolutional neural network-based subway carriage congestion degree identification method |
| CN113989229A (en) * | 2021-10-28 | 2022-01-28 | 杭州图谱光电科技有限公司 | Cell counting method based on random forest and convolutional neural network |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109271960A (en) * | 2018-10-08 | 2019-01-25 | 燕山大学 | A kind of demographic method based on convolutional neural networks |
| CN109858461A (en) * | 2019-02-21 | 2019-06-07 | 苏州大学 | A kind of method, apparatus, equipment and storage medium that dense population counts |
| CN109948593A (en) * | 2019-04-04 | 2019-06-28 | 浙江工业大学 | Based on the MCNN people counting method for combining global density feature |
-
2019
- 2019-11-20 CN CN201911142695.6A patent/CN110889360A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109271960A (en) * | 2018-10-08 | 2019-01-25 | 燕山大学 | A kind of demographic method based on convolutional neural networks |
| CN109858461A (en) * | 2019-02-21 | 2019-06-07 | 苏州大学 | A kind of method, apparatus, equipment and storage medium that dense population counts |
| CN109948593A (en) * | 2019-04-04 | 2019-06-28 | 浙江工业大学 | Based on the MCNN people counting method for combining global density feature |
Non-Patent Citations (1)
| Title |
|---|
| DEEPAK BABU SAM ET AL.: "Switching Convolutional Neural Network for Crowd Counting", 《IEEE XPLORE》 * |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112541891A (en) * | 2020-12-08 | 2021-03-23 | 山东师范大学 | Crowd counting method and system based on void convolution high-resolution network |
| CN113409246A (en) * | 2021-04-14 | 2021-09-17 | 宁波海棠信息技术有限公司 | Method and system for counting and positioning reinforcing steel bar heads |
| CN113553921A (en) * | 2021-07-02 | 2021-10-26 | 兰州交通大学 | Convolutional neural network-based subway carriage congestion degree identification method |
| CN113553921B (en) * | 2021-07-02 | 2022-06-10 | 兰州交通大学 | Convolutional neural network-based subway carriage congestion degree identification method |
| CN113989229A (en) * | 2021-10-28 | 2022-01-28 | 杭州图谱光电科技有限公司 | Cell counting method based on random forest and convolutional neural network |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Bautista et al. | Convolutional neural network for vehicle detection in low resolution traffic videos | |
| CN108304798B (en) | Street level order event video detection method based on deep learning and motion consistency | |
| CN110889360A (en) | Crowd counting method and system based on switching convolutional network | |
| CN104134068B (en) | Monitoring vehicle characteristics based on sparse coding represent and sorting technique | |
| CN108734719A (en) | Background automatic division method before a kind of lepidopterous insects image based on full convolutional neural networks | |
| CN102542571B (en) | Moving target detecting method and device | |
| CN110310241A (en) | A kind of more air light value traffic image defogging methods of fusion depth areas segmentation | |
| CN107590427B (en) | Method for detecting abnormal events of surveillance video based on space-time interest point noise reduction | |
| CN110929593A (en) | Real-time significance pedestrian detection method based on detail distinguishing and distinguishing | |
| CN109886159B (en) | Face detection method under non-limited condition | |
| DE102019123756A1 (en) | Neural network for performing semantic segmentation of an input image | |
| CN111914698A (en) | Method and system for segmenting human body in image, electronic device and storage medium | |
| Nguyen et al. | Hybrid deep learning-Gaussian process network for pedestrian lane detection in unstructured scenes | |
| Grigorev et al. | Depth estimation from single monocular images using deep hybrid network | |
| Zang et al. | Traffic lane detection using fully convolutional neural network | |
| Liang et al. | Cross-scene foreground segmentation with supervised and unsupervised model communication | |
| Nguyen et al. | Single-image crowd counting: a comparative survey on deep learning-based approaches | |
| CN114743126A (en) | Lane line sign segmentation method based on graph attention machine mechanism network | |
| Guo et al. | A novel transformer-based network with attention mechanism for automatic pavement crack detection | |
| Liu | Learning full-reference quality-guided discriminative gradient cues for lane detection based on neural networks | |
| CN112686242B (en) | Fine-grained image classification method based on multilayer focusing attention network | |
| CN113177956A (en) | Semantic segmentation method for unmanned aerial vehicle remote sensing image | |
| Sanyal et al. | Traffic sign recognition on Indian database using wavelet descriptors and convolutional neural network ensemble | |
| Molina-Cabello et al. | Vehicle type detection by convolutional neural networks | |
| Wang et al. | Variant-depth neural networks for deblurring traffic images in intelligent transportation systems |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |