CN114187613A - Crowd counting method based on multi-branch deep neural network and mixed density map - Google Patents
Crowd counting method based on multi-branch deep neural network and mixed density map Download PDFInfo
- Publication number
- CN114187613A CN114187613A CN202111514708.5A CN202111514708A CN114187613A CN 114187613 A CN114187613 A CN 114187613A CN 202111514708 A CN202111514708 A CN 202111514708A CN 114187613 A CN114187613 A CN 114187613A
- Authority
- CN
- China
- Prior art keywords
- crowd
- images
- image
- density map
- counting
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Image Analysis (AREA)
Abstract
A crowd counting method based on a multi-branch deep neural network comprises the following steps: s1, marking crowd images, generating corresponding density maps from the crowd images, and training a crowd counting model; s2, randomly cutting out 9 sub-images with the size of 240 × 240 from the image to be recognized according to different resolutions; s3, performing image enhancement transformation on the subimages of the crowd images for training to obtain enhanced subimages; s4, sending the enhanced sub-images obtained in the step S3 into a multi-branch depth convolution network (MCNN) to identify head images with different sizes; and S5, stacking the results obtained in the step S4, performing 1 x 1 convolutional layer processing to obtain a corresponding density map, and integrating the density map to obtain the estimated number of people. The method can effectively cope with the conditions of overhigh crowd density, serious shielding and the like, and meanwhile, the method can effectively avoid the error of crowd counting under the condition of too sparse crowd by adjusting the density map generation mode according to the crowd scale.
Description
Technical Field
The invention relates to the fields of artificial intelligence, machine vision and ultra-high definition display, in particular to a crowd counting method based on a multi-branch deep neural network and a mixed density map.
Background
The frequent trampling events in large activities at home and abroad have caused serious casualties, and for example, the trampling event in overseas beaches in 2015 has reached the level of serious casualty accidents specified in China. Therefore, research aiming at the crowd counting problem is getting hot, and if the crowd density of the current scene can be accurately estimated and corresponding security measures are arranged, the occurrence of the events can be effectively reduced or avoided. People counting specifically refers to estimating the number of people in a certain area by using a computer vision technology. Traditional population counting methods mainly fall into two broad categories: 1. a detection-based method of counting persons in a crowd scene by using a detector to detect their heads or shoulders; 2. although the regression-based method can obtain reasonable results in partial scenes, the detection-based method is not good in effect when facing complex scenes such as crowding and blocking, and the problems can be well alleviated through regression counting. Si-yue Yu and Jian Pu in 8 months in 2020 propose a population counting method combining a density map and foreground characteristics, a full convolution network with two tasks is firstly constructed, multi-scale spatial context information is extracted to learn the density map, and through actual detection, a regression-based method is superior to a detection-based method.
Meanwhile, with the continuous progress of ultra-high-definition video acquisition devices, the image acquisition technology has leap-type development.
The main problems of the background art are: the crowded crowd often causes accidents such as treading and serious casualty accidents. The feasible preventive measures against these malignant events are people flow statistics such as people counting. However, the traditional people counting means has various problems which are difficult to solve, the people counting mainly comprises a detection-based method and a regression-based method, the detection-based method mainly detects the head or shoulder contour of a person in a people scene by using a detector to count the person, and the detection effect based on the detector is often poor due to the limited visual field range and image resolution of a camera and the existence of the target occlusion condition; the regression-based method cannot achieve a good effect even under the conditions of crowding and large size difference between the foreground head and the background head. The difficulty in solving the above problems and defects is: the problems are difficult to solve by using a traditional method, and the only solution is to adopt video image acquisition equipment with higher resolution and wider visual angle aiming at the limitations of the use range and the image resolution; aiming at the problem of occlusion and the problem of the size of the background foreground head, the traditional solution is difficult to process densely overlapped crowds, and once the crowd scale rises, the existing method can generate larger errors.
The significance of solving the problems and the defects is as follows: the crowd counting method has the advantages that the problems of narrow visual field range, poor resolution, shielding, different sizes of background foreground heads and the like can be solved properly, the number of people can be estimated more accurately, more real-time and accurate crowd flow data can be provided for supervision and security mechanisms, and the occurrence of events such as large-scale aggregation, trampling and the like can be prevented.
Disclosure of Invention
The invention provides a crowd counting method based on a multi-branch deep neural network and a mixed density map, and particularly relates to a crowd counting method combining the multi-branch deep neural network of an ultra-high-definition video acquisition device with the mixed density map. Due to the adoption of the ultra-high-definition image and the multi-branch depth convolution network, the conditions of overhigh crowd density, serious shielding and the like can be effectively dealt with, and meanwhile, the density image generation mode is adjusted according to the crowd scale, so that the error caused by crowd counting under the condition of excessively sparse crowd can be effectively avoided.
The technical scheme of the invention is as follows:
a crowd counting method based on a multi-branch deep neural network comprises the following steps: s1, marking crowd images, generating corresponding density maps from the crowd images, and training a crowd counting model; s2, randomly cutting out 9 sub-images with the size of 240 × 240 from the image to be recognized according to different resolutions; s3, performing image enhancement transformation on the subimages of the crowd images for training to obtain enhanced subimages; s4, sending the enhanced sub-images obtained in the step S3 into a multi-branch depth convolution network (MCNN) to identify head images with different sizes; and S5, stacking the results obtained in the step S4, performing 1 x 1 convolutional layer processing to obtain a corresponding density map, and integrating the density map to obtain the estimated number of people.
Preferably, in the population counting method based on the multi-branch deep neural network, in step S1, the image data of the population needs to be labeled to mark the position of the person in the image, then the image of the population is converted into a corresponding density map by convolution of an adaptive gaussian kernel function or a fixed gaussian kernel function, and the image data of the population and the generated density map are put into a population counting model to train the population counting model.
Preferably, in the population counting method based on the multi-branch deep neural network, in step S2, in the training stage, an image with a resolution of 1080p is given, and the image to be recognized is randomly cropped into 9 sub-images with a size of 240 × 240 according to the resolution.
Preferably, in the population counting method based on the multi-branch deep neural network, in step S3, the image enhancement transformation includes rotation or contrast adjustment.
Preferably, in the population counting method based on the multi-branch depth neural network, in step S4, the multi-branch depth convolutional network uses three branches to identify head images with different sizes, and finally, the features of the original images extracted by the multi-branch depth convolutional network are stacked to obtain a merged feature map, and the merged feature map is mapped to a density map through a convolution of 1 × 1, and the difference between the predicted result density map and the mark value is measured by using the euclidean distance.
Preferably, in the population counting method based on the multi-branch deep neural network, the multi-branch deep convolutional network comprises three branches: the convolution kernels used by the large scale convolution kernel, the medium scale convolution kernel and the small scale convolution kernel are respectively (9 × 9), (7 × 7) and (7 × 7); (7 × 7), (5 × 5), and (5 × 5), (3 × 3).
According to the technical scheme of the invention, the beneficial effects are as follows:
compared with the prior art, the method provided by the invention has the following improvements:
1.) a multi-branch deep learning network is provided, an ImageNet pre-training model is introduced, and recognition of different sizes of human figures in the image is formed;
2.) the traditional method cannot cope with the crowd counting (such as serious occlusion, crowd size larger than 200 people and the like) in a complex scene, and the crowd counting method based on deep learning can expand the number of detected crowds to more than 1000 people;
3.) according to the size of the crowd size, a new density map generation mode is provided, and when the crowd density is higher, the average distance between the head of a person and the 4 surrounding heads of the person can be adopted to represent the actual size of the head of the person due to more overlapping and smaller distance between the persons; when the crowd density is low, the distance between people is large, the average distance is adopted to represent the size of the head, so that a large error is caused, at the moment, the head is regarded as a round point with uniform size by adopting a fixed Gaussian kernel function, the crowd technology is converted into a key point counting problem, and the computational power consumption is reduced;
4.) the traditional crowd counting algorithm based on deep learning has higher sensitivity to the input image, an 8K image ultrahigh-definition acquisition device is adopted, so that the resolution of the existing image is greatly improved, the detail precision is superior to that of the traditional camera/video camera, and meanwhile, the adaptive image pyramid technology is matched to select the proper image resolution according to the crowd density, thereby balancing the system efficiency and the computational power consumption.
When the method is used for counting the crowd, a multi-branch deep convolutional network is utilized, and a 1080p/4K/8K image is used as an identification object, so that the defect that the traditional crowd counting algorithm is easy to occur is effectively overcome.
For a better understanding and appreciation of the concepts, principles of operation, and effects of the invention, reference will now be made in detail to the following examples, taken in conjunction with the accompanying drawings, in which:
drawings
In order to more clearly illustrate the detailed description of the invention or the technical solutions in the prior art, the drawings that are needed in the detailed description of the invention or the prior art will be briefly described below.
FIG. 1 is a flow chart of a population counting method based on a multi-branch deep neural network according to the present invention;
FIG. 2 is a test image and corresponding density map;
FIG. 3 is a schematic diagram of the coordinates of a crowd;
FIG. 4 is a graph of density for different Gaussian kernel functions;
FIG. 5 is a block diagram of a multi-branch deep convolutional network; and
FIG. 6 is a comparison graph of image details at different resolutions.
Detailed Description
In order to make the objects, technical means and advantages of the present invention more apparent, the present invention will be described in detail with reference to the accompanying drawings and specific examples. These examples are merely illustrative and not restrictive of the invention.
The invention aims to provide a crowd counting method based on a multi-branch deep neural network and a mixed density map, which is a crowd counting method of a multi-branch deep convolutional network combined with self-adaptive image pyramid optimization. In particular, the method learns a mapping from crowd image features to an image density map using deep learning techniques. Firstly, marking training data, convolving the marked image by a self-adaptive Gaussian kernel function or a fixed Gaussian kernel function according to the dense situation of people groups in the image to obtain a corresponding density map, then putting the training data and the generated density map into a network for training to obtain a trained model, then reasoning out the density map corresponding to the image in a test data set according to the model, and finally integrating the density map to obtain the estimated people number.
Fig. 1 is a flowchart of a population counting method based on a multi-branch deep neural network and a mixed density map, which specifically includes the following steps from beginning to end:
s1, in a training stage, marking crowd images, generating corresponding density maps from the crowd images, and training a crowd counting model. Specifically, firstly, people image data (i.e., training data) needs to be labeled, positions of people in the images are marked, then, the people images are converted into corresponding density maps through convolution of a self-adaptive gaussian kernel function or a fixed gaussian kernel function, and the people image data and the generated density maps are put into a people counting model to train the people counting model.
Population counting in an open scene requires the conversion of a population image (i.e., a test image) into a corresponding density map, as shown in fig. 2. Fig. 3 is a schematic diagram of the coordinates of the human population, and the left part "1" of fig. 3 indicates the position of the center of the human head, i.e., (3,6), (12,9), (17,15) where a human exists. The right part is a position schematic diagram after the adaptive Gaussian processing, and the area of 3 x 3 in the diagram represents the actual range of the human.
In the training phase, the sum of the population counts in the graph is represented by H (x), and the function delta (x-x)i) Is represented at a pixel point xiProbability of having a human head, function in (x ═ x)i) The place is 1, and all other places are 0. Then it is determined that,if a figure has N heads, it can be described as a function of equation (1).
Where H (x) is a discrete function, which is converted to a continuous density function using a Gaussian kernel Gσ(x) And obtaining a probability density function F (x) as shown in the formula (2).
F(x)＝H(x)×Gσ(x)………(2)
However, the density function F (x) described in equation (2) is assumed to be xiAre independent samples of the planar image, and in fact each xiThe samples are crowd density samples in a real scene, and due to perspective distortion, pixel points related to different crowd samples correspond to areas with different sizes in a two-dimensional plane image. Considering that the influence of distortion on the counting is large, and most of the data used for training is data in the training set, it is difficult to obtain the true distortion magnitude corresponding to the distortion magnitude, and considering that in a crowded scene, the size of the head is usually related to the central distance between two adjacent persons. Thus for an arbitrary head center xiThe distance between the human head and the surrounding human head is recorded as
Average distance
Therefore, considering the adaptive pitch, equation (2) can be converted into equation (3).
Wherein the content of the first and second substances,
the optimum value of β is 0.3, which is the standard deviation of the gaussian kernel function, and is inversely derived by repeated tests in combination with experience. However, in the case of sparsely populated people, they cannot be reused
As an indication of the size of the human head, a Gaussian kernel function σ of fixed size is therefore usediAnd 3 (obtained according to The training results on The TRANCOS and The UCSD dataset), and through practical experiments, a fixed gaussian kernel function is selected, so that The estimation error can be effectively reduced in a sparse crowd scene. The comparison of the density map generated by the fixed Gaussian kernel function and the adaptive Gaussian kernel function in the sparse scene is shown in FIG. 4, wherein the left map is a photograph in the real scene, the middle map is the density map generated by the adaptive Gaussian kernel function, and the right map is the density map generated by the fixed Gaussian kernel function.
S2, randomly cutting out 9 sub-images of 240 x 40 size from the image to be recognized (namely the original image) according to different resolutions. In the training stage, given an image with a resolution of 1080p (4k/8k), taking an image with a resolution of 1080p as an example, the image to be recognized (i.e. the original image) is randomly cropped into 9 sub-images with a size of 240 × 240 according to different resolutions.
And S3, in the training stage, carrying out image enhancement transformation, such as random changes of rotation, contrast adjustment and the like, on the sub-images of the used training images (namely the crowd images in the S1) to obtain enhanced sub-images so as to improve the generalization and robustness of the algorithm model.
And S4, sending the image obtained in the step S3 (namely the enhanced sub-image) into a multi-branch depth convolution network (MCNN) to identify the human head images with different sizes. At this time, the ImageNet pre-training model needs to be loaded in advance, so that the problem of poor training result caused by random initialization is solved. The structure of the MCNN network is shown in fig. 5. Fig. 5 shows the relationship between the convolutional layer, the pooling layer, the input image, the fused feature image, and the generated density map. The MCNN network consists of three branches, a first column denoted as L column (Large, Large scale convolution kernel), a second column denoted as M column (Medium, Medium scale convolution kernel) and a third column denoted as S column (Small, Small scale convolution kernel). Convolution kernels used for the L, M, and S columns are (9 × 9), (7 × 7), (7 × 7), (7 × 7); (7 × 7), (5 × 5), (5 × 5), (5 × 5) and (5 × 5), (33), (3 × 3), (3 × 3); because the human heads in the crowd images are different in size, the convolution kernel with the uniform size is designed to be not beneficial to identifying the small or overlarge human head images, and therefore the three branches are adopted to identify the human head images with different sizes respectively. And finally, stacking the features of the original image extracted by the convolution network (namely the output of the convolution network) to obtain a combined feature map, and mapping the combined feature map to a density map through (1 x 1) convolution. At this time, the difference between the predicted result density map and the marker value is measured by using the euclidean distance, as shown in the following formula.
And S5, stacking the characteristics obtained in the step S4 (namely the result obtained in the step S4), processing the stacked characteristics by a (1 x 1) convolutional layer to obtain a corresponding density map, and integrating the density map to obtain the estimated number of people.
According to the invention, an ultrahigh-definition image acquisition device is adopted to capture an image, the acquired image has default resolution of 8k (7680 × 4320 pixels), however, when the number of people is small and no obvious shielding condition exists, the system response time is greatly improved by using the image with the resolution of 8k, and meanwhile, the consumption of computing power of the cloud is increased. In fig. 6, Full HD indicates Full high definition, resolution is 1080p, Ultra HD is Ultra high definition, resolution is 2160p, Full Ultra HD is Full Ultra high definition, and resolution is 4320 p. In order to optimize the overall system performance, an image pyramid strategy is adopted in combination with the intelligent large-screen actual working condition to perform downsampling on the original 8K image, and images of 2160p and 1080p are generated respectively. When the system detects that the number of crowds is not large and the pedestrian flow is sparse, 1080p images are adopted for identification, and the images with higher resolution are selected according to actual conditions along with the increase of the pedestrian flow so as to balance the operation consumption and the energy consumption of the whole system.
The embodiment of the invention trains and tests on a population counting data set ACC (awesome-crowd-counting) and Free-view, and evaluates the experimental result by using the currently accepted evaluation standard mAP (mean Average precision). The method provided by the invention achieves the leading detection precision at present.
The foregoing description is of the preferred embodiment of the concepts and principles of operation in accordance with the invention. The above-described embodiments should not be construed as limiting the scope of the claims, and other embodiments and combinations of implementations according to the inventive concept are within the scope of the invention.
Claims (6)
1. A crowd counting method based on a multi-branch deep neural network is characterized by comprising the following steps:
s1, marking crowd images, generating corresponding density maps from the crowd images, and training a crowd counting model;
s2, randomly cutting out 9 sub-images with the size of 240 × 240 from the image to be recognized according to different resolutions;
s3, performing image enhancement transformation on the subimages of the crowd images for training to obtain enhanced subimages;
s4, sending the enhanced sub-images obtained in the step S3 into a multi-branch depth convolution network (MCNN) to identify human head images with different sizes; and
and S5, stacking the results obtained in the step S4, performing 1 × 1 convolutional layer processing to obtain a corresponding density map, and integrating the density map to obtain the estimated number of people.
2. The method for counting people according to claim 1, wherein in step S1, the crowd image data is labeled to mark the position of a person in the image, and then the crowd image is convolved into the corresponding density map by an adaptive gaussian kernel function or a fixed gaussian kernel function, and the crowd image data and the generated density map are put into the crowd counting model to train the crowd counting model.
3. The method for counting people based on multi-branch deep neural network as claimed in claim 1, wherein in step S2, given an image with a resolution of 1080p, the image to be recognized is randomly cropped into 9 sub-images with a size of 240 × 240 according to different resolutions.
4. The method for people population counting based on multi-branch deep neural network of claim 1, wherein in step S3, the image enhancement transformation comprises rotation or contrast adjustment.
5. The method of claim 1, wherein in step S4, the multi-branch depth convolutional network uses three branches to identify head images with different sizes, and finally stacks features of the original images extracted by the multi-branch depth convolutional network to obtain a merged feature map, and then maps the merged feature map to a density map by a convolution of 1 × 1, and uses euclidean distance to measure the difference between the predicted density map and the labeled value.
6. The method of claim 1, wherein the multi-branch deep convolutional network comprises three branches: a large scale convolution kernel, a medium scale convolution kernel and a small scale convolution kernel, wherein the convolution kernels used by the large scale convolution kernel, the medium scale convolution kernel and the small scale convolution kernel are (9 × 9), (7 × 7) and (7 × 7), respectively; (7 × 7), (5 × 5), and (5 × 5), (3 × 3).
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111514708.5A CN114187613A (en) | 2021-12-13 | 2021-12-13 | Crowd counting method based on multi-branch deep neural network and mixed density map |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111514708.5A CN114187613A (en) | 2021-12-13 | 2021-12-13 | Crowd counting method based on multi-branch deep neural network and mixed density map |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114187613A true CN114187613A (en) | 2022-03-15 |
Family
ID=80604614
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111514708.5A Pending CN114187613A (en) | 2021-12-13 | 2021-12-13 | Crowd counting method based on multi-branch deep neural network and mixed density map |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114187613A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116311084A (en) * | 2023-05-22 | 2023-06-23 | 青岛海信网络科技股份有限公司 | Crowd gathering detection method and video monitoring equipment |
-
2021
- 2021-12-13 CN CN202111514708.5A patent/CN114187613A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116311084A (en) * | 2023-05-22 | 2023-06-23 | 青岛海信网络科技股份有限公司 | Crowd gathering detection method and video monitoring equipment |
| CN116311084B (en) * | 2023-05-22 | 2024-02-23 | 青岛海信网络科技股份有限公司 | Crowd gathering detection method and video monitoring equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108615027B (en) | Method for counting video crowd based on long-term and short-term memory-weighted neural network | |
| CN111695522B (en) | In-plane rotation invariant face detection method and device and storage medium | |
| CN104850850B (en) | A kind of binocular stereo vision image characteristic extracting method of combination shape and color | |
| CN111563447A (en) | Crowd density analysis and detection positioning method based on density map | |
| CN104835182A (en) | Method for realizing dynamic object real-time tracking by using camera | |
| Li et al. | Image Matching Algorithm based on Feature-point and DAISY Descriptor. | |
| Wang et al. | Ship detection based on fused features and rebuilt YOLOv3 networks in optical remote-sensing images | |
| CN103353941B (en) | Natural marker registration method based on viewpoint classification | |
| CN104050685A (en) | Moving target detection method based on particle filtering visual attention model | |
| CN110197185B (en) | Method and system for monitoring space under bridge based on scale invariant feature transform algorithm | |
| CN113743322A (en) | Offshore ship detection method based on improved YOLOv3 algorithm | |
| CN114332921A (en) | Pedestrian detection method based on improved clustering algorithm for Faster R-CNN network | |
| CN114187613A (en) | Crowd counting method based on multi-branch deep neural network and mixed density map | |
| CN113095316B (en) | Image rotation target detection method based on multilevel fusion and angular point offset | |
| CN106504211A (en) | Based on the low-light-level imaging method for improving SURF characteristic matchings | |
| CN110910497B (en) | Method and system for realizing augmented reality map | |
| CN113052110A (en) | Three-dimensional interest point extraction method based on multi-view projection and deep learning | |
| CN112001448A (en) | Method for detecting small objects with regular shapes | |
| Wang et al. | LBP-based edge detection method for depth images with low resolutions | |
| Kumar et al. | Intelligent parking vehicle identification and classification system | |
| CN116453048A (en) | Crowd counting method combined with learning attention mechanism | |
| Li et al. | A new algorithm of vehicle license plate location based on convolutional neural network | |
| CN110363101A (en) | A kind of flowers recognition methods based on CNN Fusion Features frame | |
| CN115170832A (en) | Weak texture surface microstructure feature extraction method based on visible light single image | |
| Li et al. | Research on ROI algorithm of ship image based on improved YOLO |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |