KR20220056399A - Crowded scenes image real-time analysis apparatus using dilated convolutional neural network and method thereof - Google Patents

Abstract

The present invention relates to an apparatus and method for analyzing a crowded scene image in real time and, more specifically, to an apparatus and method for analyzing a crowded scene image in real time using a dilated convolutional neural network (CNN), which train a crowd density distribution map model to which an end-to-end dilated CNN is applied, with training datasets including crowd density distribution map information for people included in a crowd scene image, generate a high-quality real-time crowd density distribution map by applying a crowd scene image to be analyzed to the crowd density distribution map model, and count the number of people in the crowd scene image on the basis of the generated real-time crowd density distribution map. According to the present invention, the apparatus comprises a dataset generation unit, a dataset preprocessing unit, a training unit, a density distribution map generation unit, and a count unit.

Description

Crowded scenes image real-time analysis apparatus using dilated convolutional neural network and method thereof

The present invention relates to an apparatus and method for real-time analysis of crowd scene images, and more particularly, end-to-end extended synthesis of a training dataset including crowd density distribution map information for people included in the crowd scene image. Real-time generated after learning a crowd density distribution map model to which a convolutional neural network (CNN) is applied, and applying a crowd scene image requested for analysis to the crowd density distribution map model to generate a high-quality real-time crowd density distribution map A crowd scene image real-time analysis apparatus and method using an extended convolutional neural network that counts the number of people in a crowd scene image based on a crowd density distribution map.

In general, Closed Circuit Television (CCTV) is being installed on roads, squares, and major spaces in buildings to protect citizens safely and to track and monitor offenders.

In addition, various image analysis devices for collecting, storing, and analyzing images obtained through such CCTV have been developed and applied.

A general image analysis apparatus is used to detect a person from images collected through CCTV, analyze and estimate the detected person's behavior, or count the number of people, that is, a crowd.

In particular, recently, for security, an image analysis device for calculating or estimating the number of people, that is, the number of people, from still images and moving images (hereinafter referred to as "crowd scene image") in which a large number of people, that is, the crowd is gathered, has been developed and is being applied

A conventional image analysis apparatus detects the entire body of a person or a specific part of the body (usually: a face) in an image obtained for counting the number of crowds by a feature pattern for the corresponding body part, direct detection for counting the number of people A density distribution map method that counts the number of crowds by generating a crowd density distribution map according to the method and crowd distribution is being applied.

The image analysis apparatus for counting the number of crowds by the direct detection method and the whole body detection has a problem in that it is impossible to detect the entire body of a person because a large number of people are entangled in the image.

The image analysis apparatus for counting the number of crowds by the direct detection method and body parts also has a problem in that the number of crowds cannot be accurately counted because a large number of people are entangled in the image and the corresponding parts are often not detected.

1 is a view showing an actual image and a basic density distribution map including a similar number of crowds in a general image analysis apparatus.

Referring to FIG. 1 , the image analysis apparatus to which the density distribution map method is applied is a first image (1) of each of (a) and (b) having a similar number of people as shown in FIGS. 1 (a) and (b). And it can be seen that the distributions of the first density distribution map 3 and the second density distribution map 4 corresponding to the second image 2 are not similar and completely different. That is, the image analysis apparatus to which the general density distribution map method is applied cannot accurately count the number of people in the crowd.

Therefore, the development of an image analysis apparatus capable of accurately counting the number of people in real time from a crowd scene image is required.

Republic of Korea Patent Registration No. 10-1467307 (2014.12.01. Announcement)

Therefore, it is an object of the present invention to provide a training dataset including crowd density distribution map information for people included in the crowd scene image to end-to-end extended convolutional neural network (CNN) applied crowd density. The number of people in the crowd scene image based on the generated real-time crowd density distribution map after learning the distribution map model, generating a high-quality real-time crowd density distribution map by applying the crowd scene image requested for analysis to the crowd density distribution map model An object of the present invention is to provide an apparatus and method for real-time analysis of crowd scene images using an extended convolutional neural network for counting.

Crowd scene image real-time analysis apparatus using an extended convolutional neural network according to the present invention for achieving the above object: Generating a dataset that generates a dataset including a plurality of crowd scene images annotated on the number of crowds wealth; a dataset pre-processing unit that generates a basic density distribution map for the crowd scene images of the dataset, and increases data including the crowd scene image and the basic density distribution map to generate a training dataset; The dataset preprocessor applies an end-to-end convolutional neural network including a separable convolutional layer for each depth and an extended convolutional layer to learn a crowd density distribution map for the number of heads in a crowd scene image. a learning unit for learning by applying the learning data set input from a density distribution map generator for generating and outputting a crowd density distribution map for a crowd scene image input in real time to which the learned crowd density distribution map model is applied; and a counting unit for receiving the crowd density distribution map and counting the number of crowds of the crowd scene image input in real time.

The dataset preprocessor may include: a basic density distribution map generator for generating and outputting a basic density distribution map by applying an adaptive geometry kernel expressed by the following equation to each of the crowd scene images of the dataset; and a data increasing unit for increasing the amount of data for the annotated crowd scene image.

[Equation]

Here, si denotes all target objects, δ denotes the basic density distribution map, di denotes the distance average of k nearest neighbors, and s denotes the position of a pixel in the input image.

The data increasing unit is characterized in that the data is increased by dividing the crowd scene image into a predetermined number of patches reduced by a predetermined ratio with respect to the size of the crowd scene.

The crowd density distribution map model includes: a front-end processing unit for extracting and outputting 2D features from crowd scene images having different resolutions of a preprocessed training dataset; and a back-end processing unit that generates and outputs a crowd density distribution map by performing extended convolution of 2D features output from the front-end processing unit.

The front-end processing unit includes three separable convolutional layers by depth and filters of size 3 * 3, but sequentially has 16, 32, 64, 128, and 256 different numbers of filters, 5 depth-wise separable convolutional layer modules that apply ReLU as an activation function; a max pooling layer having a size of 2*2, each configured after the separable convolutional layer module by depth and after the separable convolutional layer module by depth among the separable convolutional layer modules by depth and after the fourth and fifth separable convolutional layer modules by depth; and a standard convolutional layer module including two standard convolutional layers and 512 filters having a size of 3*3, and applying ReLU as an activation function.

The back-end processing unit includes: three first extended convolutional layer modules including two extended convolution layers and a filter having a size of 3*3, sequentially having 512, 256, and 128 filters; a second extended convolutional layer module including one extended convolutional layer and 64 filters having a size of 3*3; and a standard convolutional layer module having one standard convolutional layer and one filter having a size of 1*1, and applying ReLU as an activation function.

The extension convolution layer is characterized in that it extends the accommodation field using extra kernels.

A method for real-time analysis of crowd scene images using an extended convolutional neural network according to the present invention for achieving the above object is: A dataset generating unit generates a dataset including a plurality of crowd scene images annotated on the number of crowds data set creation process; A dataset preprocessing process in which a dataset preprocessing unit generates a basic density distribution map for the crowd scene images of the dataset, and increases data including the crowd scene image and the basic density distribution map to generate a training dataset ; The dataset in the crowd density distribution map model, where the learning unit learns the crowd density distribution map for the number of heads in the crowd scene image by applying an end-to-end convolutional neural network including a separable convolutional layer and extended convolutional layer by depth. a learning process of learning by applying the learning dataset input from the preprocessor; a density distribution map generation process in which a density distribution map generator generates and outputs a crowd density distribution map for a crowd scene image input in real time to which the learned crowd density distribution map model is applied; and a counting process of counting the number of crowds of the crowd scene image input in real time by a counting unit receiving the crowd density distribution map.

In the dataset preprocessing process, the basic density distribution map generator generates and outputs a basic density distribution map by applying an adaptive geometry kernel expressed by the following equation to each of the crowd scene images of the dataset. map creation step; and a data increasing step in which the data increasing unit increases the amount of data for the annotated crowd scene image.

[Equation]

Here, si denotes all target objects, δ denotes the basic density distribution map, di denotes the distance average of k nearest neighbors, and s denotes the position of a pixel in the input image.

In the data increasing step, the data increasing unit divides the crowd scene image into a certain number of patches reduced by a predetermined ratio with respect to the size of the crowd scene to increase the data.

The learning process includes: a front-end processing step of extracting and outputting 2D features from crowd scene images having different resolutions of the pre-processed training dataset by the front-end processing unit; and a back-end processing step in which a back-end processing unit generates and outputs a crowd density distribution map by performing extended convolution of 2D features output from the front-end processing unit.

The present invention blurs and normalizes the head of an annotated person included in the crowd scene image included in the obtained dataset, and then applies it to the adaptive geometry kernel to generate a basic density distribution map, and the generated basic density distribution map By adding to the dataset, it is possible to improve the quality of the dataset to improve the counting accuracy of the number of people included in the crowd scene image.

In addition, since the present invention increases data by overlapping and dividing the crowd scene image of the dataset and applying pseudo noise, it is possible to improve the counting accuracy of the number of people in the crowd scene image.

In addition, the present invention has the effect of reducing a large amount of computation in the convolution process and reducing the size of the model by applying a front end including a convolution layer separable by depth to the crowd density distribution map model.

In addition, the present invention minimizes the maximum and average pooling layers in the crowd density distribution map model to prevent loss of some special information of the map, and eliminates the need to use an inverse convolutional layer according to the minimization of the pooling layer, thereby reducing complexity and execution. It has the effect of saving time.

1 is a view showing an actual image and a basic density distribution map including a similar number of crowds in a general image analysis apparatus.
2 is a diagram showing the configuration of a crowd scene image real-time analysis apparatus using an extended convolutional neural network according to the present invention.
3 is a diagram conceptually illustrating each component of a crowd scene image real-time analysis apparatus using an extended convolutional neural network according to the present invention.
4 is a diagram for explaining the data increase concept of the dataset preprocessor of the apparatus for real-time analysis of crowd scene images using an extended convolutional neural network according to the present invention.
5 is a diagram showing the configuration of a standard 2D convolutional process layer for single and multiple filters according to the present invention.
6 is a diagram illustrating the configuration of a separable convolutional layer for each depth according to the present invention.
7 is a view showing the effect of expanding the receiving field according to the increase in the expansion rate according to the present invention.

Hereinafter, the configuration and operation of an apparatus for real-time analysis of crowd scene images using an extended convolutional neural network according to the present invention will be described with reference to the accompanying drawings, and a method for real-time analysis of crowd scene images in the apparatus will be described.

2 is a diagram showing the configuration of an apparatus for real-time analysis of crowd scene images using an extended convolutional neural network according to the present invention, and FIG. It is a diagram conceptually shown, and FIG. 4 is a diagram for explaining the data increase concept of the dataset preprocessor of the crowd scene image real-time analysis apparatus using the extended convolutional neural network according to the present invention, and FIG. It is a diagram showing the configuration of a standard 2D convolutional process layer for a filter, FIG. 6 is a diagram showing the configuration of a separable convolution layer for each depth according to the present invention, and FIG. It is a diagram showing the effect of expanding the receiving field. Hereinafter, it will be described with reference to FIGS. 2 to 7 .

First, referring to FIG. 2 , the apparatus for real-time analysis of crowd scene images using an extended convolutional neural network according to the present invention includes a dataset generator 100 , a crowd image acquisition unit 200 , a dataset preprocessor 300 , and a learning unit. 400 , a density distribution map generating unit 500 and a crowd counting unit 600 .

The dataset generating unit 100 collects crowd scene images, which are images in which a large number of overcrowded crowds are gathered through CCTV or uploaded to an online server for a certain period of time, and a person included in the collected crowd scene images Create a dataset containing crowd scene images including annotations for heads obtained from managers. The crowd scene image may be a still image or a frame-by-frame image of a moving picture.

In addition, the dataset generating unit 100 may be obtained by loading a dataset including a plurality of crowd scene images including annotations on human heads that are obtained in advance and stored in the storage means as shown in Table 1 below. will be.

In Table 1, the crowd scene images of ShanghaiTech part (A) and UCF_CC_50 are very crowded with large crowds, and ShanghaiTech part (B), UCSD and WorldExpo'10 are relatively sparsely populated and the surveillance view sample (crowd scenes) image) is shown.

In Table 1, T _n represents the entire sample, and M _n and M _x represent the minimum and maximum number of individuals in the image, respectively.

Used to represent the average number of people per image (A _v ), R is the image resolution, T _h is the full annotated head, ROI is the region of interest, and the presence or absence of the region of interest is indicated as Yes or No.

In order to improve the quality of the data included in the dataset, that is, the crowd scene image, the dataset generating unit 100 may perform a resizing process for each frame by applying bilinear interpolation or the like. For example, the dataset generator 100 may adjust the size of the overcrowded crowd video image for each frame to a resolution of 900*600.

When a dataset is generated or acquired, the dataset generator 100 provides the generated or acquired dataset to the dataset preprocessor 300 .

The crowd image acquisition unit 200 acquires a crowd scene image in real time with CCTV, a server, etc. or obtains a crowd scene image to be analyzed and provides it directly to the crowd density distribution map generation unit 300, or a dataset preprocessor 300 After performing data pre-processing in , it is transmitted to the density distribution map generator 300 .

The dataset preprocessor 300 includes a basic density distribution map generation module 310 and a data increase module 320 to generate new data derived from existing data and perform preprocessing to increase the existing data.

Specifically, the basic density distribution map generation module 310 applies a geometric kernel to each crowd scene image including the annotation on the human head of the dataset input from the dataset generation unit 100, as shown in 311 of FIG. Create a basic density distribution map for real photos (crowd scene images).

The basic density distribution map generation module 310 applies a Gaussian kernel to blur everyone's heads in the annotated crowd scene image and normalizes them to one, and takes into account the special distribution of all images, the basic density distribution map create

A geometry adaptive filter for applying the Gaussian kernel may be expressed as in Equation 1 below.

where s _i represents all target objects, δ is the crowd density distribution map, d _i is the distance average of k nearest neighbors, and S represents the position of a pixel in the input image.

A convolution process is applied to the parameter (δ(ss _i )) and a Gaussian filter with standard deviation (σ _i ) to generate a crowd density distribution map.

If there are many crowd scene images as in the Shanghai Tech part (A) and UCF_CC_50 of Table 1 above, a geometry adaptation kernel may be used.

And Shanghai Tech Part (B), UCSD, and WorldExpo'10 are datasets with crowd scene images with relatively small crowds, and a fixed kernel can be applied, and δ=15 can be applied to Shanghai Tech Part (B), WorldExpo '10 may apply δ=3.

Depending on the configuration used to generate the basic density distribution map, β=0.3, k=3 may be applied.

A Gaussian kernel with a fixed standard deviation (σ _i ) is used to blur head annotations and average head sizes in sparse crowd scene images.

The data increase module 320 increases the data amount of the crowd scene image of the dataset by applying the data increase method to increase the learning efficiency of the model.

As the data increase method, a partitioning method, a partitioning method, and a pseudo-noise insertion method may be applied.

4 as an example, the data increasing module 320 divides the crowd scene image of (a) into 4 equal parts, the first divided crowd scene image 11, the second divided crowd scene image 12, and the third divided crowd A scene image 13 and a fourth divided crowd scene image 14 are generated, and a fifth divided crowd scene image 15 of the same size as the divided crowd scene image is generated based on the center of the crowd scene image, and the The 6th to ninth divided crowd scene images 16, 17, 18, and 19 are generated by dividing the crowd scene image at random positions with the same size as the divided crowd scene image, and the data is increased as shown in (b). The divided crowd scene image is also referred to as a patch, and the patch has a size of 1/4 of the original image of the crowd scene image.

In addition, the data augmentation module 320 may horizontally invert the generated segmented crowd scene image (patch), and add some pseudo-noise.

The learning unit 400 has a crowd density distribution map model including a front-end processing unit 410 and a back-end processing unit 420 , and uses the pre-processed dataset input from the dataset pre-processing unit 300 as the crowd density. After learning the crowd density distribution map model by applying it to the distribution map model, the crowd density distribution map model is provided to the density distribution map generating unit 500 .

Specifically, the front-end processing unit 410 extracts and outputs 2D features from the crowd scene images having different resolutions of the pre-processed training dataset.

The front-end processing unit 410 includes five depth-by-depth separable convolutional layer modules 411 , 412 , 414 , 415 , 417 , three max pooling layers 413 , including three depth-wise separable convolutional layers. 416, 418) and a standard convolutional layer module 419 comprising two standard convolutional layers.

The five depth separable convolutional layer modules 411 , 412 , 414 , 415 , and 417 have filters of size 3*3, respectively, with 16, 32, 64, 128, and 256 filters. is comprised of

In addition, the depth separable convolutional layer modules 411 , 412 , 414 , 415 , and 417 apply the ReLU function as an activation function.

The max pooling layer 413 is connected to the output terminal of the second sequentially second separable convolutional layer module 412 by depth, and its size is 2*2.

The max pooling layer 416 is connected to the output terminal of the fourth separable convolutional layer module 455 by depth, which is sequentially fourth, and has a size of 2*2.

The max pooling layer 418 is connected to the output terminal of the fifth separable convolutional layer module 417 by depth, which is sequentially fifth, and has a size of 2*2.

The standard convolutional layer module 419 is configured at the rearmost stage, includes 512 filters of 3*3, and ReLU is applied as an activation function.

2D convolution is divided into depth-by-depth and point-by-point convolution, which is 1*1 convolution.

The front-end processing unit 410 of the present invention includes five depth-by-depth convolutional layer modules 411, 412, 414, 415, and 417 for performing depth convolution and one standard convolutional layer module 419 for performing point-by-point convolution. includes

The depth-by-depth convolution layer modules 411, 412, 414, 415, and 417 apply a single filter to all input channels by depth-by-depth convolution, and generate physical feature maps for all channels.

The standard convolutional layer module 419 generates and outputs a new output dataset by performing a point-by-point operation of 1*1 convolution in order to combine the outputs of the convolutional layer module by depth.

The front-end processing unit 410 performs a single depth-by-depth convolution process, and may be distributed into two additional operations for filtering, and uses a separate convolution for filtering, while used for other convolutional combinations, This distribution reduces the amount of computation in the convolution process and significantly reduces the model size.

Typical 2D convolution can be distinguished from separable convolution by depth.

Referring to FIG. 5, the 2D standard convolution layer includes an input having a size of 7*7*3 and a filter having a size of 3*3*3 in the form of height*width*channel as shown in 20 of FIG. If it has, the output size of that output layer is 5*5*1.

If 64 filters are included, the output layer will constitute an output layer having 64 output maps of size 5*5*1.

However, as shown in 30 of FIG. 5 , it may be configured to output as an output layer having a single size of 5*5*64 by applying 64 filters of 3*3*3 size. By doing this, it can be seen that the height and width decrease while the channel increases, as shown in 30 of FIG. 5 .

An operation of the front-end processing unit 410 that performs a separable convolution process for each depth in order to achieve the above-described transformation will be described.

The front-end processing unit 410 performs a depth-by-depth convolution operation in a first step, and performs a point-by-point convolution operation in a second step.

In the first step, the depth-by-depth convolution operation is applied to the input layer and rather than applying one filter having a 3*3*3 shape in 2D convolution, as shown in 40 of FIG. 6, each 3*3*1 It would be desirable to apply three filters having a size of .

Each filter is convolved with one channel of the input layer of size form 5*5*1, and these individual maps are wrapped together to produce a map of the form 5*5*3.

40 in FIG. 6 shows that the spatial dimension of the input layer is reduced, but the channel is still the same as before.

In the second step, as shown in 50 of FIG. 6 , the point-by-point convolution operation is applied by convolution of a filter of size 1*1*3 and an input image of size 5*5*3 providing a map of size 5*5*1. .

Therefore, the point-by-point convolution layer convolutions 64 filters of size 1*1*3 with the input image of size 5*5*3 as in 60 in FIG. create

The front-end processing unit 410 converts an input layer with a size of 7*7*3 into an output layer with a size of 5*5*64 together with the above two steps (convolution by depth and convolution by point).

As described above, since separable convolution for each depth is used, the number of operations performed by the front-end processing unit 410 can be reduced, and since fewer parameters are used compared to standard 2D convolution, the size of the model can be reduced.

The back-end processing unit 420 includes three first extended convolutional layer modules 421 including two extended convolution layers and a filter having a size of 3*3, and having a number of 512, 256, and 128 filters sequentially two each. , 422, 423), a second extended convolution layer module 424 including one extended convolution layer and 64 filters of size 3*3, and one standard convolution layer and size 1*1 It includes a standard convolutional layer module 425 that has one filter and applies ReLU as an activation function.

The extended convolutional layer modules 421 , 422 , 423 , 424 are applied for the partition problem, achieve very high accuracy and can be used instead of the pooling layer.

The extended convolution layers 421 , 422 , 423 , and 424 can be used as a replacement for the pooling layer, and the execution time is shortened and complexity is reduced by eliminating the need to use the deconvolution layer.

These extended convolutional layer modules 421 , 422 , 423 , and 424 do not use additional parameters or operations, and use extra kernels that extend the receiving field.

For example, a simple kernel of size m*m can be increased to m+(m-1)(d-1)(d-1)f _h . where d is the expansion rate.

By adjusting the expansion rate, the receiving field of the kernel having a size of 3*3 can be expanded as shown in FIG. 7 .

As shown in FIG. 7 , when the extension rate d=1, the extended convolutional layer acquires a 3*3 acceptance field like a normal convolutional layer.

When the extension factor d=2 is applied, the extended convolutional layer acquires a receiving field of 5*5, if d=3, it obtains a receiving field of size 7*7, and if d=4, a receiving field of 9*9 is obtained. will be able to obtain

The extended convolutional layer has many advantages in preserving the feature map resolution compared to mechanisms using conventional convolutional layers, pooling layers and deconvolutional layers.

These advantages will be described by way of example.

I have an overcrowded crowd scene image, and the crowd scene image is processed independently by two different techniques, but even after these techniques are applied to the image, the shape of the original input image and the shape of the output should be the same.

There are three main techniques of existing mechanism techniques: downsampling, convolutional layer, and upsampling.

The existing mechanism applies maximum pooling to shrink the shape of the initial input image, and for work, keeps the window size at 2, reducing the image size to half the shape of the initial image.

After the pooling operation, a 3*3 Sober filter is convolved with the processed image, and doublet interpolation is applied to the finally upsampling processed image so that the size becomes similar to the size of the original image.

The 2D extended convolution layer may be expressed by the following Equation (2).

where f(i, j) is the filter of height (H) and width (B), x(h,b) is the input, and g(h,b) is the output of the extended convolutional layer, The variable d represents the expansion rate.

If the parameter value d is 1, the extended convolutional layer behaves like a standard convolutional layer, but as the value of d increases, no additional parameters (weights) are used, and the receptive field of the filter is increased.

The above-described crowd density distribution map model including the front-end processing unit 410 and the back-end processing unit 420 applies a stochastic gradient descent method (SGD).

The stochastic gradient descent method is used as an optimization tool to train the proposed architecture at a constant learning rate, f _h .

As the loss channel, the Euclidean distance is used to find the distance between the predicted density distribution and the actual crowd density distribution map. The loss function can be expressed using Equation 3 below.

where F(Xi;θ) is the output of our proposed model, M is the batch size of training, θ is the parameter Xi used to display the input image, and D _i ^GT is the default density distribution map. am.

The density distribution map generation unit 500 receives and operates the crowd density distribution map model learned from the learning unit 400 , and directly receives the crowd scene image output by the crowd image acquisition unit 200 or a dataset preprocessing unit The input through 300 is applied to the crowd density distribution map model, and the crowd density distribution map according to the present invention is output to the crowd counting unit 600 .

The crowd counting unit 600 receiving the crowd density distribution map counts the number of people in the crowd by counting distribution points corresponding to the heads of people in the crowd density distribution map.

On the other hand, the extended convolution layer according to the present invention applies the extended convolution concept by convolution of the same filter, that is, a 3*3 Sobel filter having an extension rate d=2, with the original input image.

The output image has the same shape as the initial input image without any pooling and upsampling operations.

In addition, the extended convolution output includes additional comprehensive information.

On the other hand, it is common knowledge in the art that the present invention is not limited to the typical preferred embodiments described above, but can be improved, changed, replaced, or added in various ways within the scope of the present invention. Those who have will be able to understand it easily. If implementation by such improvement, change, substitution or addition falls within the scope of the appended claims below, the technical idea should also be considered to belong to the present invention.

100: data set generation unit 200: crowd image acquisition unit
300: data set preprocessor 310: basic density distribution map generation module
320: data increase module 400: learning unit
410: front-end processing unit 420: back-end processing unit
500: density distribution map generation unit 600: crowd counting unit

Claims (11)

a dataset generating unit for generating a dataset including a plurality of crowd scene images annotated on the number of crowds;
a dataset pre-processing unit that generates a basic density distribution map for the crowd scene images of the dataset, and increases data including the crowd scene image and the basic density distribution map to generate a training dataset;
The dataset preprocessor applies an end-to-end convolutional neural network including a separable convolutional layer for each depth and an extended convolutional layer to learn a crowd density distribution map for the number of heads in a crowd scene image. a learning unit for learning by applying the learning dataset input from
a density distribution map generator to which the learned crowd density distribution map model is applied to generate and output a crowd density distribution map for a crowd scene image input in real time; and
and a counting unit for receiving the crowd density distribution map and counting the number of crowds of the crowd scene image input in real time.
According to claim 1,
The data set preprocessor,
a basic density distribution map generator for generating and outputting a basic density distribution map by applying an adaptive geometry kernel expressed by the following equation to each of the crowd scene images of the dataset; and
Crowd scene image real-time analysis apparatus using an extended convolutional neural network, characterized in that it further comprises a data increasing unit for increasing the amount of data for the annotated crowd scene image.
[Equation]

Here, si denotes all target objects, δ denotes the basic density distribution map, di denotes the distance average of k nearest neighbors, and s denotes the position of a pixel in the input image.
3. The method of claim 2,
The data increase unit,
Crowd scene image real-time analysis apparatus using an extended convolutional neural network, characterized in that the data is increased by dividing the crowd scene image into a predetermined number of patches reduced by a predetermined ratio with respect to the size of the crowd scene.
According to claim 1,
The crowd density distribution map model is,
a front-end processing unit for extracting and outputting 2D features from crowd scene images having different resolutions of the preprocessed training dataset; and
Crowd scene image real-time analysis apparatus using an extended convolutional neural network, characterized in that it comprises a back-end processing unit that generates and outputs a crowd density distribution map by performing extended convolution of the 2D features output from the front-end processing unit.
5. The method of claim 4,
The front-end processing unit,
Including 3 separable convolutional layers by depth and filters of size 3*3, with different numbers of filters sequentially 16, 32, 64, 128 and 256, using ReLU as the activation function 5 depth-by-depth separable convolutional layer modules to apply;
a max pooling layer having a size of 2*2 and configured after the separable convolutional layer module for each depth and after the separable convolution layer module for each depth and after the separable convolutional layer module for each depth among the separable convolutional layer modules for each depth; and
Real-time analysis of crowd scene images using an extended convolutional neural network, comprising two standard convolutional layers and 512 filters with a size of 3*3, and a standard convolutional layer module that applies ReLU as an activation function Device.
5. The method of claim 4,
The back-end processing unit,
three first extended convolution layer modules comprising two extended convolution layers and a filter of size 3*3, sequentially having 512, 256 and 128 filter numbers;
a second extended convolutional layer module including one extended convolutional layer and 64 filters having a size of 3*3; and
Crowd scene image real-time analysis using an extended convolutional neural network, comprising one standard convolutional layer and one filter with a size of 1*1, and a standard convolutional layer module that applies ReLU as an activation function Device.
7. The method of claim 6,
The extended convolution layer is
Crowd scene image real-time analysis device using an extended convolutional neural network, characterized in that it extends the receiving field using extra kernels.
a dataset creation process in which the dataset creation unit generates a dataset including a plurality of crowd scene images annotated on the number of crowds;
A dataset preprocessing process in which a dataset preprocessing unit generates a basic density distribution map for the crowd scene images of the dataset, and increases data including the crowd scene image and the basic density distribution map to generate a training dataset ;
The dataset in the crowd density distribution map model, where the learning unit learns the crowd density distribution map for the number of heads in the crowd scene image by applying an end-to-end convolutional neural network including a separable convolutional layer and extended convolutional layer by depth. a learning process of learning by applying the learning dataset input from the preprocessor;
a density distribution map generation process in which a density distribution map generator generates and outputs a crowd density distribution map for a crowd scene image input in real time to which the learned crowd density distribution map model is applied; and
Crowd scene image real-time analysis method using an extended convolutional neural network, characterized in that the counting unit receives the crowd density distribution map and includes a counting process for counting the number of crowds of the crowd scene image input in real time.
9. The method of claim 8,
The data set preprocessing process is
a basic density distribution map generating step in which a basic density distribution map generator generates and outputs a basic density distribution map by applying an adaptive geometry kernel expressed by the following equation to each of the crowd scene images of the dataset; and
Crowd scene image real-time analysis method using an extended convolutional neural network, characterized in that the data increasing unit further comprises a data increasing step of increasing the data amount for the annotated crowd scene image.
[Equation]

Here, si denotes all target objects, δ denotes the basic density distribution map, di denotes the distance average of k nearest neighbors, and s denotes the position of a pixel in the input image.
10. The method of claim 9,
Crowd scene image real-time analysis method using an extended convolutional neural network, characterized in that the data increasing unit increases the data by dividing the crowd scene image into a certain number of patches reduced by a predetermined ratio with respect to the size of the crowd scene in the data increasing step .
9. The method of claim 8,
The learning process is
a front-end processing step of extracting and outputting 2D features from the crowd scene images having different resolutions of the pre-processed training dataset by the front-end processing unit; and
Crowd scene image real-time using extended convolutional neural network, characterized in that the back-end processing unit performs extended convolution of the 2D features output from the front-end processing unit to generate and output a crowd density distribution map. analysis method.

Images