KR102486083B1 - Crowded scenes image real-time analysis apparatus using dilated convolutional neural network and method thereof - Google Patents

Abstract

The present invention relates to an apparatus and method for analyzing a crowd scene image in real time, and more particularly, to end-to-end extended synthesis of a training dataset including crowd density distribution map information for people included in a crowd scene image. After learning a crowd density distribution map model to which a convolutional neural network (CNN) is applied, and applying the crowd scene image requested for analysis to the crowd density distribution map model, a high-quality real-time crowd density distribution map is generated. An apparatus and method for real-time analysis of a crowd scene image using an extended convolutional neural network that counts the number of people in a crowd scene image based on a crowd density distribution map.

Description

Crowded scenes image real-time analysis apparatus using dilated convolutional neural network and method thereof}

The present invention relates to an apparatus and method for analyzing a crowd scene image in real time, and more particularly, to end-to-end extended synthesis of a training dataset including crowd density distribution map information for people included in a crowd scene image. After learning a crowd density distribution map model to which a convolutional neural network (CNN) is applied, and applying the crowd scene image requested for analysis to the crowd density distribution map model, a high-quality real-time crowd density distribution map is generated. An apparatus and method for real-time analysis of a crowd scene image using an extended convolutional neural network that counts the number of people in a crowd scene image based on a crowd density distribution map.

In general, closed circuit television (CCTV) is installed in roads, plazas, and major spaces in buildings to safely protect citizens and track and monitor offenders.

In addition, various image analysis devices for collecting, storing, and analyzing images obtained through such CCTVs have been developed and applied.

A general image analysis device is used to detect a person from an image collected through CCTV, analyze and estimate the behavior of the detected person, or count a large number of people, that is, a crowd.

In particular, recently, an image analysis device for calculating or estimating the number of people, that is, the number of people, has been developed from still images and videos (hereinafter referred to as "crowd scene images") in which a large number of people, that is, crowds, are gathered for security. is being applied

A conventional image analysis device detects the entire body of a person or a specific part of the body (typically: a face) in an image obtained to count the number of crowds by a feature pattern for the corresponding body part, and counts the number of people. The density distribution map method, which counts the number of crowds by generating a crowd density distribution map according to the method and crowd distribution, is being applied.

An image analysis device that counts the number of crowds by direct detection and whole-body detection has a problem in that it is impossible to detect the entire body of a person because many people are entangled in the image.

An image analysis device that counts the number of crowds using a direct detection method and a body part also has a problem in that the number of crowds cannot be accurately counted because many people are entangled in the image and the corresponding part is often not detected.

1 is a diagram showing a real image including a similar number of crowds and a basic density distribution map in a general image analysis device.

Referring to FIG. 1, the image analysis device to which the density distribution map method is applied is a first image 1 of each of (a) and (b) having a similar number of people as shown in (a) and (b) of FIG. And it can be seen that the distributions of the first density distribution map 3 and the second density distribution map 4 corresponding to the second image 2 are not similar but completely different. That is, an image analysis device to which a general density distribution map method is applied cannot accurately count the number of people in a crowd.

Therefore, the development of an image analysis device capable of accurately counting the number of people in real time from a crowd scene image is required.

Republic of Korea Patent Registration No. 10-1467307 (2014.12.01. Notice)

Therefore, an object of the present invention is to develop a training dataset containing crowd density distribution map information for people included in a crowd scene image, to which a convolutional neural network (CNN) is applied to crowd density. After learning the distribution map model and applying the crowd scene image requested for analysis to the crowd density distribution map model to generate a high-quality real-time crowd density distribution map, the number of people in the crowd scene image is based on the generated real-time crowd density distribution map. It is to provide a crowd scene image real-time analysis device and method using an extended convolutional neural network that counts.

An apparatus for real-time analysis of crowd scene images using an extended convolutional neural network according to the present invention for achieving the above object is to generate a dataset including a plurality of crowd scene images annotated with respect to the number of crowds. wealth; a dataset pre-processing unit generating a basic density distribution map for the crowd scene images of the dataset, and generating a learning dataset by increasing data including the crowd scene image and the basic density distribution map; The dataset preprocessor is applied to a crowd density distribution map model that learns a crowd density distribution map for the number of heads of a crowd scene image by applying an end-to-end convolutional neural network including a separable convolutional layer by depth and an extended convolutional neural network. a learning unit for learning by applying the learning data set input from; a density distribution map generator for generating and outputting a crowd density distribution map for a crowd scene image input in real time by applying the learned crowd density distribution map model; and a counting unit receiving the crowd density distribution map and counting the number of crowds of the crowd scene image input in real time.

The dataset pre-processing unit includes: a basic density distribution map generating unit generating and outputting a basic density distribution map by applying an adaptive geometry kernel expressed by the following equation to each of the crowd scene images of the dataset; and a data increasing unit for increasing a data amount for the annotated crowd scene image.

[mathematical expression]

Here, si represents all target objects, δ is the basic density distribution map, di is the distance average of k nearest neighbors, and s represents the position of a pixel in the input image.

The data increasing unit may increase data by dividing the crowd scene image into a predetermined number of patches reduced by a predetermined ratio with respect to the size of the crowd scene.

The crowd density distribution map model may include a front-end processor extracting and outputting 2D features from crowd scene images having different resolutions of a preprocessed training dataset; and a back-end processing unit generating and outputting a crowd density distribution map by performing an extended convolution of the 2D features output from the front-end processing unit.

The front-end processing unit includes three separable convolution layers by depth and filters of size 3 * 3, sequentially having different numbers of filters of 16, 32, 64, 128, and 256, 5 per-depth separable convolutional layer modules applying ReLU as activation function; a max pooling layer having a size of 2*2 and configured behind a second separable convolution layer module for each depth and behind a fourth and fifth separable convolution layer modules for each depth among the separable convolution layer modules for each depth; and a standard convolution layer module including two standard convolution layers and 512 filters having a size of 3*3 and applying ReLU as an activation function.

The back-end processing unit includes three first extended convolution layer modules including two extended convolution layers and filters having a size of 3*3, and sequentially having 512, 256, and 128 filters; A second extended convolution layer module including one extended convolution layer and 64 filters of size 3*3; and a standard convolution layer module having one standard convolution layer and one filter having a size of 1*1 and applying ReLU as an activation function.

The extended convolution layer is characterized in that the receptive field is extended using redundant kernels.

A real-time analysis method for crowd scene images using an extended convolutional neural network according to the present invention for achieving the above object is: A dataset generator generates a dataset including a plurality of crowd scene images annotated with respect to the number of crowds. Data set creation process; A dataset pre-processing process in which the dataset pre-processing unit generates a basic density distribution map for the crowd scene images of the dataset, and creates a training dataset by increasing data including the crowd scene image and the basic density distribution map. ; The crowd density distribution map model in which the training unit learns a crowd density distribution map for the number of heads of a crowd scene image by applying an end-to-end convolutional neural network including a depth-separable convolutional layer and an extended convolutional layer to the dataset. a learning process of applying and learning the learning data set input from the pre-processing unit; a density distribution map generation process in which a density distribution map generating unit generates and outputs a crowd density distribution map for a crowd scene image input in real time to which the learned crowd density distribution map model is applied; and a counting step in which a counting unit receives the crowd density distribution map and counts the number of crowds of the crowd scene image input in real time.

In the dataset pre-processing process, the basic density distribution map generation unit generates and outputs a basic density distribution map by applying an adaptive geometry kernel expressed by the following equation to each of the crowd scene images of the dataset. Default density distribution map creation step; and a data increasing step in which the data increasing unit increases the amount of data for the annotated crowd scene image.

[mathematical expression]

Here, si represents all target objects, δ is the basic density distribution map, di is the distance average of k nearest neighbors, and s represents the position of a pixel in the input image.

The data increasing unit may increase data by dividing the crowd scene image into a predetermined number of patches reduced by a predetermined ratio with respect to the size of the crowd scene in the data increasing step.

The learning process may include a front-end processing step of extracting and outputting 2D features from crowd scene images having different resolutions of a pre-processed training dataset by a front-end processing unit; and a back-end processing step of generating and outputting a crowd density distribution map by performing an extended convolution of the 2D features output from the front-end processing unit by the back-end processing unit.

The present invention blurs and normalizes the annotated person's head included in the crowd scene image included in the acquired dataset, applies it to the adaptive geometry kernel to generate a basic density distribution map, and generates a basic density distribution map By adding to the dataset, there is an effect of improving the accuracy of counting the number of people included in the crowd scene image by increasing the quality of the dataset.

In addition, the present invention has an effect of improving the accuracy of counting the number of people in the crowd scene image by increasing the data by overlapping and segmenting the crowd scene images of the dataset and applying pseudo noise.

In addition, the present invention has an effect of reducing a large amount of computation in the convolution process and reducing the size of the model by applying a front end including a convolution layer separable by depth to the crowd density distribution map model.

In addition, the present invention prevents the loss of some special information of the map by minimizing the maximum and average pooling layers in the crowd density distribution map model, and does not require the use of an inverse convolution layer according to the minimization of the pooling layer, thereby reducing complexity and performance. It has the effect of reducing time.

1 is a diagram showing a real image including a similar number of crowds and a basic density distribution map in a general image analysis device.
2 is a diagram showing the configuration of a crowd scene image real-time analysis apparatus using an extended convolutional neural network according to the present invention.
3 is a diagram conceptually showing each component of an apparatus for analyzing a crowd scene image in real time using an extended convolutional neural network according to the present invention.
4 is a diagram for explaining the concept of data increase in the dataset pre-processing unit of the crowd scene image real-time analysis apparatus using the extended convolutional neural network according to the present invention.
5 is a diagram showing the configuration of a standard 2D convolution process layer for single and multiple filters according to the present invention.
6 is a diagram showing the configuration of a separable convolution layer for each depth according to the present invention.
7 is a diagram illustrating an effect of expanding an accommodation field according to an increase in an expansion rate according to the present invention.

Hereinafter, with reference to the accompanying drawings, the configuration and operation of a crowd scene image real-time analysis device using an extended convolutional neural network according to the present invention will be described, and a crowd scene image real-time analysis method in the device will be described.

Figure 2 is a diagram showing the configuration of a crowd scene image real-time analysis device using an extended convolutional neural network according to the present invention, Figure 3 shows each component of the crowd scene image real-time analysis device using an extended convolutional neural network according to the present invention 4 is a diagram for explaining the concept of data increase in the dataset pre-processing unit of the apparatus for real-time analysis of crowd scene images using an extended convolutional neural network according to the present invention, and FIG. It is a diagram showing the configuration of a standard 2D convolution process layer for a filter, and FIG. 6 is a diagram showing the configuration of a separable convolution layer for each depth according to the present invention. It is a diagram showing the expansion effect of the receiving field. It will be described with reference to FIGS. 2 to 7 below.

First of all, referring to FIG. 2, the device for real-time analysis of a crowd scene image using an extended convolutional neural network according to the present invention includes a dataset generator 100, a crowd image acquisition unit 200, a dataset preprocessor 300, and a learning unit. 400, a density distribution map generating unit 500, and a crowd counting unit 600.

The dataset generation unit 100 collects crowd scene images, which are images of a large number of dense crowds collected through CCTV or uploaded to an online server for a certain period of time, and the people included in the collected crowd scene images Create a dataset containing images of crowd scenes with annotations about heads obtained from administrators. The crowd scene image may be a still image or a frame-by-frame image of a moving picture.

In addition, the dataset generation unit 100 may acquire a dataset by loading a dataset including a plurality of crowd scene images including annotations on a person's head that have been obtained in advance and stored in a storage device as shown in Table 1 below. will be.

In Table 1, ShanghaiTech part (A) and crowd scene images of UCF_CC_50 are very crowded with many crowds, ShanghaiTech part (B), UCSD and WorldExpo'10 are relatively sparsely populated and lookout sample (crowd scene image) is present.

In Table 1, T _n represents the entire sample, and M _n and M _x represent the minimum and maximum number of individuals in the image, respectively.

It is used to represent the average number of people per image (A _v ), R is the image resolution, T _h is the entire annotated head, ROI is the region of interest, and whether the region of interest is present is indicated as Yes or No.

In order to improve the quality of the data included in the dataset, that is, the crowd scene image, the dataset generator 100 may perform a resizing process for each frame by applying bilinear interpolation or the like. For example, the dataset generation unit 100 may adjust the size of an overcrowded crowd video image for each frame to a resolution of 900*600.

When a dataset is generated or acquired, the dataset generator 100 provides the generated or acquired dataset to the dataset preprocessor 300.

The crowd image acquisition unit 200 obtains a crowd scene image to be analyzed in real time by CCTV, a server, etc., or directly provides it to the crowd density distribution map generator 300, or the dataset pre-processing unit 300 After performing data pre-processing, it is transmitted to the density distribution map generation unit 300.

The dataset preprocessor 300 includes a basic density distribution map generation module 310 and a data augmentation module 320 to generate new data derived from existing data and perform preprocessing to augment existing data.

Specifically, the basic density distribution map generation module 310 applies a geometric kernel to each crowd scene image including an annotation on a person's head in the dataset input from the dataset generator 100, as shown in 311 of FIG. Generate a basic density distribution map for real photos (crowd scene images).

The basic density distribution map generation module 310 applies a Gaussian kernel to blur all the heads of annotated crowd scene images, normalize them into one, and consider the special distribution of all images to obtain a basic density distribution map generate

A geometry adaptive filter for applying the Gaussian kernel can be expressed as in Equation 1 below.

where s _i denotes all target objects, δ is the crowd density distribution map, d _i is the distance average of k nearest neighbors, and S denotes the position of a pixel in the input image.

A convolution process is applied to the parameter (δ(ss _i )) and a Gaussian filter with standard deviation (σ _i ) to generate a crowd density distribution map.

If there are many crowd scene images, such as Shanghai Tech part (A) and UCF_CC_50 in Table 1, the geometry adaptation kernel can be used.

And Shanghai Tech Part (B), UCSD and WorldExpo'10 are datasets with crowd scene images with relatively few crowds, to which a fixed kernel can be applied, Shanghai Tech Part (B) to which δ = 15 can be applied, and WorldExpo '10 would be able to apply δ=3.

Depending on the configuration used when generating the basic density distribution map, β = 0.3 and k = 3 may be applied.

A Gaussian kernel with a fixed standard deviation (σ _i ) is used to blur head annotations and average head size in sparse crowd scene images.

The data augmentation module 320 increases the amount of data of the crowd scene images in the dataset by applying a data augmentation method to increase the learning efficiency of the model.

As the data augmentation method, a division method, a division method, and a pseudo noise insertion method may be applied.

Referring to FIG. 4 as an example, the data augmentation module 320 divides the crowd scene image of (a) into quarters to obtain a first split crowd scene image 11, a second split crowd scene image 12, and a third split crowd. A scene image 13 and a fourth divided crowd scene image 14 are generated, and a fifth divided crowd scene image 15 having the same size as the divided crowd scene image is generated based on the center of the crowd scene image, The sixth to ninth divided crowd scene images 16, 17, 18, and 19 are generated by dividing the crowd scene image into the same size as the divided crowd scene image at random locations, and the data is increased as shown in (b). The segmented crowd scene image is also referred to as a patch, and the patch has a size of 1/4 of the size of the original crowd scene image.

Also, the data augmentation module 320 may horizontally invert the generated segmented crowd scene image (patch) and add some pseudo noise.

The learning unit 400 has a crowd density distribution map model including a front-end processing unit 410 and a back-end processing unit 420, and the pre-processed dataset input from the dataset pre-processing unit 300 is converted to the crowd density After learning the crowd density distribution map model by applying it to the distribution map model, the crowd density distribution map model is provided to the density distribution map generator 500.

Specifically, the front-end processing unit 410 extracts 2D features from crowd scene images having different resolutions of the preprocessed training dataset and outputs them.

The front-end processing unit 410 includes five separable convolution layer modules for each depth including three separable convolution layers for each depth (411, 412, 414, 415, 417), three max pooling layers (413, 416, 418) and a standard convolution layer module 419 including two standard convolution layers.

The five separable convolutional layer modules 411, 412, 414, 415, and 417 for each depth have filters with a size of 3*3, and 16, 32, 64, 128, and 256 filters, respectively. It is composed of.

In addition, the depth-separable convolution layer modules 411, 412, 414, 415, and 417 apply a ReLU function as an activation function.

The max pooling layer 413 is connected to the output terminal of the second separable convolution layer module 412, which is second in order, and has a size of 2*2.

The max pooling layer 416 is connected to the output terminal of the fourth separable convolution layer module 455, which is sequentially fourth, and has a size of 2*2.

The max pooling layer 418 is connected to the output terminal of the fifth separable convolution layer module 417, which is 5th in order, and has a size of 2*2.

The standard convolution layer module 419 is configured at the last stage, includes 512 filters of 3*3, and ReLU is applied as an activation function.

2D convolutions are divided into depth-by-depth convolutions and point-by-point convolutions, which are 1*1 convolutions.

The front-end processing unit 410 of the present invention includes five depth-specific convolution layer modules 411, 412, 414, 415, and 417 that perform depth convolution and one standard convolution layer module 419 that performs point-by-point convolution. includes

The depth-specific convolution layer modules 411, 412, 414, 415, and 417 apply a single filter to all input channels as depth-specific convolution, and generate physical feature maps for all channels.

The standard convolution layer module 419 generates and outputs a new output dataset by performing a point-by-point operation, which is a 1*1 convolution, to combine the outputs of the depth-by-depth convolution layer modules.

The front-end processing unit 410 performs a single depth-by-depth convolution process, which can be split into two additional operations for filtering, while using a separate convolution for filtering, and for another convolutional combination, This distribution reduces the amount of computation in the convolution process and significantly reduces the model size.

Classical 2D convolutions can be distinguished from separable convolutions by depth.

Referring to FIG. 5, the 2D standard convolution layer has an input having a size of 7*7*3 and a filter having a size of 3*3*3 in the form of a height*width*channel as shown in 20 of FIG. , the output size of the output layer is 5*5*1.

In the case of including 64 filters, the output layer will constitute an output layer with 64 output maps of 5*5*1 size.

However, as shown in 30 of FIG. 5, it may be configured to apply 64 filters of a size of 3*3*3 to output an output layer having a single size of 5*5*64. By doing this, it can be seen that the height and width decrease while the channel increases, as shown at 30 in FIG. 5 .

An operation of the front-end processor 410 that performs a separable convolution process for each depth to achieve the above transformation will be described.

The front-end processing unit 410 performs a depth-by-depth convolution operation in a first step, and performs a point-by-point convolution operation in a second step.

In the first step, the depth-by-depth convolution operation is applied to the input layer and rather than applying one filter having the form 3*3*3 in 2D convolution, each 3*3*1 as shown at 40 in FIG. It would be desirable to apply three filters with a size of

Each filter is convolved with one channel of the input layer of size shape 5*5*1, and these individual maps are wrapped together to create a map of shape 5*5*3.

40 in Fig. 6 shows that the spatial dimension of the input layer is reduced, but the channels are still the same as before.

In the second step, the point-by-point convolution operation, as shown in 50 in FIG. 6, is applied by convolutional multiplication of a filter of size 1*1*3 and an input image of size 5*5*3 that provides a map of size 5*5*1. .

Therefore, the point-by-point convolution layer convolutions 64 filters of size 1*1*3 with an input image of size 5*5*3, as shown in 60 in FIG. generate

The front-end processing unit 410 converts an input layer of size 7*7*3 into an output layer of size 5*5*64 along with the above two steps (convolution by depth and convolution by point).

Since separable convolution by depth is used in this way, the operation performed by the front-end processing unit 410 can be reduced, and the size of the model can be reduced because fewer parameters are used than standard 2D convolution.

The back-end processing unit 420 includes two extended convolution layers and filters having a size of 3*3, including three first extended convolution layer modules (421 , 422, 423), a second extended convolution layer module 424 including one extended convolution layer and 64 filters of size 3*3, and one standard convolution layer and size 1*1 It includes a standard convolution layer module 425 that has one filter and applies ReLU as an activation function.

The extended convolution layer modules 421, 422, 423, 424 are applied for segmentation problems, achieve very high accuracy and can be used instead of pooling layers.

The extended convolution layers 421, 422, 423, and 424 can be used instead of the pooling layer, and reduce the execution time and complexity by eliminating the need to use the deconvolution layer.

These extended convolution layer modules 421, 422, 423, and 424 do not use additional parameters or operations, but use redundant kernels that extend the receptive field.

For example, a simple kernel of size m*m can be increased to m+(m-1)(d-1)(d-1)f _h . where d is the expansion rate.

By adjusting the expansion rate, as shown in FIG. 7, the receiving field of the 3*3 kernel can be expanded.

As shown in FIG. 7, when the expansion rate d = 1, the extended convolution layer obtains a 3*3 accommodation field like a normal convolution layer.

When the growth rate d=2 is applied, the extended convolutional layer obtains a 5*5 receptive field, d=3 obtains a 7*7 receptive field, and d=4 obtains a 9*9 receptive field. will be able to obtain

The extended convolutional layer has many advantages in preserving the feature map resolution compared to mechanisms using conventional convolutional layers, pooling layers, and deconvolutional layers.

These advantages are explained with an example.

I have a dense crowd scene image, and the crowd scene image is processed independently by two different techniques, but even after these techniques are applied to the image, the shape of the original input image and the shape of the output should be the same.

Existing mechanism technologies include three main technologies: downsampling, convolution layer, and upsampling.

Existing mechanisms apply max pooling to reduce the shape of the initial input image, and keep the window size at 2 for operation so that the image size is reduced to half of the initial image shape.

After pooling, a 3*3 Sober filter is convolved with the processed image, and finally, the upsampling processed image is subjected to double line interpolation so that its size is similar to that of the original image.

The 2D extended convolution layer can be expressed by Equation 2 below.

where f(i, j) is a filter of height (H) and width (B), x(h,b) is an input, g(h,b) is the output of the extended convolution layer, and The variable d represents the expansion rate.

When the parameter value d is 1, the extended convolutional layer behaves like a standard convolutional layer, but as the value of d increases, it does not use additional parameters (weights) and increases the filter's acceptance field.

The crowd density distribution map model including the front-end processing unit 410 and the back-end processing unit 420 described above applies stochastic gradient descent (SGD).

Stochastic gradient descent is used as an optimization tool to train the proposed architecture at a constant learning rate, f _h .

As the loss number, Euclidean distance is used to find the distance between the predicted density distribution and the actual crowd density distribution map. The loss function can be expressed using Equation 3 below.

where F(Xi;θ) is the output of our proposed model, M is the batch size for training, θ represents the parameter Xi is used to display the input image, and D _i ^GT is the base density distribution map. to be.

The density distribution map generation unit 500 receives and drives the crowd density distribution map model learned from the learning unit 400, and directly receives a crowd scene image output from the crowd image acquisition unit 200 or is a dataset pre-processing unit. 300, the crowd density distribution map is applied to the crowd density distribution map model, and the crowd density distribution map according to the present invention is output to the crowd counting unit 600.

Upon receiving the crowd density distribution map, the crowd counting unit 600 counts the number of people in the crowd by counting distribution points corresponding to the head of a person in the crowd density distribution map.

On the other hand, the extended convolution layer according to the present invention applies the extended convolution concept by convolutional-multiplying the same filter, that is, a 3*3 Sobel filter having an expansion rate d=2 with the original input image.

The output image has the same shape as the initial input image without pooling and upsampling.

In addition, the extended convolution output includes additional comprehensive information.

On the other hand, it is common knowledge in the art that the present invention is not limited to the above-described typical preferred embodiments, but can be implemented by various improvements, changes, substitutions, or additions within the scope of the present invention. If you have the , you can easily understand. If the implementation by such improvement, change, substitution or addition falls within the scope of the appended claims below, the technical idea should also be regarded as belonging to the present invention.

100: dataset generation unit 200: crowd image acquisition unit
300: dataset pre-processing unit 310: basic density distribution map generation module
320: data increase module 400: learning unit
410: front end processing unit 420: back end processing unit
500: density distribution map generation unit 600: crowd counting unit

Claims (11)

a dataset generating unit that creates a dataset including a plurality of crowd scene images annotated with respect to the number of crowds;
a dataset pre-processing unit generating a basic density distribution map for the crowd scene images of the dataset, and generating a learning dataset by increasing data including the crowd scene image and the basic density distribution map;
The dataset preprocessor is applied to a crowd density distribution map model that learns a crowd density distribution map for the number of heads of a crowd scene image by applying an end-to-end convolutional neural network including a separable convolutional layer by depth and an extended convolutional neural network. a learning unit for learning by applying the learning data set input from;
a density distribution map generator for generating and outputting a crowd density distribution map for a crowd scene image input in real time by applying the learned crowd density distribution map model; and
A counting unit for receiving the crowd density distribution map and counting the number of crowds of the crowd scene image input in real time;
The dataset pre-processing unit,
a basic density distribution map generating unit for generating and outputting a basic density distribution map by applying an adaptive geometry kernel expressed by the following equation to each of the crowd scene images of the dataset; and
A crowd scene image real-time analysis device using an extended convolutional neural network, characterized in that it further comprises a data increaser for increasing the amount of data for the annotated crowd scene image.
[mathematical expression]

Here, si represents all target objects, δ is the basic density distribution map, di is the distance average of k nearest neighbors, and s represents the position of a pixel in the input image.
delete According to claim 1,
The data augmentation unit,
A crowd scene image real-time analysis device using an extended convolutional neural network, characterized in that the crowd scene image is divided into a predetermined number of patches reduced by a predetermined ratio with respect to the size of the crowd scene to increase data.
According to claim 1,
The crowd density distribution map model,
a front-end processor extracting and outputting 2D features from crowd scene images having different resolutions of the preprocessed training dataset; and
A crowd scene image real-time analysis device using an extended convolutional neural network, characterized in that it comprises a back-end processing unit for generating and outputting a crowd density distribution map by performing an extended convolution of the 2D features output from the front-end processing unit.
According to claim 4,
The front-end processing unit,
Including three separable convolution layers by depth and filters of size 3*3, sequentially having different numbers of filters of 16, 32, 64, 128, and 256, and using ReLU as an activation function. Five depth-separable convolutional layer modules to apply;
a max pooling layer having a size of 2*2 and configured behind a second separable convolution layer module for each depth and behind a fourth and fifth separable convolution layer modules for each depth among the separable convolution layer modules for each depth; and
Crowd scene image real-time analysis using an extended convolutional neural network, characterized in that it includes a standard convolutional layer module that includes two standard convolutional layers and 512 filters of size 3 * 3 and applies ReLU as an activation function. Device.
According to claim 4,
The back end processing unit,
Three first extended convolution layer modules including two extended convolution layers and filters having a size of 3*3, sequentially having 512, 256, and 128 filters;
A second extended convolution layer module including one extended convolution layer and 64 filters of size 3*3; and
Crowd scene image real-time analysis using an extended convolutional neural network, characterized in that it includes a standard convolutional layer module having one standard convolutional layer and one filter having a size of 1*1 and applying ReLU as an activation function. Device.
According to claim 6,
The extended convolution layer,
An apparatus for real-time analysis of crowd scene images using an extended convolutional neural network, characterized in that the receiving field is expanded using redundant kernels.
a dataset creation process in which the dataset creation unit creates a dataset including a plurality of crowd scene images annotated on the number of crowds;
A dataset pre-processing process in which the dataset pre-processing unit generates a basic density distribution map for the crowd scene images of the dataset, and creates a training dataset by increasing data including the crowd scene image and the basic density distribution map. ;
The crowd density distribution map model in which the training unit learns a crowd density distribution map for the number of heads of a crowd scene image by applying an end-to-end convolutional neural network including a depth-separable convolutional layer and an extended convolutional layer to the dataset. a learning process of applying and learning the learning data set input from the pre-processing unit;
a density distribution map generation process in which a density distribution map generating unit generates and outputs a crowd density distribution map for a crowd scene image input in real time to which the learned crowd density distribution map model is applied; and
A counting process in which a counting unit receives the crowd density distribution map and counts the number of crowds of the crowd scene image input in real time;
The dataset preprocessing process,
a basic density distribution map generating step in which a basic density distribution map generating unit generates and outputs a basic density distribution map by applying an adaptive geometry kernel expressed by the following equation to each of the crowd scene images of the dataset; and
The crowd scene image real-time analysis method using an extended convolutional neural network, characterized in that the data augmentation unit further comprises a data augmentation step of increasing the amount of data for the annotated crowd scene image.
[mathematical expression]

Here, si represents all target objects, δ is the basic density distribution map, di is the distance average of k nearest neighbors, and s represents the position of a pixel in the input image.
delete According to claim 8,
The crowd scene image real-time analysis method using an extended convolutional neural network, characterized in that the data increaser divides the crowd scene image into a predetermined number of patches reduced by a predetermined ratio with respect to the size of the crowd scene in a data increase step to increase data. .
According to claim 8,
The learning process is
a front-end processing step of extracting and outputting 2D features from crowd scene images having different resolutions of the pre-processed training dataset by the front-end processing unit; and
A back-end processing step in which the back-end processing unit performs extended convolution of the 2D features output from the front-end processing unit to generate and output a crowd density distribution map Real-time crowd scene image using an extended convolutional neural network analysis method.

Images