KR101983684B1 - A People Counting Method on Embedded Platform by using Convolutional Neural Network - Google Patents

Abstract

A method of counting a person on an embedded platform using a convolutional neural network, the method comprising: (a) initializing and generating a background model using at least two consecutive previous frames; (b) receiving a current frame of an image; (c) updating a band top line and a bottom line of the background model using the deviation of the pixels of the background model; (d) extracting a pedestrian candidate area map using the background model and upper and lower lines of the band, and for each pixel of the current frame, extracting pixels out of the upper band line and the lower line of the updated background model, Extracting a candidate region map; (e) updating the background model using the current frame of the image, using only the background model of the previous frame in the pedestrian candidate area and updating only the remaining area; (f) inputting the pedestrian candidate area map into a convolutional neural network (CNN) pedestrian classifier and classifying the pedestrian candidate area map; And (g) counting pedestrians according to the result of the classification. Thus, the performance of the system is robust to the installation environment such as the camera photographing distance and angle, and can be operated in real time on the currently commercialized embedded board have.

Description

[0001] A People Counting Method on Embedded Platform using Convolutional Neural Network [

The present invention relates to a people counting method that operates in real time in an embedded environment and is a method for counting people on an embedded platform using a convolutional neural network that generates a background image without high learning or parameter adjustment, Counting method.

A key element of the people counting method is fast operation time, accuracy and unrestricted camera installation environment. In general, a background model generation method is used to quickly classify pedestrians. The brightness value of the image pixel has various characteristics according to the light source at the place where the image is taken and the type of the camera lens used for the image photographing.

In addition, the present invention proposes a convolutional neural network (CNN) -based pedestrian classification model having a more reliable pedestrian candidate group than an existing region proposal method, and relates to a method of counting people on an embedded platform using a convolutional neural network will be.

People counting is a video image analysis technique that detects how many people in a video image pass through a certain point, such as a store's entrance. It is one of the element technologies used in video-based surveillance system and field management. Especially, it is utilized in marketing field such as customer's movement in the store or the number of visitors at a specific time.

The people counting method has various installation environments such as the installation angle of the camera and the distance, and the name is different according to the installation angle. When the camera is installed on a ceiling, it is expressed as a top view system, and when a camera is installed in an oblique direction to photograph a pedestrian, the system is referred to as a bird's eye view system.

In the case of the top view method, the camera is installed on the ceiling of the building, so that the camera has a relatively limited installation environment. Since the photographing angle of view of the camera is viewed from above, the image mainly captured is limited to the head or shoulder of the pedestrian. Also, since the distance between the camera and the pedestrian is relatively close, it is easy to extract the semantic information from the image. Techniques that utilize semantic information have been actively studied in the top view type of people counting since system implementation is simple and advantageous for real-time processing. Such top view type people counting systems acquire images in a limited situation, so there is little degradation in recognition rate due to overlapping of outside lights, obstructed areas or pedestrians. For this reason, it shows comparatively satisfactory recognition rate by using only semantic information of a person without generating a classification model through machine learning.

Bird's eye view, on the other hand, is a method in which a camera shoots images in the form of birds flying in the sky and looking at the ground. Since the pedestrian images having various variants are captured according to the change of the shooting angle and the shooting distance, a high level of algorithm level such as machine learning is required to solve the people counting problem. People counting in Bird's eye view has traditionally been studied in the form of classification tasks.

A representative vector feature such as HOG (histogram of oriented gradient) [Non-Patent Document 1], ICF (integral channel feature) [Non Patent Document 2] is extracted and then SVM (support vector machine) or cascade This is the method of creating an object classification model using a learning machine. People counting with Bird's eye view method is still actively researched. Recently, CNN (convolutional neural network) based object recognition model which shows good performance in object detection task [ Non-Patent Documents 3 and 4] are also being studied. [Non-Patent Document 5] proposed a method of performing people counting using R-CNN (Non-Patent Document 6).

Deep learning methods for constructing artificial neural networks and learning image recognition models using big data show successful results in various image analysis fields. [Non-Patent Document 6-8] proposed a method of performing an object detection task for predicting the type and position of an object in an image using a CNN model and a region proposal method. In general, 2000 proposal regions are extracted from a single image and methods such as edge boxes (non-patent document 9) and selective search (non-patent document 10) are mainly used.

 DALAL, Navneet and TRIGGS, Bill: " Histograms of oriented gradients for human detection ", Computer Vision and Pattern Recognition, 2005. CVPR 2005. IEEE Computer Society Conference on. IEEE, Vol. 1, pp. 886-893 (2005)  Dollar, P., Tu, Z., Perona, P., and Belongie, S .: " Integral channel features ", In: BMVC, pp. 91-1 (2009)  Simonyan, K., and Zisserman, A .: " Very deep convolutional networks for large-scale image recognition " arXiv preprint arXiv: 1409.1556. (2014)  Proceedings of the IEEE conference on "Going deeper with convolutions", Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., on computer vision and pattern recognition, pp. 1-9 (2015)  Gao, C., Li, P., Zhang, Y., Liu, J., and Wang, L .: Neurocomputing, 208, pp. 108-116. (2016)  Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2014). Rich feature hierarchies for accurate object detection and semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 580-587).  Girshick, R. (2015). Fast r-cnn. In Proceedings of the IEEE international conference on computer vision (pp. 1440-1448).  Ren, S., He, K., Girshick, R., & Sun, J. (2015). Faster R-CNN: Towards real-time object detection with region proposal networks. In Advances in neural information processing systems (pp. 91-99).  Zitnick, C. L., & Doll, P. (2014, September). Edge boxes: Locating object proposals from edges. In European Conference on Computer Vision (pp. 391-405). Springer, Cham.  Uijlings, J. R., Van De Sande, K. E., Gevers, T., & Smeulders, A. W. (2013). Selective search for object recognition. International journal of computer vision, 104 (2), 154-171.  Li, G., Ren, P., Lyu, X., & Zhang, H. (2016, December). Real-Time Top-View People Counting Based on a Kinect and NVIDIA Jetson TK1 Integrated Platform. In Data Mining Workshops (ICDMW), IEEE 16th International Conference on 2016 (pp. 468-473). IEEE.  García, J., Gardel, A., Bravo, I., L ㅱ zaro, J. L., Martínez, M., & Rodríguez, D. (2013). Directional people counter based on head tracking. IEEE Transactions on Industrial Electronics, 60 (9), 3991-4000.  Hsieh, J. W., Peng, C. S., and Fan, K. C. (2007, October). Grid-based template matching for people counting. In Multimedia Signal Processing, 2007. MMSP 2007. IEEE 9th Workshop on (pp. 316-319). IEEE.  Chen, T. H., Chen, T. Y., & Chen, Z. X. (2006, June). An intelligent people-flow counting method for passing through a gate. In Robotics, Automation and Mechatronics, 2006 IEEE Conference on (pp. 1-6). IEEE.  Ma, H., Lu, H., & Zhang, M. (2008, June). A real-time effective system for tracking people using a single camera. In Intelligent Control and Automation, 2008. WCICA 2008. 7th World Congress on (pp. 6173-6177). IEEE  Hou, Y. L., & Pang, G. K. (2011). People counting and human detection in a challenging situation. IEEE transactions on systems, man, and cybernetics-part a: systems and humans, 41 (1), 24-33.  Chan, A. B., Liang, Z. S. J., & Vasconcelos, N. (2008, June). Privacy preserving crowd monitoring: Counting people without people models or tracking. In Computer Vision and Pattern Recognition, 2008. CVPR 2008. IEEE Conference on (pp. 1-7). IEEE.  Zeng, C., & Ma, H. (2010, August). Robust head-shoulder detection by pca-based multilevel hog-lbp detector for people counting. In Pattern Recognition (ICPR), 2010 20th International Conference on (pp. 2069-2072). IEEE.  Subburaman, V. B., Descamps, A., & Carincotte, C. (2012, September). Counting people in the crowd using a generic head detector. In Advanced Video and Signal-Based Surveillance (AVSS), 2012 IEEE Ninth International Conference on (pp. 470-475). IEEE.  Topkaya, I. S., Erdogan, H., & Porikli, F. (2014, August). Counting people by clustering person detector outputs. In Advanced Video and Signal Based Surveillance (AVSS), 2014 11th IEEE International Conference on (pp. 313-318). IEEE.  Gao, L., Wang, Y., & Wang, J. (2016, October). People counting with block histogram features and network flow constraints. In Image and Signal Processing, BioMedical Engineering and Informatics (CISP-BMEI), International Congress on (pp. 515-520). IEEE.  Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012). Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems (pp. 1097-1105).  Lin, M., Chen, Q., & Yan, S. (2013). Network in network. arXiv preprint arXiv: 1312.4400.  Felzenszwalb, P. F., Girshick, R. B., McAllester, D., & Ramanan, D. (2010). Object detection with discriminatively trained part-based models. IEEE transactions on pattern analysis and machine intelligence, 32 (9), 1627-1645.  Srivastava, N., Hinton, G. E., Krizhevsky, A., Sutskever, I., & Salakhutdinov, R. (2014). Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15 (1), 1929-1958.  C920. Available online: http://www.logitech.com/en-roeu/product/hd-pro-webcam-c920 (accessed on 19 July 2017).  NVIDIA Jetson TX2. Available online: http: //www.nvidia .com / object / embedded-systems-dev-kits-modules.html (accessed on 19 July 2017).

An object of the present invention is to solve the above problems and to provide a method of counting people who operate in real time in an embedded environment, And to provide a method of counting people on an embedded platform using a convolutional neural network, which generates a background image without using a convolutional neural network.

Particularly, the present invention provides a method of counting people on an embedded platform using a convolution neural network, which performs a pedestrian detection based on a CNN, which generates a pedestrian candidate region using a background model and inputs it as an input.

In addition, the present invention generates a highly reliable pedestrian candidate group using a background model instead of the existing region proposal method, uses a CNN (convolutional neural network) -based pedestrian classification model as an input, To provide a method of counting people on an embedded platform using a convolutional neural network.

According to an aspect of the present invention, there is provided a method for counting people on an embedded platform using a convolutional neural network that counts pedestrians in an image composed of consecutive frames, the method comprising: (a) Initializing and generating a model; (b) receiving a current frame of an image; (c) updating a band top line and a bottom line of the background model using the deviation of the pixels of the background model; (d) extracting a pedestrian candidate area map using the background model and upper and lower lines of the band, and for each pixel of the current frame, extracting pixels out of the upper band line and the lower line of the updated background model, Extracting a candidate region map; (e) updating the background model using the current frame of the image, using only the background model of the previous frame in the pedestrian candidate area and updating only the remaining area; (f) inputting the pedestrian candidate area map into a convolutional neural network (CNN) pedestrian classifier and classifying the pedestrian candidate area map; And (g) counting the pedestrian according to the classification result.

In the method of counting people on an embedded platform using a convolutional neural network, in the step (a), the background model is initialized by averaging N background frames (N is a natural number of 2 or more) ), The N frames are averaged up to the previous frame including the current frame.

In the method of counting people on an embedded platform using a convolutional neural network, the background model is updated by the following equation (1) in step (c).

[Equation 1]

However, B _m (i, j) represents a pixel (i, j) of the background model of the m-th frame is the current frame, Ik (i, j) is the pixel value of the pixel (i, j) of the k-th frame , F _m (i, j) is a pedestrian candidate area map indicates whether the pedestrian candidate region and the pixel (i, j) of the m-th frame, N is a natural number of at least 2.

According to another aspect of the present invention, there is provided a method of counting people on an embedded platform using a convolutional neural network, wherein in step (d), the upper line U_band and the lower line L_band of the background model are updated by the following equation (2).

[Equation 2]

However,? Is a predetermined constant as a weight.

Further, the present invention is a method for counting people on an embedded platform using a convolutional neural network, wherein in the step (e), the pedestrian candidate area map is obtained by the following equation (3).

[Equation 3]

However, F _m (i, j) is the pixel ¹ a represents the pixel value of the pixel of the pedestrian candidate region map (i, j), I _m (i, j) is the pixel (i, j) of the current frame m.

According to another aspect of the present invention, there is provided a method of counting people on an embedded platform using a convolutional neural network, wherein the CNN pedestrian classifier is added with a layer of an inception module, And a structure shown in FIG.

The present invention also provides a method for counting people on an embedded platform using convolutional neural networks, comprising the steps of: generating and learning a given learning image by applying movement and rotation operations to the CNN pedestrian classifier; .

According to another aspect of the present invention, there is provided a method of counting people on an embedded platform using a convolutional neural network, the method comprising: generating a new learning image by multiplying the given learning image (x, y) Thereby generating a learning image.

[Equation 4]

(X ', y') represents a newly generated training image, a and b are rotation parameters, and c and d are parallel movement parameters, respectively.

Further, the present invention is a method for counting people on an embedded platform using a convolutional neural network, the method comprising performing a pruning operation on the incessant module.

The present invention also relates to a computer-readable recording medium on which a program for performing a person counting method on an embedded platform using a convolutional neural network is recorded.

As described above, according to the method of counting people on an embedded platform using the convolution neural network according to the present invention, the performance of the system is robust to installation environments such as camera shooting distance and angle, and can be operated in real time on a commercially available embedded board Is obtained.

Particularly, the people counting method according to the present invention proves that it operates in real time on nvidia's TX1 and TX2 embedded boards through experiments and has robust characteristics such as camera shooting distance and angle .

BRIEF DESCRIPTION OF THE DRAWINGS Fig. 1 is a diagram showing a configuration of an overall system for carrying out the present invention; Fig.
Figure 2 is a table showing comparisons for people counting methods.
3 is a flow chart illustrating a method of counting people on an embedded platform using a convolutional neural network according to an embodiment of the present invention.
4 is a flow diagram illustrating a structure of a method of counting people according to an embodiment of the present invention.
5 is a graph illustrating a brightness value variation characteristic of an image according to an illumination environment according to an exemplary embodiment of the present invention.
6 is a graph showing a top band U_band and a bottom band L_band of a brightness value variation band of an image according to an exemplary embodiment of the present invention.
7 is a table showing a structure of a convolutional neural network of GoogleNet used in the present invention.
FIG. 8 illustrates an inception module of GoogleNet used in the present invention. FIG.
9 is a table showing the structure of CNNs for pedestrian classification according to an embodiment of the present invention.
10 shows an image and a specification of a C920 camera used in the experiment of the present invention.
11 is a table showing specifications of Jetson TX2 of Nvidia used in the experiment of the present invention.
12 illustrates an installation environment of a people counting method according to an experiment of the present invention.
Figure 13 is a pedestrian image obtained in various environments according to the experiment of the present invention.
Fig. 14 is an image of a pedestrian motion scenario according to the experiment of the present invention. Fig. 14 (b) shows a situation in which a bag is used, (b) (F) running, (g) moving at an intersection, (h) moving a crowd.
15 is a table showing the accuracy of the person counting for the experimental data according to the experiments of the present invention;
16 is a table showing an error matrix of people counting for experimental data according to the experiments of the present invention.
17 is a graphical representation of a people counting error for each pedestrian motion scenario according to the experiment of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, the present invention will be described in detail with reference to the drawings.

In the description of the present invention, the same parts are denoted by the same reference numerals, and repetitive description thereof will be omitted.

First, examples of the configuration of the entire system for carrying out the present invention will be described with reference to Fig.

1, a method of counting people on an embedded platform using a convolution neural network according to the present invention includes a step of recognizing a pedestrian with respect to the image (or image) May be implemented in a program system on the computer terminal 20 that counts the number. That is, the method of counting people can be implemented by a program and installed in the computer terminal 20 and executed. A program installed in the computer terminal 20 can operate as a single program system 30. [

Meanwhile, as another embodiment, the method of counting people according to the present invention may be implemented by a single electronic circuit such as an ASIC (on-demand semiconductor) in addition to being operated by a general-purpose computer. Alternatively, the dedicated computer terminal 20 may be developed to recognize only pedestrians and count the number of pedestrians on an image basis. This will be referred to as a people counting device or a people counting system 30. Other possible forms may also be practiced.

On the other hand, the image 10 is composed of consecutive frames in time. One frame has one image. Also, the image 10 may have one frame (or image). That is, the image 10 may correspond to one image.

Next, a method of counting people used in the present invention will be described in more detail with reference to FIG.

The people counting method has various algorithm approach according to the installation angle and distance of the camera, and is largely divided into a method using semantic information and a method using machine learning.

In the Top view method, the camera is installed on the ceiling of the building, so it has a relatively limited installation environment. Therefore, the system mainly uses semantic information (non-patent document 11-16) for the purpose of system implementation or real-time operation of the system.

Bird's eye view method has various variants according to the change of shooting angle and shooting distance. Methods for generating a pedestrian recognition model by learning handcrafted-features in order to solve the problem of people counting (Non-Patent Document 17-21) have been studied as mainstream since pedestrian images are shot . Figure 2 compares various methods of performing people counting.

Methods that use semantic information perform the people counting reflecting the depth map, template matching, histogram analysis, and the like. [Non-Patent Document 15] solves the problem by analyzing the horizontal and vertical histograms of the candidate area of the pedestrian. [Non-Patent Documents 12 and 13] use template matching through a preprocessing method such as a distance transform or an edge extraction method. In [Non-Patent Document 11], a head portion of a person is searched using depth information. It is difficult to install the people counting method because it is difficult to obtain depth information at a relatively long distance.

Machine learning techniques include a method using hand-crafted features and a method using convolutional neural networks. It is also possible to use widely known hand-crafted features such as histogram of oriented gradient (HOG), color features, canny edges, and the like to define areas, perimeters, a method using a linear feature such as a perimeter edge (Non-Patent Document 17) was also studied. Recently, there have been studies using CNN (convolutional neural network) structure which shows good performance in object detection. [Non-Patent Document 5] proposed a method of performing people counting using region based CNN (Non-Patent Document 6).

On the other hand, the background model is mainly used to generate a pedestrian candidate region in the input image. A pedestrian candidate region is expressed as an ROI (region of interest). The pedestrian classifier can selectively process ROIs instead of processing all areas of the input image to eliminate redundancy in the inference pipe-line.

A method of generating a background model includes a method using a Gaussian mixture model and a method of threshold processing brightness values of an image. When a Gaussian mixture model is used, re-learning of the background model is required when the type of input image is changed. The method of threshold processing the brightness of the image depends heavily on the user's hyper-parameter. Since the threshold value changes depending on the brightness value variation characteristic of the input image, it often depends on the experimental method to find an appropriate threshold value.

In the present invention, a background model is generated by observing fluctuation characteristics of intrinsic brightness values of an image. Update the background model locally and create a pedestrian candidate area. In addition, a convolutional neural network (CNN) based pedestrian classifier structure with pedestrian candidate area as input is presented.

Next, a method of counting people on an embedded platform using a convolutional neural network according to an embodiment of the present invention will be described with reference to FIGS. 3 and 4. FIG. FIG. 3 shows an overall flow chart of the method of counting people according to the present invention, and FIG. 4 shows the structure of a method of counting people according to the present invention.

The method for counting people according to the present invention comprises a preprocessing section for generating a background model and a pedestrian candidate region and a pedestrian classification section for performing a people counting operation. In the preprocessing section, background model generation and update, and pedestrian candidate areas are detected. In the pedestrian classification part, pedestrian classification is performed by taking the detected pedestrian candidate area as an input, and finally the people counting operation is performed with the result.

As shown in FIG. 3, the method of counting people on an embedded platform using convolution neural network according to the present invention includes generating an initial background model (S10), receiving a frame of an image (S20) A step S50 of extracting a pedestrian candidate area map, a step S50 of updating a background model, a step S60 of inputting to a CNN classifier, and a step of counting the number of pedestrians according to the result of classification S70). The steps S20 to S70 are repeated until the last frame.

Before describing each step, the overall process of creating a background model is described first. Background model generation is a classical image processing method used mainly for finding moving objects in an image. Many non-patent documents 12, 13, 15, 17, and 22 have been studied to generate a candidate region of a moving pedestrian with a difference image between an input frame and a background model. A method of generating a background model is classified into a method using a gaussian mixture model (GMM) and a method of processing a brightness value of an image pixel with a threshold value. GMM method is not used well in real system because re-learning of background model is necessary when the type of input image is changed. What is important in the method of processing the brightness value of the image pixel as a threshold value is to obtain a proper threshold value for distinguishing the background and the object. In the conventional method, the variation characteristics of image brightness value are analyzed heuristically to obtain good performance.

The variation characteristics of the brightness value of the image have various characteristics according to the light source at the place where the image is acquired and the type of the used camera lens. For example, when using only artificial light and using natural light and artificial light at the same time, the variation characteristics of brightness values of acquired images are very different.

5 shows the variation of brightness values of different images. 5 (a) is a variation characteristic of an image obtained in a place where artificial light and natural light exist at the same time. 5 (b) is an image obtained under dark illumination, and FIG. 5 (c) is a brightness value variation characteristic of an image obtained in an artificial illumination environment. In the present invention, a background model is created by observing brightness characteristic of an image.

It is assumed that the brightness value variation characteristic of the background image has a standard normal distribution function, and a brightness value band moving up and down by the width of the background image is formed according to the brightness value change of the background image. In Fig. 6, the blue solid line is the brightness value variation characteristic of Fig. 5 (a), and the red solid line is the brightness value variation characteristic of Fig. 5 (b). As shown by the dotted line in FIG. 6, upper band U_band and lower band L_band are defined as a band higher than each brightness value. Based on the determined band, pixels having a brightness value deviating from this band are judged to be a pedestrian candidate area. The pixels in the band are used for background image update.

First, an initial background model is generated (S10).

The initial background model B _init is determined as an average value of N background frames as shown in Equation (1). At this time, the background frame is selected as an arbitrary monochrome image that does not include a pedestrian.

 [Equation 1]

Here, I _k is the k-th input image frame, and X and Y denote the horizontal and vertical sizes of the image, respectively.

Next, a frame of the image to be counted is input (S20). The image is composed of consecutive frames, and performs the person counting sequentially for successive frames. That is, the subject counting is performed for one frame, and the subject counting is performed for the next frame. At this time, the background model is updated every time the person counting of each frame is performed.

Next, upper and lower band lines U_band and L_band of the background model are updated (S30).

L_band (i, j) and U_band (i, j) are defined by adding or subtracting the weighted standard deviation value to the pixels of the background model image of the previous frame.

&Quot; (4) "

Here, α is the weight, which determines the width of the brightness value band. The above procedure is repeated for a new input image.

Next, the pedestrian candidate area map is extracted using the background model and the upper and lower disconnection lines of the corresponding model (S40).

Using the m-th input I _m of the system and the background model image B _m , an area map F _m of a pedestrian candidate is generated as shown in Equation (3). The pedestrian candidate area map F _m is given as input to the pedestrian classifier.

&Quot; (3) "

Here, L_band (i, j) and U_band (i, j) are a low brightness value band and a high brightness value band for each pixel in the given image of FIG.

Next, the background model is updated by the input image frame (S50).

As described above, when the initial background model B _init is determined in Equation (1), the background model B _m of the m-th frame is updated with respect to some pixels of the image satisfying the condition as shown in Equation (2). That is, the background model of the previous frame is used as it is in the pedestrian candidate area, and the average is taken only in the area excluding the pedestrian candidate area in the background area.

&Quot; (2) "

Here, B _m (i, j) is a background model value at the pixel position (i, j) of the m-th frame, and I _k is the k-th input image frame.

F _m (i, j) is obtained by using B _{m - 1} (i, j) and upper and lower disconnection (L_band, U_band). Since initial B _{m -1} (i, j) is not defined in the beginning, F _m (i, j) is obtained by using initial background model B _init (i, j) F _m (i, j) is calculated back to update the _init B (i, j) is obtained by using this, the B _m (i, j). And B _m (i, j) is used to find F _{m + 1} (i, j).

Next, the pedestrian candidate area map is input to the CNN pedestrian classifier and classified (S60). The pedestrian classifier using CNN will be described in detail below.

Next, the number of pedestrians is counted according to the classification result (S70). Then, in steps S20 to S70, that is, the next frame is received, and the person counting operation for the next frame is repeated (S70). The iterative process proceeds to the last frame (S80).

Once the classification operation is completed, the position information of the pedestrian can be acquired in the image. Based on this location information, pedestrian counting is performed when a specific reference line or reference area is passed.

Next, a method for performing pedestrian classification using CNN according to an embodiment of the present invention will be described in more detail.

Pedestrian classification has been studied variously. [Non-Patent Document 1] greatly improves the performance of pedestrian recognition by proposing a hand-crafted feature called a histogram of oriented gradient (HOG) with high repeatability for a human being. Then, DPM (deformable part models) (Non-Patent Document 25) partially applying HOG to human's arms, legs, and head were proposed and performance was further improved. Since then, deep learning techniques have been used for pedestrian classification, which shows good performance in image recognition.

Deep learning is a method of building an artificial neural network and learning image recognition models using big data. [Non-Patent Document 6-8] proposed an object extraction technique for predicting the type and position of an object in an image using CNN and a region proposal method. Generally, 2000 object candidate regions are extracted from one image, and methods such as Edgeboxes (Non-Patent Document 14) and Selective Search (Non-Patent Document 13) are mainly used.

In the method of the present invention, a pedestrian candidate region is created using the background model described above, and the pedestrian is classified using the CNN method. That is, in the present invention, in particular, a CNN structure in which GoogleNet (Non-Patent Document 4) is pruned is used. GoogleNet is a representative CNN-based object classifier that has demonstrated good performance in the classification of the International Large Scale Visual Recognition Competition (ILSVRC) in 2014. GoogleNet is composed of 22 layers in total, and a structure in which only a convolutional layer is simply stacked as VGG-Net [Non-Patent Document 3] or Alex-Net [Non Patent Document 23] A new layer structure called the inception module has been added. The table in FIG. 7 represents the CNN structure of GoogleNet.

The inception module of GoogleNet is a type of network in network (NIN) [Non-Patent Document 24] and uses a convolutional filter having different sizes in the same layer So that features with different scales can be obtained. FIG. 8 shows an inception module structure of GoogleNet.

The NIN structure has the advantage that various scale features can be generated as described above. However, since the computational complexity is larger than that of the general CNN structure, the number of layers in the network increases the time required for object inference. GoogleNet proposes a method to reduce the number of feature-maps through 1 × 1 convolution operation, so that the computation volume is evenly distributed and thus a deep network can be constructed.

Inception modules are a kind of NIN structure. A subset of the NIN structure is an INSESSION module. The general CNN structure has difficulties in extracting non-linear features because the characteristics of the filter are linear. The NIN structure has a good performance for non-linear feature extraction by inserting another CNN in CNN as a way to overcome this problem.

In the present invention, a CNN structure for classifying a pedestrian using an inception module is proposed. The input image is a 3-channel (RGB) color image and the image size is 96 × 180. Because it performs classification for a single class, pruning is performed to reduce the number of network depths and feature maps for the basic GoogleNet structure. 9 is a proposed CNN structure.

In general, as the network depth and feature map number of CNN structure increases, the recognition rate improves, but at the same time, the program operation time increases. For the real-time operation of the program, a CNN structure with an appropriate amount of network depth and number of feature maps is needed. Pruning means reducing the number of nodes included in each layer or deleting the layer itself to reduce the size of existing CNN structures. Because pruning is the task of deleting layers or reducing the number of feature maps in a CNN structure, the number of network depths and feature maps resulting from pruning operations is reduced.

The method according to the present invention includes a regression head operation for determining the position of a pedestrian in addition to the operation for determining whether the object is a pedestrian. The regression head operation predicts the position of the upper left corner of the bounding box, the width, the height, and the like. Classification and regression head work share the weights of the CNN network so that object classification and bounding box regression are performed through a single feedforward without additional computation. Can be performed.

In general, a deep neural network model consists of a large number of parameters. The number of parameters constituting the model increases as the number of feature maps (feature-maps) used in the CNN structure increases, and as the network depth increases. The number of data actually used for learning is limited. Therefore, when learning a deep neural network model, overfitting occurs in which the model parameters are learned over the learning data. Overfitting is a phenomenon that works well for learning data but error for real data increases.

To solve the overfitting, a dropout (Non-Patent Document 27) and a data augmentation method are mainly used. Dropout is a method of partially using the fully connected layer (FCL) of the CNN structure in the learning phase. The layers used and the unused layers are randomly determined. The deep running model of the method according to the present invention solves overfitting through data augmentation because it does not include FCL.

In the present invention, a translation and a rotation operation are applied to a learning image for data augmentation. The original image is moved in parallel in the up, down, left, and right directions, and rotation calculation is performed on the resulting image. The rotation operation is applied to the center of the image as 0, 5, and 10 degrees in the left and right directions. As a result of data augmentation, 20 new images are created for one input image. Equation (5) is an equation for rotation and translation.

&Quot; (5) "

Where a and b are the rotation parameters and c and d are the translation parameters, respectively.

Next, the effects of the present invention will be described in more detail through experiments.

First, the experimental environment will be described.

A Logitec C920 camera was used in the experiment. FIG. 10 shows a product specification and a real photograph of the C920 used in the experiment (Non-Patent Document 27). The resolution of the input image is 1280 × 720 and the frame rate is 20 fps. The embedded board, on which the people counting method is implemented, is connected via USB 2.0.

The experiment uses Nvidia's Jetson TX2 embedded board. The Jetson TX2 board includes a GPU, which also makes it possible to use the NVIDIA CUDA platform. Figure 7 shows the product specification of Jetson TX2 [Non-Patent Document 27].

Figure 12 is an installation environment of the people counting method used in the experiment. One camera was installed in Bird's eye view mode.

Next, the experimental results will be described.

Experimental images were taken in various scenarios in various lighting environments. These include scenarios that you think are possible in everyday life, such as pedestrians wearing arms, pedestrians holding hands, jogging or running fast, or having a bag.

The distance between the camera and the pedestrian is a very important factor in the installation of the people counting method. As the shooting distance increases, the degree of loss of information of the pedestrian depends on the shooting angle and also has a great influence on the recognition rate. In the present invention, images were acquired at various photographing distances. Figure 13 is a pedestrian image obtained in various environments.

13 (a) was photographed in the room. In this case, the camera is installed to look at the entrance of the building and the distance from the pedestrian is 3m. Fig. 13 (b) is an image obtained from outside, and the distance between the pedestrian and the camera is 15 m, which is a relatively long distance. 13 (c) and 13 (d) are images obtained by changing the photographing angle and the photographing distance in the same room. In this case there is no special lighting and backlighting is occurring at the entrance of the building, which is a very bad environment for pedestrian classification.

Figure 14 is a scenario of various pedestrian movements. We assumed various situations in everyday life, such as when a pedestrian is holding a box or carrying a bag, moving at an intersection, moving shoulders side by side, or moving in a crowd. 14A, 14B, 14C, and 14D include the case where all of the pedestrian motion scenarios shown in FIG. 14 are included.

FIG. 15 shows the results of the people counting of the method according to the present invention for each of the experimental videos described above. The experimental video shown in FIG. 13 (a) includes a total of 304 pedestrians, (b) 317 pedestrians, and (c) and (d) 520 pedestrians. In the table of FIG. 15, the total number means the number of pedestrians corresponding to each scenario. "Detection" means the number of pedestrians that the proposed people counting method has succeeded in detection, and finally, the percentage of people counting accuracy.

The method according to the present invention has an average of 89.9% accuracy. The experimental image (b) shows a relatively low recognition rate as compared with the other images because the pedestrians were photographed from the side. Taking the pedestrian from the side, information about the shape of the pedestrian is lost, which causes a decrease in the recognition rate. Particularly in the case of images acquired from the side, the area where the pedestrian crosses the people counting line frequently occurs.

The table of FIG. 16 shows the accuracy of people counting per scenario. All the pedestrian movements defined in FIG. 14 are included. The number of accurately detected pedestrians for the total number of pedestrians included in each scenario is expressed, and finally the percentage of pedestrian detection accuracy is also shown. Fig. 17 shows the error rate when the pedestrian is detected.

The people counting method proposed in the present invention shows a detection result robust to the shape change of a pedestrian. As shown in the scenarios (a) and (b), when the pedestrian has the object or the pedestrian type change is relatively large as in the scenarios (c) and (d) Even if a pedestrian is running, a blur phenomenon is observed in consecutive frames in the moving image, but there is no significant influence on the pedestrian detection rate.

On the other hand, the scenarios (g) and (h) show a relatively low recognition rate. Scenario (g) is a situation where a pedestrian moves at an intersection at an entrance and exit, and scenario (h) moves in a crowded state. When pedestrians cross or densely move, there is a blocked area between pedestrians. The CNN-based pedestrian classifier according to the present invention recognizes pedestrians existing in the obstructed area as one person, and an error occurs. The area of occlusion that occurs when pedestrians are overlapped can always occur irrespective of the angle of camera installation or distance, and in this case, counting error is difficult to avoid. However, when the masked area occurs during a short frame period, it operates normally.

In the present invention, a method of counting people in Bird's eye view form has been described. In the method according to the present invention, a background model is generated using a characteristic that a variation in pixel brightness value of an image is different according to an illumination environment, and a background model is updated only in a part of a given image. The pedestrian candidate area generated by the method according to the present invention is given as an input of a CNN-based pedestrian classifier. The pedestrian classifier performs an operation of determining whether the inputted pedestrian candidate area is a pedestrian.

The method according to the present invention has a robust performance in a lighting environment, a camera installation distance, and shows good performance in various pedestrian scenario situations. In the case of overlapping pedestrians or transporting a large object, the obstruction area occurs and the detection performance is relatively low, but it can be confirmed that the operation is robust against the system installation environment in general.

Although the present invention has been described in detail with reference to the above embodiments, it is needless to say that the present invention is not limited to the above-described embodiments, and various modifications may be made without departing from the spirit of the present invention.

10: video 20: computer terminal
30: Program system

Claims (10)

A method of counting people on an embedded platform using a convolutional neural network that counts pedestrians in an image composed of consecutive frames,
(a) initializing and generating a background model using at least two consecutive previous frames;
(b) receiving a current frame of an image;
(c) updating a band top line and a bottom line of the background model using the deviation of the pixels of the background model;
(d) extracting a pedestrian candidate area map using the background model and the upper and lower lines of the band, and for each pixel of the current frame, extracting a pedestrian candidate area map from the area between the upper end line and the lower end line of the updated background model Extracting the pedestrian candidate area map as the pedestrian candidate area map;
(e) updating the background model using the current frame of the image, using only the background model of the previous frame in the pedestrian candidate area and updating only the remaining area;
(f) inputting the pedestrian candidate area map into a convolutional neural network (CNN) pedestrian classifier and classifying the pedestrian candidate area map; And
(g) counting the pedestrians according to the classification results. < Desc / Clms Page number 19 >
The method according to claim 1,
In the step (a), the background model is initialized by averaging N background frames (N is a natural number of 2 or more)
Wherein, in step (e), N frames including the current frame are averaged up to a previous frame, and the updated N frames are averaged to update the counted number of frames on the embedded platform using the convolution neural network.
The method according to claim 1,
Wherein, in step (e), the background model is updated by the following equation (1): " (1) "
[Equation 1]

However, B _m (i, j) represents a pixel (i, j) of the background model of the m-th frame is the current frame, Ik (i, j) is the pixel value of the pixel (i, j) of the k-th frame , F _m (i, j) is a pedestrian candidate area map indicates whether the pedestrian candidate region and the pixel (i, j) of the m-th frame, N is a natural number of at least 2.
The method of claim 3,
Wherein the top line U_band and the bottom line L_band of the background model are updated using Equation (2) in the step (c).
[Equation 2]

However,? Is a predetermined constant as a weight.
5. The method of claim 4,
Wherein, in the step (d), the pedestrian candidate area map is obtained by the following equation (3).
[Equation 3]

However, F _m (i, j) is the pixel ¹ a represents the pixel value of the pixel of the pedestrian candidate region map (i, j), I _m (i, j) is the pixel (i, j) of the current frame m.
The method according to claim 1,
Wherein the CNN pedestrian classifier comprises a layer of an inception module and the insception module comprises a network in network (NIN) structure. The CNN pedestrian classifier includes a person counting module on an embedded platform using a convolution neural network, Way.
The method according to claim 1,
Wherein the learning image is generated as a plurality of new learning images by applying movement and rotation operations when learning the CNN pedestrian classifier, and then learning is performed.
8. The method of claim 7,
And a learning image is generated by generating a plurality of new learning images according to the following equation (4) for the given learning image (x, y), but with different rotation parameters and parallel movement parameters: Counting method.
[Equation 4]

(X ', y') represents a newly generated training image, a and b are rotation parameters, and c and d are parallel movement parameters, respectively.
The method according to claim 6,
And performing a pruning operation on the inc Sense module.
A computer-readable recording medium having recorded thereon a program for performing a person counting method on an embedded platform using the convolutional neural network according to any one of claims 1 to 9.

Images