KR102332229B1 - Method for Augmenting Pedestrian Image Data Based-on Deep Learning - Google Patents

Abstract

Disclosed is a method for augmenting pedestrian image data based on deep learning.
In this embodiment, a drivable area and a pedestrian are detected from an input image using deep learning-based object detection models. Provided is a pedestrian image data augmentation method capable of automatically generating an augmented image for a pedestrian abnormal behavior situation based on a detected drivable area and masks for the pedestrian.

Description

Method for Augmenting Pedestrian Image Data Based-on Deep Learning

The present invention relates to a method for augmenting pedestrian image data based on deep learning.

The content described below merely provides background information related to the present invention and does not constitute the prior art.

An intelligent system for collecting and analyzing information related to road traffic safety has been widely studied. The intelligent traffic safety system performs road anomaly detection that automatically detects abnormal situations occurring on the road based on images captured by CCTV (Closed-Circuit Television). Since the occurrence cycle of an abnormal situation is usually very long, it is very difficult for a person to continuously monitor and detect the abnormal situation. Therefore, the intelligent traffic safety system aims to reduce the need for people to continuously monitor by automatically detecting abnormal situations such as abnormal behavior of pedestrians.

On the other hand, an image processing technology based on a deep learning model represented by a Convolutional Neural Network (CNN) is widely used in intelligent systems. Since the structure or learning method of the deep learning model can determine the performance of road anomaly detection, the intelligent system employing the deep learning model mainly focused on the use and improvement of the deep learning model. However, most intelligent systems have a problem of quantitative or qualitative limitations of the dataset used for training the used deep learning model. Therefore, some intelligent systems also conduct research on datasets (see Non-Patent Document 3).

The most widely used pedestrian-related dataset is the pedestrian anomaly dataset of the University of California, San Diego (UCSD) (see Non-Patent Document 1). The UCSD dataset contains videos (referred to as Ped1 and Ped2, respectively) taken with two different static cameras, each video showing a pedestrian walkway. Ped1 video contains 34 training videos and 36 evaluation videos in 238 x 158 resolution, and Ped2 video includes 16 training videos and 12 evaluation videos in 360 x 240 resolution. Each video consists of 120 to 200 frames.

Training videos from the UCSD dataset are images of people walking along a pedestrian pathway. The video for evaluation includes abnormal frames, and there are pixelwise spatial labels for the positions of the abnormal objects for each frame.

The UCSD dataset has a disadvantage in that the number of total frames constituting each video is small and the types of abnormal situations are limited. The UCSD dataset contains five types of anomalies, such as the behavior of throwing a bag upward, and the number of frames for anomalies compared to the total frame is high, so the situations included in the video are unnatural. It also has the disadvantage that the label for location information is only provided for some test videos.

As another dataset, there is a subway dataset containing two long videos of subway entrances and exits, which mainly focus on people entering and exiting through ticket gates (see Non-Patent Document 2). Anomalies in the subway dataset include people running around turnstiles, people walking in the wrong direction, or people cleaning walls.

Since the subway dataset only provides two long videos, we need an additional definition of the rate at which frames are extracted, and we don't know exactly which frames are marked as anomalous. Also, there is a disadvantage in that there is no label for the location information of the object that is the target of the abnormal situation.

Another dataset is the avenue dataset from the Chinese University of Hong Kong (CUHK), which contains short video clips taken from a single outdoor camera looking at the side of the building, for a building containing a pedestrian walkway. (See Non-Patent Document 3). CUHK's Avenue dataset contains 16 training videos and 21 evaluation videos with 640 x 360 resolution, including people entering and leaving the building. A total of 47 abnormal events are included in the evaluation video, but there is a disadvantage that the situation included in the video is unnatural because many unusual behaviors occur.

As another dataset, there is a UMN (University of Minnesota) dataset including eleven three short video clips shot around an outdoor field, an outdoor yard, or an indoor lobby (see Non-Patent Document 4). The abnormal situation image in each clip image includes a person who suddenly runs away as an evacuation scenario. Each video clip contains a point in time when a single anomaly event occurs.

In the UMN dataset, the division between the training frame and the evaluation frame is not clear. In addition, the label for the abnormal situation is only temporarily provided for the entire frame of each video, so it has a disadvantage that it is not suitable for use in training.

In addition to the above problems, most of the public datasets do not include data corresponding to traffic accident risk situations such as abnormal behavior according to the location of pedestrians, for example, jaywalking in which pedestrians cross the roadway rather than crosswalk. Therefore, there is a need for a method for generating various learning data for detecting situations such as jaywalking.

Non-Patent Document 1: W. Li, V. Mahadevan, and N. Vasconcelos. Anomaly detection and localization in crowded scenes. PAMI, 2014. Non-Patent Document 2: A. Adam, E. Rivlin, I. Shimshoni, and D. Reinitz. Robust real-time unusual event detection using multiple fixed-location monitors. PAMI, 2008. Non-Patent Document 3: C. Lu, J. Shi, and J. Jia. Abnormal event detection at 150 fps in matlab. In ICCV, 2013. Non-Patent Document 4: Mehran, Ramin, Alexis Oyama, and Mubarak Shah. Abnormal crowd behavior detection using social force model. 2009 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 2009. Non-Patent Document 5: He, Kaiming, et al. "Mask r-cnn." Proceedings of the IEEE international conference on computer vision. 2017. Non-Patent Document 6: Szegedy, Christian, et al "Going deeper with convolutions" Proceedings of the IEEE conference on computer vision and pattern recognition 2015.

The present disclosure detects a drivable area and a pedestrian from an input image using deep learning-based object detection models. By automatically generating an augmented image for a pedestrian abnormal behavior situation based on the detected drivable area and masks for pedestrians, the road anomaly of the intelligent traffic safety system ), the main purpose of which is to provide a pedestrian image data augmentation method capable of improving detection performance.

According to an embodiment of the present invention, there is provided an image augmentation method using a pedestrian image data augmentation apparatus, the method comprising: acquiring an input image; extracting a drivable area from the input image using a deep learning-based area detection model; extracting a pedestrian from the input image using a deep learning-based pedestrian detection model; and generating augmented images of an abnormal behavior situation of the pedestrian based on the drivable area and masks for the pedestrian. Provided is an image augmentation method implemented.

According to another embodiment of the present invention, an input unit for obtaining an input image (image); an area extraction unit for extracting a drivable area from the input image using an area detection model; a pedestrian extraction unit for extracting a pedestrian from the input image using a pedestrian detection model; And Pedestrian image data augmentation device comprising a data augmentation unit for generating augmented images for the abnormal behavior situation of the pedestrian based on the driving area and the pedestrian masks (masks) provides

According to another embodiment of the present invention, there is provided a computer program stored in a computer-readable recording medium to execute each step included in the image augmentation method.

As described above, according to this embodiment, by providing a pedestrian image data augmentation method capable of generating an augmented image for a pedestrian abnormal behavior situation based on deep learning, augmentation Based on the learning process using the collected data set, there is an effect that it becomes possible to improve the detection performance of the intelligent traffic safety system for abnormal behavior of pedestrians in traffic situations.

In addition, according to this embodiment, by providing a pedestrian image data augmentation method capable of generating an augmented image for a pedestrian abnormal behavior situation that may occur in various places based on deep learning, a pedestrian abnormality with a very low probability of occurrence There is an effect that it becomes possible to construct a detection system that can be universally applied to the detection of behavior.

1 is a block diagram of a pedestrian image data augmentation apparatus according to an embodiment of the present invention.
2 is a block diagram of Mask R-CNN, which is a basis for a region detection model and a pedestrian detection model according to an embodiment of the present invention.
3 is a flowchart of a pedestrian image data augmentation method according to an embodiment of the present invention.
4 is an exemplary diagram illustrating an augmented image generation process according to an image augmentation method according to an embodiment of the present invention.
5 is a block diagram of GoogleNet, which is the basis of a classifier model according to an embodiment of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to exemplary drawings. In adding reference numerals to the components of each drawing, it should be noted that the same components are given the same reference numerals as much as possible even though they are indicated on different drawings. In addition, in describing the present embodiments, if it is determined that a detailed description of a related well-known configuration or function may obscure the gist of the present embodiments, the detailed description thereof will be omitted.

Also, in describing the components of the present embodiments, terms such as first, second, A, B, (a), (b), etc. may be used. These terms are only for distinguishing the elements from other elements, and the essence, order, or order of the elements are not limited by the terms. Throughout the specification, when a part 'includes' or 'includes' a certain component, it means that other components may be further included, rather than excluding other components, unless otherwise stated. . In addition, the '... Terms such as 'unit' and 'module' mean a unit that processes at least one function or operation, which may be implemented as hardware or software or a combination of hardware and software.

DETAILED DESCRIPTION The detailed description set forth below in conjunction with the appended drawings is intended to describe exemplary embodiments of the present invention and is not intended to represent the only embodiments in which the present invention may be practiced.

This embodiment discloses details on a method for augmenting pedestrian image data based on deep learning. In more detail, a drivable area and a pedestrian are detected from an input image using deep learning-based object detection models. Provided is a pedestrian image data augmentation method capable of automatically generating an augmented image for a pedestrian abnormal behavior situation based on a detected drivable area and masks for a person.

1 is a block diagram of a pedestrian image data augmentation apparatus according to an embodiment of the present invention.

In an embodiment of the present invention, the pedestrian image data augmentation apparatus 100 (hereinafter, 'image augmentation apparatus') detects a drivable area and a person from an input image, and based on the detected drivable area and a mask for the person Generates augmented images for pedestrian abnormal behavior situations. The image augmentation apparatus 100 includes all or a part of the input unit 101 , the region extraction unit 102 , the pedestrian extraction unit 103 , and the data augmentation unit 104 . Here, components included in the image augmentation apparatus 100 according to the present embodiment are not necessarily limited thereto. For example, a training unit (not shown) for training the detection model may be additionally provided on the image augmentation apparatus 100 , or may be implemented in the form of interworking with an external training unit.

The input unit 101 according to the present embodiment acquires an input image. It is assumed that the input image includes a normal situation and includes a drivable area and/or a plurality of pedestrians.

The area extraction unit 102 according to the present embodiment extracts the drivable area from the input image based on an area detection model. The region detection model is a deep learning-based object detection model, and may be trained in advance using learning data including a drivable region. The region extraction unit 102 uses a Mask Region-based Convolutional Neural Network (R-CNN) as a region detection model (see Non-Patent Document 5).

The pedestrian extraction unit 103 according to the present embodiment extracts the drivable area from the input image based on a pedestrian detection model. The pedestrian detection model is a deep learning-based object detection model, and may be trained in advance using data for learning including people. The pedestrian detection unit 103 also uses Mask R-CNN as a pedestrian detection model.

2 is a block diagram of Mask R-CNN, which is a basis for a region detection model and a pedestrian detection model according to an embodiment of the present invention. Mask R-CNN includes a path for detecting a mask for an object in parallel with an object detection path that generates an ID (identification) of an object and a bounding box based on a region of interest (RoI).

The data augmentation unit 104 according to the present embodiment generates an augmented image for a pedestrian abnormal behavior situation, such as jaywalking, based on the extracted drivable area and the mask for the pedestrian.

The device (not shown) on which the image augmentation apparatus 100 according to the present embodiment is mounted may be a programmable computer and includes at least one communication interface that can be connected to a server (not shown).

Training for the area detection model and the pedestrian detection model as described above may be performed in the device using the computing power of the device on which the image augmentation apparatus 100 is mounted.

Training for the area detection model and the pedestrian detection model as described above may be performed in the server. The training unit of the server may perform training on the deep learning model having the same structure as the area detection model and the pedestrian detection model, which are components of the image augmentation apparatus 100 mounted on the device. Using a communication interface connected to the device, the server transmits the parameters of the trained deep learning model to the device, and the image augmentation apparatus 100 uses the received parameters to update the parameters of the area detection model and the pedestrian detection model. have. Also, parameters of the area detection model and the pedestrian detection model may be set when the device is shipped or when the image augmentation apparatus 100 is mounted on the device.

Hereinafter, a pedestrian image data augmentation method (hereinafter, 'image augmentation method') using the image augmentation apparatus 100 will be described with reference to FIGS. 3 and 4 .

3 is a flowchart of a pedestrian image data augmentation method according to an embodiment of the present invention. 4 is an exemplary diagram illustrating an augmented image generation process according to an image augmentation method according to an embodiment of the present invention.

The training unit of the image augmentation apparatus 100 performs pre-training on the area detection model and the pedestrian detection model (S301).

Each of the area detection model and the pedestrian detection model is a deep learning-based object detection model based on Mask R-CNN. The region detection model is trained in advance using a vehicle black box dataset called BDD100K (Black box Driving Dataset 100K) as training data. The BDD100K includes approximately 100,000 black box video clips, and most of the images include roads, so that the training unit can pre-train the area detection model to be specialized in the detection of the drivable area.

The pedestrian detection model is trained in advance using a dataset called MS-COCO 2014 (Microsoft Common Object in Context 2014) as training data. The MS-COCO dataset includes images for object detection and segmentation, etc., and using an image containing a person among the images included in the dataset, the training unit develops a pedestrian detection model to specialize in pedestrian detection. It can be pre-trained.

The image augmentation apparatus 100 acquires an input image (S302). As the input image, the Avenue CCTV image collected at CUHK (see Non-Patent Document 3) and the CCTV image collected from the street of Y city were used. The input image includes a normal situation, and includes a drivable area and at least one pedestrian as shown in FIG. 4 .

The image augmentation apparatus 100 detects the drivable area from the input image using the area detection model (S303). The region detection model may detect a mask for the drivable region from the input image as shown in FIG. 4 .

The image augmentation apparatus 100 detects a pedestrian from the input image using the pedestrian detection model (S304). The pedestrian detection model may detect a mask for the pedestrian from the input image as shown in FIG. 4 .

로 표현한 후, 수학식 1에 표현된 규칙을 기반으로 한 명의 보행자를 선택한다.When the input image includes a plurality of pedestrians, the following rule is followed. The pedestrian detection model detects all included pedestrians, but detects pedestrians according to the detection order.

After expressing as , one pedestrian is selected based on the rule expressed in Equation (1).

Here, D is the drivable area included in the input image. According to the rule expressed in Equation 1, a pedestrian having the earliest detection order is selected with respect to a detected pedestrian that is not included in the drivable area.

The image augmentation apparatus 100 generates an augmented image for a pedestrian abnormal behavior situation based on the drivable area and the pedestrian's mask (S305). The image augmentation apparatus 100 generates a composite image for a jaywalking situation by placing the pedestrian's mask within the drivable area mask as shown in FIG. 4 . In addition, by combining a plurality of pedestrian masks and a pedestrian mask to which a horizontal flip is applied, and a plurality of drivable area masks, an augmented image for a pedestrian abnormal behavior situation may be generated.

The augmented image for the jaywalking situation is classified as an abnormal situation and includes a label for the location information of pedestrians performing abnormal behavior. The augmented image and the label included in the augmented image may be used later in the training process for the intelligent traffic safety system.

Hereinafter, experimental examples and experimental results in which a dataset including an augmented image generated based on the image augmentation method according to the present embodiment is applied to training of an intelligent traffic safety system will be described.

In the experimental example, as an example of a traffic safety system, a classifier model that distinguishes the jaywalking situation from the normal situation from the input image was used. Deep learning-based GoogLeNet was used as the classifier model (see Non-Patent Document 6). As shown in Fig. 5, GoogLeNet is a deep layered structure-based CNN. In order to improve classification performance, the depth of the neural network, that is, the number of layers is increased, but each layer is simplified to suppress the increase in the amount of computation. It's CNN.

The classifier model is pre-trained based on the ImageNet dataset containing more than 20,000 image categories, and then retrained with an added layer for classifying trespassing.

An augmented image dataset for retraining the classifier model, ie, an image including a jaywalking situation, may be generated based on the image augmentation method as described above. In the experimental example, a total of 500 augmented jogging images were generated. After retraining the classifier model using the augmented jabs image and the equal number of normal situation images, classification accuracy (hereinafter, 'accuracy') was measured using the validation dataset.

The validation dataset contains 337 natural jaywalking images and 400 normal situation images. The data set for verification was collected from CCTV images of Google or Y city.

For comparison, we used a classifier model retrained using a natural dataset including natural gait images and normal situation images.

Table 1 shows the measured accuracy of the classifier model to which this embodiment is applied and the classifier model using a natural dataset.

In Table 1, the accuracy is expressed by Equation 2.

Here, a positive means a jaywalking situation and a negative means a normal situation. Also, each of TP and TN properly classifies jaywalking and normal situation, FN classifies trespassing as normal situation, and FP classifies normal situation as trespassing.

As shown in Table 1, the accuracy of the classifier model using the image augmentation method according to the present embodiment was improved by 8% or more with respect to the comparison target. In particular, the overall performance of the classifier model for trespassing classification was improved as the FP case, which classifies the normal situation as a trespassing situation, was reduced to about 1/5 or less compared to the comparison target.

As described above, according to this embodiment, by providing a pedestrian image data augmentation method capable of generating an augmented image for a pedestrian abnormal behavior situation based on deep learning, augmentation Based on the learning process using the collected data set, there is an effect that it becomes possible to improve the detection performance of the intelligent traffic safety system for abnormal behavior of pedestrians in traffic situations.

In addition, according to this embodiment, by providing a pedestrian image data augmentation method capable of generating an augmented image for a pedestrian abnormal behavior situation that may occur in various places based on deep learning, a pedestrian abnormality with a very low probability of occurrence There is an effect that it becomes possible to construct a detection system that can be universally applied to the detection of behavior.

Although it is described that each process is sequentially executed in each flowchart according to the present embodiment, the present invention is not limited thereto. In other words, since it may be applicable to change and execute the processes described in the flowchart or to execute one or more processes in parallel, the flowchart is not limited to a time-series order.

Various implementations of the systems and techniques described herein include digital electronic circuitry, integrated circuits, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), computer hardware, firmware, software, and/or combination can be realized. These various implementations may include being implemented in one or more computer programs executable on a programmable system. The programmable system includes at least one programmable processor (which may be a special purpose processor) coupled to receive data and instructions from, and transmit data and instructions to, a storage system, at least one input device, and at least one output device. or may be a general-purpose processor). Computer programs (also known as programs, software, software applications or code) contain instructions for a programmable processor and are stored on a "computer-readable recording medium".

The computer-readable recording medium includes all types of recording devices in which data readable by a computer system is stored. These computer-readable recording media are non-volatile or non-transitory, such as ROM, CD-ROM, magnetic tape, floppy disk, memory card, hard disk, magneto-optical disk, and storage device. media, and may further include transitory media such as carrier waves (eg, transmission over the Internet) and data transmission media. In addition, the computer-readable recording medium is distributed in network-connected computer systems, and computer-readable codes may be stored and executed in a distributed manner.

Various implementations of the systems and techniques described herein may be implemented by a programmable computer. Here, the computer includes a programmable processor, a data storage system (including volatile memory, non-volatile memory, or other types of storage systems or combinations thereof), and at least one communication interface. For example, a programmable computer may be one of a server, a network appliance, a set-top box, an embedded device, a computer expansion module, a personal computer, a laptop, a Personal Data Assistant (PDA), a cloud computing system, or a mobile device.

The above description is merely illustrative of the technical idea of this embodiment, and various modifications and variations will be possible by those skilled in the art to which this embodiment belongs without departing from the essential characteristics of the present embodiment. Accordingly, the present embodiments are intended to explain rather than limit the technical spirit of the present embodiment, and the scope of the technical spirit of the present embodiment is not limited by these embodiments. The protection scope of this embodiment should be interpreted by the following claims, and all technical ideas within the equivalent range should be interpreted as being included in the scope of the present embodiment.

100: image augmentation device 101: input unit
102: area extraction unit 103: pedestrian extraction unit
104: data augmentation unit

Claims (11)

In the image augmentation method using the pedestrian image data augmentation apparatus,
obtaining an input image;
extracting a drivable area from the input image using a deep learning-based area detection model;
extracting a pedestrian from the input image using a deep learning-based pedestrian detection model; and
A process of generating augmented images of an abnormal behavior situation of the pedestrian based on the drivable area and masks for the pedestrian
including,
The process of generating the augmented image is,
By generating a composite image in which the mask for the pedestrian is positioned within the mask for the drivable area, and combining the mask of the pedestrian and the mask of the pedestrian to which the horizontal flip is applied, and the mask of the drivable area An image augmentation method implemented on a computer, characterized in that generating an augmented image for an abnormal behavior situation of a pedestrian. According to claim 1,
The image augmentation method implemented on a computer, characterized in that it further comprises the step of performing pre-training on the area detection model and the pedestrian detection model based on datasets for learning. According to claim 1,
The input image is
An image augmentation method implemented on a computer, characterized in that it includes the drivable area and/or at least one pedestrian. According to claim 1,
When the input image includes a plurality of pedestrians, the image augmentation method implemented on a computer, characterized in that selecting one pedestrian extracted first from an area excluding the drivable area from among the input image. delete delete According to claim 1,
The augmented image is
Classified as an abnormal situation, the image augmentation method implemented on a computer, characterized in that it includes a label for the location information of the pedestrian performing the abnormal behavior. an input unit for acquiring an input image;
an area extraction unit for extracting a drivable area from the input image using an area detection model;
a pedestrian extraction unit for extracting a pedestrian from the input image using a pedestrian detection model; and
A data augmentation unit that generates augmented images for an abnormal behavior situation of the pedestrian based on the drivable area and masks for the pedestrian
including,
The augmented image is
a composite image in which the mask for the pedestrian is positioned within the mask for the drivable area; and
Augmented image for abnormal behavior situation of the pedestrian, generated by combining the mask of the pedestrian and the mask of the drivable area to which the mask of the pedestrian and the horizontal flip are applied
Pedestrian image data augmentation device comprising a. 9. The method of claim 8,
The area detection model and the pedestrian detection model are implemented as a deep learning-based neural network, and are trained in advance based on training datasets. 9. The method of claim 8,
The augmented image is
Pedestrian image data augmentation apparatus classified as an abnormal situation and comprising a label for location information of a pedestrian who executes the abnormal behavior. A computer program stored in a computer-readable recording medium to execute each step included in the image augmentation method according to any one of claims 1 to 4 or 7.

Images