KR102504320B1 - Garbage dumper detection system and method using multi-characteristic information in surveillance camera environment - Google Patents

Abstract

The present invention relates to a garbage dump actor detection system and method using multiple feature information in a surveillance camera environment, comprising: a feature database storing feature data for extracting features of a target; An image acquisition unit for obtaining an image captured by a pre-installed camera; a target setting unit that sets a person holding the object as a feature extraction target by using a deep learning model pre-learned from the image acquired through the image acquisition unit; a feature extraction unit for extracting feature data for determining whether a specific action is performed in a set feature extraction target; and a specific action detection unit for detecting a garbage dumping behavior using the feature information extracted through the feature extraction unit.

Description

Garbage dumper detection system and method using multi-characteristic information in surveillance camera environment}

The present invention relates to a system and method for detecting a garbage dump actor using multi-characteristic information in a surveillance camera environment, and more particularly, to a system and method for detecting a garbage dump actor of a specific behavior from a captured image provided through a surveillance camera. will be.

The development of artificial intelligence for the environment and the public is a hot topic. Many environments are being destroyed due to the indiscriminate use and disposal of plastics and garbage. In addition, as a public problem, a lot of social costs are incurred in clearing the heap of garbage dumped without permission. In order to solve this problem, the controller uses a method using a CCTV camera or a simple motion sensor, but the effect of use is insufficient due to inefficiency.

There is a method for detecting an object left behind as a similar research in the past. If the movement that occurred once persists for a long time, we use the assumption that a particular object was brought into the scene by a person and then left behind.

However, this method has a problem in that even in the case of a parked vehicle, it is determined as a left object, and there is a limit in that the abandoned object is clearly visible without being covered.

[0003] Conventional garbage dumping detection method techniques make great use of pre-determined assumptions.

For example, there are many restrictions such as a person appearing and staying in an area where garbage is dumped, one or two moving objects appearing under a dump surveillance camera, or movement around a hand.

The present invention has been made to solve the conventional problems, and an apparatus and method for detecting a person's posture and the shape of background information occurring in an actual dumping act through a learning-based method and notifying the user of the event. would like to present

In addition, the present invention is intended to provide a garbage dumping detection system and method that supports not only garbage dumping behavior but also general action classification through an additional general behavior classifier, and uses the same to provide a garbage dumping detection system and method with fewer false positives.

The object of the present invention is not limited to the object mentioned above, and other objects not mentioned will be clearly understood by those skilled in the art from the description below.

In order to achieve the above object, a system for detecting a garbage dump actor using multiple feature information in a surveillance camera environment according to an embodiment of the present invention includes a feature database storing feature data for extracting features of a target; An image acquisition unit for obtaining an image captured by a pre-installed camera; a target setting unit that sets a person holding the object as a feature extraction target by using a deep learning model pre-learned from the image acquired through the image acquisition unit; a feature extraction unit for extracting feature data for determining whether a specific action is performed in a set feature extraction target; and a specific action detection unit for detecting a garbage dumping behavior using the feature information extracted through the feature extraction unit.

The image acquisition unit is an IP camera accessed through the web.

The feature database may include image data around an object extracted from an image taken by an image acquisition unit in order to extract features of the object; joint data for extracting joint area data of the extracted object; movement data for extracting a specific behavior of the extracted target; And, in order to identify an object held by a target selected through the motion data, surrounding background data in an image captured by the image acquisition unit is stored and learned.

As for the objects held by the target, only objects that can be held by a person are learned.

The feature extraction unit may include: an image extraction unit for extracting image data around the object from the acquired image; a target joint area estimator for extracting joint area data of the extracted subject; a motion extractor around the object to extract garbage dumping motion data of the extracted object; and a background extracting unit for extracting surrounding background data of an object in which the movement of the object extracted through the motion extracting unit around the object is performed.

The feature extractor further includes a multi-feature learner for learning a plurality of multi-feature information extracted using a deep neural network (DNN), and the multi-feature learner learns a trash dumping image and a general action recognition image, respectively. .

The specific action detection unit uses a CCTV data classifier and uses at least one of a detailed human state classifier, a human posture predictor, and a human state classifier to verify speculation behavior detected through the specific behavior detection unit. It includes a behavior verification unit.

Further, the speculation behavior verification unit determines whether to send an event to the controller according to whether or not the speculation behavior detected through the specific behavior detection unit is verified.

In addition, a method for detecting a garbage dump actor using multi-characteristic information in a surveillance camera environment according to an embodiment of the present invention includes the steps of acquiring, by an image acquisition unit, an image captured through a pre-installed camera; setting a person holding an object as a feature extraction target by using a deep learning model pre-learned from the acquired image by a target setting unit; extracting, by a feature extraction unit, feature data for determining whether a specific action is performed in the set feature extraction target; and detecting, by a specific behavior detection unit, a garbage dumping behavior using the extracted characteristic information.

The step of acquiring the image is an IP camera accessed through the web, and it is preferable to access using an address that can access the Internet through the web after being authenticated through the IP, ID, and password.

As for the objects held by the target, only objects that can be held by a person can be learned.

The feature extraction step may include extracting image data around the object from the obtained image by an image extractor around the object; extracting joint region data of the extracted target joint region by a target joint region estimator; extracting garbage dumping motion data of the extracted object by a motion extractor around the object; and extracting, by a background extracting unit around the target, surrounding background data of the object in which the motion of the extracted target is performed.

In addition, a step of detecting a speculation behavior using a CCTV data classifier and a step of determining, by the speculation behavior verification unit, to send an event to a controller according to whether or not the speculation behavior detected through the specific behavior detection unit is verified; do.

In the step of verifying the detected speculative behavior, the speculative behavior verifier detects the speculative action through the specific behavior detector using at least one of a detailed state classifier, a human posture predictor, and a human state classifier. It is desirable to perform verification of speculation behavior.

According to another embodiment of the present invention, a garbage dumping agent detection learning method using multi-characteristic information in a surveillance camera environment includes the steps of storing image data around a target in an acquired image by an image acquiring unit; Storing joint data for extracting joint area data of a subject; storing motion data of an extracted object in order to determine a specific motion; and storing, by the image acquiring unit, surrounding background data of the object in the obtained image in order to determine an object that the extracted object is holding.

The step of storing each data further includes learning each of a plurality of multi-characteristic information extracted using a deep neural network (DNN).

In the step of storing each data, it is preferable to learn each of a garbage dumping image and a general action recognition image.

According to an embodiment of the present invention, a classifier capable of distinguishing speculative behavior from non-speculative behavior is learned through images collected in various ways as actual learning data, resulting in fewer false positives and improved There is an effect that can produce results.

1 is a functional block diagram for explaining a garbage dumping actor detection system using multi-characteristic information in a monitoring camera environment according to an embodiment of the present invention.
2 is a view for explaining an example of an image captured through an image acquisition unit in one embodiment of the present invention.
3 is a functional block diagram for explaining in detail the feature extraction unit of FIG. 1;
4 to 7 are reference diagrams for explaining a process for detecting a specific behavior in the garbage dump actor detection system using multi-characteristic information in a monitoring camera environment according to an embodiment of the present invention.
8 is a reference diagram for explaining a state in which a plurality of objects are set in an image obtained in an embodiment of the present invention;
9 is a reference diagram for explaining objects held by a plurality of subjects in an image obtained in an embodiment of the present invention;
10 is a functional block diagram for explaining a process for verifying a specific action in an embodiment of the present invention.
11 is a flowchart illustrating a method of detecting a garbage dump actor using multi-characteristic information in a surveillance camera environment according to an embodiment of the present invention.
12 is a flowchart for explaining the feature data extraction step of FIG. 11 .

Advantages and features of the present invention, and methods of achieving them, will become clear with reference to the detailed description of the following embodiments taken in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but will be implemented in various different forms, only these embodiments make the disclosure of the present invention complete, and common knowledge in the art to which the present invention belongs. It is provided to fully inform the holder of the scope of the invention, and the present invention is only defined by the scope of the claims. Meanwhile, terms used in this specification are for describing the embodiments and are not intended to limit the present invention. In this specification, singular forms also include plural forms unless specifically stated otherwise in a phrase. As used herein, "comprises" and/or "comprising" means that a stated component, step, operation, and/or element is the presence of one or more other components, steps, operations, and/or elements. or do not rule out additions.

1 is a functional block diagram for explaining a garbage dump actor detection system using multi-characteristic information in a monitoring camera environment according to an embodiment of the present invention.

As shown in FIG. 1, the garbage dumping agent detection system using multiple feature information in a monitoring camera environment according to an embodiment of the present invention includes a feature database 100, an image acquisition unit 200, and a target setting unit 300. , a feature extraction unit 400 and a specific action detection unit 500 are included.

The feature database 100 stores feature data for extracting features of an object.

The image acquiring unit 200 acquires an image captured through a pre-installed camera. 2 is an example of an image captured through the image acquisition unit 200. This image acquisition unit 200 may be a CCTV camera, an IP camera accessed through the web, and after being authenticated through IP, ID, and password, it is accessed using an address capable of accessing the Internet through the web.

As shown in FIG. 2 , the target setting unit 300 sets a person holding an object as a feature extraction target by using a pre-learned deep learning model from an image obtained through the image acquiring unit 200 . The target setting unit 300 may detect a person in an image captured by the image acquisition unit 200 and an object owned by the person.

The feature extraction unit 400 extracts feature data for determining whether a specific action is performed in a set feature extraction target. Here, a specific action in one embodiment of the present invention is an action of dumping garbage.

FIG. 3 is a block diagram illustrating the feature extraction unit 400 of FIG. 1 .

As shown in FIG. 3 , the feature extraction unit 400 according to an embodiment of the present invention includes an image extraction unit 410 around the target, a target joint area estimation unit 420, and a movement extraction unit 430 around the target. ) and a background extraction unit 440 around the target.

The image extractor 410 extracts image data around the object from the image, as shown in FIG. 4 , using specific information stored in the feature database 100 .

The target joint region estimator 420 extracts joint region data of the extracted target, as shown in FIG. 5 , from an acquired image using specific information stored in the feature database 100 .

The movement extractor 430 around the object extracts garbage dumping movement data of the extracted object, as shown in FIG. 7 , from an acquired image using specific information stored in the feature database 100 .

The surrounding background extractor 440 uses the specific information stored in the feature database 100, as shown in FIG. extract data

Thereafter, the specific behavior detection unit 500 detects a specific behavior, such as garbage dumping, by using the feature information extracted through the feature extraction unit 400 .

For example, as shown in FIG. 2 , when image information captured by the image acquisition unit 200 is collected, the target setting unit 300 performs deep learning pre-learned from the image acquired through the image acquisition unit 200. Using the model, a feature extraction target for a person holding an object and an object (T1) held by the target are set. Meanwhile, as shown in FIG. 8 , a feature extraction target may be a plurality of people O1 to O3 set as feature extraction targets or an object T1 held by the corresponding target.

To this end, the feature extractor 400 extracts feature data for determining whether a specific action is performed in a set feature extraction target.

First, the image extractor 410 extracts image data around the subject from the image using specific information stored in the feature database 100 .

The target joint region estimator 420 extracts joint region data of the target extracted from an acquired image using specific information stored in the feature database 100 .

The movement extraction unit 430 around the object extracts garbage dumping movement data of the object extracted from the acquired image using specific information stored in the feature database 100 .

The background extractor 440 extracts surrounding background data of an object in which the motion of the object extracted through the object motion extractor 430 is performed using specific information stored in the feature database 100 .

Thereafter, the specific behavior detection unit 500 detects a specific behavior, such as garbage dumping, by using the feature information extracted through the feature extraction unit 400 .

The specific behavior detection unit 500 may detect a characteristic behavior of a feature extraction target extracted by using the feature data extracted by the feature extraction unit 400 .

That is, the specific action detection unit 500 compares image data around the extracted feature with respect to the extracted feature extraction target to detect an object held by a person, that is, garbage. In this way, when an object being held by a person is detected, only the object holding the object may be selected and precisely analyzed.

Further, the feature action detection unit 500 compares joint area data with respect to the extracted feature extraction target to detect garbage dumping behavior. That is, by comparing joint area data at the time of garbage dumping, it is possible to detect whether the object is intended to perform garbage dumping behavior.

In addition, the feature action detection unit 500 may detect that the detected object performs a garbage dumping motion by comparing the extracted feature extraction motion data with target garbage dumping motion data.

In addition, the feature action detection unit 500 compares the extracted feature extraction target with surrounding background data of the target to determine whether the target is holding a characteristic object or detects whether the object is dumping the object that the target is holding so as to detect a garbage dumping target. can do it accurately

Meanwhile, the feature extractor 400 may further include a multi-feature learner for learning each of a plurality of multi-feature information extracted using a deep neural network (DNN). Here, it is preferable to learn only objects that can be held by a person as objects held by the hand in the present embodiment. In this way, by limiting the learning target, the object can be learned more accurately.

Meanwhile, the multi-feature learning unit may learn a trash dumping image and a general action recognition image, respectively. In this way, by learning the garbage dumping image and the general action recognition image respectively, the learning target image data can be learned and the accuracy thereof can be increased.

10 is a functional block diagram for explaining a process for verifying a specific action in an embodiment of the present invention.

Meanwhile, in an embodiment of the present invention, the detection accuracy may be increased by verifying the specific action extracted through the specific action extraction unit 500 .

To this end, in an embodiment of the present invention, a target for dumping garbage is detected using a CCTV data classifier, and a specific action is detected using at least one of a detailed human state classifier, a human posture predictor, and a human state classifier. A speculation behavior verifying unit 600 performing verification of the speculative behavior detected through the unit 500 may be included. Here, the person's detailed state classifier is an MPHB classifier capable of classifying a person's comprehensive behavioral characteristics (eg, standing, lying, sitting, etc.), and the person's state classifier is a person's detailed behavioral characteristic (eg, riding a bicycle , push a cart, etc.) is a Standford 40 classifier.

Further, the speculation behavior verifying unit 600 may determine whether to send an event to the controller according to whether or not the speculative behavior detected through the specific behavior detection unit 500 is verified.

Hereinafter, a garbage dump actor detection method using multi-characteristic information in a surveillance camera environment according to an embodiment of the present invention will be described with reference to FIG. 11 .

First, an image captured through a pre-installed camera is acquired by the image acquisition unit 200 (S100). The step of acquiring the image (S100) is preferably an IP camera accessed through the web, and can be accessed using an address that can access the Internet through the web after being authenticated through IP, ID, and password.

Thereafter, the target setting unit 300 sets a person holding an object as a feature extraction target using a deep learning model pre-learned from the obtained image (S200).

Next, the feature extraction unit 400 extracts feature data for determining whether the set feature extraction target performs an act of dumping garbage (S300).

Hereinafter, the feature extraction step (S300) according to an embodiment of the present invention will be described with reference to FIG. 12.

First, image data around the subject is extracted from the obtained image by the image extraction unit 410 around the subject (S310).

Then, the joint region data of the extracted target joint region is extracted by the target joint region estimation unit 420 (S320).

In addition, the motion data around the object is extracted by the motion extractor 430 for dumping garbage of the extracted object (S330).

Background data around the object where the motion of the extracted object has been performed is extracted by the object surrounding background extractor 440 (S340).

Thereafter, the specific behavior detection unit 500 detects a garbage dumping behavior using the extracted feature information (S400).

Thereafter, the detected garbage dumping behavior is verified (S500). That is, the detection accuracy may be increased by verifying the specific action extracted through the specific action extractor 500 .

In the verifying step, the dumping behavior verifying unit 600 may detect a target for dumping garbage using the CCTV data classifier. Further, in the verification step, the speculation behavior detected through the specific behavior detection unit 500 may be verified using at least one of the detailed human state classifier, the human posture predictor, and the human state classifier. Here, the person's detailed state classifier is an MPHB classifier capable of classifying a person's comprehensive behavioral characteristics (eg, standing, lying, sitting, etc.), and the person's state classifier is a person's detailed behavioral characteristic (eg, riding a bicycle , push a cart, etc.) is a Standford 40 classifier.

In addition, according to whether or not the detected speculation behavior is verified by the speculation behavior verifying unit 600, it may be determined to send the event to the controller.

On the other hand, only objects that can be held by a person can be learned as objects that the target is holding by hand.

In the garbage dump actor learning method using multi-characteristic information in a monitoring camera environment according to an embodiment of the present invention, image data around an object in an acquired image is stored in the feature database 100. Such image data is used to detect an object that the detected object is holding by hand or an object piled up in a garbage pile.

Subsequently, joint data for extracting joint area data of the target is stored in the feature database 100 . Here, the joint data is used to detect whether the target performs a specific action.

Then, in order to determine a specific movement, the extracted movement data of the object is stored in the feature database 100 . Here, motion data is used to detect whether a target performs a specific action. To this end, in the case of joint data and motion data, by learning using specific actions and general action recognition images as data, not only images captured through actual CCTV, but also images captured through other video media to detect specific movements can be used

In addition, in order to determine the object that the extracted object is holding, the surrounding background data of the object in the obtained image is stored in the feature database 100 .

Here, the surrounding background data is used as data for detecting an object that actually dumps garbage, that is, an object that performs a specific action, when a plurality of objects are detected in an image captured through CCTV. Therefore, it is desirable to store and learn data obtained from images captured through actual CCTV.

Here, in the step of storing each data, it is preferable to learn each of a plurality of multi-characteristic information extracted using a deep neural network (DNN).

Also, in the step of storing each data, a trash dumping image and a general action recognition image may be learned.

In the above, the configuration of the present invention has been described in detail with reference to the accompanying drawings, but this is only an example, and various modifications and changes within the scope of the technical idea of the present invention to those skilled in the art to which the present invention belongs Of course this is possible. Therefore, the scope of protection of the present invention should not be limited to the above-described embodiments, but should be defined by the description of the claims below.

Claims (20)

Image data around the object extracted from the image captured by the image acquisition unit, joint data for extracting joint area data of the extracted object, extracted using DNN (Deep Neural network) to extract features of the extracted object, extracted In order to identify the motion data for extracting a specific behavior of the target and the object held by the target selected through the motion data, the surrounding background data in the image captured by the image acquisition unit is learned and stored to extract the characteristics of the target. a feature database storing feature data;
An image acquisition unit for obtaining an image captured by a pre-installed camera;
a target setting unit that sets a person holding the object as a feature extraction target by using a deep learning model pre-learned from the image acquired through the image acquisition unit;
a feature extraction unit for extracting feature data for determining whether a specific action is performed in a set feature extraction target; and
A specific action detection unit for detecting a garbage dumping behavior using the feature information extracted through the feature extraction unit,
The feature extraction unit,
an image extraction unit for extracting image data around the object from the obtained image; a target joint area estimator for extracting joint area data of the extracted subject; a motion extractor around the object to extract garbage dumping motion data of the extracted object; and a background extracting unit for extracting surrounding background data of an object in which a motion of the object extracted through the motion extracting unit around the object is performed,
The specific action detection unit
Garbage dumping using multi-characteristic information in a surveillance camera environment, characterized in that the extracted feature extraction target is compared with the surrounding background data of the target to determine whether the target is holding a characteristic object or to detect whether the target throws the object it is holding actor detection system.
According to claim 1,
The image acquisition unit,
A garbage dump actor detection system using multi-characteristic information in a surveillance camera environment, which is an IP camera accessed through the web.
delete According to claim 1,
The object held by the target is
A garbage dump actor detection system using multi-characteristic information in a surveillance camera environment that learns only objects that people can lift.
delete delete According to claim 1,
The multi-feature learning unit,
A garbage dumping actor detection system using multi-characteristic information in a surveillance camera environment that learns garbage dumping images and general behavior recognition images separately.
According to claim 1,
The specific action detection unit,
A garbage dump actor detection system using multi-characteristic information in a surveillance camera environment using a CCTV data classifier.
According to claim 8,
Multi-characteristic information in a surveillance camera environment including a speculative behavior verification unit that verifies the speculative behavior detected through the specific behavior detection unit using at least one of a detailed human state classifier, a human posture predictor, and a human state classifier. A garbage dumping agent detection system using .
According to claim 9,
The speculation behavior verification unit,
A system for detecting trash dumping actors using multi-characteristic information in a surveillance camera environment, which determines whether to send an event to a controller according to whether the dumping behavior detected through the specific behavior detection unit is verified.
Obtaining an image captured by a pre-installed camera by an image acquisition unit;
setting a person holding an object as a feature extraction target by using a deep learning model pre-learned from the acquired image by a target setting unit;
extracting, by a feature extraction unit, feature data for determining whether a specific action is performed in the set feature extraction target; and
Detecting, by a specific behavior detection unit, a garbage dumping behavior using the extracted feature information,
In the feature extraction step,
extracting image data around the object from the obtained image by an image extraction unit around the object; extracting joint region data of the extracted target joint region by a target joint region estimator; extracting garbage dumping motion data of the extracted object by a motion extractor around the object; and extracting, by a target surrounding background extracting unit, surrounding background data of an object in which the extracted object moves,
The step of detecting the garbage dumping act,
Garbage dumping using multi-characteristic information in a surveillance camera environment, characterized in that the extracted feature extraction target is compared with the surrounding background data of the target to determine whether the target is holding a characteristic object or to detect whether the target throws the object it is holding Actor detection methods.
According to claim 11,
The step of obtaining the image is,
A method for detecting a garbage dump actor using multi-characteristic information in a surveillance camera environment, which is an IP camera accessed through the web.
According to claim 12,
The step of obtaining the image is,
A method of detecting a garbage dump actor using multi-characteristic information in a surveillance camera environment, which is accessed using an address that can access the Internet through the web after being authenticated through IP, ID, and password.
According to claim 11,
The object that the subject is holding by hand,
A garbage dump actor detection method using multi-characteristic information in a surveillance camera environment that learns only objects that people can lift.
delete According to claim 11,
Further comprising the step of verifying the detected dumping behavior after the step of detecting the dumping behavior;
The step of verifying the detected speculation behavior,
Detecting speculation behavior using a CCTV data classifier; and
A method of detecting a garbage dumping actor using multi-characteristic information in a surveillance camera environment, comprising: determining whether to send an event to a controller according to whether the dumping behavior detected through the specific behavior detection unit is verified by the dumping behavior verification unit.
According to claim 16,
The step of verifying the detected speculation behavior,
In a surveillance camera environment in which the speculation behavior verification unit verifies the speculation behavior detected through the specific behavior detection unit using at least one of a detailed human state classifier, a human posture predictor, and a human state classifier. A method for detecting garbage dumping actors using multi-characteristic information.
delete delete delete

Images