KR20210062256A - Method, program and system to judge abnormal behavior based on behavior sequence - Google Patents

Abstract

A method, program, and system for judging an abnormal behavior based on a behavior sequence are provided. The method, performed by a computer, for judging the abnormal behavior based on the behavior sequence may include the steps of: receiving image data; recognizing one or more objects in the image data; recognizing a plurality of unit actions for each of the recognized objects; classifying the plurality of unit behaviors into normal behaviors or abnormal behaviors as a behavior category; generating sequence data by sequentially sequencing the plurality of unit actions; and judging an abnormal behavior based on a ratio of unit behaviors in which the behavior category is classified as the abnormal behaviors in the sequence data. When there are a plurality of objects, it is possible to recognize the behavior while tracking each of the objects.

Description

Method, program and system to determine abnormal behavior based on behavior sequence {METHOD, PROGRAM AND SYSTEM TO JUDGE ABNORMAL BEHAVIOR BASED ON BEHAVIOR SEQUENCE}

The present invention relates to a method, program, and system for determining abnormal behavior based on an action sequence, and more particularly, a method and program for determining whether an abnormal behavior corresponds to an abnormal behavior by recognizing an object's behavior and analyzing it based on an action sequence. And a system.

Today, incidents such as theft, extortion, and assault and accidents such as fire and safety occur frequently in stores, stores, and affiliated stores in various fields. In order to prevent such an event or an accident from occurring, or to quickly respond when it occurs, the role of a control system that takes an image of the field and uses the captured image is becoming important.

CCTV (closed-circuit television) is a system that transmits an image to a specific recipient for a specific purpose, and is called a closed circuit television. CCTV is a system that transmits images only to specific recipients using wired or special wireless transmission paths so that the general public cannot receive them arbitrarily depending on the purpose. CCTV is used for various purposes such as industrial, educational, medical, traffic control surveillance, disaster prevention, and image information transmission within the company.

CCTV is composed of a camera and a digital video recorder (DVR) that plays a role of recording the video captured by the camera. In recent years, CCTVs have been utilized by grafting technology to recognize and track objects through system linkage, beyond the role of simply capturing and transmitting images.

Artificial intelligence technology is a technology that realizes human learning ability, reasoning ability, perceptual ability, and understanding of natural language through computer programs, and it is a field of technology that enables computers to imitate human intelligent behavior. .

Machine learning, or machine learning, is a field of artificial intelligence that has evolved from the study of pattern recognition and computer learning theory. Machine learning is a technology that studies and builds a system that learns, performs prediction, and improves its own performance based on empirical data and algorithms for it. Meanwhile, deep learning is a field of machine learning. Deep learning differs from general machine learning in that it can learn by itself and predict future situations even if it omits the human teaching process.

Unexamined Patent Publication No. 10-2019-0051128, 2019.05.15

The problem to be solved by the present invention is to provide a method, a program, and a system for determining abnormal behavior based on an action sequence capable of recognizing an action while tracking each object when a plurality of objects exist.

In addition, a problem to be solved by the present invention is to provide a method, program, and system for determining abnormal behavior based on not only an image but also an action sequence in order to improve recognition rate and accuracy.

In addition, a problem to be solved by the present invention is to provide a method, program, and system for determining whether an abnormal behavior is based on an action sequence capable of understanding the intention of the object by analyzing the behavior of the recognized object.

In addition, the problem to be solved by the present invention is to provide a system with improved processing speed by using a storage space like a cache.

The problems to be solved by the present invention are not limited to the problems mentioned above, and other problems that are not mentioned will be clearly understood by those skilled in the art from the following description.

A method of determining whether an abnormal behavior based on an action sequence according to an aspect of the present invention for solving the above-described problems is performed by a computer, receiving image data, and recognizing one or more objects from the image data. , Recognizing a plurality of unit actions for each of the recognized objects, classifying the plurality of unit actions into normal or abnormal actions, and sequentially sequencing the plurality of unit actions to obtain sequence data. And determining whether or not an abnormal behavior is based on a ratio of the unit behavior classified as an abnormal behavior in the sequence data.

In addition, the method of determining whether or not abnormal behavior based on the behavior sequence includes determining whether or not the abnormal behavior includes determining as abnormal behavior if the value of the ratio occupied by one or more unit behaviors classified as abnormal behavior is greater than or equal to a preset value. It may include steps.

In addition, the method of determining whether abnormal behavior based on the behavior sequence further includes classifying an object category as a background or a person for each recognized object, and the step of recognizing the plurality of unit behaviors includes the object category May be recognizing a plurality of unit actions for an object that is a person.

In addition, the method of determining whether an abnormal behavior is based on the behavior sequence may include classifying the plurality of unit behaviors as normal behaviors or abnormal behaviors by comparing and matching the plurality of unit behaviors with a pre-stored database.

In addition, the method of determining whether or not abnormal behavior based on the behavior sequence includes the steps of recognizing the object, dividing an image frame into grids of the same size, and at least one bounding box including an object included in the image frame. And extracting a grid including a center point of the bounding box.

In addition, the method of determining whether an abnormal behavior is based on the behavior sequence may include recognizing the object, analyzing a similarity between the object recognized in the first image frame and the object recognized in the second image frame. have.

In addition, the method of determining whether an abnormal behavior based on the behavior sequence includes the step of recognizing the behavior, extracting an image included in an image frame, and extracting a behavior vector from the image frame and an image frame adjacent to the image frame. And recognizing a behavior based on the image and the behavior vector.

In addition, the method of determining whether or not an abnormal behavior based on the behavior sequence includes the step of extracting the behavior vector, applying an optical filter to an image included in an image frame, and vectorizing the behavior of the object from the image to which the optical filter is applied. It may include the step of.

A program for determining abnormal behavior based on an action sequence according to another aspect of the present invention for solving the above-described problem is combined with hardware to execute a method of determining abnormal behavior based on the above-mentioned behavior sequence. Is saved.

A system for determining abnormal behavior based on an action sequence according to another aspect of the present invention for solving the above-described problem includes an input unit receiving image data, an object recognition unit recognizing one or more objects from the image data, and the recognition. A behavior recognition unit for recognizing a plurality of unit actions for each of the objects, a classification unit for classifying the plurality of unit actions into normal or abnormal behaviors, and sequence data by sequentially sequencing the plurality of unit actions. It may include a generating unit to generate and a determination unit to determine whether or not the abnormal behavior based on a ratio of the unit behavior classified as an abnormal behavior in the sequence data.

Other specific details of the present invention are included in the detailed description and drawings.

According to the present invention, even when a plurality of objects appear in an image, it is possible to distinguish and recognize and track each object, and to recognize an action for each object to determine whether or not an abnormal behavior occurs.

In addition, according to the present invention, it is possible to maintain a high level of recognition rate and accuracy even when a plurality of objects appear in an image because it determines whether an abnormal behavior is based not only on the image but also on the basis of the behavior sequence data that sequenced the recognized behavior. .

In addition, according to the present invention, an abnormal symptom is not determined only by whether or not an object has performed a specific abnormal behavior, but by analyzing the behavior of the object in a time series to understand the intention of a specific unit behavior in a context, it is more precise. You can judge whether you are acting or not.

In addition, according to the present invention, by using the storage space as a cache, it is possible not to transmit/receive all necessary data such as image data through a network, thereby improving the processing speed.

The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the following description.

1 is a block diagram of a system for determining whether an abnormal behavior is based on an action sequence according to an embodiment of the present invention.
2 is a flowchart schematically illustrating a method of determining whether an abnormal behavior is based on an action sequence according to an embodiment of the present invention.
3 is a detailed flowchart illustrating a step of recognizing an object according to an embodiment of the present invention.
4 is an exemplary view showing a state of recognizing a plurality of objects according to an embodiment of the present invention.
5 is a flowchart further including analyzing the similarity of the object in the step of recognizing an object according to an embodiment of the present invention.
6 is a detailed flowchart illustrating a step of recognizing a unit action according to an embodiment of the present invention.
7 is an exemplary view showing an image and a motion vector extracted from an image frame according to an embodiment of the present invention.
8 is an exemplary diagram illustrating a pre-stored database in which behavior categories are matched for each unit behavior according to an embodiment of the present invention.
9 is an exemplary view showing a state in which sequence data is generated by sequentially sequencing a plurality of unit actions according to an embodiment of the present invention.
10 is a flowchart schematically illustrating a process of determining whether an abnormal behavior has occurred according to an embodiment of the present invention.
11 is an exemplary view showing a state of determining whether an abnormal behavior is based on a ratio occupied by a unit behavior classified as an abnormal behavior according to an embodiment of the present invention.
12 is a flowchart illustrating a process of determining whether or not an abnormal behavior is based on a ratio occupied by a unit behavior classified as an abnormal behavior according to an embodiment of the present invention.
13 is a flowchart further including the step of classifying an object category in a method of determining whether an abnormal behavior is based on an action sequence according to an embodiment of the present invention.

Advantages and features of the present invention, and a method of achieving them will become apparent with reference to the embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in a variety of different forms. It is provided to fully inform the skilled person of the scope of the present invention, and the present invention is only defined by the scope of the claims.

The terms used in the present specification are for describing exemplary embodiments and are not intended to limit the present invention. In this specification, the singular form also includes the plural form unless specifically stated in the phrase. As used herein, “comprises” and/or “comprising” do not exclude the presence or addition of one or more other elements other than the mentioned elements. Throughout the specification, the same reference numerals refer to the same elements, and "and/or" includes each and all combinations of one or more of the mentioned elements. Although "first", "second", and the like are used to describe various elements, it goes without saying that these elements are not limited by these terms. These terms are only used to distinguish one component from another component. Therefore, it goes without saying that the first component mentioned below may be the second component within the technical idea of the present invention.

In the present specification, "unit action" refers to an action that constitutes a movement of an object.

In the present specification, "normal behavior" refers to a unit behavior that is generally performed as a daily behavior that does not deviate from the standard defined by the group to which an individual belongs.

In the present specification, "abnormal behavior" refers to a behavior that is not a normal behavior and refers to a unit behavior that is not generally performed.

In the present specification, "abnormal behavior" refers to an action to be finally detected, such as a criminal activity.

Unless otherwise defined, all terms (including technical and scientific terms) used in the present specification may be used with meanings that can be commonly understood by those of ordinary skill in the art to which the present invention belongs. In addition, terms defined in a commonly used dictionary are not interpreted ideally or excessively unless explicitly defined specifically.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a block diagram of a system for determining whether an abnormal behavior is based on an action sequence according to an embodiment of the present invention.

Referring to FIG. 1, a system 1000 for determining abnormal behavior based on an action sequence according to an embodiment of the present invention includes an input unit 10, an object recognition unit 20, a behavior recognition unit 30, and a classification unit ( 40), a generation unit 50 and a determination unit 60 are included.

The input unit 10 serves to receive image data.

"Image data" is data about an image captured through a photographing device installed in the field. The image data includes video streaming data for reproducing the captured image in real time and data on the stored image. On the other hand, the photographing device installed in the field is an arbitrary device including a camera capable of taking a video of the field. As a specific example, the photographing device may be in the form of a CCTV, and if necessary, a configuration for performing a sensor or an artificial intelligence function may be additionally provided.

The input unit 10 may receive image data by being integrally configured with a photographing device or by receiving image data photographed by the photographing device through wired or wireless communication.

Meanwhile, in an embodiment, image data may be input and transmitted using a storage cash technology. "Storage cache technology" is a technology that can use storage space like a cache. That is, in the process of transmitting and receiving necessary data, the data stored in the storage space serves as a cache without the need to transmit and receive the entire data through the network every time, and some or all of the updated data is transmitted and received by matching the intermediate stored data. It is a technology that allows you to do it. By applying the storage cache technology, it is possible to process high-capacity data (for example, video data such as video streaming) faster. Through this, the task of recognizing the object and its behavior in real time and determining whether it corresponds to an abnormal behavior can be seamlessly processed.

The object recognition unit 20 recognizes and tracks (tracking) one or more objects from the received image data.

When a plurality of objects are included in the image, the object recognition unit 20 may distinguish and recognize each object. That is, even if the number of objects included in the image frame is different, the object recognition unit 20 may recognize each object of interest in the image frame by distinguishing it from the background.

The object recognition unit 20 includes an object recognition model. The "object-recognition model" is a model that recognizes an object by analyzing image data using a computer, and may include an algorithm or data for efficiently searching for an object or utilizing machine learning (or deep learning). have.

In an embodiment, the object recognition model of the object recognition unit 20 may include algorithms of two-stage methods or single-stage methods.

The two-stage method is a method of applying a region proposal netwrok (RPN) based on deep learning or computer vision technology that selectively searches for regions that are likely to contain objects. In other words, the object is detected by collecting the set of windows of the candidate group and formulating the number of regression models and classification models. The object recognition model is an example of a two-stage algorithm, and may include algorithms such as Region based CNN (R-CNN), Faster R-CNN, and Region-based Fully Convolutional Networks (R-FCN).

The single-step method is a method of searching for an object based on a predetermined position and size. The object recognition model is an example of a single-step algorithm, and may include an algorithm such as a You only look once (YOLO) algorithm, a Single Shot Mutibox Detector (SSD), and RetinaNet. A detailed description of a process in which the object recognition model recognizes an object by applying the YOLO algorithm will be described later.

The behavior recognition unit 30 plays a role of recognizing a plurality of unit actions for each object recognized by the object recognition unit 20.

The behavior recognition unit 30 includes a behavior recognition model. The "behavior recognition model" is a model that recognizes the behavior of an object by analyzing image data using a computer, and the behavior recognition model recognizes the behavior of an object of interest that has passed through the object recognition model.

In an embodiment, the behavior recognition model may include an algorithm for improving recognition rate and accuracy. As a specific example, the behavior recognition model includes an algorithm of the two-stream model method. The two-stream model is a model that distinguishes image data into spatial and temporal streams, and extracts and combines images (31) and motion vectors (32) from each of the spatial and temporal streams to recognize behavior. As another specific example, a 3D CNN method may be applied to improve the recognition rate of the two-stream model. 3D CNN is a method of inputting input values in 3D instead of 2D, and when 3D CNN is applied, a time axis can be applied, thereby improving the recognition rate. A detailed description of the process in which the behavior recognition model recognizes the behavior through the two-stream model will be described later.

The classification unit 40 serves to classify an object category or an action category.

"Object category" is a category for classifying the properties of an object. For example, the object category may include people, animals, backgrounds, and the like, but is not limited thereto, and is an arbitrary category capable of classifying the properties or types of objects.

When the object recognition unit 20 recognizes each object of interest in the image frame by distinguishing it from the background, the classification unit 40 determines and designates an object category for the recognized objects.

In one embodiment, the object category is used by the behavior recognition unit 30 to determine an object to recognize the behavior. As a specific example, the behavior recognition unit 30 may perform a behavior recognition task only on an object whose object category is classified as a person according to its purpose. Through this, it is possible to improve the processing speed and performance of the task by excluding the action recognition task for unnecessary objects.

The "behavior category" is a category for classifying the types of behavior of an object. For example, the behavioral category may include normal behavior and abnormal behavior.

The classification unit 40 compares a plurality of unit actions recognized by the behavior recognition unit 30 with a pre-stored database (refer to FIG. 8) to match the behavior category.

The generation unit 50 plays a role of generating sequence data by sequentially sequencing a plurality of unit actions.

"Sequence data" is data obtained by sequentially sequencing recognized unit actions. That is, sequence data is data in which unit actions recognized from an object are sequentially arranged in order. The sequence data includes data on the number of times each unit action is detected by dividing the unit actions recognized from the object for each predetermined unit action (see FIG. 11). The sequence data is used to determine whether an abnormal behavior has occurred, and a specific method will be described later.

The determination unit 60 serves to determine whether an action (or behavior flow) of an object recognized based on the sequence data generated by the generation unit 50 corresponds to an abnormal behavior.

The determination unit 60 may include a sequence classification model. The "sequence classification model" is a model that analyzes sequence data generated by the generation unit 50 to determine whether the motion of an object corresponds to an abnormal behavior.

In one embodiment, the sequence classification model determines whether an abnormal behavior is based on a ratio of the number of unit behaviors in which the behavior category is classified as abnormal behavior in sequence data. A detailed description of this will be described later with reference to FIG. 11.

In another embodiment, the sequence classification model compares the sequence data generated by the generation unit 50 with sequence data that is pre-stored or learned through machine learning to determine whether there is an abnormal behavior. That is, the sequence data stored in advance or learned through machine learning is classified as abnormal behavior or normal behavior for the behavior sequence according to the number of cases, and the sequence data generated by the generation unit 50 is classified as abnormal behavior. If it matches the set action sequence, it is determined as an abnormal action, and when the sequence data generated by the generation unit 50 matches the action sequence classified as a normal action, it is judged as a normal action.

In another embodiment, the sequence classification model calculates a first score as a result of comparing the sequence data generated by the generation unit 50 with the sequence data that is pre-stored or learned through machine learning, and calculates the behavior category from the sequence data. A secondary score may be calculated based on the ratio of the number of unit actions classified as abnormal behaviors to the total number of unit actions, and the abnormal behavior can be determined by combining the first score and the second score.

2 is a flowchart schematically illustrating a method of determining whether an abnormal behavior is based on an action sequence according to an embodiment of the present invention.

Referring to FIG. 2, a method of determining whether an abnormal behavior is based on an action sequence according to an embodiment of the present invention includes receiving image data (S100), recognizing one or more objects from image data (S200), Recognizing a plurality of unit actions for each recognized object (S300), classifying an action category for a plurality of unit actions (S400), generating sequence data based on the plurality of unit actions (S500) And determining whether the abnormal behavior is based on the sequence data (S600).

Step S100 is a step in which the input unit 10 receives image data. Image data may be streamed in real time or received in a stored form.

Step S200 is a step in which the object recognition unit 20 recognizes one or more objects from image data input through the input unit 10.

Step S300 is a step in which the behavior recognition unit 30 recognizes a plurality of unit actions for each object recognized by the object recognition unit 20.

In step S400, the classification unit 40 classifies a behavior category as normal behavior or abnormal behavior with respect to a plurality of unit behaviors recognized by the behavior recognition unit 30.

Step S500 is a step in which the generation unit 50 sequentially sequence a plurality of unit actions to generate sequence data.

In step S600, the determination unit 60 determines whether an abnormal behavior occurs based on the sequence data generated by the generation unit 50.

3 is a detailed flowchart illustrating a step of recognizing an object according to an embodiment of the present invention.

Referring to FIG. 3, in the step of recognizing an object according to an embodiment of the present invention (S200), a YOLO algorithm may be applied, and the step of forming one or more bounding boxes including objects included in an image frame (S210), It may include dividing the image frame into grids having the same size (S220) and extracting a grid including a center point of the bounding box (S230).

Step S210 is a step of dividing the image frame into a plurality of grids having the same size.

Step S220 is a step of forming a bounding box having a size surrounding the object image included in the image frame. When a plurality of objects exist in an image frame, a bounding box is formed for each object.

In an embodiment, the bounding box may be formed by predicting the number of bounding boxes (anchor boxes) required for each object based on a preset shape centered on the center of the grid for each grid. The number of bounding boxes can be determined from data by the K-means algorithm.

Step S230 is a step of extracting a grid including a center point of the bounding box, and determining a grid for identifying the recognized object.

Since each object recognized through the step S230 is matched to one grid smaller in size than the bounding box and can be identified, each of the plurality of objects can be more accurately distinguished and recognized. That is, when there are a plurality of objects, even if a boundary box surrounding each object overlaps, the center grid matched to each object does not overlap, so the recognition rate is improved.

4 is an exemplary view showing a state of recognizing a plurality of objects according to an embodiment of the present invention.

Fig. 4(a) is an exemplary diagram showing the appearance of a plurality of objects (people) entering a store, and Fig. 4(b) is a picture frame divided into a plurality of grids of the same size, and a bounding box for each object It is an exemplary diagram showing a state in which is formed, and FIG. 4(c) is an exemplary diagram illustrating a state in which a grid corresponding to the center point of each bounding box is matched.

Referring to FIG. 4, it can be seen that bounding boxes formed for a plurality of objects overlap. In order to distinguish and recognize objects overlapping the bounding box as described above, the position of the object is recognized based on a coordinate value corresponding to the center point of each bounding box. The coordinate values corresponding to the center point of the bounding box formed for each object are (x1, y1), (x2, y2), and (x3, y3), respectively. Objects are recognized based on a grid (refer to FIG. 4(c)) corresponding to each coordinate value.

5 is a flowchart further including analyzing the similarity of the object in the step of recognizing an object according to an embodiment of the present invention.

Referring to FIG. 5, the step S200 of recognizing an object according to an embodiment of the present invention may apply a Siamese algorithm, and further includes a step S240 of analyzing the similarity of the object.

Step S240 is a step of analyzing the similarity between the object recognized in the first image frame and the object recognized in the second image frame adjacent to the first image frame.

The Siamese algorithm is an algorithm that recognizes and classifies objects and analyzes the similarity of the recognized objects for each frame. That is, by calculating the vector value of the recognized object for each frame and analyzing the similarity of the vector value, the recognized object for each frame is matched to have a clustering effect for the same object. This enables more accurate and effective object tracking.

6 is a detailed flowchart illustrating a step of recognizing a unit action according to an embodiment of the present invention.

Referring to FIG. 6, in the step of recognizing a unit action according to an embodiment of the present invention (S300), a two-stream model may be applied, and the step of extracting an image included in an image frame (S310), an image frame, and And extracting a motion vector from an image frame adjacent to the image frame (S320), and recognizing a behavior based on the extracted image and motion vector (S330).

Step S310 is a step of extracting the image 31 included in the image frame of the spatial stream.

Step S320 is a step of extracting a specific image frame (image frame corresponding to the image frame from which the image 31 is extracted) from the temporal stream and the motion vector 32 from the image frame adjacent to the image frame before and after the corresponding image frame.

Step S320 includes applying an optical filter to the image included in the image frame and vectorizing the behavior of the object from the image to which the optical filter has been applied. That is, an optical filter is applied to an image frame, and a vector value, which is a characteristic capable of identifying the behavior of an object, is calculated from the image frame to which the optical filter is applied.

Step S330 is a step of recognizing the behavior of the object based on the image 31 and the motion vector 32 extracted from the image frame. That is, the action is recognized based on the score obtained by combining the extracted image 31 and the action vector 32.

7 is an exemplary view showing an image and a motion vector extracted from an image frame according to an embodiment of the present invention.

Referring to FIG. 7, image data is classified into a spatial stream and a temporal stream, an image 31 is extracted from the spatial stream, and a motion vector 32 is extracted from the temporal stream.

8 is an exemplary diagram illustrating a pre-stored database in which behavior categories are matched for each unit behavior according to an embodiment of the present invention.

Referring to FIG. 8, identification numbers may be assigned to a plurality of preset unit actions, and action categories matching each may be matched to be stored and managed.

Multiple unit actions include Entering the store, Walking, Scanning the store, Watching CCTVs, Picking up things, and Putting things. in a pocket), Putting things in a bag, Putting things in a shopping basket, Putting down things, and Standing. In addition, identification numbers 1 to 10 may be assigned to each unit action. In addition, each of the unit behaviors is matched with the behavior category determined in advance as normal behavior or abnormal behavior.

9 is an exemplary view showing a state in which sequence data is generated by sequentially sequencing a plurality of unit actions according to an embodiment of the present invention.

The generation unit 50 of the system 1000 that determines whether an abnormal behavior is based on the behavior sequence is arranged in order to generate behavior sequence data for each object by placing a plurality of unit behaviors recognized for each object.

Referring to FIG. 9, sequence data for each object is generated for each row. In an embodiment, the sequence data may include information on an action category of the arranged unit actions. That is, sequence data may be generated by discriminating whether the behavior category of the arranged unit blocks is a normal behavior or an abnormal behavior. Referring to the example shown in FIG. 8, the unit behaviors with identification numbers 3, 4, 6, and 7 are classified as behavioral categories abnormal behaviors. Accordingly, the sequence data includes information on the probability of an abnormal behavior according to the number or ratio of the unit behaviors (unit behaviors shown in shading in Fig. 9) of 3, 4, 6, and 7 are included. I can.

10 is a flowchart schematically illustrating a process of determining whether an abnormal behavior has occurred according to an embodiment of the present invention.

Referring to FIG. 10, in the process of determining whether or not abnormal behavior according to an embodiment of the present invention, input image data is an object recognition model (shown as model 1), a behavior recognition model (shown as model 2), and a sequence classification model. It proceeds through (shown as model 3) in order.

The first input data is image data input through the input unit 10. The image data includes an image of an object's movement over time. The object recognition model recognizes one or more objects from the input image data. When object recognition is completed, the behavior recognition model recognizes the behavior of the recognized object. When the behavior recognition is completed, the generation unit 50 generates sequence data based on the recognized unit behaviors. The sequence classification model finally determines whether the motion of the recognized object corresponds to an abnormal behavior based on the generated sequence data. The above-described process continuously tracks the recognized object and is repeated in real time.

11 is an exemplary view showing a state of determining whether an abnormal behavior is based on a ratio occupied by a unit behavior classified as an abnormal behavior according to an embodiment of the present invention.

11 shows sequence data for each object for each row. On the left, the total number of recognized unit actions and the number of each unit action (unit actions 1 to 10 are shown as examples in FIG. 11) are shown. On the right, the ratio of the number of each unit action to the number of recognized unit actions and the result of determining whether it corresponds to an abnormal action based on the ratio is shown.

In an embodiment, the determination unit 60 may determine whether the behavior category is abnormal based on a ratio of the unit behavior classified as the abnormal behavior in the sequence data. As a specific example, if the value of the ratio occupied by one or more unit actions classified as abnormal behaviors is equal to or greater than a preset value, it may be determined as an abnormal behavior. As another specific example, it is determined based on the value of the ratio occupied by one or more unit actions classified as abnormal behaviors, but weights are assigned for each unit behavior, and the result calculated by reflecting the weights is compared with a preset value to determine whether or not abnormal behaviors. I can judge.

12 is a flowchart illustrating a process of determining whether or not an abnormal behavior is based on a ratio occupied by a unit behavior classified as an abnormal behavior according to an embodiment of the present invention.

Referring to FIG. 12, comparing with FIG. 2, determining whether the ratio of the unit behavior classified as abnormal behavior is greater than or equal to a preset value (S610), determining as abnormal behavior (S620), and determining as normal behavior (S630). ).

Step S610 is a step of determining whether a ratio of one or more unit actions classified as abnormal actions relative to the total unit actions in the sequence data is equal to or greater than a preset value.

In step S620, if the ratio of one or more unit actions classified as abnormal actions relative to the total unit actions is greater than or equal to a preset value, it is determined as an abnormal action.

In step S630, if the ratio of one or more unit actions classified as abnormal actions relative to the total unit actions is less than a preset value, it is determined as a normal action.

13 is a flowchart further including the step of classifying an object category in a method for determining whether an abnormal behavior is based on an action sequence according to an embodiment of the present invention.

Referring to FIG. 13, the step of classifying an object category for each recognized object compared with FIG. 2 (S250) is further included.

In step S250, the classification unit 40 designates an object category to which the recognized object belongs. As a specific example, the classification unit 40 may classify an object category as a background, a person, or an animal with respect to the recognized object. The classified object category can be used to proceed with the object tracking and behavior recognition process only for objects of interest that require behavior recognition according to the purpose.

The steps of a method or algorithm described in connection with an embodiment of the present invention may be implemented directly in hardware, implemented as a software module executed by hardware, or a combination thereof. Software modules include Random Access Memory (RAM), Read Only Memory (ROM), Erasable Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), Flash Memory, hard disk, removable disk, CD-ROM, or It may reside on any type of computer-readable recording medium well known in the art to which the present invention pertains.

In the above, embodiments of the present invention have been described with reference to the accompanying drawings, but those skilled in the art to which the present invention pertains can be implemented in other specific forms without changing the technical spirit or essential features. You will be able to understand. Therefore, the embodiments described above are illustrative in all respects, and should be understood as non-limiting.

1000: A system that determines whether an abnormal behavior is based on an action sequence
10: input
20: object recognition unit
30: Behavior Recognition Department
31: image
32: action vector
40: classification unit
50: generation unit
60: judgment unit

Claims (10)

Performed by the computer,
Receiving image data;
Recognizing one or more objects from the image data;
Recognizing a plurality of unit actions for each of the recognized objects;
Classifying the plurality of unit behaviors into behavioral categories as normal behaviors or abnormal behaviors;
Generating sequence data by sequentially sequencing the plurality of unit actions; And
Including, determining whether or not an abnormal behavior is based on a ratio of the unit behavior classified as an abnormal behavior in the sequence data.
A method of determining abnormal behavior based on the sequence of actions. The method of claim 1,
The step of determining whether the abnormal behavior or not,
Including the step of determining the abnormal behavior if the value of the ratio occupied by one or more unit behaviors classified as abnormal behaviors is equal to or greater than a preset value,
A method of determining abnormal behavior based on the sequence of actions. The method of claim 2,
For each recognized object, further comprising the step of classifying the object category as a background or a person,
Recognizing the plurality of unit actions,
Recognizing a plurality of unit actions for an object whose object category is a person,
A method of determining abnormal behavior based on the sequence of actions. The method of claim 3,
The step of classifying the plurality of unit actions into normal actions or abnormal actions,
Matching the plurality of unit actions by comparing them with a pre-stored database,
A method of determining abnormal behavior based on the sequence of actions. The method of claim 1,
Recognizing the object,
Dividing the image frame into grids of the same size;
Forming one or more bounding boxes including objects included in the image frame; And
Including; extracting a grid including the center point of the bounding box;
A method of determining abnormal behavior based on the sequence of actions. The method of claim 5,
Recognizing the object,
Including the step of analyzing the similarity between the object recognized in the first image frame and the object recognized in the second image frame,
A method of determining abnormal behavior based on the sequence of actions. The method of claim 1,
Recognizing the behavior,
Extracting an image included in the image frame;
Extracting a motion vector from the image frame and an image frame adjacent to the image frame; And
Recognizing an action based on the image and the action vector; Including,
A method of determining abnormal behavior based on the sequence of actions. The method of claim 7,
The step of extracting the behavior vector,
Applying an optical filter to an image included in the image frame; And
Including; vectorizing the behavior of the object from the image to which the optical filter has been applied,
A method of determining abnormal behavior based on the sequence of actions. A program that is combined with a computer that is hardware and stored in a medium to execute the method of any one of claims 1 to 8, and determines whether an abnormal behavior is based on an action sequence. An input unit for receiving image data;
An object recognition unit for recognizing one or more objects from the image data;
A behavior recognition unit for recognizing a plurality of unit actions for each of the recognized objects;
A classification unit for classifying a behavior category from the plurality of unit behaviors into normal behaviors or abnormal behaviors;
A generator configured to sequentially sequence the plurality of unit actions to generate sequence data; And
Including; a determination unit that determines whether or not an abnormal behavior based on a ratio of the unit behavior classified as an abnormal behavior in the sequence data,
A system that judges abnormal behavior based on the sequence of actions.

Images