KR20210027778A - Apparatus and method for analyzing abnormal behavior through object detection and tracking - Google Patents

Abstract

The present invention discloses a device and a method for analyzing abnormal behavior through object detection and tracking. According to the present invention, the device for analyzing abnormal behavior through object detection and tracking includes a processor and a memory connected to the processor. The memory detects an object existing in the input image in a preset frame unit based on a neural network, tracks the detected object using a kernerlize correlation filter (KCF) using weighted gray transform in units of each frame and Hue-Histogram comparison of objects existing in different frames, determines whether the tracked object has abnormal behavior based on whether it enters a preset region of interest or a pre-trained neural network, and stores program instructions executable by the processor, in the device for analyzing abnormal behavior.

Description

Apparatus and method for analyzing abnormal behavior through object detection and tracking}

The present invention relates to an apparatus and method for analyzing abnormal behavior through object detection and tracking.

Recently, in order to prevent crime or secure post-evidence, CCTVs are installed everywhere, and a system for analyzing images has been generally provided.

In particular, for the prevention and monitoring of crimes, it is important to detect objects in the video and maintain tracking accuracy.In addition, through this, it is important to immediately analyze whether the object invades or roams the prohibited area, assault or fall situation. It is important to do.

Conventionally, a number of techniques for analyzing anomalous behavior through object detection and tracking have been proposed, but when a general object tracker is used, there are many cases of missing objects due to surrounding occlusion.

In addition, there is a problem in that an error of determining an abnormal behavior occurs frequently even though it is not an abnormal behavior according to the movement of an object.

Korean Patent Publication 10-2019-0074898

In order to solve the problems of the prior art, the present invention proposes an apparatus and method for analyzing abnormal behavior through object detection and tracking, in which object tracking can be accurately performed.

In order to achieve the above object, according to an embodiment of the present invention, an apparatus for analyzing abnormal behavior through object detection and tracking, comprising: a processor; And a memory connected to the processor, wherein the memory detects an object present in the input image based on a neural network in units of a preset frame, and uses a weighted gray transformation in each frame unit, and the memory includes a Kernerlize Correlation Filter (KCF) and each other. The detected object is tracked using color histogram comparison of objects existing in different frames, and whether the tracked object has abnormal behavior based on a pre-set region of interest or a pre-learned neural network In order to determine, an abnormal behavior analysis apparatus for storing program instructions executable by the processor is provided.

The program instructions may give different weights to each of the RGB channels in consideration of distribution of each of the RGB channels of the input image for tracking the object.

The program instructions manage the x and y coordinates of the detected object, the initial detection time, and the last detection time through tracking as an object of interest list, and the object detected in the current frame is the same as the object managed in the object of interest list. The object of interest list may be updated by determining whether it is an object.

The program instructions are the object corresponding to the cropped image when the image of the tracked object is cropped every frame, the bundle of the cropped images is continuous, and the cropped image reaches a preset quantity. It is possible to determine whether or not to behave abnormally.

The program instructions may determine whether the object is intrusive or roaming through geometric analysis of the preset region of interest and a movement trajectory of the tracked object.

The program commands may determine whether the object is intrusive or roaming by using a reference line for the preset region of interest and whether the movement direction of the object is a positive “‡ term or a reverse direction with respect to the reference line. .

The program instructions may determine whether the object falls or is violated based on the pre-learned neural network.

According to another aspect of the present invention, there is provided a method for analyzing anomalous behavior through object detection and tracking in a device including a processor and a memory, the method comprising: detecting an object existing in an input image based on a neural network in units of a preset frame; Tracking the detected object using a Kernerlize Correlation Filter (KCF) using a weighted gray transformation for each frame and a color histogram of objects existing in different frames; And determining whether the tracked object has an abnormal behavior based on whether to enter a predetermined region of interest or a previously learned neural network.

According to another aspect of the present invention, a program stored on a recording medium for performing a method is provided.

According to the present invention, object detection and tracking can be accurately performed, and through this, it is possible to accurately determine whether or not the tracked object has abnormal behavior.

1 is a diagram showing the configuration of an apparatus for analyzing abnormal behavior through object detection and tracking according to an embodiment of the present invention.
2 is a diagram showing the configuration of a program module for analyzing abnormal behavior according to an embodiment of the present invention.
3 is a diagram illustrating a process of tracking movement of an object.
4 shows a situation in which object tracking fails due to occlusion between objects.
5 is a diagram for describing a KCF-based object tracking process.
6 shows a cyclic shifts method for analyzing a degree of agreement with an object of interest.
7 shows a response map obtained by obtaining a correlation value for all cyclic shifts.
8 is a diagram for describing an object tracking process according to an embodiment of the present invention.
9 is a diagram for explaining a process of removing unnecessary background information according to an existing method and the present embodiment.
10 is a diagram for describing a process of managing object information according to the present embodiment.
11 shows an image generated as a result of an operation of an object crop module.
12 is a diagram for explaining an operation process of the object crop module according to the present embodiment.
13 is a diagram for explaining a process of determining the intersection of vectors according to the present embodiment.
14 shows an object intrusion determination process based on geometric analysis.
15 is a diagram illustrating a residual learning block of ResNet.
16 is a diagram showing a residual block according to the present embodiment.
17 shows training sample images generated to learn an abnormal behavior analysis network.

In the present invention, various modifications may be made and various embodiments may be provided, and specific embodiments will be illustrated in the drawings and described in detail.

However, this is not intended to limit the present invention to a specific embodiment, it should be understood to include all changes, equivalents, and substitutes included in the spirit and scope of the present invention.

1 is a diagram showing the configuration of an apparatus for analyzing abnormal behavior through object detection and tracking according to an embodiment of the present invention.

As shown in FIG. 1, the abnormal behavior analysis apparatus according to the present embodiment may include a processor 100 and a memory 102.

The processor 100 may include a central processing unit (CPU) capable of executing a computer program or a virtual machine.

The memory 102 may include a nonvolatile storage device such as a fixed hard drive or a removable storage device. The removable storage device may include a compact flash unit, a USB memory stick, or the like. Memory 102 may also include volatile memories such as various random access memories.

According to an embodiment of the present invention, the processor 100 detects an object for each preset frame unit by using program instructions stored in the memory 102, tracks the detected object, and determines whether the object has abnormal behavior. Judge

2 is a diagram showing the configuration of a program module for analyzing abnormal behavior according to an embodiment of the present invention.

As shown in FIG. 2, the program module according to the present embodiment includes an object detection module 200, an object tracking module 202, an object management module 204, an object crop module 206, and an abnormal behavior analysis module ( 208).

In this embodiment, an object detection method based on a Convolution Neural Network (CNN) is used for object detection in an outdoor environment and various lighting conditions. With the existing background difference method, it is difficult to detect an object in an actual outdoor image such as background change, camera shake, and lighting change. Therefore, the accuracy of object detection is improved by using a CNN-based object detection algorithm.

The object detection network used in the object detection module 200 of this embodiment is based on the YOLOv3 network. Because the object is detected through Unified Detection, it shows faster processing speed compared to two-stage methods such as R-CNN. The input image is divided into each grid cell and the score for each object is calculated. The score is calculated by the equation below, and if there is no object in the image, the score is measured as 0.

Each grid cell has a probability corresponding to each of the C trained classes. The probability corresponding to each class is expressed as the following equation.

Finally, five pieces of information including the center coordinates (x, y) of each object, information about the horizontal and vertical lengths (width, height), and the probability of matching the learned class are displayed. Object tracking is performed by setting the coordinates of the object detected later as an input of the object tracking module 202 (Object Tracker).

According to the present embodiment, object detection and tracking may be performed every n frame units (eg, 6 frames).

The object tracking module 202 performs object tracking based on the coordinates input up to the next point in time, and if object tracking fails, the object detection module 200 performs tracking again on the object detected at the next detection point. .

It is necessary to update whether an object is newly detected or disappeared at a specific point in the input image.

3A is a situation in which an object goes out of the image, and FIG. 3B is a situation in which an object enters into a window. If tracking is performed with only the existing tracker, since the tracker continues to track the most similar patch, in the case of FIG. 3A, the outer portion of the image and in the case of FIG. 3B, the window area is continuously tracked.

In the present embodiment, for example, since the object detection module 200 detects an object every 6 frames, the object tracking module 202 is updated according to whether an object is detected in the corresponding area to determine whether or not the object is tracked. Decide.

As shown in FIG. 4, a situation in which object tracking fails due to occlusion between objects may occur. In the present embodiment, for example, since the object detection module 200 detects an object every 6 frames, the object coordinates are newly updated to enable correct tracking.

4A is an image in which tracking fails because the object is covered by the barbed wire when an object intrudes into the barbed wire. Thereafter, when the object crosses the barbed wire and reappears, the object detection module 200 updates the object information so that the tracking can be performed correctly. 4B and C show a situation in which tracking fails due to overlapping between objects. Thereafter, when the object is out of the overlapping situation, the object detection module 200 may newly update the object information so that each object can be correctly tracked.

A plurality of objects may exist in an image, and the present invention proposes a technique capable of maintaining the accuracy of object detection and tracking even in such a case.

Object tracking refers to acquiring an area of an object of interest in an image as initial information, and then tracking the position of the object in a series of subsequent frames.

According to this embodiment, an object detection network and an object tracking algorithm are fused. The object detection network detects an object of interest that can be observed in an image to obtain the position and size of the object, and the object tracking algorithm receives the obtained object information and continuously tracks the object in successive frames.

Object tracking is performed within a range of a certain range centering on the position of the object in the previous frame. This is performed under the assumption that the change in the position of the object between adjacent frames will be less than a certain amount. Therefore, in a real-time streaming environment, if the object tracking operation speed is less than an appropriate level, the position change of the object between adjacent frames increases (frame drop phenomenon), and tracking performance may be significantly degraded.

According to the present embodiment, multiple objects are tracked using an object tracking technology based on a Kernelized Correlation Filter (KCF).

5 is a diagram for describing a KCF-based object tracking process.

Referring to FIG. 5, in this embodiment, an input image is converted into a frequency domain and processed. Compared to other object tracking algorithms these days, it requires only a much smaller amount of operations, so smooth tracking is possible even in environments with limited hardware performance.

With the position of the object of interest in the previous frame as the center, a certain area for searching the position of the object is defined as a search window.

The KCF-based tracking algorithm uses the cyclic shifts method as shown in FIG. 6 in order to analyze the degree of agreement with the object of interest for all coordinates in the search window.

This is a method of obtaining all possible shifts by calculating with the matrix when there is data x obtained in the form of a matrix. That is, by using the image patch of the search window as a source, images for all possible shifts in the horizontal and vertical directions are generated, and the degree of agreement with the object of interest is calculated.

The degree of correspondence with the object of interest is defined as a correlation value, and a response map is obtained by obtaining a correlation value for all cyclic shifts.

The response map is expressed as a 3D graph as shown in FIG. 7, and the height at an arbitrary point of the graph means the degree of correspondence with the object of interest. It has the highest value (peak) at a specific point, and horizontal or vertical at the peak point. It can be seen that the degree of correspondence decreases as it moves in the direction. The KCF-based tracking algorithm operates on the principle of tracking the coordinates of the peak value, and calculates the expected tracking coordinates every frame.

In the present invention, in order to increase the accuracy of object tracking, an object of interest is identified and tracked by comparing the coordinates of a previous frame with a hue histogram.

8 is a diagram for describing an object tracking process according to an embodiment of the present invention.

Referring to FIG. 8, the apparatus for analyzing abnormal behavior detects an object from an input image based on deep learning. Here, deep learning may be YOLO v3.

Next, the detected object information is tracked.

Object tracking may be performed through a Kernerlize Correlation Filter (KCF) tracker using weighted gray transformation.

Here, the KCF tracker may be defined as the object tracking module 202 of FIG. 2.

The KCF tracker searches for the RGB-based index value with the largest distribution in the image. A channel space suitable for an object is obtained by using the RGB ratio of the corresponding index value.

Based on the acquired channel space, a grayscale image is generated as follows.

는 RGB 색상 값의 비율을 나타내고,

는 출력 그레이스케일 이미지,

는 타겟 객체 패치이다. here,

Represents the ratio of RGB color values,

Is the output grayscale image,

Is the target object patch.

As a result, a weight is applied to each element in consideration of the distribution of each of the RGB channels to improve tracking performance of multiple objects having different colors.

According to the present embodiment, for continuous management of an object, a comparison of ROI color histograms is performed.

To obtain the ROI, we compare the similarity between the color histograms of the objects.

According to the present embodiment, first, a histogram of color channels is compared using a correlation coefficient operator. The similarity is calculated as follows.

Here, H ₁ and H ₂ represent histogram values between objects in different frames, respectively. In order to remove unnecessary background information, d(H _1, H ₂ ) is maximized as shown in FIG. 9.

9A shows the background removal result using DeepLab V3 semantic segmentation. Despite its accurate segmentation performance, DeepLab V3 is not suitable for real-time tracking due to its high computational cost and occasional malfunction.

9B shows the result of simply removing the background by designating an ellipse ROI in the bounding box according to the present embodiment.

9B shows that the background is not completely removed compared to FIG. 9A.

In the method according to the present embodiment, the average histogram correlation is slightly lower in performance, but the processing speed is much higher than that of DeepLab V3 as follows.

는 히스토그램 상관 관계의 가중치를 결정하는 매개변수이며 실험적으로 선택된 0.8이 선택될 수 있다.

은 히스토그램 상관값이고

는 거리 정보이다. 비교 점수 S_c는 관심 객체를 식별하기 위한 척도로 사용된다.here

Is a parameter that determines the weight of the histogram correlation, and 0.8, which is experimentally selected, can be selected.

Is the histogram correlation value

Is the distance information. The comparison score S _c is used as a measure to identify the object of interest.

More specifically, if the maximum S _c is higher than a preset threshold, it is determined to be the same object. On the other hand, if all S _c is lower than the threshold value, the object is identified as a new object and stored in a new list. Through the above process, the device according to the present embodiment can identify and track the same object in various difficult situations.

As described above, the object of interest is initially detected by the object detection module 200 and its location is tracked by the object tracking module 202.

In order to analyze the abnormal behavior of the object of interest, it is necessary to systematically manage the information obtained from the object detection module 200 and the object tracking module 202.

To this end, the object management module 204 according to the present embodiment structure and manages each object information as shown in FIG. 10.

Here, x,y mean the latest coordinates of the object of interest.

The first-seen time refers to the time (initial time) when the object of interest is first detected by the object detection module 200, and the last-seen time is detected by the object detection module 200 or the object tracking module 202 or It means the last time tracked (last time).

The information on one object is composed of four items of x and y coordinates, the first time and the last time, and the object management module 204 manages a plurality of objects of interest by listing them.

Referring to FIG. 10, when the object management module 204 receives coordinate information of an object of interest from the object detection module 200 and the object tracking module 202, the input object is an object currently managed in the object of interest list. Compared with, it is determined whether or not the object is the same. Whether the object is the same object may be determined based on the distance of the coordinates.

If the input object information is determined to be the same object as the object included in the object of interest list, the object object information is updated with the input object information, and if it is determined that it is not the same object, it is added as new object information.

If the last detected or tracked time of an object is not updated for a certain period of time, the object is considered to be no longer observed in the input image and is removed from the list of objects of interest to manage only valid objects.

Abnormal behavior analysis acquires several close-up images of a specific object of interest on a continuous frame and uses them as input data. To this end, the object crop module 206 crops the image of the object of interest being tracked every frame, manages the cropped images for the same object as a bundle, and provides it to the abnormal behavior analysis module 208.

11 shows an image generated as a result of an operation of an object crop module.

Since multiple objects of interest may exist in an image at a specific point in time, the object crop module 206 must be able to manage cropped images corresponding to the same object in a bundle.

In FIG. 11, (a) to (h) are the first objects, and (i) to (p) are the second objects, and the object crop module 206 according to the present embodiment classifies and manages the objects of interest.

12 is a diagram for explaining an operation process of the object crop module according to the present embodiment.

Referring to FIG. 12, the object crop module 206 receives coordinates and size information (x, y, width, height) of an object of interest.

The object of interest is cropped by calculating an appropriate crop area based on the input information.

The image of the object of interest, which is cropped every frame, is divided and managed for each object.

When the cropped image reaches a preset quantity, that is, when the cropped image continuously exists for a preset number of frames, a sub-thread for image dump corresponds to a preset folder. Save the images.

The stored image is input to the abnormal behavior analysis module 208.

Since recording the cropped image in the auxiliary memory device may cause a serious bottleneck and deteriorate the overall performance, a sub-thread for storing the cropped image is operated, so that the above process is performed with only a few operations in the main-thread. Make it possible to complete it.

In order to collect and manage cropped images, continuity must be met.

In the process of collecting the cropped image of the object of interest, if the bundles of the cropped images are not continuous, the abnormal behavior analysis module 208 receives non-contiguous frame information, which may cause performance degradation of the abnormal behavior analysis.

Accordingly, the object cropping module 208 periodically discards the cropped image bundle whose continuity is lost without passing it to the abnormal behavior analysis module 208.

The abnormal behavior analysis module 208 determines whether an object has abnormal behavior through geometric analysis of a predefined area in the image and a tracked object trajectory.

The abnormal behavior may include intrusion, roaming, and assault. Hereinafter, a process of determining the intrusion situation of an object will be first described.

In order to determine the intrusion situation, a region of interest (ROI) is defined in advance, and the movement path of the object is analyzed and then determined.

In the ROI area setting, a reference line is set in an image, and forward and reverse directions are set for the set reference line. Whether the object is intruded is determined whether the reference line and the trajectory of the object are vectorized to determine whether the reference line has crossed the reference line, that is, has entered the ROI.

13 is a diagram for explaining a process of determining the intersection of vectors according to the present embodiment.

In FIG. 13, line 1 represents a predefined ROI (reference line), and line 2 represents a movement trajectory of an object. It is expressed as Equation 6 below as an equation of a straight line using the parameters of two lines.

Line 1 is _{composed of K 1} and K ₂ , line 2 is composed of K ₃ and K ₄ , the range of t and s is [0,1]. If the intersection point where two lines meet is summarized, Equation 6 can be expressed as follows.

Equation 7 can be separated by x and y and expressed as the following equation.

This can be summarized as t and s as follows.

That is, t and s above represent values when an intersection occurs in two lines, and the intersection can be expressed as follows.

If the values of s and t are in the range of [0,1], it is determined that the two lines intersect.

When it is determined that the two lines intersect, it is determined whether the direction of the moving object is the forward direction or the reverse direction using the _{direction of each line vector and the angle θ at the intersection point K i.}

_{The angle with the vector of line 2 is measured based on K 2} of line 1 at 0°, and the direction of the moving object according to the angle is determined by the following equation.

14 shows an object intrusion determination process based on geometric analysis.

Referring to FIG. 14B, the abnormal behavior analysis module 208 determines that the object has invaded through whether the trajectory of the object and the line of the intrusion area intersect as the tracking object enters the predefined intrusion area.

Since the tracked object enters the building and object trajectory information disappears, it is determined that the object intrusion has been completed in FIG. 14C.

According to this embodiment, it is determined that it is an intrusion situation that one or more people (objects) enter the area of interest, and when one or more people stay near the area of interest for longer than a preset time, the roaming situation It is judged to be.

In addition, according to the present embodiment, it is also possible to determine an assault and fall situation based on a neural network.

The abnormal behavior analysis module 208 according to the present embodiment analyzes an assault and fall situation of an object using a 3D Convolutional Neural Network (CNN) based on a ResNet (Residual Network).

15 is a diagram illustrating a residual learning block of ResNet.

는

이 된다. ResNet은 loss function

가 0이 되는 방향으로 학습이 진행되고,

이기 때문에

를 학습한다는 것은 나머지(residual)를 학습한다고 볼 수 있다. 추가적인 연산 없이 입력 x를 skip connection하기 때문에 연산량은 기존과 동일하고, 네트워크의 깊이가 깊어져도 gradient vanishing 문제를 줄일 수 있기 때문에, 학습 성능이 높아진다. Referring to FIG. 15, a skip connection that directly connects the input x to the output of the layer is used, and the output of the network

Is

Becomes. ResNet is a loss function

Learning proceeds in the direction where is 0,

Because

Learning to can be seen as learning the remaining (residual). Since the input x is skipped without additional calculations, the amount of calculation is the same as before, and the gradient vanishing problem can be reduced even when the depth of the network is deepened, thus increasing the learning performance.

The residual block constructed in this embodiment is shown in FIG. 16.

는 convolutional filter

의 feature map의 개수, F는 group convolution에서의 group의 수를 나타낸다.conv is the kernel size,

Is a convolutional filter

The number of feature maps, F denotes the number of groups in the group convolution.

To learn 3D CNN, the network was trained using stochastic gradient descent with momentum, and a training sample was created from the training data image to perform data augmentation. To create a training sample, the training data image was cropped into a clip image of 48 frames.

17 shows training sample images generated to learn an abnormal behavior analysis network.

The final result of the network is classified into three classes, each of which is a general situation, a fall situation, and an assault situation. In the general situation, learning was conducted by consisting of a situation where an object is walking, a situation where the object is stopped, and a general background image where no object is detected. Regarding the fall situation, the learning was carried out by cropping the moment when one object or several objects fell. The assault situation consisted of two or more objects punching each other, kicking, and shoving.

For 3D CNN learning, ResNet-34 pre-trained with Kinetics data was used, and network training was performed with the Train/Test dataset of the training sample created as shown in [Table 1].

General situation Fall situation Assault situation Train 709 382 687 Test 35 26 200

The batch size was set to 128 and training was performed until Epoch=200. Training Accuracy and Loss of the learning network were measured as 0.9748 and 0.0689, respectively, and Validation Accuracy and Loss were measured as 0.9433 and 0.1745, respectively.

The above-described embodiments of the present invention are disclosed for the purpose of illustration, and those skilled in the art who have ordinary knowledge of the present invention will be able to make various modifications, changes, and additions within the spirit and scope of the present invention. It should be seen as falling within the scope of the following claims.

Claims (9)

As an abnormal behavior analysis device through object detection and tracking,
Processor; And
Including a memory connected to the processor,
The memory,
Detects an object existing in the input image in units of a preset frame based on a neural network,
The detected object is tracked by using a Kernerlize Correlation Filter (KCF) using a weighted gray transformation for each frame and a color histogram comparison of objects existing in different frames,
To determine whether or not to enter a preset region of interest or whether the tracked object has abnormal behavior based on a pre-learned neural network,
Abnormal behavior analysis device for storing program instructions executable by the processor. The method of claim 1,
The program instructions,
In order to track the object, an abnormal behavior analysis device that gives different weights to each of the RGB channels in consideration of the distribution of each of the RGB channels of the input image. The method of claim 1,
The program instructions,
Manage the x,y coordinates of the detected object, the first detection time, and the last detection time through tracking as an object of interest list,
An abnormal behavior analysis apparatus for updating the interest object list by determining whether the object detected in the current frame is the same object as the object managed in the interest object list. The method of claim 1,
The program instructions,
Crop the image of the tracked object every frame unit,
An abnormal behavior analysis device that determines whether an object corresponding to the cropped image has abnormal behavior when the bundle of the cropped images is continuous and the cropped image reaches a preset quantity. The method of claim 1,
The program instructions,
An abnormal behavior analysis device that determines whether the object is invading or roaming through geometric analysis of the preset region of interest and a movement trajectory of the tracked object. The method of claim 1,
The program instructions,
An abnormal behavior analysis device that determines whether the object is intrusive or roaming by using a reference line with respect to the preset region of interest and whether the movement direction of the object is a positive or reverse direction with respect to the reference line. The method of claim 1,
The program instructions,
An abnormal behavior analysis device that determines whether the object falls or is violated based on the pre-learned neural network. A method for analyzing abnormal behavior through object detection and tracking in a device including a processor and a memory,
Detecting an object existing in the input image in units of a preset frame based on a neural network;
Tracking the detected object using a Kernerlize Correlation Filter (KCF) using a weighted gray transformation for each frame and a hue-histogram comparison of objects existing in different frames; And
An abnormal behavior analysis method comprising the step of determining whether the tracked object has an abnormal behavior based on whether to enter a predetermined region of interest or a pre-learned neural network. A program stored on a recording medium performing the method according to claim 8.

Images