CN107330410B - Anomaly detection method based on deep learning in complex environment - Google Patents

Anomaly detection method based on deep learning in complex environment Download PDF

Info

Publication number
CN107330410B
CN107330410B CN201710535492.8A CN201710535492A CN107330410B CN 107330410 B CN107330410 B CN 107330410B CN 201710535492 A CN201710535492 A CN 201710535492A CN 107330410 B CN107330410 B CN 107330410B
Authority
CN
China
Prior art keywords
time
motion
adjacent
lstm model
objects
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201710535492.8A
Other languages
Chinese (zh)
Other versions
CN107330410A (en
Inventor
邱鹏
霍瑛
黄陈蓉
陈行
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Nanjing electronic Mdt InfoTech Ltd.
Original Assignee
Nanjing Institute of Technology
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Nanjing Institute of Technology filed Critical Nanjing Institute of Technology
Priority to CN201710535492.8A priority Critical patent/CN107330410B/en
Publication of CN107330410A publication Critical patent/CN107330410A/en
Application granted granted Critical
Publication of CN107330410B publication Critical patent/CN107330410B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/20—Movements or behaviour, e.g. gesture recognition
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • G06T7/246—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/40—Scenes; Scene-specific elements in video content
    • G06V20/41—Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items
    • G06V20/42—Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items of sport video content
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10016—Video; Image sequence
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30196—Human being; Person

Abstract

The invention provides an anomaly detection method based on deep learning in a complex environment, which comprises the steps of inputting object space-time characteristics extracted by a convolutional neural network regression method into an LSTM model, and tracking the motion tracks of multiple objects in the complex environment; capturing nonlinear space-time actions of adjacent individuals under the condition that multiple objects move irregularly, evaluating the dependency of motion tracks between the adjacent individuals, and predicting the future motion tracks of the adjacent individuals; and completing anomaly detection according to the anomaly probability of the future motion trail of the individual. The LSTM model predicts the future motion trail of the object by using a coding and decoding framework through evaluating the dependency among the coherent individuals, thereby obtaining a more accurate result when carrying out abnormal detection on the motion of a plurality of objects.

Description

Anomaly detection method based on deep learning in complex environment
Technical Field
The invention relates to an anomaly detection method based on deep learning in a complex environment.
Background
In general, anomaly detection refers to detecting anomalous or unexpected behavior data in an environment. With the popularization of deep learning in the field of artificial intelligence, anomaly detection in complex environments such as subways, stadiums, airports and the like is widely applied to computer vision technology, but the high-density environments bring huge challenges to anomaly detection. In the face of continuous and irregular motion of a large number of objects, how to solve the problem of mutual interference between the objects and how to detect abnormality under the condition that motion tracks of a plurality of objects mutually influence each other becomes an important problem in the current abnormality detection research.
The anomaly may be an unusual shape or action, and in the existing research results, a normal video frame region is obtained through machine learning, and the region is used as a reference model, and the reference model comprises a normal event or a normal training data set. In the testing phase, the researcher treats regions other than the reference model as anomalies. However, such non-standard reference models are often difficult to define accurately because of the presence of normal events with special attributes, and furthermore, it is difficult to prepare large data sets for training covering different domains.
At present, the methods for anomaly detection in a complex environment mainly comprise the following two methods: 1) based on a trajectory method: an object is said to be abnormal if it does not follow a normal trajectory or if it occurs less frequently; 2) based on the dynamic method: compared with the normal moving object, the abnormal object has a obviously different action mode.
The problem of anomaly detection in a complex environment can be summarized as the problem of anomaly detection by constructing a deep learning model, and the research difficulty is how to design a learning model with static data, sequence data and spatial data and how to effectively apply the data to the learning model. Particularly, the prior art is mainly responsible for detecting the space-time characteristics of a single object, and does not consider the condition that the motion tracks of adjacent individuals interfere with each other in a complex environment, so that the anomaly detection effect is not ideal. Therefore, the method for carrying out anomaly detection in the complex environment by designing the deep learning model has important theoretical significance and application value. However, there is no description in the prior art.
Disclosure of Invention
The invention aims to provide an anomaly detection method based on deep learning in a complex environment, which can reduce the false detection rate of images under the condition that the motions of adjacent individuals have mutual influence, has better performance particularly when anomaly detection is carried out in a crowded environment, provides a new idea for solving the problem of anomaly detection in the complex environment, and solves the problem that the anomaly detection effect is not ideal because the motion tracks of the adjacent individuals interfere with each other under the condition that the complex environment is not considered in the prior art.
The technical solution of the invention is as follows:
a method for detecting abnormality based on deep learning in a complex environment comprises the following steps of tracking multiple object trajectories through a long-term and short-term memory model, capturing nonlinear space-time actions between adjacent individuals, predicting future motion trajectories of the adjacent individuals, and completing abnormality detection according to the abnormality probability of the future motion trajectories of the individuals, wherein the method comprises the following steps:
step 1, inputting the object space-time characteristics extracted by a convolutional neural network regression method into an LSTM model, and tracking the motion tracks of multiple objects in a complex environment;
step 2, capturing nonlinear space-time actions of adjacent individuals under the condition that multiple objects move irregularly, evaluating the dependency of motion tracks between the adjacent individuals, and predicting the future motion tracks of the adjacent individuals;
and 3, completing anomaly detection according to the anomaly probability of the future motion trail of the individual.
Further, step 1 specifically comprises:
step 11, inputting the object space-time characteristics extracted by the convolutional neural network regression method into an LSTM model; introducing YOLO to solve the object detection as a regression problem, finishing the input from an original image to the output of the positions, the categories and the corresponding confidence probabilities of all objects in the image, and taking a feature vector obtained through the YOLO as an input frame of an LSTM model; function of input frame as
Figure GDA0002426881730000022
Wherein phi (x)t) Input frame function, x, representing an LSTM modeltFor the original image input frame at time t, convθc(.) is the parameter θcThe convolutional neural network of (a) is,
Figure GDA0002426881730000023
representing the previous frame xt-1A predicted object position;
step 12, tracking the motion tracks of multiple objects in a complex environment; the LSTM model is a deep recursive network for regressing the pixel brightness and position of the object bounding box, and using them as the original input frame to perform frame-by-frame detection and tracking, and the mathematical expression of the whole-course trajectory tracking probability is
Figure GDA0002426881730000021
Wherein, BTAnd XTRespectively the position of the object at the maximum time TAnd input frame, BtThen it is the position of the object at time T, and 1. ltoreq. t.ltoreq.T, B<TIs all positions, X, before time T≤TAre all input frames up to time T.
Further, step 2 specifically comprises:
step 21, capturing nonlinear space-time motion of adjacent individuals under the condition that multiple objects move irregularly, specifically:
step 211, judging whether the two motion tracks of the two adjacent objects in the time-space domain are coherent, specifically if the motion tracks of the two dynamic objects are coherent in the time-space domain, that is, when the relative speed of the adjacent objects is kept unchanged, the two dynamic objects have similar hidden states;
step 212, for the motion trajectory of each object, the LSTM model will create and track a series of nonlinear spatio-temporal actions of the object, and the states of the object and the neighboring objects are integrated through coherent regularization, so as to update the states of the storage unit of the LSTM model, where the coherent regularization expression is:
Figure GDA0002426881730000031
wherein, ctAccumulator representing the state information of the memory cell, ftFor forgetting the gate, for resetting the state of the memory cell, if the forgetting gate is activated, the state c of the memory cell at the last timet-1It will be forgotten that the user is,
Figure GDA0002426881730000032
representing an array operation; i.e. itFor the input gate, x is input at the current timetAnd a time-last hidden layer ht-1Is activated; w is a weight matrix, WxcFor recursive memory cell state input matrices, WhcRecursive hidden state input matrix, bcIs a deviation vector;
Figure GDA0002426881730000033
representing the trajectory and spatio-temporal characteristics of adjacent objects, λj(t) is the weight of dependence between objects, fj tAnd cj t-1Respectively, the current heritage of coherent objects of the LSTM modelForgetting a door state and a last time storage unit state;
step 22, evaluating the dependency of the motion trail between the adjacent individuals, which specifically comprises the following steps:
step 221, obtaining time-varying characteristics of adjacent individual movements through the hidden state information of the LSTM model;
step 222, evaluating the dependency of the motion trail between the adjacent individuals by using the pairwise velocity correlation, and the weight lambda of the dependency of the motion trail between the adjacent individualsjThe expression of (t) is:
Figure GDA0002426881730000034
where i and j represent the motion trajectories of adjacent individuals, vi(t) and vj(t) represents the respective velocities of the adjacent objects, σ is a normalization constant, the two velocity values are multiplied and normalized by the normalization constant, γiFor obtaining the dependency weight, when the deviation of the motion tracks i and j of the adjacent individuals is larger, lambda isjThe closer the value of (t) is to 0, the higher the similarity between the motion trajectories i and j of the adjacent individuals, the higher the λjThe closer the value of (t) approaches 1;
and step 23, training the LSTM model by adopting an encoding-decoding framework to predict the future motion trail of the object.
Further, step 23 specifically includes:
231, mapping the input of the motion trajectory to a fixed-length vector by an LSTM model-based encoder through learning training, wherein an implicit vector in the encoding stage is described as follows by an expression: h isT＝LSTMe(ZT,hT-1)，hTFor implicit vectors of current time, LSTMeInput Z representing motion trajectory of object by LSTM model-based encoderTMapping to a last-in-time implicit vector hT-1；
Step 232, in the learning training, the decoder based on the LSTM model predicts the future motion trajectory of the object by using a fixed-length implicit vector, where the implicit vector expression is:
Figure GDA0002426881730000035
LSTMdrepresenting decoder based on LSTM model using last-time implicit vector hT-1Obtaining an implicit vector h of the current timeTThen, inputting Z through the motion track of the object at the current timeTPredicting next-time object motion trajectory as output
Figure GDA0002426881730000041
The invention has the beneficial effects that: compared with the prior art, the anomaly detection method based on deep learning in the complex environment has the remarkable advantages that:
firstly, the false detection rate of the prior art is high, the object space-time characteristics extracted by the convolutional neural network regression method are input into an LSTM model, and the object detection is solved as a regression problem by introducing YOLO, so that the false detection rate of the image is reduced.
The LSTM model predicts the future motion trail of the object by using a coding and decoding framework through evaluating the dependency among the coherent individuals by evaluating the dependency among the coherent individuals, thereby optimizing the space-time robustness of an abnormal detection system and obtaining a more accurate result when performing abnormal detection on the motion of multiple objects.
Drawings
Fig. 1 is an overview of the tracking process of the motion trajectory of an object.
FIG. 2 is a graph of the relationship between adjacent individuals in the LSTM model.
Fig. 3 is a diagram of an encoding-decoding framework of the LSTM model.
FIG. 4 is a diagram of a convolutional neural network tracing invisible sequences over crowded streets.
Fig. 5 is a time robustness tre (temporal robustness evaluation) evaluation diagram of the anomaly detection method based on deep learning in a complex environment according to the embodiment.
Fig. 6 is a spatial robustness sre (spatial robustness evaluations) evaluation diagram of the anomaly detection method based on deep learning in the complex environment according to the embodiment.
Fig. 7 is an abnormality detection accuracy chart of the abnormality detection method based on deep learning in the complex environment according to the embodiment.
Detailed Description
Preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings.
The anomaly detection method based on deep learning in the complex environment carries out multi-object trajectory tracking through a long-short term memory model, captures nonlinear space-time actions of adjacent individuals and predicts two parts of future motion trajectories of the adjacent individuals. In the process of tracking multiple object tracks through a long-short term memory model (LSTM), by introducing YOLO (only one eye of YouOnly Look Once-a real-time rapid target detection technology) detection, object space-time characteristics extracted by a convolutional neural network regression method are input into the LSTM model, and the motion tracks of multiple objects in a complex environment are tracked; in capturing nonlinear space-time motion between adjacent individuals and predicting future motion trajectories thereof, under the condition that multiple objects move irregularly, the dependency of the motion trajectories between the adjacent individuals is evaluated, the future motion trajectories of the objects are predicted, and the abnormality detection is completed according to the abnormality probability of the future motion trajectories of the individuals. The anomaly detection method provided by the invention can reduce the false detection rate of the image under the condition that the motions of adjacent individuals have mutual influence, has better performance particularly when the anomaly detection is carried out in a crowded environment, and provides a new thought for solving the problem of anomaly detection in a complex environment.
Examples
An anomaly detection method based on deep learning in a complex environment comprises the steps of firstly tracking multiple object tracks through a long-short term memory model (LSTM), collecting space-time characteristics of objects and preliminarily deducing the object running tracks by introducing YOLO training and detection, inputting the object space-time characteristics extracted by a convolutional neural network regression method into the LSTM, and tracking the motion tracks of multiple objects in the complex environment; and then capturing nonlinear space-time motion between adjacent individuals, evaluating the dependency of motion tracks between the adjacent individuals under the condition that the multiple objects move irregularly, predicting the future motion track of the object, and completing anomaly detection according to the anomaly probability of the future motion track of the individual. The method specifically comprises the following steps:
step 1, inputting the object space-time characteristics extracted by the convolutional neural network regression method into an LSTM model, and tracking the motion tracks of multiple objects in a complex environment, wherein FIG. 1 is an overview diagram of the tracking process of the motion tracks of the objects.
Step 11, inputting the object space-time characteristics extracted by the convolutional neural network regression method into an LSTM model; introducing YOLO to solve the object detection as a regression problem, finishing the input from an original image to the output of the positions, the categories and the corresponding confidence probabilities of all objects in the image, and taking a feature vector obtained through the YOLO as an input frame of an LSTM model; function of input frame as
Figure GDA0002426881730000051
Wherein phi (x)t) Input frame function, x, representing an LSTM modeltFor the original image input frame at time t, convθc(.) is the parameter θcThe convolutional neural network of (a) is,
Figure GDA0002426881730000052
representing the previous frame xt-1A predicted object position;
step 12, tracking the motion tracks of multiple objects in a complex environment; the LSTM model is a deep recursive network for regressing the pixel brightness and position of the object bounding box, and using them as the original input frame to perform frame-by-frame detection and tracking, and the mathematical expression of the whole-course trajectory tracking probability is
Figure GDA0002426881730000053
Wherein, BTAnd XTRespectively object position and input frame at maximum time T, BtThen it is the position of the object at time T, and 1. ltoreq. t.ltoreq.T, B<TIs all positions, X, before time T≤TAre all input frames up to time T.
And 2, capturing nonlinear space-time actions of adjacent individuals under the condition that multiple objects move irregularly, evaluating the dependency of motion tracks between the adjacent individuals, and predicting the future motion tracks of the adjacent individuals.
Step 21, capturing nonlinear space-time motion of adjacent individuals under the condition of irregular motion of multiple objects.
And step 211, judging whether the two motion tracks of the two adjacent objects in the time-space domain are coherent, specifically if the motion tracks of the two dynamic objects are coherent in the time-space domain, that is, when the relative speed of the adjacent objects is kept unchanged, the two motion tracks have similar hidden states.
Step 212, for the motion trajectory of each object, the LSTM model will create and track a series of nonlinear spatio-temporal actions of the object, and the states of the object and the neighboring objects are integrated through coherent regularization, so as to update the states of the storage unit of the LSTM model, where the coherent regularization expression is:
Figure GDA0002426881730000061
wherein, ctAccumulator representing the state information of the memory cell, ftFor forgetting the gate, for resetting the state of the memory cell, if the forgetting gate is activated, the state c of the memory cell at the last timet-1It will be forgotten that the user is,
Figure GDA0002426881730000062
representing an array operation; i.e. itFor the input gate, x is input at the current timetAnd a time-last hidden layer ht-1Is activated; w is a weight matrix, WxcFor recursive memory cell state input matrices, WhcRecursive hidden state input matrix, bcIs a deviation vector;
Figure GDA0002426881730000063
representing the trajectory and spatio-temporal characteristics of adjacent objects, λj(t) is the weight of dependence between objects, fj tAnd cj t-1Respectively representing the current forgetting gate state and the last time storage unit state of the LSTM model coherent object;
and step 22, evaluating the dependency of the motion trail between adjacent individuals, wherein the relation between the adjacent individuals in the LSTM model is described in figure 2.
Step 221, obtaining time-varying characteristics of adjacent individual movements through the hidden state information of the LSTM model;
step 222, evaluating the dependency of the motion trail between the adjacent individuals by using the pairwise velocity correlation, and the weight lambda of the dependency of the motion trail between the adjacent individualsjThe expression of (t) is:
Figure GDA0002426881730000064
where i and j represent the motion trajectories of adjacent individuals, vi(t) and vj(t) represents the respective velocities of the adjacent objects, σ is a normalization constant, the two velocity values are multiplied and normalized by the normalization constant, γiFor obtaining the dependency weight, when the deviation of the motion tracks i and j of the adjacent individuals is larger, lambda isjThe closer the value of (t) is to 0, the higher the similarity between the motion trajectories i and j of the adjacent individuals, the higher the λjThe closer the value of (t) approaches 1;
and step 23, as shown in fig. 3, training the LSTM model by using an encoding-decoding framework to predict the future motion trajectory of the object.
231, mapping the input of the motion trajectory to a fixed-length vector by an LSTM model-based encoder through learning training, wherein an implicit vector in the encoding stage is described as follows by an expression: h isT＝LSTMe(ZT,hT-1)，hTFor implicit vectors of current time, LSTMeInput Z representing motion trajectory of object by LSTM model-based encoderTMapping to a last-in-time implicit vector hT-1。
Step 232, in the learning training, the decoder based on the LSTM model predicts the future motion trajectory of the object by using a fixed-length implicit vector, where the implicit vector expression is:
Figure GDA0002426881730000071
LSTMdrepresenting decoder based on LSTM model using last-time implicit vector hT-1Obtaining an implicit vector h of the current timeTThen, the motion track of the object is input according to the current timeZTPredicting next-time object motion trajectory as output
Figure GDA0002426881730000072
And 3, obtaining the abnormal probability of the future motion trail of the individual according to the future motion trail of the individual obtained in the step 2, and completing abnormal detection according to the abnormal probability of the future motion trail of the individual.
Experimental validation analysis was as follows:
different data sets are run by using an artificial intelligence learning system Tensorflow to detect, track and identify abnormal objects, and the screens for training in the YOLO detection and LSTM models are 20FPS (Frames Per second) and 60FPS respectively. Tracking the motion tracks of multiple objects in a complex environment according to step 1, and in order to avoid an over-fitting problem, a deep learning structure requires training a large number of data sets, so that a convolutional neural network is put in advance in the ImageNet with the largest image recognition in the world for data training, and then an LSTM model is used for Fine-tuning (Fine-tuning) on a smaller data set PASCAL VOC 2012, so that the data set PASCAL VOC 2012 can detect 20 different types of objects. FIG. 4 is a diagram of tracing invisible sequence traces over crowded streets through a convolutional neural network.
Step 2, capturing nonlinear space-time actions of adjacent individuals under the condition that multiple objects move irregularly, evaluating the dependency of motion tracks between the adjacent individuals, and predicting the future motion tracks of the adjacent individuals specifically comprises the following steps:
step 21, capturing nonlinear space-time motion of adjacent individuals under the condition of irregular motion of multiple objects.
The LSTM model was trained and tested in the OTB-30 dataset. 80% of the data was used in the training phase and the remaining 20% in the experimental testing phase, and the model was trained item by item for better experimental results. The results of the experiment are shown in fig. 5 and 6. As can be seen from fig. 5 and 6, the proposed method is superior to other trajectory tracking methods in both temporal and spatial robustness evaluation.
Step 22, evaluating the dependency of the motion tracks between adjacent individuals, step 23, training the LSTM model to predict the future motion track of the object by using an encoding-decoding framework, wherein a step size (step size) concept is used in the experiment, the step size represents the number of all previous frames considered when the encoding-decoding framework based on the LSTM model predicts the future track, and the experimental result is shown in fig. 7. Fig. 7 shows an abnormality detection accuracy chart according to the detection evaluation function interaction-over-unity (iou) in the proposed method under the condition of step size change.
Therefore, the embodiment method has advantages compared with other algorithms in both space-time robustness and false detection rate, and particularly has more obvious advantages in a complex environment in which motion tracks of multiple objects influence each other, so that the effectiveness of the embodiment method is verified.

Claims (2)

1. An anomaly detection method based on deep learning in a complex environment is characterized in that: the method comprises the following steps of tracking multiple object tracks through a long-term and short-term memory model, capturing nonlinear space-time actions between adjacent individuals, predicting future motion tracks of the adjacent individuals, and completing anomaly detection according to the anomaly probability of the future motion tracks of the individuals, wherein the method specifically comprises the following steps:
step 1, inputting the object space-time characteristics extracted by a convolutional neural network regression method into an LSTM model, and tracking the motion tracks of multiple objects in a complex environment;
step 2, capturing nonlinear space-time actions of adjacent individuals under the condition that multiple objects move irregularly, evaluating the dependency of motion tracks between the adjacent individuals, and predicting the future motion tracks of the adjacent individuals; the step 2 specifically comprises the following steps:
step 21, capturing nonlinear space-time motion of adjacent individuals under the condition that multiple objects move irregularly, specifically:
step 211, judging whether the two motion tracks of the two adjacent objects in the time-space domain are coherent, specifically if the motion tracks of the two dynamic objects are coherent in the time-space domain, that is, when the relative speed of the adjacent objects is kept unchanged, the two dynamic objects have similar hidden states;
step 212, for the motion trajectory of each object, the LSTM model will create and track a series of nonlinear spatio-temporal actions of the object, and the states of the object and the neighboring objects are integrated through coherent regularization, so as to update the states of the storage unit of the LSTM model, where the coherent regularization expression is:
Figure FDA0002426881720000011
wherein, ctAccumulator representing the state information of the memory cell, ftFor forgetting the gate, for resetting the state of the memory cell, if the forgetting gate is activated, the state c of the memory cell at the last timet-1Will be forgotten, ⊙ represents an array operation, itFor the input gate, x is input at the current timetAnd a time-last hidden layer ht-1Is activated; w is a weight matrix, WxcFor recursive memory cell state input matrices, WhcRecursive hidden state input matrix, bcIs a deviation vector;
Figure FDA0002426881720000012
representing the trajectory and spatio-temporal characteristics of adjacent objects, λj(t) is the weight of dependence between objects, fj tAnd cj t-1Respectively representing the current forgetting gate state and the last time storage unit state of the LSTM model coherent object;
step 22, evaluating the dependency of the motion trail between the adjacent individuals, which specifically comprises the following steps:
step 221, obtaining time-varying characteristics of adjacent individual movements through the hidden state information of the LSTM model;
step 222, evaluating the dependency of the motion trail between the adjacent individuals by using the pairwise velocity correlation, and the weight lambda of the dependency of the motion trail between the adjacent individualsjThe expression of (t) is:
Figure FDA0002426881720000021
where i and j represent the motion trajectories of adjacent individuals, vi(t) and vj(t) represents the respective velocities of adjacent objects, σ is a normalization constant, and the two velocity values are multiplied and passedNormalization with an over-normalization constant, gammaiFor obtaining the dependency weight, when the deviation of the motion tracks i and j of the adjacent individuals is larger, lambda isjThe closer the value of (t) is to 0, the higher the similarity between the motion trajectories i and j of the adjacent individuals, the higher the λjThe closer the value of (t) approaches 1;
step 23, adopting an encoding-decoding frame to train an LSTM model to predict the future motion track of the object; step 23 specifically comprises:
231, mapping the input of the motion trajectory to a fixed-length vector by an LSTM model-based encoder through learning training, wherein an implicit vector in the encoding stage is described as follows by an expression: h isT＝LSTMe(ZT,hT-1)，hTFor implicit vectors of current time, LSTMeInput Z representing motion trajectory of object by LSTM model-based encoderTMapping to a last-in-time implicit vector hT-1；
Step 232, in the learning training, the decoder based on the LSTM model predicts the future motion trajectory of the object by using a fixed-length implicit vector, where the implicit vector expression is:
Figure FDA0002426881720000022
LSTMdrepresenting decoder based on LSTM model using last-time implicit vector hT-1Obtaining an implicit vector h of the current timeTThen, inputting Z through the motion track of the object at the current timeTPredicting next-time object motion trajectory as output
Figure FDA0002426881720000023
And 3, completing anomaly detection according to the anomaly probability of the future motion trail of the individual.
2. The anomaly detection method based on deep learning in the complex environment according to claim 1, wherein the step 1 specifically comprises:
step 11, inputting the object space-time characteristics extracted by the convolutional neural network regression method into an LSTM model; introducing YOLO derivativesThe body detection is used as regression problem solving to finish the output of the positions, the categories and the corresponding confidence probabilities of all objects in the image from the input of the original image, and the feature vector obtained by the YOLO is used as an input frame of the LSTM model; function of input frame as
Figure FDA0002426881720000024
Wherein phi (x)t) Input frame function, x, representing an LSTM modeltFor the original image input frame at time t, convθc(.) is the parameter θcThe convolutional neural network of (a) is,
Figure FDA0002426881720000025
representing the previous frame xt-1A predicted object position;
step 12, tracking the motion tracks of multiple objects in a complex environment; the LSTM model is a deep recursive network for regressing the pixel brightness and position of the object bounding box, and using them as the original input frame to perform frame-by-frame detection and tracking, and the mathematical expression of the whole-course trajectory tracking probability is
Figure FDA0002426881720000031
Wherein, BTAnd XTRespectively object position and input frame at maximum time T, BtThen it is the position of the object at time T, and 1. ltoreq. t.ltoreq.T, B<TIs all positions, X, before time T≤TAre all input frames up to time T.
CN201710535492.8A 2017-07-03 2017-07-03 Anomaly detection method based on deep learning in complex environment Active CN107330410B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201710535492.8A CN107330410B (en) 2017-07-03 2017-07-03 Anomaly detection method based on deep learning in complex environment

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201710535492.8A CN107330410B (en) 2017-07-03 2017-07-03 Anomaly detection method based on deep learning in complex environment

Publications (2)

Publication Number Publication Date
CN107330410A CN107330410A (en) 2017-11-07
CN107330410B true CN107330410B (en) 2020-06-30

Family

ID=60198872

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201710535492.8A Active CN107330410B (en) 2017-07-03 2017-07-03 Anomaly detection method based on deep learning in complex environment

Country Status (1)

Country Link
CN (1) CN107330410B (en)

Families Citing this family (22)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107909084B (en) * 2017-11-15 2021-07-13 电子科技大学 Haze concentration prediction method based on convolution-linear regression network
CN107948166B (en) * 2017-11-29 2020-09-25 广东亿迅科技有限公司 Deep learning-based traffic anomaly detection method and device
CN108090558B (en) * 2018-01-03 2021-06-08 华南理工大学 Automatic filling method for missing value of time sequence based on long-term and short-term memory network
CN108288032B (en) * 2018-01-08 2020-11-10 深圳市腾讯计算机系统有限公司 Action characteristic acquisition method, device and storage medium
CN108537818B (en) * 2018-03-07 2020-08-14 上海交通大学 Crowd trajectory prediction method based on cluster pressure LSTM
CN108320297B (en) * 2018-03-09 2020-06-19 湖北工业大学 Video target real-time tracking method and system
CN108564118B (en) * 2018-03-30 2021-05-11 陕西师范大学 Crowd scene pedestrian trajectory prediction method based on social affinity long-term and short-term memory network model
CN108805015B (en) * 2018-04-26 2021-09-03 常州大学 Crowd abnormity detection method for weighted convolution self-coding long-short term memory network
CN109086797B (en) * 2018-06-29 2021-12-28 中国地质大学（武汉） Abnormal event detection method and system based on attention mechanism
CN109359519B (en) * 2018-09-04 2021-12-07 杭州电子科技大学 Video abnormal behavior detection method based on deep learning
CN109409499B (en) * 2018-09-20 2022-03-15 北京航空航天大学 Track recovery method based on deep learning and Kalman filtering correction
CN109787699B (en) * 2018-10-18 2020-09-08 国网江苏省电力有限公司信息通信分公司 Wireless sensor network routing link state prediction method based on mixed depth model
CN109977840A (en) * 2019-03-20 2019-07-05 四川川大智胜软件股份有限公司 A kind of airport scene monitoring method based on deep learning
CN110532852B (en) * 2019-07-09 2022-10-18 长沙理工大学 Subway station pedestrian abnormal event detection method based on deep learning
CN111179365A (en) * 2019-12-11 2020-05-19 中国科学院高能物理研究所 Mobile radioactive source radiation image self-adaptive superposition optimization method based on recurrent neural network
CN111200540A (en) * 2019-12-27 2020-05-26 合肥学院 Big data computer system fault detection method based on deep recursion network
CN111831870B (en) * 2020-06-12 2024-02-13 北京百度网讯科技有限公司 Abnormality detection method and device for spatiotemporal data, electronic equipment and storage medium
CN111783960A (en) * 2020-07-09 2020-10-16 中国人民解放军国防科技大学 Ship track prediction method and system based on automatic encoder and bidirectional LSTM
CN111783738A (en) * 2020-07-29 2020-10-16 中国人民解放军国防科技大学 Abnormal motion trajectory detection method for communication radiation source
CN112036267A (en) * 2020-08-14 2020-12-04 珠海格力电器股份有限公司 Target detection method, device, equipment and computer readable storage medium
CN113516304B (en) * 2021-06-29 2024-01-23 上海师范大学 Regional pollutant space-time joint prediction method and device based on space-time diagram network
CN115019206B (en) * 2022-06-13 2023-02-03 北京拙河科技有限公司 Airplane landing environment safety detection method and system

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101141633A (en) * 2007-08-28 2008-03-12 湖南大学 Moving object detecting and tracing method in complex scene
CN104272377A (en) * 2012-02-06 2015-01-07 莱金德3D有限责任公司 Motion picture project management system
CN105791705A (en) * 2016-05-26 2016-07-20 厦门美图之家科技有限公司 Video anti-shake method and system suitable for movable time-lapse photography and shooting terminal

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9916538B2 (en) * 2012-09-15 2018-03-13 Z Advanced Computing, Inc. Method and system for feature detection
US10108325B2 (en) * 2014-12-11 2018-10-23 Rdi Technologies, Inc. Method of analyzing, displaying, organizing and responding to vital signals

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101141633A (en) * 2007-08-28 2008-03-12 湖南大学 Moving object detecting and tracing method in complex scene
CN104272377A (en) * 2012-02-06 2015-01-07 莱金德3D有限责任公司 Motion picture project management system
CN105791705A (en) * 2016-05-26 2016-07-20 厦门美图之家科技有限公司 Video anti-shake method and system suitable for movable time-lapse photography and shooting terminal

Also Published As

Publication number Publication date
CN107330410A (en) 2017-11-07

Similar Documents

Publication Publication Date Title
CN107330410B (en) Anomaly detection method based on deep learning in complex environment
Chang et al. Clustering driven deep autoencoder for video anomaly detection
CN108805015B (en) Crowd abnormity detection method for weighted convolution self-coding long-short term memory network
CN111860162B (en) Video crowd counting system and method
Xu et al. Learning deep representations of appearance and motion for anomalous event detection
Chetverikov et al. Dynamic texture as foreground and background
Doulamis Adaptable deep learning structures for object labeling/tracking under dynamic visual environments
CN104219488B (en) The generation method and device and video monitoring system of target image
Medel Anomaly detection using predictive convolutional long short-term memory units
Wei et al. Detecting video anomaly with a stacked convolutional LSTM framework
Soyguder Intelligent control based on wavelet decomposition and neural network for predicting of human trajectories with a novel vision-based robotic
Ansari et al. An expert video surveillance system to identify and mitigate shoplifting in megastores
Wei et al. End-to-end video saliency detection via a deep contextual spatiotemporal network
CN112488014B (en) Video prediction method based on gated cyclic unit
Yu et al. Regularity learning via explicit distribution modeling for skeletal video anomaly detection
Yan et al. Memory clustering autoencoder method for human action anomaly detection on surveillance camera video
US11810351B2 (en) Video analytic processing with neuro-symbolic artificial intelligence
Halim Intelligent Human Anomaly Identification and Classification in Crowded Scenes via Multi-fused Features and Restricted Boltzmann Machines
Li Moving object detection for unseen videos via truncated weighted robust principal component analysis and salience convolution neural network
Muhamad et al. A comparative study using improved LSTM/GRU for human action recognition
Qiu et al. Anomaly detection in a crowd using a cascade of deep learning networks
Ashok et al. FINGER RECONGITION AND GESTURE BASED VIRTUAL KEYBOARD
Gayal et al. Timber–Prairie Wolf Optimization-Dependent Deep Learning Classifier for Anomaly Detection in Surveillance Videos
Chen et al. Spatial-temporal context-aware abnormal event detection based on incremental sparse combination learning
Wu et al. Skeleton-based pedestrian abnormal behavior detection with spatio-temporal model in public places

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant
TR01 Transfer of patent right
TR01 Transfer of patent right

Effective date of registration: 20210401

Address after: Room 503-2, 5 / F, R & D East Building, Guodian Nanzi, No. 9, Huida Road, Jiangbei new district, Nanjing City, Jiangsu Province, 210032

Patentee after: Nanjing electronic Mdt InfoTech Ltd.

Address before: 1 No. 211167 Jiangsu city of Nanjing province Jiangning Science Park Hongjing Road

Patentee before: NANJING INSTITUTE OF TECHNOLOGY