CN117541994A - Abnormal behavior detection model and detection method in dense multi-person scene - Google Patents

Abstract

The invention provides an abnormal behavior detection model and a detection method aiming at a dense multi-person scene. The skeleton gesture extraction module is used for extracting skeleton joint point information of each person under video monitoring. The behavior classification module models long-term time dynamics inside and between actions, predicting action tags with video sequences. Meanwhile, a full connection layer is introduced into the action recognition module to classify normal and abnormal behaviors. In order to improve the accuracy and precision of the target detection module, the invention replaces the common convolution layer of the YOLOv5 with the snake-shaped convolution layer, and simultaneously introduces an ASFF feature fusion module which can effectively fuse the extracted features of the network in a multi-scale manner. The invention also adopts a separated characteristic training strategy and a method for calculating Euclidean distance so as to solve the problem of lack of abnormal behavior data samples and under fitting of a deep neural network.

Description

An abnormal behavior detection model and detection method in dense multi-person scenarios

Technical field

The invention belongs to the technical field of computer vision. Specifically, it involves detection models and methods that combine image processing, target detection, behavior recognition and pattern recognition.

Background technique

Behavior recognition aims to find out the specific location in time or space of behaviors that people are interested in from long videos, and is one of the most basic video understanding tasks. In current behavior recognition research, detecting abnormal behaviors in video sequences is crucial. In particular, the abnormal behavior of many people in the crowd under public surveillance videos is a major potential threat. If it cannot be discovered in time, it will seriously affect the safety of people's lives and property. The diversity and rapid changes in human behavior make it difficult to detect abnormal behaviors. Although some classic algorithms have achieved good results in identifying single-person behaviors on public data sets, the accuracy of detecting abnormal behaviors of multiple people is very low. At the same time, the current mainstream algorithms with better performance have a large amount of calculations and high model complexity, making them difficult to practice and industrialize.

Most early abnormal behavior recognition research used artificial feature descriptors to represent pedestrian appearance features and motion features extracted from corresponding feature information. These methods use traditional machine learning algorithms for human action recognition. Artificial feature descriptors include trajectories, Histogram of Oriented Gradient (HOG), Histogram of Optical Flow (HOF), hybrid dynamic textures and other low-level visual features. Traditional behavior detection methods rely on a simple understanding of image features. However, with the rapid development of deep learning, researchers have begun to explore abnormal behavior identification research based on deep learning. They have achieved a series of results. Deep learning methods can extract high-level features of human behaviors in videos, thereby more effectively distinguishing normal behaviors from abnormal behaviors.

Human skeleton posture extraction uses deep learning methods to detect and track the coordinate positions of key joints (skeleton joints) of the human body from images or videos. Different from the behavior recognition method based on image features, the behavior recognition based on skeleton pose extraction is a method that uses human body pose information for action classification. It captures behavioral features through skeleton pose extraction and uses machine or deep learning methods for action classification. Human skeleton posture extraction can conduct detailed analysis of human body posture and joint motion. This enables behavior recognition to capture minute motion changes and detailed information, providing more accurate and granular behavioral analysis results.

Traditional algorithms are not stable in complex scenes and are easily affected by challenging factors such as lighting changes, angle differences, deformation and occlusion. Normal behavior is usually more common than abnormal behavior, which leads to the problem of class imbalance in the data set. This causes the model to be biased towards normal behavior that is easier to identify and less accurate at identifying abnormal behavior. At the same time, the number of people may change in different video frames, and this change in the number of people will cause the fitness of most models to decrease.

Current mainstream methods usually do not directly consider time or timing information, so they may not fully exploit time series action relationships for abnormal behavior identification tasks involving time series. Abnormal behavior is often related to specific patterns or trends in the time series. If the algorithm does not consider time information, it may not be able to accurately capture the evolution and timing relationships of abnormal behaviors, resulting in false positives or false negatives. In dense multi-person scenes, there are also complex temporal relationships between actions between people. Ignoring these relationships may also lead to oversimplified judgments of abnormal behavior and an inability to accurately distinguish between normal and abnormal behavior.

Problems with existing technologies: (1) Behavior category imbalance in the data set: Normal behaviors are usually more common than abnormal behaviors, causing the current model to be biased towards normal behaviors that are easier to identify, resulting in lower accuracy in identifying abnormal behaviors. .

(2) Adaptability decrease and number of people changes: In dense scenes, the number of people in the video frame may change, which will lead to a decrease in the adaptability of most models. The model of the present invention aims to solve this problem and can adapt to changes in the number of people in different frames, thereby improving the robustness and accuracy of the model.

(3) The problem of incorrect connection of multi-person joint points: When performing behavior recognition in dense scenes, there is the problem of incorrect connection of multi-person joint points.

Contents of the invention

The present invention provides an abnormal behavior detection model and detection method in a dense multi-person scenario, and aims to propose an abnormal behavior detection model in a dense multi-person scenario to improve the shortcomings of current mainstream algorithms in this scenario. This model aims to improve the accuracy of abnormal behavior detection in dense scenes.

The solution of the present invention to solve the technical problem is to adopt an abnormal behavior detection model in a dense multi-person scene, including a skeleton posture extraction module YH-Pose and a behavior classification module BR-LSTM. The YH-Pose module first extracts data from the original video data. Extract behavioral feature information, and then input these behavioral feature information into the BR-LSTM module to complete action classification; the YH-Pose module: is a top-down skeleton pose extraction module, which also contains human body detection detector and skeleton pose extractor. The human body detector is implemented based on YOLOv5. First, each person detected by YOLOv5 in the scene will add a bounding box. The bounding box contains the position information of the person in the image; YH -Pose module integrates the high-resolution skeleton pose extraction network HRNet, which predicts the coordinates of n skeleton joint points of each person in the image, and then connects these skeleton joint points in an orderly manner according to the structure of the human skeleton to form a human pose Skeleton model; the YH-pose module uses the input RGB video stream data to combine the human body bounding box and the posture skeleton network. The combined output result is two-dimensional human posture information, which includes the two-dimensional coordinates of k joints of the human body in each frame. information, as well as the coordinate position and confidence score of the human target frame; the BR-LSTM module: predicts action labels based on video sequences to complete the behavior classification task; the model introduces a separate feature training strategy: in the data preprocessing stage , BR-LSTM will divide the two-dimensional joint coordinate data into x coordinate sequence and y coordinate sequence, and calculate the Euclidean distance from each joint to the root node to expand the data sample; this module includes a data preprocessing module and a six-dimensional A bidirectional behavior classification network composed of LSTM units. The data preprocessing module maps action features into feature vectors through a linear layer, and inputs these feature vectors into the forward LSTM layer and the reverse LSTM layer, within each time step. , LSTM will calculate the hidden state and output of the current time step based on the current input and the hidden state of the previous time step. Finally, a fully connected layer will be used to further classify normal and abnormal behaviors.

The human detector extracts detailed skeleton data from the captured RGB video data, including skeleton joint point positions and time information of joint points between frames; in order to detect people in the image first and then detect the human skeleton posture, the YH-Pose module The high-resolution skeleton pose extraction network HRNet is combined with the YOLOv5 target detection framework; the prediction head part of YOLOv5 is improved by introducing multiple ASFF modules, which perform multi-scale weighted fusion of feature information; the input image is processed with different layers of features In the feature pyramid network of graphs, the ASFF module completely fuses the feature information between each layer and merges them into a new feature representation through weighted averaging; in traditional convolution operations, the convolution kernels are usually in a fixed order. Scan the input feature map from left to right and top to bottom; while serpentine convolution uses a non-linear scanning method, designing the scanning path of the convolution kernel into a snake shape or a curve shape, thus changing the order of scanning ;Serpentine convolution can also reduce the number of parameters of the model; in traditional convolution operations, adjacent convolution kernels usually have similar weights, so the number of parameters can be reduced by sharing parameters; and serpentine convolution due to its Nonlinear scan paths often result in different weights between adjacent convolution kernels, so parameters cannot be shared directly. In this way, each convolution kernel requires independent parameters, but it also increases the expression ability of the model. ; Therefore, in the YOLOv5 backbone network part, the serpentine convolution layer is used to replace the ordinary convolution layer, so that each convolution kernel can see a wider range of input information, thereby improving the model's perception of global features.

α ³ , β ³ , γ ³ are the weighted scaling factors of the third layer, x ^1→3 , x ^2→3 , x ^3→3 are the feature tensors of each layer; as shown in formula (1), the third layer ASFF module The new weighted feature is ASFF3:

ASFF3＝x ^1→3 *α ³ +x ^2→3 *β ³ +x ^3→3 *γ ³ (1).

Using HRNet as a model for human skeleton posture prediction, when the network wants to output N classification key points, it first outputs an N-dimensional feature map, and then builds a Gaussian kernel based on the key points on the N-dimensional annotated feature map and generates human skeleton key points. Point heat map encoding, then, obtain the (x, y) coordinates and confidence scores of n skeleton joint points from the skeleton key point heat map.

Focal Loss loss function is used in separated feature training, such as formula (2). p is the probability that the model predicts belonging to the foreground, and its value range is 0 to 1; y's value range is 1 and -1; α _t It plays a modulating role and can reduce the attention on unimportant sample features and increase the attention on challenging sample features. It is a certain parameter with a parameter range of 0 to 5;

FL(p _t )＝-α _t (1-p _t ) ^γ log(p _t ) (2)

Separate feature training uses a separate feature encoding strategy to extract skeleton features. First, a vector representing the human body posture is generated. This vector is spliced from three vectors: the normalized joint position x-axis coordinate vector, the normalized joint position y The axis coordinate vector and the distance vector from the joint to the root joint point. The distance between the person and the camera in different areas in each frame of the image is different, so the proportion of the joint positions in the image is also different. Each joint in the image is composed of its abscissa and Description of the ordinate, define the original position of the i-th joint as (x _i , y _i ), and use formula (4) to normalize the joint position of each person detected in each frame of image, Represents the normalized coordinate position of the joint. Each joint in the image is described by its abscissa and ordinate. Therefore, the normalized joint position vector contains 2k features corresponding to k joints,

After determining the coordinate positions of the human skeleton joints, the second component vector is obtained by calculating the distance from each of the p joints to the human root joint point O (center of mass). The Euclidean distance calculation formula from each joint to the root joint is (5), the joint distance vector contains k features, corresponding to k distances (d1-dp) and the Euclidean distance (x0, y0) from the joint to the root joint;

The BR-LSTM module uses bidirectional LSTM to perform feature extraction and behavior classification on skeleton information. First, k coordinates are split into x coordinate values (x1, x2,....,xk) and y coordinate values (y1, y2, ....,yk), and then calculate each joint point (d1, d2,....,dk) to the root node as the third feature component; when continuous image frames are detected, the time-varying x coordinate sequence, y coordinate sequence and distance sequence; next, the data will be processed into a length and size suitable for LSTM training, and input into three LSTM networks for temporal feature extraction, and each time a new image is detected data, new coordinate values will be added to the sequence and the old coordinate sequence will be deleted; finally, the classification information of behavioral actions is merged and input to the fully connected layer to classify normal behavior and abnormal behavior to determine whether the behavior for abnormal behavior.

The LSTM neural unit includes input gate i _t , forgetting gate f _t , unit state C _t and output gate O _t . The long and short-term memory is controlled through the gate and cell state. The calculation process is represented by formulas (6) to (11); in In formula (6), the information at time t of the input gate is a combination of the hidden output at the previous moment and the input information at time t; in formula (7), the candidate unit state at time t is calculated by h _t-1 and x _t It is obtained that h _t-1 and x _t represent the hidden output at the previous moment and the input information at time t respectively; in formula (8), the forgetting gate is used to control which information in the memory state at the previous moment should be forgotten or retained. ; In formula (9), the outputs of the input gate and the forget gate are combined to update the cell state at time t; then, the two-layer LSTM network is connected end to end, and the cells of each LSTM layer are connected in turn to predict the forward learning action. Feature sequence and reverse learned action feature sequence;

i _t =σ(W _i ·[h _t-1 , x _t ]+b _i ) (6);

C _t = tanh (W _c ·[h _t-1 , x _t ]+b _c ) (7);

f _t =σ(W _f ·[h _t-1 ,x _t ])+b _f (8);

C _t =f _t *C _t-1 +i _t *C _t (9);

O _t =σ(W _o ·[h _t-1 ,x _t ]+b _o ) (10);

h _t =O _t *tanh(C _t ) (11);

In the structure of the two-layer bidirectional LSTM, the forward layer and the backward layer are jointly connected to the output layer. The output layer contains six shared weights w ₁ -w ₆ . The forward propagation of behavioral features is in the forward layer from time 1 to Calculation is performed at time t. In the backward layer, reverse calculation is performed from time t to time 1 to obtain and save the output of the backward hidden layer at each time. The forward layer and backward layer are calculated at each time. The corresponding output results at each time are combined to obtain the final output. The mathematical formula is as (12)-(15), where, is the bias; o' _t ,o″ _t is the result of two layers of LSTM processing the action feature vector output at the corresponding moment;

An abnormal behavior detection method in a dense crowd scene, which is characterized by including the following steps: Step 1, video collection: for the dense crowd scene to be analyzed for abnormal behavior, record or obtain relevant video data; Step 2, human body Detection: Based on the detection framework of YOLOv5, add a bounding box to each person in the video and mark the position information of the person in the image; Step 3, Skeleton pose extraction: Use YH-Pose integrated with the high-resolution skeleton pose extraction network HRNet module, calculate and determine the positions of k key skeleton nodes for each person in the video; Step 4, pose skeleton model generation: According to the skeleton structure of the human body, connect the key skeleton nodes confirmed in the previous step in an orderly manner to generate the pose skeleton of the human body Model; Step 5, feature fusion: The YH-pose network uses the input RGB video frame data to fuse the human body bounding box and the posture skeleton model to generate the fused two-dimensional coordinate information and bounding box containing k joints of the human body in each frame. Position and confidence human posture information; Step 6, data preprocessing: Use the BR-LSTM module in the behavior classification stage to perform data preprocessing on the generated human posture information, including dividing the two-dimensional joint coordinates into independent x and y Coordinate sequence, and calculate the Euclidean distance from each joint to the root node; Step 7, behavioral feature extraction: The feature extraction part in the BR-LSTM module accepts the preprocessed data and extracts the spatiotemporal features of the action through the long short-term memory network; Step 8, classification and prediction: After data processing and behavioral feature extraction, the fully connected layer is used for final classification calculation, and abnormal and normal behaviors are identified and predicted through the trained model; Step 9, output result: based on classification and prediction As a result, abnormal behaviors in the video are flagged for further analysis and processing.

Beneficial effects of the present invention: through a variety of improved modules and strategies, the network within the framework is expanded so that abnormal human behavior in dense scenes can be accurately detected.

(1) The key to the target detection module is to combine HRNet with YOLOv5, use HRNet to extract high-resolution posture features, and use the improved YOLOv5 framework for target detection. By introducing the ASFF feature fusion module, the features of the two networks can be effectively fused to improve the accuracy and precision of target detection.

(2) During the training process, it was noted that the lack of sufficient abnormal behavior data samples may lead to underfitting of the model. To solve this problem, a separated feature training strategy is adopted. Specifically, the fused two-dimensional coordinate data is divided into x coordinate sequence and y coordinate sequence, and then respectively input into the network for training. In this way, existing normal behavior data can be more fully utilized to train the model, improving its performance and generalization capabilities. To enhance the model's sensitivity to abnormal behavior, the Euclidean distance from each 2D coordinate to the root node is calculated. The purpose of this is to introduce more information about the spatial relationships between joint points, making the model more capable of detecting abnormal behavior. By introducing a separated feature training strategy and a method of calculating Euclidean distance, the problem of lack of abnormal behavior data samples and underfitting of deep neural networks can be effectively solved. These improvements can improve the performance and accuracy of the model in abnormal behavior detection tasks.

Description of drawings

Figure 1 is the overall framework diagram of the abnormal behavior recognition model technical solution;

Figure 2 is a visual comparison of the skeleton joint connections of the two models;

Figure 3 is a flow chart of the multi-scale spatial feature fusion strategy;

Figure 4 is a representation of the action characteristics from the joint point to the root joint point;

Figure 5 is the internal network structure diagram of the LSTM unit;

Figure 6 is a detailed structural diagram of the two-layer bidirectional LSTM.

Detailed ways

The present invention studies an abnormal behavior detection model for dense multi-person scenes, which consists of two modular networks to realize skeleton posture extraction and behavior classification respectively. The skeleton pose extraction network is a top-down skeleton pose extraction module (YH-Pose), which can extract 17 (when K is 17) skeleton joint points of each person under video surveillance. This module describes human action information by analyzing changes in skeleton joint points in time series. This method combines the high-resolution skeleton pose extraction network (HRNet) with the improved YOLOv5 target detection frame, and introduces the feature fusion module ASFF to improve the detection accuracy of YOLOv5. At the same time, it solves the problem of skeleton joint disconnection when the HRNet network performs skeleton pose extraction.

The Behavior Classification Network is a BR-Lstm module that models long-term temporal dynamics within and between actions, leveraging approximately 2 seconds of video sequences to predict action labels. It includes a data preprocessing module and a bidirectional behavior classification network with six LSTM units. In order to solve the problem that there are few abnormal behavior data samples and deep neural networks are prone to under-fitting during the training process, a separated feature training strategy is proposed. In the BR-Lstm network, the fused two-dimensional coordinate data is divided into x-coordinate sequences. and y-coordinate sequences and compute the Euclidean distance from each 2D coordinate to the root node to augment the data sample. The data preprocessing module encodes the 2D human joint coordinates containing action features in a separate manner. It calculates the Euclidean distance of all joints to the root joint, thereby generating feature vectors for different actions. The data preprocessing module maps action detail features into action feature vectors through linear layers, and inputs the action feature vectors into the forward LSTM layer and the reverse LSTM layer. Within each time step, the forward LSTM calculates the hidden state and output of the current time step based on the current input and the hidden state of the previous time step. The reverse LSTM calculates the hidden state and output of the current time step based on the current input and the hidden state of subsequent time steps. The hidden states of forward LSTM and backward LSTM are concatenated at each time step to form a richer representation while preserving past and future contextual information. Finally, a fully connected layer is added after the bidirectional LSTM to further classify normal and abnormal behaviors.

Specifically, the overall block diagram of the intelligent abnormal behavior identification framework proposed by the present invention is shown in Figure 1. The framework integrates the skeleton pose extraction module (YH-Pose) and the behavior classification module (BR-Lstm). YH-Pose includes a human body detection module and a skeleton pose extraction module. The present invention will be further described below in conjunction with the accompanying drawings and examples.

I. Overall technical plan:

Our overall framework architecture integrates the skeleton pose extraction module (YH-Pose) and the behavior classification module (BR-Lstm). Among them, the target detector in the skeleton pose extraction module is designed based on YOLOv5 and adds a bounding box for each detected person in the scene. The human bounding box contains the position information of the person in the image. The High-Resolution Skeleton Posture Extraction Network (HRNet) is improved by using the Focal Loss loss function instead of the mean square error (MSE) loss function, which makes the model converge faster. The joint detection module incorporates an improved high-resolution skeleton pose extraction network (HRNet) and is responsible for estimating the coordinates of the 17 skeleton joint points of each person in the image. The human pose skeleton model is a structured model used to represent and capture human poses. It simulates the structure of the human skeletal system and defines the connections and constraints between joints. This module connects these joint points in an orderly manner according to the structure of the human skeleton to form a unique human posture skeleton model. The YH-pose module processes the input RGB video frames and outputs human body bounding boxes. The 2D human pose with bounding box includes the 2D coordinate information of the 17 joints of the human body in each frame, as well as the coordinate position and confidence score of the human target box.

BR-LSTM is a bidirectional convolutional extended short-term memory network that can extract spatiotemporal feature information. It integrates a preprocessing data module with posture information as input, and an action feature extraction composed of six bidirectionally linked LSTM units. module and a fully connected layer (FC layer). The preprocessing data module of the BR-LSTM network divides the two-dimensional joint coordinates of the 17 skeleton joints into independent sequences of xi and yi coordinates. It calculates the Euclidean distance di (i=1,2,...,17) from each skeleton joint to the root node. The feature extraction module extracts the spatiotemporal features of human body posture. The Dropout strategy is introduced after feature fusion to avoid overfitting. II. Human body detector module:

Detailed skeleton data is extracted from the captured RGB video data of human behavior, including detailed information of skeleton joint point positions and temporal information of joint points between frames. The YOLOv5 algorithm detects human bodies in the target area. There are four models to choose from in YOLOv5: YOLOv5s, YOLOv5m, YOLOv5l and YOLOv5x. According to the YOLOv5 paper, the YOLOv5s model has the smallest depth, the smallest width of the feature map, the lowest model complexity, and the fastest detection speed. The YOLOv5s network model has low detection accuracy when applied to target detection and cannot extract multi-scale features of the target in the image. Therefore, we introduced multiple ASFF modules before the YOLOv5s prediction head and used snake-shaped convolution layers to replace ordinary convolutions. The stacking method improves the yolov5s network model, thereby improving the accuracy without reducing the detection speed. Convolutional neural networks (CNN) require fixed-size input images. Due to different image sizes, traditional cropping methods may lose some effective information and cause unnecessary loss of accuracy. In addition, repeated convolution calculations on candidate regions will lead to problems such as computational redundancy. In order to solve the problem of inconsistent scale feature fusion, multiple ASFF modules are introduced before the YOLOv5 prediction head to perform weighted fusion of feature information. As shown in Figure 2, the input image passes through a feature pyramid network with different layers of feature maps. Without the ASFF module, each layer can only output the feature prediction results of each layer. The ASFF module can completely fuse feature information between layers. By taking a weighted average of features from different layers, they are combined into a new feature representation. α ³ , β ³ , γ ³ are the weighted scaling factors of the third layer. x ^1→3 , x ^2→3 , x ^3→3 are the feature tensors of each layer. In traditional convolution operations, the convolution kernel usually scans the input feature map from left to right and top to bottom in a fixed order. Serpentine convolution uses a non-linear scanning method and designs the scanning path of the convolution kernel into a snake shape or curve, thereby changing the scanning order. Serpentine convolution can also reduce the number of parameters of the model. In traditional convolution operations, adjacent convolution kernels usually have similar weights, so the amount of parameters can be reduced by sharing parameters. Due to its nonlinear scan path, the serpentine convolution often has different weights between adjacent convolution kernels, so parameters cannot be shared directly. In this way, each convolution kernel requires independent parameters, but it also increases the expressive ability of the model. Therefore, in the YOLOv5 backbone network part, the serpentine convolution layer is used to replace the ordinary convolution layer, so that each convolution kernel can see a wider range of input information, thus improving the model's ability to perceive global features.

ASFF3＝x ¹ _→ ³ *α ³ +x ² _→ ³ *β ³ +x ³ _→ ³ *γ ³ (1)

The current mainstream two-dimensional human skeleton pose extraction method is key point detection. It is mainly based on two aspects: joint position regression task and joint heatmap estimation task. Among the current mainstream skeleton pose extraction networks, the HRNet network has good detection results. Its entire network can maintain high resolution throughout, rather than restoring resolution through a low-to-high process, resulting in heat map predictions with higher spatial accuracy. Therefore, the YH-Pose scheme uses HRNet as the framework for skeleton pose extraction. When the network wants to output key points for N classifications, it will output N-dimensional feature maps. At the same time, a Gaussian kernel is constructed based on the key points on the N-dimensional annotated feature map, and the key points of the human skeleton are detected, thereby obtaining the (x, y) coordinates and confidence scores of the 17 skeleton joint points in the image. The Focal Loss loss function can dynamically reduce the weight of nodes that have a small impact on behavior during the training process, thereby focusing attention more quickly on nodes that have a greater impact on behavior. This method can effectively deal with the situation of uneven data samples and uneven node weight distribution, and improve the performance of the model in abnormal behavior identification tasks. Since the mean square error (MSE) loss function is very sensitive to outliers (outliers). Due to the existence of the square term, MSE will amplify the impact of outliers, causing the model to pay excessive attention to these outliers and ignore the importance of other data points. This can lead to unstable model performance in the face of outliers. Therefore, the Focal Loss loss function is used to replace the original mean square error (MSE) loss function. The Focal Loss loss function is shown in formula (2), p ranges from 0 to 1, and p _t is the probability that the model predicts belonging to the foreground. The value range of y is 1 and -1, and α _t is the weighting factor introduced. _αt is a modulation factor that can reduce the loss contribution of unimportant samples, thereby increasing the loss proportion under challenging samples. It is a definite parameter. The parameter range is 0 to 5.

FL(p _t )＝-α _t (1-p _t ) ^γ log(p _t ) (2)

III. Separate feature encoding strategy:

Skeleton feature extraction is the most critical task in the entire process. In this task, a vector representing the human posture is generated, which is composed of three vectors: the normalized joint position x-axis coordinate vector, the normalized joint position y-axis coordinate vector, and the distance from the joint to the root joint point. vector. The distance between the person and the camera in different areas of each frame of the image is different, so the proportions of joint positions in the image are also different. Each joint in the image is described by its abscissa and ordinate. Define the original position of the i-th joint as (xi _, y _i ). Use this method to normalize the joint positions of each person detected in each frame. As shown in Equation 4, Represents the normalized coordinate position of the joint. Each joint in the image is described by its abscissa and ordinate. Therefore, the normalized joint position vector contains 34 features corresponding to 17 joints.

As shown in Figure 4, after determining the coordinate positions of the human skeleton joints, the second component vector is obtained by calculating the distance from each of the 17 joints to the human body root joint point O (center of mass). The Euclidean distance from each joint to the root joint is calculated using formula (5). The joint distance vector contains 17 features, corresponding to 17 distances (d1-d17). The Euclidean distance (x0, y0) from the joint to the root joint.

IV.Double-layer bidirectional BR-Lstm module:

The BR-Lstm module uses bidirectional LSTM to perform feature extraction and behavior classification on skeleton information. First, split the 17 coordinates into x coordinate values (x1,x2,....,x17) and y coordinate values (y1,y2,....,y17), and then calculate each joint point (d1, d2,....,d17) to the root node as the third feature component. When continuous image frames are detected, the x-coordinate sequence, y-coordinate sequence and distance sequence changing over time can be obtained. Next, the data will be processed into a length and size suitable for LSTM training, and input into three LSTM networks for temporal feature extraction. After that, every time a new frame of image data is detected, new coordinate values will be added to sequence and delete the old coordinate sequence. Finally, the classification information of behavioral actions is merged and input into the fully connected layer to classify normal behavior and abnormal behavior to determine whether the behavior is abnormal. Figure 3 shows a schematic structural diagram of an LSTM neuron. The LSTM neural unit includes input gate i _t , forget gate f _t , unit state C _t and output gate O _t . Long and short-term memory is controlled through gates and cell states, and its calculation process can be expressed by the following formulas 6 to 11. In formula (6), the information at time t of the input gate is the combination of the hidden output at the previous time and the input information at time t. In formula (7), the candidate unit state at time t is calculated from h _t-1 and x _t , where h _t-1 and x _t represent the hidden output at the previous time and the input information at time t respectively. In formula (8), the forgetting gate is used to control which information in the memory state at the previous moment should be forgotten or retained. In formula (9), the outputs of the input gate and the forget gate are combined to update the cell state at time t. Then, the two-layer LSTM network is connected end to end, and the cells of each LSTM layer are connected in turn to predict the forward learning action feature sequence and the reverse learning action feature sequence.

i _t =σ(W _i ·[h _t-1 , x _t ]+b _i ) (6)

C _t = tanh (W _c ·[h _t-1 , x _t ]+b _c 0 (7)

f _t =σ(W _f ·[h _t-1 ,x _t ])+b _f (8)

C _t ＝f _t *C _t-1 +i _t *C _t (9)

O _t =σ(W _o ·[h _t-1 ,x _t ]+b _o ) (10)

h _t =O _t *tanh(C _t ) (11)

The specific structure of the two-layer bidirectional LSTM is shown in Figure 4, in which the forward layer and the backward layer are jointly connected to the output layer, and the output layer contains six shared weights w ₁ -w ₆ . The forward propagation of behavioral features is computed in the forward layer along the time from time 1 to time t. In the backward layer, reverse calculation is performed from time t to time 1 to obtain and save the output of the backward hidden layer at each time. The final output is obtained by combining the corresponding output results of the forward layer and the backward layer at each moment. The mathematical formula is as (12)-(15). In the above formula, It's bias. o′ _t ,o″ _t are the results of two layers of LSTM processing the action feature vector output at the corresponding moment.

V. Data set collection:

The public data sets used for training are HMDB51 and NTU_RGB+D and the self-built Shanghai traffic road data set. The HMDB51 dataset contains 51 types of actions and a total of 6849 videos, each of which contains at least 101 videos with a resolution of 320x320. HMDB51 contains 51 different action categories, such as "brushing", "clapping", "running" and "waving". These categories represent common human action behaviors in daily life. These action categories cover different viewing angles, movement speeds, lighting conditions, and background distractions. Therefore, accurate classification and recognition of actions is an extremely challenging task. NTU_RGB+D is a widely used skeleton-based human action recognition dataset. It contains 56,880 skeletal action sequences. There are two evaluation benchmarks, including cross-subject (X-Sub) and cross-view (X-View) settings. For X-Sub, the training and test sets are from two disconnected sets, each with 20 subjects. For X-View, the training set contains 37,920 samples captured by the camera, and the test set contains 18,960 sequences captured by the camera. The Shanghai Traffic Roads dataset consists of 100 two-minute videos that were legally captured by static overhead high-definition cameras on 32 traffic roads in Shanghai, China. The dataset is divided into three parts: 60 training videos, 13 validation videos, and 27 test videos. Videos are tagged with fine-grained action start and end times for 10 different action categories. In the study, five normal behaviors that often appear on traffic zebra crossings were selected: walking, jumping, running, turning, and smoking, and five abnormal behaviors: falling, kicking, hitting, vomiting, and looking down at mobile phones.

VI. Model training:

The equipment used for training is a server equipped with a 12th generation Intel(R) Core(TM) i9-12900K 3.19GHz CPU, 64GB memory, 64-bit Windows 10 operating system and Nvidia 3090 GPU. The program uses Anaconda3 version 5.2.0 as the integrated development environment, and the programming language is Python version 3.6.5. Various modular networks of my own design were built under the PyTorch deep learning framework.

VII. Loss function:

The cross-entropy loss function is used to calculate the difference between the predicted output of the action recognition model and the true label. For each sample, the cross-entropy loss can be calculated by the following formula (16): y_true is the true label vector and y_pred is the predicted output vector of the model.

L=-∑(y _- true ^* log(y _- pred)) (16).

VIII. Model training parameter settings:

Each sample in the three data sets is uniformly adjusted to 50 frames, and then the statistics and domain knowledge proposed by Zhang et al. are used to identify and process outliers, such as deletion, replacement or smoothing operations for data preprocessing. For the HMDB51 and NTU_RGB+D datasets, the model was trained for 110 epochs with a batch size of 200. The initial learning rate is set to 0.1 and reduced by 10 times at 80 and 120 epochs, the weight decay is set to 5e-4, and the loss is set to 0.1 as the iteration termination condition of the model. For the self-built traffic road data set, the initial learning rate is set to 0.001, the number of training rounds is 10,000, the adam optimizer is used for optimization, and the dropout rate of the last layer of FC is set to 0.5.

IX. Evaluation indicators:

The action classification performance of this framework is measured by two metrics: average precision (map) and accuracy (Acc). The two indicators Giga Floating Point Operations per Second (GFLOPs) and Frames Per Second (FPS) are used to analyze the computational complexity and detection speed of the model. Object key point similarity (OKS) is an evaluation index for joint detection and is used to compare the performance differences between the skeleton pose extraction model and other models.

The above-described specific embodiments of the present invention are only used to illustrate or explain the principles of the present invention, and do not constitute a limitation of the present invention. Therefore, any modifications, equivalent substitutions, improvements, etc. made without departing from the spirit and scope of the present invention shall be included in the protection scope of the present invention.

Claims (9)

1. An abnormal behavior detection model in dense multi-person scenes, which is characterized by including the skeleton pose extraction module YH-Pose and the behavior classification module BR-LSTM. The YH-Pose module first extracts behavioral feature information from the original video data. , and then input these behavioral feature information into the BR-LSTM module to complete action classification; the YH-Pose module: is a top-down skeleton pose extraction module, which also includes a human detector and skeleton pose extraction The human detector is implemented based on YOLOv5. First, each person detected by YOLOv5 in the scene will add a bounding box. The bounding box contains the position information of the person in the image; the YH-Pose module integrates The high-resolution skeleton pose extraction network HRNet predicts the coordinates of n skeleton joint points for each person in the image, and then connects these skeleton joint points in an orderly manner according to the structure of the human skeleton to form a human pose skeleton model; YH- The pose module uses the input RGB video stream data to combine the human body bounding box and the pose skeleton network. The combined output result is two-dimensional human pose information, which includes the two-dimensional coordinate information of k joints of the human body in each frame, as well as the human target. The coordinate position and confidence score of the box; the BR-LSTM module: predicts action labels based on video sequences to complete the behavior classification task; the model introduces a separated feature training strategy: in the data preprocessing stage, BR-LSTM will Divide the two-dimensional joint coordinate data into an x-coordinate sequence and a y-coordinate sequence, and calculate the Euclidean distance from each joint to the root node to expand the data sample; this module includes a data preprocessing module and an LSTM unit composed of six Bidirectional behavior classification network, the data preprocessing module will map the action features into feature vectors through the linear layer, and input these feature vectors into the forward LSTM layer and the reverse LSTM layer. In each time step, the LSTM will be based on the current The input and the hidden state of the previous time step are used to calculate the hidden state and output of the current time step. Finally, a fully connected layer is used to further classify normal and abnormal behaviors. 2. The abnormal behavior detection model in dense multi-person scenes according to claim 1, characterized in that the human detector extracts detailed skeleton data from the captured RGB video data, including skeleton joint point positions and inter-frame Temporal information of joint points; in order to detect people in the image first and then detect the human skeleton pose, the YH-Pose module combines the high-resolution skeleton pose extraction network HRNet with the YOLOv5 target detection framework; the prediction head part of YOLOv5 is introduced into multiple ASFF modules are improved, and these modules perform multi-scale weighted fusion of feature information; the input image passes through a feature pyramid network with different layers of feature maps, and the ASFF module completely fuses the feature information between each layer and combines them through a weighted average. Merge into a new feature representation; while serpentine convolution uses a non-linear scanning method, designing the scanning path of the convolution kernel into a snake shape or curve, thus changing the order of scanning; serpentine convolution also It can reduce the number of parameters of the model; due to its nonlinear scan path, the serpentine convolution often has different weights between adjacent convolution kernels, so parameters cannot be shared directly; in this way, each convolution kernel needs independent parameters, but also increases the expressive ability of the model; therefore, in the YOLOv5 backbone network part, the snake-shaped convolution layer is used to replace the ordinary convolution layer, so that each convolution kernel can see a wider range of input information , thereby improving the model’s ability to perceive global features. 3. The abnormal behavior detection model in dense multi-person scenes according to claim 2, characterized in that α ³ , β ³ , γ ³ are the weighted scaling factors of the third layer, x ^1→3 , x ^2→3 , x ^3→3 is the feature tensor of each layer; as shown in formula (1), the new feature after weighting of the third layer ASFF module is ASFF3: ASFF3＝x ^1→3 *α ³ +x ^2→3 *β ³ +x ^3→3 *γ ³ (1). 4. The abnormal behavior detection model in dense multi-person scenes according to claim 2, characterized in that HRNet is used as a model for human skeleton posture prediction. When the network wants to output N classification key points, it first outputs N dimensions. feature map, then, build a Gaussian kernel based on the key points on the N-dimensional annotated feature map, and generate the human skeleton key point heat map encoding. Then, obtain the (x, y) of n skeleton joint points from the skeleton key point heat map. Coordinates and confidence scores. 5. The abnormal behavior detection model in a dense multi-person scene according to claim 1, characterized in that the FocalLoss loss function is used in the separated feature training, such as formula (2), p is the probability of the model prediction belonging to the foreground, Its value range is 0 to 1; y's value range is 1 and -1; α _t plays a modulating role, which can reduce the attention to unimportant sample features and increase the attention to challenging sample features. It is a certain parameter, ranging from 0 to 5; FL(p _t )＝-α _t (1-p _t ) ^γ log(p _t ) (2) 6. The abnormal behavior detection model in a dense multi-person scene according to claim 1, characterized in that the separated feature training adopts a separated feature encoding strategy to extract skeleton features. First, a vector representing the human body posture is generated. The vector It is composed of three vectors: the normalized joint position x-axis coordinate vector, the normalized joint position y-axis coordinate vector and the joint-to-root joint point distance vector. The distance between the person and the camera in different areas in each frame of the image is The distance is different, so the proportion of the joint position in the image is also different. Each joint in the image is described by its abscissa and ordinate. The original position of the i-th joint is defined as ( _xi , _yi ), using formula (4) The joint positions of each person detected in each frame of the image are normalized, Represents the normalized coordinate position of the joint. Each joint in the image is described by its abscissa and ordinate. Therefore, the normalized joint position vector contains 2k features corresponding to k joints, After determining the coordinate positions of the human skeleton joints, the second component vector is obtained by calculating the distance from each of the p joints to the human root joint point O (center of mass). The Euclidean distance calculation formula from each joint to the root joint is (5), the joint distance vector contains k features, corresponding to k distances (d1-dp) and the Euclidean distance (x0, y0) from the joint to the root joint; 7. The abnormal behavior detection model in a dense multi-person scene according to claim 1, characterized in that the BR-LSTM module uses a bidirectional LSTM to perform feature extraction and behavior classification on the skeleton information. First, the k coordinates are split into x coordinate value (x1,x2,....,xk) and y coordinate value (y1,y2,....,yk), and then calculate each joint point (d1,d2,....,dk) to the root node as the third feature component; when continuous image frames are detected, the x coordinate sequence, y coordinate sequence and distance sequence changing over time are obtained; next, the data will be processed into a length and size suitable for LSTM training , and are respectively input into three LSTM networks for temporal feature extraction. After that, every time a new frame of image data is detected, the new coordinate value will be added to the sequence and the old coordinate sequence will be deleted; finally, the behavioral action The classification information is merged and input into the fully connected layer to classify normal behavior and abnormal behavior to determine whether the behavior is abnormal behavior. 8. The abnormal behavior detection model in dense multi-person scenarios according to claim 7, characterized in that the LSTM neural unit includes an input gate _it , a forgetting gate _ft , a unit state _Ct and an output gate _Ot . Memory is controlled through gates and cell states, and its calculation process is expressed by formulas (6) to (11); in formula (6), the information at time t of the input gate is the hidden output of the previous moment and the input information at time t. Combination; in formula (7), the candidate unit state at time t is calculated from h _t-1 and x _t , h _t-1 and x _t respectively represent the hidden output at the previous moment and the input information at time t; in the formula In (8), the forget gate is used to control which information in the memory state at the previous moment should be forgotten or retained; in formula (9), the output of the input gate and the forget gate are combined to update the cell state at time t; then , connect the two-layer LSTM network end to end, and connect the cells of each LSTM layer in turn to predict the action feature sequence of forward learning and the action feature sequence of reverse learning; i _t =σ(W _i ·[h _t-1 , x _t ]+b _i ) (6); C _t = tanh (W _c ·[h _t-1 , x _t ]+b _c ) (7); f _t =σ(W _f ·[h _t-1 ,x _t ])+b _f (8); C _t =f _t *C _t-1 +i _t *C _t (9); O _t =σ(W _o ·[h _t-1 ,x _t ]+b _o ) (10); h _t =O _t *tanh(C _t ) (11); In the structure of the two-layer bidirectional LSTM, the forward layer and the backward layer are jointly connected to the output layer. The output layer contains six shared weights w ₁ -w ₆ . The forward propagation of behavioral features is in the forward layer from time 1 to Calculation is performed at time t. In the backward layer, reverse calculation is performed from time t to time 1 to obtain and save the output of the backward hidden layer at each time. The forward layer and backward layer are calculated at each time. The corresponding output results at each time are combined to obtain the final output. The mathematical formula is as (12)-(15), where, is the bias; o' _t ,o" _t is the result of two layers of LSTM processing the action feature vector output at the corresponding moment; 9. An abnormal behavior detection method in a dense multi-person scenario, which is characterized by including the following steps: Step 1, video collection: Record or obtain relevant video data for dense crowd scenes where abnormal behavior analysis is to be performed; Step 2, human detection: Based on the YOLOv5 detection framework, add a bounding box to each person in the video and mark the person's position information in the image; Step 3, Skeleton pose extraction: Use the YH-Pose module integrated with the high-resolution skeleton pose extraction network HRNet to calculate and determine the k key skeleton node positions of each person in the video; Step 4, pose skeleton model generation: According to the skeleton structure of the human body, connect the key skeleton nodes confirmed in the previous step in an orderly manner to generate the pose skeleton model of the human body; Step 5, feature fusion: The YH-pose network uses the input RGB video frame data to fuse the human body bounding box and the posture skeleton model, and generates the fused two-dimensional coordinate information, bounding box position and k joints of the human body in each frame. Confidence of human posture information; Step 6, data preprocessing: Use the BR-LSTM module in the behavior classification stage to perform data preprocessing on the generated human posture information, including dividing the two-dimensional joint coordinates into independent x and y coordinate sequences, and calculating each joint to Euclidean distance of the root node; Step 7, behavioral feature extraction: The feature extraction part in the BR-LSTM module accepts the preprocessed data and extracts the spatiotemporal features of the action through the long short-term memory network; Step 8, classification and prediction: After data processing and behavioral feature extraction, the fully connected layer is used for final classification calculation, and abnormal and normal behaviors are identified and predicted through the trained model; Step 9, output results: Based on the classification and prediction results, mark abnormal behaviors in the video for further analysis and processing.