TW202309772A - Behavior recognition method and apparatus, electronic device, and storage medium - Google Patents

Abstract

A behavior recognition method and apparatus, an electronic device, and a storage medium. The method comprises: receiving an input video frame and extracting character features in the video frame; clustering the multiple character features in the video frame to obtain a clustering result; determining attention allocation weights of the character features in the video frame on the basis of the clustering result; updating the character features on the basis of the attention allocation weights; extracting character spatio-temporal features on the basis of the updated character features; and performing behavior recognition on the video frame on the basis of the character spatio-temporal features to obtain a recognition result.

Description

Behavior recognition method, electronic device and storage medium

The present invention relates to the field of computer technology, in particular to a behavior recognition method, electronic equipment and storage media.

In group activity recognition (Group Activity Recognition) technology, computer vision algorithms are used to identify the action categories of different people in the movie screen, and the crowd behavior categories described in the screen, which are often used for behavior recognition in scenes such as sports events. For example, for a video of a volleyball match, the task requires identifying the action category of each volleyball player and the category of crowd activity described in the video (left pass, right pass, left spike, etc.). For this task, we can usually detect the human body in the film first, and then use Individual Action Recognition (Individual Action Recognition) to further infer the crowd behavior category of the film based on the individual's actions.

In recent years, with the development of deep learning in computer vision, many works in the past usually use convolutional neural networks to detect the actions of each person in the film, and use global pooling to obtain the overall characteristics of the crowd to identify crowd behavior categories. Crowd behavior recognition not only depends on individual actions and video background information, but also depends on the relationship information between individual actions. In addition to using convolutional neural networks, some methods also use models such as graph convolutional networks, recurrent neural networks, and Transformers to capture and analyze the relationship information between individual actions.

Therefore, the present invention proposes a behavior recognition technical solution.

Therefore, according to the first object of the present invention, a behavior recognition method is provided, comprising: receiving an input movie frame, and extracting character features in the movie frame; Clustering to obtain a clustering result; determining the attention distribution weight of the character feature in the film frame based on the clustering result; updating the character feature based on the attention distribution weight; based on the updated character feature, Extracting the spatio-temporal features of the characters; performing behavior recognition on the movie frames based on the spatio-temporal features of the characters to obtain a recognition result.

In a possible implementation manner, the determining the attention distribution weights of the character features in the film frame based on the clustering result includes: determining the Attention distribution weights among the character features.

In a possible implementation manner, the determining the attention distribution weights between the character features based on the association relationship between the character features in the clustering result includes: determining the same group obtained by clustering The first similarity between the character features of the group; based on the first similarity, determine the first attention distribution weight among the character features in the group.

In a possible implementation manner, determining the first similarity between the character features in the same group obtained by clustering includes: dividing the feature matrix of the character features into N parts; The features are respectively corresponding to the calculation of similarities, and N first similarities are obtained; said determining the first attention distribution weight between the character features in the group based on said first similarities includes: based on said N first similarities degree, determine the N first attention distribution weights among the characters in the group.

In a possible implementation manner, the determining the attention distribution weights between the character features based on the association relationship between the character features in the clustering result includes: determining the weights of each group obtained by clustering Overall features; determining a second similarity between the overall features of each group obtained by clustering; based on the second similarity, determining a second attention distribution weight between the character features.

In a possible implementation manner, the updating of the character features based on the attention distribution weight includes: for a target character feature in the character features of a single group, using the target character feature and the first character feature of each character feature in the group - Attention distribution weights, performing weighted summation on the features of each character in the group, and obtaining the update features in the group corresponding to each character in the group, as the updated character features.

In a possible implementation manner, the updating of the character features based on the attention distribution weights includes: for the target general features of the target groups in each group, using the second Attention is assigned to weights to obtain the inter-group update features corresponding to each group; the inter-group update features are respectively added to the intra-group update features corresponding to the characters in the target group to obtain updated character features.

In a possible implementation manner, the extracting the spatio-temporal features of the character based on the updated character features includes: performing spatial decoding on the updated character features to obtain the spatio-temporal features of the character.

In a possible implementation manner, performing spatial decoding on the updated character features to obtain the character spatio-temporal features includes: performing spatial decoding on the updated character features to obtain the character spatial features; Time-domain encoding and decoding are performed to obtain the time-domain characteristics of the person; the space-time characteristics of the person are fused with the time-domain characteristics of the person to obtain the time-space characteristics of the person.

In a possible implementation manner, the time-domain encoding and decoding of the character features of multiple film frames to obtain the character time-domain features includes: based on the self-attention mechanism, the character features of multiple film frames Encoding is performed to obtain time-domain coding features; the time-domain coding features are decoded based on a self-attention mechanism, and/or the time-domain coding features are decoded based on spatial coding features to obtain character time-domain features; wherein, The spatially encoded features are the updated character features.

In a possible implementation manner, the spatial decoding of the updated character features to obtain the character spatial features includes: decoding the spatial encoding features based on a self-attention mechanism, and/or decoding the spatial encoding features based on the time domain encoding The feature decodes the spatially encoded features to obtain the character's spatial features.

In a possible implementation, the method further includes: extracting global features of the movie frame; using the character spatio-temporal features to determine a third attention distribution weight in the global features; using the third attention The force distribution weight updates the global features; the performing behavior recognition on the film frame based on the character spatio-temporal features, and obtaining the recognition result includes: performing crowd behavior recognition on the film frames based on the updated global features, Obtain the result of crowd behavior recognition.

In a possible implementation manner, after using the third attention distribution weight to update the global feature, the method further includes: using the updated global feature as a new global feature, and using the character spatiotemporal feature as For new character features, iteratively update the global features and character spatio-temporal features until the iterative stop condition is met, and obtain the iteratively updated global features and character spatio-temporal features; based on the character spatio-temporal features, the Performing behavior recognition on the film frame to obtain a recognition result includes: performing crowd behavior recognition on the film frame based on the iteratively updated global features to obtain a crowd behavior recognition result.

In a possible implementation manner, after obtaining the iteratively updated global features and character spatio-temporal features, the method further includes: performing character behavior recognition based on the iteratively updated character spatio-temporal features to obtain a character behavior recognition result .

In a possible implementation manner, the extracting the character features in the film frame includes: performing human body recognition in the film frame to obtain the target rectangular frame of each character; extracting the feature in the film frame, and using the The target rectangular frame is matched with the features of the extracted movie frame to obtain the corresponding character features.

According to the second object of the present invention, a behavior recognition device is provided, including: a character feature extraction unit for receiving input film frames and extracting character features in the film frames; a clustering unit for A plurality of the character features in the film frame are clustered to obtain a clustering result; an attention distribution unit is used to determine the attention distribution weight of the character feature in the film frame based on the clustering result; A character feature update unit, used to update the character feature based on the attention distribution weight; a character spatio-temporal feature extraction unit, used to extract the character spatio-temporal feature based on the updated character feature; and a behavior recognition unit, with Based on the spatio-temporal characteristics of the person, behavior recognition is performed on the video frame to obtain a recognition result.

In a possible implementation manner, the attention allocation unit is configured to determine an attention allocation weight between the character features based on the association relationship between the character features in the clustering result.

In a possible implementation manner, the attention allocating unit includes: a first similarity determining unit, configured to determine a first similarity between character features in the same group obtained by clustering; and a first An attention determination unit, configured to determine a first attention distribution weight among characters in the group based on the first similarity.

In a possible implementation manner, the first similarity determination unit is configured to divide the feature matrix of the character features into N parts; and calculate the similarity for the N features of different character features to obtain N A first similarity; and the first attention determination unit, configured to determine N first attention distribution weights among the characters in the group based on the N first similarities.

In a possible implementation manner, the attention allocation unit includes: an overall feature determination unit, configured to determine the overall features of each group obtained by clustering; a second similarity determination unit, configured to determine the clustering obtained A second similarity between the overall features of each group; and a second attention determination unit, configured to determine a second attention distribution weight between the character features based on the second similarity.

In a possible implementation manner, the character feature updating unit is configured to, for a target character feature in the character features of a single group, use the target character feature and the first attention distribution weight of each character feature in the group to assign weights to the group Each character feature in the group is weighted and summed to obtain the update feature in the group corresponding to each character in the group, which is used as the updated character feature.

In a possible implementation manner, the character feature update unit includes: an inter-group update feature determination unit, configured to use the difference between the target overall feature and the overall feature of each group for the target overall feature of the target grouping in each group. The second attention distribution weight obtains the update features between groups corresponding to each group; and a character feature update subunit, which is used to add the update features between the groups to the update features in the group corresponding to each character in the target grouping , to get the updated character traits.

In a possible implementation manner, the character spatio-temporal feature extraction unit is configured to perform spatial decoding on the updated character features to obtain the character spatio-temporal features.

In a possible implementation manner, the character spatio-temporal feature extraction unit includes: a spatial decoding unit, configured to spatially decode the updated character features to obtain character spatial features; a time-domain encoding and decoding unit, configured to convert multiple Time-domain encoding and decoding are performed on the character features of each film frame to obtain character time-domain features; and a fusion unit is used to fuse the character space features with the character time-domain features to obtain character space-time features.

In a possible implementation manner, the time-domain encoding and decoding unit includes: a time-domain encoding unit, configured to encode the character features of multiple film frames based on a self-attention mechanism to obtain time-domain encoding features; and A time-domain decoding unit, configured to decode the time-domain coding features based on a self-attention mechanism, and/or decode the time-domain coding features based on spatial coding features to obtain character time-domain features; wherein, the spatial The encoded feature is the updated character feature.

In a possible implementation manner, the spatial decoding unit is configured to decode the spatially encoded features based on a self-attention mechanism, and/or decode the spatially encoded features based on the time domain encoded features, to obtain character space features.

In a possible implementation manner, the device further includes: a global feature extraction unit, configured to extract global features of the film frame; a third attention determination unit, configured to determine the The third attention distribution weight in the global feature; and a global feature update unit, used to update the global feature with the third attention distribution weight; the behavior recognition unit includes: a crowd behavior recognition unit, The method is used for performing crowd behavior recognition on the film frame based on the updated global feature to obtain a crowd behavior recognition result.

In a possible implementation manner, the device further includes: an iterative updating unit, configured to use the updated global features as new global features, and use the character spatio-temporal features as new character features, iteratively The global feature and the character spatio-temporal feature are updated until the iterative stop condition is satisfied, and the iteratively updated global feature and the character spatio-temporal feature are obtained; The frame is used for crowd behavior recognition, and the result of crowd behavior recognition is obtained.

In a possible implementation manner, the device further includes: a character behavior recognition unit, configured to perform character behavior recognition based on the iteratively updated spatio-temporal characteristics of the character, and obtain a character behavior recognition result.

In a possible implementation manner, the character feature extraction unit includes: a target rectangular frame determination unit, configured to perform human body recognition in movie frames to obtain the target rectangular frame of each character; and a character feature extraction subunit, It is used to extract the features in the film frame, and use the target rectangular frame in the film frame to match the features of the extracted film frame to obtain the corresponding character features.

According to the third object of the present invention, an electronic device is provided, including: a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to call the instructions stored in the memory, to perform the above method.

According to the fourth object of the present invention, there is provided a computer-readable storage medium on which computer program instructions are stored, and the above-mentioned method is implemented when the computer program instructions are executed by a processor.

According to the fifth object of the present invention, a computer program product is provided, including computer-readable codes, or a non-volatile computer-readable storage medium carrying computer-readable codes, when the computer-readable codes are processed in an electronic device When running in the processor, the processor in the electronic device executes the above method.

The effect of the present invention is that: in the embodiment of the present invention, after receiving the input film frame, the character features in the film frame are extracted, and the clustering result is obtained by clustering a plurality of character features in the film frame, and then based on the clustering Determine the attention distribution weight of the character features in the movie frame based on the class results, update the character features based on the attention distribution weight, and then extract the character spatio-temporal features based on the updated character features, and conduct behavior recognition on the movie frame based on the character spatio-temporal features, and get recognition result. Therefore, the relationship between each individual feature (action) is obtained based on clustering, and the attention distribution weight of character features is obtained based on the clustering results to highlight the importance of different character features, so that the most important character features If the action information can be highlighted, then, the behavior recognition based on the updated character feature extraction can reduce the calculation redundancy and information interference caused by the relationship between unimportant individual actions, and improve the accuracy of behavior recognition. Accuracy.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention. Other features and aspects of the present invention will become apparent from the following detailed description of exemplary embodiments with reference to the accompanying drawings.

Before the present invention is described in detail, it should be noted that in the following description, similar elements are denoted by the same numerals.

Various exemplary embodiments, features, and aspects of the invention will be described in detail below with reference to the drawings. Identical drawing symbols in the drawings indicate functionally identical or similar elements. While various aspects of the embodiments are shown in the drawings, the drawings are not necessarily drawn to scale unless specifically indicated.

The word "exemplary" is used exclusively herein to mean "serving as an example, embodiment, or illustration." Any embodiment described herein as "exemplary" is not necessarily to be construed as superior or better than other embodiments.

The term "and/or" in this article is just an association relationship describing associated objects, which means that there can be three relationships, for example, A and/or B can mean: A exists alone, A and B exist at the same time, and B exists alone. three conditions. In addition, the term "at least one" herein means any one or any combination of at least two of the plurality, for example, including at least one of A, B, and C, may mean including the composition consisting of A, B, and C Any one or more elements selected in the collection.

In addition, in order to better illustrate the present invention, numerous specific details are given in the specific embodiments below. It will be understood by those skilled in the art that the present invention may be practiced without certain of the specific details. In some instances, methods, means, components and circuits well known to those skilled in the art have not been described in detail in order to highlight the gist of the present invention.

In related technologies, crowd behavior recognition methods based on deep learning often try to establish a larger-scale spatio-temporal relationship model and more diverse input features (video optical flow, human body key point information) to improve the accuracy of crowd behavior recognition .

An embodiment of the present invention provides a behavior recognition method, which further takes into account that not everyone's actions are important to the crowd behavior recognition in the crowd behavior recognition. For example, in a volleyball game, only the actions of players approaching or touching the volleyball are decisive for the identification of crowd behavior categories. Therefore, the present invention obtains the clustering result by clustering multiple character features in the film frame, then determines the attention distribution weight of the character feature in the film frame based on the clustering result, and updates the character based on the attention distribution weight Features; based on the updated character features, extract the character spatio-temporal features; based on the character spatio-temporal features, conduct behavior recognition on the film frame to obtain the recognition result. Thus, the relationship between each individual feature (action) is obtained based on clustering, and the attention distribution weight of character features is obtained based on the clustering results to highlight the importance of different character features, so that important action information can be It is highlighted, which reduces the calculation redundancy and information interference caused by analyzing the relationship between unimportant individual actions, and improves the accuracy of behavior recognition.

In a possible implementation, the behavior recognition method may be executed by electronic equipment such as a terminal device or a server, and the terminal device may be user equipment (User Equipment, UE), mobile device, user terminal, terminal, cellular phone, wireless Phones, personal digital assistants (Personal Digital Assistant, PDA), handheld devices, computing devices, vehicle-mounted devices, wearable devices, etc., the method can be implemented by calling computer-readable instructions stored in the memory by the processor.

For ease of description, in one or more embodiments of this specification, the execution subject of the behavior recognition method may be a server, and the implementation of the method will be introduced later by taking the execution subject as a server as an example. It can be understood that the implementation subject of the method is the server is only an exemplary description, and should not be understood as a limitation of the method.

Fig. 1 shows the flowchart of the behavior recognition method according to the embodiment of the present invention, as shown in Fig. 1, described behavior recognition method comprises: in step S11, receives the film frame of input, and extracts the characters in the film frame feature.

The movie frame here may be any movie frame in the movie frame sequence, or may be multiple movie frames in the movie frame sequence. The film frames may be input in the form of film frame sequences, and the length of a single film frame sequence may be predetermined, for example, 20 frames.

The video frame can be a video frame stored in the local storage space, then the video frame can be read from the terminal local storage space to realize the input of the video frame, for example, it can be a video frame of a locally stored sports event video The video frame, for another example, may be a video frame in a locally stored store management video.

Alternatively, the movie frame may also be a movie frame captured by the acquired image acquisition device in real time, for example, it may be a movie frame in a live video of a sports event, and for another example, it may be captured by an image acquisition device located at the entrance of a shopping mall in real time movie frame.

In a possible implementation manner, extracting the character features in the film frame includes: performing human body recognition in the film frame to obtain the target rectangular frame of each character; extracting the feature in the film frame, and using all the features in the film frame The target rectangular frame is matched to obtain the corresponding character features from the features of the extracted movie frame.

Specifically, for the characters in the movie frame, the area where the characters in the movie frame are located can be identified through human body recognition technology. This area is often represented by a rectangular frame, and the area selected by the rectangular frame is where the recognized character is located. Area. Since multiple rectangular frames may be obtained when identifying the same person, then the non-maximum suppression algorithm (Non-Maximum Suppression, NMS) can be used to deduplicate multiple rectangular frames, and for a single movie frame One of the characters retains a rectangular frame as the area where the recognized character is located.

For the multi-frame movie frames in the movie frame sequence, the regions where the characters in the multi-frame movie frames are located can be obtained through the above method.

In the embodiment of the present invention, the global feature extraction can be performed on the movie frame to obtain the global feature of the whole movie frame. The output of the last layer of the road is used as the global feature. Then, feature extraction is performed on the intermediate features output by the intermediate layer of the I3D network to obtain the characteristics of each character. Specifically, the position of the character in the film frame is the position of the multiple rectangular frames obtained after deduplication by the NMS above. The position of the rectangular frame corresponds to the intermediate feature extracted by the I3D network, and the RoIAlign technology is used to extract the features corresponding to the position of the rectangular frame in the intermediate feature to obtain the character features in the movie frame. In addition, other ways may also be used to acquire the character features, and the present invention does not limit the specific way of acquiring the character features.

In step S12, clustering is performed on a plurality of the character features in the film frame to obtain a clustering result.

In the process of clustering, multiple character features are divided into different groups according to a certain standard (such as similarity), so that the similarity of the character features in the same group is as large as possible, and at the same time they are not in the same group. The differences between the characteristics of the characters in the book are also as large as possible. That is to say, after clustering, the character features of the same group are gathered together as much as possible, and the character features of different groups are separated as much as possible.

In the embodiment of the present invention, the character features can be clustered based on the K-means algorithm. Specifically, the character features to be classified can be formed into a feature set and the number K of categories to be divided can be specified (for example, it can be 3) , randomly select K characters from the feature set as the initial cluster centers of K categories, and for each character feature in the feature set except the K initial cluster centers, calculate the character features and the K initial cluster centers respectively. The distance between the features of each initial cluster center (for example, it can be the Euclidean distance, which is used to characterize the similarity between the features), and the character features are assigned to the initial cluster with the closest distance to the character features. In the category corresponding to the cluster center, then according to the characteristics of the characters included in the K categories, recalculate the new cluster centers of the K categories, and then reclassify the characteristics of the characters in the feature set until each of the K categories The distance between the cluster centers of two adjacent clusters of a category is within a preset distance.

After clustering the character features, the obtained clustering result is that multiple character features are divided into multiple groups, and finally, the cluster center of each group is also determined after multiple updates. For example, group A contains character features {a, b, c}; group B contains character features {d, e, f, g}; group C contains character features {h, i}; and the final determined group A The characteristic of the cluster center of group B is α, the characteristic of the cluster center of group B is β, and the characteristic of the cluster center of group C is γ.

In step S13, based on the clustering result, the attention distribution weights of the character features in the movie frame are determined.

The clustering result can represent the potential connection between multiple character features, help to capture more key potential features, cluster key features in a group, and enhance its representation ability, which can improve the behavior recognition of film frames Therefore, attention can be assigned to character features in movie frames based on the clustering results to enhance the expressiveness of key features.

There are many ways to specifically determine the weight of attention distribution. For example, it can be determined according to the similarity between the characteristics of the characters in the group. Possible implementation methods are not described here.

In step S14, the character features are updated based on the attention distribution weights.

After the attention distribution weights are determined, the person features can be updated to add attention information in the person features. Specifically, the attention distribution weight can be used to weight the character features to update the character features. For example, in computer language, the feature matrix is used to represent the character features, so the attention weight can be multiplied by the feature matrix to realize the The character characteristics endow the weight of attention distribution to realize the updating of the character characteristics.

In the embodiment of the present invention, the first attention distribution weight can be obtained based on the first similarity of character characteristics in the group obtained by clustering, and the second attention distribution weight can be obtained based on the similarity of overall characteristics between groups. For the process of updating the character features by using the first attention distribution weight and the second attention weight, please refer to the possible implementation methods provided by the present invention, which will not be repeated here.

In step S15, based on the updated character features, the spatio-temporal features of the character are extracted.

The spatio-temporal characteristics of a character can represent the characteristics of a character in the time dimension and space dimension. For the character features in the same video frame, the spatial distribution information of the characters can be represented, and based on the updated character features, the character spatial features can be obtained. The spatial characteristics of the person can be fused with the temporal characteristics of the person to obtain the temporal and spatial characteristics of the person. For the specific fusion process, please refer to the possible implementation methods provided by the present invention, which will not be repeated here.

In step S16, based on the spatiotemporal characteristics of the person, perform behavior recognition on the film frame to obtain a recognition result.

The character features updated based on the attention distribution weights enhance the expression ability of key features, then, the character spatio-temporal features extracted based on character features also enhance the expression ability of key features. Therefore, based on the extracted spatiotemporal features of characters, the behavior recognition of movie frames can improve the accuracy of recognition results.

Behavior recognition in the embodiment of the present invention may include individual character behavior recognition and/or group crowd behavior recognition. The spatio-temporal features are used to identify the behavior of individual characters; another example is to use the spatio-temporal features of the characters to perform attention weighting on the global features of the film frame, and use the global features after the attention weighting to identify crowd behaviors. For details, reference may be made to the possible implementation manners provided by the present invention, and details are not described here.

In the embodiment of the present invention, after receiving the input film frame, the character features in the film frame are extracted, the clustering result is obtained by clustering a plurality of character features in the film frame, and then based on the clustering result, the The attention distribution weight of the character features, the character features are updated based on the attention distribution weights, and then based on the updated character features, the spatiotemporal features of the characters are extracted, and based on the spatiotemporal features of the characters, the behavior recognition is performed on the movie frame, and the recognition result is obtained. Therefore, the relationship between each individual feature (action) is obtained based on clustering, and the attention distribution weight of character features is obtained based on the clustering results to highlight the importance of different character features, so that the most important character features If the action information can be highlighted, then, the behavior recognition based on the updated character feature extraction can reduce the calculation redundancy and information interference caused by the relationship between unimportant individual actions, and improve the accuracy of behavior recognition. Accuracy.

In a possible implementation manner, the determining the attention distribution weights of the character features in the film frame based on the clustering result includes: determining the Attention distribution weights among the character features.

The association relationship here is used to represent the correlation between character features, for example, it can be the similarity between character features, then, the attention distribution weight between character features can be determined based on the similarity between character features .

In the embodiment of the present invention, two implementations are provided to determine the weight of attention allocation based on the association relationship between the character features in the clustering results, respectively determining the first attention allocation based on the similarity of the character features in the group weight, and determining the second attention distribution weight based on the similarity between the overall features of the groups, the two implementations will be described in detail below.

In a possible implementation manner, the determining the attention distribution weights between the character features based on the association relationship between the character features in the clustering result includes: determining the same group obtained by clustering the first similarity between the personal characteristics; and based on the first similarity, determine the first attention distribution weight among the personal characteristics in the group.

The first similarity between character features in the same group can be the similarity between multiple character features in the same group. There are many ways to calculate the feature similarity, for example, it can be based on Euclidean The similarity calculation method based on the distance can also be used as another example, the similarity calculation method based on the cosine similarity, and so on. The present invention does not specifically limit the calculation method of the similarity.

After determining the similarity between character features, the similarity can be normalized (normalize), specifically through a normalization function (for example, softmax function), after normalization, the character feature can be obtained The first attention assigns weights.

The first attention distribution weight can be applied to the character features, and the character features can be updated to add attention information to the character features. After using the first attention distribution weight to update multiple character features, they can be updated The updated character features, based on the updated character features of the attention distribution weight, enhance the expression ability of key features. Therefore, based on the updated character features, the behavior recognition of the movie frame can improve the accuracy of the recognition results.

In the embodiment of the present invention, by determining the first similarity between the character features in the same group obtained by clustering; and based on the first similarity, determining the first attention distribution weight between the character features in the group . Thus, based on the similarity between the same group of character features, the correlation between the character features in the group can be determined, and the first attention distribution weight determined according to the similarity can enhance the expression of key features in the character features Therefore, based on the updated character characteristics, the behavior recognition of the movie frame can improve the accuracy of the recognition result.

In a possible implementation manner, determining the first similarity between the character features in the same group obtained by clustering includes: dividing the feature matrix of the character features into N parts; Computing the similarities corresponding to the features respectively, and obtaining N first similarities; said determining the first attention distribution weight between the character features in the group based on said first similarities, including: based on said N first The similarity determines the N first attention distribution weights among the characteristics of the characters in the group.

In computer technology, the specific expression form of character features is a feature matrix, for example, the size of the feature matrix of character features is T×1024. Then, the feature matrix can be divided into N parts, and N is an integer greater than 1. For example, when N is 8, the matrix of T×1024 is divided into 8 parts, and the divided matrix can be expressed as 8×T×128; For another example, when N is 4, the T×1024 matrix is divided into 4 parts, and the divided matrix can be expressed as 4×T×256. Among them, T is a dimension of the feature matrix.

Then, when calculating the similarity for different character features, the corresponding similarity can be calculated for N features, so that N first similarities will be obtained. For example, for a feature matrix with a size of T×1024, in N When 8 is selected, the similarity between 8 sub-feature matrices with a size of T×128 and the sub-feature matrix of T×128 in another feature matrix will be calculated. In this way, 8 similarities will be obtained, and a size of 8 can be used The matrix is represented; while calculating the two feature matrices of T×1024, only one similarity can be obtained. Therefore, compared with 1 similarity, 8 similarities can enhance the diversity of the relationship between character features, and can more accurately describe the relationship between character features.

After obtaining the N first similarities, N first attention distribution weights among the characters in the group can be determined based on the N first similarities. For the specific method of determining the attention distribution weights, please refer to the relevant description above. , which will not be described here.

In the embodiment of the present invention, by dividing the feature matrix of the character features into N parts, and then correspondingly calculating similarities for the N features of different character features, N first similarities are obtained, and based on the N The first similarities determine the N first attention distribution weights between the character features in the group. In this way, the diversity of the relationship between the character features can be enhanced to more accurately describe the relationship between the character features, and thus the accuracy of behavior recognition based on the character features can be improved.

In a possible implementation manner, the determining the attention distribution weights between the character features based on the association relationship between the character features in the clustering result includes: determining the weights of each group obtained by clustering Overall features; determining a second similarity between the overall features of each group obtained by clustering; and determining a second attention distribution weight between the character features based on the second similarity.

The overall characteristics of a certain group of clustering results can represent the characteristics of the group as a whole. The overall characteristics are calculated based on the characteristics of the persons in the group. For example, it can be the average pooling of the characteristics of the persons in the group. Alternatively, it may also be obtained by performing a random pooling operation on the characteristics of the characters in the group. The embodiment of the present invention does not specifically limit the method of determining the overall characteristics.

In addition, the overall characteristics of each group can also be the characteristics of the cluster centers of each group in the clustering result, then, in a possible implementation, the determination of the overall characteristics of each group obtained by clustering includes: The characteristics of the cluster centers of each group in the above clustering results are used as the overall characteristics of each group. Because in the process of clustering, other human characteristics determine whether they belong to the class by calculating the distance (similarity) with the cluster center, therefore, the similarity between the characteristics of the cluster center and all human characteristics in the group is relatively high , then the features of the cluster center can be used to accurately characterize the overall characteristics of the group.

The second similarity of the overall features between different groups can be the similarity between the overall features of each group. There are many ways to specifically calculate the second similarity of the overall features, for example, it can be based on Euclidean The similarity calculation method based on the distance can also be used as another example, the similarity calculation method based on the cosine similarity, and so on. The present invention does not specifically limit the calculation method of the second similarity degree of the overall feature.

After determining the second similarity between the overall features of each group, the second similarity can be normalized (normalize), specifically through a normalization function (for example, softmax function), after normalization, The second attention distribution weight of the character feature can be obtained.

The second attention distribution weight can be applied to the character features, and the character features can be updated to add attention information to the character features. After using the second attention distribution weight to update multiple character features, they can be updated The updated character features, based on the updated character features of the attention distribution weight, enhance the expression ability of key features. Therefore, based on the updated character features, the behavior recognition of the movie frame can improve the accuracy of the recognition results.

In the embodiment of the present invention, the overall characteristics of each group are obtained according to the characteristics of the characters in each group; the second similarity between the overall characteristics of each group obtained by clustering is determined; and based on the second similarity, the characters are determined Second attention distributes weights between features. Thus, based on the similarity between the overall characteristics of different groups, the association relationship between the character characteristics between the groups can be determined, and the second attention distribution weight determined according to the second similarity of the overall characteristics can enhance the character characteristics. Therefore, based on the updated character features, the behavior recognition of the movie frame can improve the accuracy of the recognition results.

It can be understood that "first" and "second" in the embodiments of the present invention are used to distinguish the described objects, and should not be interpreted as other restrictions on the order of the described objects, indicating or implying relative importance, etc.

After the attention distribution weights are determined, the character features can be updated to add attention information to the character features. As mentioned above, the present invention can obtain the first attention distribution weight and the second attention distribution weight. The following describes the process of updating character features by using the first attention distribution weight and the second attention weight respectively.

In a possible implementation manner, the updating of the character features based on the attention distribution weight includes: for a target character feature in the character features of a single group, using the target character feature and the first character feature of each character feature in the group - Attention distribution weights, performing weighted summation on the features of each character in the group, and obtaining the update features in the group corresponding to each character in the group, as the updated character features.

For a target character feature in a certain group of character features, use the target character feature and the first attention distribution weight of each character feature in the group, and each character feature in the group (including the target character feature itself) Carry out weighted summation to obtain the update features in the group corresponding to each person in the group, as the updated character features.

For example, the first degree of similarity between target character V ₁ and n character traits (V ₁ , V ₂ , V ₃ . . . V _n ) is P ₁ , P ₂ , P _{3 .} . . The weights _of the first attention _distribution obtained after similarity normalization are W ₁ , W ₂ , W _{3 .} '= V ₁ •W ₁ +V ₂ •W ₂ +V ₃ •W ₃ +...+V _n •W _n (1)

Therefore, the updated target character feature V ₁ ' comprehensively considers the degree of similarity with all character features in the group, and can enhance the expression ability of key features when the similarity with all character features in the group is high. Therefore, based on The updated character features can be used for behavior recognition on video frames, which can improve the accuracy of recognition results.

In a possible implementation manner, the updating of the character features based on the attention distribution weights includes: for the target general features of the target groups in each group, using the second Attention is assigned to weights to obtain the inter-group update features corresponding to each group; the inter-group update features are respectively added to the intra-group update features corresponding to the characters in the target group to obtain updated character features.

The inter-group update feature of the target group can be obtained by weighting and summing the overall features of each group by using the second attention distribution weight, and then adding the weighted sum result to the characteristics of each person in the target group.

For example, the target overall feature of the target group is C ₁ , and the second similarity with the m overall features (C ₁ , C ₂ , C ₃ ... C _m ) of each group is Q ₁ , Q ₂ , Q ₃ ... Q _m , the second attention distribution weight obtained after normalizing the second similarity is U ₁ , U ₂ , U ₃ ... U _m , then the component update feature C ₁ ' can be obtained by formula (2) Determine: C ₁ '= C ₁ •U ₁ +C ₂ •U ₂ +C ₃ •U ₃ +...+C _m •U _m (2)

Further, the inter-group update features are respectively added to the intra-group update features corresponding to the characters in the target group, that is, the values of n character features in the target group are V ₁ '+C ₁ ', V ₂ '+C ₁ ', V ₃ '+C ₁ '... V _n '+C ₁ '.

After updating multiple character features with the second attention distribution weight, the updated character features can be obtained. The updated character features based on the second attention distribution weight enhance the expression ability of key features. Therefore, based on the updated Based on the characteristics of the final character, the behavior recognition of the film frame can improve the accuracy of the recognition result.

In a possible implementation manner, the extracting the spatio-temporal features of the character based on the updated character features includes: performing spatial decoding on the updated character features to obtain the spatio-temporal features of the character.

The spatio-temporal features of the characters include the spatial features of the characters, and the character features in the same movie frame can represent the spatial distribution in the movie frame. Therefore, spatial decoding of the character features updated based on the attention distribution weight can obtain the character spatial features, and the character Spatial features can represent the spatial distribution information of character features in video frames. For a specific manner of spatially decoding the updated character features, reference may be made to the possible implementation manners provided by the present invention, which will not be repeated here.

In addition, the character spatio-temporal features also include character space features. In a possible implementation, the updated character features are spatially decoded to obtain the character spatio-temporal features, including: spatially decoding the updated character features to obtain the character space Features: performing time-domain encoding and decoding on the character features of multiple film frames to obtain character time-domain features; and merging the character space features with the character time-domain features to obtain character space-time features.

In the embodiment of the present invention, it is also possible to determine the time-domain characteristics of the person. In a film frame sequence, the character characteristics of the same person will change in different film frames. The time-domain characteristics of the person here can represent the character characteristics of the same person. Information that changes over time in different video frames. Therefore, the time-domain encoding and decoding can be performed on the character features of the same character in different movie frames to obtain the character time-domain features.

The time-domain features of the person can be fused with the space-time characteristics of the person to obtain the time-space characteristics of the person. The specific fusion method can be, for example, adding processing of the time-domain characteristics of the person and the space-time characteristics of the person. Then, based on the spatio-temporal characteristics of the characters, behavior recognition is performed on the film frames. Therefore, not only the spatial characteristics of the characters, but also the temporal characteristics of the characters are considered during the behavior recognition, which can improve the accuracy of the recognition results.

In a possible implementation manner, the time-domain encoding and decoding of the character features of multiple film frames to obtain the character time-domain features includes: based on the self-attention mechanism, the character features of multiple film frames Encoding to obtain time-domain coding features; and decoding the time-domain coding features based on a self-attention mechanism, and/or decoding the time-domain coding features based on spatial coding features to obtain character time-domain features; wherein, The spatially encoded feature is the updated character feature.

After the character features of multiple movie frames are extracted, the character features of multiple movie frames can be input into the time-domain encoder, and encoded based on the self-attention mechanism to obtain time-domain coding features. In this encoding process, the self-attention mechanism can be used to calculate the alignment probability of the character features in each film frame and the human body features at other moments, and then the probability is weighted and summed with the human body features at the corresponding time, as the time domain Encoding features.

In the process of decoding the time-domain coding features, the time-domain coding features can be decoded based on the self-attention mechanism to obtain the first time-domain features, and the time-domain coding features can also be decoded based on the spatial coding features , to obtain the second time-domain feature, and then fuse the first time-domain feature with the second time-domain feature to obtain the person's time-domain feature.

In the process of decoding the temporal coding feature based on the spatial coding feature, the similarity between the spatial coding feature and the temporal coding feature can be determined, and the temporal coding feature can be weighted by using the similarity to obtain the second temporal feature. Decoding the time-domain coding features based on the spatial coding features makes the obtained second time-domain features incorporate spatial context information, enhances the feature representation in the second time-domain features, and can improve the accuracy of the final behavior recognition result.

In addition, there are many ways to decode the spatially encoded features. In a possible implementation, the spatial decoding of the updated character features to obtain the character spatial features includes: based on the self-attention mechanism. The spatial coding features are decoded, and/or the spatial coding features are decoded based on the time domain coding features to obtain the character spatial features.

In the process of decoding the spatial encoding, the spatial encoding feature can be decoded based on the self-attention mechanism to obtain the first spatial feature, and the spatial encoding feature can also be decoded based on the time-domain encoding feature to obtain the second spatial feature , and then fuse the first spatial feature with the second spatial feature to obtain the character spatial feature.

In the process of decoding the spatial encoding features based on the self-attention mechanism, the degree of correlation (for example, similarity) between the spatial encoding features of different characters can be determined, and then the degree of association can be used to weight each spatial encoding feature, Get the first space features.

In the process of decoding the spatial encoding feature based on the temporal encoding feature, the similarity between the temporal encoding feature and the spatial encoding feature can be determined, and then the similarity is used to weight the spatial encoding feature to obtain the second spatial feature. Decoding the spatially encoded features based on the time-domain encoded features makes the obtained second spatial features incorporate temporal context information, enhances the feature representation in the second spatial features, and can improve the accuracy of the final behavior recognition result.

In the embodiment of the present invention, the time-domain coding feature is decoded based on the spatial coding feature to obtain the character's time-domain feature, and based on the time-domain coding feature, the spatial coding feature is decoded to obtain the character's spatial feature, and the character's space The feature is fused with the character's time-domain feature. In the obtained character's time-space feature, the semantic association based on the spatial context and time context is used to enhance the character's feature representation. Then, the behavior recognition based on the character's time-space feature can improve the accuracy of the behavior recognition result. .

In a possible implementation, the method further includes: extracting global features of the movie frame; using the character spatio-temporal features to determine a third attention distribution weight in the global features; using the third attention The force distribution weight updates the global feature; and the behavior recognition of the video frame based on the character spatio-temporal feature to obtain the recognition result includes: performing crowd behavior recognition on the video frame based on the updated global feature , to get the result of crowd behavior recognition.

The global feature of the movie frame can be obtained by extracting the feature of the whole picture of the movie frame, and it can also be called the scene feature of the movie frame. The specific extraction method can be extracted through the I3D network, and the output of the last layer of the I3D network , input to the group representation generator (Group Representation Generator, GRG), GRG is a preprocessing element, used to initialize the features of the I3D output, and obtain the global features after initialization, the specific extraction method will not be described here.

For the extracted global features, attention can be allocated to the global features. Specifically, attention can be allocated to the global features by using the spatio-temporal features of the characters. Therefore, the third attention allocation in the global features can be determined by using the spatio-temporal features of the characters. weights, and then utilize the third attention distribution weights to update the global features.

Specifically, the transformer model can be used to use the spatiotemporal characteristics of the characters to allocate attention to the global features. Specifically, the global feature can be used as the query parameter in the transformer model, and the character spatiotemporal feature can be used as the key parameter in the transformer model to obtain the third attention allocation weight, and realize the attention allocation in the global feature. In this way, it can be based on The characteristics of each person in the movie frame to optimize the global characteristics.

Therefore, the optimized global feature is the feature after the attention is allocated by the character's spatiotemporal features, which can highlight the key features in the global feature and weaken the irrelevant features in the global feature. Then, based on the updated global features, behavior recognition is performed on movie frames, which can improve the accuracy of recognition results.

In the process of crowd behavior recognition for the movie frame, the updated global features can be input into the fully connected layer of the neural network, and the fully connected layer can be used for classification, and multiple groups of people will be preset in the fully connected layer Behavior category, according to the global characteristics, the fully connected layer can output the global characteristics as the reliability of each crowd behavior category, and the crowd behavior category with the highest reliability can be used as the crowd behavior recognition result.

For example, in the video frame of a volleyball game, the extracted global features are input to the fully connected layer, and the reliability of the character action category "serve left" is 0.9, and the reliability of "pass left" is 0.3, " The reliability of "Smash left" is 0.4, the reliability of "Serve right" is 0.1, and the reliability of "Block right" is 0.1. Then, the most reliable "Serve left" can be output as the recognition result.

In a possible implementation manner, after using the third attention distribution weight to update the global feature, the method further includes: using the updated global feature as a new global feature, and using the character spatiotemporal feature as For new character features, iteratively update the global features and character spatio-temporal features until the iterative stop condition is met, and obtain the iteratively updated global features and character spatio-temporal features; based on the character spatio-temporal features, the Performing behavior recognition on the film frame to obtain a recognition result includes: performing crowd behavior recognition on the film frame based on the iteratively updated global features to obtain a crowd behavior recognition result.

Using the character spatio-temporal features as new character features, iteratively update the new character features, specifically, iteratively perform steps S12-S15 to obtain the iteratively updated character spatio-temporal features, of course, the The iterative process can also be one or more possible implementations of steps S12-S15 in the embodiment of the present invention, for example, an implementation of updating the character features based on the first attention distribution weight among the character features in the group ; Based on the second attention distribution weight between the features between the groups, the implementation of updating the character features; the spatial decoding of the updated character features, to obtain the realization of the character space-time features, etc., not listed here one by one, For details, refer to possible implementation manners of steps S12-S15 provided in the present invention.

Similarly, the updated global feature can be used as a new global feature, and the global feature can be iteratively updated. When the iteration stop condition is met, the iteration can be stopped, and the iteratively updated global feature and character spatio-temporal features can be obtained. , the condition for stopping iteration may be, for example, a preset number of iterations.

After the iterative update, crowd behavior recognition can be performed on the film frame based on the iteratively updated global feature, and the crowd behavior recognition result can be obtained. Since key features are highlighted in the iteratively updated global feature, the global feature is weakened. The irrelevant features in the model improve the accuracy of crowd behavior recognition.

In a possible implementation manner, after obtaining the iteratively updated global features and character spatio-temporal features, the method further includes: performing character behavior recognition based on the iteratively updated character spatio-temporal features to obtain a character behavior recognition result .

In the embodiment of the present invention, the behavior of individual characters can also be identified. For example, in a volleyball game, the characters will have corresponding game actions, such as serving, kicking, passing, smashing, blocking and so on. Therefore, the behavior of a single character can also be identified based on the spatiotemporal characteristics of the character.

Specifically, the spatiotemporal characteristics of the characters can be input into the fully connected layer of the neural network, and the fully connected layer can be used for classification. The fully connected layer will have multiple game actions in advance. According to the spatiotemporal characteristics of the characters, the fully connected layer can output the character spatiotemporal The feature is the reliability of each action, and the game action with the highest reliability can be used as the result of character action recognition.

For example, in the video frame of a volleyball game, the extracted spatiotemporal features of the characters are input to the fully connected layer, and the reliability of the character action "serve" is 0.9, the reliability of "throwing the ball" is 0.3, and the reliability of "passing the ball" is 0.3. The reliability of "is 0.4, the reliability of "smash" is 0.1, and the reliability of "block" is 0.1, then the "serve" action with the highest reliability can be output as the recognition result.

In the embodiment of the present invention, since the spatio-temporal features of the characters are the features after attention allocation, the key features in the global features can be highlighted, and the irrelevant features in the global features can be weakened, so the character action recognition is performed based on the spatio-temporal features of the characters, and the The accuracy of the result of character action recognition is high.

An application scenario of the embodiment of the present invention is described below. In this application scenario, the behavior recognition method of the present invention can be implemented based on an end-to-end network. The network can input a video frame sequence. In this application scenario, the input video frame is a video frame in a volleyball match, and the video A frame sequence contains multiple movie frames. Then extract the global feature of each movie frame and each character feature.

For the extracted character features, each character feature in the same film frame is clustered to obtain a clustering result, based on the clustering result, determine the attention distribution weight of each character feature; update the character feature based on the attention distribution weight, The spatial encoding features are obtained, and then the spatial encoding features are decoded to obtain the character spatial features. In addition, the time-domain features of the characters can also be obtained based on the features of the characters in different movie frames. With regard to the obtained character spatial features and the character temporal features, the two may be fused to obtain the fused character spatio-temporal features. The spatio-temporal features of the person can be iteratively updated as a new character feature until the obtained spatio-temporal features of the character meet the preset convergence condition.

For the extracted global features, attention can be allocated to the global features based on the spatio-temporal features of the characters, and the spatio-temporal features of the characters can be updated to realize the optimization of the spatio-temporal features of the characters. The optimized spatio-temporal features of the characters can be used as input again to perform iterative optimization of the global features. This process can be repeated multiple times until the obtained global features meet the preset convergence conditions.

The updated spatio-temporal features of the person can be input into the fully connected layer, and the fully connected layer can be used to classify, identify the action of a single person, and obtain the action recognition result of a single person. It is also possible to input the updated global features into the fully connected layer, use the fully connected layer to classify, identify crowd behavior, and obtain the crowd behavior recognition result of the entire movie frame sequence.

For example, the iteratively updated global features are input to the fully connected layer, and the reliability of the action category of "left serve" is 0.9, the reliability of "left pass" is 0.3, and the reliability of "left spike" is obtained. is 0.4, the reliability of "right serve" is 0.1, and the reliability of "right block" is 0.1, then the most reliable "left serve" can be output as the crowd behavior recognition result of the video frame in the volleyball game . Input the extracted spatio-temporal features of the characters into the fully connected layer, and obtain the reliability of the characters' actions as "serve the ball" with a reliability of 0.9, "pad the ball" with a reliability of 0.3, "pass the ball" with a reliability of 0.4, and "smash the ball" with a reliability of 0.9. The reliability of is 0.1, and the reliability of "block" is 0.1. Then, the action of "serve" with the highest reliability can be output as the result of character behavior recognition.

It can be understood that the above-mentioned method embodiments mentioned in the present invention can all be combined with each other to form a combined embodiment without violating the principle and logic. Due to space limitations, the present invention will not repeat them. Those skilled in the art can understand that, in the above method in the specific implementation manner, the specific execution order of each step should be determined according to its function and possible internal logic.

In addition, the present invention also provides behavior recognition devices, electronic equipment, computer-readable storage media, and programs, all of which can be used to implement any behavior recognition method provided by the present invention, corresponding technical solutions and descriptions, and corresponding records in the method section, No longer.

Fig. 2 shows the block diagram of the behavior recognition device according to the embodiment of the present invention, as shown in Fig. 2, described device 20 comprises: a character feature extraction unit 21, is used for receiving the film frame of input, and extracts described film frame character features in; a clustering unit 22, for clustering a plurality of the character features in the film frame, to obtain a clustering result; an attention distribution unit 23, for based on the clustering result Determine the attention distribution weights of character features in the film frame; a character feature update unit 24, for updating the character features based on the attention distribution weight; a character spatio-temporal feature extraction unit 25, for based on the update The final character feature, extracting the character spatio-temporal feature; and a behavior recognition unit 26, used to perform behavior recognition on the video frame based on the character spatio-temporal feature, and obtain the recognition result.

In a possible implementation manner, the attention allocation unit 23 is configured to determine an attention allocation weight between the character features based on the association relationship between the character features in the clustering result.

In a possible implementation manner, the attention allocation unit 23 includes: a first similarity determination unit, configured to determine the first similarity between the character features in the same group obtained by clustering; and a first An attention determination unit, configured to determine a first attention distribution weight among characters in the group based on the first similarity.

In a possible implementation manner, the first similarity determination unit is configured to divide the feature matrix of the character features into N parts; and calculate the similarity for the N features of different character features to obtain N The first degree of similarity; the first attention determination unit is configured to determine N first attention distribution weights between the characters in the group based on the N first similarities.

In a possible implementation manner, the attention allocation unit 23 includes: an overall feature determination unit, used to determine the overall features of each group obtained by clustering; a second similarity determination unit, used to determine the clustering The obtained second similarity between the overall features of each group; and a second attention determination unit, configured to determine a second attention distribution weight among the character features based on the second similarity.

In a possible implementation manner, the character feature update unit 24 is configured to use the target character feature and the first attention distribution weights of each character feature in the group for a target character feature in a single group of character features to The characteristics of each person in the group are weighted and summed to obtain the update characteristics in the group corresponding to each person in the group, which is used as the updated person characteristic.

In a possible implementation manner, the character feature update unit 24 includes: an inter-group update feature determination unit, configured to use the overall feature of the target and the overall feature of each group for the target overall feature of the target group in each group The second attention distribution weight of each group is obtained corresponding to the update feature between groups; and a character feature update subunit, which is used to add the update feature between the groups to the update feature in the group corresponding to each character in the target grouping , get the updated character features.

In a possible implementation manner, the character spatio-temporal feature extraction unit is configured to perform spatial decoding on the updated character features to obtain the character spatio-temporal features.

In a possible implementation manner, the character spatio-temporal feature extraction unit includes: a spatial decoding unit, configured to spatially decode the updated character features to obtain character spatial features; a time-domain encoding and decoding unit, configured to convert multiple Time-domain encoding and decoding are performed on the character features of each film frame to obtain character time-domain features; and a fusion unit is used to fuse the character space features with the character time-domain features to obtain character space-time features.

In a possible implementation manner, the time-domain encoding and decoding unit includes: a time-domain encoding unit, configured to encode the character features of multiple film frames based on a self-attention mechanism to obtain time-domain encoding features; and A time-domain decoding unit, configured to decode the time-domain coding features based on a self-attention mechanism, and/or decode the time-domain coding features based on spatial coding features to obtain character time-domain features; wherein, the spatial The encoded feature is the updated character feature.

In a possible implementation manner, the spatial decoding unit is configured to decode the spatially encoded features based on a self-attention mechanism, and/or decode the spatially encoded features based on the time domain encoded features, to obtain character space features.

In a possible implementation manner, the device further includes: a global feature extraction unit, configured to extract global features of the film frame; a third attention determination unit, configured to determine the The third attention distribution weight in the global feature; and a global feature update unit, used to update the global feature with the third attention distribution weight; the behavior recognition unit includes: a crowd behavior recognition unit, The method is used for performing crowd behavior recognition on the film frame based on the updated global feature to obtain a crowd behavior recognition result.

In a possible implementation manner, the device further includes: an iterative updating unit, configured to use the updated global features as new global features, and use the character spatio-temporal features as new character features, iteratively The global feature and the character spatio-temporal feature are updated until the iterative stop condition is satisfied, and the iteratively updated global feature and the character spatio-temporal feature are obtained; The frame is used for crowd behavior recognition, and the result of crowd behavior recognition is obtained.

In a possible implementation manner, the device further includes: a character behavior recognition unit, configured to perform character behavior recognition based on the iteratively updated spatio-temporal characteristics of the character, and obtain a character behavior recognition result.

In a possible implementation manner, the character feature extraction unit 21 includes: a target rectangular frame determination unit, configured to perform human body recognition in movie frames to obtain the target rectangular frame of each character; and a character feature extraction subunit , for extracting features in the film frame, and using the target rectangular frame in the film frame to match the features of the extracted film frame to obtain corresponding character features.

In some embodiments, the functions or modules included in the device provided by the embodiments of the present invention can be used to execute the methods described in the above method embodiments, and its specific implementation and technical effects can refer to the descriptions of the above method embodiments, For the sake of brevity, no more details are given here.

The embodiment of the present invention also proposes a computer-readable storage medium on which computer program instructions are stored, and the above-mentioned method is implemented when the computer program instructions are executed by a processor. Computer readable storage media can be volatile or non-volatile computer readable storage media.

An embodiment of the present invention also proposes an electronic device, including: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to call the instructions stored in the memory to execute the above method.

An embodiment of the present invention also provides a computer program product, including computer-readable codes, or a non-volatile computer-readable storage medium carrying computer-readable codes, when the computer-readable codes are run in a processor of an electronic device , the processor in the electronic device executes the above method.

Electronic devices may be provided as terminals, servers, or other types of devices.

Fig. 3 shows a block diagram of an electronic device 800 according to an embodiment of the present invention. For example, the electronic device 800 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant and other terminals.

Referring to FIG. 3, the electronic device 800 may include one or more of the following elements: a processing element 802, a memory 804, a power supply element 806, a multimedia element 808, an audio element 810, an input/output (I/O) interface 812, a sensor device element 814, and communication element 816.

The processing element 802 generally controls the overall operations of the electronic device 800, such as operations associated with display, telephone calls, data communications, camera operations, and recording operations. The processing element 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above method. Additionally, processing element 802 may include one or more modules to facilitate interaction between processing element 802 and other elements. For example, the processing element 802 may include a multimedia module to facilitate interaction between the multimedia element 808 and the processing element 802 .

The memory 804 is configured to store various types of data to support operations in the electronic device 800 . Examples of such data include instructions for any application or method operating on the electronic device 800, contact data, phonebook data, messages, pictures, videos, and the like. Memory 804 may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electronically erasable programmable read-only memory (EEPROM), erasable In addition to programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

The power supply element 806 provides power to various elements of the electronic device 800 . Power components 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for electronic device 800 .

The multimedia component 808 includes a screen providing an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch panel to receive input signals from a user. The touch panel includes one or more touch sensors to sense touch, swipe and gestures on the touch panel. The touch sensor may not only sense the boundary of a touch or swipe action, but also detect the duration and pressure associated with the touch or swipe action. In some embodiments, the multimedia element 808 includes a front camera and/or a rear camera. When the electronic device 800 is in an operation mode, such as a shooting mode or a movie mode, the front camera and/or the rear camera can receive external multimedia data. Each front camera and rear camera can be a fixed optical lens system or have focal length and optical zoom capability.

The audio component 810 is configured to output and/or input audio signals. For example, the audio component 810 includes a microphone (MIC), which is configured to receive an external audio signal when the electronic device 800 is in an operation mode, such as a calling mode, a recording mode and a voice recognition mode. The received audio signal can be further stored in the memory 804 or sent via the communication element 816 . In some embodiments, the audio component 810 further includes a speaker for outputting audio signals.

The I/O interface 812 provides an interface between the processing element 802 and the peripheral interface module, which may be a keyboard, a click wheel, a button, and the like. These buttons may include, but are not limited to: a home button, volume buttons, start button, and lock button.

Sensor element 814 includes one or more sensors for providing various aspects of status assessment for electronic device 800 . For example, the sensor element 814 can detect the open/closed state of the electronic device 800, the relative positioning of elements such as the display and keypad of the electronic device 800, the sensor element 814 can also detect the electronic device 800 or The position of a component of the electronic device 800 changes, the presence or absence of user contact with the electronic device 800 , the orientation or acceleration/deceleration of the electronic device 800 and the temperature of the electronic device 800 change. The sensor element 814 may include a proximity sensor configured to detect the presence of nearby objects in the absence of any physical contact. The sensor element 814 may also include a light sensor, such as a complementary metal oxide semiconductor (CMOS) or charge coupled device (CCD) image sensor, for use in imaging applications. In some embodiments, the sensor element 814 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor or a temperature sensor.

The communication element 816 is configured to facilitate wired or wireless communication between the electronic device 800 and other devices. The electronic device 800 can access a wireless network based on a communication standard, such as a wireless network (WiFi), a second generation mobile communication technology (2G) or a third generation mobile communication technology (3G), or a combination thereof. In an exemplary embodiment, the communication component 816 receives broadcast signals or broadcast related messages from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication element 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on Radio Frequency Identification (RFID) technology, Infrared Data Association (IrDA) technology, Ultra Wide Band (UWB) technology, Bluetooth (BT) technology and other technologies.

In an exemplary embodiment, electronic device 800 may be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field Programmable gate array (FPGA), controller, microcontroller, microprocessor or other electronic component implementation for performing the method described above.

In an exemplary embodiment, there is also provided a non-volatile computer-readable storage medium, such as the memory 804 including computer program instructions, which can be executed by the processor 820 of the electronic device 800 to implement the above method.

Fig. 4 shows a block diagram of an electronic device 1900 according to an embodiment of the present invention. For example, electronic device 1900 may be provided as a server. Referring to FIG. 4 , the electronic device 1900 includes a processing element 1922 , which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions executable by the processing element 1922 , such as application programs. The application programs stored in the memory 1932 may include one or more modules each corresponding to a set of instructions. In addition, the processing element 1922 is configured to execute instructions to perform the above method.

Electronic device 1900 may also include a power supply element 1926 configured to perform power management of electronic device 1900, a wired or wireless network interface 1950 configured to connect electronic device 1900 to a network, and an input/output (I/O) Interface 1958. The electronic device 1900 can operate based on the operating system stored in the memory 1932, such as the Microsoft server operating system (Windows Server ^TM ), the operating system based on the graphical user interface (Mac OS X ^TM ) introduced by Apple Inc., multi-user multi-process computer Operating System (Unix ^™ ), Free and Open Source Unix-like Operating System (Linux ^™ ), Open Source Unix-like Operating System (FreeBSD ^™ ) or similar.

In an exemplary embodiment, there is also provided a non-volatile computer-readable storage medium, such as the memory 1932 including computer program instructions, which can be executed by the processing element 1922 of the electronic device 1900 to implement the above method.

The present invention can be a system, method and/or computer program product. The computer program product may include a computer-readable storage medium loaded with computer-readable program instructions for causing a processor to implement various aspects of the invention.

A computer readable storage medium may be a tangible device that can hold and store instructions for use by an instruction execution device. A computer readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: pen drives, hard drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only Memory (EPROM or Flash), Static Random Access Memory (SRAM), Compact Disc Read Only Memory (CD-ROM), Digital Versatile Disc (DVD), Memory Stick, Floppy Disk, Mechanically Encoded Devices, such as Punched cards or raised-in-recess structures having instructions stored thereon, and any suitable combination of the foregoing. As used herein, computer-readable storage media are not to be construed as transient signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., pulses of light through fiber optic cables), or transmitted electrical signals.

The computer readable program instructions described herein may be downloaded from a computer readable storage medium to each computing/processing device, or to an external computer over a network, such as the Internet, a local area network, a wide area network, and/or a wireless network or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers and/or edge servers. A network adapter card or network interface in each computer/processing device receives computer-readable program instructions from the network and forwards the computer-readable program instructions for computer-readable storage in each computer/processing device in storage media.

Computer program instructions for performing the operations of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or any Composition of source or object code written in programming languages including object-oriented programming languages such as Smalltalk, C++, etc., and conventional procedural programming languages such as "C" or similar programming languages. Computer readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or executed on the server. In cases involving a remote computer, the remote computer can be connected to the user computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (for example, using the Internet) Internet Service Provider to connect via the Internet). In some embodiments, electronic circuits, such as programmable logic circuits, field programmable gate arrays (FPGAs) or programmable logic arrays (PLAs), are customized by utilizing state information of computer readable program instructions, the electronic circuits Computer readable program instructions can be executed to implement various aspects of the invention.

Aspects of the present invention are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart and/or block diagram and combinations of blocks in the flowchart and/or block diagram can be implemented by computer readable program instructions.

These computer-readable program instructions may be provided to a processor of a general-purpose computer, special-purpose computer, or other programmable data-processing device to produce a machine such that, when executed by the processor of the computer or other programmable data-processing device, Means that implement the function/action specified in one or more blocks in the flowchart and/or block diagram are produced. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause a computer, a programmable data processing device and/or other equipment to work in a specific way, so that a computer-readable medium storing instructions includes An article of manufacture comprising instructions for implementing various aspects of the functions/acts specified in one or more blocks in flowcharts and/or block diagrams.

It is also possible to load computer-readable program instructions into a computer, other programmable data processing device, or other equipment, so that a series of operation steps are executed on the computer, other programmable data processing device, or other equipment to produce a computer-implemented process , so that instructions executed on computers, other programmable data processing devices, or other devices implement the functions/actions specified in one or more blocks in the flowcharts and/or block diagrams.

The flowchart and block diagrams in the drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction that contains one or more logic for implementing the specified Executable instructions for a function. In some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks in succession may, in fact, be executed substantially concurrently, or they may sometimes be executed in the reverse order, depending upon the functionality involved. It is also to be noted that each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, can be implemented by a dedicated hardware-based system that performs the specified function or action implemented, or may be implemented using a combination of dedicated hardware and computer instructions.

The computer program product can be realized by hardware, software or a combination thereof. In an optional embodiment, the computer program product is embodied as a computer storage medium. In another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (Software Development Kit, SDK), etc. wait.

Having described various embodiments of the present invention, the foregoing description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and alterations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principle of each embodiment, practical application or improvement of technology in the market, or to enable other ordinary skilled in the art to understand each embodiment disclosed herein.

But the above-mentioned ones are only embodiments of the present invention, and should not limit the scope of the present invention. All simple equivalent changes and modifications made according to the patent scope of the present invention and the content of the patent specification are still within the scope of the present invention. Within the scope covered by the patent of the present invention.

S11~S16: Steps 20: Behavior recognition device 21: Character feature extraction unit 22: Clustering unit 23: Attention allocation unit 24:Characteristic update unit 25: Character space-time feature extraction unit 26: Behavior recognition unit 800, 1900: Electronic equipment 802, 1922: Processing elements 804, 1932: memory 806, 1926: Power components 808:Multimedia components 810:Audio components 812: input/output interface 814: sensor element 816: Communication components 820: Processor 1950: Web interface 1958: Input and output interface

Other features and effects of the present invention will be clearly presented in the implementation manner with reference to the drawings, wherein: Fig. 1 shows a flow chart of a behavior recognition method according to an embodiment of the present invention; FIG. 2 shows a block diagram of a behavior recognition device according to an embodiment of the present invention; FIG. 3 shows a block diagram of an electronic device according to an embodiment of the present invention; and Fig. 4 shows a block diagram of an electronic device according to an embodiment of the present invention.

S11~S16: Steps

Claims (17)

A behavior recognition method, comprising: receiving an input movie frame, and extracting character features in the movie frame; Clustering a plurality of the character features in the movie frame to obtain a clustering result; Determining the attention distribution weights of the character features in the film frame based on the clustering result; updating the character features based on the attention distribution weights; Based on the updated character features, extracting character spatio-temporal features; and Based on the spatiotemporal characteristics of the person, conduct behavior recognition on the film frame to obtain a recognition result. The method as described in claim 1, wherein said determination of the attention distribution weights of the character features in the film frame based on the clustering results includes: Based on the association relationship between the character features in the clustering result, determine the attention distribution weights among the character features. The method according to claim 2, wherein, based on the association relationship between the character features in the clustering result, determining the attention distribution weight between the character features includes: determining the first similarity between the character features in the same group obtained by clustering; and Based on the first similarity, a first attention distribution weight among characters in the group is determined. The method as described in claim 3, wherein determining the first similarity between the characteristics of the characters in the same group obtained by clustering includes: Dividing the feature matrix of the character features into N parts; Computing corresponding similarities for N features of different character features, to obtain N first similarities; and Based on the first similarity, determining the first attention distribution weight between the characters in the group includes: Based on the N first similarities, determine N first attention distribution weights among the characters in the group. The method according to any one of claim items 2-4, wherein, based on the association relationship between the character features in the clustering result, determining the attention distribution weight between the character features includes: Determine the overall characteristics of each group obtained by clustering; Determining a second degree of similarity between the overall characteristics of each group obtained by clustering; and Based on the second similarity, a second attention distribution weight between the character features is determined. The method according to claim 1, wherein updating the character features based on the attention distribution weights includes: For the target character feature in the character features of a single group, use the target character feature and the first attention distribution weight of each character feature in the group, carry out weighted summation on each character feature in the group, and obtain the corresponding weight of each character in the group The update feature within the group is used as the updated character feature. The method according to claim 1, wherein updating the character features based on the attention distribution weights includes: For the target overall feature of the target grouping in each group, using the second attention distribution weight of the target overall feature and the overall feature of each group, obtain the update feature between groups corresponding to each group; and The inter-group update features are respectively added to the intra-group update features corresponding to the characters in the target group to obtain updated character features. The method according to claim 1, wherein said extracting the spatio-temporal features of the character based on the updated character features includes: Spatial decoding is performed on the updated character features to obtain the spatio-temporal features of the character. The method as described in claim item 8, wherein spatial decoding is performed on the updated character features to obtain the character spatio-temporal features, including: Perform spatial decoding on the updated character features to obtain the character spatial features; performing time-domain encoding and decoding on the character features of multiple film frames to obtain character time-domain features; and The space-time feature of the person is fused with the time-domain feature of the person to obtain the space-time feature of the person. The method according to claim 9, wherein the time-domain encoding and decoding of the character features of multiple film frames to obtain the character time-domain features includes: Encoding the character features of multiple film frames based on a self-attention mechanism to obtain time-domain encoding features; and Decoding the time-domain coding features based on a self-attention mechanism, and/or decoding the time-domain coding features based on spatial coding features, to obtain character time-domain features; wherein, the spatial coding features are the updated character traits. The method according to claim 10, wherein the spatial decoding of the updated character features to obtain the character spatial features includes: The spatial encoding features are decoded based on a self-attention mechanism, and/or the spatial encoding features are decoded based on the time-domain encoding features to obtain character spatial features. The method according to any one of claim items 8-11, wherein the method further comprises: extracting global features of the film frame; Using the character spatio-temporal features to determine the third attention distribution weight in the global features; updating the global features using the third attention distribution weights; The step of performing behavior recognition on the movie frame based on the spatio-temporal characteristics of the character to obtain a recognition result includes: Based on the updated global features, crowd behavior recognition is performed on the movie frame to obtain a crowd behavior recognition result. The method according to claim 12, wherein, after using the third attention distribution weight to update the global features, the method further includes: The updated global feature is used as a new global feature, and the person's spatio-temporal feature is used as a new character feature, and the global feature and the person's spatio-temporal feature are iteratively updated until the iterative stop condition is satisfied, and the iteratively updated Global characteristics and character spatiotemporal characteristics; and The step of performing behavior recognition on the movie frame based on the spatio-temporal characteristics of the character to obtain a recognition result includes: Based on the iteratively updated global features, crowd behavior recognition is performed on the movie frame to obtain a crowd behavior recognition result. The method according to claim 13, wherein, after obtaining the iteratively updated global features and character spatio-temporal features, the method further includes: Performing character behavior recognition based on the iteratively updated spatiotemporal features of the character to obtain a character behavior recognition result. The method as claimed in claim 1, wherein said extracting character features in said movie frame includes: Carry out human body recognition in the video frame to obtain the target rectangular frame of each character; and The features in the film frame are extracted, and the corresponding character features are obtained by matching the features of the extracted film frame by using the target rectangular frame in the film frame. An electronic device comprising: a processor; and a memory for storing processor-executable instructions; Wherein, the processor is configured to invoke instructions stored in the memory to execute the method described in any one of claim items 1 to 15. A computer-readable storage medium, on which computer program instructions are stored, wherein the computer program instructions implement the method described in any one of claims 1 to 15 when executed by a processor.

Images