JP2022546153A - Action recognition method, device, computer equipment and storage medium - Google Patents

Abstract

Embodiments of the present invention disclose a motion recognition method, apparatus, computer equipment and storage medium, comprising obtaining a first image and recognizing a target image region including a target object in the first image. and performing motion detection processing on the target image region using a motion detection network having a plurality of motion detection branches to generate a plurality of types of first motion detection results corresponding to the target object. obtaining a second motion detection of the target object based on different categories of motion detected by different motion detection branches and first motion detection results respectively corresponding to the plurality of motion detection branches; determining a result. [Selection drawing] Fig. 1

Description

(Cross reference to related applications)
This disclosure is filed under and claims priority from a Chinese patent application with application number 202010755553.3 and filing date of July 31, 2020, is incorporated into this disclosure by reference.

TECHNICAL FIELD The present disclosure relates to the technical field of computer vision, and in particular to motion recognition methods, devices, computer equipment, and storage media.

At present, the Internet education industry is developing rapidly, providing students and teachers with a convenient and comfortable teaching environment. Intelligentization of classroom interaction is an important direction of current Internet education, mainly including intelligentization based on student's action recognition and facial expression recognition. In the conventional Internet education, the interaction between the student and the teacher is completed mainly by ringing an electronic doorbell, etc., which makes it difficult to determine the student's condition and limits the sense of experience.

Embodiments of the present invention provide at least a motion recognition method, apparatus, computing device and storage medium.

In a first aspect, embodiments of the present invention provide a motion recognition method. The method for motion recognition comprises: acquiring a first image; recognizing a target image region containing a target object in the first image; performing motion detection processing on an image region to obtain a plurality of types of first motion detection results corresponding to the target object, wherein categories of motion detected by different motion detection branches are: different; and determining a second motion detection result for the target object based on first motion detection results respectively corresponding to a plurality of motion detection branches.

Thus, by recognizing a student's motion using a motion detection network having multiple motion detection branches, different motion detection branches can now detect different categories of motion, and furthermore, a single detection process In the process, detection results are obtained for each of the various actions made by the student, thereby allowing a comprehensive and accurate recognition of the student's actions.

In one optional embodiment, recognizing a target image region containing a target object in said first image comprises performing a feature extraction process on said first image to obtain first features of said first image. obtaining a map, wherein the first feature map includes feature submaps respectively corresponding to a plurality of feature channels, wherein features included in different feature submaps are different; determining first coordinate information of a center point of the target object in the first feature map based on features contained in a first feature submap; and determining first coordinate information of the center point in the first feature map. and determining first size information of the target object on the first feature map based on features contained in a second feature submap of the plurality of feature submaps; and determining the first coordinate information and the determining the target image area based on the first size information.

Thus, a target image region containing the target object can be accurately determined from the first image.

In one optional embodiment, determining first coordinate information of a center point of the target object on the first feature map based on features included in a first feature submap of the plurality of feature submaps. According to a preset pooling size and pooling stride, maximum pooling processing is performed on the first feature submap to correspond to a plurality of pooling values and each of the plurality of pooling values. obtaining a position index, wherein the position index is used to identify the position of the pooling value in the first feature submap; and based on each of the pooling values and a first threshold, a plurality of the determining a target pooling value belonging to the center point from pooling values; determining first coordinate information of the center point in the first feature map based on a position index corresponding to the target pooling value; including.

Thus, by performing the max pooling operation on the first feature submap, the target pooling value belonging to the center point of the target object is determined even more accurately from the plurality of pooling values, thereby obtaining the target from the first image. The position of the object can be determined more accurately.

In one optional embodiment, determining the target image area based on the first coordinate information and the first size information comprises: the first coordinate information, the first size information, and the first feature second coordinate information of the center point in the first image, and Determining second size information of the target object, and determining the target image area based on the second coordinate information and the second size information.

In one optional embodiment, determining the target image area based on the second coordinate information and the second size information comprises: determining the first target image area based on the second coordinate information and the second size information; determining a first area extent containing the target object from an image; and based on the first area extent containing the target object, determining a second area extent containing the target object, wherein the second area an area greater than the first area extent and the second area extent encompassing the first area extent; and determining the target image area from the first image based on the second area extent. ,including.

Thus, by extending the first area extent to obtain the second area extent, the target object can be more completely encompassed, thereby detecting the motion made by the target object based on the target image area. At times, more accurate detection results can be obtained.

In one optional embodiment, the motion detection network comprises a feature extraction network and a plurality of motion detection branch networks connected to the feature extraction network, using a motion detection network having a plurality of motion detection branches. and obtaining a plurality of types of first motion detection results corresponding to the target object by performing a motion detection process on the target image region using the feature extraction network on the target image region. obtaining a second feature map of the target image region by performing feature extraction processing on the second feature map; and performing motion detection processing on each of the second feature maps using a plurality of the motion detection branch networks. , obtaining a first motion detection result respectively corresponding to each of the motion detection branch networks.

In this way, by performing motion detection processing on each of the second feature maps of the target image region using a plurality of motion detection branch networks, various motion categories can be detected for each target image region. detection is realized, and a more comprehensive motion detection result for each target object is obtained.

In one optional embodiment, a plurality of said motion detection branch networks are used to respectively perform a motion detection process on said second feature map to detect a first motion respectively corresponding to each said motion detection branch network. Obtaining a detection result includes, for each of a plurality of motion detection branch networks, performing motion detection processing on the second feature map using the motion detection branch network, wherein the target object detects the motion. obtaining a probability of categorizing a motion detected by a detection branch network; determining a first motion detection result corresponding to the motion detection branch network based on the probability and a second predetermined threshold; including.

In a second aspect, embodiments of the present invention further provide a motion recognition device. The motion recognition device comprises an acquisition module configured to acquire a first image, a recognition module configured to recognize a target image region including a target object in the first image, and a plurality of motion detection branches. a detection module configured to perform a motion detection process on the target image region to obtain a plurality of types of first motion detection results corresponding to the target object using a motion detection network comprising wherein a second motion detection result of the target object is determined based on detection modules with different categories of motion detected by different motion detection branches and first motion detection results respectively corresponding to a plurality of motion detection branches; a decision module configured to decide.

In one possible embodiment, the recognition module performs a feature extraction process on the first image to obtain a first feature map of the first image, the first feature map comprising a plurality of features. including feature submaps respectively corresponding to channels, wherein features included in different feature submaps are different, and based on features included in a first feature submap of a plurality of feature submaps, Determining first coordinate information of a center point of the target object and including first coordinate information of the center point on the first feature map and a feature included in a second one of the plurality of feature submaps. and determining the target image region based on the first coordinate information and the first size information.

In one possible embodiment, the recognition module performs a maximum pooling operation on the first feature submap according to a preset pooling size and pooling stride to obtain multiple pooling values and multiple pooling values. obtaining a position index corresponding to each of said pooling values, said position index being used to identify the position of said pooling value in said first feature submap, and for each said pooling value and a first threshold; determining a target pooling value belonging to the center point from the plurality of pooling values; and determining first coordinate information of the center point in the first feature map according to a position index corresponding to the target pooling value. configured as

In one possible embodiment, the recognition module stores the first coordinate information, the first size information, and the first feature point in the first feature map and each pixel point in the first image. determining second coordinate information of the center point in the first image and second size information of the target object in the first image based on a positional mapping relationship between the second coordinate information and the second It is configured to determine the target image area based on two size information.

In one possible embodiment, the recognition module determines, from the first image, a first area extent containing the target object based on the second coordinate information and the second size information; determining a second area extent including the target object based on the first area extent, the second area extent encompassing the first area extent, and the target from the first image based on the second area extent; It is configured to determine an image region.

In one possible embodiment, the motion detection network comprises a feature extraction network and a plurality of motion detection branch networks connected to the feature extraction network, the detection module using the feature extraction network to detect the target performing feature extraction processing on an image region to obtain a second feature map of the target image region, and performing motion detection processing on each of the second feature maps using a plurality of the motion detection branch networks. Doing so is configured to obtain a first motion detection result respectively corresponding to each of said motion detection branch networks.

In one possible embodiment, the detection module performs motion detection processing on the second feature map using the motion detection branch network for each of a plurality of motion detection branch networks, Obtaining a probability that the target object performs a category of motion detected by the motion detection branch network, and determining a first motion detection result corresponding to the motion detection branch network based on the probability and a predetermined second threshold. configured to determine.

In a third aspect, optional embodiments of the present invention further provide computer equipment. The computing device includes a processor, a memory in which machine-readable instructions executable by the processor are stored, the processor for executing the machine-readable instructions stored in the memory. used to perform the steps in any one possible embodiment of the first aspect or the first aspect above when the machine-readable instructions are executed by the processor.

In a fourth aspect, an alternative embodiment of the invention provides that, when a computer program is stored and the computer program is executed, the steps of any one of the first aspect or any one possible embodiment of the first aspect above are performed. Further provided is a computer readable storage medium for execution.

In a fifth aspect, an optional embodiment of the invention further provides a computer program product for causing a computer to perform the steps in any one possible embodiment of the first aspect or the first aspect above.

In order to make the above objects, features and advantages of the present invention more comprehensible, preferred embodiments will be particularly mentioned below and will be described in detail with reference to the accompanying drawings.

In order to describe the technical solutions of the embodiments of the present invention more clearly, the following briefly introduces the drawings required for the embodiments, and the drawings here are incorporated into the specification to be incorporated into the specification. These drawings show embodiments consistent with the present invention and are used together with the description to explain the technical solutions of the present invention. The following drawings show only some embodiments of the present invention and should not be considered as limiting the scope, and those skilled in the art will be able to make other modifications based on these drawings without creative effort. It should be understood that the relevant drawings of the can also be obtained.
4 is a flow chart illustrating a method for motion recognition provided by an embodiment of the present invention; 4 is a flow chart illustrating a specific method for recognizing a target image region containing a target object in a first image provided by an embodiment of the present invention; 1 is a schematic diagram illustrating the structure of a motion recognition network provided by an embodiment of the present invention; FIG. 1 is a schematic diagram of a motion recognition device provided by an embodiment of the present invention; FIG. 1 is a schematic diagram of a computing device provided by an embodiment of the present invention; FIG.

In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the following clearly and completely describes the technical solutions in the embodiments of the present invention with reference to the drawings in the embodiments of the present invention. It should be understood that the described embodiments are only some, but not all embodiments of the present invention. In general, the components of the embodiments of the invention described and illustrated in the drawings herein can be arranged and designed in a variety of different configurations. Accordingly, the following detailed description of embodiments of the invention provided in the drawings is not intended to limit the scope of the invention for which protection is sought, but merely selected embodiments of the invention. It is nothing more than an indication of All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

Studies show that computer vision-based student action recognition in the classroom mainly analyzes student actions in the classroom, including actions such as standing up, raising hands, and lying down on a desk, mainly through body detection, tracking and action classification technology. . When classifying and recognizing the actions of students, the design of the model structure is generally a multi-class classification design. The probability that the student will perform each of the three actions of standing up, raising his hand, or lying down on the desk is then predicted, and then the action with the highest probability is determined as the action that the student has taken. However, in an actual classroom, students may perform multiple actions at the same time, such as raising their hands and standing up at the same time, or raising their hands and lying down on a desk at the same time. However, current detection methods cannot detect multiple actions performed by students at the same time, so there is a problem that the actions of students in the classroom cannot be fully recognized.

Based on the above research, an embodiment of the present invention provides a motion recognition method, using a motion detection network with multiple motion detection branches to recognize a student's motion, where different motion detection branches can be detected. The categories of actions are different, and a single detection processing process can obtain detection results for each of the various actions performed by the student, thereby recognizing the student's actions comprehensively and accurately.

The drawbacks of the above-mentioned solutions are the result of the inventor's practice and meticulous research. The inventors contributed to the present invention in the present invention.

It should be noted that similar symbols and letters represent similar items in the following drawings, and therefore, if a term is defined in one drawing, it need not be further defined and construed in subsequent drawings. want to be

In order to facilitate understanding of this embodiment, first, the motion recognition method disclosed by the embodiment of the present invention will be described in detail. The execution body of the motion recognition method provided by the embodiments of the present invention is generally a motion recognition device, which includes, for example, a terminal device or a server or other processing device, and the terminal device is a user device. (UE: User Equipment), mobile devices, user terminals, terminals, cellular phones, cordless phones, personal digital assistants (PDA: Personal Digital Assistant), portable devices, computing devices, in-vehicle devices, wearable devices, and the like. In some possible embodiments, the motion recognition method may be implemented by a processor invoking computer readable instructions stored in memory.

In the following, the action recognition method provided by the embodiments of the present invention will be described by taking the action recognition device as an execution body as an example. It should be noted that the motion recognition method provided by the embodiment of the present invention not only can recognize the motion of the students in the classroom, but also can be applied to other motion detection scenes in which various motions can be performed simultaneously.

FIG. 1 is a flowchart of a motion recognition method provided by an embodiment of the present invention, the method comprising:
step S101 of obtaining a first image;
recognizing a target image region containing a target object in said first image S102;
In step S103, obtain a plurality of types of first motion detection results corresponding to the target object by performing motion detection processing on the target image region using a motion detection network having multiple motion detection branches. a step S103 in which categories of motions detected by different motion detection branches are different;
determining a second motion detection result of the target object based on first motion detection results respectively corresponding to a plurality of motion detection branches.

In the motion recognition method provided by the embodiments of the present invention, the target object includes, for example, any one of a person, an animal, a mechanical device, a vehicle, a robot, and the like.

A motion detection network is, for example, a neural network model. Illustratively, after training the neural network, a neural network model is obtained that can detect actions made by the target object contained in the first image. The neural network model includes a plurality of motion detection branches, also called detection heads, which are branch networks in the motion detection network. Each detection head can obtain a respective probability that the target object will perform certain categories of actions. Different detection heads detect different categories of motion. A first action detection result corresponding to each action detection branch may indicate whether the target subject performs the category of action detected by the corresponding action detection branch. The second action detection result may indicate whether the target subject performs a category of actions respectively detected by the plurality of action detection branches.

In an embodiment of the present invention, after acquiring a first image, a target image region of a target object contained in the first image is recognized, and a motion detection network having multiple motion detection branches is used to perform motion detection on the target image region. performing motion detection processing on the target object to obtain a plurality of types of first motion detection results corresponding to the target object; and further based on the first motion detection results respectively corresponding to the plurality of motion detection branches to perform a second motion of the target object. A detection result is determined, and a single detection processing process obtains a detection result for each of the various actions performed by the student, so that the student's action can be recognized comprehensively and accurately.

In the embodiment of the present invention, S101 to S104 will be described in detail by taking as an example the application of the motion recognition method to motion detection for students.

I: In the above S101, the method of acquiring the first image differs depending on the application scene.

Exemplarily, when the method is applied to a classroom scene, a motion recognition device can be installed in a classroom where a teacher teaches, the motion recognition device is, for example, a terminal device, and the motion recognition device is a class room. A first image can be acquired in real time when a student takes a class by a camera attached to the camera, or a camera is provided in the action recognition device, and the action recognition device is provided with a camera so that the student can take a lesson. A first image can be acquired when receiving

In the scene of an online class, the motion recognition device is, for example, a teacher terminal, a student terminal, or a server. acquires a first image including the students by and transmits the first image to the teacher terminal, the teacher terminal receives the first image transmitted from the student terminal, and the student based on the first image is Detects motion. When the motion recognition device is a student-side terminal, a camera is connected to the student-side terminal, and the student-side terminal obtains a first image including the student by the camera connected thereto, and recognizes the action performed by the student based on the first image. detect, and then send the detection result to the teacher-side terminal, so that the teacher can know in real time what the student has done through the teacher-side terminal. When the action recognition device is a server, the server receives the first image transmitted from the student terminal, detects the action performed by the student based on the first image, and further transmits the detection result to the teacher terminal.

II: In S102 above, in addition to the target object, the acquired first image also contains other image background information, and the image background information may interfere with the second motion detection result of the target object to some extent. , it is possible to first detect the target image area of the target object contained in the first image, and then realize motion detection for the target object based on the target image area.

In some optional embodiments, as shown in FIG. 2, embodiments of the present invention provide a specific method for recognizing a target image region containing a target object in the first image, the method comprising: , including the following steps:

In S201, perform feature extraction processing on the first image to obtain a first feature map of the first image, the first feature map including feature submaps respectively corresponding to a plurality of feature channels, and different The features included in the feature submaps are different.

In some alternative embodiments, a convolutional neural network may be used to perform feature extraction processing on the first image to obtain a first feature map of the first image. After performing feature extraction processing on the first image using a convolutional neural network, a first feature map of the first image is obtained.

Illustratively, the first feature map is composed of feature sub-maps of a plurality of channels, and the first feature map is obtained after superimposing the plurality of feature sub-maps.

In S202, determining first coordinate information of the center point of the target object on the first feature map based on features included in a first feature submap of a plurality of feature submaps; determining a first size information of the target object at the first feature map based on first coordinate information of the center point at and features contained in a second one of the plurality of feature submaps. .

Exemplarily, among the plurality of feature submaps that make up the first feature map, the features included in the feature submap of the i-th channel (that is, the first feature submap) are each It is used to characterize whether the first feature point is the center point of the target object. activating the first feature submap with a sigmoid activation function to convert the feature value of each first feature point in the first feature map in the first feature submap to a numerical value between a1 and a2; can be done. Illustratively, a1 is 0, for example, and a2 is 1, for example.

Here, for a certain first feature point, after the feature value of the first feature point in the first feature submap is converted into a numerical value between 0 and 1, the more the corresponding numerical value approaches 1, the more belongs to the center point of the target object.

Further, based on the numerical value after the feature value of each first feature point in the first feature submap is transformed between 0 and 1, the corresponding first feature point in the first feature map of the center point of each target object. A feature point may be determined, and first coordinate information of the determined first feature point may be determined as first coordinate information of a center point of the target object on the first feature map.

In another possible embodiment, the actual prediction process involves transforming the feature value of each first feature point in the first feature map in the first feature submap to a number between 0 and 1, then the position Since the numerical values corresponding to first feature points that are close to each other may also be close to each other, and a unique center point can be determined for each target object, embodiments of the present invention further employ the following method to employ the first feature point A first coordinate information of a center point of the target object on a map may be determined.

performing a maximum pooling operation on the first feature submap according to a preset pooling size and pooling stride to obtain a plurality of pooling values and a position index corresponding to each of the plurality of pooling values. , the position index is used to identify the position of the pooling value in the first feature submap, and based on the respective pooling value and a first threshold, a target pooling from a plurality of the pooling values belonging to the center point; A value is determined, and a first coordinate information of the center point in the first feature map is determined based on the position index corresponding to the target pooling value.

Illustratively, for example, a maximum pooling operation of 3×3 and a stride of 1 can be performed on the first feature submap, and when performing the pooling operation, every 3×3 For the feature value of the first feature point, determine the maximum response value of the 3×3 first feature points and the position index of the maximum response value in the first feature map. At this time, the quantity of the maximum response value is related to the size of the first feature map. After that, there are a total of 80×60 maximum response values obtained, and for each maximum response value, there may be at least one other maximum response value, all of which have the same position index. Further, the maximum response values with the same position index are merged to obtain M maximum response values and a position index corresponding to each of the M maximum response values. Furthermore, each of the M maximum response values is compared with a first threshold, and if a maximum response value is greater than the first threshold, the maximum response value is determined as the target pooling value. The position index corresponding to the target pooling value is the first coordinate information of the center point of the target object in the first feature map.

Here, activation processing is performed on the first feature submap, and the feature value of each first feature point in the first feature map in the first feature submap is converted into a numerical value between 0 and 1. , the max pooling process may be performed, or the max pooling process may be performed directly on the first feature submap.

When performing the maximum pooling process directly on the first feature submap, after performing the maximum pooling process on the first feature submap, the activation process is performed on each pooling value by an activation function, and each pooling After transforming the value into a number between 0 and 1, based on the pooling value transformed into a number between 0 and 1 and the first threshold, a target pooling value belonging to the center point of the target object from the plurality of pooling values. can be determined.

Further, when performing the maximum pooling process directly on the first feature submap, after performing the maximum pooling process on the first feature submap, the target It is also possible to directly determine the target pooling value belonging to the center point of interest, where the first threshold is different from the value of the first threshold in some examples above where the activation process needs to be performed. Specifically, it can be specifically selected according to actual needs.

In another example, in the plurality of feature submaps that make up the first feature map, the features included in the feature submaps of the jth channel and the kth channel (that is, the second feature submap) are: A first size information of the target object in the first image is used to characterize the first size information in the first feature map.

Exemplarily, the feature value of each first feature point in the first feature map in the feature submaps of the j-th channel is the length value in the first size information corresponding to each first feature point. Characterizing, the feature value of each first feature point in the feature submap of the kth channel characterizes the width value in the first size information corresponding to each first feature point.

For example, if the number of feature submaps is 3, i may be 0, j may be 1, and k may be 2, for example. Specific values of i, j and k are set according to the actual neural network processing process.

After obtaining the first coordinate information of the center point on the first feature map, based on the first coordinate information, from the second feature submap, the first feature point characterizing the center point on the second feature submap. Reading the feature values and determining the read feature values as the first size information of the target object in the first feature map.

In S203, the target image area is determined based on the first coordinate information and the first size information.

In some alternative embodiments, after performing feature extraction processing on the first image to obtain a first feature map of the first image, embodiments of the present invention further adopt the following method to: A target image region may be determined. generating a positional mapping relationship between each pixel point in the first image and a first feature point in the first feature map, based on the first coordinate information, the first size information, and the positional mapping relationship; , second coordinate information of a center point in a first image (the second coordinate information of the center point in a first image can be determined based on the location mapping relationship and the first coordinate information); determining a second size information of the target object in one image (based on the first size information to determine the second size information of the target object in a first image), and a center point in a first image; and second size information of the target object in the first image.

When determining the target image area based on the second coordinate information and the second size information, one possible embodiment directly determines the target image area based on the second coordinate information and the second size information. Alternatively, in another possible embodiment, based on the second coordinate information and the second size information, a first area extent containing the target object is determined from the first image; determining a second area extent including the target object, the second area extent being greater than the first area extent, and the second area extent encompassing the first area extent; and based on the second area extent, a first A target image region may be determined from the image.

Illustratively, in an embodiment of the present invention, based on the center point of the first area extent and the four vertices of the first area extent, move along the direction from the center point to the vertex (i.e., the each moving away from the center point and other vertices) to expand the area range based on the first area range, and after each vertex is moved, the position of the four vertices of the second area range and the second area range can be obtained based on the positions of the four vertices of the second area range. Here, the movement distance of each vertex may be the same or different, that is, in the process of expanding the area range based on the first area range, the expansion width/extension size of the area around each vertex is , which may be the same or different, and are not limited here.

III: In S103 above, the motion detection network includes, for example, a feature extraction network and a plurality of motion detection branch networks connected to the feature extraction network. Here, each motion detection branch network corresponds to one motion detection branch, and the categories of motions detected by different motion detection branch networks are different.

In performing motion detection processing on the target image region using a motion detection network having a plurality of motion detection branches to obtain a plurality of types of first motion detection results corresponding to the target object, in part: In alternative embodiments of , for example, the following process may be employed.

performing feature extraction processing on the target image region using the feature extraction network to obtain a second feature map of the target image region; and using a plurality of the motion detection branch networks to obtain the second feature. A motion detection process is performed on each map to obtain a first motion detection result respectively corresponding to each of the motion detection branch networks.

In some alternative embodiments, a plurality of the motion detection branch networks are used to respectively perform motion detection processing on the second feature map, and a first motion detection corresponding to each of the motion detection branch networks respectively. Obtaining results may employ, for example, the following processes. For each of a plurality of motion detection branch networks, performing motion detection processing on the second feature map using the motion detection branch network, wherein the target object is motion detected by the motion detection branch network. Obtaining a categorization probability, and determining a first motion detection result corresponding to the motion detection branch network based on the probability and a predetermined second threshold.

Illustratively, as shown in FIG. 3, embodiments of the present invention provide examples of specific configurations of motion detection networks. When the action recognition method provided by the embodiment of the present invention is applied to student's action recognition in the classroom, there are three action detection branch networks, respectively A, B and C, and the action detection branch network A stands up. The motion detection branch network B detects the motion category of raising hands, and the motion detection branch network C detects the motion category of lying face down on a desk. After obtaining the first image and determining the target image regions in the first image for each student in the first image, feature extraction network M is used to extract features for the target image regions corresponding to each student. processing to obtain a second feature map corresponding to each student, and using the motion detection branch network A to perform motion detection processing on the second feature map to obtain the probability that the student will stand up. , and based on the probability that the student will stand up and the corresponding second threshold, determine a first motion detection result that the student will stand up, e.g. If the probability of performing the action is greater than a corresponding second threshold, then it is determined that the student has performed a standing action.

Similarly, the motion detection branch network B is used to perform motion detection processing on the second feature map to obtain the probability that the student will raise his hand, and the probability that the student will raise his hand and the corresponding second Based on the threshold, a first motion detection result of the student's hand-raising motion is determined. Perform motion detection processing on the second feature map using the motion detection branch network C, obtain the probability that the student will make the motion of lying down on the desk, and the probability that the student will make the motion of lying down on the desk and the correspondence. Based on the second threshold, the first motion detection result that the student makes a motion of lying face down on the desk is determined.

Finally, the final second motion detection result of the student based on the first motion detection results respectively corresponding to the standing motion, the raising of the hand, and the motion of lying down on the desk. to decide.

For example, the first motion detection result obtained by the motion detection branch network A is not standing, the first motion detection result obtained by the motion detection branch network B is that the hand is raised, and the motion detection branch network C is obtained. If the first motion detection result is that the person is not lying face down on the desk, the corresponding second motion detection result is that the person is not standing up, raising his hand, and not lying face down on the desk.

Here, it should be noted that the second thresholds corresponding to different motion detection branch networks can be the same or different, and can be specifically set according to actual needs.

Exemplarily, the image size of the target image region input to the feature extraction network is 112×112, and the feature extraction network downsamples the target image region four times to obtain a second feature map of size 7×7. where the downsampling process for the target image region is, for example, to perform four sequential convolution operations with a stride of 2 on the target image region. The 7×7 second feature maps are then input to different motion detection branch networks respectively. For each motion detection branch network, first convolution processing is performed on the second feature map, and average pooling processing is performed on the results of the convolution processing to obtain one-dimensional data, and then the one-dimensional data is activated by sigmoid. and finally obtain the probabilities corresponding to the motion detection branch network.

Also, in the related art, prior to using a neural network to detect motion of a target object contained in an image, it is common practice to train the neural network on sample images from multiple image acquisition devices to obtain different image acquisitions. Sample images derived from the device contain different image features due to differences in imaging parameters, and the neural network can learn different features derived from different sample images during the training process, thereby using the neural network to improve the image quality. The generalization ability of the neural network can be enhanced when detecting motion of the target object involved. When performing a motion detection process on an image using such a neural network, the neural network outputs a probability that the target object in the image will perform a certain motion, and the probability is combined with a preset probability threshold. and determine a motion detection result corresponding to the target object based on the comparison result. However, due to the different image acquisition parameters of different cameras, there will be differences in the quality of the images acquired by different cameras, and due to the different image quality, there will also be differences in the features of the images involved, thereby making the same Using the probability threshold of 1 as a uniform criterion to obtain motion detection results for images captured by different cameras does not always give optimal results, and some images may be misjudged, resulting in detection errors. A problem arises in that the accuracy of

To solve the above problem, embodiments of the present invention further provide specific methods for determining the second threshold. The method uses the motion detection network to classify each of a plurality of second images associated with the first image to obtain a classification prediction probability for each of the second images; determining a second threshold based on predicted classification probabilities corresponding to each of the second images and pre-labeled actual classification results respectively corresponding to a plurality of the second images.

Here, the relationship between the first image and the second image is
the similarity between the imaging parameters of the first image and the second image is greater than a preset similarity threshold;
and acquiring a plurality of the first images and the second images with the same image acquisition device.

In this way, based on the classification results of the plurality of second images related to the first image, the second threshold is obtained, and during the process of classifying the first image, the first image and the second image are classified. is related, if the first classification threshold is taken as one of the judgment criteria during the classification process, the motion detection result of the second image can be obtained with higher accuracy, thereby increasing the accuracy rate of the classification result can increase

The relationship between the first image and the second image includes, for example, at least one of the following.

(1) The degree of similarity between the imaging parameters of the first image and the second image is greater than a preset similarity threshold.

For example, the imaging parameters of different images can be parameter vectors, and the similarity of the imaging parameters of different images can be characterized, for example, by the vector distance between the parameter vectors of different images, and the parameter vectors of different images If the vector distance between is less than a preset distance threshold, the similarity of the shooting parameters of the different images is characterized as being greater than a preset similarity threshold.

(2) A plurality of the first images and the second images are acquired by the same image acquisition device.

Before starting to use the image capturing device, the first probability threshold of the image capturing device can be determined in the above steps S101-S102, and after the image capturing device is used, the determined first probability threshold Based on this, classification processing is performed on the acquired second image.

The second threshold needs to be determined so that the accuracy rate of the discrimination result reaches the preset accuracy rate threshold when the classification result of the second image is discriminated with the determined second threshold value.

Exemplarily, the following method may be employed to determine the second threshold.

determining a plurality of candidate thresholds, and for each of the plurality of candidate thresholds, predictions corresponding to each of the candidate thresholds based on classification prediction probabilities and actual classification results respectively corresponding to the plurality of second images; An accuracy rate is determined, and the second threshold is determined from the plurality of candidate thresholds based on predicted accuracy rates respectively corresponding to the plurality of candidate thresholds.

Illustratively, a plurality of candidate thresholds can be determined within the bidding range based on the bidding range of the second threshold and the preset bidding stride.

Exemplarily, after classifying the second image using the motion detection network, the result of the classification process is activated with, for example, a sigmoid activation function, and the change value of the result of the classification process is changed from 0 to 1. , where the classified results characterize the classification prediction probabilities of the second image. Accordingly, the range of values for the second threshold becomes [0, 1]. Assuming 0.05 as the valuation stride, the determined candidate thresholds are respectively 0, 0.05, 0.1, 0.15, 0.2, 0.25, 0.3, 0.35, at 0.4, 0.45, 0.5, 0.55, 0.6, 0.65, 0.7, 0.75, 0.8, 0.85, 0.9, 0.95, 1 There may be.

Here, the bidding range of the second threshold can be determined according to the actual situation, and the bidding stride can also be determined according to actual needs, for example, the bidding stride can be 0.01, Note that it may be determined to be 0.02 or the like and is not limited to this in the present example.

Illustratively, the result of classifying the second image by the motion detection network includes that the target object in the second image made a certain action or that the target object in the second image did not make that action; Suppose that after classifying a plurality of second images using a motion detection network, we obtain the classification prediction probability of the nth second image denoted by score_n. Assuming that the picking range of the second threshold is [0,1] and the picking stride is 0.001, the possible values of the second threshold with the stride of 0.001 are thrd=0+0.001 xk, where kε[0,1000]. For the p-th traversal, the determined candidate threshold is thrd_p=0+0.001×p, with this candidate threshold thrd_p, if score_n is greater than thrd_p, the predicted classification result of the second image has taken the corresponding action. otherwise, the predictive classification result of the second image is that the corresponding action is not performed.

Subsequently, statistically based on the predicted classification results corresponding to the n second images and the actual classification results corresponding to the n second images respectively,
TP: number of second images predicted to have acted and acted at a candidate threshold of thrd_p;
TN: number of second images predicted to actually have motion and not have motion at the candidate threshold of thrd_p;
FP: the number of second images predicted to have had no motion and have had motion at a candidate threshold of thrd_p; Obtain a parameter, the number of second images that are expected to be absent.

Subsequently, the prediction accuracy rate F corresponding to the candidate threshold of thrd_p is obtained based on the following formulas (1) to (3).

After obtaining the prediction accuracy rates of all the candidate thresholds, the candidate threshold with the highest prediction accuracy rate F is determined as the second threshold.

In another embodiment of the present invention, in order to more accurately determine the second threshold, the bidding range of the second threshold is further divided into a plurality of bidding intervals, followed by each bidding interval , based on the classification prediction probability and the actual classification result corresponding to each of the plurality of second images, determine the prediction accuracy rate corresponding to each pricing interval, and then determine the prediction accuracy rate corresponding to each of the plurality of pricing intervals determine a target bidding interval from the multiple bidding intervals, then determine multiple candidate thresholds from the multiple target bidding intervals, and determine the prediction accuracy rate corresponding to each candidate threshold according to the above process. may be used, thereby reducing the amount of computation required when determining the second threshold, saving computational resources and computing time.

Also, embodiments of the present invention may further employ a stepwise approach to determine the second threshold.

In the specific embodiment of the above method, the description order of each step implies a strict execution order and does not constitute any limitation to the implementation process, and the specific execution order of each step is not limited to its function and possible intrinsic logic. It is understood by those skilled in the art that should be determined by

Based on the same inventive idea, the embodiment of the present invention further provides a motion recognition device corresponding to the method of motion recognition, and the principle of the device in the embodiment of the present invention to solve the problem is the above operation of the embodiment of the present invention. Since it is the same as the recognition method, the implementation of the device can be referred to the implementation of the method, and redundant descriptions will be omitted.

FIG. 4 is a schematic diagram of a motion recognition device provided by an embodiment of the present invention, said device comprising an acquisition module 41, a recognition module 42, a detection module 43 and a determination module 44, wherein:
the acquisition module 41 is configured to acquire the first image;
the recognition module 42 is configured to recognize a target image region containing a target object in the first image;
A detection module 43 performs motion detection processing on the target image region using a motion detection network having multiple motion detection branches to obtain multiple types of first motion detection results corresponding to the target object. where the categories of behavior detected by different behavior detection branches are different,
A determination module 44 is configured to determine a second motion detection result of the target object based on first motion detection results respectively corresponding to a plurality of motion detection branches.

In one possible embodiment, the recognition module 42 performs a feature extraction process on the first image to obtain a first feature map of the first image, the first feature map comprising a plurality of feature channels. , wherein features included in different feature submaps are different, and based on features included in a first feature submap of a plurality of feature submaps, the Determining first coordinate information of a center point of a target object, based on the first coordinate information of the center point on the first feature map and features contained in a second one of the plurality of feature submaps. , determining first size information of the target object in the first feature map, and determining the target image region based on the first coordinate information and the first size information.

In one possible embodiment, the recognition module 42 performs a max pooling operation on the first feature submap according to a preset pooling size and pooling stride, and calculates multiple pooling values and multiple pooling values and multiple pooling strides. obtaining a position index corresponding to each of the pooling values, the position index being used to identify the position of the pooling value in the first feature submap, based on each of the pooling values and a first threshold; determining a target pooling value belonging to the center point from a plurality of the pooling values, and determining first coordinate information of the center point in the first feature map based on a position index corresponding to the target pooling value. configured to

In one possible embodiment, the recognition module 42 recognizes the first coordinate information, the first size information, and the first feature point in the first feature map and each pixel point in the first image. determining second coordinate information of the center point in the first image and second size information of the target object in the first image based on a positional mapping relationship between the second coordinate information and the second It is configured to determine the target image area based on two size information.

In one possible embodiment, the recognition module 42 determines, from the first image, a first area extent containing the target object based on the second coordinate information and the second size information; determining a second area extent containing the target object based on a first area extent containing the target object, wherein the second area extent encompasses the first area extent, and based on the second area extent from the first image the It is configured to determine a target image region.

In one possible embodiment, the motion detection network comprises a feature extraction network and a plurality of motion detection branch networks connected to the feature extraction network;
The detection module 43 performs feature extraction processing on the target image region using the feature extraction network to obtain a second feature map of the target image region, and uses a plurality of the motion detection branch networks. Then, motion detection processing is performed on each of the second feature maps to obtain first motion detection results corresponding to each of the motion detection branch networks.

In one possible embodiment, the detection module 43, for each of a plurality of motion detection branch networks, performs a motion detection process on the second feature map using the motion detection branch network; Obtaining a probability that a target object performs a category of motion detected by the motion detection branch network, and based on the probability and a predetermined second threshold, determining a first motion detection result corresponding to the motion detection branch network. configured to determine.

For the description of the processing flow of each module in the device and the interaction flow between each module, please refer to the relevant descriptions in the above method embodiments, which will not be described in detail herein.

In an embodiment of the present invention, the acquisition module 41, the recognition module 42, the detection module 43, and the determination module 44 in the motion recognition device are all implemented as a central processing unit (CPU), a digital It can be realized by a signal processor (DSP), a microcontroller unit (MCU) or a field-programmable gate array (FPGA).

An embodiment of the present invention further provides a computer device, and FIG. 5 is a structural schematic diagram of a computer device provided by an embodiment of the present invention, said computer device comprising a processor 11 and a memory 12, wherein said Machine-readable instructions executable by the processor 11 are stored in the memory 12 to implement the steps of the motion recognition method according to the embodiments of the present invention when the computer equipment operates. Machine-readable instructions are executed by the processor.

For the specific execution process of the above instructions, please refer to the steps of the motion recognition method described in the embodiments of the present invention, and the description is omitted here.

An embodiment of the present invention further provides a computer readable storage medium on which a computer program is stored and which, when the computer program is executed by a processor, performs the steps of the motion recognition method described in the above method embodiments. . Here, the storage medium may be a volatile or non-volatile computer readable storage medium.

A computer program product of a motion recognition method provided by an embodiment of the present invention includes a computer readable storage medium storing program code, wherein instructions contained in the program code are adapted to perform the operations described in the above method embodiments. It can be used to carry out the steps of the recognition method, specifically refer to the above method embodiments, and the description is omitted here.

An embodiment of the present invention further provides a computer program product that, when executed by a processor, implements the method of any one of the above embodiments. The computer program product may be specifically implemented in hardware, software or a combination thereof. In one alternative embodiment, said computer program product is tangibly embodied as a computer storage medium, and in another alternative embodiment, said computer program product is tangibly embodied as a software development kit (SDK: It is embodied as a software product such as Software Development Kit).

Features disclosed in several method or apparatus embodiments provided by the present invention may be combined arbitrarily to obtain new method embodiments or apparatus embodiments, provided they do not conflict.

Those skilled in the art will clearly understand that the specific operating processes of the above-described systems and devices can refer to the corresponding processes in the method embodiments for convenience and brevity of description, Description is omitted here. It should be understood that in some embodiments provided by the present invention, the disclosed systems, devices and methods may be embodied in other forms. The embodiments of the device described above are only exemplary, for example, the division of the units is only the division of logical functions, and there may be other division modes when actually implemented, and For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not performed. Also, mutual couplings or direct couplings or communication connections shown or discussed may be indirect couplings or communication connections through some communication interface, device or unit, and may be electrical, mechanical or otherwise. .

The units described as separate members above may or may not be physically separated, and the members indicated as units may or may not be physical units, and may or may not be located in one place. or distributed over multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the present embodiment.

Also, each functional unit in each embodiment of the present invention may be integrated into one processing unit, individual units may physically exist alone, and two or more units may be combined into one processing unit. may be integrated into the unit.

When the functions are implemented in the form of software functional units and sold or used as stand-alone products, they may be stored in a processor-executable non-volatile computer-readable storage medium. Based on this understanding, the technical solution of the present invention can be implemented in the form of a software product essentially or the part contributing to the prior art or the part of the technical solution can be implemented in the form of a computer software product. are stored in a storage medium, and are used to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. including instructions for The storage medium includes USB flash memory, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk, optical disk, etc. Including media.

Finally, the above-described examples to be described are only specific embodiments of the present invention, and are only for describing the technical solutions of the present invention, not for limiting it. , the scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the above embodiments, any person skilled in the art can still modify the technical solutions described in the above embodiments within the technical scope disclosed in the present invention. or can easily conceive of changes or equivalent replacements for some technical features thereof, but these modifications, changes or replacements do not affect the essence of the corresponding technical solution. It should be understood that all should fall within the protection scope of the present invention without departing from the spirit and scope of the technical solutions in the embodiments of the invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the above claims.

Claims (11)

A motion recognition method comprising:
obtaining a first image;
recognizing a target image region containing a target object in the first image;
obtaining a plurality of types of first motion detection results corresponding to the target object by performing motion detection processing on the target image region using a motion detection network having a plurality of motion detection branches; where different categories of motion detected by different motion detection branches and
determining a second motion detection result for the target object based on first motion detection results respectively corresponding to a plurality of motion detection branches;
Action recognition method. Recognizing a target image region containing a target object in the first image comprises:
performing a feature extraction process on the first image to obtain a first feature map of the first image, the first feature map including feature submaps respectively corresponding to a plurality of feature channels; , the features included in different feature submaps are different;
determining first coordinate information of a center point of the target object on the first feature map based on features included in a first feature submap of a plurality of feature submaps; Determining first size information of the target object on the first feature map based on first coordinate information of the center point and features included in a second one of the plurality of feature submaps. ,
2. The method of claim 1, further comprising determining the target image area based on the first coordinate information and the first size information. Determining first coordinate information of a center point of the target object on the first feature map based on features included in a first feature submap of a plurality of feature submaps;
A plurality of pooling values and a position index corresponding to each of the plurality of pooling values are obtained by performing a maximum pooling process on the first feature submap according to a preset pooling size and pooling stride. obtaining, wherein the position index is used to identify the position of the pooled value in the first feature submap;
determining a target pooling value belonging to the center point from a plurality of the pooling values based on each pooling value and a first threshold;
3. The method of claim 2, comprising determining first coordinate information of the center point in the first feature map based on a position index corresponding to the target pooling value. Determining the target image area based on the first coordinate information and the first size information includes:
in the first image based on the first coordinate information, the first size information, and a positional mapping relationship between a first feature point in the first feature map and each pixel point in the first image; determining second coordinate information for the center point of and second size information for the target object in the first image;
4. The motion recognition method according to claim 2, further comprising determining the target image area based on the second coordinate information and the second size information. Determining the target image area based on the second coordinate information and the second size information includes:
determining a first area extent containing the target object from the first image based on the second coordinate information and the second size information;
determining a second area range that includes the target object based on a first area range that includes the target object, wherein the second area range encompasses the first area range;
5. The method of claim 4, comprising determining the target image region from the first image based on the second region extent. the motion detection network includes a feature extraction network and a plurality of motion detection branch networks connected to the feature extraction network;
obtaining a plurality of types of first motion detection results corresponding to the target object by performing motion detection processing on the target image region using a motion detection network having a plurality of motion detection branches;
performing a feature extraction process on the target image region using the feature extraction network to obtain a second feature map of the target image region;
respectively performing motion detection processing on the second feature map using a plurality of the motion detection branch networks to obtain a first motion detection result respectively corresponding to each of the motion detection branch networks. The motion recognition method according to any one of claims 1 to 5, characterized by: obtaining a first motion detection result corresponding to each of the motion detection branch networks by performing motion detection processing on each of the second feature maps using a plurality of the motion detection branch networks;
For each of a plurality of motion detection branch networks, performing motion detection processing on the second feature map using the motion detection branch network, such that the target object is detected by the motion detection branch network. obtaining the probability of doing a category of behavior;
determining a first motion detection result corresponding to the motion detection branch network based on the probability and a predetermined second threshold. A motion recognition device,
an acquisition module configured to acquire a first image;
a recognition module configured to recognize a target image region containing a target object in the first image;
configured to perform motion detection processing on the target image region using a motion detection network having a plurality of motion detection branches to obtain a plurality of types of first motion detection results corresponding to the target object; a detection module in which different categories of motion detected by different motion detection branches are different;
a determination module configured to determine a second motion detection result of the target object based on first motion detection results respectively corresponding to a plurality of motion detection branches;
motion recognition device. a computer device,
including a memory and a processor;
the memory stores machine-readable instructions executable by the processor;
the processor is used to execute machine-readable instructions stored in the memory;
When the machine-readable instructions are executed by the processor, the processor performs the steps of the action recognition method according to any one of claims 1 to 7,
computer equipment. A computer readable storage medium,
A computer program is stored on the computer-readable storage medium,
A computer readable storage medium for performing the steps of the action recognition method according to any one of claims 1 to 7, when the computer program is run on a computer device. A computer program,
causing a computer to execute the action recognition method according to any one of claims 1 to 7;
computer program.

Images