CN112115746A - Human body action recognition device and method and electronic equipment - Google Patents

Abstract

Embodiments of the present invention provide a human body motion recognition device and method, and electronic equipment. First, the bounding box of the human body is detected in the input image, and there are selected key points based on the human body in the detected bounding box to detect the action of the human body and/or based on the convolutional neural network. The detection method is faster and has higher recognition accuracy, and by combining the two detection methods, different detection methods can be selected according to different situations, which can flexibly respond to various scenarios and needs.

Description

Human body motion recognition device and method, electronic equipment

technical field

The present invention relates to the field of information technology.

Background technique

In recent years, research in the field of computer vision has made great progress with the help of deep learning. Deep learning refers to a collection of algorithms that use various machine learning algorithms on hierarchical neural networks to solve various problems such as images and texts. The core of deep learning is feature learning, which aims to obtain hierarchical feature information through hierarchical neural networks, so as to solve the important problems that required manual design of features in the past.

Security monitoring is one of the important applications of deep learning, and human action and behavior recognition is an important part of security monitoring.

It should be noted that the above description of the technical background is only for the convenience of clearly and completely describing the technical solutions of the present invention and facilitating the understanding of those skilled in the art. It should not be assumed that the above-mentioned technical solutions are well known to those skilled in the art simply because these solutions are described in the background section of the present invention.

SUMMARY OF THE INVENTION

However, due to the complex movements of the human body and the changeable application scenarios, the existing action recognition methods have slow processing speed and low recognition accuracy. In addition, they cannot flexibly respond to various scenarios and needs.

Embodiments of the present invention provide a human body action recognition device and method, and electronic equipment. First, a bounding box of a human body is detected in an input image, and human body-based key points are selected in the detected bounding box to detect the human body's actions and actions. / or based on the convolutional neural network to detect human actions, in this way, through the hierarchical detection method, the processing speed is faster and the recognition accuracy is higher, and by combining the two detection methods, different detection methods can be selected according to different situations. It can flexibly respond to various scenarios and needs.

According to a first aspect of the embodiments of the present invention, there is provided a human motion recognition device, the device includes: a target detection unit, which is used for detecting the bounding box of the human body in an input image; and a first detection unit, which is used for detecting In the bounding box of the human body, the features of the human body are calculated based on the key points of the human body, and the movements of the human body are detected according to the features of the human body to obtain the first recognition result; the second detection unit, which uses In the bounding box of the detected human body, the action of the human body is detected based on the convolutional neural network to obtain a second recognition result; and a selection unit is used to select the first detection unit and the second detection unit. At least one of the units is used to detect the motion of the human body, so as to obtain at least one of the first recognition result and the second recognition result.

According to a second aspect of the embodiments of the present invention, there is provided an electronic device, the electronic device includes the apparatus according to the first aspect of the embodiments of the present invention.

According to a third aspect of the embodiments of the present invention, there is provided a human action recognition method, the method comprising: detecting a bounding box of a human body in an input image; selecting and performing at least one of the following detections: at the detected boundary of the human body In the frame, the features of the human body are calculated based on the key points of the human body, and the movements of the human body are detected according to the features of the human body to obtain a first recognition result; and in the bounding box of the detected human body, based on A convolutional neural network is used to detect the action of the human body, and a second recognition result is obtained.

The beneficial effect of the present invention is: firstly, the bounding box of the human body is detected in the input image, and the human body-based key points are selected in the detected bounding box to detect the human body's action and/or the convolutional neural network to detect the human body. In this way, through the hierarchical detection method, the processing speed is faster and the recognition accuracy is higher, and by combining the two detection methods, different detection methods can be selected according to different situations, which can flexibly respond to various scenarios and needs.

With reference to the following description and drawings, specific embodiments of the invention are disclosed in detail, indicating the manner in which the principles of the invention may be employed. It should be understood that embodiments of the present invention are not thereby limited in scope. Embodiments of the invention include many changes, modifications and equivalents within the spirit and scope of the appended claims.

Features described and/or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, in combination with, or instead of features in other embodiments .

It should be emphasized that the term "comprising/comprising" when used herein refers to the presence of a feature, integer, step or component, but does not exclude the presence or addition of one or more other features, integers, steps or components.

Description of drawings

The accompanying drawings, which are included to provide a further understanding of the embodiments of the invention, constitute a part of the specification, are used to illustrate embodiments of the invention, and together with the written description, serve to explain the principles of the invention. Obviously, the drawings in the following description are only some embodiments of the present invention, and for those of ordinary skill in the art, other drawings can also be obtained from these drawings without creative effort. In the attached image:

1 is a schematic diagram of a human body motion recognition device according to Embodiment 1 of the present invention;

FIG. 2 is a schematic diagram of the first detection unit 102 according to Embodiment 1 of the present invention;

3 is a schematic diagram of the detection result of the key points of the human body according to Embodiment 1 of the present invention;

4 is a schematic diagram of obtaining the features of the human body based on key points according to Embodiment 1 of the present invention;

FIG. 5 is a schematic diagram of performing human motion recognition using the human motion recognition device 100 according to Embodiment 1 of the present invention;

6 is a schematic diagram of an electronic device according to Embodiment 2 of the present invention;

7 is a schematic block diagram of a system configuration of an electronic device according to Embodiment 2 of the present invention;

FIG. 8 is a schematic diagram of a method for recognizing human action according to Embodiment 3 of the present invention.

Detailed ways

The foregoing and other features of the present invention will become apparent from the following description with reference to the accompanying drawings. In the specification and drawings, specific embodiments of the invention are disclosed in detail, which are indicative of some of the embodiments in which the principles of the invention may be employed, it being understood that the invention is not limited to the described embodiments, but rather The invention includes all modifications, variations and equivalents falling within the scope of the appended claims.

Example 1

Embodiments of the present invention provide a human motion recognition device. FIG. 1 is a schematic diagram of a human body motion recognition device according to Embodiment 1 of the present invention.

As shown in FIG. 1 , the human body motion recognition device 100 includes:

A target detection unit 101, which is used to detect the bounding box of the human body in the input image;

The first detection unit 102 is used to calculate the features of the human body based on the key points of the human body in the detected bounding box of the human body, and detect the movements of the human body according to the features of the human body to obtain a first recognition result;

The second detection unit 103 is configured to detect the motion of the human body based on the convolutional neural network in the detected bounding box of the human body, and obtain a second recognition result; and

The selection unit 104 is configured to select at least one of the first detection unit 102 and the second detection unit 103 to detect the motion of the human body, so as to obtain at least one of the first recognition result and the second recognition result.

It can be seen from the above embodiment that the bounding box of the human body is first detected in the input image, and selected key points based on the human body are selected in the detected bounding box to detect the action of the human body and/or detect the movement of the human body based on the convolutional neural network. In this way, through the hierarchical detection method, the processing speed is faster and the recognition accuracy is higher, and by combining the two detection methods, different detection methods can be selected according to different situations, which can flexibly respond to various scenarios. and demand.

In this embodiment, the input image may be an image obtained in real time or obtained in advance. For example, the input image is a video image captured by a monitoring device, and each input image corresponds to a frame of the video image.

In this embodiment, the target detection unit 101 is used to detect the bounding box of the human body in the input image. The target detection unit 101 can perform detection based on various target detection methods, for example, Faster R-CNN, FPN, Yolo network, and the like.

In this embodiment, different networks can be used for detection according to different requirements. For example, the Yolo network can be used when the processing speed is high, and the Faster R-CNN network can be used when the recognition accuracy is high.

Through the target detection unit 101, when there is at least one human body in the input image, the bounding box of at least one human body is detected. After detecting the bounding box of the human body, the selection unit 104 selects at least one of the first detection unit 102 and the second detection unit 103 to detect the motion of the human body to obtain at least one of the first recognition result and the second recognition result.

In this embodiment, the selection unit 104 may select at least one of the first detection unit 102 and the second detection unit 103 to detect the motion of the human body according to actual needs or application scenarios, so as to obtain the first recognition result and the second recognition result at least one of the results. When the target detection unit 101 detects multiple bounding boxes of multiple human bodies in the input image, according to the selection result of the selection unit 104, the first detection unit 102 and/or the second detection unit 103 detect the multiple bounding boxes one by one .

For example, for a situation where only simple movements need to be detected, such as simple trunk movements such as walking, standing, and sitting, in this case, the selection unit 104 selects the second detection unit 103 to detect the movements of the human body, ie, outputs the second recognition result.

In this embodiment, the second detection unit 103 detects the motion of the human body based on a convolutional neural network (CNN) in the detected bounding box of the human body, and obtains the second recognition result.

In this embodiment, a popular CNN network can be used to detect the torso action, for example, an AlexNet network can be used for detection.

In this embodiment, when training the CNN network, a training data set can be established first, and the training data set includes actions marked as "walking", "standing", "sit", "running", "squatting" and Images of the "lying" human body, which are available from open datasets.

For another example, in the case where it is necessary to detect simple movements and more complex movements at the same time, for example, in addition to simple trunk movements such as walking, standing, and sitting, it also includes more complex partial movements such as raising the head and raising the hand, in this case, select the unit 104 Select the first detection unit 102 to detect the motion of the human body, that is, to output the first recognition result. Alternatively, the first detection unit 102 and the second detection unit 103 may also be selected to perform detection at the same time, that is, to output the first recognition result and the second recognition result.

In this embodiment, the first detection unit 102 calculates the features of the human body based on the key points of the human body in the detected bounding box of the human body, and detects the movements of the human body according to the features of the human body to obtain the first recognition result.

FIG. 2 is a schematic diagram of the first detection unit 102 according to Embodiment 1 of the present invention. As shown in FIG. 2, the first detection unit 102 includes:

a first detection module 201, which is used to detect the key points of the human body in the detected bounding box of the human body;

a calculation module 202, which is used to calculate the features of the human body according to the detected key points of the human body; and

The second detection module 203 is configured to detect the movement of the human body based on the classifier and/or preset rules according to the calculated features of the human body, and obtain the first recognition result.

In this embodiment, the first detection module 201 may detect the key-points of the human body based on various methods. For example, the first detection module 201 detects the human body based on the Cascaded Pyramid Network (CPN, Cascaded Pyramid Network). key point. Alternatively, it can also be detected based on methods such as Open-pose or Alpha-pose.

In this embodiment, the key points of the human body may include multiple points that respectively represent the positions of multiple parts of the human body, for example, two ears, two eyes, nose, two shoulders, two elbows of the human body, respectively , two wrists, two hips, two knees, and two ankle points.

FIG. 3 is a schematic diagram of the detection result of the key points of the human body according to Embodiment 1 of the present invention. As shown in Figure 3, in the bounding box of a human body, key points representing various parts of the human body are detected by CPN and the position information of these key points can be output.

In this embodiment, the calculation module 202 calculates the features of the human body according to the key points of the human body detected by the first detection module 201. For example, the features of the human body may include: two points representing the positions of multiple parts of the human body respectively. dimensional coordinates; and at least one angle between lines connecting multiple points.

In this embodiment, the characteristics of the human body to be calculated can be determined according to actual needs.

FIG. 4 is a schematic diagram of obtaining features of a human body based on key points according to Embodiment 1 of the present invention. As shown in Figure 4, the key points used to calculate the features include the points where the following body parts are located: nose, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee , right knee, left ankle, and right ankle. The calculated features of the human body include the two-dimensional coordinates of these points, for example, the X and Y coordinates of these points. In addition, the features of the human body may also include the first angle between the left leg and the torso, the angle between the right leg and the torso. 2nd angle, 3rd angle between left calf and left thigh, and 4th angle between right calf and right thigh.

After the calculation module 202 calculates the characteristics of the human body, the second detection module 203 detects the movements of the human body based on the classifier and/or preset rules according to the calculated characteristics of the human body, and obtains a first recognition result.

In this embodiment, the second detection module 203 may detect the body movement of the human body based on the classifier according to the calculated characteristics of the human body, and detect the head movement and the upper limb movement of the human body based on a preset rule.

In this embodiment, the second detection module 203 may detect the torso action of the human body based on various classifiers. For example, the second detection module 203 may perform detection based on a Multi-Layer Perceptron (MLP, Multi-Layer Perceptron) classifier. According to the calculated features and detection based on the MLP classifier, better detection performance can be obtained.

In this embodiment, the second detection module 203 may also detect head movements and upper limb movements of the human body based on preset rules, for example, movements such as looking up, looking down, raising hands, and the like. Preset rules can be set for different actions according to actual needs. For example, when the height of the two ears is higher than the height of the two eyes, it is judged as "looking down"; when the height of the wrist is higher than the elbow When the height is high, it is judged as "raised hand".

FIG. 5 is a schematic diagram of performing human motion recognition using the human motion recognition apparatus 100 according to Embodiment 1 of the present invention. As shown in FIG. 5 , an input image containing multiple human bodies is input to the target detection unit 101 , and the target detection unit 101 detects the bounding boxes of each human body in the input image and outputs to the first detection unit 102 and the second detection unit respectively. In step 103, the first detection unit 102 and the second detection unit 103 perform detection according to the selection result of the selection unit 104, and output at least one of the first recognition result and the second recognition result.

It can be seen from the above embodiment that the bounding box of the human body is first detected in the input image, and selected key points based on the human body are selected in the detected bounding box to detect the action of the human body and/or detect the movement of the human body based on the convolutional neural network. In this way, through the hierarchical detection method, the processing speed is faster and the recognition accuracy is higher, and by combining the two detection methods, different detection methods can be selected according to different situations, which can flexibly respond to various scenarios. and demand.

Example 2

An embodiment of the present invention further provides an electronic device, and FIG. 6 is a schematic diagram of the electronic device according to Embodiment 2 of the present invention. As shown in FIG. 6 , the electronic device 600 includes a human body motion recognition device 601 , and the structure and function of the human body motion recognition device 601 are the same as those described in Embodiment 1, and are not repeated here.

FIG. 7 is a schematic block diagram of a system configuration of an electronic device according to Embodiment 2 of the present invention. As shown in FIG. 7 , the electronic device 700 may include a central processing unit 701 and a memory 702 ; the memory 702 is coupled to the central processing unit 701 . This figure is exemplary; other types of structures may be used in addition to or in place of this structure to implement telecommunication functions or other functions.

As shown in FIG. 7 , the electronic device 700 may further include: an input unit 703 , a display 704 , and a power supply 705 .

In one embodiment, the functions of the human motion recognition device described in Embodiment 1 may be integrated into the central processing unit 701 . Wherein, the central processor 701 may be configured to: detect the bounding box of the human body in the input image; select and perform at least one of the following detections: in the detected bounding box of the human body, calculate the detected bounding box based on the key points of the human body The feature of the human body is detected, and the action of the human body is detected according to the feature of the human body to obtain a first recognition result; and in the bounding box of the detected human body, the action of the human body is detected based on a convolutional neural network, A second identification result is obtained.

For example, in the bounding box of the detected human body, the features of the human body are calculated based on the key points of the human body, and the movements of the human body are detected according to the features of the human body, and the first recognition result is obtained, including: Detecting the key points of the human body in the bounding box of the detected human body; calculating the features of the human body according to the detected key points of the human body; and according to the calculated features of the human body, based on a classifier and/or The preset rules detect the movements of the human body to obtain a first recognition result.

In another implementation manner, the human body motion recognition device described in Embodiment 1 may be configured separately from the central processing unit 701. For example, the human body motion recognition device may be configured as a chip connected to the central processing unit 701, through the central processing unit 701. 701 to realize the function of the human action recognition device.

In this embodiment, the electronic device 700 does not necessarily include all the components shown in FIG. 7 .

As shown in FIG. 7 , the central processing unit 701 , also sometimes referred to as a controller or operational control, may include a microprocessor or other processor device and/or logic device, and the central processing unit 701 receives input and controls the various components of the electronic device 700 operation.

The memory 702 may be, for example, one or more of a cache, flash memory, hard drive, removable media, volatile memory, non-volatile memory, or other suitable devices. And the central processing unit 701 can execute the program stored in the memory 702 to realize information storage or processing. The functions of other components are similar to the existing ones, and are not repeated here. The various components of electronic device 700 may be implemented by dedicated hardware, firmware, software, or a combination thereof, without departing from the scope of the present invention.

It can be seen from the above embodiment that the bounding box of the human body is first detected in the input image, and selected key points based on the human body are selected in the detected bounding box to detect the action of the human body and/or detect the movement of the human body based on the convolutional neural network. In this way, through the hierarchical detection method, the processing speed is faster and the recognition accuracy is higher, and by combining the two detection methods, different detection methods can be selected according to different situations, which can flexibly respond to various scenarios. and demand.

Example 3

An embodiment of the present invention further provides a method for recognizing human body motion, and the method corresponds to the device for recognizing human body motion in Embodiment 1. FIG. 8 is a schematic diagram of a method for recognizing human action according to Embodiment 3 of the present invention. As shown in Figure 8, the method includes:

Step 801: Detect the bounding box of the human body in the input image; and

Step 802: Select and perform at least one of the following detections: in the detected bounding box of the human body, calculate the features of the human body based on the key points of the human body, and detect the human body according to the features of the human body. action to obtain a first recognition result; and in the bounding box of the detected human body, the action of the human body is detected based on a convolutional neural network to obtain a second recognition result.

In this embodiment, the specific implementation methods of the above steps are the same as those described in Embodiment 1, and are not repeated here.

It can be seen from the above embodiment that the bounding box of the human body is first detected in the input image, and selected key points based on the human body are selected in the detected bounding box to detect the action of the human body and/or detect the movement of the human body based on the convolutional neural network. In this way, through the hierarchical detection method, the processing speed is faster and the recognition accuracy is higher, and by combining the two detection methods, different detection methods can be selected according to different situations, which can flexibly respond to various scenarios. and demand.

An embodiment of the present invention further provides a computer-readable program, wherein when the program is executed in a human motion recognition apparatus or electronic device, the program causes a computer to execute Embodiment 3 in the human motion recognition apparatus or electronic equipment The described method for human action recognition.

An embodiment of the present invention further provides a storage medium storing a computer-readable program, wherein the computer-readable program enables a computer to execute the human motion recognition method described in Embodiment 3 in a human motion recognition apparatus or electronic device.

The method for performing human motion recognition in the human motion recognition apparatus or electronic device described in conjunction with the embodiments of the present invention may be directly embodied as hardware, a software module executed by a processor, or a combination of the two. For example, one or more of the functional block diagrams shown in FIG. 1 and/or one or more combinations of the functional block diagrams may correspond to either each software module of the computer program flow, or each hardware module. These software modules may correspond to the respective steps shown in FIG. 8 . These hardware modules can be implemented by, for example, solidifying these software modules using a Field Programmable Gate Array (FPGA).

A software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art. A storage medium can be coupled to the processor, such that the processor can read information from, and write information to, the storage medium; or the storage medium can be an integral part of the processor. The processor and storage medium may reside in an ASIC. The software module can be stored in the memory of the mobile terminal, or can be stored in a memory card that can be inserted into the mobile terminal. For example, if the electronic device adopts a larger-capacity MEGA-SIM card or a large-capacity flash memory device, the software module can be stored in the MEGA-SIM card or a large-capacity flash memory device.

One or more of the functional block diagrams described with respect to FIG. 1 and/or one or more combinations of the functional block diagrams can be implemented as a general purpose processor, a digital signal processor (DSP), a special purpose processor for performing the functions described herein Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or any suitable combination thereof. One or more of the functional block diagrams and/or one or more combinations of the functional block diagrams described with respect to FIG. 1 can also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, multiple microprocessors, One or more microprocessors or any other such arrangement in communication with the DSP.

The present invention has been described above with reference to the specific embodiments, but those skilled in the art should understand that these descriptions are all exemplary and do not limit the protection scope of the present invention. Various variations and modifications of the present invention can be made by those skilled in the art in accordance with the spirit and principles of the present invention, and these variations and modifications are also within the scope of the present invention.

Claims (10)

1. A human body motion recognition device, the device comprising: a target detection unit for detecting the bounding box of the human body in the input image; The first detection unit is configured to calculate the features of the human body based on the key points of the human body in the bounding box of the detected human body, and detect the movements of the human body according to the features of the human body to obtain the first detection unit. identification results; a second detection unit, configured to detect the motion of the human body based on the convolutional neural network in the bounding box of the detected human body, and obtain a second recognition result; and a selection unit, configured to select at least one of the first detection unit and the second detection unit to detect the motion of the human body, so as to obtain at least one of the first recognition result and the second recognition result. 2. The apparatus according to claim 1, wherein the first detection unit comprises: a first detection module, which is used for detecting the key points of the human body in the bounding box of the detected human body; a computing module for computing the features of the human body according to the detected key points of the human body; and The second detection module is configured to detect the movement of the human body based on the classifier and/or preset rules according to the calculated characteristics of the human body, and obtain a first recognition result. 3. The apparatus of claim 2, wherein, The key points of the human body include a plurality of points respectively representing the positions of the multiple parts of the human body. 4. The apparatus of claim 2, wherein, The characteristics of the human body include: two-dimensional coordinates of a plurality of points respectively representing the positions of the plurality of parts of the human body; and at least one angle between lines connecting the plurality of points. 5. The apparatus of claim 2, wherein, The first detection module detects the key points of the human body based on a Cascaded Pyramid Network (CPN). 6. The apparatus of claim 2, wherein, The classifier is a multilayer perceptron (MLP) classifier. 7. The apparatus of claim 1, wherein, The second detection module detects the movement of the torso of the human body based on the classifier according to the calculated characteristics of the human body, and detects the movement of the head and the upper limbs of the human body based on a preset rule. 8. An electronic device comprising the apparatus of any of claims 1-7. 9. A method for human action recognition, the method comprising: Detect the bounding box of the human body in the input image; Select and perform at least one of the following detections: in the bounding box of the detected human body, calculate the features of the human body based on the key points of the human body, and detect the movements of the human body according to the features of the human body, and obtain a first recognition result; and, in the bounding box of the detected human body, detecting the motion of the human body based on a convolutional neural network to obtain a second recognition result. 10. The method according to claim 9, wherein, in the bounding box of the detected human body, the feature of the human body is calculated based on the key points of the human body, and the feature of the human body is detected according to the feature of the human body. action to obtain the first recognition result, including: Detect key points of the human body in the bounding box of the detected human body; Calculate the features of the human body according to the detected key points of the human body; and According to the calculated features of the human body, the movements of the human body are detected based on a classifier and/or a preset rule, and a first recognition result is obtained.

Images