CN112183155B

CN112183155B - Method and device for establishing action posture library, generating action posture and identifying action posture

Info

Publication number: CN112183155B
Application number: CN201910588266.5A
Authority: CN
Inventors: 康凯; 侯琦; 冀志龙
Original assignee: Beijing Xintang Sichuang Educational Technology Co Ltd
Current assignee: Beijing Xintang Sichuang Educational Technology Co Ltd
Priority date: 2019-07-02
Filing date: 2019-07-02
Publication date: 2022-05-17
Anticipated expiration: 2039-07-02
Also published as: CN112183155A

Abstract

The embodiment of the invention provides a method for establishing an action attitude library, a method for generating and identifying an action attitude and a device thereof, wherein the method for establishing the action attitude library comprises the following steps: acquiring a human body image to be put in a warehouse; acquiring a human body shape characteristic vector and a human body angle characteristic vector to be warehoused; determining a class center human body image and a human body image to be put in storage for judgment; calculating the overall similarity of the human body image to be warehoused and judged and the class center human body image; and when the overall similarity meets an overall similarity threshold, classifying the human body image to be warehoused as the class of the selected class center human body image, and storing or deleting the human body image to be warehoused until all the human body images to be warehoused are traversed. The method for establishing the action posture library, the method for generating and identifying the action posture and the device thereof provided by the embodiment of the invention ensure the precision and the effect of the action posture of the human body.

Description

Method and device for establishing action posture library, generating action posture and identifying action posture

Technical Field

The embodiment of the invention relates to the field of computers, in particular to a method and a device for establishing an action attitude library, generating an action attitude and identifying the action attitude.

Background

With the development of computer vision technology, the application requirements for human body action gestures are more and more extensive, and therefore, the requirements for recognition and establishment of human body action gestures are higher and higher.

Such as: in the field of human-computer interaction, the human body action posture needs to be recognized to determine the actual posture of a human body, so that further machine feedback is carried out, and therefore enough human body posture data need to be established to ensure that actions can be recognized smoothly; in the field of motion generation, it is necessary to determine an initial motion and a final motion and generate a human body motion posture therebetween.

Therefore, a reasonable and effective modeling mode is needed to model the action posture of the human body so as to meet the requirements of various application scenes.

Therefore, how to ensure the accuracy and effect of human body action posture modeling becomes a technical problem which needs to be solved urgently.

Disclosure of Invention

The embodiment of the invention provides a method and a device for establishing an action posture library, generating an action posture and identifying the action posture so as to ensure the accuracy and the effect of human action posture modeling.

In order to solve the above problem, an embodiment of the present invention provides a method for establishing an action posture library, including:

taking a human body image to be put in a warehouse;

acquiring a shape characteristic vector and an angle characteristic vector of the human body to be warehoused according to the human body image to be warehoused;

determining a class center human body image used for calculating the overall similarity and a human body image to be warehoused in the human body image to be warehoused;

calculating the overall similarity of the human body image to be warehoused and the quasi-central human body image according to the human body shape characteristic vector to be warehoused, the quasi-central human body shape characteristic vector of the human body angle characteristic vector to be warehoused and the quasi-central human body image and the quasi-central human body angle characteristic vector;

and when the overall similarity meets an overall similarity threshold, classifying the human body image to be warehoused as the class of the selected class center human body image, storing or deleting the human body image to be warehoused, obtaining an action posture library after classifying the human body image to be warehoused until all the human body images to be warehoused are traversed.

In order to solve the above problem, an embodiment of the present invention further provides an action posture generating method, including:

acquiring a human body action initial frame and a human body action termination frame;

acquiring a starting human body shape characteristic vector and a starting human body angle characteristic vector of the human body action starting frame, and acquiring a terminating human body shape characteristic vector and a terminating human body angle characteristic vector of the human body action terminating frame;

acquiring various central human body shape characteristic vectors and various central human body angle characteristic vectors of various central human body images in the action attitude library established according to the establishment method of the action attitude library;

acquiring a nearest similar start similar center human body image of the human body action start frame according to the start human body shape feature vector, the start human body angle feature vector, each similar center human body shape feature vector and each type of center human body angle feature vector, and acquiring a nearest similar stop similar center human body image of the human body action stop frame according to the stop human body shape feature vector, the stop human body angle feature vector, each similar center human body shape feature vector and each type of center human body angle feature vector;

and acquiring the action posture of the quasi-center human body image arranged between the human body action initial frame and the human body action termination frame by utilizing an overall similarity shortest path algorithm according to the closest similar initial quasi-center human body image, the closest termination quasi-center human body image and the overall similarity matrix.

In order to solve the above problem, an embodiment of the present invention further provides an action gesture recognition method, including:

acquiring a human body image to be recognized;

acquiring a human body shape characteristic vector to be recognized and a human body angle characteristic vector to be recognized of the human body image to be recognized;

calculating the overall similarity of the human body image to be recognized and each class center human body image according to the human body shape feature vector to be recognized, the human body angle feature vector to be recognized, each class center human body shape feature vector and each class center human body angle feature vector;

and comparing the overall similarity with a similarity threshold to obtain a class center human body image of which the overall similarity meets the similarity threshold, and identifying the human body image to be identified as the class center human body image meeting the similarity threshold.

In order to solve the above problem, an embodiment of the present invention further provides an apparatus for establishing an action posture library, including:

the human body image acquisition unit is suitable for acquiring human body images to be warehoused;

the characteristic vector acquisition unit is suitable for acquiring the shape characteristic vector of the human body to be warehoused and the angle characteristic vector of the human body to be warehoused according to the human body image to be warehoused;

the to-be-warehoused calculation image determining unit is suitable for determining a class center human body image used for calculating the overall similarity and a to-be-warehoused judgment human body image in the to-be-warehoused human body image;

the integral similarity calculation unit is suitable for calculating the integral similarity of the human body image to be warehoused and judged and the quasi-central human body image according to the human body shape characteristic vector to be warehoused and judged, the human body angle characteristic vector to be warehoused and the quasi-central human body shape characteristic vector and the quasi-central human body angle characteristic vector of the quasi-central human body image;

and the classification action posture library obtaining unit is suitable for classifying the human body images to be warehoused as the class of the selected class center human body images or deleting the human body images to be warehoused when the overall similarity meets an overall similarity threshold value to obtain an action posture library after classifying the human body images to be warehoused until all the human body images to be warehoused are traversed to obtain a classification action posture library.

To solve the above problem, an embodiment of the present invention further provides an action posture generating device, including:

the to-be-generated human body image acquisition unit is suitable for acquiring a human body action initial frame and a human body action termination frame;

the to-be-generated characteristic vector acquisition unit is suitable for acquiring a starting human body shape characteristic vector and a starting human body angle characteristic vector of the human body action starting frame and acquiring a terminating human body shape characteristic vector and a terminating human body angle characteristic vector of the human body action terminating frame;

a to-be-generated center-like feature vector acquisition unit adapted to acquire various center human body shape feature vectors and various center human body angle feature vectors of each center-like human body image in the action attitude library established according to the establishment method of the action attitude library;

a quasi-center human body image obtaining unit, adapted to obtain a nearest similar start quasi-center human body image of the human body motion start frame according to the start human body shape feature vector, the start human body angle feature vector, each quasi-center human body shape feature vector and each type of center human body angle feature vector, and obtain a nearest similar stop quasi-center human body image of the human body motion stop frame according to the stop human body shape feature vector, the stop human body angle feature vector, each quasi-center human body shape feature vector and each type of center human body angle feature vector;

and the action posture acquisition unit is suitable for acquiring the action posture of the quasi-center human body image arranged between the human body action initial frame and the human body action termination frame by utilizing an overall similarity shortest path algorithm according to the closest similar initial quasi-center human body image, the closest termination quasi-center human body image and the overall similarity matrix.

In order to solve the above problem, an embodiment of the present invention further provides an action gesture recognition apparatus, including:

the human body image acquisition unit to be recognized is suitable for acquiring a human body image to be recognized;

the to-be-recognized characteristic vector acquisition unit is suitable for acquiring a to-be-recognized human body shape characteristic vector and a to-be-recognized human body angle characteristic vector of the to-be-recognized human body image;

the to-be-recognized center feature vector acquisition unit is suitable for acquiring various center human body shape feature vectors and various center human body angle feature vectors of various center human body images in the action attitude library established according to the establishment method of the action attitude library;

the overall similarity calculation unit is suitable for calculating the overall similarity of the human body image to be recognized and each class center human body image according to the human body shape feature vector to be recognized, the human body angle feature vector to be recognized, each class center human body shape feature vector and each class center human body angle feature vector;

and the identification unit is suitable for comparing the overall similarity with a similarity threshold value to obtain a class center human body image of which the overall similarity meets the similarity threshold value, and identifying the human body image to be identified as the class center human body image meeting the similarity threshold value.

In order to solve the above problem, an embodiment of the present invention further provides an electronic device, including at least one memory and at least one processor; the memory stores a program that the processor calls to execute the method of establishing the motion gesture library, or the method of generating the motion gesture, or the method of recognizing the motion gesture as described above.

In order to solve the above problem, an embodiment of the present invention further provides a storage medium, where a program suitable for video classroom interaction is stored, so as to implement the method for establishing the motion gesture library, the method for generating the motion gesture, or the method for recognizing the motion gesture.

Compared with the prior art, the technical scheme of the invention has the following advantages:

the method for establishing the action attitude library comprises the steps of firstly obtaining a human body shape characteristic vector to be warehoused and a human body angle characteristic vector to be warehoused based on a specific human body image to be warehoused, then calculating overall similarity based on the human body shape characteristic vector to be warehoused and the human body angle characteristic vector to be warehoused, and a class center human body shape characteristic vector and a class center human body angle characteristic vector of a class center human body image, comparing the relationship between the overall similarity and a similarity threshold, and classifying the human body image to be warehoused as the class of the class center human body image or deleting the human body image to be warehoused when the overall similarity meets the similarity threshold, so that the classified human body action attitude library can be obtained. It can be seen that the method for establishing an action posture library provided by the embodiment of the present invention processes an existing human body image to be warehoused, determines the overall similarity between the human body image to be warehoused and a selected class center human body image by using a human body shape feature vector to be warehoused and a human body angle feature vector to be warehoused, determines to classify the human body image to be warehoused as the class center human body image based on the relationship between the overall similarity and a similarity threshold, or directly deletes the human body image to be warehoused, on one hand, establishes the action posture library by using the existing human body image to be warehoused, improves the precision of the human body action posture to be warehoused in the action posture library, that is, ensures the precision of an action posture resource when the action posture is established, thereby improving the precision and effect of the generated action posture; on the other hand, in order to reduce the memory space and ensure the precision requirement of generating the action gesture, when an action gesture library is established, a large number of human body images to be warehoused are classified and analyzed, and when the classification and analysis are performed, the overall similarity between different human body images is determined by using the human body shape characteristic vector to be warehoused and the human body angle characteristic vector to be warehoused, the position state of a human body is used as a comparison factor, the action angle of the human body is also used as a part for determining the overall similarity, so that the determination accuracy is improved, after the classification is completed, the human body images to be warehoused, the overall similarity of which meets the threshold value requirement, and the class center human body images can be classified into one class for storage, preparation is made for the generation of subsequent action gestures and the re-classification based on the requirement, and can also be deleted, on the basis of ensuring the generation of subsequent action gestures, the storage space is reduced; furthermore, by classifying the obtained action posture library, the accuracy of the obtained action posture can be ensured, and the human body images meeting the similarity threshold are classified into one class, so that the calculation amount during the selection of the action posture is reduced, the multiple acquisition of the human body images with very close action postures can be avoided, the change of the acquired action postures is realized, and the variation amount is in an allowable range.

The action gesture generating method comprises the steps of firstly obtaining action gestures before and after generated action, namely a human action initial frame and a human action termination frame, respectively obtaining an initial human body shape feature vector, an initial human body angle feature vector, a termination human body shape feature vector and a termination human body angle feature vector based on the human action initial frame and the human action termination frame, further obtaining a nearest similar initial center human body image of the human action initial frame according to the initial human body shape feature vector, the initial human body angle feature vector, each similar center human body shape feature vector and each type of center human body angle feature vector, and obtaining the human action termination frame according to the termination human body shape feature vector, the termination human body angle feature vector, each similar center human body shape feature vector and each type of center human body angle feature vectors (ii) nearest similar termination class center body images; and finally, acquiring the action posture of the class center human body image arranged between the human body action initial frame and the human body action termination frame by utilizing an overall similarity shortest path algorithm according to the most approximate initial class center human body image, the most approximate termination class center human body image and an overall similarity matrix in the action posture library. It can be seen that the action gesture generating method provided in the embodiment of the present invention can obtain the quasi-center human body image that can be set between the human body action start frame and the human body action end frame through the overall similarity calculation on the basis of establishing the action gesture library according to the real human body image, on one hand, the calculation of the overall similarity is performed, and the human body position and the human body action angle are combined, so that the calculation accuracy is improved, and on the other hand, the minimum action transition of the human body image that is set between the human body action start frame and the human body action end frame is ensured by using the principle of the shortest path of the similarity.

The invention provides a method and a device for establishing an action posture library, generating an action posture and identifying the action posture, wherein the action posture identification method comprises the steps of firstly acquiring a human body image to be identified of the action posture to be identified; further obtaining the human body shape characteristic vector to be recognized and the human body angle characteristic vector to be recognized of the human body image to be recognized, and various central human body shape characteristic vectors and various central human body angle characteristic vectors of various central human body images in the action posture library established according to the establishing method of the action posture library, and then according to the shape characteristic vector of the human body to be recognized, the angle characteristic vector of the human body to be recognized, the shape characteristic vectors of various central human bodies and the angle characteristic vectors of various central human bodies, calculating the overall similarity between the human body image to be recognized and each class-center human body image, comparing the overall similarity with a similarity threshold value to obtain a class-center human body image with the overall similarity meeting the similarity threshold value, and recognizing the human body image to be recognized as the class-center human body image meeting the similarity threshold value. In this way, the class center human body image meeting the similarity threshold is determined by calculating the overall similarity between the human body image to be recognized and each class center human body image in the action posture library, the human body image to be recognized is the class center human body image meeting the similarity threshold, and when the overall similarity is calculated, the human body shape characteristic vector and the human body angle characteristic vector are simultaneously utilized, so that the accuracy of similarity calculation is ensured, and the accuracy of recognition is further ensured; meanwhile, by using the established action attitude library, the calculation amount can be reduced, the recognition time can be shortened, and the requirements on equipment can be reduced on the basis of ensuring the recognition accuracy.

Drawings

Fig. 1 is a schematic flow chart of a method for establishing an action posture library according to an embodiment of the present invention;

fig. 2 is a schematic flow chart of a method for establishing an action posture library according to an embodiment of the present invention, in which a shape feature vector of a human body to be warehoused is acquired according to the human body image to be warehoused;

fig. 3 is a schematic flow chart of the method for establishing the motion posture library according to the embodiment of the present invention, in which the angle feature vector of the human body to be warehoused is acquired according to the human body image to be warehoused;

fig. 4 is a schematic flow chart illustrating a method for establishing an action posture library according to an embodiment of the present invention, in which the overall similarity between the human body image to be put into a storage and judged and the class center human body image is calculated;

FIG. 5 is a flowchart illustrating a method for generating motion gestures according to an embodiment of the present invention;

FIG. 6 is a flowchart illustrating a method for recognizing motion gestures according to an embodiment of the present invention;

FIG. 7 is a block diagram of an apparatus for creating a motion gesture library according to an embodiment of the present invention;

FIG. 8 is a block diagram of an action gesture generating apparatus according to an embodiment of the present invention;

FIG. 9 is a block diagram of an apparatus for recognizing motion gestures according to an embodiment of the present invention;

fig. 10 shows an alternative hardware electronic device architecture of the electronic device provided in the embodiment of the present invention.

Detailed Description

In the prior art, the accuracy and the effect of human body posture modeling are poor, or the cost is high.

In the two-dimensional human body motion posture generation method, the coordinates of the skeleton points in the human body image are used for modeling, and the established motion posture has low precision and poor effect.

In order to solve the above problem, a method for generating an action gesture based on three-dimensional information is provided, such as: the three-dimensional laser scanner is used for directly and quickly acquiring the high-precision three-dimensional point cloud data of the target object, but the equipment has the disadvantages of very complex structure, very high price, very high requirement on the environment, huge mention, unsuitability for popularization and large-scale use, huge three-dimensional point cloud data and high requirement on data transmission and storage.

In order to ensure the precision and effect of human body action attitude modeling under the condition of meeting the requirement of lower cost, the embodiment of the invention provides a method and a device for establishing an action attitude library, generating an action attitude and identifying the action attitude, wherein the method for establishing the action attitude library is used for acquiring a human body image to be put in a warehouse; acquiring a shape characteristic vector and an angle characteristic vector of the human body to be warehoused according to the human body image to be warehoused; determining a class center human body image for calculating similarity; calculating the overall similarity of the human body image to be warehoused and the quasi-central human body image according to the human body shape characteristic vector to be warehoused, the human body angle characteristic vector to be warehoused and the quasi-central human body shape characteristic vector and the quasi-central human body angle characteristic vector of the quasi-central human body image; and when the overall similarity meets an overall similarity threshold, classifying the human body image to be warehoused as the class of the selected class center human body image, storing or deleting the human body image to be warehoused, and obtaining a classified action posture library.

The embodiment of the invention provides a method for establishing an action attitude library, which comprises the steps of firstly obtaining a human body shape characteristic vector to be warehoused and a human body angle characteristic vector to be warehoused based on a specific human body image to be warehoused, then calculating the overall similarity based on the human body shape characteristic vector to be warehoused and the human body angle characteristic vector to be warehoused, the class center human body shape characteristic vector and the class center human body angle characteristic vector of the class center human body image, comparing the overall similarity with a similarity threshold, and classifying the human body image to be warehoused as the class of the class center human body image or deleting the human body image to be warehoused when the overall similarity meets the similarity threshold, so that the classified human body action attitude library can be obtained.

It can be seen that the method for establishing an action posture library provided by the embodiment of the present invention processes an existing human body image to be warehoused, determines the overall similarity between the human body image to be warehoused and a selected class center human body image by using a human body shape feature vector to be warehoused and a human body angle feature vector to be warehoused, determines to classify the human body image to be warehoused as the class center human body image based on the relationship between the overall similarity and a similarity threshold, or directly deletes the human body image to be warehoused, on one hand, establishes the action posture library by using the existing human body image to be warehoused, improves the precision of the human body action posture to be warehoused in the action posture library, that is, ensures the precision of an action posture resource when the action posture is established, thereby improving the precision and effect of the generated action posture; on the other hand, in order to reduce the memory space and ensure the precision requirement of generating the action gesture, when an action gesture library is established, a large number of human body images to be warehoused are classified and analyzed, and when the classification and analysis are performed, the overall similarity between different human body images is determined by using the human body shape characteristic vector to be warehoused and the human body angle characteristic vector to be warehoused, the position state of a human body is used as a comparison factor, the action angle of the human body is also used as a part for determining the overall similarity, so that the determination accuracy is improved, after the classification is completed, the human body images to be warehoused, the overall similarity of which meets the threshold value requirement, and the class center human body images can be classified into one class for storage, preparation is made for the generation of subsequent action gestures and the re-classification based on the requirement, and can also be deleted, on the basis of ensuring the generation of subsequent action gestures, the storage space is reduced; furthermore, by classifying the obtained action posture library, the accuracy of the obtained action posture can be ensured, and the human body images meeting the similarity threshold are classified into one class, so that the calculation amount during the selection of the action posture is reduced, the multiple acquisition of the human body images with very close action postures can be avoided, the change of the acquired action postures is realized, and the variation amount is in an allowable range.

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

Referring to fig. 1, fig. 1 is a flowchart illustrating a method for establishing an action posture library according to an embodiment of the present invention.

As shown in the figure, the method for establishing the action posture library provided by the embodiment of the invention comprises the following steps:

step S10: and acquiring a human body image to be put in storage.

It can be understood that the establishment of the action attitude library of the invention can be a brand-new action attitude library from scratch, or can be further expanded and perfected on the basis of the established action attitude library to obtain a more comprehensive action attitude library.

In any case, in order to establish the motion posture library, it is necessary to first acquire the to-be-warehoused human body image required for establishing the motion posture library, and it is easy to understand that the to-be-warehoused human body image includes an image of the whole body of the human body.

It can be understood that, in order to establish the action posture library, 1 or more to-be-warehoused human body images are acquired, and certainly, under the condition of establishing a completely new action posture library, a plurality of to-be-warehoused human body images need to be acquired, while the existing action posture library is perfected and expanded, only 1 to-be-warehoused human body image can be acquired, and when a plurality of to-be-warehoused human body images are acquired simultaneously, the plurality of to-be-warehoused human body images can be from the whole human body image of the same person or from the whole human body images of different persons. However, in the process of establishing the action posture library, even if a plurality of human body images to be warehoused are obtained, a single human body image is classified, judged and classified until the classification of the plurality of human body images to be warehoused is completed.

In a specific embodiment, a common camera may be directly used to capture a human body to obtain an image of the human body to be put in storage, but in other embodiments, other methods may also be used, such as: and acquiring the human body image to be put in storage by means of video disassembly and the like.

Step S11: and acquiring the shape characteristic vector of the human body to be warehoused and the angle characteristic vector of the human body to be warehoused according to the human body image to be warehoused.

When the action posture of a person changes, the shape of the human body changes along with the action posture, so that the shape of the human body can be used for marking the action posture of the person, and on the other hand, when the action posture of the person changes, an included angle between a connecting line between different parts of the human body and a reference direction also changes, so that the angle of the human body can be used for marking the action posture of the person.

In order to improve the accuracy of grasping the action posture of the human body, in the embodiment provided by the invention, the feature vectors of the human body shape and the human body angle which can indicate the action posture of the human body are simultaneously obtained, and specifically, after the human body image to be warehoused is obtained, the human body shape feature vector to be warehoused and the human body angle feature vector to be warehoused are extracted according to the human body image to be warehoused.

Referring to fig. 2, fig. 2 is a schematic flow chart illustrating a method for establishing an action posture library according to an embodiment of the present invention, where the method obtains a shape feature vector of a human body to be warehoused according to the human body image to be warehoused.

As shown in the figure, in the method for establishing an action posture library according to the embodiment of the present invention, the step of obtaining the shape feature vector of the human body to be put in storage according to the human body image to be put in storage may include:

step S1101: and acquiring a human body mask image to be warehoused of the human body image to be warehoused according to the human body image to be warehoused.

And acquiring a human body mask image to be warehoused from the human body image to be warehoused by using a human body semantic image segmentation algorithm.

Specifically, a human face mask image to be binned, including eyes, a nose, a mouth, eyebrows, ears and the like, may be acquired from a human body image to be binned by using a human face semantic image segmentation algorithm, and a hand mask image to be binned and a body mask image to be binned are acquired from the human body image to be binned by using a body semantic image segmentation algorithm, where the body mask image to be binned may include a trunk, legs and the like.

In a specific embodiment, when the number of the human body images to be put in storage is large, in order to facilitate subsequent processing, different parts of the human body mask image to be put in storage may be marked with colors to obtain a pixel-marked human body mask image, and the same part of different human body mask images to be put in storage is marked with the same color, so that in the subsequent processing, different parts may be distinguished by color blocks.

Such as: different parts of the human face mask image to be warehoused are marked by pixels with colors, different parts of the human face in the same human face mask image to be warehoused are marked by different colors, the same parts of the human face in different human face mask images to be warehoused are marked by the same colors, and different parts of the body mask image to be warehoused are marked by the pixels with colors different from the colors marked by the human face mask image to be warehoused.

The acquisition of the mask image of the human body to be put in storage can eliminate the influence of the difference of the skin color, illumination, scene and the like of different people on the subsequent processing.

Step S1102: and acquiring the shape characteristic vector of the human body to be warehoused according to the mask image of the human body to be warehoused.

The method comprises the steps of obtaining a human body mask image to be warehoused, and then extracting a human body shape feature vector to be warehoused from the human body mask image to be warehoused.

It is understood that the histogram feature vector of human face directional gradient, the histogram feature vector of hand directional gradient and the histogram feature vector of body directional gradient may be specifically included.

To obtain the shape feature vector of the human body to be put in storage, and to ensure accuracy, it is further required to obtain the angle feature vector of the human body to be put in storage, please refer to fig. 3, where fig. 3 is a schematic flow diagram of the method for establishing the action posture library according to the human body image to be put in storage provided by the embodiment of the present invention.

As shown in the figure, in the method for establishing an action posture library according to the embodiment of the present invention, the step of obtaining an angle feature vector of a human body to be warehoused according to an image of the human body to be warehoused may include:

step S1103: and acquiring human body skeleton points to be warehoused of the human body image to be warehoused by using a skeleton point detection model.

In order to obtain the angle feature vector of the human body to be put in storage, the human body bone points to be put in storage are obtained first, and specifically, a bone point detection model can be adopted for obtaining, for example: openpos et al, specifically, skeletal points may include:

body part skeletal points such as: cervical, abdominal, shoulder, elbow, wrist, crotch, knee, ankle, footstep, etc.

Hand bone points may include: individual finger tip skeletal points, finger middle skeletal points, finger to palm joint skeletal points, palm skeletal points, and the like.

The face skeleton points may include: ocular, otic, labial, nasal, etc.

Of course, the number and the positions of the skeleton points can be set according to the needs, and in a specific embodiment, 68 human face skeleton points, 42 hand skeleton points and 18 body skeleton points can be set.

Step S1104: and acquiring a human body bone point line segment to be warehoused according to the human body bone point to be warehoused.

And after obtaining the human skeleton points to be warehoused, establishing human skeleton point line segments to be warehoused according to the skeleton connection relation of different parts, and forming the line segments of the human skeleton to be warehoused.

It can be understood that the segment of the human bone point to be put in storage may be formed along the human bone, such as: the wrist skeleton point and the elbow skeleton point are connected to form a forearm skeleton point line segment which is closer to the actual human body action, so that the actual human body action of the person in the image can be obtained more conveniently.

Step S1105: and acquiring the angle characteristic vector of the human body to be warehoused according to the included angle between the line segment of the human body bone point to be warehoused and the reference direction.

After the bone point line segment of the human body to be warehoused is obtained, the included angle between the bone point line segment of the human body to be warehoused and the reference direction can be obtained, the included angle between the bone point line segment of the human body to be warehoused and the reference direction forms a vector, and the angle characteristic vector of the human body to be warehoused is obtained.

In the embodiment of dividing a human body into a body part, a face part and a hand part, obtaining a body angle feature vector to be warehoused based on a skeleton point line segment of the body to be warehoused; obtaining a human face angle feature vector to be put in storage based on the human face skeleton point line segment; and obtaining a hand angle feature vector to be warehoused based on the hand skeleton point line segment.

In order to obtain the above angular feature vector, in one embodiment, a horizontal direction may be selected as the reference direction, and further, a horizontal right direction or a horizontal left direction may be selected as the reference direction. In other embodiments, other directions may be selected as reference directions, such as: the vertical direction.

Of course, the order of obtaining the angle feature vector of the human body to be put in storage and obtaining the shape feature vector of the human body to be put in storage is not limited, and the angle feature vector of the human body to be put in storage may be obtained first, the shape feature vector of the human body to be put in storage may be obtained first, or both may be obtained simultaneously.

After obtaining the angle feature vector and the shape feature vector of the human body to be put in storage, please continue to refer to fig. 1 for determining whether the human body image to be put in storage can become a part of the motion posture library or what manner is called as a part of the motion posture library.

Step S12: and determining a class center human body image used for calculating the overall similarity and a human body image to be warehoused in the human body image to be warehoused.

It can be understood that, when the method for establishing the motion posture library provided by the present invention is applied to establishing a completely new motion posture library from scratch, after the shape feature vector of the human body to be warehoused and the angle feature vector of the human body to be warehoused of each human body image to be warehoused are obtained, one of the human body to be warehoused and the shape feature vector of the human body to be warehoused are selected at will to be determined as a class center human body image for calculating the overall similarity, and the other is selected as a human body image to be warehoused and judged for further judgment.

When the action posture library is further improved and expanded on the basis of the established action posture library, a human body image which is determined as a class center human body image is selected from the action posture library as a determined class center human body image used for calculating the overall similarity, and one human body image is selected from the human body images to be stored as a human body image to be judged for subsequent processing.

Step S13: and calculating the overall similarity of the human body image to be warehoused and the quasi-center human body image according to the human body shape characteristic vector to be warehoused, the human body angle characteristic vector to be warehoused and the quasi-center human body shape characteristic vector and the quasi-center human body angle characteristic vector of the quasi-center human body image.

After a class center human body image and a human body image to be warehoused are determined for calculating the overall similarity, a class center human body shape characteristic vector and a class center human body angle characteristic vector of the class center human body image are obtained, a human body shape characteristic vector to be warehoused and judged and a human body angle characteristic vector to be warehoused of the human body image to be warehoused and judged are obtained, and then the overall similarity of the human body image to be warehoused and judged and the class center human body image is calculated according to the human body shape characteristic vector to be warehoused and judged, the human body angle characteristic vector to be warehoused and judged, the class center human body shape characteristic vector and the class center human body angle characteristic vector.

Specifically, referring to fig. 4, fig. 4 is a schematic flow chart illustrating a process of calculating the overall similarity between the human body image to be put into a storage and the center-like human body image according to the method for establishing the motion posture library provided in the embodiment of the present invention.

As shown in the figure, the overall similarity between the human body image to be put in storage and the class center human body image can be obtained through the following steps.

Step S131: and acquiring the human body shape similarity according to the human body shape characteristic vector to be warehoused and judged and the quasi-central human body shape characteristic vector.

Based on the foregoing content, the to-be-warehoused human body shape feature vector includes a to-be-warehoused human body shape feature vector, a to-be-warehoused hand shape feature vector, and a to-be-warehoused trunk shape feature vector, and in consideration of different degrees of influence of different parts on the action posture, in order to further improve the accuracy of the calculated human body shape similarity, in a specific implementation manner, the human body shape similarity, the hand shape similarity, and the trunk shape similarity may be respectively obtained, and then the human body shape similarity is obtained based on the human body shape similarity, the hand shape similarity, and the trunk shape similarity.

Specifically, the face shape similarity is obtained according to the face shape feature vector to be warehoused and judged and the class center face shape feature vector, the hand shape similarity is obtained according to the hand shape feature vector to be warehoused and judged and the class center hand shape feature vector, and the trunk shape similarity is obtained according to the body shape feature vector to be warehoused and judged and the class center body shape feature vector.

In one embodiment, after obtaining the human shape similarity by obtaining the human face shape similarity, the hand shape similarity and the torso shape similarity, the human shape similarity may be obtained by using the following formula:

wherein λ is^f-face shape similarity weight;

λ^h-a hand shape similarity weight;

λ^b-a body shape similarity weight;

-face shape similarity;

-hand shape similarity;

-body shape similarity.

It can be seen that when the human body shape similarity is obtained through the above formula, different weights are respectively added based on the influence degree of the human face shape similarity, the hand shape similarity and the trunk shape similarity on the human body shape similarity, so that the accuracy of the obtained human body shape similarity is improved.

In a specific embodiment, the human body shape similarity may be a chi-square distance between the human body shape feature vector to be put in storage and the central-like human body shape feature vector, and in this case, the human face shape similarity is a chi-square distance between the human face shape feature vector to be put in storage and the central-like human face shape feature vector, the hand shape similarity is a chi-square distance between the hand shape feature vector to be put in storage and the central-like hand shape feature vector, and the trunk shape similarity is a chi-square distance between the body shape feature vector to be put in storage and the central-like body shape feature vector.

Step S132: and obtaining human body angle similarity according to the human body angle characteristic vector to be warehoused and judged and the class center human body angle characteristic vector.

Based on the foregoing content, the to-be-warehoused judged human body angle feature vector includes a to-be-warehoused judged human face angle feature vector, a to-be-warehoused judged hand angle feature vector, and a to-be-warehoused judged trunk angle feature vector, and meanwhile, in order to further improve the accuracy of the calculated human body angle similarity, in a specific implementation manner, the human face angle similarity, the hand angle similarity, and the trunk angle similarity may be respectively obtained, and then the human body angle similarity is obtained based on the human face angle similarity, the hand angle similarity, and the trunk angle similarity.

Specifically, face angle similarity is obtained according to the face angle feature vector to be warehoused and judged and the class center face angle feature vector, hand angle similarity is obtained according to the hand angle feature vector to be warehoused and judged and the class center hand angle feature vector, and trunk angle similarity is obtained according to the body angle feature vector to be warehoused and judged and the class center body angle feature vector.

In a specific embodiment, after obtaining the human face angle similarity, the hand angle similarity and the trunk angle similarity to obtain the human body angle similarity, the human body angle similarity may be obtained by using the following formula:

wherein λ is^f-face angle similarity weight;

λ^h-hand angle similarity weight;

λ^b-a body angle similarity weight;

-face angle similarity;

-hand angle similarity;

-body angle similarity.

It can be seen that when the human body angle similarity is obtained through the formula, different weights are respectively added based on the influence degree of the human face angle similarity, the hand angle similarity and the trunk angle similarity on the human body angle similarity, so that the accuracy of the obtained human body angle similarity is improved.

In a specific embodiment, the human angle similarity may be the euclidean distance between the human angle feature vector to be stored and the class center human angle feature vector, in which case, the human angle similarity is the euclidean distance between the human angle feature vector to be stored and the class center human angle feature vector, the hand angle similarity is the euclidean distance between the hand angle feature vector to be stored and the class center hand angle feature vector, and the trunk angle similarity is the euclidean distance between the body angle feature vector to be stored and the class center body angle feature vector.

Wherein: the euclidean distance is the most common representation of the distance between two points or between multiple points, also known as the euclidean metric, defined in euclidean space. Euclidean distance between two points x1(x11, x12, …, x1n) and x2(x21, x22, …, x2n) in n-dimensional space:

here, the points x1(x11, x12, …, x1n) and x2(x21, x22, …, and x2n) are included angle vectors (representing human postures) of two image frames, and x11, x12, …, x1n, x21, x22, …, and x2n are included angles between a plurality of bone point segments and the reference direction, respectively.

Step S133: and acquiring the overall similarity according to the human body shape similarity and the human body angle similarity.

And after the human body shape similarity and the human body angle similarity are obtained, the overall similarity is further obtained.

Specifically, the overall similarity may be obtained by using the following formula:

D＝(αD_app+βD_ang)^γ

wherein D is_app-human shape similarity;

D_ang-human angle similarity;

α — weight of human shape similarity;

beta-weight of human angle similarity;

γ — weight of the weighted sum.

Based on the different influences of the human body shape similarity and the human body angle similarity on the overall similarity, corresponding weights are added respectively, the accuracy of the overall similarity is improved, and therefore the accuracy of classification in the process of establishing the action posture library in the follow-up process can be guaranteed.

Step S14: judging whether the overall similarity meets an overall similarity threshold value; if so, step S15 is performed, and if not, step S17 is performed.

And determining an overall similarity threshold value based on needs, comparing the overall similarity obtained through the calculation with the overall similarity threshold value, executing the step S15 if the overall similarity threshold value is met, otherwise executing the step S17, and judging whether the traversal of various central human body images is finished.

Step S15: and classifying the human body image to be warehoused as the class of the class center human body image, and storing or deleting the human body image to be warehoused.

When the overall similarity between the human body image to be warehoused and the determined class center human body image meets an overall similarity threshold, the human body image to be warehoused and judged and the class center human body image can be classified into one class, so that the human body image to be warehoused and judged and the class center human body image can be conveniently searched when needed, and meanwhile, materials can be provided for the follow-up human body image reclassification based on the established action posture library; certainly, in order to reduce the occupied storage space, when the overall similarity between the human body image to be warehoused and the determined class center human body image meets the overall similarity threshold, the human body image to be warehoused and judged can also be deleted.

Step S16: and obtaining an action attitude library after classifying the human body image to be warehoused.

When the overall similarity between the human body image to be warehoused and the determined class center human body image meets an overall similarity threshold, the human body image to be warehoused is classified as the class of the class center human body image to be stored or the human body image to be warehoused is deleted, and an action posture library after the human body image to be warehoused is classified can be obtained.

Of course, the overall similarity between the human body image to be put in storage and judged and the determined class center human body image may or may not meet the overall similarity threshold, and when the overall similarity threshold is not met, further processing can be performed.

And step S17, judging whether the traversal of various central human body images is finished, if so, executing step S18, and if not, executing step S19.

And when the overall similarity between the human body image to be warehoused and the determined class center human body image does not meet the overall similarity threshold, further judging whether the overall similarity between the human body image to be warehoused and judged and all the class center human body images is calculated and compared with the overall similarity threshold.

When the action posture library is a newly constructed action posture library, the central human body images are all human body images to be warehoused except the human body images to be warehoused and judged; and when the action posture library is the constructed action posture library, the various central human body images are various central human body images in the existing action posture library.

And step S18, taking the human body image to be warehoused as a new center-like human body image, and executing step S16.

And if the traversal of various central human body images is finished, the fact that the overall similarity of the human body image to be warehoused and judged and other class-center human body images does not meet the requirement of a similarity threshold is proved, the human body image to be warehoused and judged is used as a new class-center human body image to form a new class, and therefore the action posture library after the human body image to be warehoused and judged is classified is obtained.

Step S19: and replacing the quasi-center human body image, taking the replaced quasi-center human body image as a quasi-center human body image for calculating the similarity, and turning to execute the step S13.

If the traversal of various central human body images is not completed, it is indicated that other similar central human body images are not subjected to overall similarity calculation with the human body image to be warehoused, the similar central human body image is replaced, the replaced similar central human body image is used as the similar central human body image for calculating the similarity, and calculation and judgment are performed again until the motion attitude library after the human body image to be warehoused is classified is obtained.

Step S110: and judging whether the acquired human body images to be put in storage are classified, if not, executing the step S111, and if so, executing the step S112.

Because there may be a plurality of acquired human body images to be warehoused, it is further necessary to determine whether all the human body images to be warehoused have been classified, if so, the construction of the action posture library is stopped, and if there are human body images to be warehoused that have not been classified, step S111 is executed.

Step S111: and replacing the human body image to be put in storage and judging, and turning to execute the step S13.

And if the obtained human body image to be warehoused is not completely classified, replacing the human body image to be warehoused and judged, then calculating and comparing the overall similarity again, finishing the classification of the replaced human body image to be warehoused and judged, and knowing that all the human body images to be warehoused are completely classified.

Step S112: and establishing an overall similarity matrix between any two of all the class center human body images in the classified action posture library to obtain the classified action posture library.

In order to facilitate the subsequent generation of the action postures, in a specific embodiment, an overall similarity matrix of overall similarities between any two of all class-center human body images in the classified action posture library can be further established, so that the action posture library comprising each human body image, the human body shape feature vector of each human body image, the human body angle feature vector and the overall similarity matrix is obtained.

The establishment of the overall similarity matrix can facilitate the selection and the establishment of the action posture between two frames of the video, and improve the precision of the obtained action posture.

It can be seen that the method for establishing an action posture library provided in the embodiments of the present invention processes a human body image to be put in storage, sequentially determines the overall similarity between the human body image to be put in storage and a selected class-center human body image by using the human body shape feature vector to be put in storage and the human body angle feature vector to be put in storage of each human body image to be put in storage in the human body image to be judged, determines to classify the human body image to be put in storage as the class-center human body image based on the relationship between the overall similarity and the similarity threshold, or directly deletes the human body image to be put in storage or takes the human body image to be put in storage as a new class-center human body image until the classification of all the human body images to be put in storage is completed, and establishes the action posture library. On one hand, the motion attitude library is established by utilizing the existing human body image, so that the precision of the motion attitude in the motion attitude library is improved, namely the precision of motion attitude resources when the motion attitude is established is ensured, and the precision and the effect of the generated motion attitude can be improved; on the other hand, in order to reduce the memory space and ensure the precision requirement of generating the action gesture, when an action gesture library is established, a large number of human body images to be warehoused are classified and analyzed, the human body shape characteristic vector and the human body angle characteristic vector are utilized to determine the overall similarity between different human body images while the classification and analysis are carried out, the position state of the human body is taken as a comparison factor, the action angle of the human body is also taken as a part for determining the overall similarity, the determination accuracy is improved, after the classification is finished, the human body images to be warehoused, which meet the threshold requirement of the overall similarity, and class center human body images can be classified and stored, preparation is made for the subsequent generation of the action gesture and the required re-classification, and the human body images can be deleted on the basis of ensuring the subsequent generation of the action gesture, the storage space is reduced; furthermore, by classifying the obtained action posture library, the accuracy of the obtained action posture can be ensured, and the human body images meeting the similarity threshold are classified into one class, so that the calculation amount during the selection of the action posture is reduced, the multiple acquisition of the human body images with very close action postures can be avoided, the change of the acquired action postures is realized, and the variation amount is in an allowable range.

In order to ensure the accuracy requirement of human body motion gesture generation, an embodiment of the present invention further provides a motion gesture generation method, please refer to fig. 5, and fig. 5 is a flow diagram of the motion gesture generation method provided in the embodiment of the present invention.

As shown in the drawings, the motion gesture generation method provided by the embodiment of the present invention includes the following steps:

step S20: and acquiring a human body action initial frame and a human body action termination frame.

In order to generate the human body motion gesture, a start motion gesture and an end motion gesture which need to be connected are firstly acquired, and a human body motion start frame and a human body motion end frame are acquired for the purpose.

Specifically, the human motion start frame and the human motion end frame may be from the same piece of video data or from different motion videos.

Step S21: and acquiring a starting human body shape characteristic vector and a starting human body angle characteristic vector of the human body action starting frame, and acquiring a terminating human body shape characteristic vector and a terminating human body angle characteristic vector of the human body action terminating frame.

After a human body action initial frame and a human body action termination frame are obtained, an initial human body shape characteristic vector and an initial human body angle characteristic vector of the human body action initial frame and an end human body shape characteristic vector and an end human body angle characteristic vector of the human body action termination frame are respectively obtained.

Specifically, to obtain the starting body shape feature vector and the terminating body shape feature vector, a starting body mask image may be obtained according to the body motion starting frame, and a terminating body mask image may be obtained according to the body motion terminating frame; then, a starting human body shape vector is extracted from the starting human body mask image, and a terminating human body shape vector is extracted from the terminating mask image, wherein the human body shape vector can also be a gradient histogram feature vector.

For obtaining the initial human body angle characteristic vector and the termination human body angle characteristic vector, a bone point detection model is used for obtaining an initial human body bone point of an initial human body action frame and a termination human body bone point of a termination human body action frame, then a segment of the initial human body bone point is obtained according to the initial human body bone point, a segment of the termination human body bone point is obtained according to the termination human body bone point, finally the initial human body angle characteristic vector is obtained according to an included angle between the segment of the initial human body bone point and a reference direction, and the termination human body angle characteristic vector is obtained according to an included angle between the segment of the termination human body bone point and the reference direction.

It can be understood that, in order to combine the influence of different parts on the action posture, the starting human body shape feature vector may include a starting human face shape feature vector, a starting hand shape feature vector and a starting trunk shape feature vector; terminating the human shape feature vector may include terminating a human shape feature vector, terminating a hand shape feature vector, and terminating a torso shape feature vector.

And step S22, acquiring various central human body shape characteristic vectors and various central human body angle characteristic vectors of various central human body images in the action posture library established according to the establishment method of the action posture library.

And acquiring the shape characteristic vector of the quasi-central human body and the angle characteristic vector of the quasi-central human body from the established action posture library to prepare for acquiring the action posture subsequently.

Step S23: and obtaining a nearest similar initial similar central human body image of the action initial frame according to the initial human body shape characteristic vector, the initial human body angle characteristic vector, each similar central human body shape characteristic vector and each type of central human body angle characteristic vector, and obtaining a nearest similar termination similar central human body image of the human body action termination frame according to the termination human body shape characteristic vector, the termination human body angle characteristic vector, each similar central human body shape characteristic vector and each type of central human body angle characteristic vector.

According to the method for calculating the overall similarity, the overall similarity of the human body action initial frame and various central human body shape characteristic vectors in the action posture library is respectively calculated, and a class center human body image with the maximum overall similarity with the human body action initial frame, namely the class center human body image closest to the initial class center is obtained; and respectively calculating the overall similarity of the human body action termination frame and the shape feature vectors of various central human bodies in the action attitude library to obtain a class center human body image with the maximum overall similarity with the human body action termination frame, namely the class center human body image closest to the termination class center.

Step S24: and acquiring the action posture of the quasi-center human body image arranged between the human body action initial frame and the human body action termination frame by utilizing an overall similarity shortest path algorithm according to the closest similar initial quasi-center human body image, the closest termination quasi-center human body image and the overall similarity matrix.

The overall similarity matrix stores the overall similarity between any two of all class center human body images in the established action posture library, so that after the closest similar starting class center human body image and the closest similar ending class center human body image are obtained, the action posture of the class center human body image arranged between the closest similar starting class center human body image and the closest ending class center human body image can be obtained by utilizing an overall similarity shortest path algorithm according to the overall similarity matrix in the action posture library, namely the action posture of the class center human body image arranged between a human action starting frame and a human action ending frame.

It can be understood that the shortest overall similarity path means that the sum of the overall similarities is the largest from the human body motion starting frame, the quasi-center human body image arranged between the human body motion starting frame and the human body motion ending frame to the human body motion ending frame.

It can be seen that the action gesture generating method provided in the embodiment of the present invention can obtain the quasi-center human body image that can be set between the human body action start frame and the human body action end frame through the overall similarity calculation on the basis of establishing the action gesture library according to the real human body image, on one hand, the calculation of the overall similarity is performed, and the human body position and the human body action angle are combined, so that the calculation accuracy is improved, and on the other hand, the minimum action transition of the human body image that is set between the human body action start frame and the human body action end frame is ensured by using the principle of the shortest path of the similarity.

Of course, based on different application scenarios, an embodiment of the present invention further provides an action and gesture recognition method, please refer to fig. 6, where fig. 6 is a schematic flow diagram of the action and gesture recognition method provided in the embodiment of the present invention, and it can be seen that the action and gesture recognition method provided in the embodiment of the present invention includes the following steps:

step S30: and acquiring a human body image to be recognized.

In a specific embodiment, the image of the human body to be recognized may be obtained by using a camera to shoot, and of course, the image of the human body to be recognized may also be obtained from an existing image library.

Step S31: and acquiring the shape characteristic vector and the angle characteristic vector of the human body to be recognized of the human body image to be recognized.

And after obtaining the human body image to be recognized, obtaining the human body shape characteristic vector to be recognized and the human body angle characteristic vector to be recognized.

In a specific implementation manner, the human body mask image to be recognized may be first obtained according to the human body image to be recognized, and then the human body shape vector to be recognized may be extracted from the human body mask image to be recognized, where the human body shape vector to be recognized may also be a gradient histogram feature vector.

In order to obtain the human body angle feature vector to be recognized, firstly, the human body bone point to be recognized of the action frame to be recognized is obtained by using a bone point detection model, then, the human body bone point line segment to be recognized is obtained according to the human body bone point to be recognized, and finally, the human body angle feature vector to be recognized is obtained according to the included angle between the human body bone point line segment to be recognized and the reference direction.

It can be understood that, in order to combine the influence of different parts on the action posture, the human body shape feature vector to be recognized may include a human face shape feature vector to be recognized, a hand shape feature vector to be recognized, and a trunk shape feature vector to be recognized.

Step S32: and acquiring various central human body shape characteristic vectors and various central human body angle characteristic vectors of various central human body images in the action attitude library established according to the establishment method of the action attitude library.

And acquiring the shape characteristic vector of the quasi-central human body and the angle characteristic vector of the quasi-central human body from the established action posture library to prepare for subsequent action posture identification.

Step S33: and calculating the overall similarity of the human body image to be recognized and each class center human body image according to the human body shape characteristic vector to be recognized, the human body angle characteristic vector to be recognized, each class center human body shape characteristic vector and each class center human body angle characteristic vector.

And respectively calculating the overall similarity of the human body image to be recognized and the shape characteristic vectors of various central human bodies in the action posture library according to the method for calculating the overall similarity.

Step S34: and comparing the overall similarity with a similarity threshold to obtain a class center human body image of which the overall similarity meets the similarity threshold, and identifying the human body image to be identified as the class center human body image meeting the similarity threshold.

In this way, the class center human body image meeting the similarity threshold is determined by calculating the overall similarity between the human body image to be recognized and each class center human body image in the action posture library, the human body image to be recognized is the class center human body image meeting the similarity threshold, and when the overall similarity is calculated, the human body shape characteristic vector and the human body angle characteristic vector are simultaneously utilized, so that the accuracy of similarity calculation is ensured, and the accuracy of recognition is further ensured; meanwhile, by using the established action attitude library, the calculation amount can be reduced, the recognition time can be shortened, and the requirements on equipment can be reduced on the basis of ensuring the recognition accuracy.

In the following, the device for establishing an action and gesture library according to the embodiment of the present invention is introduced, and the device for establishing an action and gesture library described below may be regarded as a functional module architecture that is required to be set by an electronic device (e.g., a PC) to implement the method for establishing an action and gesture library according to the embodiment of the present invention. The contents of the device for establishing the motion gesture library described below may be referred to in correspondence with the contents of the method for establishing the motion gesture library described above.

Fig. 7 is a block diagram of an apparatus for establishing an action-gesture library according to an embodiment of the present invention, where the apparatus for establishing an action-gesture library is applied to a client or a server, and referring to fig. 7, the apparatus for establishing an action-gesture library may include:

the human body image acquisition unit 100 is suitable for acquiring human body images to be warehoused;

the to-be-warehoused characteristic vector acquisition unit 110 is suitable for acquiring a shape characteristic vector of a human body to be warehoused and an angle characteristic vector of the human body to be warehoused according to the human body image to be warehoused;

a to-be-stored calculation image determining unit 120 adapted to determine a class center human body image used for calculating the overall similarity and a to-be-stored judgment human body image in the to-be-stored human body image;

the to-be-warehoused overall similarity calculation unit 130 is adapted to calculate the overall similarity between the to-be-warehoused judged human body image and the quasi-central human body image according to the to-be-warehoused judged human body shape feature vector, the to-be-warehoused judged human body angle feature vector and the quasi-central human body shape feature vector of the quasi-central human body image;

and the classification action posture library obtaining unit 140 is adapted to, when the overall similarity satisfies an overall similarity threshold, classify the human body image to be warehoused as the class of the selected class center human body image, store or delete the human body image to be warehoused, obtain an action posture library after classifying the human body image to be warehoused until all the human body images to be warehoused are traversed, and obtain a classification action posture library.

It can be seen that the device for establishing an action posture library provided by the embodiment of the present invention processes an existing human body image to be warehoused, determines the overall similarity between the human body image to be warehoused and a selected class center human body image by using a human body shape feature vector to be warehoused and a human body angle feature vector to be warehoused, determines to classify the human body image to be warehoused as the class center human body image based on the relationship between the overall similarity and a similarity threshold, or directly deletes the human body image to be warehoused, on one hand, establishes the action posture library by using the existing human body image to be warehoused, improves the precision of the action posture of the human body to be warehoused in the action posture library, that is, ensures the precision of an action posture resource when the action posture is established, thereby improving the precision and the effect of the generated action posture; on the other hand, in order to reduce the memory space and ensure the precision requirement of generating the action gesture, when an action gesture library is established, a large number of human body images to be warehoused are classified and analyzed, and when the classification and analysis are performed, the overall similarity between different human body images is determined by using the human body shape characteristic vector to be warehoused and the human body angle characteristic vector to be warehoused, the position state of a human body is used as a comparison factor, the action angle of the human body is also used as a part for determining the overall similarity, so that the determination accuracy is improved, after the classification is completed, the human body images to be warehoused, the overall similarity of which meets the threshold value requirement, and the class center human body images can be classified into one class for storage, preparation is made for the generation of subsequent action gestures and the re-classification based on the requirement, and can also be deleted, on the basis of ensuring the generation of subsequent action gestures, the storage space is reduced; furthermore, by classifying the obtained action posture library, the accuracy of the obtained action posture can be ensured, and the human body images meeting the similarity threshold are classified into one class, so that the calculation amount during the selection of the action posture is reduced, the multiple acquisition of the human body images with very close action postures can be avoided, the change of the acquired action postures is realized, and the variation amount is in an allowable range.

In order to improve the accuracy of grasping the motion posture of the human body, in the embodiment provided by the invention, the feature vectors of the human body shape and the human body angle, which can indicate the motion posture of the human body, are simultaneously obtained, specifically, after the human body image to be warehoused is obtained, the to-be-warehoused feature vector obtaining unit 110 extracts the human body shape feature vector to be warehoused and the human body angle feature vector to be warehoused according to the human body image to be warehoused.

In order to obtain the human body shape feature vector, the to-be-put-in-storage feature vector obtaining unit 110 may specifically include: acquiring a human body mask image to be warehoused of the human body image to be warehoused according to the human body image to be warehoused; and acquiring the shape characteristic vector of the human body to be warehoused according to the mask image of the human body to be warehoused.

The acquisition of the mask image of the human body to be put in storage can eliminate the influence of the skin color, illumination, scene and the like of different people on subsequent processing.

In order to obtain the human body shape feature vector, the to-be-put-in-storage feature vector obtaining unit 110 may specifically include: acquiring human body bone points to be warehoused of the human body image to be warehoused by using a bone point detection model; acquiring human skeleton point line segments to be warehoused according to the human skeleton points to be warehoused; and acquiring the angle characteristic vector of the human body to be warehoused according to the included angle between the line segment of the human body bone point to be warehoused and the reference direction.

The hand skeletal points may include: individual finger tip skeletal points, finger middle skeletal points, finger to palm joint skeletal points, palm skeletal points, and the like.

The face skeleton points may include: ocular, otic, labial, nasal, etc.

After the angle feature vector of the human body to be warehoused and the shape feature vector of the human body to be warehoused are obtained, a to-be-warehoused calculation image determining unit 120 is further utilized to determine a class center human body image used for calculating the overall similarity and a to-be-warehoused judgment human body image in the to-be-warehoused human body image.

When the method is applied to establishing a completely new action posture library from scratch, after the shape characteristic vector of the human body to be warehoused and the angle characteristic vector of the human body to be warehoused of each human body image to be warehoused are obtained, one of the human body to be warehoused and the shape characteristic vector of the human body to be warehoused is selected at will to be determined as a class center human body image used for calculating the overall similarity, and the other human body to be warehoused and judged is selected as a human body image to be warehoused and judged for further judgment.

After the class center human body image and the class center human body angle characteristic vector of the class center human body image are determined, the to-be-warehoused overall similarity calculating unit 130 obtains the class center human body shape characteristic vector and the class center human body angle characteristic vector of the class center human body image, the to-be-warehoused judged human body shape characteristic vector and the to-be-warehoused judged human body angle characteristic vector of the to-be-warehoused judged human body image, and then calculates the overall similarity of the to-be-warehoused judged human body image and the class center human body image according to the to-be-warehoused judged human body shape characteristic vector, the to-be-warehoused judged human body angle characteristic vector, the class center human body shape characteristic vector and the class center human body angle characteristic vector.

The method specifically comprises the following steps: acquiring human body shape similarity according to the human body shape characteristic vector to be warehoused and judged and the quasi-central human body shape characteristic vector; acquiring human body angle similarity according to the human body angle characteristic vector to be warehoused and judged and the quasi-center human body angle characteristic vector; and acquiring the overall similarity according to the human body shape similarity and the human body angle similarity.

The human body shape feature vector to be put in storage can comprise a human face shape feature vector to be put in storage, a hand shape feature vector to be put in storage and a trunk shape feature vector to be put in storage, the influence degree of different parts on the action posture is considered, in order to further improve the accuracy of the calculated human body shape similarity, in a specific implementation mode, the human face shape similarity, the hand shape similarity and the trunk shape similarity can be obtained respectively, and then the human body shape similarity is obtained based on the human face shape similarity, the hand shape similarity and the trunk shape similarity.

In one embodiment, after obtaining the human face shape similarity, the hand shape similarity, and the trunk shape similarity to obtain the human body shape similarity, the human body shape similarity may be obtained by using the following formula:

wherein λ is^f-face shape similarity weight;

λ^h-a hand shape similarity weight;

λ^b-a body shape similarity weight;

-face shape similarity;

-hand shape similarity;

-body shape similarity.

Similarly, the human body angle feature vector to be warehoused includes a human face angle feature vector to be warehoused, a hand angle feature vector to be warehoused and judged and a trunk angle feature vector to be warehoused and judged, and meanwhile, in order to further improve the accuracy of the calculated human body angle similarity, in a specific implementation manner, the human face angle similarity, the hand angle similarity and the trunk angle similarity can be respectively obtained, and then the human body angle similarity is obtained based on the human face angle similarity, the hand angle similarity and the trunk angle similarity.

wherein λ is^f-face angle similarity weight;

λ^h-hand angle similarity weight;

λ^b-a body angle similarity weight;

-face angle similarity;

-hand angle similarity;

-body angle similarity.

D＝(αD_app+βD_ang)^γ；

wherein D is_app-human shape similarity;

D_ang-human body angle similarity;

α — weight of human shape similarity;

beta-weight of human angle similarity;

γ — weight of the weighted sum.

The classification action posture library obtaining unit 140 judges whether the overall similarity meets an overall similarity threshold; if so, classifying the human body image to be warehoused as the class of the class center human body image or deleting the human body image to be warehoused, and obtaining an action posture library after classifying the human body image to be warehoused; if not, further judging whether traversing of various central human body images is finished or not, if so, taking the human body image to be warehoused as a new class-center human body image, and if not, replacing the class-center human body image, and taking the replaced class-center human body image as a class-center human body image for calculating the similarity until an action posture library after classifying the human body image to be warehoused is obtained; and then judging whether the acquired human body images to be put in storage are classified, if not, replacing, further classifying, and if so, obtaining a classification action attitude library.

In another specific embodiment, the classification motion gesture library obtaining unit 140 may further establish an overall similarity matrix between any two of all the center-like human body images in the classified motion gesture library.

In order to ensure the accuracy requirement of human motion gesture generation, an embodiment of the present invention further provides a motion gesture generating apparatus, please refer to fig. 8, and fig. 8 is a block diagram of the motion gesture generating apparatus provided in the embodiment of the present invention.

As shown in the drawings, an action posture generating apparatus provided by an embodiment of the present invention includes:

a to-be-generated human body image acquisition unit 200 adapted to acquire a human body action start frame and a human body action end frame;

a to-be-generated feature vector obtaining unit 210, adapted to obtain a starting human body shape feature vector and a starting human body angle feature vector of the human body motion starting frame, and obtain a terminating human body shape feature vector and a terminating human body angle feature vector of the human body motion terminating frame;

a to-be-generated center-like feature vector obtaining unit 220, adapted to obtain various center body shape feature vectors and various center body angle feature vectors of various center-like body images in the motion posture library established according to the method for establishing a motion posture library of claim 15;

a quasi-center human body image obtaining unit 230, adapted to obtain a nearest quasi-start quasi-center human body image of the human body motion start frame according to the start human body shape feature vector, the start human body angle feature vector, each quasi-center human body shape feature vector, and each type of center human body angle feature vector, and obtain a nearest quasi-end quasi-center human body image of the human body motion end frame according to the end human body shape feature vector, the end human body angle feature vector, each quasi-center human body shape feature vector, and each type of center human body angle feature vector;

and the action posture acquiring unit 240 is adapted to acquire the action posture of the quasi-center human body image arranged between the human body action starting frame and the human body action ending frame by utilizing an overall similarity shortest path algorithm according to the closest similar starting quasi-center human body image, the closest ending quasi-center human body image and the overall similarity matrix.

In order to generate a human body motion gesture, the to-be-generated human body image acquisition unit 200 first acquires a start motion gesture and an end motion gesture that need to be connected, and for this purpose, acquires a human body motion start frame and a human body motion end frame.

After obtaining the human body motion start frame and the human body motion end frame, the to-be-generated feature vector obtaining unit 210 obtains a start human body shape feature vector and a start human body angle feature vector of the human body motion start frame, and a stop human body shape feature vector and a stop human body angle feature vector of the human body motion end frame, respectively.

It can be understood that, in order to combine the influence of different parts on the motion pose, the starting human body shape feature vector may include a starting human face shape feature vector, a starting hand shape feature vector, and a starting trunk shape feature vector; terminating the human shape feature vector may include terminating a human shape feature vector, terminating a hand shape feature vector, and terminating a torso shape feature vector.

The to-be-generated center-like feature vector obtaining unit 220 obtains a center-like human body shape feature vector and a center-like human body angle feature vector from the established motion posture library, and prepares for obtaining motion postures subsequently.

According to the method for calculating the overall similarity, the quasi-center human body image obtaining unit 230 calculates the overall similarity between the human body motion start frame and the shape feature vectors of various types of central human bodies in the motion posture library, and obtains a quasi-center human body image with the maximum overall similarity to the human body motion start frame, namely the closest quasi-center human body image; and respectively calculating the overall similarity of the human body action termination frame and the shape feature vectors of various central human bodies in the action attitude library to obtain a class center human body image with the maximum overall similarity with the human body action termination frame, namely the class center human body image closest to the termination class center.

Since the overall similarity matrix stores the overall similarity between any two of all class-center human body images in the action-posture library after the classification is established, after the closest similar starting class-center human body image and the closest similar ending class-center human body image are obtained, the action-posture obtaining unit 240 can obtain the action posture of the class-center human body image arranged between the closest similar starting class-center human body image and the closest ending class-center human body image by using the overall similarity shortest path algorithm according to the overall similarity matrix in the action-posture library, that is, the action posture of the class-center human body image arranged between the human-body action starting frame and the human-body action ending frame.

It can be seen that the action gesture generating device provided in the embodiment of the present invention can obtain the quasi-center human body image that can be set between the human body action start frame and the human body action end frame through the overall similarity calculation on the basis of establishing the action gesture library according to the real human body image, on one hand, the calculation of the overall similarity is performed, and the human body position and the human body action angle are combined, so that the calculation accuracy is improved, and on the other hand, the minimum action transition of the human body image that is set between the human body action start frame and the human body action end frame is ensured by using the principle of the shortest path of the similarity.

Based on different application scenarios, the embodiment of the present invention further provides an action gesture recognition apparatus, and the action gesture recognition apparatus described below may be considered as a functional module architecture that is required to be set by an electronic device (e.g., a PC) to implement the action gesture recognition method provided by the embodiment of the present invention. The contents of the motion gesture recognition apparatus described below may be referred to in correspondence with the contents of the motion gesture recognition method described above.

Referring to fig. 9, fig. 9 is a block diagram of an action gesture recognition apparatus according to an embodiment of the present invention, and it can be seen that the action gesture recognition apparatus according to the embodiment of the present invention includes:

a to-be-recognized human body image obtaining unit 300 adapted to obtain a to-be-recognized human body image;

a to-be-recognized feature vector obtaining unit 310, adapted to obtain a to-be-recognized human body shape feature vector and a to-be-recognized human body angle feature vector of the to-be-recognized human body image;

a to-be-recognized center feature vector obtaining unit 320, adapted to obtain various types of center body shape feature vectors and various types of center body angle feature vectors of various types of center body images in the action posture library established according to the method for establishing the action posture library of any one of claims 1 to 16;

an overall similarity calculation unit 330, adapted to calculate overall similarities between the human body image to be recognized and the central human body image according to the human body shape feature vector to be recognized, the human body angle feature vector to be recognized, the central human body shape feature vectors of each class and the central human body angle feature vectors of each class;

the identifying unit 340 is adapted to compare the overall similarity with a similarity threshold to obtain a quasi-center human body image with the overall similarity meeting the similarity threshold, and identify the human body image to be identified as the quasi-center human body image meeting the similarity threshold.

In one embodiment, the human body image to be recognized acquiring unit 300 may acquire the human body image to be recognized by using a camera, and of course, may also acquire the human body image to be recognized from an existing image library.

After the human body image to be recognized is obtained, the feature vector to be recognized obtaining unit 310 obtains the shape feature vector of the human body to be recognized and the angle feature vector of the human body to be recognized.

The to-be-recognized center feature vector obtaining unit 320 obtains a center-like human body shape feature vector and a center-like human body angle feature vector from the established motion posture library, and prepares for subsequent motion posture recognition.

According to the method for calculating the overall similarity, the overall similarity calculation unit 330 calculates the overall similarity between the human body image to be recognized and the shape feature vectors of various central human bodies in the motion posture library.

The identifying unit 340 compares the overall similarity with a similarity threshold to obtain a quasi-center human body image with the overall similarity meeting the similarity threshold, and identifies the to-be-identified human body image as the quasi-center human body image meeting the similarity threshold.

In this way, the action gesture recognition device provided by the embodiment of the present invention determines the class center human body image satisfying the similarity threshold by calculating the overall similarity between the human body image to be recognized and each class center human body image in the action gesture library, recognizes that the human body image to be recognized is the class center human body image satisfying the similarity threshold, and simultaneously utilizes the human body shape feature vector and the human body angle feature vector when performing the overall similarity calculation, thereby ensuring the accuracy of the similarity calculation and further ensuring the accuracy of the recognition; meanwhile, by using the established classification action attitude library, the calculation amount can be reduced, the recognition time can be shortened and the requirements on equipment can be reduced on the basis of ensuring the recognition accuracy.

In addition, the embodiment of the present invention further provides an electronic device, where the electronic device may load the program module architecture in a program form to implement the method for establishing the motion gesture library, the method for generating the motion gesture, or the method for recognizing the motion gesture provided in the embodiment of the present invention; the hardware device can be applied to an electronic device with specific data processing capacity, and the electronic device can be: such as a terminal device or a server device.

Optionally, fig. 10 shows an optional hardware electronic device architecture of the electronic device provided in the embodiment of the present invention, which may include: at least one memory 3 and at least one processor 1; the memory stores a program, the processor calls the program to execute the aforementioned establishment method, action posture generation method or action posture recognition method of the action posture library, and in addition, at least one communication interface 2 and at least one communication bus 4; the processor 1 and the memory 3 may be located in the same electronic device, for example, the processor 1 and the memory 3 may be located in a server device or a terminal device; the processor 1 and the memory 3 may also be located in different electronic devices.

As an optional implementation of the disclosure of the embodiment of the present invention, the memory 3 may store a program, and the processor 1 may call the program to execute the method for establishing the motion gesture library, the method for generating the motion gesture, or the method for recognizing the motion gesture provided in the above embodiment of the present invention.

In the embodiment of the invention, the electronic equipment can be a tablet computer, a notebook computer and other equipment capable of establishing the action posture library, and the establishment of the action posture library is realized.

In the embodiment of the present invention, the number of the processor 1, the communication interface 2, the memory 3, and the communication bus 4 is at least one, and the processor 1, the communication interface 2, and the memory 3 complete mutual communication through the communication bus 4; it is clear that the communication connection of the processor 1, the communication interface 2, the memory 3 and the communication bus 4 shown in fig. 10 is only an alternative;

optionally, the communication interface 2 may be an interface of a communication module, such as an interface of a GSM module;

the processor 1 may be a central processing unit CPU or a Specific Integrated circuit asic (application Specific Integrated circuit) or one or more Integrated circuits configured to implement an embodiment of the invention.

The memory 3 may comprise a high-speed RAM memory and may also comprise a non-volatile memory, such as at least one disk memory.

It should be noted that the above terminal device may further include other devices (not shown) that may not be necessary for the disclosure of the embodiment of the present invention; these other components may not be necessary to understand the disclosure of embodiments of the present invention, which are not individually described herein.

An embodiment of the present invention further provides a storage medium, where the storage medium stores a program suitable for video classroom interaction, so as to implement the method for establishing an action posture library according to any one of the foregoing embodiments, or the method for generating an action posture according to any one of the foregoing embodiments, or the method for recognizing an action posture according to any one of the foregoing embodiments.

When the program suitable for the interaction of the video classroom stored in the storage medium provided by the embodiment of the invention is used for realizing the establishment method of the action gesture library, processing the existing human body image to be warehoused, determining the overall similarity between the human body image to be warehoused and the selected class center human body image by utilizing the shape characteristic vector of the human body to be warehoused and the angle characteristic vector of the human body to be warehoused, determining the class of the human body image to be warehoused as the class center human body image based on the relationship between the overall similarity and the similarity threshold value, or directly deleting the human body image to be put in storage, on one hand, establishing an action posture library by utilizing the existing human body image to be put in storage, improving the precision of the action posture of the human body to be put in storage in the action posture library, the precision of action posture resources when the action posture is established is ensured, so that the precision and the effect of the generated action posture can be improved; on the other hand, in order to reduce the memory space and ensure the precision requirement of generating the action gesture, when an action gesture library is established, a large number of human body images to be warehoused are classified and analyzed, and when the classification and analysis are performed, the overall similarity between different human body images is determined by using the human body shape characteristic vector to be warehoused and the human body angle characteristic vector to be warehoused, the position state of a human body is used as a comparison factor, the action angle of the human body is also used as a part for determining the overall similarity, so that the determination accuracy is improved, after the classification is completed, the human body images to be warehoused, the overall similarity of which meets the threshold value requirement, and the class center human body images can be classified into one class for storage, preparation is made for the generation of subsequent action gestures and the re-classification based on the requirement, and can also be deleted, on the basis of ensuring the generation of subsequent action gestures, the storage space is reduced; furthermore, by classifying the obtained action posture library, the accuracy of the obtained action posture can be ensured, and the human body images meeting the similarity threshold are classified into one class, so that the calculation amount during the selection of the action posture is reduced, the multiple acquisition of the human body images with very close action postures can be avoided, the change of the acquired action postures is realized, and the variation amount is in an allowable range.

When the action posture generation method is implemented, based on establishing an action posture library according to a real human body image, a class center human body image which can be set between a human body action starting frame and a human body action ending frame can be obtained through overall similarity calculation, on one hand, the overall similarity is calculated, and meanwhile, a human body position and a human body action angle are combined, so that the calculation precision is improved, and on the other hand, the minimum action transition of the human body image which is set between the human body action starting frame and the human body action ending frame is ensured by using a similarity shortest path principle.

When the action posture recognition method is realized, the class center human body image meeting the similarity threshold is determined by calculating the overall similarity between the human body image to be recognized and each class center human body image in the action posture library, the human body image to be recognized is the class center human body image meeting the similarity threshold, and the human body shape feature vector and the human body angle feature vector are simultaneously utilized during the overall similarity calculation, so that the accuracy of similarity calculation is ensured, and the accuracy of recognition is further ensured; meanwhile, the established action attitude library is utilized, so that the calculation amount can be reduced, the identification time can be shortened and the requirement on equipment can be reduced on the basis of ensuring the identification accuracy.

The embodiments of the present invention described above are combinations of elements and features of the present invention. Unless otherwise mentioned, the elements or features may be considered optional. Each element or feature may be practiced without being combined with other elements or features. In addition, the embodiments of the present invention may be configured by combining some elements and/or features. The order of operations described in the embodiments of the present invention may be rearranged. Some configurations of either embodiment may be included in the class center embodiment and may be replaced with corresponding configurations of the class center embodiment. It is obvious to those skilled in the art that claims that are not explicitly cited in each other in the appended claims may be combined into an embodiment of the present invention or may be included as new claims in a modification after the filing of the present application.

Embodiments of the invention may be implemented by various means, such as hardware, firmware, software, or a combination thereof. In a hardware configuration, the method according to an exemplary embodiment of the present invention may be implemented by one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGAs), processors, controllers, micro-controllers, microprocessors, and the like.

In a firmware or software configuration, embodiments of the present invention may be implemented in the form of modules, procedures, functions, and the like. The software codes may be stored in memory units and executed by processors. The memory unit is located inside or outside the processor, and may transmit and receive data to and from the processor via various known means.

The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Although the embodiments of the present invention have been disclosed, the present invention is not limited thereto. Various changes and modifications may be effected therein by one skilled in the art without departing from the spirit and scope of the invention as defined in the appended claims.

Claims

1. A method for establishing an action posture library is characterized by comprising the following steps:

acquiring a human body image to be put in a warehouse;

according to the human body shape characteristic vector to be warehoused, the class center human body shape characteristic vector of the human body angle characteristic vector to be warehoused and the class center human body image and the class center human body angle characteristic vector, calculating the overall similarity of the human body image to be warehoused and judged and the class center human body image, wherein the step of calculating the overall similarity of the human body image to be warehoused and judged and the class center human body image comprises the following steps:

acquiring human body shape similarity according to the human body shape characteristic vector to be warehoused and judged and the quasi-central human body shape characteristic vector;

acquiring human body angle similarity according to the human body angle characteristic vector to be warehoused and judged and the quasi-center human body angle characteristic vector;

acquiring the overall similarity according to the human body shape similarity and the human body angle similarity; and when the overall similarity meets an overall similarity threshold, classifying the human body image to be warehoused as the class of the selected class center human body image, storing or deleting the human body image to be warehoused, obtaining an action posture library after classifying the human body image to be warehoused until all the human body images to be warehoused are traversed.

2. The method for establishing the action posture library according to claim 1, wherein the human shape feature vector to be warehoused comprises a human face shape feature vector to be warehoused, a hand shape feature vector to be warehoused and a trunk shape feature vector to be warehoused;

the step of obtaining the human body shape similarity according to the human body shape characteristic vector to be put in storage and the quasi-central human body shape characteristic vector comprises the following steps:

acquiring face shape similarity according to the face shape feature vector to be warehoused and the center-like face shape feature vector, acquiring hand shape similarity according to the hand shape feature vector to be warehoused and the center-like hand shape feature vector, and acquiring trunk shape similarity according to the body shape feature vector to be warehoused and the center-like body shape feature vector;

and acquiring the human body shape similarity according to the human face shape similarity, the hand shape similarity and the body shape similarity.

3. The method for establishing an action posture library according to claim 2, wherein the step of obtaining the human body similarity according to the human face shape similarity, the hand shape similarity and the body shape similarity comprises:

obtaining the human body shape similarity by using the following formula:

wherein λ is^f-face shape similarity weight;

λ^h-a hand shape similarity weight;

λ^b-a body shape similarity weight;

-face shape similarity;

-hand shape similarity;

-body shape similarity.

4. The method for establishing the action posture library according to claim 2, wherein the human shape similarity is a chi-square distance between the human shape feature vector to be put in storage and the center-like human shape feature vector.

5. The method for establishing the action posture library according to claim 1, wherein the human body angle feature vector comprises a human face angle feature vector to be warehoused, a hand angle feature vector to be warehoused, and a trunk angle feature vector to be warehoused;

the step of obtaining the human body angle similarity according to the human body angle characteristic vector to be warehoused and judged and the class center human body angle characteristic vector comprises the following steps:

acquiring face angle similarity according to the face angle feature vector to be warehoused and the class center face angle feature vector, acquiring hand angle similarity according to the hand angle feature vector to be warehoused and the class center hand angle feature vector, and acquiring trunk angle similarity according to the body angle feature vector to be warehoused and the class center body angle feature vector;

and acquiring the human body angle similarity according to the human face angle similarity, the hand angle similarity and the body angle similarity.

6. The method for establishing the motion pose library of claim 5, wherein the step of obtaining the human body angle similarity according to the human face angle similarity, the hand angle similarity and the body angle similarity comprises:

obtaining the human body angle similarity by using the following formula:

wherein λ is^f-face angle similarity weight;

λ^h-hand angle similarity weight;

λ^b-a body angle similarity weight;

-face angle similarity;

-hand angle similarity;

-body angle similarity.

7. The method for establishing an action posture library according to claim 5, wherein the human body angle similarity is the Euclidean distance between the human body angle feature vector to be warehoused and judged and the quasi-central human body angle feature vector.

8. The method for establishing the action posture library according to claim 1, wherein the step of obtaining the overall similarity according to the human body shape similarity and the human body angle similarity comprises:

obtaining the overall similarity by using the following formula:

D＝(αD_app+βD_ang)^γ；

wherein D is_app-human shape similarity;

D_ang-human angle similarity;

α - -weight of human shape similarity;

beta-weight of human angle similarity;

γ — weight of the weighted sum.

9. The method for establishing the action posture library according to any one of claims 1 to 8, further comprising, when the overall similarity does not satisfy the similarity threshold and traversal of various types of central human body images is not completed, replacing the class-center human body image, taking the replaced class-center human body image as a class-center human body image for calculating the similarity until the overall similarity does not satisfy the similarity threshold and various types of central human body images have been traversed, taking the human body image to be warehoused and judged as a new class-center human body image, and obtaining the action posture library after classification of the human body image to be warehoused and judged.

10. The method for establishing the motion gesture library according to any one of claims 1 to 8, wherein the step of traversing all the human body images to be warehoused further comprises the following steps:

and establishing an overall similarity matrix between any two of all the class center human body images in the classified action posture library to obtain the classified action posture library.

11. The method for establishing the action posture library according to any one of claims 1 to 8, wherein the step of obtaining the shape feature vector of the human body to be warehoused according to the human body image to be warehoused comprises the following steps:

acquiring a human body mask image to be warehoused of the human body image to be warehoused according to the human body image to be warehoused;

and acquiring the shape feature vector of the human body to be warehoused according to the mask image of the human body to be warehoused.

12. The method for establishing the action posture library according to claim 11, wherein the step of obtaining the human body mask image to be warehoused of the human body image to be warehoused according to the human body image to be warehoused comprises:

acquiring a face mask image to be warehoused by using a face semantic image segmentation algorithm;

and acquiring a hand mask image to be warehoused and a body mask image to be warehoused by using a body semantic image segmentation algorithm.

13. The method for establishing the motion gesture library according to claim 12, wherein the shape feature vector of the human body to be put in the library is a histogram of oriented gradients feature vector.

14. The method for establishing the action posture library according to any one of claims 1 to 8, wherein the step of obtaining the angle feature vector of the human body to be warehoused according to the human body image to be warehoused comprises the following steps:

acquiring human body bone points to be warehoused of the human body image to be warehoused by using a bone point detection model;

acquiring human skeleton point line segments to be warehoused according to the human skeleton points to be warehoused;

and acquiring the angle characteristic vector of the human body to be warehoused according to the included angle between the line segment of the human body bone point to be warehoused and the reference direction.

15. The method of building a library of action poses according to claim 14, wherein the reference direction is a horizontal direction.

16. An action gesture generation method, comprising:

acquiring various types of central human body shape feature vectors and various types of central human body angle feature vectors of various types of central human body images in the action posture library established according to the method for establishing the action posture library of claim 10;

acquiring a nearest similar initial similar central human body image of the human body action initial frame according to the initial human body shape characteristic vector, the initial human body angle characteristic vector, each similar central human body shape characteristic vector and each type of central human body angle characteristic vector, and acquiring a nearest similar termination similar central human body image of the human body action termination frame according to the termination human body shape characteristic vector, the termination human body angle characteristic vector, each similar central human body shape characteristic vector and each type of central human body angle characteristic vector;

and acquiring the action attitude of the class center human body image arranged between the human body action initial frame and the human body action termination frame by utilizing an overall similarity shortest path algorithm according to the closest similar initial class center human body image, the closest similar termination class center human body image and the overall similarity matrix.

17. An action gesture recognition method, comprising:

acquiring a human body image to be recognized;

acquiring various types of central human body shape feature vectors and various types of central human body angle feature vectors of various types of central human body images in the action posture library established according to the method for establishing the action posture library of any one of claims 1 to 15;

calculating the overall similarity of the human body image to be recognized and each class center human body image according to the human body shape feature vector to be recognized, the human body angle feature vector to be recognized, each class center human body shape feature vector and each class center human body angle feature vector, wherein the step of calculating the overall similarity of the human body image to be put in storage and judged and the class center human body image comprises the following steps:

acquiring the overall similarity according to the human body shape similarity and the human body angle similarity;

18. An apparatus for creating a library of action gestures, comprising:

the to-be-warehoused overall similarity calculation unit is suitable for calculating the overall similarity between the to-be-warehoused judged human body image and the quasi-central human body image according to the to-be-warehoused judged human body shape characteristic vector, the to-be-warehoused judged human body angle characteristic vector and the quasi-central human body shape characteristic vector of the quasi-central human body image, and the step of calculating the overall similarity between the to-be-warehoused judged human body image and the quasi-central human body image comprises the following steps:

acquiring human body shape similarity according to the human body shape characteristic vector to be warehoused and judged and the class center human body shape characteristic vector;

19. An action posture generating apparatus, characterized by comprising:

a to-be-generated center-like feature vector acquisition unit adapted to acquire each type of central human body shape feature vector and each type of central human body angle feature vector of each center-like human body image in the action posture library established by the method for establishing an action posture library according to claim 10;

20. An action gesture recognition apparatus, comprising:

a to-be-recognized center feature vector acquisition unit adapted to acquire various center human body shape feature vectors and various center human body angle feature vectors of various center human body images in the action posture library established according to the method for establishing the action posture library of any one of claims 1 to 15;

the overall similarity calculation unit is adapted to calculate overall similarities between the human body image to be recognized and the center-like human body images according to the human body shape feature vector to be recognized, the human body angle feature vector to be recognized, the center-like human body shape feature vectors and the center-like human body angle feature vectors, and the step of calculating the overall similarities between the human body image to be put in storage and the center-like human body images comprises the steps of:

21. An electronic device comprising at least one memory and at least one processor; the memory stores a program that the processor calls to execute the method of establishing the motion gesture library according to any one of claims 1 to 15, or the method of generating the motion gesture according to claim 16, or the method of recognizing the motion gesture according to claim 17.

22. A storage medium storing a program suitable for video classroom interaction to implement the method of establishing the motion gesture library according to any one of claims 1 to 15, or the method of generating the motion gesture according to claim 16, or the method of recognizing the motion gesture according to claim 17.