CN109274883B - Posture correction method, device, terminal and storage medium - Google Patents
Posture correction method, device, terminal and storage medium Download PDFInfo
- Publication number
- CN109274883B CN109274883B CN201810820856.1A CN201810820856A CN109274883B CN 109274883 B CN109274883 B CN 109274883B CN 201810820856 A CN201810820856 A CN 201810820856A CN 109274883 B CN109274883 B CN 109274883B
- Authority
- CN
- China
- Prior art keywords
- user
- posture
- information
- image
- module
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/80—Camera processing pipelines; Components thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Image Analysis (AREA)
Abstract
The embodiment of the invention discloses a posture correction method, a posture correction device, a terminal and a storage medium. The posture correction method comprises the following steps: acquiring a posture image of a user, wherein the posture image displays at least one part of the user; recognizing position information of at least one part of the user from the attitude image by adopting a part recognition model; calculating the posture information of the user according to the position information of the at least one part; and prompting posture correction information to the user according to the posture information of the user and the reference posture information. The posture correction method provided by the embodiment of the invention can realize automatic correction and timely correction of the user posture.
Description
Technical Field
The embodiment of the invention relates to a computer vision technology, in particular to a posture correction method, a posture correction device, a posture correction terminal and a storage medium.
Background
With the development of the live broadcast technology, a large amount of video information is pushed to user side equipment through the internet, and a user can receive the video information from all over the world without going out.
The user can participate in the posture through live video and correct, for example yoga posture is corrected etc.. Currently, the methods for users to participate in posture correction include the following two methods: the first method is that the user side equipment plays a pre-recorded teaching video, the posture of the user is projected to the user side equipment, and the user automatically compares the difference between the projected image and the teaching video and automatically corrects the difference. The second method is that the anchor and the user perform real-time interactive teaching, the user side equipment transmits the gesture of the user to the anchor side equipment through a live video stream, and the anchor side equipment transmits the correction suggestion of the anchor to the user side equipment.
The first method needs self correction of a user, and the second method needs real-time on-line correction of an anchor, so that the problems of correction errors or untimely correction are easy to occur.
Disclosure of Invention
The embodiment of the invention provides a posture correction method, a posture correction device, a terminal and a storage medium, which are used for realizing automatic correction and timely correction of user postures.
In a first aspect, an embodiment of the present invention provides a posture correction method, including:
acquiring a posture image of a user, wherein the posture image displays at least one part of the user;
recognizing position information of at least one part of the user from the attitude image by adopting a part recognition model;
calculating the posture information of the user according to the position information of the at least one part;
and prompting posture correction information to the user according to the posture information of the user and the reference posture information.
In a second aspect, an embodiment of the present invention further provides a posture correction device, including:
the system comprises an acquisition module, a display module and a display module, wherein the acquisition module is used for acquiring a posture image of a user, and the posture image displays at least one part of the user;
the recognition module is used for recognizing the position information of at least one part of the user from the gesture image by adopting a part recognition model;
the calculation module is used for calculating the posture information of the user according to the position information of the at least one part;
and the prompting module is used for prompting the posture correction information to the user according to the posture information of the user and the reference posture information.
In a third aspect, an embodiment of the present invention further provides a terminal, including:
one or more processors;
a memory for storing one or more programs;
the camera is used for acquiring a posture image of a user;
when executed by the one or more processors, cause the one or more processors to implement the posture improvement method of any of the embodiments.
In a fourth aspect, the present invention further provides a computer-readable storage medium, on which a computer program is stored, where the computer program is executed by a processor to implement the posture correction method according to any one of the embodiments.
In the embodiment of the invention, the gesture information of the user is obtained through the part recognition model by acquiring the gesture image of the user and recognizing the position information of the part from the gesture image by adopting the part recognition model; through posture information and benchmark posture information according to the user, the posture correction information is prompted to the user, so that the benchmark posture can be compared, the irregular posture of the user can be automatically corrected, the automatic and intelligent correction of the posture of the user is realized, the user is not required to compare and correct by itself, the anchor is not required to correct online in real time, and the problem of correction error or untimely correction is avoided.
Drawings
Fig. 1 is a flowchart of a posture correction method according to an embodiment of the present invention;
fig. 2a is a flowchart of a posture correction method according to a second embodiment of the present invention;
FIG. 2b is a schematic structural diagram of a part identification model according to a second embodiment of the present invention;
FIG. 2c is a schematic structural diagram of another location identification model according to a second embodiment of the present invention;
FIG. 2d is a thermodynamic diagram provided in accordance with a second embodiment of the present invention;
FIG. 2e is a schematic diagram of the posture information of the user according to the second embodiment of the present invention;
fig. 3 is a schematic structural diagram of a posture correction device according to a fifth embodiment of the present invention;
fig. 4 is a schematic structural diagram of a terminal according to a sixth embodiment of the present invention.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting of the invention. It should be further noted that, for the convenience of description, only some of the structures related to the present invention are shown in the drawings, not all of the structures.
Example one
Fig. 1 is a flowchart of a posture correction method according to an embodiment of the present invention, where this embodiment is applicable to a situation where a terminal collects a posture image of a user and corrects a posture of the user according to the posture image, and the method may be executed by a posture correction device, which may be formed by hardware and/or software and integrated in the terminal, and specifically includes the following steps:
s110, acquiring a posture image of the user, wherein at least one part of the user is displayed on the posture image.
The terminal is provided with a camera facing to one side of the user, and the user is shot through the camera so as to acquire the posture image of the user. Alternatively, the posture image may be acquired at a preset time, acquired after a photographing instruction, or periodically acquired. The pose image may be one, two or more.
The user parts displayed on the gesture image include, but are not limited to, head, neck, left/right shoulders, left/right elbows, left/right wrists, left/right hips, left/right knees, left/right ankles. In some embodiments, the user locations may be referred to as key points of the user.
And S120, recognizing the position information of at least one part of the user from the posture image by adopting a part recognition model.
The input of the part recognition model is a pose image, and the output is position information of at least one part in the pose image. Based on this, the posture image is input to the part recognition model, and the position information of at least one part output by the part recognition model is acquired. Alternatively, the position information of the at least one location may be position coordinates or a range of areas, such as a thermodynamic diagram.
And S130, calculating the posture information of the user according to the position information of at least one part.
When the user's body parts are in different positions, the user exhibits different gestures, for example, the left wrist is above the right wrist, and the user exhibits different gestures than the right wrist is above the left wrist. In this embodiment, the posture information of the user includes information such as an angle and a position of a limb formed by at least one part, and/or a relative angle and a relative distance between the limb and the limb. Of course, the requirements for the posture are different according to the different types of the user's motions, and any posture information that can be obtained from the position information of the part is within the scope of the present application.
And S140, prompting posture correction information to the user according to the posture information of the user and the reference posture information.
In the present embodiment, reference posture information corresponding to the posture information of the user is stored in advance, and for example, if the posture information of the user is tree-type, the reference posture information is reference tree-type posture information.
Posture correction information is obtained by comparing the posture information of the user with the reference posture information, and the posture correction information is displayed on a terminal display screen and/or is subjected to voice prompt, so that the posture is corrected automatically and timely.
In the embodiment of the invention, the gesture information of the user is obtained through the part recognition model by acquiring the gesture image of the user and recognizing the position information of the part from the gesture image by adopting the part recognition model; through posture information and benchmark posture information according to the user, the posture correction information is prompted to the user, so that the benchmark posture can be compared, the irregular posture of the user can be automatically corrected, the automatic correction of the posture of the user is realized, the user is not required to compare and correct by itself, the anchor real-time online correction is also not required, and the problem of correction error or untimely correction is avoided.
Example two
In this embodiment, the above embodiment is further optimized, and specifically, the position information of at least one part of the user is recognized from the gesture image by using a part recognition model, which is defined as: inputting the attitude image into a part recognition model to obtain a thermodynamic diagram output by the part recognition model and corresponding to each part; accordingly, the calculation of the posture information of the user from the position information of the at least one part is defined as: respectively determining the position coordinates of each part from the thermodynamic diagrams corresponding to each part; and calculating the posture information of the user according to the position coordinates of the at least one part. It should be noted that if the position information output by the part recognition model is position coordinates, the posture information of the user may be directly calculated from the position coordinates of at least one part. In addition, the structure of the part recognition model is limited in the embodiment.
Fig. 2a is a flowchart of a posture correction method according to a second embodiment of the present invention, and as shown in fig. 2a, the method includes the following steps:
s210, acquiring a posture image of the user, wherein at least one part of the user is displayed on the posture image.
S210 is the same as S110, and is not described herein again.
And S220, inputting the posture image into the part recognition model to obtain a thermodynamic diagram which is output by the part recognition model and corresponds to each part.
Fig. 2b is a schematic structural diagram of a part identification model according to a second embodiment of the present invention. As shown in fig. 2b, the part recognition model includes a feature extraction submodel 21 and a part detection submodel 22.
The feature extraction submodel 21 includes: and the plurality of groups of convolution modules 23 and down-sampling modules 24 which are connected in sequence are used for inputting the attitude image and outputting the characteristics in the attitude image.
The convolution module 23 is used to perform feature extraction on the input content. The convolution module 23 includes at least one convolution layer for extracting features. The down-sampling module 24 is configured to down-sample the features output by the convolution module 23, and optionally 1/2 down-sample the features output by the convolution module 23 by using a max boosting method. The purpose of downsampling is to scale the received features, thereby reducing feature dimensions and reducing data redundancy.
The site detection submodel 22 includes: a plurality of detection modules 25 (shown in a dashed box in fig. 2 b) and convolution modules 23 connected in series for inputting features in the pose image and outputting position information corresponding to each part, such as a thermodynamic diagram;
wherein, the detection module 25 includes: a connected down-sampling module 24, a plurality of groups of convolution modules 23 and up-sampling modules 26 connected in sequence. The recognition result is corrected step by a plurality of connected detection modules 25, thereby achieving a better recognition effect. Fig. 2b shows 2 detection modules 25 connected in sequence, but the present invention is not limited thereto, and the sub-site detection model 22 may further include a plurality of detection modules 25 connected in sequence, and experiments prove that the recognition effect achieved by using 5 detection modules 25 is ideal.
The up-sampling module 26 is configured to up-sample the features output by the convolution module 23, and optionally 1/2 up-sample the features output by the convolution module 23 by using a nearest neighbor difference or bilinear difference algorithm. The purpose of up-sampling is to convert a received low resolution image into a high resolution image in order to acquire an image with sufficient resolution.
Further, fig. 2c is a schematic structural diagram of another part recognition model provided in the second embodiment of the present invention. Through a large number of experiments, if the part recognition model comprises a feature extraction submodel and a part detection submodel, the recognition result is not ideal enough, for example, the situation of part recognition error occurs. Based on this, as shown in fig. 2c, a skip layer 27 is added between the feature extraction submodel and the part detection submodel. The skip layer 27 is used for splicing the features output by at least one down-sampling module 24 in the feature extraction submodel 21 and then splicing the features output by the corresponding up-sampling module 26 in the part detection submodel 22.
Wherein, the jump layer 27 splices the features output by different down-sampling modules 24 in the feature extraction submodel 21; then, the above-mentioned spliced features are spliced with the features of the same resolution output by the up-sampling module 26 in the region detection sub-model 22. It is worth noting that the resolution of the features that are stitched to each other should be the same. If the resolutions are different, the resolutions may be adjusted to be the same using up-sampling module 26 or down-sampling module 24. In fig. 2c, 3 modules for adjusting the resolution are shown, the fill color of the modules for adjusting the resolution being black.
By adding the skip layer 27 in this way, the part detection submodel 22 learns multi-scale global features, so that the part detection submodel 22 can acquire sufficient global information, and the part recognition model 22 can accurately recognize position information of a corresponding part from a complex attitude image. In some embodiments, each convolution module 23 includes M bottleeck 28, each bottleeck 28 including connected dilation, convolution, and compression layers. This is because the features extracted by the convolutional layer are limited by the number of channels input, and if the channels are "compressed" before the features are extracted, fewer features can be extracted. Thus, the channels are expanded, features are extracted by convolution, and then compressed to extract a sufficient number of features. Like the bottleck in the MobileNet-V2 network structure, in this embodiment, the expanded layer includes 1 × 1 convolution kernel, the convolutional layer includes 3 × 3 convolution kernel, and the compressed layer includes 1 × 1 convolution kernel. Optionally, a large number of experiments prove that the identification accuracy is higher when M is more than or equal to 3 and less than or equal to 5.
In the embodiment, the number of output channels in the bottleeck 28 is N, N is more than or equal to 16 and less than or equal to 96, and is less than that of output channels in the MobileNet-V2 network structure, so that network parameters are reduced, and the calculation efficiency is improved; meanwhile, the space required by a storage network is saved, so that the part recognition model can run at the terminal.
The number of liters per channel, convolution size, number of output channels and number of repetitions (i.e., the number of bottleeck) of the bottleeck 28 are labeled above the corresponding convolution module as shown in fig. 2 c. Of course, the parameters of the convolution module 23 in fig. 2c are only examples, and those skilled in the art can adjust the parameters of the convolution module to obtain a better recognition effect.
It should be noted that, in general, the more convolution modules in the feature extraction submodel 21 and the part detection submodel 22, the more bottleeck contained in each convolution module, the deeper the extracted features, and the better the recognition effect. But results in poor performance when the network reaches a certain depth. Those skilled in the art can adjust the number of detection modules, the number of convolution modules, and the number of bottleeck contained in each convolution module based on the network structure shown in fig. 2c to achieve better recognition effect.
In this embodiment, the posture image is input to the part recognition model, the part recognition model outputs thermodynamic diagrams corresponding to the respective parts, and one thermodynamic diagram shows one part. Of course, a thermodynamic diagram may also be two or more locations. The thermodynamic diagram in this embodiment is a diagram showing the region where the corresponding portion is located in a particularly highlighted form.
As shown in fig. 2d, it is assumed that the part recognition model outputs 3 thermodynamic diagrams in total, the first thermodynamic diagram corresponding to the left wrist, the second thermodynamic diagram corresponding to the right wrist, and the third thermodynamic diagram corresponding to the neck.
And S230, respectively determining the position coordinates of each part from the thermodynamic diagrams corresponding to each part.
Optionally, a point (x, y) is selected from the highlighted area on the thermodynamic diagram of each part as the position coordinates of the corresponding part. Preferably, the center point or the brightest point is selected as the position coordinates of the corresponding part from the highlighted area on the thermodynamic diagram of each part. Specifically, the thermodynamic diagrams of the respective portions are filtered (for example, gaussian filtering, wiener filtering, or mean filtering), and the coordinates of the point with the lowest gradation or the highest brightness are selected as the position coordinates of the corresponding portion in the filtered image.
And S240, calculating the posture information of the user according to the position coordinates of at least one part.
Optionally, calculating a bending angle between the user's connected limbs according to the position coordinates of the at least one part; and/or calculating the orientation angle of each limb of the user relative to the preset direction according to the position coordinates of at least one part.
It is worth mentioning that if the position information is position coordinates, the bending angle between the limbs connected by the user is calculated directly according to the position information of at least one part; and/or calculating the azimuth angle of each limb of the user relative to the preset direction according to the position information of at least one part. If the position information is the thermodynamic diagram, S230 is executed to determine corresponding position coordinates according to the position information of at least one part, and then the bending angle between the connected limbs of the user is calculated according to the position coordinates of at least one part; and/or calculating the orientation angle of each limb of the user relative to the preset direction according to the position coordinates of at least one part.
The connected limbs are, for example, upper and lower arms, head and neck, thigh and calf, etc. The bending angle between the connected limbs is the angle between the two connected limbs. The preset direction may be a horizontal direction and/or a vertical direction.
Fig. 2e is a schematic diagram of the posture information of the user provided by the second embodiment of the present invention, and fig. 2e shows the parts including the left shoulder, the left elbow and the left wrist. Vector for upper arm
It is shown that,
left shoulder coordinate-left elbow coordinate, vector for lower arm
It is shown that,
bending angle between upper and lower arms
The method for calculating the azimuth angle of the upper arm comprises the following steps: assuming that the predetermined direction is a vertical direction, the corresponding vector
Computing vectors
To vector
Angle h of (d).
And S250, prompting the posture correction information to the user according to the posture information of the user and the reference posture information.
Wherein the reference attitude information includes: a reference bending angle between the connected limbs, and/or a reference orientation angle of each limb relative to a predetermined direction.
In this embodiment, the posture correction information is presented to the user according to the posture information of the user and the reference posture information, which includes the following two implementation manners.
The first embodiment: first, a first score for the connected limbs is calculated based on the bend angles between the connected limbs and the corresponding reference bend angle ranges. Wherein, the reference angle range is defined by 3 thresholds: std, min, max. std denotes the median value, min denotes the minimum value, and max denotes the maximum value. Optionally, the first score of the connected limb is calculated according to the following piecewise function. Of course, the first score may be directly obtained by the difference between the bending angle and std.
Where x represents the bend angle between the connected limbs.
Then, a second score of each limb is calculated based on the azimuth angle of each limb with respect to the preset direction and the corresponding reference azimuth angle range.
Alternatively, as in the calculation method of the first score, the reference azimuth angle range is also calculated by 3 thresholds: std, min, max. And calculating a second score of each limb according to the segmentation function. At this time, x in the piecewise function represents the azimuth angle of each limb.
Then, carrying out weighted summation on the first score and the second score to obtain a comprehensive score of the user; and prompting the posture correction information corresponding to the comprehensive score to the user.
By the formula
A composite score for the user is calculated. Where n is the sum of the number of azimuth angles of each limb and the number of flexion angles between the connected limbs, xiIs the azimuth angle of the ith limb or the azimuth angle of the connected limb, f (x)i) Is the score for the ith limb or connected limb, e.g. f (x)1) Score, f (x), representing the azimuth angle of the upper arm2) A score representing the bending angle between the upper and lower arms. w is aiIs corresponding to f (x)i) The weight of (a) is determined,
the weight of the first score for each connected limb and the weight of the second score for each limb may be the same or different. The score for the limb with the highest requirement is given a first weight and the score for the limb with the lowest requirement is given a second weight, taking into account the different requirements of the different poses for the different limbs. The first weight is greater than the second weight.
The different composite scores correspond to different posture correction information, and the posture correction information is correction of the body dimension of the user. For example, if the composite score of the user is greater than the first segment value, e.g., 80 points, the user is prompted to "better flexibility, stretch of limbs, and overall reach a higher level"; if the comprehensive score of the user is greater than the second segmentation value, for example, 60 points, and less than the first segmentation value, the method prompts that the flexibility of the user is normal, the limbs are stretched, and the whole standard level is reached; if the comprehensive score of the user is smaller than the second segmentation value, the user is prompted to have the general flexibility, the limbs of the user are not enough to stretch, and the general level of the user is reached.
The second embodiment: calculating a first difference value between the bending angle between the connected limbs and the corresponding reference bending angle, and prompting posture correction information corresponding to the first difference value to a user; and/or calculating a second difference value between the azimuth angle of each limb relative to the preset direction and the corresponding reference azimuth angle, and prompting posture correction information corresponding to the second difference value to the user.
The posture correction information in the present embodiment is correction of the user limb dimension. For example, a first difference between the flexion angle between the connected limbs minus the corresponding reference flexion angle is greater than 0, prompting the user to tighten the corresponding limb; the first difference is less than 0, prompting the user to expand the corresponding limb. For another example, if the second difference between the orientation angle of the limb relative to the preset direction and the corresponding reference orientation angle is greater than 0, prompting the user to move the limb in the direction of reducing the second difference; the second difference is less than 0, prompting the user to move the limb in a direction that increases the second difference.
In the embodiment, at least one bottleeck forms a convolution module for feature extraction, so that multiple channels and sufficient number of features can be extracted; the number of output channels in the convolution module is less than that of output channels in a MobileNet-V2 network structure, so that network parameters are reduced, and the calculation efficiency is improved; meanwhile, the space required by a storage network is saved, so that image recognition can be carried out on the terminal; the feature output by the convolution module is subjected to down sampling, namely the feature is scaled, so that the feature dimension is reduced, and the data redundancy is reduced; the part detection submodel learns multi-scale and global characteristics through layer skipping, so that the part detection submodel can acquire enough global information, and the part recognition model can accurately recognize the position information of the corresponding part from a complex attitude image.
In addition, in the embodiment, the posture of the user is obtained by calculating the bending angle between the connected limbs and/or the azimuth angle of the limbs relative to the preset direction, and the posture is consistent with the requirement on the posture in real life, so that the posture correction is more reasonable and effective. And then, calculating the score of the user by combining the reference bending angle range and/or the reference azimuth angle range, scoring the posture from the dimensions of the bending angle and the azimuth angle, accurately evaluating the accuracy of the posture of the user, and achieving the effect of effectively correcting the posture.
In some embodiments, to increase the rate of operation of the site detection model at the terminal, the bottleeck is replaced with a separable convolution module.
The specific demonstration process is as follows:
assume that the input size of the convolution is (D)F,DFM), using a convolution kernel K of (D)K,DKM, N), step size is 1, and size of output feature graph G is (D)G,DGN). Then the amount of computation for the conventional convolution is: dK·DK·M·N·DF·DFWhere M is the number of input channels and N is the number of output channels.
The separable convolution is to divide the conventional convolution into a depth convolution and a point-by-point convolution, and the corresponding calculation amount is: dK·DK·M·DF·DF+M·N·DF·DFThe ratio of the two convolution calculations is as follows:
as can be seen from the ratio, under the same hyper-parameter, the calculation amount of the separable convolution is the calculation amount of the traditional convolution
And (4) doubling. Furthermore, the precision is not very different, and in the case of the same model and input data, the probability of correlation keypoint (PCKh) of the conventional convolution is 98, while the PCKh of the separable convolution is 97.2).
EXAMPLE III
The embodiment of the invention provides an application scenario of a posture correction method, and describes the process of posture correction of a user through a terminal in detail.
In an application scenario of the embodiment, the user turns on the posture correction function of the terminal and selects one posture, for example, a pigeon posture. The terminal starts the camera and collects the area image in the camera collecting area. The camera acquisition area can be an area in front of the camera, and no blocking object or obstacle is required in the area, so that the camera acquisition area is suitable for the movement of a user. And identifying the user part in the preset area of the area image by adopting a part identification model. The preset region may be a central region of the region image, and the waist and the chest are preferentially identified. When the user is identified within a preset area, for example, the waist and chest of the user, it is determined that the user is located at a suitable position and the activity can be started. On the contrary, if the user is not identified in the preset area, the moving direction is prompted to the user according to the identification result in the whole area image, so that the user is located in the preset area.
And when the user is identified in the preset area of the area image, playing a teaching video corresponding to the reference posture information through the display screen. The reference attitude information refers to reference attitude information corresponding to the attitude selected by the user. The reference attitude information is stored in a configuration file of the terminal.
After the teaching video is played or in the playing process, the terminal prompts a user to start intelligent correction, and the camera collects the posture image of the user in real time, projects the posture image of the user onto a display screen of the terminal and displays the posture image of the user and the teaching video in a regional mode.
Then, the posture correction device collects one or more posture images of the user and adopts a part recognition model to recognize the position information of at least one part of the user from the posture images; calculating the posture information of the user according to the position information of at least one part; based on the posture information of the user and the reference posture information, posture correction information such as "arm re-height point", "good hold", and the like is presented to the user.
After the posture correction is finished, the terminal displays the posture correction information of the previous time and the corresponding comprehensive score for the user to compare and check so as to obtain the correction effect.
Example four
In this embodiment, the above embodiment is further optimized, and a part recognition model is adopted, and the position information of at least one part of the user recognized from the gesture image specifically includes: and inputting the attitude model into a part recognition model, and acquiring the position information of at least one part output by the part recognition model.
Further, the obtaining of the position information of the at least one part output by the part recognition model comprises: and acquiring a thermodynamic diagram corresponding to each part output by the part identification model.
The part identification model in this embodiment is described in detail in the above embodiments, and is not described herein again.
In some embodiments, before the position information of at least one part of the user is recognized by the part recognition model, a training process of the part recognition model is further included. The training process of the part recognition model comprises the following steps: the method comprises the steps of obtaining a part recognition model to be trained, obtaining a sample image displaying at least one part of a user and position label information corresponding to the at least one part, and optionally, the position label information corresponding to the at least one part refers to position coordinates or an area range corresponding to the part. Wherein the area range can be represented by a coordinate range, a geometric mark or a thermodynamic diagram.
Then, a part recognition model is trained according to the sample image and the position label information corresponding to at least one part. Specifically, a sample image is input to a part recognition model to be trained, and parameters in the part recognition model are iterated, so that the model outputs position label information approximate to the corresponding position of at least one part in the input sample image.
Further, training a part recognition model according to the position label information corresponding to the sample image and the at least one part, comprising: generating a thermodynamic diagram corresponding to each part according to the position label information corresponding to at least one part; and training the part recognition model according to the sample image and the thermodynamic diagrams corresponding to all parts.
Illustratively, 1 ten thousand sample images are taken in a hundred-degree crowd-sourced form, and then each part is marked on the sample images. The marked sample images are processed by a gaussian fuzzy algorithm to obtain thermodynamic diagrams corresponding to each part, such as a thermodynamic diagram corresponding to the head, a thermodynamic diagram corresponding to the left shoulder, a thermodynamic diagram corresponding to the right shoulder and the like. Inputting the unmarked sample image into a part recognition model to be trained, and iterating parameters in the part recognition model to enable the model output to approach a thermodynamic diagram corresponding to at least one part in the input sample image.
EXAMPLE five
Fig. 3 is a schematic structural diagram of a posture correction device according to a fifth embodiment of the present invention, which includes an acquisition module 41, a recognition module 42, a calculation module 43, and a prompt module 44.
An acquisition module 41, configured to acquire a posture image of a user, where the posture image shows at least one part of the user;
a recognition module 42 for recognizing position information of at least one part of the user from the gesture image using a part recognition model;
a calculation module 43, configured to calculate pose information of the user according to the position information of the at least one portion;
and the prompting module 44 is used for prompting the posture correction information to the user according to the posture information of the user and the reference posture information.
In the embodiment, the gesture information of the user is obtained through the part recognition model by acquiring the gesture image of the user and recognizing the position information of the part from the gesture image by adopting the part recognition model, so that the gesture information of the user is calculated; through posture information and benchmark posture information according to the user, the posture correction information is prompted to the user, so that the benchmark posture can be compared, the irregular posture of the user can be automatically corrected, the automatic correction of the posture of the user is realized, the user is not required to compare and correct by itself, the anchor real-time online correction is also not required, the labor cost and the operation cost are saved, and the problem of correction error or untimely correction is avoided.
Optionally, the recognition module 42, when recognizing the position information of at least one part of the user from the gesture image by using the part recognition model, is specifically configured to: and inputting the attitude image into the part recognition model to obtain a thermodynamic diagram which is output by the part recognition model and corresponds to each part.
Optionally, the recognition module 42, when recognizing the position information of at least one part of the user from the gesture image by using the part recognition model, is specifically configured to: and inputting the attitude model into a part recognition model, and acquiring the position information of at least one part output by the part recognition model.
Further, the part identification model comprises a feature extraction submodel and a part detection submodel, and further comprises a jump layer; the feature extraction submodel includes: the system comprises a plurality of groups of convolution modules and down-sampling modules which are connected in sequence and used for inputting attitude images and outputting characteristics in the attitude images; the part detection submodel includes: the detection modules and the convolution modules are connected in sequence and used for inputting the features in the attitude image and outputting thermodynamic diagrams corresponding to all parts; the detection module includes: the system comprises a down-sampling module, a plurality of groups of convolution modules and an up-sampling module which are connected in sequence; the layer jump is used for splicing the characteristics output by at least one down-sampling module in the characteristic extraction submodel with the characteristics output by the corresponding up-sampling module in the position detection submodel; each convolution module comprises M bottleecks, each bottleeck comprises an expansion layer, a convolution layer and a compression layer which are connected, the number of output channels is N, wherein M, N is a natural number, M is more than or equal to 3 and less than or equal to 5, and N is more than or equal to 16 and less than or equal to 96.
Optionally, when the calculating module 43 calculates the posture information of the user according to the position information of the at least one portion, it is specifically configured to: calculating the bending angle between the limbs connected by the user according to the position information of at least one part; and/or calculating the azimuth angle of each limb of the user relative to the preset direction according to the position information of at least one part.
Optionally, each bottleeck includes a depth convolution and a point-by-point convolution.
Optionally, the reference posture information includes: a reference bending angle range between the connected limbs and a reference azimuth angle range of each limb relative to a preset direction; accordingly, the prompting module 44 is specifically configured to, when prompting the posture correction information to the user according to the posture information of the user and the reference posture information: calculating a first score of the connected limbs according to the bending angle between the connected limbs and the corresponding reference bending angle range; calculating a second score of each limb according to the azimuth angle of each limb relative to the preset direction and the corresponding reference azimuth angle range; carrying out weighted summation on the first score and the second score to obtain a comprehensive score of the user; and prompting the posture correction information corresponding to the comprehensive score to the user.
Optionally, the apparatus further comprises a collection module and a playing module. The acquisition model is used for acquiring an area image in an acquisition area of the camera before acquiring the attitude image. When the recognition module 42 recognizes the user in the preset region of the region image, the teaching video corresponding to the reference posture information is played through the playing module.
Optionally, the apparatus further includes a training module, configured to obtain a part recognition model to be trained, obtain a sample image showing at least one part of a user and position label information corresponding to the at least one part, and train the part recognition model according to the sample image and the position label information corresponding to the at least one part.
The posture correction device provided by the embodiment of the invention can execute the posture correction method provided by any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method.
EXAMPLE six
Fig. 4 is a schematic structural diagram of a terminal according to a sixth embodiment of the present invention, and as shown in fig. 4, the terminal includes a processor 50, a memory 51, an input device 52, an output device 53, and a camera 54; the number of the processors 50 in the terminal may be one or more, and one processor 50 is taken as an example in fig. 4; the processor 50, the memory 51, the input device 52, the output device 53 and the camera 54 in the terminal may be connected by a bus or other means, which is exemplified in fig. 4.
The memory 51 is a computer-readable storage medium, and can be used for storing software programs, computer-executable programs, and modules, such as program instructions/modules corresponding to the posture correction method in the embodiment of the present invention (for example, the acquisition module 41, the recognition module 42, the calculation module 43, and the prompt module 44 in the posture correction apparatus). The processor 50 executes various functional applications of the terminal and data processing by running software programs, instructions, and modules stored in the memory 51, thereby implementing the posture correction method described above.
The memory 51 may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function; the storage data area may store data created according to the use of the terminal, and the like. Further, the memory 51 may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid state storage device. In some examples, the memory 51 may further include memory located remotely from the processor 50, which may be connected to the terminal over a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The input device 52 may be used to receive input numeric or character information and to generate key signal inputs related to user settings and function controls of the terminal, such as user selection of a gesture through the input device 52. The output device 53 may include a display device such as a display screen and an audio device such as a speaker.
And the camera 54 is used for acquiring a posture image of the user. The camera may be a front-facing camera on the terminal.
EXAMPLE seven
An embodiment of the present invention further provides a computer-readable storage medium having stored thereon a computer program, which when executed by a computer processor is configured to perform a method for posture correction, the method including:
acquiring a posture image of a user, wherein the posture image displays at least one part of the user;
recognizing position information of at least one part of the user from the attitude image by adopting a part recognition model;
calculating the posture information of the user according to the position information of at least one part;
and prompting the posture correction information to the user according to the posture information of the user and the reference posture information.
Of course, the embodiments of the present invention provide a computer-readable storage medium having a computer program stored thereon, where the computer program is not limited to the above method operations, and can also perform related operations in the posture correction method provided by any embodiments of the present invention.
From the above description of the embodiments, it is obvious for those skilled in the art that the present invention can be implemented by software and necessary general hardware, and certainly, can also be implemented by hardware, but the former is a better embodiment in many cases. Based on such understanding, the technical solutions of the present invention may be embodied in the form of a software product, which can be stored in a computer-readable storage medium, such as a floppy disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a FLASH Memory (FLASH), a hard disk or an optical disk of a computer, and includes several instructions for enabling a computer device (which may be a personal computer, a server, or a network device) to execute the methods of the embodiments of the present invention.
It should be noted that, in the embodiment of the posture correction device, the units and modules included in the embodiment are merely divided according to functional logic, but are not limited to the above division as long as the corresponding functions can be realized; in addition, specific names of the functional units are only for convenience of distinguishing from each other, and are not used for limiting the protection scope of the present invention.
It is to be noted that the foregoing is only illustrative of the preferred embodiments of the present invention and the technical principles employed. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments illustrated herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, although the present invention has been described in greater detail by the above embodiments, the present invention is not limited to the above embodiments, and may include other equivalent embodiments without departing from the spirit of the present invention, and the scope of the present invention is determined by the scope of the appended claims.
Claims (9)
1. A posture correction method, comprising:
acquiring a posture image of a user, wherein the posture image displays at least one part of the user;
recognizing position information of at least one part of the user from the attitude image by adopting a part recognition model;
the recognizing, by using the part recognition model, the position information of at least one part of the user from the gesture image includes:
inputting the attitude image into the part recognition model to obtain a thermodynamic diagram output by the part recognition model and corresponding to each part;
the part identification model comprises a feature extraction submodel, a part detection submodel and a jump layer;
the feature extraction submodel includes: the system comprises a plurality of groups of convolution modules and down-sampling modules which are connected in sequence and used for inputting attitude images and outputting characteristics in the attitude images;
the part detection submodel includes: the detection modules and the convolution modules are sequentially connected and are used for inputting the features in the attitude image and outputting thermodynamic diagrams corresponding to all parts;
the detection module comprises: the system comprises a down-sampling module, a plurality of groups of convolution modules and an up-sampling module which are connected in sequence;
the skip layer is used for splicing the characteristics output by at least one down-sampling module in the characteristic extraction submodel with the characteristics output by the corresponding up-sampling module in the position detection submodel;
calculating the posture information of the user according to the position information of the at least one part;
and prompting posture correction information to the user according to the posture information of the user and the reference posture information.
2. The method of claim 1, wherein each convolution module comprises M bottleecks, each bottleeck comprises an expansion layer, a convolution layer and a compression layer which are connected, the number of output channels is N, wherein M, N is a natural number, M is greater than or equal to 3 and less than or equal to 5, and N is greater than or equal to 16 and less than or equal to 96.
3. The method of claim 2, wherein each bottleeck comprises a depth convolution and a point-by-point convolution.
4. The method of claim 1, wherein the calculating the pose information of the user according to the position information of the at least one part comprises:
calculating the bending angle between the limbs connected by the user according to the position information of the at least one part; and/or the presence of a gas in the gas,
and calculating the azimuth angle of each limb of the user relative to the preset direction according to the position information of the at least one part.
5. The method of claim 4, wherein the reference pose information comprises: a reference bending angle range between the connected limbs and a reference azimuth angle range of each limb relative to a preset direction;
the prompting of the posture correction information to the user according to the posture information of the user and the reference posture information comprises the following steps:
calculating a first score of the connected limbs according to the bending angle between the connected limbs and the corresponding reference bending angle range;
calculating a second score of each limb according to the azimuth angle of each limb relative to the preset direction and the corresponding reference azimuth angle range;
weighting and summing the first score and the second score to obtain a comprehensive score of the user;
and prompting the posture correction information corresponding to the comprehensive score to the user.
6. The method of any one of claims 1-5, further comprising, prior to acquiring the pose image:
collecting an area image in a camera collecting area;
and when the user is identified in the preset area of the area image, playing a teaching video corresponding to the reference posture information.
7. A posture improvement device, comprising:
the system comprises an acquisition module, a display module and a display module, wherein the acquisition module is used for acquiring a posture image of a user, and the posture image displays at least one part of the user;
the recognition module is used for recognizing the position information of at least one part of the user from the gesture image by adopting a part recognition model;
the recognition module, when recognizing the position information of at least one part of the user from the gesture image by using the part recognition model, is specifically configured to: inputting the attitude image into a part recognition model to obtain a thermodynamic diagram output by the part recognition model and corresponding to each part;
the recognition module, when recognizing the position information of at least one part of the user from the gesture image by using the part recognition model, is specifically configured to: inputting the attitude model into a part recognition model, and acquiring position information of at least one part output by the part recognition model;
the part identification model comprises a feature extraction submodel, a part detection submodel and a jump layer;
the feature extraction submodel includes: the system comprises a plurality of groups of convolution modules and down-sampling modules which are connected in sequence and used for inputting attitude images and outputting characteristics in the attitude images;
the part detection submodel includes: the detection modules and the convolution modules are connected in sequence and used for inputting the features in the attitude image and outputting thermodynamic diagrams corresponding to all parts;
the detection module includes: the system comprises a down-sampling module, a plurality of groups of convolution modules and an up-sampling module which are connected in sequence;
the layer jump is used for splicing the characteristics output by at least one down-sampling module in the characteristic extraction submodel with the characteristics output by the corresponding up-sampling module in the position detection submodel;
the calculation module is used for calculating the posture information of the user according to the position information of the at least one part;
and the prompting module is used for prompting the posture correction information to the user according to the posture information of the user and the reference posture information.
8. A terminal, comprising:
one or more processors;
a memory for storing one or more programs;
the camera is used for acquiring a posture image of a user;
when executed by the one or more processors, cause the one or more processors to implement a posture improvement method as recited in any of claims 1-6.
9. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the posture-correcting method of any one of claims 1-6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810820856.1A CN109274883B (en) | 2018-07-24 | 2018-07-24 | Posture correction method, device, terminal and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810820856.1A CN109274883B (en) | 2018-07-24 | 2018-07-24 | Posture correction method, device, terminal and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109274883A CN109274883A (en) | 2019-01-25 |
| CN109274883B true CN109274883B (en) | 2022-02-01 |
Family
ID=65153187
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810820856.1A Active CN109274883B (en) | 2018-07-24 | 2018-07-24 | Posture correction method, device, terminal and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109274883B (en) |
Families Citing this family (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110211670B (en) * | 2019-05-14 | 2022-06-03 | 广州虎牙信息科技有限公司 | Index prediction method, index prediction device, electronic equipment and storage medium |
| CN110264539A (en) * | 2019-06-18 | 2019-09-20 | 北京字节跳动网络技术有限公司 | Image generating method and device |
| CN110443148B (en) * | 2019-07-10 | 2021-10-22 | 广州市讯码通讯科技有限公司 | Action recognition method, system and storage medium |
| CN112241653A (en) * | 2019-07-16 | 2021-01-19 | 北京百度网讯科技有限公司 | Singing posture correcting method and device, electronic equipment and storage medium |
| CN113298013A (en) * | 2021-06-08 | 2021-08-24 | Tcl通讯（宁波）有限公司 | Motion correction method, motion correction device, storage medium and electronic equipment |
| CN113538873A (en) * | 2021-07-28 | 2021-10-22 | 东莞全芯物联科技有限公司 | AI position of sitting corrects camera based on image recognition technology |
| CN114220123B (en) * | 2021-12-10 | 2022-11-04 | 江苏泽景汽车电子股份有限公司 | Posture correction method and device, projection equipment and storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104200491A (en) * | 2014-08-15 | 2014-12-10 | 浙江省新华医院 | Motion posture correcting system for human body |
| CN107506707A (en) * | 2016-11-30 | 2017-12-22 | 奥瞳系统科技有限公司 | Using the Face datection of the small-scale convolutional neural networks module in embedded system |
| CN107895150A (en) * | 2016-11-30 | 2018-04-10 | 奥瞳系统科技有限公司 | Face datection and head pose angle based on the small-scale convolutional neural networks module of embedded system are assessed |
| CN108062536A (en) * | 2017-12-29 | 2018-05-22 | 纳恩博（北京）科技有限公司 | A kind of detection method and device, computer storage media |
| CN108182456A (en) * | 2018-01-23 | 2018-06-19 | 哈工大机器人（合肥）国际创新研究院 | A kind of target detection model and its training method based on deep learning |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| TW201123031A (en) * | 2009-12-24 | 2011-07-01 | Univ Nat Taiwan Science Tech | Robot and method for recognizing human faces and gestures thereof |
| WO2012039467A1 (en) * | 2010-09-22 | 2012-03-29 | パナソニック株式会社 | Exercise assistance system |
-
2018
- 2018-07-24 CN CN201810820856.1A patent/CN109274883B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104200491A (en) * | 2014-08-15 | 2014-12-10 | 浙江省新华医院 | Motion posture correcting system for human body |
| CN107506707A (en) * | 2016-11-30 | 2017-12-22 | 奥瞳系统科技有限公司 | Using the Face datection of the small-scale convolutional neural networks module in embedded system |
| CN107895150A (en) * | 2016-11-30 | 2018-04-10 | 奥瞳系统科技有限公司 | Face datection and head pose angle based on the small-scale convolutional neural networks module of embedded system are assessed |
| CN108062536A (en) * | 2017-12-29 | 2018-05-22 | 纳恩博（北京）科技有限公司 | A kind of detection method and device, computer storage media |
| CN108182456A (en) * | 2018-01-23 | 2018-06-19 | 哈工大机器人（合肥）国际创新研究院 | A kind of target detection model and its training method based on deep learning |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109274883A (en) | 2019-01-25 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109274883B (en) | Posture correction method, device, terminal and storage medium | |
| CN109117753B (en) | Part recognition method, device, terminal and storage medium | |
| US9183431B2 (en) | Apparatus and method for providing activity recognition based application service | |
| CN110688929B (en) | Human skeleton joint point positioning method and device | |
| CN110427900B (en) | Method, device and equipment for intelligently guiding fitness | |
| CN112819852A (en) | Evaluating gesture-based motion | |
| CN110544301A (en) | Three-dimensional human body action reconstruction system, method and action training system | |
| CN110428486B (en) | Virtual interaction fitness method, electronic equipment and storage medium | |
| CN110544302A (en) | Human body action reconstruction system and method based on multi-view vision and action training system | |
| JP2009157767A (en) | Face image recognition apparatus, face image recognition method, face image recognition program, and recording medium recording this program | |
| CN112163479A (en) | Motion detection method, motion detection device, computer equipment and computer-readable storage medium | |
| CN112528812A (en) | Pedestrian tracking method, pedestrian tracking device and pedestrian tracking system | |
| CN112258555A (en) | Real-time attitude estimation motion analysis method, system, computer equipment and storage medium | |
| CN115131879B (en) | Action evaluation method and device | |
| CN110969110A (en) | Face tracking method and system based on deep learning | |
| CN108875506B (en) | Face shape point tracking method, device and system and storage medium | |
| CN113409651A (en) | Live broadcast fitness method and system, electronic equipment and storage medium | |
| CN113191200A (en) | Push-up test counting method, device, equipment and medium | |
| CN113255429A (en) | Method and system for estimating and tracking human body posture in video | |
| CN111353347B (en) | Action recognition error correction method, electronic device, and storage medium | |
| CN111428609A (en) | Human body posture recognition method and system based on deep learning | |
| CN116012942A (en) | Sign language teaching method, device, equipment and storage medium | |
| WO2021054217A1 (en) | Image processing device, image processing method and program | |
| CN114360052A (en) | Intelligent somatosensory coach system based on AlphaPose and joint point angle matching algorithm | |
| CN113963202A (en) | Skeleton point action recognition method and device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |