CN109711271A - A kind of action determination method and system based on joint connecting line - Google Patents
A kind of action determination method and system based on joint connecting line Download PDFInfo
- Publication number
- CN109711271A CN109711271A CN201811470739.3A CN201811470739A CN109711271A CN 109711271 A CN109711271 A CN 109711271A CN 201811470739 A CN201811470739 A CN 201811470739A CN 109711271 A CN109711271 A CN 109711271A
- Authority
- CN
- China
- Prior art keywords
- skeletal joint
- image
- training
- joint
- model
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Abstract
The invention discloses a kind of action determination method and system based on joint connecting line, described method includes following steps: step S1, skeletal joint identification model is established, and repetition training is carried out by training image, so that the skeletal joint identification model is able to achieve image skeletal joint automatic identification；Step S2 trains each training action image by obtaining, and establishes standard operation database and the corresponding standard section of each movement based on the skeletal joint identification model；Step S3 obtains multiple still images of each training action of user, is judged based on the skeletal joint identification model and standard operation database and the corresponding standard section of each movement the movement of user, and the present invention can make the judgement of movement more acurrate effectively.
Description
Technical field
The present invention relates to image recognition processing technical fields, judge more particularly to a kind of movement based on joint connecting line
Method and system.
Background technique
Movement, is the kinematic system for having certain motivation and purpose and being directed toward certain object, ceases manner of breathing with all sectors of society
It closes, such as: the movement of mankind's items, the various runnings of machine.
In reality, either the mankind or machine, the assessment of movement are generally sentenced by the naked eyes of related personnel and experience
It is disconnected.Since assessment is not able to satisfy accurate, specification, perfect strict demand, the performance level of movement is also tended to not as good as people's will.This
Outside, since the assessment of movement depends on the site supervision of manpower, it cannot achieve intelligent automation, it is difficult to socially popularize.
With the continuous development of computer technology, based on the above issues, a series of actions assessment product has been emerged in large numbers in market.Root
It studies according to investigations, the core of current movement assessment product movement judgement is mainly to find out the difference of implementation movement and standard operation
It is different, and its movement is compared and is generally all based on generally speaking, such as: human body left upper extremity often ignores part, such as the judgement of left wrist,
It is easy to cause the situation of implementation movement position identification mistake to occur in this way, in addition, the same implementation position, the implementation that system obtains
The difference that movement will also tend to be formed skeletal joint by the position puts influence, the reason is that each skeletal joint difference is put,
Affiliated central point is also different；And current product does not account for the deficiency of this respect generally, leads to implementation movement and standard operation
Difference inaccuracy.
Summary of the invention
In order to overcome the deficiencies of the above existing technologies, purpose of the present invention is to provide a kind of based on joint connecting line
Action determination method and system, so that the judgement of movement is more acurrate effectively.
In order to achieve the above object, the present invention proposes a kind of action determination method based on joint connecting line, include the following steps:
Step S1 establishes skeletal joint identification model, and carries out repetition training by training image, so that the bone closes
Section identification model is able to achieve image skeletal joint automatic identification；
Step S2 trains each training action image by obtaining, and it is dynamic to establish standard based on the skeletal joint identification model
Make database and the corresponding standard section of each movement；
Step S3 obtains multiple still images of each training action of user, based on the skeletal joint identification model and
Standard operation database and the corresponding standard section of each movement judge the movement of user.
Preferably, step S1 further comprises:
Step S100 establishes skeletal joint sample by obtaining the still image of different angle of each human body skeletal joint point
Database；
Step S101 establishes skeletal joint identification model, passes through the bone of the sample image to skeletal joint sample database
Bone joint carries out rectangle frame label, obtains each parameter of each skeletal joint indicia framing of each image, and utilize skeletal joint sample
The training image of database carries out repetition training to the skeletal joint identification model established, and makes it that can realize oneself of skeletal joint
Dynamic identification.
Preferably, step S100 further comprises:
Step S100a obtains the static state of the different angle of each human body skeletal joint point based on the distribution in skeleton joint
Image, and gray proces are carried out to it；
Step S100b, the total quantity based on gray level image, divides the image into training image and two class of test image is deposited
Storage, to establish the skeletal joint sample database.
Preferably, step S101 further comprises:
Step S101a makees rectangle frame label to the skeletal joint of all sample images of skeletal joint sample database, obtains
To each skeletal joint indicia framing coordinate, width and height parameter；
Step S101b establishes skeletal joint identification model, using the skeletal joint sample database training image as mould
The input picture of type training carries out repetition training, obtains target prediction frame parameter, realize the automatic identification of skeletal joint.
Preferably, after step S101b, further includes:
After model training stopping, model and skeletal joint sample database test image are subjected to trial operation test, root
Model recognition accuracy is obtained according to loss function, and final skeletal joint identification model is determined according to model recognition accuracy.
Preferably, if the model recognition accuracy reaches preset threshold, it is determined that currently skeletal joint identification model is
Final model, and the skeletal joint identification model is applied to on-site identification；Otherwise, prototype network structure is readjusted,
And skeletal joint sample database training image repetition training is continued with, or increase training figure in skeletal joint sample database
As continuing repetition training, until model recognition accuracy reaches preset threshold.
Preferably, step S2 further comprises:
Step S200 obtains multiple still images for training each training action by image collecting device；
Step S201 trains the skeletal joint of each training action image based on skeletal joint identification model identification, defeated
Target prediction frame parameter out；
Step S202 converts the target prediction frame parameter to the output of above-mentioned model, obtains each image bone
The affiliated center point coordinate in joint is based on above-mentioned central point, the adjacent skeletal joint of each image is connected two-by-two, forms a large amount of human bodies
Skeletal joint figure establishes standard operation database；
Step S203 is based on identical skeletal joint, is formed by angle with its connecting line to each training action of coach
Each repetitive operation is analyzed, obtains the corresponding standard section of each movement.
Preferably, step S3 further comprises:
Step S300 obtains multiple still images of each training action of user by image collecting device；
Step S301, it is defeated based on the skeletal joint of skeletal joint identification model identification each training action image of user
Target prediction frame parameter out, and the target prediction frame parameter to the output of above-mentioned model is converted, obtain each image bone
The affiliated center point coordinate in bone joint is based on above-mentioned central point, the adjacent skeletal joint of each image is connected two-by-two, forms user's bone
Bone joint figure；
User's skeletal joint figure skeletal joint figure corresponding with standard operation database is compared step S302, defeated
Comparison result out.
Preferably, in step S302, according to sequence from top to bottom, from left to right user's skeletal joint figure and standard
The corresponding skeletal joint figure of action database is compared, connecting line identical for skeletal joint, is overlapped based on a line segment,
Then judge that another line segment connected to it is formed by whether angle exceeds standard section；If so, prompt mistake, and will use
Family movement is distinctly displayed with standard operation；Otherwise, then it is considered as standard operation, does not prompt.
In order to achieve the above objectives, the present invention also provides a kind of movements based on joint connecting line to judge system, comprising:
Skeletal joint identification model is established and training unit, for establishing skeletal joint identification model, and passes through training figure
As carrying out repetition training, so that the skeletal joint identification model is able to achieve image skeletal joint automatic identification；
Unit is established in standard operation database and each action criteria section, for training each training action figure by obtaining
Picture establishes standard operation database and the corresponding standard section of each movement based on the skeletal joint identification model；
Judging unit is acted, for obtaining multiple still images of each training action of user, is known based on the skeletal joint
Other model and standard operation database and the corresponding standard section of each movement judge the movement of user.
Compared with prior art, a kind of action determination method and system based on joint connecting line of the present invention is based on human body bone
The distribution in bone joint establishes the characteristic information of each skeletal joint multi-angle of skeletal joint identification model repetition training, realizes figure
As skeletal joint automatic identification, so that movement judge careful to each skeletal joint, avoids implementation from acting position and easily identify mistake
The case where occur, make movement judge it is more acurrate, comprehensive, meanwhile, the present invention is based on the affiliated central points of each skeletal joint, adjacent bone
Bone joint connects two-by-two, forms skeleton joint figure, thus using central point as tie point, regardless of each skeletal joint of human body
It puts, also can accurately obtain and be formed by angle between current implementation movement and its skeletal joint, it is sufficient to it is any dynamic to cope with human body
Implementation movement is simplified to skeletal joint figure by the acquisition of work, and movement convenient to carry out is compared with standard operation, improves computer picture
Treatment effeciency and saving memory space.
Detailed description of the invention
Fig. 1 is a kind of step flow chart of the action determination method based on joint connecting line of the present invention；
Fig. 2 is the detailed flowchart of step S1 in the specific embodiment of the invention；
Fig. 3 is the detailed flowchart of step S101 in the specific embodiment of the invention；
Fig. 4 is the detailed flowchart of step S2 in the specific embodiment of the invention；
Fig. 5 is the detailed flowchart of step S3 in the specific embodiment of the invention；
Fig. 6 is a kind of system architecture diagram of the movement judgement system based on joint connecting line of the present invention；
Fig. 7 is the foundation and training, testing process schematic diagram of skeletal joint identification model in the specific embodiment of the invention；
Fig. 8 is that the movement based on joint connecting line based on skeletal joint identification model in the specific embodiment of the invention is sentenced
Disconnected process schematic.
Specific embodiment
Below by way of specific specific example and embodiments of the present invention are described with reference to the drawings, those skilled in the art can
Understand further advantage and effect of the invention easily by content disclosed in the present specification.The present invention can also pass through other differences
Specific example implemented or applied, details in this specification can also be based on different perspectives and applications, without departing substantially from
Various modifications and change are carried out under spirit of the invention.
Fig. 1 is a kind of step flow chart of the action determination method based on joint connecting line of the present invention.As shown in Figure 1, this
A kind of action determination method based on joint connecting line is invented, is included the following steps:
Step S1 establishes skeletal joint identification model, and carries out repetition training by training image, so that the bone closes
Section identification model is able to achieve image skeletal joint automatic identification.
Specifically, as shown in Fig. 2, step S1 further comprises:
Step S100 establishes skeletal joint sample database.
Specifically, step S100 further comprises:
Step S100a obtains each human body skeletal joint point (such as: head, left wrist, the right side based on the distribution in skeleton joint
Ankle etc.) different angle still image, and gray proces are carried out to it.Specifically, big from different perspectives using camera
Amount captures the still image of above-mentioned skeletal joint point, and carries out gray proces to it, and skeletal joint image is passed through gray proces
After become gray level image.The present invention carries out subsequent processing by first carrying out gray proces to image again, not only improves image procossing
Efficiency, and will not influence image texture characteristic.
Step S100b, the total quantity based on gray level image, divides the image into training image and two class of test image is deposited
Storage, to establish the skeletal joint sample database；Wherein, the former is used for model training, and the latter is used for model measurement, so far,
The foundation of skeletal joint sample database finishes.In the specific embodiment of the invention, gray level image is stored on local server.
Step S101 establishes skeletal joint identification model, passes through the bone of the sample image to skeletal joint sample database
Bone joint carries out rectangle frame label, obtains each parameter of each skeletal joint indicia framing of each image, and utilize skeletal joint sample
The training image of database carries out repetition training to the skeletal joint identification model established, and makes it that can realize oneself of skeletal joint
Dynamic identification.
Specifically, as shown in figure 3, step S101 further comprises:
Step S101a makees rectangle frame label to the skeletal joint of all sample images of skeletal joint sample database, obtains
To each skeletal joint indicia framing coordinate, width and height parameter.
Based on the skeletal joint sample database that step S100 is established, a large amount of skeleton arthrosis images have been obtained.Due to
Skeletal joint is together with human body adjacent thereto, it is difficult to realize that specific aim is captured, such as:, also can be left small when shooting left wrist
A part of arm and left hand is shot together.In order to ensure the quality of model training sample, first by label program, to all samples
The skeletal joint of this image carries out rectangle frame label, and the corresponding skeletal joint of a sample image obtains the affiliated bone of each sample
Indicia framing coordinate, width and the height parameter in bone joint.
Step S101b establishes skeletal joint identification model, using the skeletal joint sample database training image as mould
The input picture of type training carries out repetition training, obtains target prediction frame parameter, realize the automatic identification of skeletal joint.
In the specific embodiment of the invention, skeletal joint identification model uses SSD (Single Shot MultiBox
Detector, single object detector) algorithm realization.The direct predicted boundary frame coordinate of the algorithm and class detection, do not have
Proposal process is generated, model structure is simpler, and detection speed is faster.For different size of object detection, traditional method
Different size is usually converted images into, is then handled respectively, finally integrates result；And SSD algorithm utilizes difference
The feature map of convolutional layer, which carries out synthesis, can also reach effect same, it is ensured that object detection accuracy.
Specifically, in step S101b, skeletal joint identification model is established, SSD algorithm is based on, skeletal joint sample
Input picture of the training image of database as model training, above-mentioned input picture, image skeletal joint indicia framing parameter
And image skeletal joint title is transferred to model repetition training, obtains target prediction frame parameter (that is: coordinate, width and height
Degree), realize the effect based on object coordinates, classification detection identification skeletal joint.Generally, in step S101b, when model with
When the number of skeletal joint sample database training image repetition training reaches preset threshold, such as 200,000 times, model training stops.
Preferably, further include following steps after step S101b:
Model and skeletal joint sample database test image are carried out test run after model training stopping by step S101c
Row test, obtains model recognition accuracy according to loss function, determines that final skeletal joint is known according to model recognition accuracy
Other model.
Specifically, after being stopped according to step S101b model training, model and skeletal joint sample database test image
Trial operation test is carried out, model recognition accuracy is obtained according to loss function.If accuracy rate reaches preset threshold, such as: 80%, then
It determines that current skeletal joint identification model is final model, and model is applied to on-site identification；Otherwise, model is readjusted
Network structure and skeletal joint sample database training image repetition training is continued with, or in skeletal joint sample database
Increase training image and continues repetition training until model recognition accuracy reaches preset threshold.In the specific embodiment of the invention,
Adjustment for prototype network structure, can be accurate come the model identification for improving model training by adjusting the number of plies of network structure
Rate, such as: network structure originally is 16 layers, since recognition accuracy test is lower than certain threshold values, is then needed to current network knot
The hidden layer of structure is extended, even if the number of plies of network structure becomes greater than 16, but invention is not limited thereto.
In the specific embodiment of the invention, loss function uses following formula:
Wherein, x is the Jaccard coefficient to match, and c is confidence level, and l is prediction block, and g is indicia framing, and N is prediction block number
Amount, conf are confidence loss, and loc is positioning loss, and α is weight term, default setting 1.
Step S2 trains each training action image by obtaining, and it is dynamic to establish standard based on the skeletal joint identification model
Make database and the corresponding standard section of each movement.
Specifically, as shown in figure 4, step S2 further comprises:
Step S200 obtains multiple still images for training each training action by image collecting device, has in the present invention
In body embodiment, image collecting device uses camera assembly, that is to say, that in the field, religion is largely captured by camera assembly
Practice each training action video, extract video and extracts multiple still images in the form of frame to take out.The camera assembly is imaged by 5
Head composition is arranged in surface, front, dead astern, front-left and the front-right of human body, and it is polygonal that realization acts details
Degree is captured.
Step S201 trains the skeletal joint of each training action image based on skeletal joint identification model identification, defeated
Target prediction frame parameter out.
Step S202 converts the target prediction frame parameter to the output of above-mentioned model, obtains each image bone
The affiliated center point coordinate in joint is based on above-mentioned central point, the adjacent skeletal joint of each image is connected two-by-two, forms a large amount of human bodies
Skeletal joint figure establishes standard operation database.
Since a skeletal joint difference is put, such as: side, front, the affiliated central point of the skeletal joint are also different.Cause
This, can be by existing tool, such as Photoshop, to upper in step S202 in order to keep the implementation obtained movement more acurrate
The target prediction frame parameter for stating model output is converted, and obtains the affiliated center point coordinate of each identification image skeletal joint, then
Based on above-mentioned central point, the adjacent skeletal joint of each image is connected two-by-two, forms a large amount of skeleton joints figure, standard operation
Database finishes.
Step S203 is based on identical skeletal joint, is formed by angle with its connecting line to each training action of coach
Each repetitive operation is analyzed, obtains the corresponding standard section of each movement.
N times can be repeated due to training each training action, in step S203, it is based on identical skeletal joint, with
Its connecting line is formed by angle to analyze each repetitive operation, obtains the corresponding standard section of each movement.
Step S3 obtains multiple still images of each training action of user, based on the skeletal joint identification model and
Standard operation database and the corresponding standard section of each movement judge the movement of user.
Specifically, as shown in figure 5, step S3 further comprises:
Step S300 obtains multiple still images of each training action of user by image collecting device, has in the present invention
In body embodiment, image collecting device uses camera assembly, that is to say, that in the field, the religion that user has recorded according to terminal
Journey is trained, and largely captures each training action video of user by camera assembly, extract video and by take out extract in the form of frame it is more
Still image, the camera assembly are made of 5 cameras, be arranged in the surface of human body, front, just after
Side, front-left and front-right.
Step S301, it is defeated based on the skeletal joint of skeletal joint identification model identification each training action image of user
Target prediction frame parameter out, and the target prediction frame parameter to the output of above-mentioned model is converted, obtain each image bone
The affiliated center point coordinate in bone joint is based on above-mentioned central point, the adjacent skeletal joint of each image is connected two-by-two, forms user's bone
Bone joint figure.That is, identifying the skeletal joint of each training action image of user by skeletal joint identification model；Pass through
Existing tool, such as photoshop conversion, obtain each affiliated center point coordinate of image skeletal joint, and be based on above-mentioned center
Point two-by-two connects the identified adjacent skeletal joint of image, forms user's skeletal joint figure.
User's skeletal joint figure skeletal joint figure corresponding with standard operation database is compared step S302, defeated
Comparison result out.
Since the connection of 2 skeletal joints can form a line segment, the connection of two lines section can shape in an angle, therefore in step
In rapid S302, according to sequence from top to bottom, from left to right user's skeletal joint figure bone corresponding with standard operation database
Bone joint figure is compared, connecting line identical for skeletal joint, is overlapped based on a line segment, then judges connected to it another
One line segment is formed by whether angle exceeds standard section；If so, prompt mistake, user action solid line is shown, standard
Movement dotted line is shown, in order to user's difference；Otherwise, then it is considered as standard operation, does not prompt.For malfunction, Yong Huxiu
Direct action, camera assembly obtain in real time, terminal real-time update comparison result.
Fig. 6 is a kind of system architecture diagram of the movement judgement system based on joint connecting line of the present invention.As shown in fig. 6, this
It invents a kind of movement based on joint connecting line and judges system, comprising:
Skeletal joint identification model is established and training unit 601, for establishing skeletal joint identification model, and passes through training
Image carries out repetition training, so that the skeletal joint identification model is able to achieve image skeletal joint automatic identification.
Specifically, skeletal joint identification model is established and training unit 601 further comprises:
Skeletal joint sample database establishes unit, for establishing skeletal joint sample database.
Specifically, skeletal joint sample database establishes the distributed acquisition human body bone that unit is primarily based on skeleton joint
The still image of the different angle of bone artis (such as: head, left wrist, right ankle), and gray proces are carried out to it, it is then based on
The total quantity of gray level image, divides the image into training image and two class of test image is stored, to establish the skeletal joint
Sample database；Wherein, the former is used for model training, and the latter is used for model measurement.Specifically, using camera from different angles
Degree largely captures the still image of above-mentioned skeletal joint, and gray proces are carried out to it, and skeletal joint sample database is established single
Skeletal joint image is become gray level image by member after gray proces.The present invention by image is first carried out gray proces again into
Row subsequent processing not only improves image processing efficiency, and will not influence image texture characteristic.In the specific embodiment of the invention
In, gray level image is stored on local server
Model foundation training unit, for establishing skeletal joint identification model, by skeletal joint sample database
The skeletal joint of sample image carries out rectangle frame label, obtains each parameter of each skeletal joint indicia framing of each image, and utilize
The training image of skeletal joint sample database carries out repetition training to the skeletal joint identification model established, and realize it can
The automatic identification of skeletal joint.
In the specific embodiment of the invention, model foundation training unit is specifically used for:
Rectangle frame label is made to the skeletal joint of all sample images of skeletal joint sample database, obtains each bone and closes
Shackle mark frame coordinate, width and height parameter.
Skeletal joint identification model is established, using the skeletal joint sample database training image as the defeated of model training
Enter image, carries out repetition training, target prediction frame parameter is obtained, to realize the automatic identification of skeletal joint.Of the invention specific
In embodiment, skeletal joint identification model uses SSD (Single Shot MultiBox Detector, single target detection
Device) algorithm realization.The direct predicted boundary frame coordinate of the algorithm and class detection, do not generate proposal process, model
Structure is simpler, and detection speed is faster.That is, establishing skeletal joint identification model, it is then based on SSD algorithm, bone
Input picture of the training image of joint sample database as model training, above-mentioned input picture, image skeletal joint mark
Note frame parameter and image skeletal joint title are transferred to model repetition training, obtain target prediction frame parameter (that is: coordinate, width
Degree and height), realize the effect based on object coordinates, classification detection identification skeletal joint.Generally, when model and bone close
When the number of section sample database training image repetition training reaches preset threshold, such as 200,000 times, model training stops.
After model training stopping, model and skeletal joint sample database test image are subjected to trial operation test, root
Model recognition accuracy is obtained according to loss function, and final skeletal joint identification model is determined according to model recognition accuracy.?
That is model and skeletal joint sample database test image are carried out trial operation test after model training stops, according to
Loss function obtains model recognition accuracy.If accuracy rate reaches preset threshold, such as: 80%, it is determined that current skeletal joint is known
Other model is final model, and model is applied to on-site identification；Otherwise, prototype network structure is readjusted, and continues benefit
With skeletal joint sample database training image repetition training, or increases training image in skeletal joint sample database and continue
Repetition training is until model recognition accuracy reaches preset threshold.
Unit 602 is established in standard operation database and each action criteria section, for training each training action by obtaining
Image establishes standard operation database and the corresponding standard section of each movement based on the skeletal joint identification model.
Specifically, standard operation database and each action criteria section establish unit 602 and further comprise:
Image acquisition units, for obtaining multiple still images for training each training action by image collecting device,
In the specific embodiment of the invention, image collecting device uses camera assembly, that is to say, that in the field, image acquisition units are logical
It crosses camera assembly and largely captures each training action video of coach, extract video and extract multiple still images in the form of frame to take out.Institute
It states camera assembly to be made of 5 cameras, is arranged in the surface of human body, front, dead astern, front-left and just
Right, realization act details multi-angle and capture.
Skeletal joint recognition unit, for training each training action image based on skeletal joint identification model identification
Skeletal joint exports target prediction frame parameter.
Standard operation Database unit, for turning to the target prediction frame parameter to the output of above-mentioned model
It changes, obtains each affiliated center point coordinate of image skeletal joint, be based on above-mentioned central point, two-by-two the adjacent skeletal joint of each image
Connection, forms a large amount of skeleton joints figure, establishes standard operation database.
Since a skeletal joint difference is put, such as: side, front, the affiliated central point of the skeletal joint are also different.Cause
This, in order to make obtain implementation movement it is more acurrate, standard operation Database unit can by existing tool, such as
Photoshop converts the target prediction frame parameter of above-mentioned model output, obtains in belonging to each identification image skeletal joint
Heart point coordinate, is then based on above-mentioned central point, and the adjacent skeletal joint of each image is connected two-by-two, forms a large amount of skeletons and closes
Section figure, to establish standard operation database.
Each action criteria interval determination unit is based on identical skeletal joint, for each training action to coach with it
Connecting line is formed by angle to analyze each repetitive operation, obtains the corresponding standard section of each movement.
N times can be repeated due to training each training action, in step S203, it is based on identical skeletal joint, with
Its connecting line is formed by angle to analyze each repetitive operation, obtains the corresponding standard section of each movement.
Judging unit 603 is acted, for obtaining multiple still images of each training action of user, is based on the skeletal joint
Identification model and standard operation database and the corresponding standard section of each movement judge the movement of user.
Specifically, movement judging unit 603 further comprises:
Image acquisition units obtain multiple still images of each training action of user by image collecting device, in this hair
In bright specific embodiment, image collecting device uses camera assembly, that is to say, that in the field, user has recorded according to terminal
Study course be trained, largely capture each training action video of user by camera assembly, extract video and mentioned in the form of frame by taking out
Multiple still images are taken, the camera assembly is made of 5 cameras, is arranged in the surface of human body, front, just
Rear, front-left and front-right.
User's skeletal joint figure acquiring unit, for identifying each training action of user based on the skeletal joint identification model
The skeletal joint of image exports target prediction frame parameter, and carries out to the target prediction frame parameter to the output of above-mentioned model
Conversion, obtains each affiliated center point coordinate of image skeletal joint, above-mentioned central point is based on, the adjacent skeletal joint two of each image
Two connections, form user's skeletal joint figure.
Comparing unit, for comparing user's skeletal joint figure skeletal joint figure corresponding with standard operation database
It is right, export comparison result.
Since the connection of 2 skeletal joints can form a line segment, the connection of two lines section can shape in an angle, therefore compare
Unit is according to sequence from top to bottom, from left to right user's skeletal joint figure skeletal joint corresponding with standard operation database
Figure is compared, connecting line identical for skeletal joint, is overlapped based on a line segment, then judges another line connected to it
Section is formed by whether angle exceeds standard section；If so, prompt mistake, user action solid line is shown, standard operation is empty
Line is shown, in order to user's difference；Otherwise, then it is considered as standard operation, does not prompt.For malfunction, user's corrective action,
Camera assembly obtains in real time, terminal real-time update comparison result.
Fig. 7 is the foundation and training, testing process schematic diagram of skeletal joint identification model in the specific embodiment of the invention, figure
8 be the movement deterministic process signal based on joint connecting line based on skeletal joint identification model in the specific embodiment of the invention
Figure.Illustrate process of the invention below in conjunction with Fig. 7 and Fig. 8:
S1 establishes skeletal joint sample database.
In the specific embodiment of the invention, based on the distribution in skeleton joint, human body is defined by 20 skeletal joints
Composition, be respectively: skull (head), cervical vertebra (neck), thoracic vertebrae (chest), lumbar vertebrae (waist), left shoulder, right shoulder, left elbow, right elbow,
Left wrist, right wrist, left phalanges (left hand), right phalanges (right hand), left hip, right hip, left knee, right knee, left ankle, right ankle, left phalanx are (left
Foot), right phalanx (right crus of diaphragm).
Based on the distribution in skeleton joint, the static state of above-mentioned skeletal joint is largely captured from different perspectives by camera
Image extracts image and carries out gray proces to it；Wherein, gray proces not only improve image processing efficiency, and will not shadow
Ring image texture characteristic.
Above-mentioned skeletal joint image passes through gray proces, becomes grayscale image.Based on grayscale image total quantity, system is image point
It is these two types of at training image and test image；Wherein, the former is used for model training, and the latter is used for model measurement, and grayscale image is deposited
On local server, the foundation of skeletal joint sample database is finished for storage.
S2: establishing model, realizes skeletal joint identification.
S2.1, by marking program to obtain indicia framing coordinate, width and the height parameter of each sample image skeletal joint.
Based on skeletal joint sample database, a large amount of skeleton arthrosis images have been obtained.Due to skeletal joint be together with
Human body adjacent thereto, it is difficult to realize that specific aim is captured, such as:, also can be one of left forearm and left hand when shooting left wrist
Divide and shoots together.In order to ensure the quality of model training sample, system closes the bone of all sample images by label program
Section makees rectangle frame label, and the corresponding skeletal joint of a sample image show that the indicia framing of the affiliated skeletal joint of each sample is sat
Mark, width and height parameter.
S2.2 establishes model, with training image repetition training, realizes image skeletal joint automatic identification.
In the specific embodiment of the invention, skeletal joint identification model uses SSD algorithm.The direct predicted boundary frame of the algorithm
Coordinate and class detection, do not generate proposal process, and model structure is simpler, and detection speed is faster.In step
In S2.2, model is established, SSD algorithm is based on, using skeletal joint sample database training image as the input figure of model training
Picture is transferred to above-mentioned input picture, image skeletal joint indicia framing parameter and image skeletal joint title model and instructs repeatedly
Practice, obtain target prediction frame parameter (that is: coordinate, width and height), realizes based on object coordinates, classification detection identification bone
The effect in joint.It is the calculation formula of target prediction frame relevant parameter below:
It is the calculation formula of target prediction frame relevant parameter in the specific embodiment of the invention below:
1, the calculation formula of prediction block size:
Wherein, SminIt is 0.2, indicates that bottom size is 0.2；SmaxIt is 0.9, indicates that top size is 0.9；M is represented
Characteristic pattern (feature map) number.
2, following data is the aspect ratio obtained based on prediction block size, one shares 5 kinds of aspect ratios here.
3, prediction block width calculation formula:
4, prediction block height calculation formula is calculated:
Whether S2.3, training of judgement reach system setting threshold value, if not having, return step S2.2 continues to train, otherwise
Enter step S2.4.That is
When the number of model and the training image repetition training of skeletal joint sample database reaches default threshold values, such as:
200000 times, model training stops.
Model and skeletal joint sample database test image are carried out trial operation test by step S2.4, according to loss letter
Number obtains model recognition accuracy.
Step S2.5, whether judgment models recognition accuracy reaches system setting threshold value, if model recognition accuracy reaches
A certain threshold values, such as: 80%, then model is applied to on-site identification；Otherwise, prototype network structure, and continuation and bone are readjusted
The sample database training image repetition training of bone joint, or increase training image in skeletal joint sample database and continue repeatedly
Training is until model recognition accuracy reaches preset threshold.Loss function uses following formula:
Wherein, x is the Jaccard coefficient to match, and c is confidence level, and l is prediction block, and g is indicia framing, and N is prediction block number
Amount, conf are confidence loss, and loc is positioning loss, and α is weight term, default setting 1.
S3 is largely captured each training action video of coach by camera assembly, is based on model and ready-made tool, obtains image
The skeletal joint of identification and its affiliated central point.That is, in the field, it is each largely to capture coach by camera assembly first
Training action video, system extract video and extract multiple still images in the form of frame to take out.The camera assembly is imaged by 5
Head composition is arranged in surface, front, dead astern, front-left and the front-right of human body, and it is polygonal that realization acts details
Degree is captured；It is then based on model, the skeletal joint of each training action image is trained in identification；Since a skeletal joint difference is put
It puts, such as: side, front, the affiliated central point of the skeletal joint is also different, in order to keep the implementation obtained movement more acurrate, by existing
At tool, such as: Photoshop converts the target prediction frame parameter of above-mentioned model output, obtains each identification image bone
The affiliated center point coordinate in joint；
Step S4 is based on above-mentioned central point, the adjacent skeletal joint of each image is connected two-by-two, obtains a large amount of standard operations
Skeletal joint figure establishes standard operation database and each action criteria section.
In the specific embodiment of the invention, head is turned to and is judged, according to the eyes size of the same horizontal position
Rotation direction and angle are defined, such as: head turns to the left side, and left eye is smaller than right eye, obtains according to current two sizes
Rotational angle.
N times can be repeated due to training each training action, system is based on identical skeletal joint, is formed with its connecting line
Angle analyze each repetitive operation, obtain the corresponding standard section of each movement.
S5, largely captured by camera assembly, model identification, ready-made tool conversion and adjacent skeletal joint connection, obtain
To user's skeletal joint figure.That is, in the field, user is trained according to the study course that terminal has been recorded.Pass through camera shooting
Component largely captures each training action video of user, and system extracts video and extracts multiple still images in the form of frame to take out.It is described
Camera assembly is made of 5 cameras, is arranged in surface, front, dead astern, front-left and the positive right side of human body
Side；By model, the skeletal joint of each training action image of user is identified；It is converted by ready-made tool, obtains each image bone
The affiliated center point coordinate in joint.Based on above-mentioned central point, the identified adjacent skeletal joint of image is connected two-by-two, forms user
Skeletal joint figure.
User's skeletal joint figure is compared S6 with corresponding standard operation skeletal joint figure.Due to 2 skeletal joints
Connection can form a line segment, two lines section connection can shape in an angle, therefore, can be according to from top to bottom, from left to right
User's skeletal joint figure skeletal joint figure corresponding with standard operation database is compared in sequence, identical for skeletal joint
Connecting line, be overlapped based on line segment, judge that another line segment connected to it is formed by whether angle exceeds standard regions
Between；If so, terminal notifying mistake, user action solid line is shown, standard operation dotted line is shown, is distinguished convenient for user；Otherwise, it is considered as
Standard operation does not prompt, and for malfunction, user's corrective action, camera assembly obtains in real time, and terminal real-time update compares
As a result.
In conclusion a kind of action determination method and system based on joint connecting line of the present invention is based on skeleton joint
Distribution, establish the characteristic information of each skeletal joint multi-angle of skeletal joint identification model repetition training, realize image bone
Joint automatic identification avoids implementation movement position from easily identifying wrong situation so that movement judge careful to each skeletal joint
Occur, judges movement more acurrate, comprehensive, meanwhile, the present invention is based on the affiliated central points of each skeletal joint, adjacent skeletal joint
It connects two-by-two, forms skeleton joint figure, thus using central point as tie point, no matter how each skeletal joint of human body is put,
Also it can accurately obtain between current implementation movement and its skeletal joint and be formed by angle, it is sufficient to cope with obtaining for any movement of human body
It takes, implementation movement is simplified to skeletal joint figure, movement convenient to carry out is compared with standard operation, improves Computer Image Processing effect
Rate and saving memory space.
Invention is not limited to the movement judgement of mankind's items movement, can also be applied in the various runnings of machine, to reach
Judge machine whether the effect of normal operation.
The above-described embodiments merely illustrate the principles and effects of the present invention, and is not intended to limit the present invention.Any
Without departing from the spirit and scope of the present invention, modifications and changes are made to the above embodiments by field technical staff.Therefore,
The scope of the present invention, should be as listed in the claims.
Claims (10)
1. a kind of action determination method based on joint connecting line, includes the following steps:
Step S1 establishes skeletal joint identification model, and carries out repetition training by training image, so that the skeletal joint is known
Other model is able to achieve image skeletal joint automatic identification；
Step S2 trains each training action image by obtaining, establishes standard operation number based on the skeletal joint identification model
According to library and the corresponding standard section of each movement；
Step S3 obtains multiple still images of each training action of user, is based on the skeletal joint identification model and standard
Action database and the corresponding standard section of each movement judge the movement of user.
2. a kind of action determination method based on joint connecting line as described in claim 1, which is characterized in that step S1 is into one
Step includes:
Step S100 establishes skeletal joint sample data by obtaining the still image of different angle of each human body skeletal joint point
Library；
Step S101 establishes skeletal joint identification model, is closed by the bone of the sample image to skeletal joint sample database
Section carries out rectangle frame label, obtains each parameter of each skeletal joint indicia framing of each image, and utilize skeletal joint sample data
The training image in library carries out repetition training to the skeletal joint identification model established, and makes it that can realize the automatic knowledge of skeletal joint
Not.
3. a kind of action determination method based on joint connecting line as claimed in claim 2, which is characterized in that step S100 into
One step includes:
Step S100a obtains the static map of the different angle of each human body skeletal joint point based on the distribution in skeleton joint
Picture, and gray proces are carried out to it；
Step S100b, the total quantity based on gray level image, divides the image into training image and two class of test image is stored,
To establish the skeletal joint sample database.
4. a kind of action determination method based on joint connecting line as claimed in claim 2, which is characterized in that step S101 into
One step includes:
Step S101a makees rectangle frame label to the skeletal joint of all sample images of skeletal joint sample database, obtains each
Skeletal joint indicia framing coordinate, width and height parameter；
Step S101b establishes skeletal joint identification model, instructs the skeletal joint sample database training image as model
Experienced input picture carries out repetition training, obtains target prediction frame parameter, realize the automatic identification of skeletal joint.
5. a kind of action determination method based on joint connecting line as claimed in claim 4, which is characterized in that in step
After S101b, further includes:
After model training stopping, model and skeletal joint sample database test image are subjected to trial operation test, according to damage
It loses function call and goes out model recognition accuracy, final skeletal joint identification model is determined according to model recognition accuracy.
6. a kind of action determination method based on joint connecting line as claimed in claim 5, it is characterised in that: if the model
Recognition accuracy reaches preset threshold, it is determined that current skeletal joint identification model is final model, and the bone pass
It saves identification model and is applied to on-site identification；Otherwise, prototype network structure is readjusted, and continues with skeletal joint sample data
Library training image repetition training, or increase training image in skeletal joint sample database and continue repetition training, until model
Recognition accuracy reaches preset threshold.
7. a kind of action determination method based on joint connecting line as described in claim 1, it is characterised in that: step S2 is into one
Step includes:
Step S200 obtains multiple still images for training each training action by image collecting device；
Step S201 is trained the skeletal joint of each training action image based on skeletal joint identification model identification, exports mesh
Mark prediction block parameter；
Step S202 converts the target prediction frame parameter to the output of above-mentioned model, obtains each image skeletal joint
Affiliated center point coordinate is based on above-mentioned central point, the adjacent skeletal joint of each image is connected two-by-two, forms a large amount of skeletons
Joint figure establishes standard operation database；
Step S203 is based on identical skeletal joint, is formed by angle with its connecting line to divide to each training action of coach
Each repetitive operation is analysed, obtains the corresponding standard section of each movement.
8. a kind of action determination method based on joint connecting line as described in claim 1, it is characterised in that: step S3 is into one
Step includes:
Step S300 obtains multiple still images of each training action of user by image collecting device；
Step S301 exports mesh based on the skeletal joint of skeletal joint identification model identification each training action image of user
Prediction block parameter is marked, and the target prediction frame parameter to the output of above-mentioned model is converted, show that each image bone closes
Center point coordinate belonging to saving is based on above-mentioned central point, the adjacent skeletal joint of each image is connected two-by-two, forms user's bone and closes
Section figure；
User's skeletal joint figure skeletal joint figure corresponding with standard operation database is compared step S302, exports ratio
To result.
9. a kind of action determination method based on joint connecting line as claimed in claim 8, it is characterised in that: in step S302
In, according to sequence from top to bottom, from left to right user's skeletal joint figure skeletal joint corresponding with standard operation database
Figure is compared, connecting line identical for skeletal joint, is overlapped based on a line segment, then judges another line connected to it
Section is formed by whether angle exceeds standard section；If so, prompt mistake, and user action and standard operation are distinguished and shown
Show；Otherwise, then it is considered as standard operation, does not prompt.
10. a kind of movement based on joint connecting line judges system, comprising:
Skeletal joint identification model is established and training unit, for establishing skeletal joint identification model, and by training image into
Row repetition training, so that the skeletal joint identification model is able to achieve image skeletal joint automatic identification；
Unit is established in standard operation database and each action criteria section, for training each training action image, base by obtaining
Standard operation database and the corresponding standard section of each movement are established in the skeletal joint identification model；
Judging unit is acted, for obtaining multiple still images of each training action of user, mould is identified based on the skeletal joint
Type and standard operation database and the corresponding standard section of each movement judge the movement of user.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811470739.3A CN109711271A (en) | 2018-12-04 | 2018-12-04 | A kind of action determination method and system based on joint connecting line |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811470739.3A CN109711271A (en) | 2018-12-04 | 2018-12-04 | A kind of action determination method and system based on joint connecting line |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN109711271A true CN109711271A (en) | 2019-05-03 |
Family
ID=66254575
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811470739.3A Pending CN109711271A (en) | 2018-12-04 | 2018-12-04 | A kind of action determination method and system based on joint connecting line |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109711271A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2020252918A1 (en) * | 2019-06-20 | 2020-12-24 | 平安科技（深圳）有限公司 | Human body-based gesture recognition method and apparatus, device, and storage medium |
| WO2021217906A1 (en) * | 2020-04-28 | 2021-11-04 | 深圳壹账通智能科技有限公司 | Posture detection method, apparatus and device based on gait features, and storage medium |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20110234589A1 (en) * | 2009-10-07 | 2011-09-29 | Microsoft Corporation | Systems and methods for tracking a model |
| CN103390174A (en) * | 2012-05-07 | 2013-11-13 | 深圳泰山在线科技有限公司 | Physical education assisting system and method based on human body posture recognition |
| CN103810724A (en) * | 2014-03-06 | 2014-05-21 | 西安电子科技大学 | Human motion tracking method based on space embedded extreme learning machine |
| CN107292886A (en) * | 2017-08-11 | 2017-10-24 | 厦门市美亚柏科信息股份有限公司 | Object intrusion detection method and device based on mesh generation and neutral net |
| CN107451568A (en) * | 2017-08-03 | 2017-12-08 | 重庆邮电大学 | Use the attitude detecting method and equipment of depth convolutional neural networks |
| CN108734104A (en) * | 2018-04-20 | 2018-11-02 | 杭州易舞科技有限公司 | Body-building action error correction method based on deep learning image recognition and system |
| CN108853953A (en) * | 2018-09-03 | 2018-11-23 | 合肥智慧龙图腾知识产权股份有限公司 | A kind of shape up exercise method, apparatus, system, computer equipment and storage medium |
-
2018
- 2018-12-04 CN CN201811470739.3A patent/CN109711271A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20110234589A1 (en) * | 2009-10-07 | 2011-09-29 | Microsoft Corporation | Systems and methods for tracking a model |
| CN103390174A (en) * | 2012-05-07 | 2013-11-13 | 深圳泰山在线科技有限公司 | Physical education assisting system and method based on human body posture recognition |
| CN103810724A (en) * | 2014-03-06 | 2014-05-21 | 西安电子科技大学 | Human motion tracking method based on space embedded extreme learning machine |
| CN107451568A (en) * | 2017-08-03 | 2017-12-08 | 重庆邮电大学 | Use the attitude detecting method and equipment of depth convolutional neural networks |
| CN107292886A (en) * | 2017-08-11 | 2017-10-24 | 厦门市美亚柏科信息股份有限公司 | Object intrusion detection method and device based on mesh generation and neutral net |
| CN108734104A (en) * | 2018-04-20 | 2018-11-02 | 杭州易舞科技有限公司 | Body-building action error correction method based on deep learning image recognition and system |
| CN108853953A (en) * | 2018-09-03 | 2018-11-23 | 合肥智慧龙图腾知识产权股份有限公司 | A kind of shape up exercise method, apparatus, system, computer equipment and storage medium |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2020252918A1 (en) * | 2019-06-20 | 2020-12-24 | 平安科技（深圳）有限公司 | Human body-based gesture recognition method and apparatus, device, and storage medium |
| WO2021217906A1 (en) * | 2020-04-28 | 2021-11-04 | 深圳壹账通智能科技有限公司 | Posture detection method, apparatus and device based on gait features, and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110222665B (en) | Human body action recognition method in monitoring based on deep learning and attitude estimation | |
| CN109635875A (en) | A kind of end-to-end network interface detection method based on deep learning | |
| Li et al. | Intelligent sports training system based on artificial intelligence and big data | |
| CN108345869A (en) | Driver's gesture recognition method based on depth image and virtual data | |
| CN104200200B (en) | Fusion depth information and half-tone information realize the system and method for Gait Recognition | |
| CN109635727A (en) | A kind of facial expression recognizing method and device | |
| CN107145845A (en) | The pedestrian detection method merged based on deep learning and multi-characteristic points | |
| CN108304820A (en) | A kind of method for detecting human face, device and terminal device | |
| CN109670396A (en) | A kind of interior Falls Among Old People detection method | |
| CN107844797A (en) | A kind of method of the milking sow posture automatic identification based on depth image | |
| CN107767335A (en) | A kind of image interfusion method and system based on face recognition features' point location | |
| CN112560741A (en) | Safety wearing detection method based on human body key points | |
| CN108764066A (en) | A kind of express delivery sorting working specification detection method based on deep learning | |
| CN110490080A (en) | A kind of human body tumble method of discrimination based on image | |
| CN110263768A (en) | A kind of face identification method based on depth residual error network | |
| CN110363742A (en) | A kind of notch of switch machine detection method based on CNN and image procossing | |
| CN110084192A (en) | Quick dynamic hand gesture recognition system and method based on target detection | |
| CN109840478A (en) | A kind of movement appraisal procedure, device, mobile terminal and readable storage medium storing program for executing | |
| CN109934182A (en) | Object behavior analysis method, device, electronic equipment and computer storage medium | |
| CN109117753A (en) | Position recognition methods, device, terminal and storage medium | |
| CN105069745A (en) | face-changing system based on common image sensor and enhanced augmented reality technology and method | |
| CN116092199B (en) | Employee working state identification method and identification system | |
| CN109711271A (en) | A kind of action determination method and system based on joint connecting line | |
| CN109271918A (en) | The method for distinguishing balanced capacity obstacle crowd based on centre-of gravity shift model | |
| CN111709365A (en) | Automatic human motion posture detection method based on convolutional neural network |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20190503 |