CN109598234A - Critical point detection method and apparatus - Google Patents
Critical point detection method and apparatus Download PDFInfo
- Publication number
- CN109598234A CN109598234A CN201811474069.2A CN201811474069A CN109598234A CN 109598234 A CN109598234 A CN 109598234A CN 201811474069 A CN201811474069 A CN 201811474069A CN 109598234 A CN109598234 A CN 109598234A
- Authority
- CN
- China
- Prior art keywords
- network
- human body
- feature
- image
- input
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/103—Static body considered as a whole, e.g. static pedestrian or occupant recognition
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Physics & Mathematics (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Biology (AREA)
- Evolutionary Computation (AREA)
- Bioinformatics & Computational Biology (AREA)
- General Engineering & Computer Science (AREA)
- Artificial Intelligence (AREA)
- Life Sciences & Earth Sciences (AREA)
- Human Computer Interaction (AREA)
- Multimedia (AREA)
- Image Analysis (AREA)
Abstract
The embodiment of the present invention provides a kind of critical point detection method and apparatus, wherein, the critical point detection method includes using the current frame image in video information as the input of human body detector, to calculate and export the pose probability value in the human testing frame vector and the current frame image for being cut out to the current frame image；The current frame image is cut out according to the human testing frame to obtain human body image block；Using the pose probability value and the human body image block as the input of property detector, to calculate and export the key point in the current frame image.The present invention can effectively solve the problem that characteristics of human body's detection is difficult to the problem of mobile terminal executes in real time, reduces the network complexity during critical point detection, provides detection accuracy.
Description
Technical field
The present invention relates to technical field of image processing, in particular to a kind of critical point detection method and apparatus.
Background technique
It mainly include top-down and bottom-up in the existing human body critical point detection method based on deep learning
Two model framework design methods.Wherein, top-down system usually obtains the detection of personage using human testing network first
Then frame obtains the key point of each limbs of personage in frame using a feature detection network again；And bottom-up approach is then first
It first detects limbs key point all in image, then these is put by certain concatenate rule and connects into different people.
But since human body attitude is rich and changeful, and it is easy to be blocked by background object, itself clothing, therefore either bottom-up still
Top-down mode often requires bigger neural network and goes to complete limbs Detection task, once network is not enough
Ability to express, it may be difficult to respect to all complex scenes, lead to the human body critical point detection method based on deep learning
Data processing speed is not good enough, it is difficult to apply on real-time scene, especially mobile terminal.
Summary of the invention
In view of this, the present invention provides a kind of critical point detection method and apparatus, the above problem can effectively solve the problem that.
In order to achieve the above object, present pre-ferred embodiments provide a kind of critical point detection method, are applied to mobile whole
End, the critical point detection method includes feature detection process, and this feature detection process includes:
Using the current frame image in video information as the input of human body detector, to calculate and export for working as to described
The pose probability value in human testing frame and the current frame image that prior image frame is cut out；
The current frame image is cut out according to the human testing frame to obtain human body image block；
Using the pose probability value and the human body image block as the input of property detector, so that this feature detector
It chooses with the matched feature detection network of the pose probability value to calculate and export the key point in the current frame image.
In the selection of present pre-ferred embodiments, the human body detector includes that fisrt feature extracts network, region is built
Discuss network and classification Recurrent networks；It calculates and exports the human testing frame for being cut out to the current frame image and institute
The step of stating the pose probability value in current frame image, comprising:
The input of network is extracted using the current frame image as the fisrt feature to extract and export the present frame
Characteristics of image in image；
Using the characteristics of image extracted as the input of region suggestion network to generate initial detecting frame, and according to institute
Initial detecting frame is stated to cut the characteristics of image in the current frame image to obtain initial pictures characteristic block；
Using the initial pictures characteristic block as the input of the classification Recurrent networks, to calculate for characterizing human body attitude
The pose probability value of classification, and refine is carried out to the initial detecting frame and corrects to obtain human testing frame.
In the selection of present pre-ferred embodiments, the property detector includes that second feature extracts network and multiple
Feature detects network；Using the pose probability value and the human body image block as the input of property detector, to calculate and defeated
The step of key point in the current frame image out, comprising:
The input of network is extracted using the human body image block as the second feature to calculate and extract human body image
Characteristics of human body in block；
Corresponding feature detection network is chosen from multiple feature detection networks according to the pose probability value and is made
For target detection network, the characteristics of human body is detected to the pass of the characteristics of human body as the input of the target detection network
Key point.
In the selection of present pre-ferred embodiments, input of the pose probability value as property detector is being executed,
It is described before the step of detecting network from selection in multiple features detection network and the most matched feature of the pose probability value
Method further include:
Training dataset is obtained, which is divided into multiple training subsets, the training subset and the spy
Sign detection network corresponds；
For training subset described in each, using the training subset as the input of character pair detection network to calculate simultaneously
The test feature point for exporting the training subset, which is calculated as the input of Recurrent networks and export test with
Track value；
The loss function value of the feature detection network is calculated according to the test feature point and the test pursuit gain, and
Feature detection network is optimized until the output of the loss function value meets preset need according to loss function value.
In the selection of present pre-ferred embodiments, the calculating step of the loss function value Loss includes:
Wherein, ocRepresent test feature point；δXcRepresent test tracking
Value, HcRepresent fact characteristic point, δ YcActual tracking value is represented, C represents the quantity of feature detection network, and c represents c-th of training
Collection.
In the selection of present pre-ferred embodiments, the critical point detection method further includes signature tracking process, the spy
Levying tracking process includes:
Using the human testing frame as the input of detection Recurrent networks, to carry out refine correction to human body detection block,
And human body tracking is carried out based on the human testing frame after correction.
In the selection of present pre-ferred embodiments, operation has first thread and the second thread in the mobile terminal；
The first thread is used to be based on the first thread for executing the feature detection process, second thread
Operation result execute the signature tracking process, wherein the first thread and second thread are handed over according to predetermined period
For operation.
Present pre-ferred embodiments also provide a kind of critical point detection device, are applied to mobile terminal, the key point inspection
Surveying device includes:
Pose probability computing module, for using the current frame image in video information as the input of human body detector, with
It calculates and exports the posture in the human testing frame and the current frame image for being cut out to the current frame image
Probability value；
Image cropping module, for being cut out the current frame image to obtain human body according to the human testing frame
Image block；
Key point extraction module, for using the pose probability value and the human body image block as the defeated of property detector
Enter, so that this feature detector chooses and the matched feature detection network of the pose probability value is described current to calculate and export
Key point in frame image.
In the selection of present pre-ferred embodiments, the human body detector includes that fisrt feature extracts network, region is built
Discuss network and classification Recurrent networks；The pose probability computing module includes；
Fisrt feature extraction unit, for using the current frame image as the fisrt feature extract network input with
It extracts and exports the characteristics of image in the current frame image；
Image cropping unit, for suggesting the input of network for the characteristics of image extracted as the region to generate just
Beginning detection block, and the characteristics of image in the current frame image is cut to obtain initial graph according to the initial detecting frame
As characteristic block；
Pose probability computing unit, for using the initial pictures characteristic block as it is described classification Recurrent networks input,
To calculate the pose probability value for characterizing human body attitude classification, and refine is carried out to the initial detecting frame and corrects to obtain people
Body detection block.
In the selection of present pre-ferred embodiments, the property detector includes that second feature extracts network and multiple
Feature detects network, and the key point extraction module includes:
Second feature extraction unit, for using the human body image block as the second feature extract network input with
It calculates and extracts the characteristics of human body in human body image block；
Critical point detection unit is corresponded to for being chosen from multiple feature detection networks according to the pose probability value
Feature detection network and as target detection network, using the characteristics of human body as the input of the target detection network to examine
Survey the key point of the characteristics of human body.
Compared with prior art, the embodiment of the present invention provides a kind of critical point detection method and apparatus, wherein the present invention adopts
Property detector is made of a kind of multiple mininet models for handling posture respectively, so that detection model be effectively reduced
Network training difficulty, improve data processing speed, each small network realized in its corresponding posture and is compared
High precision.Meanwhile the present invention while detecting human body also can synchronism output human body attitude type, according to the type
Suitable feature detection network is selected to carry out critical point detection.
In addition, the present invention uses parallel detection logic when carrying out critical point detection to further increase the speed of service
Detailed description of the invention
In order to illustrate the technical solution of the embodiments of the present invention more clearly, below will be to needed in the embodiment attached
Figure is briefly described, it should be understood that the following drawings illustrates only certain embodiments of the present invention, therefore is not construed as pair
The restriction of range for those of ordinary skill in the art without creative efforts, can also be according to this
A little attached drawings obtain other relevant attached drawings.
Fig. 1 is the frame structure schematic diagram of mobile terminal provided in an embodiment of the present invention.
Fig. 2 is the flow diagram of critical point detection method provided in an embodiment of the present invention.
Fig. 3 is the sub-process schematic diagram of step S11 shown in Fig. 2.
Fig. 4 is the sub-process schematic diagram of step S13 shown in Fig. 2.
Fig. 5 is the schematic network structure of property detector provided in an embodiment of the present invention.
Fig. 6 is another flow diagram of critical point detection method provided in an embodiment of the present invention.
Fig. 7 is a kind of functional block diagram of critical point detection device provided in an embodiment of the present invention.
Icon: 10- mobile terminal；100- critical point detection device；110- pose probability computing module；1100- first is special
Levy extraction unit；1101- image cropping unit；1102- pose probability computing unit；120- image cropping module；130- is crucial
Point extraction module；1300- second feature extraction unit；1301- critical point detection unit；200- memory；300- storage control
Device；400- processor.
Specific embodiment
In order to make the object, technical scheme and advantages of the embodiment of the invention clearer, below in conjunction with the embodiment of the present invention
In attached drawing, technical scheme in the embodiment of the invention is clearly and completely described, it is clear that described embodiment is
A part of the embodiment of the present invention, instead of all the embodiments.The present invention being usually described and illustrated herein in the accompanying drawings is implemented
The component of example can be arranged and be designed with a variety of different configurations.
Therefore, the detailed description of the embodiment of the present invention provided in the accompanying drawings is not intended to limit below claimed
The scope of the present invention, but be merely representative of selected embodiment of the invention.Based on the embodiments of the present invention, this field is common
Technical staff's every other embodiment obtained without creative efforts belongs to the model that the present invention protects
It encloses.
It should also be noted that similar label and letter indicate similar terms in following attached drawing, therefore, once a certain Xiang Yi
It is defined in a attached drawing, does not then need that it is further defined and explained in subsequent attached drawing.
As shown in Figure 1, being the frame structure schematic diagram of mobile terminal 10 provided in an embodiment of the present invention, the mobile terminal 10
Including critical point detection device 100, memory 200, storage control 300 and processor 400.Wherein, the memory
200, storage control 300 and each element of processor 400 are directly or indirectly electrically connected between each other, to realize the biography of data
Defeated or interaction.It is electrically connected for example, being realized between these elements by one or more communication bus or signal wire.The key
Point detection device 100 includes that at least one can be stored in the memory 200 in the form of software or firmware or be solidificated in institute
State the software function module in the operating system of mobile terminal 10.Control of the processor 400 in the storage control 300
The lower access memory 200, for executing the executable module stored in the memory 200, such as the key point
Software function module included by detection device 100 and computer program etc., and then realize the key point in the embodiment of the present invention
Detection method.
Herein it should be understood that the structure of mobile terminal shown in FIG. 1 10 is only to illustrate, such as mobile terminal 10 can
With the more perhaps less component than shown in Fig. 1 or with the configuration different from shown in Fig. 1.Wherein, shown in FIG. 1
Each component can be realized by software, hardware or a combination thereof.
As shown in Fig. 2, a kind of flow diagram of the critical point detection method provided for present pre-ferred embodiments, described
Critical point detection method is applied to mobile terminal 10 shown in Fig. 2.Below in conjunction with Fig. 2 to the critical point detection method
Detailed process and step are described in detail.It should be noted that the reality for the critical point detection method that the present embodiment provides
Border implementation steps are not limitation with Fig. 2 in the following sequence.
Step S11 is used for using the current frame image in video information as the input of human body detector with calculating and exporting
The pose probability value in human testing frame and the current frame image that the current frame image is cut out；
Step S12 is cut out the current frame image according to the human testing frame to obtain human body image block；
Step S13, using the pose probability value and the human body image block as the input of property detector, so that the spy
Detector is levied to choose with the matched feature detection network of the pose probability value to calculate and export in the current frame image
Key point.
The characteristics of human body's detection method provided in above-mentioned steps S11- step S13 is applied to mobile terminal 10, with realization pair
The real-time detection of human body key point.Specifically, detector (such as human testing during human body critical point detection in order to balance
Device, property detector etc.) the speed of service and precision, the present invention abandoned in the prior art using a biggish detection model
The mode of progress critical point detection, but the design method based on top-down human body critical point detection frame, using more points
The property detector of branch realizes critical point detection network.Wherein, the property detector in the critical point detection network includes multiple
Feature detects network, and each feature detection network is used to be responsible for handling the critical point detection under a kind of posture, so as to effectively mention
High data processing speed reduces the training difficulty when carrying out feature detection network training, so that each small feature detects net
Network can realize relatively high detection accuracy in its corresponding posture, while ensure to realize human body key on mobile terminal 10
The real-time detection of point.
It in detail, is the classification that human body attitude is obtained by the way of cluster in step S11, it is assumed that N is shared in training set
A human sample, each human body share L key point, by the Unitary coordinateization of these key points between section [0,1], then the
First point of normalized coordinate on n human body can be expressed asSo, for each human body attitude, by its all the points
CoordinateAs its feature vector, then all postures are clustered using hierarchical clustering algorithm, cluster numbers
Mesh is C, and the linking criteria used is maximum linkage, in this way, can be by all data in training set point
At C class, the recommendation numerical value of C is 6.
Specifically, in actual implementation, the human body detector may include that fisrt feature extracts network, net is suggested in region
Network (Region Proposal Network, RPN) and classification Recurrent networks, then as shown in figure 3, step S11 can be by following
Step S110- step S112 is realized, specific as follows.
Step S110 extracts the input of network for the current frame image as the fisrt feature to extract and export institute
State the characteristics of image in current frame image；
Step S111 suggests the input of network to generate initial detecting for the characteristics of image extracted as the region
Frame, and the characteristics of image in the current frame image is cut according to the initial detecting frame to obtain initial pictures feature
Block；
Step S112 is used for table using the initial pictures characteristic block as the input of the classification Recurrent networks to calculate
The pose probability value of human body attitude classification is levied, and refine is carried out to the initial detecting frame and corrects to obtain human testing frame.
In step S110- step S112, the fisrt feature extracts network for extracting in the current frame image
Characteristics of image (such as human body image feature).Suggest that network is used to extract what network obtained according to the fisrt feature in the region
Characteristics of image generates rough detection block, i.e. initial detecting frame, then according to initial detecting frame in the current frame image
Characteristics of image is cut to obtain initial pictures characteristic block.The classification Recurrent networks can be a fully-connected network, use
In suggesting that the output result of network is calculated and exports three vectors according to the region, wherein first vector is to be used for
Refine, correction are carried out to export more accurate human testing frame vector to the initial frame that surrounds, second vector is that length can be with
For the probability vector of 2 prospect (such as human body) probability vector and background, with the classification for foreground and background, third vector is
For selecting the pose probability value vector of matched feature detection network.
It should be noted that with traditional Faster-RCNN (Faster Regions with Convolutional
Neural Network Feature) place that is slightly different is that the classification Recurrent networks in the present invention increase a use
In the output of the pose probability value of human body attitude classification, but the training method of the whole network of human body detector that the present invention provides
Consistent with traditional Faster-RCNN training method, details are not described herein for the present embodiment.
Further, in step s 12, shape, size of the human body image block etc. depend on the human testing frame
Shape, size, the present embodiment is herein with no restrictions.When actual implementation, it is assumed that human testing frame be a length be 4 to
Amount, such as [x, y, w, h], wherein x, y respectively represent the coordinate in the human testing frame upper left corner, w and h respectively represent frame width and
Highly.It, can be by the rectangle region at the abscissa x to x+w of current frame image, ordinate y to y+h when progress characteristics of image is cut out
Domain, which extracts, completes to cut out to get human body image block is arrived.
Further, in step s 13, the property detector may include that second feature extracts network and multiple spies
Sign detection network, multiple feature detection one second feature of network share extract network (i.e. basic network), and each feature is examined
Survey grid network is corresponding to be responsible for the detection of key point under human body attitude a kind of.In the present invention, by using a basic network (
Two feature extraction networks) plus the feature detection structure of multiple-limb network (feature detection network), network can be greatly reduced
So that the model of property detector will not be excessively too fat to move in mobile terminal 10, while network mould can be also greatly reduced in parameter amount
The training difficulty of type.In detail, as shown in figure 4, step S13 can be realized by following step S130- step 131.
Step S130 extracts the input of network for the human body image block as the second feature to calculate and extract this
Characteristics of human body in human body image block；
Step S131 chooses corresponding feature from multiple feature detection networks according to the pose probability value and detects
Network and as target detection network, using the characteristics of human body as the input of the target detection network to detect the human body
The key point of feature.
Optionally, in the present embodiment, the spatial information of the form expression key point of thermodynamic chart can be used.Assuming that feature is examined
The input size of survey grid network is H × W, and first point of coordinate is (xl, yl), the ratio of input and output is s, then a human body appearance
The thermodynamic chart of state is oneThree-dimensional matrice H, wherein
Z indicates the third dimension of three-dimensional matrice H, and l indicates the number of human body key point, and 0 < l < L-1.
Further, the description based on above-mentioned steps S10- step S13, the critical point detection method may also include feature
Tracking process, this feature tracking process include using the human testing frame as the input of tracking Recurrent networks, to the human body
Detection block carries out refine correction, and carries out signature tracking based on the human testing frame after correction.
It should be noted that as shown in figure 5, the present embodiment is by increasing a detection Recurrent networks in property detector
Mode realize signature tracking, the detection Recurrent networks and multiple features detection network collectively form the more of the property detector
A branch extracts this basic network of network with shared second feature.In detail, the detection Recurrent networks are used for present frame
Human testing frame carry out again refine, correction, and based on after refine human testing frame realize signature tracking.Actual implementation
When, after human body detector executes primary acquisition human testing frame, which can handle future
The displacement of number frame human testing frame, liberates the pressure of human body detector, reduces runing time.
As an implementation, the present invention realizes feature detection process above-mentioned and signature tracking using parallel form
Process.Specifically, first thread and the second thread can have been run in the mobile terminal 10；The first thread is for executing institute
Feature detection process is stated, second thread executes the signature tracking mistake for the operation result based on the first thread
Journey, wherein the first thread and second thread are according to predetermined period alternate run.For example, the feature detection process
It can execute when the camera of mobile terminal 10 is opened, and be executed once every fixed frame number.Wherein, human body detector obtains general
The highest human testing frame of rate and pose probability value, subsequent property detector are cut from current frame image according to human testing frame
Area-of-interest (such as human body region) out, and select suitable feature detection network to obtain key point according to pose probability value
The refine information of testing result and human testing frame.Signature tracking process can occur between feature detection process twice, the spy
Human body detector is in sign detection process in order to save power consumption in a dormant state, and only property detector is in running order, this
When property detector detection Recurrent networks start to play a role, after the completion of every frame critical point detection, to human testing frame into
Row refine guarantees that human body detection block is aligned with the human body of the frame image, to reduce cumulative errors, then as the people of next frame
Body detection block.Feature detection process ensure that the update that can carry out personage when personage's appearing and subsiding in time, and signature tracking
Process ensure that human testing thread will not be fully loaded with, and reduce overall power.
Wherein, by this present embodiment feature detection process and signature tracking process executed by two thread parallels,
Therefore the human testing frame for the necessarily previous frame that property detector obtains, but since time interval is too short, human testing circle
Error can be ignored.
Further, according to actual needs, as shown in fig. 6, executing the pose probability value as property detector
Input, with chosen from multiple features detection network with the step of the pose probability value most matched feature detection network it
Before, the critical point detection method can also be trained property detector by following step S14- step S16, specifically such as
Under.
Step S14 obtains training dataset, which is divided into multiple training subsets, the training subset
It is corresponded with feature detection network；
Step S15, for training subset described in each, using the training subset as the input of character pair detection network
To calculate and export the test feature point of the training subset, which is calculated as the input of Recurrent networks and defeated
Pursuit gain is tested out；
Step S16 calculates the loss of the feature detection network according to the test feature point and the test pursuit gain
Functional value, and feature detection network is optimized until the output of the loss function value meets according to loss function value
Preset need.
Wherein, in step S14- step S16, the preset need refers to that loss function value reaches minimum or tends to be flat
Surely.In addition, it is assumed that training dataset is divided into C training subset, and random cropping and change in displacement are carried out to training set, every time
When iteration in turn from C training subset take out a batch (batch) training data, be respectively put into property detector with
It obtains the output of character pair detection network and detects the output of Recurrent networks.Assuming that the instruction obtained from c-th of training subset
Practicing image is Ic, standard thermodynamic chart is Hc, the difference between coordinate after the original detection block coordinate of image and random cropping, displacement
For δ Yc, the output of c-th of branch is Oc, the output of Recurrent networks is δ Xc, then loss function are as follows:
After the training for completing property detector, the position acquisition of the maximum on slice each in thermodynamic chart can be passed through
The coordinate of each human body key point.
Based on the description of aforementioned critical point detection method as can be seen that the present invention efficiently solves in the prior art based on depth
The critical point detection technology of study is difficult to the shortcomings that mobile terminal 10 executes in real time.By executing parallel, (such as feature was detected
Journey and tracking features process) and miniaturization network solve feature detection speed issue；Pass through human body attitude classification and more points
The strategy of the small network of branch (such as multiple features detect network) solves the problems, such as that small network ability to express is weak；By to multiple small-sized
Feature detection network and detection block Recurrent networks it is careful design be effectively relieved in the prior art multiple models execute when
The excessively high problem of power consumption.
Further, referring to Fig. 7, the embodiment of the present invention also provides a kind of critical point detection device 100, it is applied to Fig. 1
Shown in mobile terminal 10.The critical point detection device 100 includes pose probability computing module 110, image cropping module 120
With key point extraction module 130.
The pose probability computing module 110, for using the current frame image in video information as human body detector
Input, to calculate and export in the human testing frame and the current frame image for being cut out to the current frame image
Pose probability value；In the present embodiment, the description as described in the pose probability computing module 110 is specifically referred to step S11
Detailed description, that is, the step S11 can be executed by the pose probability computing module 110.Optionally, the posture
Probability evaluation entity 110 includes fisrt feature extraction unit 1100, image cropping unit 1101 and pose probability computing unit
1102。
The fisrt feature extraction unit 1100, for extracting network for the current frame image as the fisrt feature
Input to extract and export the characteristics of image in the current frame image；In the present embodiment, extracted about the fisrt feature
The description of unit 1100 specifically refers to the detailed description to step S110, that is, the step S110 can be by described first
Feature extraction unit 1100 executes.
Described image cuts unit 1101, for suggesting the input of network using the characteristics of image extracted as the region
To generate initial detecting frame, and the characteristics of image in the current frame image is cut to obtain according to the initial detecting frame
To initial pictures characteristic block；In the present embodiment, the description as described in described image cuts unit 1101 is specifically referred to step
The detailed description of S111 executes that is, the step S111 can cut unit 1101 by described image.
The pose probability computing unit 1102, for using the initial pictures characteristic block as the classification Recurrent networks
Input, to calculate the pose probability value for characterizing human body attitude classification, and refine school is carried out to the initial detecting frame
Just obtaining human testing frame.In the present embodiment, the description as described in the pose probability computing unit 1102 is specifically referred to step
The detailed description of rapid S112, that is, the step S112 can be executed by the pose probability computing unit 1102.
Described image cut module 120, for according to the human testing frame to the current frame image be cut out with
Obtain human body image block；In the present embodiment, the description as described in described image cuts module 120 is specifically referred to step S12
Detailed description executes that is, the step S12 can cut module 120 by described image.
The key point extraction module 130, for being examined using the pose probability value and the human body image block as feature
The input of device is surveyed, so that this feature detector is chosen with the matched feature detection network of the pose probability value to calculate and export
Key point in the current frame image.In the present embodiment, the description as described in the key point extraction module 130 is specifically referred to
To the detailed description of step S13, that is, the step S13 can be executed by the key point extraction module 130.Optionally, institute
Stating key point extraction module 130 may include that second feature extracts single 1300 yuan and critical point detection unit 1301.
The second feature extraction unit 1300, for extracting network for the human body image block as the second feature
Input to calculate and extract the characteristics of human body in human body image block；In the present embodiment, extracted about the second feature single
The description of member 1300 specifically refers to the detailed description to step S130, that is, the step S130 can be special by described second
Extraction unit 1300 is levied to execute.
The critical point detection unit 1301, for being detected in networks according to the pose probability value from multiple features
Corresponding feature detection network is chosen and as target detection network, using the characteristics of human body as the target detection network
It inputs to detect the key point of the characteristics of human body.In the present embodiment, description has as described in the critical point detection unit 1301
Body can refer to the detailed description to step S131, that is, the step S131 can be held by the critical point detection unit 1301
Row.
In conclusion the embodiment of the present invention provides a kind of critical point detection method and apparatus, wherein the spy that the present invention uses
Levying detector is made of a kind of multiple mininet models for handling posture respectively, so that the network of detection model be effectively reduced
Training difficulty, improves data processing speed, each small network is enabled to realize relatively high essence in its corresponding posture
Degree.Meanwhile the present invention while detecting human body also can synchronism output human body attitude type, according to the type i.e. may be selected close
Suitable feature detection network carries out critical point detection.
In several embodiments provided by the embodiment of the present invention, it should be understood that disclosed system and method, it can also
To realize by another way.System and method embodiment described above is only schematical, for example, in attached drawing
Flow chart and block diagram show that the systems of multiple embodiments according to the present invention, method and computer program product are able to achieve
Architecture, function and operation.In this regard, each box in flowchart or block diagram can represent module, a program
A part of section or code, a part of the module, section or code include that one or more is patrolled for realizing defined
Collect the executable instruction of function.It should also be noted that in some implementations as replacement, function marked in the box
It can occur in a different order than that indicated in the drawings.For example, two continuous boxes can actually be held substantially in parallel
Row, they can also be executed in the opposite order sometimes, and this depends on the function involved.It is also noted that block diagram and/or
The combination of each box in flow chart and the box in block diagram and or flow chart, can the function as defined in executing or dynamic
The dedicated hardware based system made is realized, or can be realized using a combination of dedicated hardware and computer instructions.
In addition, each functional module in each embodiment of the present invention can integrate one independent portion of formation together
Point, it is also possible to modules individualism, an independent part can also be integrated to form with two or more modules.
It, can be with if the function is realized and when sold or used as an independent product in the form of software function module
It is stored in a computer readable storage medium.Based on this understanding, technical solution of the present invention is substantially in other words
The part of the part that contributes to existing technology or the technical solution can be embodied in the form of software products, the meter
Calculation machine software product is stored in a storage medium, including some instructions are used so that a computer equipment (can be a
People's computer, electronic equipment or network equipment etc.) execute all or part of step of each embodiment the method for the present invention
Suddenly.And storage medium above-mentioned includes: USB flash disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), deposits at random
The various media that can store program code such as access to memory (RAM, Random Access Memory), magnetic or disk.
It should be noted that, in this document, the terms "include", "comprise" or its any other variant are intended to the packet of nonexcludability
Contain, so that the process, method, article or equipment for including a series of elements not only includes those elements, but also including
Other elements that are not explicitly listed, or further include for elements inherent to such a process, method, article, or device.
In the absence of more restrictions, the element limited by sentence "including a ...", it is not excluded that including the element
Process, method, article or equipment in there is also other identical elements.
The foregoing is only a preferred embodiment of the present invention, is not intended to restrict the invention, for the skill of this field
For art personnel, the invention may be variously modified and varied.All within the spirits and principles of the present invention, made any to repair
Change, equivalent replacement, improvement etc., should all be included in the protection scope of the present invention.
Claims (10)
1. a kind of critical point detection method is applied to mobile terminal, which is characterized in that the critical point detection method includes feature
Detection process, this feature detection process include:
Using the current frame image in video information as the input of human body detector, to calculate and export for the present frame
The pose probability value in human testing frame and the current frame image that image is cut out；
The current frame image is cut out according to the human testing frame to obtain human body image block；
Using the pose probability value and the human body image block as the input of property detector, so that this feature detector is chosen
With the matched feature detection network of the pose probability value to calculate and export the key point in the current frame image.
2. critical point detection method according to claim 1, which is characterized in that the human body detector includes fisrt feature
Extract network, network is suggested in region and classification Recurrent networks；It calculates and exports for being cut out to the current frame image
The step of pose probability value in human testing frame and the current frame image, comprising:
The input of network is extracted using the current frame image as the fisrt feature to extract and export the current frame image
In characteristics of image；
Suggest the input of network using the characteristics of image extracted as the region to generate initial detecting frame, and according to described first
Beginning detection block cuts the characteristics of image in the current frame image to obtain initial pictures characteristic block；
Using the initial pictures characteristic block as the input of the classification Recurrent networks, to calculate for characterizing human body attitude classification
Pose probability value, and to the initial detecting frame carry out refine correct to obtain human testing frame.
3. critical point detection method according to claim 1, which is characterized in that the property detector includes second feature
It extracts network and multiple features detects network；Using the pose probability value and the human body image block as property detector
Input, the step of to calculate and export the key point in the current frame image, comprising:
The input of network is extracted using the human body image block as the second feature to calculate and extract in human body image block
Characteristics of human body；
Corresponding feature detection network is chosen from multiple feature detection networks according to the pose probability value and as mesh
The characteristics of human body, is detected the key of the characteristics of human body by mark detection network as the input of the target detection network
Point.
4. critical point detection method according to claim 1, which is characterized in that executing the pose probability value as special
The input of detector is levied, detects network with the most matched feature of the pose probability value to choose from multiple features detection network
The step of before, the method also includes:
Training dataset is obtained, which is divided into multiple training subsets, the training subset and the feature are examined
Survey grid network corresponds；
For training subset described in each, using the training subset as the input of character pair detection network to calculate and export
The training subset is calculated as the input of Recurrent networks and exports test tracking by the test feature point of the training subset
Value；
According to the test feature point and the loss function value tested pursuit gain and calculate the feature detection network, and according to
Loss function value optimizes until the output of the loss function value meets preset need feature detection network.
5. critical point detection method according to claim 4, which is characterized in that the calculating of the loss function value Loss walks
Suddenly include:
Wherein, OcRepresent test feature point；δXcRepresent test pursuit gain, HcGeneration
Table fact characteristic point, δ YcActual tracking value is represented, C represents the quantity of feature detection network, and c represents c-th of training subset.
6. critical point detection method according to claim 1, which is characterized in that the critical point detection method further includes spy
Tracking process is levied, this feature tracking process includes:
Using the human testing frame as the input of detection Recurrent networks, to carry out refine correction, and base to human body detection block
Human testing frame after correction carries out human body tracking.
7. critical point detection method according to claim 6, which is characterized in that operation has First Line in the mobile terminal
Journey and the second thread；
The first thread is used for the fortune based on the first thread for executing the feature detection process, second thread
Row result executes the signature tracking process, wherein the first thread and second thread are alternately transported according to predetermined period
Row.
8. a kind of critical point detection device, it is applied to mobile terminal, which is characterized in that the critical point detection device includes:
Pose probability computing module, for using the current frame image in video information as the input of human body detector, to calculate
And the pose probability in human testing frame and the current frame image of the output for being cut out to the current frame image
Value；
Image cropping module, for being cut out the current frame image to obtain human body image according to the human testing frame
Block；
Key point extraction module, for using the pose probability value and the human body image block as the input of property detector,
So that this feature detector is chosen with the matched feature detection network of the pose probability value to calculate and export the present frame
Key point in image.
9. critical point detection device according to claim 8, which is characterized in that the human body detector includes fisrt feature
Extract network, network is suggested in region and classification Recurrent networks；The pose probability computing module includes；
Fisrt feature extraction unit, for extracting the input of network for the current frame image as the fisrt feature to extract
And export the characteristics of image in the current frame image；
Image cropping unit, for suggesting the input of network to generate initial inspection for the characteristics of image extracted as the region
Frame is surveyed, and the characteristics of image in the current frame image is cut according to the initial detecting frame to obtain initial pictures spy
Levy block；
Pose probability computing unit, for using the initial pictures characteristic block as it is described classification Recurrent networks input, in terms of
The pose probability value for characterizing human body attitude classification is calculated, and refine is carried out to the initial detecting frame and corrects to obtain human body inspection
Survey frame.
10. critical point detection device according to claim 8, which is characterized in that the property detector includes second special
Sign extracts network and multiple features detect network, and the key point extraction module includes:
Second feature extraction unit, for extracting the input of network for the human body image block as the second feature to calculate
And extract the characteristics of human body in human body image block；
Critical point detection unit, for choosing corresponding spy from multiple feature detection networks according to the pose probability value
Sign detects network and as target detection network, using the characteristics of human body as the input of the target detection network to detect
State the key point of characteristics of human body.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811474069.2A CN109598234B (en) | 2018-12-04 | 2018-12-04 | Key point detection method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811474069.2A CN109598234B (en) | 2018-12-04 | 2018-12-04 | Key point detection method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109598234A true CN109598234A (en) | 2019-04-09 |
| CN109598234B CN109598234B (en) | 2021-03-23 |
Family
ID=65960934
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811474069.2A Active CN109598234B (en) | 2018-12-04 | 2018-12-04 | Key point detection method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109598234B (en) |
Cited By (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110634160A (en) * | 2019-08-12 | 2019-12-31 | 西北大学 | Method for constructing target three-dimensional key point extraction model and recognizing posture in two-dimensional graph |
| CN110969138A (en) * | 2019-12-10 | 2020-04-07 | 上海芯翌智能科技有限公司 | Human body posture estimation method and device |
| CN111291692A (en) * | 2020-02-17 | 2020-06-16 | 咪咕文化科技有限公司 | Video scene recognition method and device, electronic equipment and storage medium |
| CN111325179A (en) * | 2020-03-09 | 2020-06-23 | 厦门美图之家科技有限公司 | Gesture tracking method and device, electronic equipment and storage medium |
| CN111599007A (en) * | 2020-05-26 | 2020-08-28 | 张仲靖 | Smart city CIM road mapping method based on unmanned aerial vehicle aerial photography |
| CN112016371A (en) * | 2019-05-31 | 2020-12-01 | 广州市百果园信息技术有限公司 | Face key point detection method, device, equipment and storage medium |
| CN112200183A (en) * | 2020-09-30 | 2021-01-08 | 北京字节跳动网络技术有限公司 | Image processing method, device, equipment and computer readable medium |
| WO2021068589A1 (en) * | 2019-10-09 | 2021-04-15 | 东软睿驰汽车技术（沈阳）有限公司 | Method and apparatus for determining object and key points thereof in image |
| CN112699265A (en) * | 2019-10-22 | 2021-04-23 | 商汤国际私人有限公司 | Image processing method and device, processor and storage medium |
| WO2021146952A1 (en) * | 2020-01-21 | 2021-07-29 | 深圳市大疆创新科技有限公司 | Following method and device, movable platform, and storage medium |
| CN113538573A (en) * | 2020-04-20 | 2021-10-22 | 中移(成都)信息通信科技有限公司 | Clothing key point positioning method and device, electronic equipment and computer storage medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102792317A (en) * | 2010-03-11 | 2012-11-21 | 高通股份有限公司 | Image feature detection based on application of multiple feature detectors |
| US20160253784A1 (en) * | 2014-12-11 | 2016-09-01 | Sharp Laboratories Of America, Inc. | System for video super resolution using semantic components |
| CN107944442A (en) * | 2017-11-09 | 2018-04-20 | 北京智芯原动科技有限公司 | Based on the object test equipment and method for improving convolutional neural networks |
| CN108520251A (en) * | 2018-04-20 | 2018-09-11 | 北京市商汤科技开发有限公司 | Critical point detection method and device, electronic equipment and storage medium |
-
2018
- 2018-12-04 CN CN201811474069.2A patent/CN109598234B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102792317A (en) * | 2010-03-11 | 2012-11-21 | 高通股份有限公司 | Image feature detection based on application of multiple feature detectors |
| US20160253784A1 (en) * | 2014-12-11 | 2016-09-01 | Sharp Laboratories Of America, Inc. | System for video super resolution using semantic components |
| CN107944442A (en) * | 2017-11-09 | 2018-04-20 | 北京智芯原动科技有限公司 | Based on the object test equipment and method for improving convolutional neural networks |
| CN108520251A (en) * | 2018-04-20 | 2018-09-11 | 北京市商汤科技开发有限公司 | Critical point detection method and device, electronic equipment and storage medium |
Cited By (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112016371A (en) * | 2019-05-31 | 2020-12-01 | 广州市百果园信息技术有限公司 | Face key point detection method, device, equipment and storage medium |
| CN110634160A (en) * | 2019-08-12 | 2019-12-31 | 西北大学 | Method for constructing target three-dimensional key point extraction model and recognizing posture in two-dimensional graph |
| CN110634160B (en) * | 2019-08-12 | 2022-11-18 | 西北大学 | Method for constructing target three-dimensional key point extraction model and recognizing posture in two-dimensional graph |
| WO2021068589A1 (en) * | 2019-10-09 | 2021-04-15 | 东软睿驰汽车技术（沈阳）有限公司 | Method and apparatus for determining object and key points thereof in image |
| CN112699265A (en) * | 2019-10-22 | 2021-04-23 | 商汤国际私人有限公司 | Image processing method and device, processor and storage medium |
| WO2021077620A1 (en) * | 2019-10-22 | 2021-04-29 | 商汤国际私人有限公司 | Image processing method and apparatus, processor, and storage medium |
| CN110969138A (en) * | 2019-12-10 | 2020-04-07 | 上海芯翌智能科技有限公司 | Human body posture estimation method and device |
| WO2021146952A1 (en) * | 2020-01-21 | 2021-07-29 | 深圳市大疆创新科技有限公司 | Following method and device, movable platform, and storage medium |
| CN111291692A (en) * | 2020-02-17 | 2020-06-16 | 咪咕文化科技有限公司 | Video scene recognition method and device, electronic equipment and storage medium |
| CN111291692B (en) * | 2020-02-17 | 2023-10-20 | 咪咕文化科技有限公司 | Video scene recognition method and device, electronic equipment and storage medium |
| CN111325179A (en) * | 2020-03-09 | 2020-06-23 | 厦门美图之家科技有限公司 | Gesture tracking method and device, electronic equipment and storage medium |
| CN111325179B (en) * | 2020-03-09 | 2023-05-02 | 厦门美图之家科技有限公司 | Gesture tracking method, gesture tracking device, electronic equipment and storage medium |
| CN113538573A (en) * | 2020-04-20 | 2021-10-22 | 中移(成都)信息通信科技有限公司 | Clothing key point positioning method and device, electronic equipment and computer storage medium |
| CN111599007A (en) * | 2020-05-26 | 2020-08-28 | 张仲靖 | Smart city CIM road mapping method based on unmanned aerial vehicle aerial photography |
| CN112200183A (en) * | 2020-09-30 | 2021-01-08 | 北京字节跳动网络技术有限公司 | Image processing method, device, equipment and computer readable medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109598234B (en) | 2021-03-23 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109598234A (en) | Critical point detection method and apparatus | |
| Li et al. | Instance-level salient object segmentation | |
| CN105518709B (en) | The method, system and computer program product of face for identification | |
| CN110176027A (en) | Video target tracking method, device, equipment and storage medium | |
| CN111160269A (en) | Face key point detection method and device | |
| CN108399386A (en) | Information extracting method in pie chart and device | |
| CN108961675A (en) | Fall detection method based on convolutional neural networks | |
| CN108334848A (en) | A kind of small face identification method based on generation confrontation network | |
| CN109255324A (en) | Gesture processing method, interaction control method and equipment | |
| CN109657583A (en) | Face's critical point detection method, apparatus, computer equipment and storage medium | |
| CN106469298A (en) | Age recognition methodss based on facial image and device | |
| CN107609519A (en) | The localization method and device of a kind of human face characteristic point | |
| CN110287873A (en) | Noncooperative target pose measuring method, system and terminal device based on deep neural network | |
| CN106874826A (en) | Face key point-tracking method and device | |
| CN107871106A (en) | Face detection method and device | |
| CN108304820A (en) | A kind of method for detecting human face, device and terminal device | |
| CN109376631A (en) | A kind of winding detection method and device neural network based | |
| CN110263768A (en) | A kind of face identification method based on depth residual error network | |
| CN108664838A (en) | Based on the monitoring scene pedestrian detection method end to end for improving RPN depth networks | |
| CN110490238A (en) | A kind of image processing method, device and storage medium | |
| CN109343920A (en) | A kind of image processing method and its device, equipment and storage medium | |
| CN107016319A (en) | A kind of key point localization method and device | |
| CN110047081A (en) | Example dividing method, device, equipment and the medium of chest x-ray image | |
| CN110222780A (en) | Object detecting method, device, equipment and storage medium | |
| CN109753864A (en) | A kind of face identification method based on caffe deep learning frame |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |