CN111738174B - Human body example analysis method and system based on depth decoupling - Google Patents
Human body example analysis method and system based on depth decoupling Download PDFInfo
- Publication number
- CN111738174B CN111738174B CN202010592997.XA CN202010592997A CN111738174B CN 111738174 B CN111738174 B CN 111738174B CN 202010592997 A CN202010592997 A CN 202010592997A CN 111738174 B CN111738174 B CN 111738174B
- Authority
- CN
- China
- Prior art keywords
- human body
- clustering
- candidate region
- integral
- body part
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000004458 analytical method Methods 0.000 title claims abstract description 47
- 238000001514 detection method Methods 0.000 claims abstract description 78
- 238000000034 method Methods 0.000 claims abstract description 37
- 238000013527 convolutional neural network Methods 0.000 claims description 7
- 238000006116 polymerization reaction Methods 0.000 claims description 4
- 238000012216 screening Methods 0.000 claims description 4
- 238000013528 artificial neural network Methods 0.000 claims description 3
- 238000010276 construction Methods 0.000 claims description 3
- 238000010586 diagram Methods 0.000 description 8
- 238000004590 computer program Methods 0.000 description 7
- 230000006870 function Effects 0.000 description 6
- 238000000605 extraction Methods 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 238000012545 processing Methods 0.000 description 3
- 230000011218 segmentation Effects 0.000 description 3
- 230000036544 posture Effects 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 239000000126 substance Substances 0.000 description 2
- 238000006467 substitution reaction Methods 0.000 description 2
- 238000012549 training Methods 0.000 description 2
- 229910017435 S2 In Inorganic materials 0.000 description 1
- 101100194363 Schizosaccharomyces pombe (strain 972 / ATCC 24843) res2 gene Proteins 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 239000000284 extract Substances 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 238000011176 pooling Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/103—Static body considered as a whole, e.g. static pedestrian or occupant recognition
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/23—Clustering techniques
- G06F18/231—Hierarchical techniques, i.e. dividing or merging pattern sets so as to obtain a dendrogram
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/26—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
- G06V10/267—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion by performing operations on regions, e.g. growing, shrinking or watersheds
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Multimedia (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- General Engineering & Computer Science (AREA)
- Evolutionary Computation (AREA)
- Computing Systems (AREA)
- Biomedical Technology (AREA)
- General Health & Medical Sciences (AREA)
- Computational Linguistics (AREA)
- Biophysics (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Molecular Biology (AREA)
- Health & Medical Sciences (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Bioinformatics & Computational Biology (AREA)
- Evolutionary Biology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Human Computer Interaction (AREA)
- Image Analysis (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention belongs to the field of computer vision, in particular to a human body example analysis method and system based on deep decoupling, aiming at solving the problem that missed detection and false detection influence the human body analysis identification precision of an example, the method comprises the following steps: acquiring an example detection frame and characteristics of a candidate region in an input image; the candidate region is a candidate region of a human body example; the human body examples comprise a human body integral example and a human body part example; acquiring an example mask and example features of an example corresponding to the candidate region based on the features of the candidate region; and carrying out human body integral-human body component association clustering through a hierarchical clustering algorithm based on the example detection boxes, the example masks and the example characteristics corresponding to the candidate regions to obtain a human body example analysis result. The method and the device can improve the identification precision of the human body analysis of the example and reduce the missing detection and the false detection of the human body example.
Description
Technical Field
The invention belongs to the field of computer vision, and particularly relates to a human body example analysis method and system based on deep decoupling.
Background
Human body analysis research extracts human body parts (such as hair, face, trunk, legs and the like) in an image from a background and divides the human body parts into different semantic regions according to part type definitions, so that semantic type labels corresponding to each pixel of a full image are given. And the example human body analysis further associates each human body part with the human body to which the human body part belongs, and divides semantic areas of each part into different human body examples. Currently, most of example human body analysis methods adopt a flow of firstly detecting and then analyzing, wherein the flow firstly positions a whole human body detection frame as an example, and then pixel-level semantic analysis is carried out on each example detection frame. In the method, the image contains a plurality of people and the postures of the human body are various, so that two problems are caused: when the deviation occurs in the detection frame, all parts of the human body example cannot be included, so that the inherited deviation in the subsequent semantic analysis stage cannot be analyzed and missed parts are omitted; when a plurality of human body examples overlap in a large area, a plurality of human body examples may be contained in one detection frame and cannot be distinguished, and the semantic analysis stage still analyzes based on that only one human body example is contained in the detection frame, so that false recognition is caused.
Disclosure of Invention
In order to solve the above problems in the prior art, that is, to solve the problem that missing detection and false detection affect the human body analysis and identification accuracy of the example, the first aspect of the present invention provides a human body example analysis method based on deep decoupling, which includes the following steps:
step S100, acquiring an example detection frame and characteristics of a candidate area in an input image; the candidate region is a candidate region of a human body example; the human body examples comprise a human body integral example and a human body part example;
step S200, acquiring an example mask and example features of an example corresponding to the candidate region based on the features of the candidate region;
and S300, carrying out human body integral-human body component association clustering through a hierarchical clustering algorithm based on the example detection boxes, the example masks and the example characteristics corresponding to the candidate regions to obtain a human body example analysis result.
In some preferred embodiments, the method for acquiring the example detection frame and feature of the candidate region in the input image in step S100 includes:
step S110, extracting image characteristics of the input image based on a convolutional neural network to serve as first characteristics;
step S120, based on the first feature, obtaining the example detection frames of the human body whole example and the human body part example in the candidate region, and extracting the feature of the internal image of each example detection frame as a second feature.
In some preferred embodiments, the method for "extracting image features of the input image based on the convolutional neural network" in step S110 includes:
and extracting image features of the input image through a depth convolution neural network, and extracting image features of different scales based on a feature pyramid network with deformable convolution.
In some preferred embodiments, the method for "acquiring an example mask and an example feature of an example corresponding to the candidate region" in step S200 includes:
step S210, extracting masks corresponding to human body examples in each candidate region as example masks based on the second features;
step S220, weighting the corresponding second characteristics based on the example mask of each candidate area, and acquiring the characteristics of the corresponding human body example in the candidate area as example characteristics.
In some preferred embodiments, in step S300, "performing human body integer-human body component association clustering by using hierarchical clustering algorithm to obtain human body example analysis result" includes:
step S310, clustering is carried out according to the example detection boxes corresponding to the candidate areas to obtain a first clustering result I S1 ；
Step S320, according to the preset human body structure constraint conditions, based on the human body integral example andexample feature similarity of human body part examples, pair I S1 Screening to obtain a second polymerization result I S2 ；
Step S330, adding I S2 Dividing the external isolated human body part examples and all human body integral examples into two groups to construct bipartite graph models and establish full connection, and combining I after bipartite graph matching is carried out under the preset human body structure constraint condition S2 Construction of the final clustering result I S3 ；
Step S340, according to I S3 And obtaining an example result of human body integral-human body part clustering, and then corresponding the example mask and the category of each human body part example in each human body integral example to obtain a human body example analysis result.
In some preferred embodiments, the first clustering result I S1 The acquisition method comprises the following steps:
clustering is carried out according to the example detection frames corresponding to the human body integral example and the human body part example, if the center point of the example detection frame of the human body part example is positioned in the example detection frame of the human body integral example, the human body part example is judged to be matched with the corresponding human body integral example, and the matching formula is
Wherein the content of the first and second substances,
an instance detection box representing an ith individual part instance,
an instance detection box representing the jth individual's whole body instance,
representing the clustering result I S1 The set of instances of (a) detect box clustering, S1 represents the first clustering stage.
In some preferred embodiments, the preset human body structure constraint condition is a preset number of each human body instance corresponding to each type of component.
In some preferred embodiments, the second classification result I S2 The acquisition method comprises the following steps:
based on the number of various preset human body examples corresponding to the parts, for each cluster, according to the similarity of the human body part examples in the cluster and the example characteristics of the human body examples, removing the human body part examples with lower similarity to obtain a second clustering result I S2 。
In some preferred embodiments, the two-classification matching in step S340 adopts the hungarian algorithm.
In a second aspect of the present invention, a human body instance analysis system based on depth decoupling is provided, which includes a first module, a second module, and a third module;
the first module is configured to acquire an example detection frame and features of a candidate region in an input image; the candidate region is a candidate region of a human body example; the human body examples comprise a human body integral example and a human body part example;
the second module is configured to obtain an example mask and an example feature of an example corresponding to the candidate region based on the feature of the candidate region;
the third module is configured to perform human body whole-human body component association clustering through a hierarchical clustering algorithm based on the example detection boxes, the example masks and the example features corresponding to the candidate regions, and obtain a human body example analysis result.
The invention has the beneficial effects that:
after the detection frames and the characteristics of the candidate regions of the human body and the human body parts are obtained, example characteristics and an example mask are extracted based on the characteristics of the candidate regions, and then hierarchical clustering of the human body and the parts is carried out on the example detection frames, the example characteristics and the example mask to gradually correct the matching relation of the human body and the human body part examples, so that the identification precision of example human body analysis is improved, and missing detection and false detection of the human body examples are reduced.
Drawings
Other features, objects and advantages of the present application will become more apparent upon reading of the following detailed description of non-limiting embodiments thereof, made with reference to the accompanying drawings in which:
FIG. 1 is a schematic flow chart of a human body example analysis method based on depth decoupling according to an embodiment of the invention;
FIG. 2 is a schematic diagram of a feature pyramid network structure based on deformable convolution according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of an example leg and a split leg network configuration in accordance with an embodiment of the present invention;
FIG. 4 is a diagram illustrating bipartite graph model matching according to an embodiment of the invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings, and it is apparent that the described embodiments are some, but not all embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The present application will be described in further detail with reference to the following drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the relevant invention and not restrictive of the invention. It should be noted that, for convenience of description, only the portions related to the related invention are shown in the drawings.
It should be noted that the embodiments and features of the embodiments in the present application may be combined with each other without conflict.
The invention discloses a human body example analysis method based on depth decoupling, which comprises the following steps of:
step S100, acquiring an example detection frame and characteristics of a candidate area in an input image; the candidate region is a candidate region of a human body example; the human body examples comprise a human body integral example and a human body part example;
step S200, acquiring an example mask and example features of an example corresponding to the candidate region based on the features of the candidate region;
and S300, carrying out human body integral-human body component association clustering through a hierarchical clustering algorithm based on the example detection boxes, the example masks and the example characteristics corresponding to the candidate regions to obtain a human body example analysis result.
In order to more clearly explain the human body example analysis method based on depth decoupling, the following is a detailed description of the steps in one embodiment of the method according to the present invention with reference to the accompanying drawings.
The human body example analysis method based on the depth decoupling comprises the following steps S100-S300.
Step S100, acquiring an example detection frame and characteristics of a candidate area in an input image; the candidate region is a candidate region of a human body example; the human body examples comprise a human body integral example and a human body part example. The steps further include step S110 and step S120.
Step S110, extracting image characteristics of the input image based on a convolutional neural network to serve as first characteristics.
In some preferred embodiments, the first feature may be obtained by: and extracting image features of the input image through a depth convolution neural network, and extracting image features of different scales as first features based on a feature pyramid network with deformable convolution. Referring to fig. 2, a network structure of the feature pyramid network with deformable convolution adopted in this embodiment is specifically shown, and a feature pyramid network based on deformable convolution replaces a 3 × 3 convolution of an original feature pyramid network with a 3 × 3 deformable convolution. In fig. 2, res2 to res5 are residual modules, respectively, F2 to F5 are corresponding layer output feature maps, respectively, and RoI is a pooling layer.
Step S120, based on the first characteristic, acquiring the example detection frames of the human body whole example and the human body part example in the candidate area, and extracting the characteristic of the internal image of each example detection frame as a second characteristic.
In this embodiment, the example detection frame and the second feature may be extracted through the detection branch, and the detection branch adopts a Mask R-CNN structure, that is, 2 full connection layers perform feature extraction
And taking, and then adopting 1 full connection layer for predicting the category and 1 full connection layer prediction detection frame.
Step S200, based on the characteristics of the candidate region, acquiring an example mask and example characteristics of an example corresponding to the candidate region. The steps further include step S210 and step S220.
And step S210, extracting masks corresponding to human body examples in each candidate region as example masks based on the second features.
In the step, the example mask can be extracted through a dividing branch, the dividing branch takes the second characteristic as input, and the example mask is extracted through training by a full convolution network and binary cross entropy loss. The structure of the full convolutional network is not limited. In the embodiment, the segmentation branch adopts a Mask R-CNN structure, 4 convolutions of 3x3 are adopted for feature extraction, 1 deconvolution is adopted for up-sampling, and 1 convolution of 1x1 is adopted for final segmentation Mask prediction.
Step S220, weighting the corresponding second characteristics based on the example mask of each candidate area, and acquiring the characteristics of the corresponding human body example in the candidate area as example characteristics.
In this step, example features may be extracted through example branches, and in this embodiment, the example branches use the result of the division branch as an attention mask to filter background information, and then perform example feature extraction by using 4 3 × 3 convolutional layers and 2 full-link layers. According to the example branch provided by the invention, the example mask is used as the attention weight and the feature weight of the candidate region, and the human body whole and human body part example features with the same dimensionality are obtained through network mapping, so that whether the human body whole and the human body part example features belong to the same example can be judged through feature measurement. Specifically, the example mask output by the dividing branch is subjected to point multiplication with the characteristics of the candidate region, then the example mask passes through 4 convolution layers with the dimension of 3 multiplied by 3 being 256 and 2 full-connection layers with the dimension of 1024, and finally 1 full-connection layer with the dimension of 64 is used for outputting the example characteristics; in addition, the training stage example branch uses contrast loss to carry out metric learning, and the distance between the human body whole body and the human body part example characteristics of the same example should be smaller than the characteristic distance of different examples from different sources. The contrast loss equation is as follows:
wherein v is i Showing an example feature of a human body part, u j Showing an example feature of the human body as a whole,
is represented by the formula i Positive sample instance characteristics of the same human body instance,
is represented by the formula i Negative examples of different human examples are characterized, and K represents the number of all negative examples.
Fig. 3 shows a schematic diagram of a network structure of an example leg and a split leg in an embodiment of the invention.
And S300, carrying out human body integral-human body component association clustering through a hierarchical clustering algorithm based on the example detection boxes, the example masks and the example characteristics corresponding to the candidate regions to obtain a human body example analysis result. Specifically, the method comprises steps S310-S340.
Step S310, clustering is carried out according to the example detection frames corresponding to the candidate areas to obtain a first clustering result I S1 。
In the step, clustering is carried out according to the positions of the human body whole and the human body part detection frame, and if the center of the human body part detection frame is positioned in the human body whole detection frame, the human body part and the human body whole detection frame are judged to belong to the same human body example.
Clustering is carried out according to the human body integral example and the example detection frames corresponding to the human body part example, if the center point of the example detection frame of the human body part example is positioned in the example detection frame of the human body integral example, the human body part example is judged to be matched with the corresponding human body integral example, and the matching formula is
Wherein the content of the first and second substances,
an instance detection box representing an ith individual part instance,
an example detection box representing the jth individual's whole body example,
represents the clustering result I S1 The set of instances of (a) detect box clustering, S1 represents the first clustering stage.
Step S320, according to the preset human body structure constraint conditions, based on the similarity of the example characteristics of the human body whole example and the human body part example, for I S1 Screening to obtain a second polymerization result I S2 。
According to the clustering result I S1 And human body structure constraint conditions, namely the number of various parts corresponding to each human body example is limited, and the matching degree (namely the clustering score w) of the human body and the human body part examples is measured by extracting example features through the example branch of the method i，j ，
Is the clustering score, p, of the features of both instances i，j The similarity score of the example characteristic representing the ith personal body part and the integral example characteristic of the jth personal body part can be calculated by adopting a cosine similarity method,
the class score representing the output of the detection branch) can remove components with low matching degree, thereby reducing the error matching of the whole human body and the human body component example, and particularly, when a plurality of human body components are contained in one detection frame due to the overlapping of a plurality of human bodies in a complex scene, the component example can be corrected to avoid the situationAnd (4) continuing the false recognition of human body analysis.
The constraint condition of the human body structure is based on a human body physical structure model, the upper limit of the number of the human body parts corresponding to one human body example is limited (for example, one person has at most two hands), the constraint condition is recorded as R and is applied to the first clustering result I S1 And respectively calculating cosine distances between the same category of human body part example characteristics which are limited by the constraint R and the corresponding human body integral example characteristics, wherein the constraint R can be expressed as the following formula:
wherein x is i，j Indicating whether the ith personal part instance matches the jth personal part instance, 0 indicating no match, 1 indicating a match, and PartM indicating a set of M identical category personal part instances. The formula indicates that the number of component matches of the same category should not exceed 1.
Based on the number of various preset parts corresponding to each human body example, for each cluster, clustering a result I according to the similarity of the human body part examples in the cluster and the example characteristics of the human body examples S1 Removing the human body part example with lower similarity to obtain a second clustering result I S2 。
Step S330, adding I S2 Dividing the external isolated human body part examples and all human body integral examples into two groups to construct bipartite graph models and establish full connection, and adding I after bipartite graph matching under the preset human body structure constraint condition S2 In the step (2), a final clustering result I is obtained S3 。
As shown in FIG. 4, through bipartite graph matching, components which are not contained in the human body detection frame can be matched with the bipartite graph matching, so that the problems that all human body components cannot be contained due to inaccuracy of the human body detection frame and the subsequent human body analysis inherits the deviation of the human body components are solved, and missing detection in example human body analysis is reduced. As shown in fig. 4, the isolated human body part example is represented as a human body part node set (including hair 1, hair 2, hat 3, face 4, face 5, overcoat 6 … right foot M), and the human body whole example is represented as a pedestrian node set (including pedestrian 1-pedestrian N).
When the optimal matching is calculated by using the Hungarian algorithm, the human body structure constraint condition R still takes effect to reduce the error matching, and the bipartite graph matching can be expressed by the following formula:
wherein x is i，j And M is the number of the human body part instances, and N is the number of the human body whole instances.
Step S340, according to I S3 And obtaining an example result of human body integral-human body part clustering, and then corresponding the example mask and the category of each human body part example in each human body integral example to obtain a human body example analysis result.
The invention obtains fine-grained component-level examples by detecting the whole human body and human body components, and carries out hierarchical clustering between human bodies and components by extracting example masks and example features through the segmentation branch and the example branch to obtain example human body analysis results, so as to improve the identification precision of example human body analysis and reduce the missing detection and false detection of human body examples, and the improvement is more remarkable particularly for complex human body posture scenes.
The human body example analysis system based on depth decoupling comprises a first module, a second module and a third module;
the first module is configured to acquire an example detection frame and features of a candidate region in an input image; the candidate region is a candidate region of a human body example; the human body examples comprise a human body integral example and a human body part example;
the second module is configured to obtain an example mask and an example feature of an example corresponding to the candidate region based on the feature of the candidate region;
the third module is configured to perform human body whole-human body component association clustering through a hierarchical clustering algorithm based on the example detection boxes, the example masks and the example features corresponding to the candidate regions, and obtain a human body example analysis result.
It can be clearly understood by those skilled in the art that, for convenience and brevity of description, the specific working process and related description of the system described above may refer to the corresponding process in the foregoing method embodiments, and will not be described herein again.
It should be noted that, the human body example analysis system based on depth decoupling provided in the foregoing embodiment is only illustrated by the division of the functional modules, and in practical applications, the functions may be distributed by different functional modules according to needs, that is, the modules or steps in the embodiment of the present invention are further decomposed or combined, for example, the modules in the foregoing embodiment may be combined into one module, or may be further split into multiple sub-modules, so as to complete all or part of the functions described above. The names of the modules and steps involved in the embodiments of the present invention are only for distinguishing the modules or steps, and are not to be construed as unduly limiting the present invention.
The invention also provides an embodiment of a storage device, wherein a plurality of programs are stored in the storage device, and the programs are suitable for being loaded and executed by a processor to realize the human body example analysis method based on the deep decoupling.
The invention also provides an embodiment of a processing device, which comprises a processor and a storage device; a processor adapted to execute various programs; a storage device adapted to store a plurality of programs; the program is adapted to be loaded and executed by a processor to implement the human instance parsing method based on depth decoupling described above.
It can be clearly understood by those skilled in the art that, for convenience and brevity of description, the specific working processes and related descriptions of the storage device and the processing device described above may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.
In particular, according to an embodiment of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method illustrated in the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network via the communication section, and/or installed from a removable medium. The computer program, when executed by a Central Processing Unit (CPU), performs the above-described functions defined in the method of the present application. It should be noted that the computer readable medium mentioned above in the present application may be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this application, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In this application, however, a computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.
Computer program code for carrying out operations for aspects of the present application may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The terms "first," "second," and the like are used for distinguishing between similar elements and not necessarily for describing or implying a particular order or sequence.
The terms "comprises," "comprising," or any other similar term are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus.
So far, the technical solutions of the present invention have been described in connection with the preferred embodiments shown in the drawings, but it is apparent to those skilled in the art that the scope of the present invention is not limited to these specific embodiments. Equivalent changes or substitutions of related technical features can be made by those skilled in the art without departing from the principle of the invention, and the technical scheme after the changes or substitutions can fall into the protection scope of the invention.
Claims (8)
1. A human body example analysis method based on depth decoupling is characterized by comprising the following steps:
step S100, acquiring an example detection frame and characteristics of a candidate area in an input image; the candidate region is a candidate region of a human body example; the human body examples comprise a human body integral example and a human body part example;
step S200, acquiring an example mask and example features of an example corresponding to the candidate region based on the features of the candidate region;
step S300, based on the example detection frame, the example mask and the example characteristics corresponding to each candidate region, carrying out human body whole-human body part association clustering through a hierarchical clustering algorithm to obtain a human body example analysis result;
carrying out human body integral-human body component association clustering through a hierarchical clustering algorithm to obtain a human body example analysis result, wherein the method comprises the following steps:
step S310, clustering is carried out according to the example detection frames corresponding to the candidate areas to obtain a first clustering result I S1 ；
Step S320, according to the preset human body structure constraint conditions, based onExample feature similarity of human body Whole example and human body part example, Pair I S1 Screening to obtain a second polymerization result I S2 ；
Step S330, adding I S2 Dividing the external isolated human body part examples and all human body integral examples into two groups to construct bipartite graph models and establish full connection, and combining I after bipartite graph matching is carried out under the preset human body structure constraint condition S2 Construction of the final clustering result I S3 ；
Step S340, according to I S3 Obtaining an example result of human body integral-human body component clustering, and then corresponding the example mask and the category of each human body component example in each human body integral example to obtain a human body example analysis result;
the first clustering result I S1 The acquisition method comprises the following steps:
clustering is carried out according to the example detection frames corresponding to the human body integral example and the human body part example, if the center point of the example detection frame of the human body part example is positioned in the example detection frame of the human body integral example, the human body part example is judged to be matched with the corresponding human body integral example, and the matching formula is
Wherein, the first and the second end of the pipe are connected with each other,
an instance detection box representing an ith individual part instance,
example detection Box, I, showing the jth Individual's Whole body example j，S1 Representing the clustering result I S1 The set of instances of (a) detect box clustering, S denotes the first clustering stage.
2. The human body example analysis method based on depth decoupling according to claim 1, wherein in step S100, "obtaining example detection frames and features of candidate regions in an input image" comprises:
step S110, extracting image characteristics of the input image based on a convolutional neural network to serve as first characteristics;
step S120, based on the first feature, obtaining the example detection frames of the human body whole example and the human body part example in the candidate region, and extracting the feature of the internal image of each example detection frame as a second feature.
3. The human body instance parsing method based on depth decoupling as claimed in claim 2, wherein in step S110, "extracting image features of the input image based on convolutional neural network" includes:
and extracting image features of the input image through a depth convolution neural network, and extracting image features of different scales based on a feature pyramid network with deformable convolution.
4. The human body instance parsing method based on depth decoupling according to claim 2, wherein in step S200, "obtain instance masks and instance features of corresponding instances of candidate regions", the method includes:
step S210, extracting masks corresponding to human body examples in each candidate region as example masks based on the second features;
step S220, weighting the corresponding second characteristics based on the example mask of each candidate area, and acquiring the characteristics of the corresponding human body example in the candidate area as example characteristics.
5. The human body example analysis method based on deep decoupling according to claim 4, wherein the preset human body structure constraint condition is the number of preset various parts corresponding to each human body example.
6. The method for human instance parsing based on deep decoupling of claim 5, wherein the second clusterResults I S2 The acquisition method comprises the following steps:
based on the number of various preset human body examples corresponding to the parts, for each cluster, according to the similarity of the human body part examples in the cluster and the example characteristics of the human body examples, removing the human body part examples with lower similarity to obtain a second clustering result I S2 。
7. The human body instance parsing method based on deep decoupling as claimed in claim 4, wherein the bipartite graph matching in step S340 adopts Hungarian algorithm.
8. A human body example analysis system based on depth decoupling is characterized by comprising a first module, a second module and a third module;
the first module is configured to acquire an example detection frame and features of a candidate region in an input image; the candidate region is a candidate region of a human body example; the human body examples comprise a human body integral example and a human body part example;
the second module is configured to obtain an example mask and an example feature of an example corresponding to the candidate region based on the feature of the candidate region;
the third module is configured to perform human body whole-human body component association clustering through a hierarchical clustering algorithm based on the example detection boxes, the example masks and the example features corresponding to the candidate regions to obtain a human body example analysis result;
carrying out human body integral-human body component association clustering through a hierarchical clustering algorithm to obtain a human body example analysis result, wherein the method comprises the following steps:
step S310, clustering is carried out according to the example detection frames corresponding to the candidate areas to obtain a first clustering result I S1 ；
Step S320, according to the preset human body structure constraint conditions, based on the similarity of the example characteristics of the human body whole example and the human body part example, for I S1 Screening to obtain a second polymerization result I S2 ；
Step S330, adding I S2 Dividing the external isolated human body part examples and all human body integral examples into two groups to construct bipartite graph models and establish full connection, and combining I after bipartite graph matching is carried out under the preset human body structure constraint condition S2 Construction of the final clustering result I S3 ；
Step S340, according to I S3 Obtaining an example result of human body integral-human body part clustering, and then corresponding the example mask and the category of each human body part example in each human body integral example to obtain a human body example analysis result;
the first clustering result I S1 The acquisition method comprises the following steps:
clustering is carried out according to the example detection frames corresponding to the human body integral example and the human body part example, if the center point of the example detection frame of the human body part example is positioned in the example detection frame of the human body integral example, the human body part example is judged to be matched with the corresponding human body integral example, and the matching formula is
Wherein, the first and the second end of the pipe are connected with each other,
an instance detection box representing an ith individual part instance,
example detection Box, I, showing the jth Individual's Whole body example j，S1 Representing the clustering result I S1 The set of instances of (a) detect box clustering, S1 represents the first clustering stage.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010592997.XA CN111738174B (en) | 2020-06-25 | 2020-06-25 | Human body example analysis method and system based on depth decoupling |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010592997.XA CN111738174B (en) | 2020-06-25 | 2020-06-25 | Human body example analysis method and system based on depth decoupling |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111738174A CN111738174A (en) | 2020-10-02 |
| CN111738174B true CN111738174B (en) | 2022-09-20 |
Family
ID=72651173
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010592997.XA Active CN111738174B (en) | 2020-06-25 | 2020-06-25 | Human body example analysis method and system based on depth decoupling |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111738174B (en) |
Families Citing this family (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112233777A (en) * | 2020-11-19 | 2021-01-15 | 中国石油大学(华东) | Gallstone automatic identification and segmentation system based on deep learning, computer equipment and storage medium |
| WO2022144603A1 (en) * | 2020-12-31 | 2022-07-07 | Sensetime International Pte. Ltd. | Methods and apparatuses for training neural network, and methods and apparatuses for detecting correlated objects |
| JP2023511241A (en) * | 2020-12-31 | 2023-03-17 | 商▲湯▼国▲際▼私人有限公司 | Neural network training method and apparatus and associated object detection method and apparatus |
| CN113627269B (en) * | 2021-07-16 | 2023-04-28 | 中国科学院合肥物质科学研究院 | Pest target detection method based on decoupling classification and regression feature optimal layer technology |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103345744A (en) * | 2013-06-19 | 2013-10-09 | 北京航空航天大学 | Human body target part automatic analytic method based on multiple images |
| CN106548194A (en) * | 2016-09-29 | 2017-03-29 | 中国科学院自动化研究所 | The construction method and localization method of two dimensional image human joint pointses location model |
| CN107194338A (en) * | 2017-05-14 | 2017-09-22 | 北京工业大学 | Traffic environment pedestrian detection method based on human body tree graph model |
| CN108460411A (en) * | 2018-02-09 | 2018-08-28 | 北京市商汤科技开发有限公司 | Example dividing method and device, electronic equipment, program and medium |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10430966B2 (en) * | 2017-04-05 | 2019-10-01 | Intel Corporation | Estimating multi-person poses using greedy part assignment |
-
2020
- 2020-06-25 CN CN202010592997.XA patent/CN111738174B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103345744A (en) * | 2013-06-19 | 2013-10-09 | 北京航空航天大学 | Human body target part automatic analytic method based on multiple images |
| CN106548194A (en) * | 2016-09-29 | 2017-03-29 | 中国科学院自动化研究所 | The construction method and localization method of two dimensional image human joint pointses location model |
| CN107194338A (en) * | 2017-05-14 | 2017-09-22 | 北京工业大学 | Traffic environment pedestrian detection method based on human body tree graph model |
| CN108460411A (en) * | 2018-02-09 | 2018-08-28 | 北京市商汤科技开发有限公司 | Example dividing method and device, electronic equipment, program and medium |
Non-Patent Citations (1)
| Title |
|---|
| Hierarchical Human Parsing With Typed Part-Relation Reasoning;Wenguan Wang 等;《arXiv:2003.04845v1》;20200310;第8929-8239页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111738174A (en) | 2020-10-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111738174B (en) | Human body example analysis method and system based on depth decoupling | |
| Zhao et al. | Hyperspectral anomaly detection based on stacked denoising autoencoders | |
| US9251420B2 (en) | System for mapping and identification of plants using digital image processing and route generation | |
| US9230168B2 (en) | Automatic generation of built-up layers from high resolution satellite image data | |
| CN110443222B (en) | Method and device for training face key point detection model | |
| CN110705584A (en) | Emotion recognition method, emotion recognition device, computer device and storage medium | |
| US20190065890A1 (en) | System and method for plant leaf identification | |
| CN113158789B (en) | Target detection method, system, device and medium for remote sensing image | |
| CN111444807A (en) | Target detection method, device, electronic equipment and computer readable medium | |
| CN113920255B (en) | High-efficient mapping system based on point cloud data | |
| CN111612075A (en) | Interest point and descriptor extraction method based on joint feature recombination and feature mixing | |
| CN115661573A (en) | Method and device for detecting infrared dim target, computing equipment and storage medium | |
| CN114168768A (en) | Image retrieval method and related equipment | |
| CN116309612B (en) | Semiconductor silicon wafer detection method, device and medium based on frequency decoupling supervision | |
| Drobnjak et al. | Testing a new ensemble vegetation classification method based on deep learning and machine learning methods using aerial photogrammetric images | |
| CN116935240A (en) | Surface coverage classification system and method for multi-scale perception pyramid | |
| CN115719428A (en) | Face image clustering method, device, equipment and medium based on classification model | |
| CN108154107B (en) | Method for determining scene category to which remote sensing image belongs | |
| CN115546586A (en) | Method and device for detecting infrared dim target, computing equipment and storage medium | |
| CN116958724A (en) | Training method and related device for product classification model | |
| CN111626373B (en) | Multi-scale widening residual error network, small target recognition and detection network and optimization method thereof | |
| CN114387465A (en) | Image recognition method and device, electronic equipment and computer readable medium | |
| CN114463764A (en) | Table line detection method and device, computer equipment and storage medium | |
| Stanski et al. | Flower detection using object analysis: new ways to quantify plant phenology in a warming tundra biome | |
| CN112232431A (en) | Watermark detection model training method, watermark detection method, system, device and medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |