CN112950637A - Human body part segmentation network training method, human body part segmentation method and device - Google Patents

Human body part segmentation network training method, human body part segmentation method and device Download PDF

Info

Publication number
CN112950637A
CN112950637A CN202110515510.2A CN202110515510A CN112950637A CN 112950637 A CN112950637 A CN 112950637A CN 202110515510 A CN202110515510 A CN 202110515510A CN 112950637 A CN112950637 A CN 112950637A
Authority
CN
China
Prior art keywords
human body
training
body part
segmentation
image
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202110515510.2A
Other languages
Chinese (zh)
Other versions
CN112950637B (en
Inventor
贾文浩
高原
刘霄
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Century TAL Education Technology Co Ltd
Original Assignee
Beijing Century TAL Education Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Century TAL Education Technology Co Ltd filed Critical Beijing Century TAL Education Technology Co Ltd
Priority to CN202110515510.2A priority Critical patent/CN112950637B/en
Publication of CN112950637A publication Critical patent/CN112950637A/en
Application granted granted Critical
Publication of CN112950637B publication Critical patent/CN112950637B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G06T7/0012—Biomedical image inspection
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/11—Region-based segmentation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10004—Still image; Photographic image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30196—Human being; Person

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • General Health & Medical Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Biophysics (AREA)
  • Software Systems (AREA)
  • Evolutionary Computation (AREA)
  • Computational Linguistics (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Biomedical Technology (AREA)
  • Mathematical Physics (AREA)
  • Data Mining & Analysis (AREA)
  • Artificial Intelligence (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Multimedia (AREA)
  • Medical Informatics (AREA)
  • Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
  • Radiology & Medical Imaging (AREA)
  • Quality & Reliability (AREA)
  • Image Analysis (AREA)

Abstract

The embodiment of the invention provides a human body part segmentation network training method, a human body part segmentation method and a device, wherein the human body part segmentation network training method comprises the steps of acquiring training human body image data and training human body characteristics of a training human body image, acquiring training human body part segmentation marks by using a human body part segmentation network to be trained, acquiring reference evaluation scores by using the training human body part segmentation marks, acquiring training evaluation scores of the training human body part segmentation marks by using an evaluation network to be trained, acquiring global losses of model training according to segmentation losses acquired by using the reference human body part segmentation marks and the training human body part segmentation marks and evaluation losses acquired by using the training evaluation scores and the reference evaluation scores, further optimizing the human body part segmentation network and the evaluation network to obtain the human body part segmentation network and the evaluation network which are trained, the accuracy of the human body part segmentation network can be improved.

Description

Human body part segmentation network training method, human body part segmentation method and device
Technical Field
The embodiment of the invention relates to the field of image recognition, in particular to a human body part segmentation network training method, a human body part segmentation method and a human body part segmentation device.
Background
Human body part segmentation belongs to a subtask of a semantic segmentation task, and aims to perform fine-grained segmentation on an image containing a human body, for example, the human body image is segmented into images of various parts such as a head region, a trunk region, an extremity region and the like.
The human body part segmentation technology is applied to various fields, such as human body appearance transfer, behavior recognition, pedestrian re-recognition, fashion composition and the like. Therefore, the human body part segmentation has important research significance and application value.
The human body part segmentation technology in the prior art has poor accuracy of human body part segmentation.
Therefore, how to improve the accuracy of human body part segmentation becomes a technical problem that needs to be solved urgently by those skilled in the art.
Disclosure of Invention
The technical problem solved by the embodiment of the invention is how to improve the quality of the human body part segmentation result.
In order to solve the above problem, an embodiment of the present invention provides a human body part segmentation network training method, including:
acquiring training human body image data, wherein the training human body image data comprises a training human body image and a reference human body part segmentation mark of the training human body image, and the reference human body part segmentation mark comprises human body part categories of all pixel points of the training human body image;
acquiring training human body characteristics of the training human body image;
acquiring training human body part segmentation marks according to the training human body characteristics by using a human body part segmentation network to be trained;
acquiring a reference evaluation score of the training human body part segmentation mark at least by using the training human body part segmentation mark, wherein the reference evaluation score is used for evaluating a segmentation result of the training human body part segmentation mark;
obtaining training evaluation scores corresponding to the training human body features according to the training human body features by using an evaluation network to be trained, wherein the evaluation network is suitable for evaluating the training human body part segmentation marks obtained by the human body part segmentation network;
and according to the segmentation loss obtained by using the reference human body part segmentation mark and the training human body part segmentation mark and the evaluation loss obtained by using the training evaluation score and the reference evaluation score, obtaining the global loss of model training, and according to the global loss, optimizing the human body part segmentation network to be trained and the evaluation network until the global loss meets a global loss threshold value, so as to obtain the trained human body part segmentation network and the trained evaluation network.
In order to solve the above problem, an embodiment of the present invention further provides a human body part segmentation method, including:
acquiring a single image to be segmented;
acquiring human body characteristics of the single image to be segmented;
and obtaining a human body part segmentation result corresponding to the single image to be segmented according to the human body characteristics by utilizing the human body part segmentation network obtained by training the human body part segmentation network training method.
In order to solve the above problem, an embodiment of the present invention further provides a human body part segmentation network training device, including:
a training human body image data acquisition unit adapted to acquire training human body image data, the training human body image data including a training human body image and a reference human body part segmentation label of the training human body image, the reference human body part segmentation label including a human body part category of each pixel point of the training human body image;
the training human body feature acquisition unit is suitable for acquiring training human body features of the training human body image;
the training human body part segmentation mark acquisition unit is suitable for acquiring a training human body part segmentation mark according to the training human body characteristics by using a human body part segmentation network to be trained;
a reference evaluation score acquisition unit adapted to acquire a reference evaluation score of the training human body part division mark by using at least the training human body part division mark, wherein the reference evaluation score is used for evaluating a division result of the training human body part division mark;
the training evaluation score obtaining unit is suitable for obtaining a training evaluation score corresponding to the training human body characteristics according to the training human body characteristics by utilizing an evaluation network to be trained, wherein the evaluation network is suitable for evaluating the training human body part segmentation marks obtained by the human body part segmentation network;
and the model training unit is suitable for acquiring the global loss of model training according to the segmentation loss acquired by the reference human body part segmentation mark and the training human body part segmentation mark and the evaluation loss acquired by the training evaluation score and the reference evaluation score, and optimizing the human body part segmentation network to be trained and the evaluation network according to the global loss until the global loss meets a global loss threshold value to obtain the trained human body part segmentation network and the trained evaluation network.
In order to solve the above problem, an embodiment of the present invention further provides a human body part segmentation apparatus, including:
the single image acquisition unit to be segmented is suitable for acquiring a single image to be segmented;
the human body characteristic acquisition unit is suitable for acquiring the human body characteristics of the single image to be segmented;
the human body part segmentation network is obtained by training through the human body part segmentation network training method according to any one of the preceding specific embodiments, and is suitable for obtaining a human body part segmentation result corresponding to the single image to be segmented according to the human body characteristics.
An embodiment of the present invention provides a storage medium, in which a program suitable for training a human body part segmentation network is stored to implement the human body part segmentation network training method according to each of the foregoing embodiments, or a storage medium, in which a program suitable for performing human body part segmentation is stored to implement the human body part segmentation method according to each of the foregoing embodiments.
The embodiment of the invention provides electronic equipment, which comprises at least one memory and at least one processor; the memory stores a program that the processor calls to execute the human body part segmentation network training method or the human body part segmentation method.
Compared with the prior art, the technical scheme of the embodiment of the invention has the following advantages:
the human body part segmentation network training method provided by the embodiment of the invention comprises the following steps: firstly, acquiring training human body image data, wherein the training human body image data comprises a training human body image and a reference human body part segmentation mark of the training human body image, and the reference human body part segmentation mark comprises human body part categories of all pixel points of the training human body image; then, training human body features of each training human body image are obtained, a training human body part segmentation mark is obtained according to the training human body features by using a human body part segmentation network to be trained, and a reference evaluation score of the training human body part segmentation mark is obtained by at least using the training human body part segmentation mark; obtaining training evaluation scores corresponding to the training human body features according to the training human body features by using an evaluation network to be trained, wherein the evaluation network is suitable for evaluating the training human body part segmentation marks obtained by the human body part segmentation network; and according to the segmentation loss obtained by using the reference human body part segmentation mark and the training human body part segmentation mark and the evaluation loss obtained by using the training evaluation score and the reference evaluation score, obtaining the global loss of model training, and according to the global loss, optimizing the human body part segmentation network to be trained and the evaluation network until the global loss meets a global loss threshold value, so as to obtain the trained human body part segmentation network and the trained evaluation network. It can be seen that, in the human body part segmentation network training method provided by the embodiment of the present invention, not only training of the human body part segmentation network is performed, but also training of the evaluation network is performed, and in the training process, the human body part segmentation network to be trained and the evaluation network are optimized according to the global loss combining the segmentation loss and the evaluation loss, and meanwhile, the accuracy of the segmentation result and the accuracy of the evaluation score are considered, and the improvement of the accuracy of the evaluation network can improve the accuracy of the human body part segmentation network, so that when the human body part is actually segmented, the accuracy of the human body part segmentation result obtained by using the trained human body part segmentation network is higher.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the provided drawings without creative efforts.
Fig. 1 is a schematic flow chart of a human body part segmentation network training method according to an embodiment of the present invention;
FIG. 2 is a schematic flow chart of a step of acquiring training human body image data of the human body part segmentation network training method according to the embodiment of the present invention;
FIG. 3 is a schematic diagram of an original image of a human body part segmentation network training method according to an embodiment of the present invention;
FIG. 4 is a schematic diagram of an image before segmentation of a human body part segmentation network training method according to an embodiment of the present invention;
FIG. 5 is a schematic diagram of an image after human body segmentation training of the human body part segmentation network training method according to the embodiment of the present invention;
fig. 6 is a schematic flow chart of a method for training a human body part segmentation network according to an embodiment of the present invention to obtain a reference pixel evaluation score;
fig. 7 is a schematic flow chart of the method for training a human body part segmentation network according to the embodiment of the present invention for obtaining a reference intersection ratio evaluation score;
fig. 8 is a schematic flow chart of the method for training a human body part segmentation network according to the embodiment of the present invention for obtaining a reference quality evaluation score;
FIG. 9 is a flowchart illustrating a human body segmentation method according to an embodiment of the present invention;
FIG. 10 is a schematic flow chart illustrating a human body segmentation method according to an embodiment of the present invention;
FIG. 11 is a schematic diagram of a human body part segmentation network training device according to an embodiment of the present invention;
FIG. 12 is a schematic view of a human body part segmentation apparatus according to an embodiment of the present invention;
fig. 13 is a schematic diagram of an electronic device according to an embodiment of the invention.
Detailed Description
As can be seen from the prior art, the accuracy of the human body part segmentation result obtained by the human body part segmentation technique in the prior art is poor.
In order to solve the foregoing problems, an embodiment of the present invention provides a human body part segmentation network training method, including:
acquiring training human body image data, wherein the training human body image data comprises a training human body image and a reference human body part segmentation mark of the training human body image, and the reference human body part segmentation mark comprises human body part categories of all pixel points of the training human body image;
acquiring training human body characteristics of the training human body image;
acquiring a training human body part segmentation mark according to the training human body characteristics by using a human body part segmentation network to be trained, wherein the training human body part segmentation mark comprises a segmentation probability value of each pixel point of the training human body image, and the segmentation probability value comprises a probability value that the pixel point is of each human body part type or non-human body part type;
acquiring a reference evaluation score of the training human body part segmentation mark at least by using the training human body part segmentation mark, wherein the reference evaluation score is used for evaluating a segmentation result of the training human body part segmentation mark;
obtaining training evaluation scores corresponding to the training human body features according to the training human body features by using an evaluation network to be trained, wherein the evaluation network is suitable for evaluating the training human body part segmentation marks obtained by the human body part segmentation network;
and according to the segmentation loss obtained by using the reference human body part segmentation mark and the training human body part segmentation mark and the evaluation loss obtained by using the training evaluation score and the reference evaluation score, obtaining the global loss of model training, and according to the global loss, optimizing the human body part segmentation network to be trained and the evaluation network until the global loss meets a global loss threshold value, so as to obtain the trained human body part segmentation network and the trained evaluation network.
It can be seen that, in the human body part segmentation network training method provided in the embodiment of the present invention, not only the training of the human body part segmentation network but also the training of the evaluation network are performed, and in the training process, the global loss of model training is obtained according to the segmentation loss obtained by using the reference human body part segmentation markers and the training human body part segmentation markers and the evaluation loss obtained by using the training evaluation scores and the reference evaluation scores, and the human body part segmentation network to be trained and the evaluation network are optimized according to the global loss, while the accuracy of the segmentation result and the accuracy of the evaluation scores are considered, and the improvement of the accuracy of the evaluation network can improve the accuracy of the human body part segmentation network. Therefore, when the human body part is actually segmented, the accuracy of the human body part segmentation result obtained by utilizing the trained human body part segmentation network is higher.
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Referring to fig. 1, fig. 1 is a schematic flow chart of a human body part segmentation network training method according to an embodiment of the present invention.
The human body part segmentation network training method provided by the embodiment of the invention comprises the following steps:
and S101, acquiring training human body image data.
The human body part segmentation network training method provided by the embodiment of the invention is used for realizing the training of the human body part segmentation network, so that human body image data used for training needs to be obtained before the training of the human body part segmentation network.
The training human body image data comprises a training human body image and a reference human body part segmentation mark of the training human body image, and the reference human body part segmentation mark comprises human body part categories of all pixel points of the training human body image.
The method for obtaining the training human body image can be selected according to needs. In an embodiment, please refer to fig. 2, and fig. 2 is a partial schematic flow chart of a human body part segmentation network training method according to an embodiment of the present invention.
In the step S101, the step of acquiring the image data of the training human body comprises the following steps:
step S1011: and carrying out single distinguishing on the original image to obtain a single distinguishing image of each human body.
Referring to fig. 3, fig. 3 is a schematic diagram of an original image of a human body part segmentation network training method according to an embodiment of the present invention.
The single distinguishing image is a part directly obtained by cutting the original image, and the size of the single distinguishing image may be different according to the different sizes of human bodies.
Step S1012, distinguishing the image by the single person to obtain an image before division;
the single person distinguishing image can be directly used as a pre-segmentation image, and the pre-segmentation image can also be obtained through further processing.
In one embodiment, the step of distinguishing the image with the single person to obtain the image before segmentation includes:
and placing the single distinguishing image in a pre-segmentation image with a standard size, filling the part except the single distinguishing image into an average value point, wherein the pixel value of the average value point is the average value of the minimum value and the maximum value of the value range of the pixel value of the image.
When the length and the width of the single distinguishing image are smaller than those of the standard size, the single distinguishing image can be directly placed in the image before segmentation of the standard size. When any one of the length and the width of the image is larger than the standard size, the side of the single-person distinguishing image larger than the standard size is aligned with the corresponding length of the standard size, and then the single-person distinguishing image is placed in the image before division of the standard size.
And the pixel value of the average value point is the average value of the minimum value and the maximum value of the value range of the image pixel value. For example, when the image pixel value ranges from (0-255, 0-255, 0-255), the pixel value of the average point may take (127,127,127). Therefore, the mean point has small influence on the human body image, and the human body part segmentation network is not easily influenced to carry out human body part segmentation.
Of course, the single distinguishing image can be placed in the image before segmentation with the standard size, and then the image before segmentation can be obtained in a mode of expanding the selection range.
The resulting pre-segmentation image is shown in fig. 4. Fig. 4 is a schematic diagram of an image before segmentation of the human body part segmentation network training method according to the embodiment of the present invention.
Step S1013: and carrying out human body part segmentation on the pre-segmentation image to obtain the training human body image data.
After the human body part segmentation is performed, each pixel point of the training human body image data is marked as a part classification, such as a head region, a trunk region, an extremity region or a non-human body part. Of course, the head region, the trunk region, and the limb region may be further subdivided.
That is, as shown in fig. 5, fig. 5 is a schematic diagram of an image after human body segmentation training of the human body part segmentation network training method provided in the embodiment of the present invention, where in the image after human body segmentation training, each pixel point is marked as a part classification.
As described above, in one embodiment, when the method for obtaining the pre-segmentation image by using the single-person segmentation image is a method in which the single-person segmentation image is placed in a pre-segmentation image of a standard size, and a portion other than the single-person segmentation image is filled with a mean value point, and a pixel value of the mean value point is a mean value of a minimum value and a maximum value of a pixel value range of an image, the step S1013: performing human body part segmentation on the pre-segmentation image to obtain the training human body image data may include: and segmenting human body parts in the pre-segmentation image with the standard size to obtain the training human body image data and reference human body part segmentation marks of all points of the training human body image data, wherein the human body part types of the parts except the single-person segmentation image in the pre-segmentation image are non-human body parts.
The division marks of the reference human body parts of the parts except the pre-division images are marked as non-human body parts, other parts of the reference human body part marks cannot be influenced, and the subsequent comparison processing of the division marks of the reference human body parts and the division marks of the training human body parts obtained according to the pre-division images with the standard sizes is facilitated.
And S102, acquiring training human body characteristics of the training human body image.
After the training human body image is obtained, training human body features can be further obtained, and the obtained human body features can be multidimensional vectors or other data forms such as matrixes.
And obtaining the training human body characteristics of the training human body image by selecting a proper characteristic extraction network.
In order to improve the quality of the obtained training human body features, in a specific embodiment, the step of obtaining the training human body features of the training human body image includes:
acquiring at least a first size training human body feature and a second size training human body feature of the training human body image;
and at least fusing the first size training human body characteristic and the second size training human body characteristic to obtain the training human body characteristic.
By fusing the first-size training human body features and the second-size training human body features, the obtained training human body features can better reflect the feature information of the human body image data under various scales.
The feature extraction networks suitable for obtaining at least the first-size training human body features and the second-size training human body features of the training human body image are multiple, and specifically, the training human body features of the training human body image can be obtained by using a feature pyramid-residual error network (ResNet-FPN).
Of course, in other embodiments, three, four, or more sizes of body features may be acquired.
And S103, acquiring training human body part segmentation marks according to the training human body characteristics by using the human body part segmentation network to be trained.
In a specific embodiment, the training human body part segmentation markers may include segmentation probability values of respective pixel points of the training human body image, wherein the segmentation probability values include probability values of the pixel points being in respective human body part categories or non-human body part categories;
such as point a (head region, 0.5; torso region, 0.2; limb region, 0.2; non-body part, 0.1), or point B (head region, 0.05; torso region, 0.1; limb region, 0.05; non-body part, 0.8).
It is easily understood that each part classification of the training human body part segmentation markers is the same as each part classification of the reference human body part segmentation markers in the training human body image data.
Of course, in other embodiments, the training human body part segmentation markers may also be predicted human body part categories of each pixel point of the training human body image, and the predicted human body part categories may be directly obtained according to the human body part segmentation network, or may be used as the predicted human body part categories according to the human body part categories corresponding to the maximum segmentation probability values in the segmentation probability values.
And step S104, acquiring a reference evaluation score of the training human body part segmentation mark at least by utilizing the training human body part segmentation mark.
In order to improve the accuracy of the human body part segmentation, it is necessary to improve the training accuracy of the human body part segmentation network, and in order to improve the training accuracy of the human body part segmentation network, it is necessary to further improve the accuracy of an evaluation network for evaluating the human body part segmentation network.
It is easily understood that in order to train the evaluation network, a reference evaluation score and a training evaluation score are also required to be obtained.
The reference evaluation score can be selected as required, and the reference evaluation score can evaluate the segmentation accuracy of the training human body part segmentation mark.
Wherein the benchmark rating score may be selected according to a variety of methods.
Referring to fig. 6, fig. 6 is a schematic flow chart of a method for training a human body part segmentation network to obtain a benchmark pixel evaluation score according to an embodiment of the present invention, in a specific implementation, the step of obtaining the benchmark evaluation score of the training human body part segmentation mark by using at least the training human body part segmentation mark includes:
step S10411: and acquiring the maximum segmentation probability value of each pixel point of the training human body image by using the segmentation probability value.
When the segmentation probability value of each pixel point of the training human body part segmentation mark is the segmentation probability value of each pixel point of the training human body image, the maximum segmentation probability value can be obtained according to the segmentation probability value.
And the maximum segmentation probability value is the maximum value in the probability values of the pixel points of the human body parts.
Such as: for the pixel point A, the probability values of the human body parts are respectively as follows:
head region, 0.5; torso region, 0.2; area of extremities, 0.2; and the maximum segmentation probability value is 0.5 when the non-human body part is 0.1.
And S10412, selecting a reference pixel point according to a preset requirement.
The pixel points can be used as the reference pixel points, so that the complexity of determining the reference pixel points can be reduced, and the flow is simplified; of course, the reference pixel point can be selected according to the requirement, so that the evaluation accuracy is improved.
In a specific embodiment, the step of selecting the reference pixel point according to the predetermined requirement includes:
and acquiring pixel points with the maximum segmentation probability value larger than or equal to the preset probability value as reference pixel points.
The maximum segmentation probability value can measure the degree of significance of classification of the result obtained by the human body part segmentation network, when the maximum segmentation probability value is smaller, the segmentation probabilities of various human body parts of each point in the training human body part segmentation mark are relatively close, and the segmentation probabilities do not account for relatively larger segmentation probabilities, so that the segmentation result of the point is considered to have poor degree of significance and poor reference significance; when the maximum segmentation probability value is larger, the segmentation probability difference of various human body parts of each point in the training human body part segmentation mark is larger, and the segmentation probability with a larger proportion exists, so that the segmentation result of the point is considered to be better in significance degree, and the possibility that the point is a certain human body part or a non-human body part is larger.
Therefore, the pixel points with the maximum segmentation probability value larger than or equal to the predetermined probability value can be obtained and used as the reference pixel points. Therefore, pixel points with high significance and high reference significance of the segmentation result are selected, pixel evaluation scores are calculated, and the pixel evaluation scores obtained subsequently are better in representativeness.
In particular, the predetermined probability value may be selected as desired. In a specific embodiment, the predetermined probability value may be 0.2-0.4, such as 0.2, 0.3, 0.35, 0.4, and the like, and when the predetermined probability value is 0.2-0.4, the selected pixel segmentation result has a strong significance, and the reference significance is large, and a large number of pixels with the reference significance are not removed, so that the significance of the segmentation result of the training human body part segmentation mark can be better reflected.
And S10413, obtaining the mean value of the maximum segmentation probability values of the reference pixel points to obtain reference pixel evaluation scores.
After the reference pixel point is determined, the reference pixel point can be used for further obtaining a reference pixel evaluation score.
For example, assume that the segmentation probability of the point C in the training human body image is (head region, 0.7; torso region, 0.17; limb region, 0.05; non-human body part, 0.09), and the predetermined probability value is 0.3.
The maximum segmentation probability value of the point C is 0.7 of the probability of the head, is greater than the preset probability value, and can be used as a reference pixel point.
And then, taking the average value of the maximum segmentation probability values of all the reference pixel points in the training human body image, namely obtaining the obtained reference pixel evaluation score of the training human body part segmentation mark. The obtained evaluation score of the reference pixel is not only simple in calculation method, but also can consider all reference pixel points.
Referring to fig. 7, fig. 7 is a schematic flow chart of a method for training a human body part segmentation network according to an embodiment of the present invention, where the method for training a human body part segmentation network includes obtaining a reference intersection ratio evaluation score, and in another specific implementation, in order to make a result of the reference evaluation score more representative, the step of obtaining the reference evaluation score of the training human body part segmentation score by using at least the training human body part segmentation marker may further include:
and S10421, acquiring intersection ratio scores of all the human body part categories by using the training human body part segmentation marks and the reference human body part segmentation marks.
The cross-over ratio score can measure the coincidence degree of each human body part type between the training human body part segmentation mark and the reference human body part segmentation mark, and when the cross-over ratio score is larger, the coincidence degree of the training human body part segmentation mark and the reference human body part segmentation mark of the human body part type is proved to be larger, and the segmentation result is more accurate; when the intersection ratio score is smaller, the coincidence degree of the human body part type on the training human body part segmentation mark and the reference human body part segmentation mark is smaller, and the accuracy degree of the segmentation result is smaller, so that the accuracy evaluation of the training human body part segmentation mark can be realized by utilizing the intersection ratio score.
The calculation method of the intersection ratio may be selected as needed, and in a specific embodiment, the step of obtaining the intersection ratio score of each human body part category by using the training human body part segmentation markers and the reference human body part segmentation markers includes:
acquiring a human body part category corresponding to the maximum segmentation probability value of each pixel point of the training human body image, wherein the human body part category is the predicted human body part category;
marking the region occupied by the pixel points of the same human body part category as a training segmentation region of the human body part category;
obtaining a reference segmentation area of each human body part type by using the human body part type of each pixel point of the training human body image;
and obtaining the intersection ratio score of each human body part type by using the training segmentation region and the reference segmentation region of each human body part type.
For example, in the reference human body part segmentation markers, the reference segmented region of the head is a1, the reference segmented region of the trunk is B1, and the reference segmented region of the limbs is C1; in the training reference human body part segmentation markers, the training segmentation area of the head is a2, the training segmentation area of the trunk is B2, and the training segmentation area of the limbs is C2, and the cross-over ratio score of the head classification, the cross-over ratio score of the trunk classification, and the cross-over ratio score of the limbs classification are the same. Thus, the obtained intersection ratio score can well measure the overlapping degree of each human body part type between the training human body part division mark and the reference human body part division mark.
S10422, obtaining the mean value of the intersection ratio scores of all the human body part types to obtain a reference intersection ratio evaluation score;
if the intersection ratio score of the head classification is obtained as described above, the intersection ratio score of the trunk classification is obtained as described above, and the intersection ratio score of the limb classification is obtained as described above, the reference intersection ratio evaluation score is an arithmetic mean of the intersection ratio scores of the three different classifications.
The reference intersection ratio evaluation score can measure the coincidence degree of all the human body part categories on the training human body part segmentation marks and the reference human body part segmentation marks, and when the reference intersection ratio evaluation score is larger, the coincidence degree of the training human body part segmentation marks and the reference human body part segmentation marks of all the human body part categories is proved to be larger, and the segmentation result is more accurate; when the reference intersection ratio evaluation score is smaller, the superposition degree of the division marks of all the human body part categories on the training human body part and the reference human body part is proved to be smaller, and the accuracy degree of the division result is proved to be smaller.
In another specific embodiment, please refer to fig. 8, and fig. 8 is a schematic flow chart illustrating a process of obtaining a reference quality evaluation score of the human body part segmentation network training method according to the embodiment of the present invention. The human body part segmentation network training method provided by the embodiment of the invention can also realize the acquisition of the reference evaluation score of the training human body part segmentation mark by simultaneously combining the reference intersection ratio evaluation score and the reference quality evaluation score.
Specifically, the step of obtaining the reference evaluation score of the training human body part segmentation marker by using at least the training human body part segmentation marker further includes:
and S10431, acquiring the evaluation score of the reference pixel and the evaluation score of the reference intersection ratio.
The method for obtaining the reference pixel evaluation score and the reference intersection ratio evaluation score is as described above, and is not described herein again.
And S10432, acquiring a reference quality evaluation score of the training human body part segmentation mark according to the reference pixel evaluation score and the reference intersection ratio evaluation score.
The reference quality evaluation score is obtained according to the reference pixel evaluation score and the reference intersection ratio evaluation score, so that the significance degree of the training human body part segmentation mark and the coincidence degree of the training human body part segmentation mark and the reference human body part segmentation mark can be comprehensively measured, and the evaluation effect is better.
In a specific embodiment, the step of obtaining the reference quality evaluation score of the training human body part segmentation marker according to the reference pixel evaluation score and the reference intersection ratio evaluation score includes:
and acquiring a weighted geometric mean of the reference pixel evaluation score and the reference intersection ratio evaluation score according to a preset weight ratio to obtain the reference quality evaluation score.
For example, in the reference quality evaluation score, if the weight ratio of the reference pixel evaluation score to the reference intersection ratio evaluation score is 1:2, the reference pixel evaluation score is 0.7, and the reference intersection ratio evaluation score is 0.68, the reference quality evaluation score is set.
The reference quality evaluation score combines the reference pixel evaluation score and the reference intersection ratio evaluation score, so that the quality of the human body part segmentation result obtained by the human body part segmentation network can be comprehensively measured, and when the reference quality evaluation score is larger, the quality of the human body part segmentation result obtained by the human body part segmentation network is proved to be better; and when the reference quality evaluation score is smaller, the quality of the human body part segmentation result obtained by the human body part segmentation network is proved to be poor.
And S105, obtaining a training evaluation score corresponding to the training human body characteristic according to the training human body characteristic by using an evaluation network to be trained.
In order to train the evaluation network, training evaluation scores need to be acquired, and therefore, the human body part segmentation network training method provided by the invention also acquires the training evaluation scores according to the training human body characteristics by using the evaluation network to be trained.
Wherein the evaluation network is adapted to evaluate the training human body part segmentation markers obtained by the human body part segmentation network, and the evaluation network can predict the similarity between the training human body part segmentation markers and the reference human body part segmentation markers, so as to predict the segmentation quality of the training human body part segmentation markers.
Similarly, the training evaluation score may also include a training pixel evaluation score, a training cross-over ratio evaluation score, and a training quality evaluation score.
Of course, in another embodiment, the evaluation network may include any two or three of a pixel evaluation network, a cross-over ratio evaluation network, and a quality evaluation network, and correspondingly, the reference evaluation score may also include a reference pixel evaluation score, a reference cross-over ratio evaluation score, and a reference quality evaluation score, and the training evaluation score may also include a training pixel evaluation score, a training cross-over ratio evaluation score, and a training quality evaluation score.
The step of obtaining the training evaluation score corresponding to the training human body feature according to the training human body feature by using the evaluation network to be trained comprises the following steps:
obtaining training pixel evaluation scores corresponding to the training human body characteristics according to the training human body characteristics by using a pixel evaluation network to be trained;
or obtaining training cross-comparison evaluation scores corresponding to the training human body characteristics according to the training human body characteristics by using a cross-comparison evaluation network to be trained;
or obtaining a training quality evaluation score corresponding to the training human body characteristic according to the training human body characteristic by using a quality evaluation network to be trained.
The evaluation networks may be identical or different in structure, and may be selected as needed.
Specifically, the structure of the evaluation network may be that refined features are extracted through a plurality of continuous convolution operations on the basis of the human body features, then the obtained results are subjected to dimensionality reduction through an average pooling operation, and finally a score is obtained through full-connection operation regression to serve as a corresponding training evaluation score.
The sequence of step S104 and step S105 is not required, and may be performed simultaneously, or may be performed in other sequences.
Step S106: a segmentation loss is obtained.
The segmentation loss obtained by the reference human body part segmentation marker and the training human body part segmentation marker can be used, and the segmentation loss can measure the accuracy of the obtained result of the human body part segmentation network.
Step S107: and obtaining the evaluation loss.
The evaluation loss may include the accuracy of the result of the evaluation network, which may be obtained by using the training evaluation score and the benchmark evaluation score.
Specifically, the evaluation loss may include a pixel evaluation loss obtained using the training pixel evaluation score and the reference pixel evaluation score, or a cross-over ratio evaluation loss obtained using the training cross-over ratio evaluation score and the reference cross-over ratio evaluation score, or a quality evaluation loss obtained using the training quality evaluation score and the reference quality evaluation score. The three can be selected simultaneously, or only one or two of them can be selected.
The completion sequence of the steps of different paths in the steps S103 to S107 is not required, and it is only required that the step S104 is performed after the step S103, the step S106 is performed after the step S103, the step S107 is performed after the step S105, and the step S107 is performed after the steps S104 and S105.
And step S108, acquiring the global loss.
The global loss is obtained based on a segmentation loss obtained using the reference human body part segmentation markers and the training human body part segmentation markers and an evaluation loss obtained using the training evaluation score and the reference evaluation score. The global loss can be obtained by adding the segmentation loss and the evaluation loss, or can be obtained by other processing methods.
Step S109: and judging whether the global loss meets a global loss threshold, if so, performing step S111, and if not, performing step S110.
And S110, optimizing the human body part segmentation network to be trained and the evaluation network according to the global loss.
The loss function used when the human body part segmentation network to be trained and the evaluation network are optimized can be a mean square error loss function.
Then, after completion of step S110, steps S102 to S108 are resumed.
And S111, obtaining the trained human body part segmentation network and the trained evaluation network.
It can be seen that, in the human body part segmentation network training method provided in the embodiment of the present invention, not only the training of the human body part segmentation network but also the training of the evaluation network are performed, and in the training process, the global loss of model training is obtained according to the segmentation loss obtained by using the reference human body part segmentation markers and the training human body part segmentation markers and the evaluation loss obtained by using the training evaluation scores and the reference evaluation scores, and the human body part segmentation network to be trained and the evaluation network are optimized according to the global loss, while the accuracy of the segmentation result and the accuracy of the evaluation scores are considered, and the improvement of the accuracy of the evaluation network can improve the accuracy of the human body part segmentation network. Therefore, when the human body part is actually segmented, the accuracy of the human body part segmentation result obtained by utilizing the trained human body part segmentation network is higher.
Referring to fig. 9, fig. 9 is a schematic flow chart of a human body part segmentation method according to an embodiment of the present invention, and the embodiment of the present invention further provides a human body part segmentation method, including:
step S201, acquiring a single image to be segmented;
the single image to be segmented is an image which only contains one human body and needs to be segmented.
The step of acquiring the single image to be segmented may be selected as needed, please refer to fig. 10, where fig. 10 is a schematic flow chart of a human body segmentation method according to an embodiment of the present invention, and in a specific implementation manner, the step S201 of acquiring the single image to be segmented may include:
step S2011, an image to be processed is acquired.
The image to be processed is an original image which may contain a plurality of human bodies and needs to be segmented.
And S2012, carrying out single person distinguishing on the image to be processed to obtain each single person image to be segmented and the image position of the single person image to be segmented in the image to be processed.
The image to be processed can be distinguished by one person by utilizing a network or a processing module which can be distinguished by one person, so that each image to be divided by one person and the image position of the image to be divided by one person in the image to be processed can be obtained.
Step S202, obtaining the human body characteristics of the single image to be segmented;
the method for obtaining the human body features may be the same as the method for obtaining the training human body features, and is not described herein again.
And S203, obtaining the human body part segmentation result corresponding to the single image to be segmented according to the human body characteristics by utilizing the human body part segmentation network obtained by training the human body part segmentation network training method.
It can be seen that in the human body part segmentation method provided by the embodiment of the invention, the accuracy of the segmentation result and the accuracy of the evaluation score are considered during training of the utilized human body part segmentation network, and the accuracy of the human body part segmentation network can be improved by improving the accuracy of the evaluation network. Therefore, when the human body part is actually segmented, the accuracy of the human body part segmentation result obtained by utilizing the trained human body part segmentation network is higher.
Of course, the obtained human body part segmentation results corresponding to the single image to be segmented may also be combined to obtain the human body part segmentation result corresponding to the image to be processed, so please refer to fig. 10 again, in an embodiment, the human body part segmentation method further includes:
and S204, combining the human body part segmentation results corresponding to the single image to be segmented according to the image position of the corresponding single image to be segmented in the image to be processed to obtain the human body part segmentation result of the image to be processed.
The image position of the corresponding single image to be segmented in the image to be processed is utilized to combine the human body part segmentation results corresponding to the single image to be segmented, so that the human body part segmentation results of the image to be processed can be obtained, the whole process can meet the practical application requirements, and the method is convenient.
The human body part segmentation network training device and the human body part segmentation device provided by the embodiments of the present invention are introduced below, and the human body part segmentation network training device and the human body part segmentation device described below may be regarded as a functional module architecture required to be arranged by an electronic device (e.g., a PC) to respectively implement the human body part segmentation network training method and the human body part segmentation method provided by the embodiments of the present invention. The contents of the human body part segmentation network training apparatus and the human body part segmentation apparatus described below may be referred to in correspondence with the contents of the human body part segmentation network training method and the human body part segmentation method described above, respectively.
Referring to fig. 11, fig. 11 is a schematic diagram of a human body part segmentation network training device according to an embodiment of the present invention, where the embodiment of the present invention provides a human body part segmentation network training device, including:
a training human body image data obtaining unit 11 adapted to obtain training human body image data, the training human body image data including a training human body image and a reference human body part segmentation label of the training human body image, the reference human body part segmentation label including a human body part category of each pixel point of the training human body image;
a training human body feature obtaining unit 12 adapted to obtain training human body features of the training human body image;
a training human body part segmentation marker acquisition unit 13 adapted to acquire a training human body part segmentation marker according to the training human body feature using a human body part segmentation network 21 to be trained;
a reference evaluation score obtaining unit 14 adapted to obtain a reference evaluation score of the training human body part division mark by using at least the training human body part division mark, wherein the reference evaluation score is used for evaluating a division result of the training human body part division mark;
a training evaluation score obtaining unit 15, adapted to obtain, by using an evaluation network 22 to be trained, a training evaluation score corresponding to the training human body feature according to the training human body feature, where the evaluation network 22 is adapted to evaluate the training human body part segmentation marker obtained by the human body part segmentation network 21;
the model training unit 16 is adapted to obtain a global loss of model training according to a segmentation loss obtained by using the reference human body part segmentation flag and the training human body part segmentation flag and an evaluation loss obtained by using the training evaluation score and the reference evaluation score, and optimize the human body part segmentation network 21 and the evaluation network 22 to be trained according to the global loss until the global loss satisfies a global loss threshold, so as to obtain the trained human body part segmentation network 21 and the trained evaluation network 22.
It can be seen that, in the human body part segmentation network training apparatus provided in the embodiment of the present invention, not only the training of the human body part segmentation network 21 but also the training of the evaluation network 22 are performed, and in the training process, a global loss of model training is obtained according to a segmentation loss obtained by using the reference human body part segmentation markers and the training human body part segmentation markers and an evaluation loss obtained by using the training evaluation scores and the reference evaluation scores, and the human body part segmentation network 21 and the evaluation network 22 to be trained are optimized according to the global loss, and meanwhile, the accuracy of the segmentation result and the accuracy of the evaluation scores are considered, and the improvement of the accuracy of the evaluation network 22 can improve the accuracy of the human body part segmentation network 21. Therefore, when the human body part segmentation is actually carried out, the accuracy of the human body part segmentation result obtained by using the trained human body part segmentation network 21 is high.
Optionally, the training human body part segmentation marker includes a segmentation probability value of each pixel point of the training human body image, the segmentation probability value includes a probability value that the pixel point is of each human body part type or non-human body part type, and the reference evaluation score obtaining unit 14 includes a reference pixel evaluation score obtaining unit adapted to obtain a maximum segmentation probability value of each pixel point of the training human body image by using the segmentation probability value; selecting a reference pixel point according to a preset requirement; obtaining the mean value of the maximum segmentation probability values of all the reference pixel points to obtain reference pixel evaluation scores;
the training evaluation score obtaining unit 15 includes a training pixel evaluation score obtaining unit, and is adapted to obtain a training pixel evaluation score corresponding to the training human body feature according to the training human body feature by using a pixel evaluation network to be trained;
the evaluation loss includes:
a pixel evaluation loss obtained using the training pixel evaluation score and the reference pixel evaluation score.
Optionally, the reference pixel evaluation score obtaining unit is adapted to select all the pixel points as reference pixel points.
Optionally, the reference pixel evaluation score obtaining unit is adapted to obtain a pixel point with a maximum segmentation probability value greater than or equal to a predetermined probability value as a reference pixel point.
Optionally, the predetermined probability value ranges from 0.2 to 0.4.
Optionally, the training human body part segmentation markers include predicted human body part categories of each pixel point of the training human body image; the reference evaluation score obtaining unit 14 includes a reference intersection ratio evaluation score obtaining unit adapted to obtain an intersection ratio score of each of the human body part categories using the training human body part segmentation markers and the reference human body part segmentation markers; acquiring the mean value of the intersection ratio scores of all the human body part categories to obtain a reference intersection ratio evaluation score;
the training evaluation score obtaining unit 15 includes a training cross-over ratio evaluation score obtaining unit, and is adapted to obtain a training cross-over ratio evaluation score corresponding to the training human body feature according to the training human body feature by using a cross-over ratio evaluation network to be trained;
the evaluation loss includes: and evaluating loss by using the training cross ratio evaluation score and the reference cross ratio evaluation score.
Optionally, the training human body part segmentation marker includes a segmentation probability value of each pixel point of the training human body image, and the segmentation probability value includes a probability value that the pixel point is of each human body part type or non-human body part type; the reference intersection ratio evaluation score obtaining unit is suitable for obtaining a human body part category corresponding to the maximum segmentation probability value of each pixel point of the training human body image, and the human body part category is the predicted human body part category; marking the region occupied by the pixel points of the same human body part category as a training segmentation region of the human body part category; obtaining a reference segmentation area of each human body part type by using the human body part type of each pixel point of the training human body image; and obtaining the intersection ratio score of each human body part type by using the training segmentation region and the reference segmentation region of each human body part type.
Optionally, the reference evaluation score obtaining unit 14 includes a reference quality evaluation score obtaining unit, and is adapted to obtain a maximum segmentation probability value of each pixel of the training human body image by using the segmentation probability value, select a reference pixel according to a predetermined requirement, and obtain a mean value of the maximum segmentation probability values of each reference pixel to obtain a reference pixel evaluation score; acquiring a reference quality evaluation score of the training human body part segmentation mark according to the reference pixel evaluation score and the reference intersection ratio evaluation score;
the training evaluation score obtaining unit 15 includes a training quality evaluation score obtaining unit adapted to obtain a training quality evaluation score corresponding to the training human body feature according to the training human body feature by using a quality evaluation network to be trained;
the evaluation loss includes: and obtaining a quality evaluation loss by using the training quality evaluation score and the reference quality evaluation score.
Optionally, the training quality evaluation score obtaining unit is adapted to obtain a weighted geometric mean of the reference pixel evaluation score and the reference intersection ratio evaluation score according to a predetermined weight ratio, so as to obtain the reference quality evaluation score.
Optionally, in the quality evaluation scores, a weight ratio of the pixel evaluation score to the intersection ratio evaluation score ranges from 1:2 to 1: 4.
Optionally, the training human body feature obtaining unit 12 is adapted to obtain at least a first size training human body feature and a second size training human body feature of the training human body image;
and at least fusing the first size training human body characteristic and the second size training human body characteristic to obtain the training human body characteristic.
Optionally, the training human body image data obtaining unit 11 is adapted to:
carrying out single distinguishing on the original image to obtain a single distinguishing image of each human body;
distinguishing the image by the single person to obtain an image before segmentation;
and carrying out human body part segmentation on the pre-segmentation image to obtain the training human body image data.
Optionally, the training human body image data obtaining unit 11 is further adapted to:
placing the single distinguishing image in a pre-segmentation image with a standard size, filling the part except the single distinguishing image into an average value point, wherein the pixel value of the average value point is the average value of the minimum value and the maximum value of the value range of the pixel value of the image;
and segmenting human body parts in the pre-segmentation image with the standard size to obtain the training human body image data and reference human body part segmentation marks of all points of the training human body image data, wherein the human body part types of the parts except the single-person segmentation image in the pre-segmentation image are non-human body parts.
Referring to fig. 12, fig. 12 is a schematic view of a human body part segmentation apparatus according to an embodiment of the present invention, which includes:
a single image-to-be-segmented acquiring unit 31 adapted to acquire a single image to be segmented;
a human body feature obtaining unit 32, adapted to obtain a human body feature of the single image to be segmented;
the human body part segmentation network 21 is obtained by training through the human body part segmentation network 21 training method and is suitable for obtaining a human body part segmentation result corresponding to the single image to be segmented according to the human body characteristics.
Optionally, the single image obtaining unit 31 to be segmented includes:
a to-be-processed image acquiring unit 311 adapted to acquire a to-be-processed image;
a single distinguishing unit 312, adapted to distinguish the to-be-processed image by a single person to obtain each single to-be-segmented image and an image position of the single to-be-segmented image in the to-be-processed image;
the human body part segmenting device further comprises: and the segmentation result combination unit 33 is adapted to combine the human body part segmentation results corresponding to the single images to be segmented according to the image positions of the corresponding single images to be segmented in the images to be processed, so as to obtain the human body part segmentation results of the images to be processed.
Certainly, the embodiment of the present invention further provides an electronic device, where the electronic device provided in the embodiment of the present invention may load a program module architecture in a program form to implement the human body part segmentation network training method and the human body part segmentation method provided in the embodiment of the present invention; the hardware device can be applied to an electronic device with specific data processing capacity, and the electronic device can be: such as a terminal device or a server device.
Therefore, referring to fig. 13, fig. 13 is a schematic view of an electronic device according to an embodiment of the invention.
The electronic device provided by the embodiment of the invention comprises: at least one memory 41 and at least one processor 42, the memory 41 storing one or more computer-executable instructions, the processor 42 invoking the one or more computer-executable instructions to perform the human body part segmentation network training method, or the human body part segmentation method.
It will be appreciated that the device may also comprise at least one communication interface 43 and at least one communication bus 44; processor 42 and memory 41 may be located on the same electronic device, for example processor 42 and memory 41 may be located on a server device or a terminal device; the processor 42 and the memory 41 may also be located on different electronic devices.
In the embodiment of the present invention, the electronic device may be a tablet computer, a notebook computer, or the like capable of performing a human body part segmentation network training method or a human body part segmentation method.
In the embodiment of the present invention, the number of the processor 42, the communication interface 43, the memory 41 and the communication bus 44 is at least one, and the processor 42, the communication interface 43 and the memory 41 complete the communication with each other through the communication bus 44; it is clear that the communication connection of the processor 42, the communication interface 43, the memory 41 and the communication bus 44 shown in fig. 12 is only an alternative.
Alternatively, the communication interface 43 may be an interface of a communication module, such as an interface of the GS10M module; processor 42 may be a central processing unit CPU or a specific integrated circuit AS10IC or one or more integrated circuits configured to implement an embodiment of the present invention; the memory 41 may comprise high speed RAM memory, and may also include non-volatile memory, such as at least one disk memory.
It should be noted that the above-mentioned apparatus may also include other devices (not shown) that may not be necessary to the disclosure of the embodiments of the present invention; these other components may not be necessary to understand the disclosure of embodiments of the present invention, which are not individually described herein.
The embodiment of the invention also provides a storage medium, wherein the storage medium stores one or more computer-executable instructions, and the one or more computer-executable instructions are used for executing the human body part segmentation network training method or the human body part segmentation method.
The embodiments of the present invention described above are combinations of elements and features of the present invention. Unless otherwise mentioned, the elements or features may be considered optional. Each element or feature may be practiced without being combined with other elements or features. In addition, the embodiments of the present invention may be configured by combining some elements and/or features. The order of operations described in the embodiments of the present invention may be rearranged. Some configurations of any embodiment may be included in another embodiment, and may be replaced with corresponding configurations of the other embodiment. It is obvious to those skilled in the art that claims that are not explicitly cited in each other in the appended claims may be combined into an embodiment of the present invention or may be included as new claims in a modification after the filing of the present application.
Embodiments of the invention may be implemented by various means, such as hardware, firmware, software, or a combination thereof. In a hardware configuration, the method according to an exemplary embodiment of the present invention may be implemented by one or more application specific integrated circuits (AS10IC), a digital signal processor DS10P), a digital signal processing device (DS10PD), a Programmable Logic Device (PLD), a Field Programmable Gate Array (FPGA), a processor, a controller, a microcontroller, a microprocessor, or the like.
In a firmware or software configuration, embodiments of the present invention may be implemented in the form of modules, procedures, functions, and the like. The software codes may be stored in memory units and executed by processors. The memory unit is located inside or outside the processor, and may transmit and receive data to and from the processor via various known means.
The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Although the embodiments of the present invention are disclosed above, the embodiments of the present invention are not limited thereto. Various changes and modifications may be effected therein by one of ordinary skill in the pertinent art without departing from the scope or spirit of the present embodiments, and it is intended that the scope of the present embodiments be defined by the appended claims.

Claims (20)

1. A human body part segmentation network training method is characterized by comprising the following steps:
acquiring training human body image data, wherein the training human body image data comprises a training human body image and a reference human body part segmentation mark of the training human body image, and the reference human body part segmentation mark comprises human body part categories of all pixel points of the training human body image;
acquiring training human body characteristics of the training human body image;
acquiring training human body part segmentation marks according to the training human body characteristics by using a human body part segmentation network to be trained;
acquiring a reference evaluation score of the training human body part segmentation mark at least by using the training human body part segmentation mark, wherein the reference evaluation score is used for evaluating a segmentation result of the training human body part segmentation mark;
obtaining training evaluation scores corresponding to the training human body features according to the training human body features by using an evaluation network to be trained, wherein the evaluation network is suitable for evaluating the training human body part segmentation marks obtained by the human body part segmentation network;
and according to the segmentation loss obtained by using the reference human body part segmentation mark and the training human body part segmentation mark and the evaluation loss obtained by using the training evaluation score and the reference evaluation score, obtaining the global loss of model training, and according to the global loss, optimizing the human body part segmentation network to be trained and the evaluation network until the global loss meets a global loss threshold value, so as to obtain the trained human body part segmentation network and the trained evaluation network.
2. The human body part segmentation network training method according to claim 1, wherein the training human body part segmentation markers comprise segmentation probability values of respective pixel points of the training human body image, the segmentation probability values comprise probability values of the pixel points being of respective human body part classes or non-human body part classes;
the step of obtaining the reference evaluation score of the training human body part segmentation mark at least by using the training human body part segmentation mark comprises the following steps:
acquiring the maximum segmentation probability value of each pixel point of the training human body image by using the segmentation probability value;
selecting a reference pixel point according to a preset requirement;
obtaining the mean value of the maximum segmentation probability values of all the reference pixel points to obtain reference pixel evaluation scores;
the step of obtaining the training evaluation score corresponding to the training human body feature according to the training human body feature by using the evaluation network to be trained comprises the following steps:
obtaining training pixel evaluation scores corresponding to the training human body characteristics according to the training human body characteristics by using a pixel evaluation network to be trained;
the evaluation loss includes:
a pixel evaluation loss obtained using the training pixel evaluation score and the reference pixel evaluation score.
3. The human body part segmentation network training method according to claim 2, wherein the step of selecting the reference pixel points according to the predetermined requirement comprises:
and selecting all the pixel points as reference pixel points.
4. The human body part segmentation network training method according to claim 2, wherein the step of selecting the reference pixel points according to the predetermined requirement comprises:
and acquiring pixel points with the maximum segmentation probability value larger than or equal to the preset probability value as reference pixel points.
5. The human body part segmentation network training method as claimed in claim 4, wherein the predetermined probability value ranges from 0.2 to 0.4.
6. The human body part segmentation network training method of claim 1,
the training human body part segmentation mark comprises a predicted human body part category of each pixel point of the training human body image;
the step of obtaining the reference evaluation score of the training human body part segmentation mark at least by using the training human body part segmentation mark comprises the following steps:
acquiring intersection ratio scores of the human body part categories by using the training human body part segmentation markers and the reference human body part segmentation markers;
acquiring the mean value of the intersection ratio scores of all the human body part categories to obtain a reference intersection ratio evaluation score;
the step of obtaining the training evaluation score corresponding to the training human body feature according to the training human body feature by using the evaluation network to be trained comprises the following steps:
obtaining training cross-comparison evaluation scores corresponding to the training human body characteristics according to the training human body characteristics by using a cross-comparison evaluation network to be trained;
the evaluation loss includes:
and evaluating loss by using the training cross ratio evaluation score and the reference cross ratio evaluation score.
7. The human body part segmentation network training method according to claim 6, wherein the training human body part segmentation markers comprise segmentation probability values of respective pixel points of the training human body image, the segmentation probability values comprise probability values of the pixel points being in respective human body part classes or non-human body part classes;
the step of obtaining the intersection ratio score of each human body part category by using the training human body part segmentation markers and the reference human body part segmentation markers comprises the following steps:
acquiring a human body part category corresponding to the maximum segmentation probability value of each pixel point of the training human body image, wherein the human body part category is the predicted human body part category;
marking the region occupied by the pixel points of the same human body part category as a training segmentation region of the human body part category;
obtaining a reference segmentation area of each human body part type by using the human body part type of each pixel point of the training human body image;
and obtaining the intersection ratio score of each human body part type by using the training segmentation region and the reference segmentation region of each human body part type.
8. The human body part segmentation network training method according to claim 6, wherein the training human body part segmentation markers comprise segmentation probability values of respective pixel points of the training human body image, the segmentation probability values comprise probability values of the pixel points being in respective human body part classes or non-human body part classes;
the step of obtaining the reference evaluation score of the training human body part segmentation mark by using at least the training human body part segmentation mark further comprises:
acquiring the maximum segmentation probability value of each pixel point of the training human body image by using the segmentation probability value, selecting a reference pixel point according to a preset requirement, and acquiring the mean value of the maximum segmentation probability values of each reference pixel point to obtain a reference pixel evaluation score;
acquiring a reference quality evaluation score of the training human body part segmentation mark according to the reference pixel evaluation score and the reference intersection ratio evaluation score;
the step of obtaining the training evaluation score corresponding to the training human body feature according to the training human body feature by using the evaluation network to be trained comprises the following steps:
obtaining a training quality evaluation score corresponding to the training human body characteristic according to the training human body characteristic by using a quality evaluation network to be trained;
the evaluation loss includes:
and obtaining a quality evaluation loss by using the training quality evaluation score and the reference quality evaluation score.
9. The human body part segmentation network training method according to claim 8, wherein the step of obtaining the reference quality evaluation score of the training human body part segmentation marker according to the reference pixel evaluation score and the reference intersection ratio evaluation score comprises:
and acquiring a weighted geometric mean of the reference pixel evaluation score and the reference intersection ratio evaluation score according to a preset weight ratio to obtain the reference quality evaluation score.
10. The human body part segmentation network training method according to claim 8, wherein in the quality evaluation scores, a weight ratio of the pixel evaluation score to the intersection ratio evaluation score ranges from 1:2 to 1: 4.
11. The human body part segmentation network training method according to any one of claims 1 to 10, wherein the step of obtaining training human body features of the training human body image comprises:
acquiring at least a first size training human body feature and a second size training human body feature of the training human body image;
and at least fusing the first size training human body characteristic and the second size training human body characteristic to obtain the training human body characteristic.
12. The human body part segmentation network training method according to any one of claims 1 to 10, wherein the step of acquiring training human body image data comprises:
carrying out single distinguishing on the original image to obtain a single distinguishing image of each human body;
distinguishing the image by the single person to obtain an image before segmentation;
and carrying out human body part segmentation on the pre-segmentation image to obtain the training human body image data.
13. The human body part segmentation network training method as claimed in claim 12, wherein the step of using the single person difference image to obtain the image before segmentation comprises:
placing the single distinguishing image in a pre-segmentation image with a standard size, filling the part except the single distinguishing image into an average value point, wherein the pixel value of the average value point is the average value of the minimum value and the maximum value of the value range of the pixel value of the image;
the step of performing human body part segmentation on the pre-segmentation image to obtain the training human body image data comprises the following steps:
and segmenting human body parts in the pre-segmentation image with the standard size to obtain the training human body image data and reference human body part segmentation marks of all points of the training human body image data, wherein the human body part types of the parts except the single-person segmentation image in the pre-segmentation image are non-human body parts.
14. A human body part segmentation method is characterized by comprising the following steps:
acquiring a single image to be segmented;
acquiring human body characteristics of the single image to be segmented;
the human body part segmentation network obtained by training by using the human body part segmentation network training method according to any one of claims 1 to 13, and obtaining a human body part segmentation result corresponding to the single image to be segmented according to the human body characteristics.
15. The human body part segmentation method as set forth in claim 14, wherein the step of acquiring the image to be segmented by the single person comprises:
acquiring an image to be processed;
carrying out single distinguishing on the image to be processed to obtain each single image to be segmented and the image position of the single image to be segmented in the image to be processed;
further comprising:
combining the human body part segmentation results corresponding to the single image to be segmented according to the image position of the corresponding single image to be segmented in the image to be processed to obtain the human body part segmentation result of the image to be processed.
16. A human body part segmentation network training device is characterized by comprising:
a training human body image data acquisition unit adapted to acquire training human body image data, the training human body image data including a training human body image and a reference human body part segmentation label of the training human body image, the reference human body part segmentation label including a human body part category of each pixel point of the training human body image;
the training human body feature acquisition unit is suitable for acquiring training human body features of the training human body image;
the training human body part segmentation mark acquisition unit is suitable for acquiring a training human body part segmentation mark according to the training human body characteristics by using a human body part segmentation network to be trained;
a reference evaluation score acquisition unit adapted to acquire a reference evaluation score of the training human body part division mark by using at least the training human body part division mark, wherein the reference evaluation score is used for evaluating a division result of the training human body part division mark;
the training evaluation score obtaining unit is suitable for obtaining a training evaluation score corresponding to the training human body characteristics according to the training human body characteristics by utilizing an evaluation network to be trained, wherein the evaluation network is suitable for evaluating the training human body part segmentation marks obtained by the human body part segmentation network;
and the model training unit is suitable for acquiring the global loss of model training according to the segmentation loss acquired by the reference human body part segmentation mark and the training human body part segmentation mark and the evaluation loss acquired by the training evaluation score and the reference evaluation score, and optimizing the human body part segmentation network to be trained and the evaluation network according to the global loss until the global loss meets a global loss threshold value to obtain the trained human body part segmentation network and the trained evaluation network.
17. A human body part segmentation apparatus, comprising:
the single image acquisition unit to be segmented is suitable for acquiring a single image to be segmented;
the human body characteristic acquisition unit is suitable for acquiring the human body characteristics of the single image to be segmented;
a human body part segmentation network obtained by training with the human body part segmentation network training method of any one of claims 1 to 13, and adapted to obtain a human body part segmentation result corresponding to the single image to be segmented according to the human body characteristics.
18. The human body part segmentation apparatus as set forth in claim 17, wherein the single image acquisition unit to be segmented comprises:
the image processing device comprises a to-be-processed image acquisition unit, a processing unit and a processing unit, wherein the to-be-processed image acquisition unit is suitable for acquiring an image to be processed;
the single distinguishing unit is suitable for distinguishing the image to be processed by a single person to obtain each single image to be segmented and the image position of the single image to be segmented in the image to be processed;
the human body part segmenting device further comprises: and the segmentation result combination unit is suitable for combining the human body part segmentation results corresponding to the single image to be segmented according to the image position of the corresponding single image to be segmented in the image to be processed to obtain the human body part segmentation result of the image to be processed.
19. A storage medium characterized in that the storage medium stores a program adapted to train a human body part segmentation network to realize the human body part segmentation network training method according to any one of claims 1 to 13, or the storage medium stores a program adapted to perform human body part segmentation to realize the human body part segmentation method according to claim 14 or 15.
20. An electronic device comprising at least one memory and at least one processor; the memory stores a program that the processor calls to perform the human body part segmentation network training method according to any one of claims 1 to 13 or the human body part segmentation method according to claim 14 or 15.
CN202110515510.2A 2021-05-12 2021-05-12 Human body part segmentation network training method, human body part segmentation method and device Active CN112950637B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110515510.2A CN112950637B (en) 2021-05-12 2021-05-12 Human body part segmentation network training method, human body part segmentation method and device

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110515510.2A CN112950637B (en) 2021-05-12 2021-05-12 Human body part segmentation network training method, human body part segmentation method and device

Publications (2)

Publication Number Publication Date
CN112950637A true CN112950637A (en) 2021-06-11
CN112950637B CN112950637B (en) 2021-08-20

Family

ID=76233766

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110515510.2A Active CN112950637B (en) 2021-05-12 2021-05-12 Human body part segmentation network training method, human body part segmentation method and device

Country Status (1)

Country Link
CN (1) CN112950637B (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114863227A (en) * 2022-05-24 2022-08-05 重庆中科云从科技有限公司 Model evaluation method, system and computer storage medium

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107123123A (en) * 2017-05-02 2017-09-01 电子科技大学 Image segmentation quality evaluating method based on convolutional neural networks
CN109242864A (en) * 2018-09-18 2019-01-18 电子科技大学 Image segmentation result quality evaluating method based on multiple-limb network
CN109934223A (en) * 2019-03-01 2019-06-25 北京地平线机器人技术研发有限公司 A kind of example segmentation determination method, neural network model training method and device neural network based
CN110930417A (en) * 2019-11-26 2020-03-27 腾讯科技（深圳）有限公司 Training method and device of image segmentation model, and image segmentation method and device
US20210089845A1 (en) * 2019-09-20 2021-03-25 Samsung Electronics Co., Ltd. Teaching gan (generative adversarial networks) to generate per-pixel annotation

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107123123A (en) * 2017-05-02 2017-09-01 电子科技大学 Image segmentation quality evaluating method based on convolutional neural networks
CN109242864A (en) * 2018-09-18 2019-01-18 电子科技大学 Image segmentation result quality evaluating method based on multiple-limb network
CN109934223A (en) * 2019-03-01 2019-06-25 北京地平线机器人技术研发有限公司 A kind of example segmentation determination method, neural network model training method and device neural network based
US20210089845A1 (en) * 2019-09-20 2021-03-25 Samsung Electronics Co., Ltd. Teaching gan (generative adversarial networks) to generate per-pixel annotation
CN110930417A (en) * 2019-11-26 2020-03-27 腾讯科技（深圳）有限公司 Training method and device of image segmentation model, and image segmentation method and device

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
宁阳: "基于图像增强和循环对抗学习的自动胰腺分割方法研究", 《中国博士学位论文全文数据库（电子期刊）医药卫生科技辑》 *

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114863227A (en) * 2022-05-24 2022-08-05 重庆中科云从科技有限公司 Model evaluation method, system and computer storage medium

Also Published As

Publication number Publication date
CN112950637B (en) 2021-08-20

Similar Documents

Publication Publication Date Title
CN107944450B (en) License plate recognition method and device
CN109740668B (en) Deep model training method and device, electronic equipment and storage medium
US10169683B2 (en) Method and device for classifying an object of an image and corresponding computer program product and computer-readable medium
CN110969175B (en) Wafer processing method and device, storage medium and electronic equipment
CN105354307A (en) Image content identification method and apparatus
CN106951911A (en) A kind of quick multi-tag picture retrieval system and implementation method
CN111126481A (en) Training method and device of neural network model
CN111340213B (en) Neural network training method, electronic device, and storage medium
CN110457677A (en) Entity-relationship recognition method and device, storage medium, computer equipment
CN111199186A (en) Image quality scoring model training method, device, equipment and storage medium
CN111723666A (en) Signal identification method and device based on semi-supervised learning
CN112950637B (en) Human body part segmentation network training method, human body part segmentation method and device
CN108921162A (en) Licence plate recognition method and Related product based on deep learning
CN109635755A (en) Face extraction method, apparatus and storage medium
CN112241736B (en) Text detection method and device
CN112364974A (en) Improved YOLOv3 algorithm based on activation function
CN110135428B (en) Image segmentation processing method and device
CN117315237B (en) Method and device for determining target detection model and storage medium
US20210124974A1 (en) Method, apparatus, and device for processing image and storage medium
CN113139581A (en) Image classification method and system based on multi-image fusion
CN112950652B (en) Robot and hand image segmentation method and device thereof
CN112040401B (en) Indoor positioning method and device, electronic equipment and storage medium
CN114419370A (en) Target image processing method and device, storage medium and electronic equipment
CN113987170A (en) Multi-label text classification method based on convolutional neural network
CN114662580A (en) Training method, classification method, device, equipment and medium of data classification model

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant