CN113449570A - Image processing method and device - Google Patents
Image processing method and device Download PDFInfo
- Publication number
- CN113449570A CN113449570A CN202010231605.7A CN202010231605A CN113449570A CN 113449570 A CN113449570 A CN 113449570A CN 202010231605 A CN202010231605 A CN 202010231605A CN 113449570 A CN113449570 A CN 113449570A
- Authority
- CN
- China
- Prior art keywords
- human body
- model
- result
- value
- image
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T13/00—Animation
- G06T13/20—3D [Three Dimensional] animation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/20—Analysis of motion
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
Abstract
The invention discloses an image processing method and device. Wherein, the method comprises the following steps: acquiring an original image; carrying out human body detection on the original image to obtain a human body image; processing the human body image by using the trained first model to obtain a processing result of the human body image, wherein the processing result comprises: two-dimensional joint points, three-dimensional joint points and a skin multi-person linear SMPL model; and generating a human body model according to the processing result of the human body image. The invention solves the technical problem of lower identification accuracy for positioning two-dimensional and three-dimensional joint points and reconstructing human body parameters in the related technology.
Description
Technical Field
The invention relates to the technical field of computer vision, in particular to an image processing method and device.
Background
Related human techniques in the industry today include human detection, two-dimensional and three-dimensional joint point localization, segmentation, and the like. Aiming at parts such as two-dimensional and three-dimensional joint point positioning, human parameter coefficient reconstruction and the like, the following scheme can be adopted at present: 1) firstly, the image is detected by a deep learning scheme, a human body area is intercepted after the detection is finished, two-dimensional joint points are estimated by a deep learning network, and then three-dimensional joint points, human body postures and shape parameters are estimated by the two-dimensional joint points. However, when a three-dimensional joint is estimated by using two-dimensional joints, there may be ambiguity of motion, for example, two-dimensional joints in the same state may correspond to three-dimensional joints in different positions, and the recognition accuracy of the three-dimensional joints depends on the recognition accuracy of the two-dimensional joints, resulting in low recognition accuracy of the three-dimensional joints. 2) Firstly, the image is detected by a deep learning scheme, a human body region is intercepted after the detection is finished, the three-dimensional joint points are directly predicted by a deep learning network, the three-dimensional joint points are changed into three-dimensional voxel grids, the possibility of each voxel grid of each joint is deduced, and therefore training and prediction are carried out. However, because the samples of the three-dimensional joint points are difficult to obtain, most training samples are collected in a laboratory environment, the robustness to outdoor scenes is not high, and the voxel grid is adopted for prediction, so that the calculation amount is large, and the real-time performance is low. 3) Firstly, detecting a human body, then carrying out human segmentation or human analysis on the detected picture, and then carrying out human model estimation by an optimization method by using segmentation and analysis results. However, the human body segmentation and analysis requirements are too high, and the result deviation can affect the human body reconstruction effect.
In view of the problems in the above solutions, no effective solution has been proposed at present.
Disclosure of Invention
The embodiment of the invention provides an image processing method and device, which at least solve the technical problem of low identification accuracy for positioning two-dimensional and three-dimensional joint points and reconstructing human body parameters in the related technology.
According to an aspect of an embodiment of the present invention, there is provided an image processing method including: acquiring an original image; carrying out human body detection on the original image to obtain a human body image; processing the human body image by using the trained first model to obtain a processing result of the human body image, wherein the processing result comprises: two-dimensional joint points, three-dimensional joint points and a skin multi-person linear SMPL model; and generating a human body model according to the processing result of the human body image.
Optionally, the method further comprises: obtaining a plurality of groups of training samples, wherein each group of training samples comprises: the method comprises the following steps of (1) obtaining a human body image, first marking information of a two-dimensional joint point, second marking information of a three-dimensional joint point and a parameter value of an SMPL model; training a preset model by utilizing a plurality of groups of training samples, and acquiring a target loss value of the preset model; stopping training the preset model under the condition that the target loss value is smaller than the preset value, and determining the preset model as a first model; and under the condition that the target loss value is larger than the preset value, continuously training the preset model by using the multiple groups of training samples until the target loss value is smaller than the preset value.
Optionally, training the preset model by using multiple sets of training samples, and obtaining the target loss value of the preset model includes: inputting a plurality of groups of training samples into a preset model, and obtaining an output result of the preset model, wherein the output result comprises: a first result of a two-dimensional joint point, a second result of a three-dimensional joint point, and a third result of an SMPL model; obtaining a first loss value of the two-dimensional joint point based on the first mark information and the first result; obtaining a second loss value of the three-dimensional joint point based on the second marking information and the second result; obtaining a third loss value of the SMPL model based on the parameter value and the third result; and obtaining a target loss value based on the first loss value, the second loss value and the third loss value.
Optionally, the parameter value of the SMPL model is real data acquired by the acquisition device, or adjustment data obtained by adjusting the parameter value acquired by the acquisition device.
Optionally, deriving a third loss value for the SMPL model based on the parameter value and the third result comprises: obtaining a third loss value based on the parameter value and the third result under the condition that the parameter value is the real value acquired by the acquisition device; and under the condition that the parameter value is an adjustment value obtained by adjusting the parameter value acquired by the acquisition device, acquiring a three-dimensional joint point based on the parameter value, projecting the three-dimensional joint point onto a two-dimensional plane to obtain a two-dimensional joint point, acquiring a fourth loss value of the two-dimensional joint point based on the projected two-dimensional joint point and the first mark information, and determining the fourth loss value as a third loss value.
Optionally, the method further comprises: processing the parameter value of the third result by using a discriminator to obtain a classification result of the parameter value of the third result, wherein the classification result is used for representing whether the parameter value of the third result is a real value acquired by an acquisition device; and determining whether to stop training the preset model or not based on the classification result and the target loss value.
Optionally, the arbiter is trained with a generative confrontation network.
Optionally, the human body detection is performed on the original image, and obtaining the human body image includes: processing the original image by using the trained second model to obtain the position information of the human body in the original image; and cutting and normalizing the original image based on the position information to obtain a human body image.
Optionally, the first model employs an hourglass-type network structure or a feature map pyramid FPN network structure.
According to another aspect of the embodiments of the present invention, there is also provided an image processing apparatus including: the acquisition module is used for acquiring an original image; the detection module is used for carrying out human body detection on the original image to obtain a human body image; the processing module is used for processing the human body image by utilizing the trained first model to obtain a processing result of the human body image, wherein the processing result comprises: two-dimensional joint points, three-dimensional joint points and a skin multi-person linear SMPL model; and the generating module is used for generating a human body model according to the processing result of the human body image.
According to another aspect of the embodiments of the present invention, there is also provided a storage medium including a stored program, wherein when the program is executed, an apparatus in which the storage medium is located is controlled to execute the above-mentioned image processing method.
According to another aspect of the embodiments of the present invention, there is also provided a processor, configured to execute a program, where the program executes the image processing method described above.
In the embodiment of the invention, after the original image is obtained, firstly, the original image is subjected to human body detection to obtain the human body image, and then the trained first model is used for processing the human body image to obtain the processing result of the human body image, so that the purposes of human body detection, two-dimensional and three-dimensional joint point positioning and SMPL model establishment are simultaneously realized, and the human body model can be further generated. It is easy to notice that, because a two-dimensional joint point, a three-dimensional joint point and an SMPL model can be obtained simultaneously by using one model, the three-dimensional joint point is not required to be estimated through the two-dimensional joint point, thereby achieving the technical effect of improving the image identification accuracy, and further solving the technical problem of lower identification accuracy for positioning the two-dimensional and three-dimensional joint points and reconstructing human body parameters in the related technology.
Drawings
The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the invention and together with the description serve to explain the invention without limiting the invention. In the drawings:
FIG. 1 is a flow chart of an image processing method according to an embodiment of the invention;
FIG. 2 is a schematic illustration of an alternative human body image according to an embodiment of the invention;
FIG. 3 is a schematic illustration of an alternative mannequin according to an embodiment of the present invention;
FIG. 4a is a schematic illustration of an alternative average shaped mannequin in accordance with an embodiment of the present invention;
FIG. 4b is a schematic diagram of an alternative human model generated with shape parameters added thereto according to an embodiment of the present invention;
FIG. 4c is a schematic diagram of an alternative human model generated after adding shape parameters and pose parameters according to an embodiment of the invention;
FIG. 4d is a schematic diagram of an alternative human model generated from detected human motion according to an embodiment of the invention;
FIG. 5 is a flow diagram of an alternative image processing method according to an embodiment of the invention;
fig. 6 is a schematic diagram of an alternative GAN network in accordance with an embodiment of the present invention; and
fig. 7 is a schematic diagram of an image processing apparatus according to an embodiment of the present invention.
Detailed Description
In order to make the technical solutions of the present invention better understood, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
It should be noted that the terms "first," "second," and the like in the description and claims of the present invention and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the invention described herein are capable of operation in sequences other than those illustrated or described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
Example 1
According to an embodiment of the present invention, there is provided an image processing method, it should be noted that the steps shown in the flowchart of the drawings may be executed in a computer system such as a set of computer executable instructions, and that although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in an order different from that here.
Fig. 1 is a flowchart of an image processing method according to an embodiment of the present invention, as shown in fig. 1, the method including the steps of:
step S102, obtaining an original image;
specifically, the original image may be an image obtained by capturing input video stream data, or may be an image directly obtained, where the original image includes a human body.
Step S104, carrying out human body detection on the original image to obtain a human body image;
specifically, the human body image may be a minimum image extracted from the original image and including a complete human body region, as shown in fig. 2.
In an alternative embodiment, the human body detection can be performed by using a deep learning model, such as fast Region Convolutional Neural Networks (fast Region Convolutional Neural Networks), yolo (young Only Look one), ssd (single Shot detector), and other detection frames and their variants. As known to those skilled in the art, under different devices and application scenarios, different detection frameworks can be selected to quickly and accurately implement human body detection to obtain a human body image.
Optionally, the human body detection is performed on the original image, and obtaining the human body image includes: processing the original image by using the trained deep learning model to obtain the position information of the human body in the original image; and cutting and normalizing the original image based on the position information to obtain a human body image. The human body position in the human body image can be represented by a minimum bounding rectangle containing a complete human body region in the original image, and is expressed in a form of two-dimensional coordinates (left, top, bottom, right).
Step S106, processing the human body image by using the trained first model to obtain a processing result of the human body image, wherein the processing result comprises: two-dimensional joint points, three-dimensional joint points, and SMPL (Skinned Multi-Person Linear) models.
Alternatively, the first model may adopt an hourglass network structure or an FPN (Feature map Pyramid) network structure. For example, when the input is a w × h image, the output feature map may be a w × h or w/4 × h/4 image.
Specifically, the above-described joint point may be a position coordinate of each joint on the human body, such as a wrist, an elbow, or the like, as shown in fig. 2.
The two-dimensional joint points can be expressed in the form of a thermodynamic diagram (Heat Map) or in the form of a coordinate vector. In the form of a thermal map, each joint point can be represented as a feature map, and assuming that the input human body image is a w × h image, the output feature map is an image with the same size or scaled in equal proportion, the value of the feature map at the position of the joint point is 1, and the values of the feature maps at other positions are 0. In one example, when there are 16 two-dimensional joint points of the human body, 16 maps of w h or w/2 h/2 or less may be used to represent the two-dimensional joint points of the human body.
The three-dimensional joint point can also have two expression modes of thermodynamic diagram and coordinate vector, wherein for the thermodynamic diagram mode, compared with the two-dimensional joint point, the three-dimensional joint point increases the z-axis information on a three-dimensional space and diffuses the thermodynamic diagram into a cuboid.
In an alternative embodiment, the human body image may be processed by using the first model to obtain a parameter value of the SMPL model; and then obtaining two-dimensional joint points or three-dimensional joint points based on the parameter values.
And step S108, generating a human body model according to the processing result of the human body image.
As shown in fig. 3, the SMPL model may include a shape (shape) parameter and a pose (position) parameter, and the human body model generated from the shape parameter and the pose parameter may include a plurality of vertices and three-dimensional joint points, each of which is a three-dimensional vector including (x, y, z) coordinates. Fig. 4a to 4c show a process of generating a human body model based on a shape parameter and an orientation parameter, wherein fig. 4a shows a human body model of an average shape, fig. 4b shows a human body model generated by adding a shape parameter to the average shape, and fig. 4c shows a human body model generated by adding a shape parameter and an orientation parameter to the average shape. Fig. 4d shows a human model generated from the detected motion of the human body, in addition to the human model generated in fig. 4 c. As can be seen by comparing fig. 4b and 4c, the difference between the two is not very large, and therefore, in some applications, human modeling can be achieved by generating a human model from only the shape parameters.
According to the embodiment of the invention, after the original image is obtained, firstly, the original image is subjected to human body detection to obtain the human body image, and then the trained first model is used for processing the human body image to obtain the processing result of the human body image, so that the purposes of human body detection, two-dimensional and three-dimensional joint point positioning and SMPL model establishment are simultaneously realized, and the human body model can be further generated. It is easy to notice that, because a two-dimensional joint point, a three-dimensional joint point and an SMPL model can be obtained simultaneously by using one model, the three-dimensional joint point is not required to be estimated through the two-dimensional joint point, thereby achieving the technical effect of improving the image identification accuracy, and further solving the technical problem of lower identification accuracy for positioning two-dimensional and three-dimensional joint points and reconstructing human parameters in the related technology.
In a first application scenario, a human body motion can be detected in real time to drive a human body animation (AVATAR) model, for example, the human body motion is captured based on two-dimensional joint points and three-dimensional joint points, so that the human body animation model performs corresponding motion along with the human body motion, and interactive interaction is realized.
In the second application scenario, the purpose of editing the human body, such as slimming, can be achieved according to the two-dimensional joint points and the three-dimensional joint points in the processing result, for example, processing image pixels at corresponding positions of an arm, a leg, and a body on the human body image, and achieving image processing effects of slimming an arm, a leg, and a waist.
Optionally, in the above embodiment of the present invention, the image processing method further includes: obtaining a plurality of groups of training samples, wherein each group of training samples comprises: the method comprises the following steps of (1) obtaining a human body image, first marking information of a two-dimensional joint point, second marking information of a three-dimensional joint point and a parameter value of an SMPL model; training a preset model by utilizing a plurality of groups of training samples, and acquiring a target loss value of the preset model; stopping training the preset model under the condition that the target loss value is smaller than the preset value, and determining the preset model as a first model; and under the condition that the target loss value is larger than the preset value, continuously training the preset model by using the multiple groups of training samples until the target loss value is smaller than the preset value. The smaller the target loss value is, the higher the recognition accuracy is, the preset value can be set in advance according to the requirements of image recognition accuracy and efficiency, and whether the model is trained can be determined through the preset value.
Optionally, training the preset model by using multiple sets of training samples, and obtaining the target loss value of the preset model includes: inputting a plurality of groups of training samples into a preset model, and obtaining an output result of the preset model, wherein the output result comprises: a first result of a two-dimensional joint point, a second result of a three-dimensional joint point, and a third result of an SMPL model; obtaining a first loss value of the two-dimensional joint point based on the first mark information and the first result; obtaining a second loss value of the three-dimensional joint point based on the second marking information and the second result; obtaining a third loss value of the SMPL model based on the parameter value and the third result; and obtaining a target loss value based on the first loss value, the second loss value and the third loss value.
Optionally, in the foregoing embodiment of the present invention, the image processing method further includes labeling the training sample with first label information of a two-dimensional joint point and second label information of a three-dimensional joint point.
In an alternative embodiment, for a two-dimensional joint point, the first loss value may be obtained based on a predicted thermodynamic diagram (i.e., the first result) and a thermodynamic diagram of the label (i.e., the first label information), or based on a predicted coordinate vector (i.e., the first result) and a coordinate vector of the label (i.e., the first label information), or based on a combined information of the thermodynamic diagram and the coordinate vector.
For a three-dimensional joint point, likewise, the second loss value may be obtained based on the predicted thermodynamic diagram (i.e., the second result) and the thermodynamic diagram of the label tag (i.e., the second label information), or based on the predicted coordinate vector (i.e., the second result) and the coordinate vector of the label tag (i.e., the second label information), or based on the integrated information of the thermodynamic diagram and the coordinate vector.
Compared with a thermodynamic diagram mode, the coordinate vector mode is more convenient to calculate.
Alternatively, the parameter value of the SMPL model may be a real value acquired by the acquisition device, or an adjusted value obtained by adjusting the parameter value acquired by the acquisition device. In an alternative embodiment, the parameter values of the SMPL model may be predicted by true values that are weighted more heavily and adjusted values that are weighted less heavily.
Specifically, the above-mentioned acquisition device may be a camera or a sensor disposed at a plurality of fixed positions in a laboratory environment or an outdoor environment.
Because only the data collected in the laboratory environment can obtain accurate and real parameter values of the SMPL model, the data collected in the outdoor environment has no way to obtain accurate parameter values of the SMPL model. Therefore, in actual calculations, the third loss value may be calculated in different ways for the SMPL model based on the type of parameter value. Optionally, when the parameter value is the real value acquired by the acquisition device, a direct regression mode may be adopted to calculate a third loss value, that is, the third loss value is obtained based on the parameter value and the third result; when the parameter value is an adjustment value obtained by adjusting the parameter value acquired by the acquisition device, a three-dimensional joint point can be obtained according to the parameter value of the SMPL model, the three-dimensional joint point is projected onto a two-dimensional plane to obtain a two-dimensional joint point, a fourth loss value of the two-dimensional joint point is calculated based on the projected two-dimensional joint point and the first mark information, and the loss value is used as a third loss value and is transmitted back to a parameter space of the SMPL model to update the parameter value of the SMPL model.
In the training process, the target loss value is the synthesis of the first loss value, the second loss value and the third loss value, and can be obtained by calculation in a manner of solving a weighted sum.
In an optional embodiment, in the model training process, parameters of the two-dimensional joint points, the three-dimensional joint points and the SMPL model may be learned at the same time, and the model may be generated by performing regression as a whole, and in addition, as shown in fig. 5, an SMPL model discriminator may be used to discriminate the parameter value of the SMPL model, and determine whether the parameter value is a value randomly generated through a network or an acquired real value, thereby improving the reality of the model effect. Specifically, the SMPL model discriminator processes the parameter value of the third result (i.e., the SMPL model output by the preset model) to obtain a classification result of the parameter value of the third result, where the classification result is used to characterize whether the parameter value of the third result is a real value acquired by the acquisition device; and determining whether to stop training the preset model or not based on the classification result and the target loss value. Among them, a D-discriminator in a countermeasure generation Network (GAN) can be adopted as the SMPL model discriminator.
In an alternative embodiment, since the data collected in the outdoor environment has no way to obtain the precise parameter values of the SMPL model, so that it may generate abnormal parameter values, in the embodiment of the present invention, a GAN network is added to train the SMPL model discriminator (i.e. D discriminator), as shown in fig. 6, the GAN network includes a G generator and a D discriminator, the D discriminator is a binary network, receives the randomly generated value from the G generator and the collected real value, and outputs a label indicating the authenticity of the data, for example, when the real value is received, the output is close to a positive label (usually, the positive label is set to 1), when the randomly generated value from the G generator is received, the output is close to a negative label (usually, the negative label is set to 0), and the difference between the randomly generated value and the real value is described by the D discriminator, and then, updating the weight of the numerical value randomly generated by the G generator according to the difference, so that the numerical value randomly generated by the G generator is closer to the real numerical value, and the capability of distinguishing the randomly generated numerical value from the real numerical value by the D discriminator is improved.
Example 2
According to an embodiment of the present invention, an image processing apparatus is provided, which can execute the image processing method described in embodiment 1, and preferred embodiments and application scenarios in this embodiment are the same as those in embodiment 1, and are not described herein again.
Fig. 7 is a schematic diagram of an image processing apparatus according to an embodiment of the present invention, as shown in fig. 7, the apparatus including:
an obtaining module 72, configured to obtain an original image;
the detection module 74 is used for performing human body detection on the original image to obtain a human body image;
a processing module 76, configured to process the human body image by using the trained first model to obtain a processing result of the human body image, where the processing result includes: parameter values of two-dimensional joint points, three-dimensional joint points and the SMPL model;
and a generating module 78, configured to generate a human body model according to the processing result of the human body image.
Optionally, in the above embodiment of the present invention, the apparatus further includes: the obtaining module is further configured to obtain a plurality of sets of training samples, where each set of training samples includes: the method comprises the following steps of (1) obtaining a human body image, first marking information of a two-dimensional joint point, second marking information of a three-dimensional joint point and a parameter value of an SMPL model; the training module is used for training the preset model by utilizing a plurality of groups of training samples and acquiring a target loss value of the preset model; the training stopping module is used for stopping training the preset model and determining the preset model as a first model under the condition that the target loss value is smaller than the preset value; the training module is further used for continuing to train the preset model by using the multiple groups of training samples under the condition that the target loss value is larger than the preset value until the target loss value is smaller than the preset value.
Optionally, the training module comprises: the obtaining unit is used for inputting a plurality of groups of training samples into a preset model and obtaining an output result of the preset model, wherein the output result comprises: a first result of a two-dimensional joint point, a second result of a three-dimensional joint point, and a third result of an SMPL model; the first processing unit is used for obtaining a first loss value of the two-dimensional joint point based on the first mark information and the first result; the second processing unit is used for obtaining a second loss value of the three-dimensional joint point based on the second marking information and the second result; a third processing unit, configured to obtain a third loss value of the SMPL model based on the parameter value and the third result; and the fourth processing unit is used for obtaining a target loss value based on the first loss value, the second loss value and the third loss value.
Optionally, the third processing unit is further configured to obtain a third loss value based on the parameter value and the third result when the parameter value is acquired by the acquisition device; and under the condition that the parameter value is obtained by adjusting the parameter value acquired by the acquisition device, acquiring a three-dimensional joint point based on the parameter value, projecting the three-dimensional joint point onto a two-dimensional plane to obtain a two-dimensional joint point, acquiring a fourth loss value of the two-dimensional joint point based on the projected two-dimensional joint point and the first mark information, and determining the fourth loss value as a third loss value.
Optionally, the apparatus further comprises: the processing module is further used for processing the parameter value of the third result by using the discriminator to obtain a classification result of the parameter value of the third result, wherein the classification result is used for representing whether the parameter value of the third result is a real value acquired by the acquisition device; the training stopping module is further used for determining whether to stop training the preset model or not based on the classification result and the target loss value.
Optionally, the training module is further configured to train the discriminator with the generation of the countermeasure network.
Optionally, the detection module comprises: the detection unit is used for processing the original image by using the trained second model to obtain the position information of the human body in the original image; and the fifth processing unit is used for cutting and normalizing the original image based on the position information to obtain a human body image.
Example 3
According to an embodiment of the present invention, there is provided a storage medium including a stored program, wherein an apparatus in which the storage medium is located is controlled to execute the image processing method in embodiment 1 described above when the program is executed.
Example 4
According to an embodiment of the present invention, there is provided a processor configured to execute a program, where the program executes the image processing method in embodiment 1.
The above-mentioned serial numbers of the embodiments of the present invention are merely for description and do not represent the merits of the embodiments.
In the above embodiments of the present invention, the descriptions of the respective embodiments have respective emphasis, and for parts that are not described in detail in a certain embodiment, reference may be made to related descriptions of other embodiments.
In the embodiments provided in the present application, it should be understood that the disclosed technology can be implemented in other ways. The above-described embodiments of the apparatus are merely illustrative, and for example, the division of the units may be a logical division, and in actual implementation, there may be another division, for example, multiple units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, units or modules, and may be in an electrical or other form.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.
The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a removable hard disk, a magnetic or optical disk, and other various media capable of storing program codes.
The foregoing is only a preferred embodiment of the present invention, and it should be noted that, for those skilled in the art, various modifications and decorations can be made without departing from the principle of the present invention, and these modifications and decorations should also be regarded as the protection scope of the present invention.
Claims (12)
1. An image processing method, comprising:
acquiring an original image;
carrying out human body detection on the original image to obtain a human body image;
processing the human body image by using the trained first model to obtain a processing result of the human body image, wherein the processing result comprises: two-dimensional joint points, three-dimensional joint points and a skin multi-person linear SMPL model;
and generating a human body model according to the processing result of the human body image.
2. The method of claim 1, further comprising:
obtaining a plurality of groups of training samples, wherein each group of training samples comprises: the method comprises the following steps of (1) obtaining a human body image, first marking information of a two-dimensional joint point, second marking information of a three-dimensional joint point and a parameter value of an SMPL model;
training a preset model by using the multiple groups of training samples, and acquiring a target loss value of the preset model;
stopping training the preset model and determining the preset model as the first model under the condition that the target loss value is smaller than a preset value;
and under the condition that the target loss value is larger than the preset value, continuing to train the preset model by using the multiple groups of training samples until the target loss value is smaller than the preset value.
3. The method of claim 2, wherein training a preset model with the plurality of sets of training samples and obtaining a target loss value of the preset model comprises:
inputting the plurality of groups of training samples into the preset model, and obtaining an output result of the preset model, wherein the output result comprises: a first result of the two-dimensional articulated point, a second result of the three-dimensional articulated point, and a third result of the SMPL model;
obtaining a first loss value of the two-dimensional joint point based on the first mark information and the first result;
obtaining a second loss value of the three-dimensional joint point based on the second marking information and the second result;
obtaining a third loss value of the SMPL model based on the parameter value and the third result;
and obtaining the target loss value based on the first loss value, the second loss value and the third loss value.
4. The method of claim 3, wherein the parameter values of the SMPL model are real values acquired by an acquisition device or adjusted values obtained by adjusting the parameter values acquired by the acquisition device.
5. The method of claim 4, wherein deriving a third loss value for the SMPL model based on the parameter value and the third result comprises:
obtaining the third loss value based on the parameter value and the third result under the condition that the parameter value is the real value acquired by the acquisition device;
and under the condition that the parameter value is an adjustment value obtained by adjusting the parameter value acquired by the acquisition device, acquiring a three-dimensional joint point based on the parameter value, projecting the three-dimensional joint point onto a two-dimensional plane to acquire a two-dimensional joint point, acquiring a fourth loss value of the two-dimensional joint point based on the projected two-dimensional joint point and the first mark information, and determining the fourth loss value as the third loss value.
6. The method of claim 3, further comprising:
processing the parameter value of the third result by using a discriminator to obtain a classification result of the parameter value of the third result, wherein the classification result is used for representing whether the parameter value of the third result is a real numerical value acquired by an acquisition device;
and determining whether to stop training the preset model or not based on the classification result and the target loss value.
7. The method of claim 6, wherein the arbiter is trained using a generative countermeasure network.
8. The method of claim 1, wherein the human body detection of the original image, and obtaining the human body image comprises:
processing the original image by using the trained second model to obtain the position information of the human body in the original image;
and cutting and normalizing the original image based on the position information to obtain the human body image.
9. The method of claim 1, wherein the first model employs an hourglass-type network structure or a feature map pyramid FPN network structure.
10. An image processing apparatus characterized by comprising:
the acquisition module is used for acquiring an original image;
the detection module is used for carrying out human body detection on the original image to obtain a human body image;
the processing module is configured to process the human body image by using the trained first model to obtain a processing result of the human body image, where the processing result includes: two-dimensional joint points, three-dimensional joint points and a skin multi-person linear SMPL model;
and the generating module is used for generating a human body model according to the processing result of the human body image.
11. A storage medium, characterized in that the storage medium includes a stored program, wherein an apparatus in which the storage medium is located is controlled to execute the image processing method according to any one of claims 1 to 9 when the program is executed.
12. A processor, characterized in that the processor is configured to run a program, wherein the program is configured to execute the image processing method according to any one of claims 1 to 9 when running.
Priority Applications (4)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010231605.7A CN113449570A (en) | 2020-03-27 | 2020-03-27 | Image processing method and device |
| PCT/CN2021/080280 WO2021190321A1 (en) | 2020-03-27 | 2021-03-11 | Image processing method and device |
| KR1020227037422A KR20220160066A (en) | 2020-03-27 | 2021-03-11 | Image processing method and apparatus |
| JP2022558577A JP7448679B2 (en) | 2020-03-27 | 2021-03-11 | Image processing method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010231605.7A CN113449570A (en) | 2020-03-27 | 2020-03-27 | Image processing method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113449570A true CN113449570A (en) | 2021-09-28 |
Family
ID=77808126
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010231605.7A Pending CN113449570A (en) | 2020-03-27 | 2020-03-27 | Image processing method and device |
Country Status (4)
| Country | Link |
|---|---|
| JP (1) | JP7448679B2 (en) |
| KR (1) | KR20220160066A (en) |
| CN (1) | CN113449570A (en) |
| WO (1) | WO2021190321A1 (en) |
Families Citing this family (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114299204B (en) * | 2021-12-22 | 2023-04-18 | 深圳市海清视讯科技有限公司 | Three-dimensional cartoon character model generation method and device |
| CN114157526B (en) * | 2021-12-23 | 2022-08-12 | 广州新华学院 | Digital image recognition-based home security remote monitoring method and device |
| CN115482557B (en) * | 2022-10-09 | 2023-11-17 | 中国电信股份有限公司 | Human body image generation method, system, equipment and storage medium |
| CN115775300A (en) * | 2022-12-23 | 2023-03-10 | 北京百度网讯科技有限公司 | Reconstruction method of human body model, training method and device of human body reconstruction model |
| CN117351432B (en) * | 2023-12-04 | 2024-02-23 | 环球数科集团有限公司 | Training system for multi-target recognition model of scenic spot tourist |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108053469A (en) * | 2017-12-26 | 2018-05-18 | 清华大学 | Complicated dynamic scene human body three-dimensional method for reconstructing and device under various visual angles camera |
| CN109285215A (en) * | 2018-08-28 | 2019-01-29 | 腾讯科技（深圳）有限公司 | A kind of human 3d model method for reconstructing, device and storage medium |
| CN109615582A (en) * | 2018-11-30 | 2019-04-12 | 北京工业大学 | A kind of face image super-resolution reconstruction method generating confrontation network based on attribute description |
| CN109859296A (en) * | 2019-02-01 | 2019-06-07 | 腾讯科技（深圳）有限公司 | Training method, server and the storage medium of SMPL parametric prediction model |
| CN110020633A (en) * | 2019-04-12 | 2019-07-16 | 腾讯科技（深圳）有限公司 | Training method, image-recognizing method and the device of gesture recognition model |
| CN110298916A (en) * | 2019-06-21 | 2019-10-01 | 湖南大学 | A kind of 3 D human body method for reconstructing based on synthesis depth data |
Family Cites Families (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20140204013A1 (en) * | 2013-01-18 | 2014-07-24 | Microsoft Corporation | Part and state detection for gesture recognition |
| JP6373026B2 (en) * | 2014-03-20 | 2018-08-15 | 株式会社東芝 | Image processing apparatus, image processing system, image processing method, and program |
| JP6912215B2 (en) * | 2017-02-09 | 2021-08-04 | 国立大学法人東海国立大学機構 | Detection method and detection program to detect the posture of an object |
| CN108345869B (en) * | 2018-03-09 | 2022-04-08 | 南京理工大学 | Driver posture recognition method based on depth image and virtual data |
| JP2020030613A (en) * | 2018-08-22 | 2020-02-27 | 富士通株式会社 | Information processing device, data calculating program and data calculating method |
| CN109702741B (en) * | 2018-12-26 | 2020-12-18 | 中国科学院电子学研究所 | Mechanical arm vision grasping system and method based on self-supervision learning neural network |
| CN110188598B (en) * | 2019-04-13 | 2022-07-05 | 大连理工大学 | Real-time hand posture estimation method based on MobileNet-v2 |
-
2020
- 2020-03-27 CN CN202010231605.7A patent/CN113449570A/en active Pending
-
2021
- 2021-03-11 KR KR1020227037422A patent/KR20220160066A/en unknown
- 2021-03-11 JP JP2022558577A patent/JP7448679B2/en active Active
- 2021-03-11 WO PCT/CN2021/080280 patent/WO2021190321A1/en active Application Filing
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108053469A (en) * | 2017-12-26 | 2018-05-18 | 清华大学 | Complicated dynamic scene human body three-dimensional method for reconstructing and device under various visual angles camera |
| CN109285215A (en) * | 2018-08-28 | 2019-01-29 | 腾讯科技（深圳）有限公司 | A kind of human 3d model method for reconstructing, device and storage medium |
| CN109615582A (en) * | 2018-11-30 | 2019-04-12 | 北京工业大学 | A kind of face image super-resolution reconstruction method generating confrontation network based on attribute description |
| CN109859296A (en) * | 2019-02-01 | 2019-06-07 | 腾讯科技（深圳）有限公司 | Training method, server and the storage medium of SMPL parametric prediction model |
| CN110020633A (en) * | 2019-04-12 | 2019-07-16 | 腾讯科技（深圳）有限公司 | Training method, image-recognizing method and the device of gesture recognition model |
| CN110298916A (en) * | 2019-06-21 | 2019-10-01 | 湖南大学 | A kind of 3 D human body method for reconstructing based on synthesis depth data |
Non-Patent Citations (2)
| Title |
|---|
| A. KANAZAWA ET AL.: "End-to-End Recovery of Human Shape and Pose", 《 2018 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION》 * |
| MATTHEW LOPER ET AL.: "SMLP: a skinned multi-person linear model", 《ACM TRANSACTIONS ON GRAPHICS》 * |
Also Published As
| Publication number | Publication date |
|---|---|
| KR20220160066A (en) | 2022-12-05 |
| JP2023519012A (en) | 2023-05-09 |
| WO2021190321A1 (en) | 2021-09-30 |
| JP7448679B2 (en) | 2024-03-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Patel et al. | AGORA: Avatars in geography optimized for regression analysis | |
| CN111126272B (en) | Posture acquisition method, and training method and device of key point coordinate positioning model | |
| CN113449570A (en) | Image processing method and device | |
| CN108717531B (en) | Human body posture estimation method based on Faster R-CNN | |
| US8582867B2 (en) | Learning-based pose estimation from depth maps | |
| Ballan et al. | Marker-less motion capture of skinned models in a four camera set-up using optical flow and silhouettes | |
| JP4951498B2 (en) | Face image recognition device, face image recognition method, face image recognition program, and recording medium recording the program | |
| CN109934847A (en) | The method and apparatus of weak texture three-dimension object Attitude estimation | |
| WO2019225547A1 (en) | Object tracking device, object tracking method, and object tracking program | |
| CN113706699B (en) | Data processing method and device, electronic equipment and computer readable storage medium | |
| US10776978B2 (en) | Method for the automated identification of real world objects | |
| KR20120048370A (en) | Object pose recognition apparatus and method using the same | |
| WO2001033508A1 (en) | Automatic generation of animation of synthetic characters | |
| JP2016099982A (en) | Behavior recognition device, behaviour learning device, method, and program | |
| CN110717391A (en) | Height measuring method, system, device and medium based on video image | |
| Shen et al. | Exemplar-based human action pose correction | |
| CN113343840B (en) | Object identification method and device based on three-dimensional point cloud | |
| Zheng et al. | 4D reconstruction of blooming flowers | |
| CN110211222A (en) | A kind of AR immersion tourism guide method, device, storage medium and terminal device | |
| JP2013501284A5 (en) | ||
| Wang et al. | Digital twin: Acquiring high-fidelity 3D avatar from a single image | |
| JP2010211732A (en) | Object recognition device and method | |
| Darujati et al. | Facial motion capture with 3D active appearance models | |
| JP7318814B2 (en) | DATA GENERATION METHOD, DATA GENERATION PROGRAM AND INFORMATION PROCESSING DEVICE | |
| Ward et al. | A model-based approach to recovering the structure of a plant from images |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |