CN116452601A - Virtual fitting method, virtual fitting device, electronic equipment and storage medium - Google Patents
Virtual fitting method, virtual fitting device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN116452601A CN116452601A CN202210015099.7A CN202210015099A CN116452601A CN 116452601 A CN116452601 A CN 116452601A CN 202210015099 A CN202210015099 A CN 202210015099A CN 116452601 A CN116452601 A CN 116452601A
- Authority
- CN
- China
- Prior art keywords
- image
- human body
- dressing
- clothing
- virtual fitting
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 66
- 238000012545 processing Methods 0.000 claims abstract description 32
- 238000003709 image segmentation Methods 0.000 claims abstract description 23
- 238000004422 calculation algorithm Methods 0.000 claims description 59
- 238000004590 computer program Methods 0.000 claims description 16
- 230000015572 biosynthetic process Effects 0.000 claims description 10
- 238000003786 synthesis reaction Methods 0.000 claims description 10
- 238000012549 training Methods 0.000 claims description 10
- 230000011218 segmentation Effects 0.000 claims description 8
- 238000012360 testing method Methods 0.000 claims description 5
- 230000007547 defect Effects 0.000 abstract description 2
- 230000009467 reduction Effects 0.000 abstract description 2
- 230000036544 posture Effects 0.000 description 35
- 230000006870 function Effects 0.000 description 12
- 238000004364 calculation method Methods 0.000 description 7
- 230000002194 synthesizing effect Effects 0.000 description 6
- 238000004891 communication Methods 0.000 description 5
- 238000013527 convolutional neural network Methods 0.000 description 5
- 238000013528 artificial neural network Methods 0.000 description 4
- 210000000746 body region Anatomy 0.000 description 4
- 230000008569 process Effects 0.000 description 4
- 238000004458 analytical method Methods 0.000 description 3
- 230000003416 augmentation Effects 0.000 description 2
- 238000006243 chemical reaction Methods 0.000 description 2
- 239000002131 composite material Substances 0.000 description 2
- 230000002354 daily effect Effects 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 230000018109 developmental process Effects 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 238000003708 edge detection Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000012067 mathematical method Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000009471 action Effects 0.000 description 1
- 230000004913 activation Effects 0.000 description 1
- 230000003044 adaptive effect Effects 0.000 description 1
- 230000004931 aggregating effect Effects 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 238000012937 correction Methods 0.000 description 1
- 125000004122 cyclic group Chemical group 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 230000003203 everyday effect Effects 0.000 description 1
- 230000008921 facial expression Effects 0.000 description 1
- 230000005057 finger movement Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/11—Region-based segmentation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/50—Image enhancement or restoration by the use of more than one image, e.g. averaging, subtraction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/187—Segmentation; Edge detection involving region growing; involving region merging; involving connected component labelling
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20212—Image combination
- G06T2207/20221—Image fusion; Image merging
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
- Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
- Y02P90/30—Computing systems specially adapted for manufacturing
Abstract
The invention provides a virtual fitting method, a device, electronic equipment and a storage medium, wherein the virtual fitting method comprises the following steps: acquiring a human body image and a target clothes image; image segmentation is carried out on the human body image, and at least one human body dressing area is determined; performing gesture estimation based on the human body image to obtain a plurality of human body key points, and processing the target clothing image based on the human body key points to obtain a clothing image to be tested; and generating a human body dressing image based on the human body dressing region and the to-be-tested clothing image. The virtual fitting method, the device, the electronic equipment and the storage medium provided by the invention are used for solving the defect of high virtual fitting cost in the prior art and realizing the reduction of the virtual fitting cost.
Description
Technical Field
The present invention relates to the field of image processing technologies, and in particular, to a virtual fitting method, a virtual fitting device, an electronic device, and a storage medium.
Background
With the progress of science and technology and the development of artificial intelligence, intelligent algorithms are increasingly applied to daily life, particularly for a wardrobe, as furniture articles which need to be used every day, the intelligent development is of great importance, and the key problem of intelligence is to provide convenience for daily life, such as recording and virtual fitting of clothes in the wardrobe.
The prior art adopts the 3D camera to gather human body posture data, and then builds human body model, but uses the 3D camera can improve the cost greatly, therefore is not fit for family expenses very much, generally uses in large-scale market, can acquire the customer figure in real time through the 3D camera. However, in a home scene, the 3D fitting mirror with larger volume is not stored in excessive space, and the 3D camera is high in cost for families and is not suitable for families.
Disclosure of Invention
The invention provides a virtual fitting method, a device, electronic equipment and a storage medium, which are used for solving the defect of high virtual fitting cost in the prior art and realizing the reduction of the virtual fitting cost.
The invention provides a virtual fitting method, which comprises the following steps:
acquiring a human body image and a target clothes image;
image segmentation is carried out on the human body image, and at least one human body dressing area is determined;
performing gesture estimation based on the human body image to obtain a plurality of human body key points, and processing the target clothing image based on the human body key points to obtain a clothing image to be tested; and generating a human body dressing image based on the human body dressing region and the to-be-tested clothing image.
According to the virtual fitting method provided by the invention, the target clothing image is processed based on the human body key points to obtain the clothing image to be tested, which comprises the following steps:
and based on the human body key points, performing twisting treatment on the target clothing image through an interpolation algorithm to obtain the clothing image to be tested, wherein the clothing image to be tested corresponds to the gesture of the clothing region of the human body.
According to the virtual fitting method provided by the invention, the generating a human body wearing image based on the human body wearing region and the to-be-tested clothing image comprises the following steps:
generating an initial synthetic image based on the human body key points corresponding to the human body dressing region and the target clothing image;
and generating the human body dressing image based on the initial synthetic image and the to-be-tested dressing image.
According to the virtual fitting method provided by the invention, the human body wearing image is generated based on the human body wearing region and the to-be-tested clothing image, and the virtual fitting method comprises the following steps:
inputting the human body dressing region and the to-be-tested clothing image into a test model to generate the human body dressing image;
the matching model is obtained through training the following steps:
Inputting the sample dressing region and the sample dressing object image into an initial model to obtain a sample dressing image; the sample dressing area is obtained according to a sample character image, and the sample dressing image is obtained according to a sample clothing image;
calculating a total loss value based on the sample try-on image and the sample fit image;
and carrying out iterative updating on the parameters of the initial model based on the total loss value until the training ending condition is met, so as to obtain the try-on model.
The virtual fitting method provided by the invention further comprises the following steps:
obtaining a first loss value according to the sample try-on clothing image and the sample clothing image;
obtaining a second loss value according to the sample character image and the sample dressing image;
and determining the total loss value according to the first loss value and the second loss value.
According to the virtual fitting method provided by the invention, the human body image and the target clothes image are acquired, and the virtual fitting method comprises the following steps:
acquiring the human body image from a terminal device; the method comprises the steps of,
and the target clothes image is acquired from the terminal equipment or the intelligent wardrobe.
The virtual fitting method provided by the invention further comprises the following steps:
And projecting the human body dressing image to a target household appliance for display.
The invention also provides a virtual fitting device, comprising:
the acquisition module is used for acquiring the human body image and the target clothes image from the terminal equipment;
the segmentation module is used for carrying out image segmentation on the human body image and determining at least one human body dressing area;
the preliminary processing module is used for estimating the gesture based on the human body image to obtain a plurality of human body key points, and processing the target clothing image based on the human body key points to obtain a clothing image to be tested;
and the synthesis module is used for generating a human body dressing image based on the human body dressing area and the to-be-tested clothing image.
The invention also provides an electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the steps of any one of the virtual fitting methods described above when executing the program.
The present invention also provides a non-transitory computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of a virtual fitting method as described in any of the above.
The invention also provides a computer program product comprising a computer program which, when executed by a processor, implements the steps of a virtual fitting method as described in any of the above.
According to the virtual fitting method, the virtual fitting device, the electronic equipment and the storage medium, which are provided by the invention, the human body image and the target clothes image are directly acquired, the three-dimensional camera is not required to be used for shooting the human body, the fitting cost is reduced, and the privacy of a user can be protected.
Drawings
In order to more clearly illustrate the invention or the technical solutions of the prior art, the following description will briefly explain the drawings used in the embodiments or the description of the prior art, and it is obvious that the drawings in the following description are some embodiments of the invention, and other drawings can be obtained according to the drawings without inventive effort for a person skilled in the art.
FIG. 1 is a schematic flow chart of a virtual fitting method according to the present invention;
FIG. 2 is a second flow chart of the virtual fitting method according to the present invention;
fig. 3 is a schematic structural view of the virtual fitting device provided by the invention;
fig. 4 is a schematic structural diagram of an electronic device provided by the present invention.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the present invention more apparent, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings, and it is apparent that the described embodiments are some embodiments of the present invention, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
The virtual fitting method, apparatus, electronic device and storage medium of the present invention are described below with reference to fig. 1 to 4.
As shown in fig. 1, the virtual fitting method provided by the present invention is applied to a terminal device, for example, a virtual fitting mirror, and may also be applied to a cloud server, where the virtual fitting method includes:
step 110, acquiring a human body image and a target clothing image.
It can be understood that the human body image and the target clothing image can be acquired from a terminal device, and the terminal device can be a mobile phone or a tablet personal computer, and the like, and also can be a virtual fitting mirror. For example, the mobile phone or the tablet computer stores a human body image and a target clothing image, and the virtual fitting mirror can execute subsequent image processing operations after acquiring the human body image and the target clothing image from the mobile phone or the tablet computer.
Or, the human body image and the target clothes image are stored in the storage space of the virtual fitting mirror, and the virtual fitting mirror can directly acquire the human body image and the target clothes image from the storage space of the virtual fitting mirror and then execute subsequent image processing operation.
Or after the cloud server acquires the human body image and the target clothing image from the terminal equipment such as the mobile phone or the tablet personal computer, the cloud server executes subsequent image processing operation.
The target clothing image may be an image of clothing that the user needs to try on. The human body image uploaded by the terminal device and the target clothes image can be shot in advance and stored in the terminal device.
The human body image and the target clothing image are two-dimensional images.
The human body image includes, in addition to graphic features of the user figure, dimensions of various parts of the user figure, for example: arm length, shoulder width, leg length, etc.; the target clothing image includes, in addition to graphic features of the clothing that the user needs to replace, dimensions of various parts of the clothing that need to be replaced, such as dimensions of sleeve length of the clothing, width of the clothing, length and width of pants, and the like.
And 120, performing image segmentation on the human body image to determine at least one human body dressing region. It can be understood that the image segmentation processing can be performed on the human body image to obtain a user figure segmented image, and each area of the user figure segmented image is identified to determine whether the area belongs to the area requiring dressing.
The wearing region of the human body, that is, the region where the human body wears clothes, may be, for example, an upper body region or a lower body region.
The body posture may include a position and a body posture of each key point of the body, for example, a walking posture, a standing posture, a sitting posture, or the like, which is not limited by the embodiment of the present invention.
And 130, estimating the posture based on the human body image to obtain a plurality of human body key points, and processing the target clothing image based on the human body key points to obtain a clothing image to be tested.
It will be appreciated that the relative positional relationship between the plurality of keypoints and the plurality of keypoints of the human body may be determined by processing the human body image, which may be indicative of the human body pose.
And 140, generating a human body dressing image based on the human body dressing region and the to-be-tested clothing image.
It will be appreciated that the human body dressing image is an actual image corresponding to the target garment after the user wears the target garment. In the embodiment, the human body wearing image can be obtained by combining the human body wearing image with the human body wearing region after the target clothes image is processed according to the human body key points corresponding to the human body posture, and the user does not need to wear the target clothes on the human body, and the target clothes are acquired through shooting, so that the user can try on the target clothes conveniently.
The human body dressing region and the target clothing image are synthesized into human body dressing images conforming to various human body postures, for example, the human body dressing region and the target clothing image are synthesized into a human body dressing image of a walking posture, or the human body dressing region and the target clothing image are synthesized into a human body dressing image of a standing posture, or the human body dressing region and the target clothing image are synthesized into a human body dressing image of a sitting posture.
In some embodiments, the image segmentation of the human body image to determine at least one human body dressing region includes:
inputting the human body image into a trained image segmentation algorithm model, and carrying out image segmentation on the human body image to obtain the human body dressing region.
Further, the human body image is input into a trained human body posture estimation model to obtain the human body posture.
It can be understood that the trained image segmentation algorithm model and the trained human body posture estimation model can be a neural network algorithm model, and are issued to each terminal device after the cloud server is trained.
The image segmentation algorithm model may be one of a partial packet network (PGN) model, a Graph algorism (Graph algorisms) model, a JPPNet network model, an SSL (Self-supervised Structure sensitive Learning) network model, and an SCHP (Self-Correction for Human Parsing) network model.
The part of the packet network model redefines instance-level human body analysis into two twin subtasks which can be learned together and perfected mutually through a unified network: 1. semantic part segmentation that designates each pixel as a human part (e.g., face, arm); 2. instance-aware edge detection, classifying semantic parts to different persona instances.
The SSL network model is a new self-supervision structure-sensitive learning framework, and is characterized in that approximate human joint information is directly generated from analysis labels, and the two labels are used as supervision signals of a structure-sensitive loss function, so that the SSL network model is called a self-supervision learning strategy.
The SCHP network model designs a cyclic learning scheduler, and the performance of the network model is improved by iteratively aggregating the currently learned model with the previous best model in an online manner to infer more reliable labels.
In some embodiments, the human body posture estimation model is obtained by training one of an openpost algorithm model, a deep algorithm model and an RMPE (region Multi-Person Pose Estimation) algorithm model by taking a preset human body image as a sample and taking a human body posture corresponding to the preset human body image as a sample label.
It can be understood that the openpore algorithm is a bottom-up algorithm, and the openpore algorithm model is based on a convolutional neural network and an open source library developed by taking Caffe (Convolutional Architecture for Fast Feature Embedding) as a framework. The openpost algorithm can realize posture estimation of human body actions, facial expressions, finger movements and the like. The method is suitable for single person and multiple persons, has excellent robustness, and is applied to real-time multi-person two-dimensional attitude estimation based on deep learning.
Wherein the data structure in the Caffe framework exists in the form of Blobs-laminates-Net. Blobs are data for storing ownership weights, activation values and forward and backward in a network through a 4-dimensional vector form (num, channel, height, width). Blobs provide a unified memory interface as a standard data format for Caffe. Layers represent the specific Layers in the neural network, such as convolutional Layers, etc., which are the essential elements of the Caffe framework and the basic units that perform the calculations. The layer receives Blobs input from the bottom layer and outputs the Blobs to the higher layer. Forward propagation and backward propagation will be achieved at each layer. Net is a directed acyclic graph consisting of multiple layers connected together. A network combines the initial data layer loading data into a whole to the final loss layer.
The openpost algorithm principle is as follows:
1. inputting an image, extracting features through a VGG (Visual Geometry Group) convolution network to obtain a group of feature images, dividing the feature images into two branches, and extracting confidence (Part Confidence Maps) and relevance (Part Affinity Fields) of the two branches by using a CNN (convolutional neural network) network respectively;
2. after obtaining the confidence and Association information, obtaining key points (Part Association) by using even matching (Bipartite Matching) in graph theory, connecting the key points of the same person, and finally merging the generated even matching into an overall skeleton of the same person due to the vectorization of the Association degree;
3. finally, based on human body key Point Affinity Fields (PAFs), multi-Person paring is solved, multi-Person paring is converted into graphics, hungarian algorithm (Hungarian Algorithm) is adopted, hungarian algorithm is the most common algorithm for partial graph matching, and the core of the algorithm is finding an augmentation path, which is an algorithm for solving the maximum matching of the binary graphs by using the augmentation path.
In some embodiments, the processing the target clothing image based on the human body key points to obtain a clothing image to be tested includes:
And based on the human body key points, performing twisting treatment on the target clothing image through an interpolation algorithm to obtain the clothing image to be tested, wherein the clothing image to be tested corresponds to the gesture of the clothing region of the human body.
It will be appreciated that when the target garment is worn on a human body, the target garment will be distorted, and therefore, when the target garment image is combined with the wearing region of the human body, the target garment image needs to be distorted first, so that the shape of the target garment image conforms to the corresponding human body posture.
The target clothing image can be distorted through an interpolation algorithm to enable the target clothing image to meet the walking gesture of the human body, the target clothing image can be distorted through the interpolation algorithm to enable the target clothing image to meet the standing gesture of the human body, and the target clothing image can be distorted through the interpolation algorithm to enable the target clothing image to meet the sitting gesture of the human body.
Further, the distorted target clothing image and the human body clothing region are combined into a human body clothing image, namely, an effect picture which is displayed after the human body wears the target clothing.
In some embodiments, the generating a human dressing image based on the human dressing region and the to-be-tested clothing image includes:
Generating an initial synthetic image based on the human body key points corresponding to the human body dressing region and the target clothing image;
and generating the human body dressing image based on the initial synthetic image and the to-be-tested dressing image.
It can be understood that the to-be-tested clothing image, namely, the target clothing image is obtained after the target clothing image is distorted according to the human body key points corresponding to the human body clothing region.
The human body key points corresponding to the human body dressing areas are synthesized with the target clothing images, and the obtained initial synthesized image is not distorted based on the human body key points, so that the initial synthesized image and the clothing images to be tested are synthesized again, and the obtained human body dressing images are the closest to reality.
In some embodiments, generating a human wearing image based on the human wearing region and the to-be-tested clothing image includes:
inputting the human body dressing region and the to-be-tested clothing image into a test model to generate the human body dressing image;
the matching model is obtained through training the following steps:
inputting the sample dressing region and the sample dressing object image into an initial model to obtain a sample dressing image; the sample dressing area is obtained according to a sample character image, and the sample dressing image is obtained according to a sample clothing image;
Calculating a total loss value based on the sample try-on image and the sample fit image;
and carrying out iterative updating on the parameters of the initial model based on the total loss value until the training ending condition is met, so as to obtain the try-on model.
It will be appreciated that the sample try-on garment image is a garment image after the twisting process. The initial model and the try-on model may be models based on the same neural network algorithm, the parameters of the two models being different.
In some embodiments, the virtual fitting method further comprises:
obtaining a first loss value according to the sample try-on clothing image and the sample clothing image;
obtaining a second loss value according to the sample character image and the sample dressing image;
and determining the total loss value according to the first loss value and the second loss value.
It will be appreciated that the loss value, i.e. the value based on solving the loss function. The penalty function is a function that maps random events or their values of related random variables to non-negative real numbers to represent the "risk" or "penalty" of the random event.
The loss function employed in this embodiment may be an absolute value loss function, a square loss function, a log-log loss function, an exponential loss function, a perceptual loss function, or a cross entropy loss function.
In some embodiments, acquiring a human body image and a target clothing image includes:
acquiring the human body image from a terminal device; the method comprises the steps of,
and the target clothes image is acquired from the terminal equipment or the intelligent wardrobe.
It can be understood that the terminal device can be a mobile phone or a computer, and replaces a camera in the prior art scheme. The intelligent wardrobe stores images corresponding to clothes in the wardrobe.
In some embodiments, the generating the human body dressing image based on the initial composite image and the to-be-tested dressing image includes:
and inputting the initial synthesized image and the to-be-tested clothing image into a trained virtual try-on network model for synthesis to obtain the human body clothing image.
It is understood that the virtual try-on network model may be one of a CP-VTON algorithm model, a CP-vton+ algorithm model, a VITON (virtual try-on network based on image) algorithm model, and an ACGPN (adaptive content generation reservation network) algorithm model. The CP-VTON+ algorithm model is a model with improved setting of a loss function in the CP-VTON algorithm model.
The CP-VTON algorithm model is issued to each virtual fitting mirror after the cloud server is trained, and the virtual fitting mirrors are not required to train the CP-VTON algorithm model locally, so that the production cost of the CP-VTON algorithm model can be reduced.
Further, the CP-VTON algorithm includes two main functional modules:
geometry matching module (Geometric Matching Module): the clothing is subjected to learning distortion by using the convolutional neural network, so that the distorted clothing is aligned with a human body.
Try-on Module (Try-on Module): and fusing the distorted clothes with the target person, and synthesizing a final try-on result.
The flow of the whole CP-VTON algorithm is divided into two stages according to the two functional modules:
stage 1: firstly, extracting the characteristics of a human body in a reference image, then respectively extracting the characteristics of the human body and the high-dimensional characteristics of the replaced clothes by utilizing two convolutional neural networks, merging the characteristics of the human body and the high-dimensional characteristics of the replaced clothes through a relevant network layer, inputting the characteristics of the human body and the high-dimensional characteristics of the replaced clothes into a regression network to obtain a set of conversion parameters, and performing thin-plate spline transformation on the initial clothes by utilizing the learned conversion parameters to obtain the distorted clothes.
Stage 2: and generating an initial composite image and a mask (mask) of clothes by using the U-Net structure, and then combining the distorted clothes images by using the mask to obtain the final result of the CP-VTON algorithm model.
In some embodiments, the human body image and the target clothing image are both two-dimensional images. And compared with the processing of the three-dimensional image, the processing of the two-dimensional image has higher efficiency and is more convenient.
In some embodiments, the difference algorithm is a thin plate spline interpolation algorithm (Thin Plate Spline, TPS).
It will be appreciated that the thin-plate spline interpolation algorithm is one of the interpolation methods and is a commonly used two-dimensional interpolation method.
The thin-plate spline interpolation algorithm is a mathematical method that uses variable spline to produce a smooth curve through a series of points. Interpolation splines are composed of polynomials, each of which is determined by two adjacent data points, such that any two adjacent polynomials and their derivatives (excluding the nine derivatives) are continuous at the connection points.
In some embodiments, the virtual fitting method further comprises:
and projecting the human body dressing image to a target household appliance for display.
It will be appreciated that the target home device may be a television or a refrigerator with a display screen. Further, the fitting mirror projects the wearing image of the human body to a television or a refrigerator with a display screen for display.
In some embodiments, the virtual fitting method further comprises:
after the human body dressing image is obtained, deleting the human body image and/or the human body dressing image based on the input deleting instruction.
It can be appreciated that in the virtual fitting method provided by the invention, the human body image and the target clothing image need to be acquired from the terminal equipment, and the human body image and the target clothing image are processed, so that the human body wearing image is finally obtained. The human body image and the human body dressing image relate to the privacy of the user, so that the human body image and/or the human body dressing image are/is used for protecting the privacy of the user from being revealed after a deleting instruction input by the user is received.
In other embodiments, as shown in fig. 2, after uploading a human body image, the mobile phone terminal device determines a human body dressing area based on the human body image, determines a human body gesture, obtains key points of the human body, and then combines the target clothing image uploaded by the mobile phone terminal device, twists the target clothing by using an interpolation algorithm, and synthesizes the human body dressing image with the human body dressing area.
In summary, the virtual fitting method provided by the invention comprises the following steps: acquiring a user figure image and a human body image and a target clothes image; image segmentation is carried out on the human body image, and at least one human body dressing area is determined; performing gesture estimation based on the human body image to obtain a plurality of human body key points, and processing the target clothing image based on the human body key points to obtain a clothing image to be tested; and generating a human body dressing image based on the human body dressing region and the to-be-tested clothing image.
In the virtual fitting method provided by the invention, the human body image and the target clothes image are directly acquired, the three-dimensional camera is not required to be used for shooting the human body, the fitting cost is reduced, and the privacy of a user can be protected.
The virtual fitting device provided by the invention is described below, and the virtual fitting device described below and the virtual fitting method described above can be referred to correspondingly to each other.
As shown in fig. 3, the virtual fitting device 300 provided by the present invention includes: an acquisition module 310, a segmentation module 320, a preliminary processing module 330, and a synthesis module 340.
The acquisition module 310 is used for acquiring a human body image and a target clothing image.
It may be appreciated that the acquiring module 310 may acquire the human body image and the target clothing image from a terminal device, where the terminal device may be a mobile phone or a tablet computer, or may be a virtual fitting mirror, for example, the mobile phone or the tablet computer stores the human body image and the target clothing image, and the virtual fitting mirror may perform subsequent image processing operations after acquiring the human body image and the target clothing image from the mobile phone or the tablet computer.
Or, the human body image and the target clothes image are stored in the storage space of the virtual fitting mirror, and the virtual fitting mirror can directly acquire the human body image and the target clothes image from the storage space of the virtual fitting mirror and then execute subsequent image processing operation.
Or after the cloud server acquires the human body image and the target clothing image from the terminal equipment such as the mobile phone or the tablet personal computer, the cloud server executes subsequent image processing operation.
The target clothing image may be an image of clothing that the user needs to try on. The human body image uploaded by the terminal device and the target clothes image can be shot in advance and stored in the terminal device.
The human body image and the target clothing image are two-dimensional images.
The human body image includes, in addition to graphic features of the user figure, dimensions of various parts of the user figure, for example: arm length, shoulder width, leg length, etc.; the target clothing image includes, in addition to graphic features of the clothing that the user needs to replace, dimensions of various parts of the clothing that need to be replaced, such as dimensions of sleeve length of the clothing, width of the clothing, length and width of pants, and the like.
The segmentation module 320 is configured to perform image segmentation on the human body image to determine at least one wearing region of the human body.
It can be understood that the image segmentation processing can be performed on the human body image to obtain a user figure segmented image, and each area of the user figure segmented image is identified to determine whether the area belongs to the area requiring dressing.
The wearing region of the human body, that is, the region where the human body wears clothes, may be, for example, an upper body region or a lower body region.
The human body posture can be the posture of walking of a person, the posture of standing of a person, and the sitting posture of a person.
The preliminary processing module 330 is configured to perform gesture estimation based on the human body image to obtain a plurality of human body key points, and process the target clothing image based on the human body key points to obtain a clothing image to be tested;
it can be understood that the human body image corresponds to different human body postures, and each human body posture can correspondingly extract a plurality of key points for representation.
The synthesizing module 340 is configured to generate a human wearing image based on the human wearing region and the to-be-tested clothing image.
It will be appreciated that the human body dressing image is an actual image corresponding to the target garment after the user wears the target garment. In the embodiment, the human body wearing image can be obtained by combining the human body wearing image with the human body wearing region after the target clothes image is processed according to the human body key points corresponding to the human body posture, and the user does not need to wear the target clothes on the human body, and the target clothes are acquired through shooting, so that the user can try on the target clothes conveniently.
In some embodiments, the segmentation module 320 is further configured to input the human body image into a trained image segmentation algorithm model, and perform image segmentation on the human body image to obtain the human body dressing region.
Further, the human body image is input into a trained human body posture estimation model to obtain the human body posture.
It can be understood that the trained image segmentation algorithm model and the trained human body posture estimation model can be a neural network algorithm model, and are issued to each terminal device after the cloud server is trained.
The image segmentation algorithm model may be one of a partial packet network model, a graph algorithm model, a JPPNet network model, an SSL network model, and an SCHP network model.
The part of the packet network model redefines instance-level human body analysis into two twin subtasks which can be learned together and perfected mutually through a unified network: 1. semantic part segmentation that designates each pixel as a human part (e.g., face, arm); 2. instance-aware edge detection, classifying semantic parts to different persona instances.
In some embodiments, the human body posture estimation model is obtained by training one of an openpost algorithm model, a deep algorithm model and an RMPE algorithm model by taking a preset human body image as a sample and taking a human body posture corresponding to the preset human body image as a sample label.
In some embodiments, the preliminary processing module 330 is further configured to perform a warping process on the target clothing image by an interpolation algorithm based on the human body key points, so as to obtain the clothing image to be tested, where the clothing image to be tested corresponds to the pose of the clothing region of the human body.
It will be appreciated that when the target garment is worn on a human body, the target garment will be distorted, and therefore, when the target garment image is combined with the wearing region of the human body, the target garment image needs to be distorted first, so that the shape of the target garment image conforms to the corresponding human body posture.
The target clothing image can be distorted through an interpolation algorithm to enable the target clothing image to meet the walking gesture of the human body, the target clothing image can be distorted through the interpolation algorithm to enable the target clothing image to meet the standing gesture of the human body, and the target clothing image can be distorted through the interpolation algorithm to enable the target clothing image to meet the sitting gesture of the human body.
Further, the distorted target clothing image and the human body clothing region are combined into a human body clothing image, namely, an effect picture which is displayed after the human body wears the target clothing.
In some embodiments, the synthesis module 340 includes: a first synthesis unit and a second synthesis unit.
The first synthesizing unit is used for generating an initial synthesized image based on the human body key points corresponding to the human body dressing area and the target clothing image.
The second synthesis unit is used for generating the human body wearing image based on the initial synthesis image and the to-be-tested wearing object image.
It can be understood that the to-be-tested clothing image, namely, the target clothing image is obtained after the target clothing image is distorted according to the human body key points corresponding to the human body clothing region.
The human body key points corresponding to the human body dressing areas are synthesized with the target clothing images, and the obtained initial synthesized image is not distorted based on the human body key points, so that the initial synthesized image and the clothing images to be tested are synthesized again, and the obtained human body dressing images are the closest to reality.
In some embodiments, the synthesizing module 340 is further configured to input the human clothing region and the clothing image to be tested into a test model, and generate the human clothing image.
The matching model is obtained through training the following steps:
Inputting the sample dressing region and the sample dressing object image into an initial model to obtain a sample dressing image; the sample dressing area is obtained according to a sample character image, and the sample dressing image is obtained according to a sample clothing image;
calculating a total loss value based on the sample try-on image and the sample fit image; and carrying out iterative updating on the parameters of the initial model based on the total loss value until the training ending condition is met, so as to obtain the try-on model.
In some embodiments, the virtual fitting device further comprises: the device comprises a first loss value calculation module, a second loss value calculation module and a total loss value calculation module.
The first loss value calculation module is used for obtaining a first loss value according to the sample try-on clothing image and the sample clothing image;
the second loss value calculation module is used for obtaining a second loss value according to the sample character image and the sample dressing image;
the total loss value calculation module is used for determining the total loss value according to the first loss value and the second loss value.
In some embodiments, the acquisition module 310 includes:
the first acquisition unit is used for acquiring the human body image from the terminal equipment; the method comprises the steps of,
The second acquisition unit is used for acquiring the target clothes image from the terminal equipment or the intelligent wardrobe.
It can be understood that the terminal device can be a mobile phone or a computer, and replaces a camera in the prior art scheme. The intelligent wardrobe stores images corresponding to clothes in the wardrobe.
In some embodiments, the second synthesizing unit is further configured to input the initial synthesized image and the to-be-tested clothing image to a trained virtual try-on network model for synthesis, so as to obtain the human body clothing image.
In some embodiments, the human body image and the target clothing image are both two-dimensional images. And compared with the processing of the three-dimensional image, the processing of the two-dimensional image has higher efficiency and is more convenient.
In some embodiments, the difference algorithm is a thin plate spline interpolation algorithm.
It will be appreciated that the thin-plate spline interpolation algorithm is one of the interpolation methods and is a commonly used two-dimensional interpolation method.
The thin-plate spline interpolation algorithm is a mathematical method that uses variable spline to produce a smooth curve through a series of points. Interpolation splines are composed of polynomials, each of which is determined by two adjacent data points, such that any two adjacent polynomials and their derivatives (excluding the nine derivatives) are continuous at the connection points.
In some embodiments, the virtual fitting device further comprises: and the screen throwing module.
The screen projection module is used for projecting the human body dressing image to a target household appliance for display.
It will be appreciated that the target home device may be a television or a refrigerator with a display screen. Further, the fitting mirror projects the wearing image of the human body to a television or a refrigerator with a display screen for display.
In some embodiments, the virtual fitting device further comprises: and deleting the module.
The deleting module is used for deleting the human body image and/or the human body dressing image based on the input deleting instruction after the human body dressing image is obtained.
It can be appreciated that in the virtual fitting method provided by the invention, the human body image and the target clothing image need to be acquired from the terminal equipment, and the human body image and the target clothing image are processed, so that the human body wearing image is finally obtained. The human body image and the human body dressing image relate to the privacy of the user, so that the human body image and/or the human body dressing image are/is used for protecting the privacy of the user from being revealed after a deleting instruction input by the user is received. In summary, the virtual fitting device provided by the present invention includes: the acquisition module 310 is used for acquiring a human body image and a target clothing image; the segmentation module 320 is configured to perform image segmentation on the human body image, and determine at least one human body dressing region; the preliminary processing module 330 is configured to perform gesture estimation based on the human body image to obtain a plurality of human body key points, and process the target clothing image based on the human body key points to obtain a clothing image to be tested; the synthesizing module 340 is configured to generate a human wearing image based on the human wearing region and the to-be-tested clothing image.
In the virtual fitting device provided by the invention, the human body image and the target clothes image are directly acquired, the three-dimensional camera is not required to be used for shooting the human body, fitting cost is reduced, and user privacy can be protected.
The electronic device, the computer program product and the storage medium provided by the invention are described below, and the electronic device, the computer program product and the storage medium described below and the virtual fitting method described above can be referred to correspondingly.
Fig. 4 illustrates a physical schematic diagram of an electronic device, as shown in fig. 4, which may include: processor 410, communication interface (Communications Interface) 420, memory 430 and communication bus 440, wherein processor 410, communication interface 420 and memory 430 communicate with each other via communication bus 440. The processor 410 may invoke logic instructions in the memory 430 to perform a virtual fitting method comprising:
step 110, acquiring a user figure image, a human body image and a target clothes image;
step 120, image segmentation is carried out on the human body image, and at least one human body dressing area is determined;
130, estimating the posture based on the human body image to obtain a plurality of human body key points, and processing the target clothing image based on the human body key points to obtain a clothing image to be tested;
And 140, generating a human body dressing image based on the human body dressing region and the to-be-tested clothing image.
Further, the logic instructions in the memory 430 described above may be implemented in the form of software functional units and may be stored in a computer-readable storage medium when sold or used as a stand-alone product. Based on this understanding, the technical solution of the present invention may be embodied essentially or in a part contributing to the prior art or in a part of the technical solution, in the form of a software product stored in a storage medium, comprising several instructions for causing a computer device (which may be a personal computer, a server, a network device, etc.) to perform all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a magnetic disk, or an optical disk, or other various media capable of storing program codes.
In another aspect, the present invention also provides a computer program product comprising a computer program, the computer program being storable on a non-transitory computer readable storage medium, the computer program, when executed by a processor, being capable of performing the virtual fitting method provided by the methods described above, the method comprising:
Step 110, acquiring a user figure image, a human body image and a target clothes image;
step 120, image segmentation is carried out on the human body image, and at least one human body dressing area is determined;
130, estimating the posture based on the human body image to obtain a plurality of human body key points, and processing the target clothing image based on the human body key points to obtain a clothing image to be tested;
and 140, generating a human body dressing image based on the human body dressing region and the to-be-tested clothing image.
In yet another aspect, the present invention also provides a non-transitory computer readable storage medium having stored thereon a computer program which, when executed by a processor, is implemented to perform the virtual fitting method provided by the above methods, the method comprising:
step 110, acquiring a user figure image, a human body image and a target clothes image;
step 120, image segmentation is carried out on the human body image, and at least one human body dressing area is determined;
130, estimating the posture based on the human body image to obtain a plurality of human body key points, and processing the target clothing image based on the human body key points to obtain a clothing image to be tested;
And 140, generating a human body dressing image based on the human body dressing region and the to-be-tested clothing image.
The apparatus embodiments described above are merely illustrative, wherein the elements illustrated as separate elements may or may not be physically separate, and the elements shown as elements may or may not be physical elements, may be located in one place, or may be distributed over a plurality of network elements. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art will understand and implement the present invention without undue burden.
From the above description of the embodiments, it will be apparent to those skilled in the art that the embodiments may be implemented by means of software plus necessary general hardware platforms, or of course may be implemented by means of hardware. Based on this understanding, the foregoing technical solution may be embodied essentially or in a part contributing to the prior art in the form of a software product, which may be stored in a computer readable storage medium, such as ROM/RAM, a magnetic disk, an optical disk, etc., including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the method described in the respective embodiments or some parts of the embodiments.
Finally, it should be noted that: the above embodiments are only for illustrating the technical solution of the present invention, and are not limiting; although the invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims (11)
1. A virtual fitting method, comprising:
acquiring a human body image and a target clothes image;
image segmentation is carried out on the human body image, and at least one human body dressing area is determined;
performing gesture estimation based on the human body image to obtain a plurality of human body key points, and processing the target clothing image based on the human body key points to obtain a clothing image to be tested; and generating a human body dressing image based on the human body dressing region and the to-be-tested clothing image.
2. The virtual fitting method according to claim 1, wherein the processing the target clothing image based on the human body key points to obtain a clothing image to be tested comprises:
And based on the human body key points, performing twisting treatment on the target clothing image through an interpolation algorithm to obtain the clothing image to be tested, wherein the clothing image to be tested corresponds to the gesture of the clothing region of the human body.
3. The virtual fitting method according to claim 2, wherein the generating a human wearing image based on the human wearing region and the to-be-tested clothing image includes:
generating an initial synthetic image based on the human body key points corresponding to the human body dressing region and the target clothing image;
and generating the human body dressing image based on the initial synthetic image and the to-be-tested dressing image.
4. A virtual fitting method according to any of claims 1-3, wherein generating a human wearing image based on the human wearing region and the subject wearing image comprises:
inputting the human body dressing region and the to-be-tested clothing image into a test model to generate the human body dressing image;
the test model is obtained through training through the following steps:
inputting the sample dressing region and the sample dressing object image into an initial model to obtain a sample dressing image; the sample dressing area is obtained according to a sample character image, and the sample dressing image is obtained according to a sample clothing image;
Calculating a total loss value based on the sample try-on image and the sample fit image;
and carrying out iterative updating on the parameters of the initial model based on the total loss value until the training ending condition is met, so as to obtain the try-on model.
5. The virtual fitting method according to claim 4, further comprising:
obtaining a first loss value according to the sample try-on clothing image and the sample clothing image;
obtaining a second loss value according to the sample character image and the sample dressing image;
and determining the total loss value according to the first loss value and the second loss value.
6. The virtual fitting method according to any of claims 1-4, wherein acquiring a human body image and a target clothing image comprises:
acquiring the human body image from a terminal device; the method comprises the steps of,
and acquiring the target clothes image from the terminal equipment or the intelligent wardrobe.
7. The virtual fitting method according to any of claims 1-4, further comprising:
and projecting the human body dressing image to a target household appliance for display.
8. A virtual fitting device, comprising:
The acquisition module is used for acquiring the human body image and the target clothes image from the terminal equipment;
the segmentation module is used for carrying out image segmentation on the human body image and determining at least one human body dressing area;
the preliminary processing module is used for estimating the gesture based on the human body image to obtain a plurality of human body key points, and processing the target clothing image based on the human body key points to obtain a clothing image to be tested;
and the synthesis module is used for generating a human body dressing image based on the human body dressing area and the to-be-tested clothing image.
9. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the steps of the virtual fitting method according to any of claims 1 to 7 when the program is executed.
10. A non-transitory computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the steps of the virtual fitting method according to any of claims 1 to 7.
11. A computer program product comprising a computer program which, when executed by a processor, implements the steps of the virtual fitting method according to any of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210015099.7A CN116452601A (en) | 2022-01-07 | 2022-01-07 | Virtual fitting method, virtual fitting device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210015099.7A CN116452601A (en) | 2022-01-07 | 2022-01-07 | Virtual fitting method, virtual fitting device, electronic equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116452601A true CN116452601A (en) | 2023-07-18 |
Family
ID=87126051
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210015099.7A Pending CN116452601A (en) | 2022-01-07 | 2022-01-07 | Virtual fitting method, virtual fitting device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116452601A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117523142A (en) * | 2023-11-13 | 2024-02-06 | 书行科技(北京)有限公司 | Virtual fitting method, virtual fitting device, electronic equipment and computer readable storage medium |
-
2022
- 2022-01-07 CN CN202210015099.7A patent/CN116452601A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117523142A (en) * | 2023-11-13 | 2024-02-06 | 书行科技(北京)有限公司 | Virtual fitting method, virtual fitting device, electronic equipment and computer readable storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109636831B (en) | Method for estimating three-dimensional human body posture and hand information | |
| US11055888B2 (en) | Appearance-flow-based image generation | |
| US9646340B2 (en) | Avatar-based virtual dressing room | |
| US9001118B2 (en) | Avatar construction using depth camera | |
| US20210142539A1 (en) | Accurately generating virtual try-on images utilizing a unified neural network framework | |
| CN110096156B (en) | Virtual reloading method based on 2D image | |
| CN110033505A (en) | A kind of human action capture based on deep learning and virtual animation producing method | |
| KR20220066366A (en) | Predictive individual 3D body model | |
| CN113496507A (en) | Human body three-dimensional model reconstruction method | |
| JP2015531098A5 (en) | ||
| Liu et al. | 3D real human reconstruction via multiple low-cost depth cameras | |
| WO2014022608A2 (en) | Avatar-based virtual dressing room | |
| US20220270324A1 (en) | Systems and methods for generating a model of a character from one or more images | |
| US11922593B2 (en) | Methods of estimating a bare body shape from a concealed scan of the body | |
| Korosteleva et al. | Neuraltailor: Reconstructing sewing pattern structures from 3d point clouds of garments | |
| CN116452601A (en) | Virtual fitting method, virtual fitting device, electronic equipment and storage medium | |
| Xu et al. | CP3: Unifying point cloud completion by pretrain-prompt-predict paradigm | |
| CN114270402A (en) | Method and apparatus for constructing three-dimensional scanning human body model | |
| CN116266408A (en) | Body type estimating method, body type estimating device, storage medium and electronic equipment | |
| CN115936796A (en) | Virtual makeup changing method, system, equipment and storage medium | |
| Wang et al. | Evaluation of hybrid deep learning and optimization method for 3D human pose and shape reconstruction in simulated depth images | |
| Li et al. | SwinGar: Spectrum-Inspired Neural Dynamic Deformation for Free-Swinging Garments | |
| Beacco et al. | Automatic 3D avatar generation from a single RBG frontal image | |
| CN117635897B (en) | Three-dimensional object posture complement method, device, equipment, storage medium and product | |
| CN110838042B (en) | Commodity display method and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |