CN111768335A - CNN-based user interactive image local clothing style migration method - Google Patents

CNN-based user interactive image local clothing style migration method Download PDF

Info

Publication number
CN111768335A
CN111768335A CN202010628294.8A CN202010628294A CN111768335A CN 111768335 A CN111768335 A CN 111768335A CN 202010628294 A CN202010628294 A CN 202010628294A CN 111768335 A CN111768335 A CN 111768335A
Authority
CN
China
Prior art keywords
style
image
content
clothing
loss
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202010628294.8A
Other languages
Chinese (zh)
Other versions
CN111768335B (en
Inventor
熊海涛
王涵颍
蔡圆媛
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Technology and Business University
Original Assignee
Beijing Technology and Business University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Technology and Business University filed Critical Beijing Technology and Business University
Priority to CN202010628294.8A priority Critical patent/CN111768335B/en
Publication of CN111768335A publication Critical patent/CN111768335A/en
Application granted granted Critical
Publication of CN111768335B publication Critical patent/CN111768335B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/04—Context-preserving transformations, e.g. by using an importance map
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T11/00—2D [Two Dimensional] image generation
    • G06T11/001—Texturing; Colouring; Generation of texture or colour
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/11—Region-based segmentation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/13—Edge detection
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/194—Segmentation; Edge detection involving foreground-background segmentation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10004—Still image; Photographic image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20076—Probabilistic image processing
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20172—Image enhancement details
    • G06T2207/20192—Edge enhancement; Edge preservation
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
    • Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
    • Y02P90/30—Computing systems specially adapted for manufacturing

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • General Health & Medical Sciences (AREA)
  • Computing Systems (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • Biomedical Technology (AREA)
  • Molecular Biology (AREA)
  • Biophysics (AREA)
  • General Engineering & Computer Science (AREA)
  • Artificial Intelligence (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Image Analysis (AREA)

Abstract

The invention discloses a CNN-based user interactive image local clothing style migration method, which comprises the following steps: (1) inputting the content graph and the style graph into a CNN (content network node) for mapping to obtain content characteristics and style characteristics; (2) interactively segmenting the content graph by using a GrabConut algorithm, framing and extracting the local clothing by using a rectangle, and generating a local clothing outline graph; (3) converting the contour map into a binary map, and performing distance transformation to generate a distance transformation matrix; (4) increasing the distance between the inside and the outside of the local clothing outline by utilizing power level operation to form outline characteristics; (5) calculating the content loss, style loss and contour loss of the random noise image according to the characteristics; (6) and synthesizing three types of losses and adding a regular term to smoothly denoise the boundary area. The invention adopts a user interactive method to introduce contour loss to the picture so as to realize the retention of the clothing shape and the limitation of the style migration area, and effectively realizes the style migration of the local clothing.

Description

CNN-based user interactive image local clothing style migration method
Technical Field
The invention relates to the technical field of image processing and recognition, in particular to a CNN-based user interactive image local clothing style migration method.
Background
Image style migration refers to extracting a style from one picture and applying the style to another picture. Originally, in the visual field, image rendering is generally regarded as an expansion problem of texture synthesis, and a new image is generated through texture modeling, but the quality of the image generated by the method is not high. Although there are studies trying to perform style migration using a generative countermeasure network (GAN), although a good migration effect is obtained, it should be noted that the GAN-based method is unstable, its generation is too free, and it needs to be well constrained to generate a reasonable result stably, and the generative countermeasure network is a data-driven method, which is premised on a large amount of data, and is difficult to be effective when the amount of data is not sufficient. In recent years, image style migration research mainly focuses on feature mapping of content and style through a convolutional neural network, losses of content features and style features are reduced continuously, new images are generated in an iterative mode, good results are obtained, but content details cannot be well reserved in the style migration process of the algorithms, and semantic and depth information contained in content images is lacked. If the method is directly applied to fashion style migration of clothes, the generated clothes picture has low resolution, the clothes shape can be deformed and the color of the original clothes is kept, the style is irregularly migrated to the background instead of the local clothes, and the clothes and the new style are difficult to fuse.
Disclosure of Invention
In order to apply style migration to the fashion field and realize style design of fashion clothing, the method utilizes the user interactive GrabCT algorithm to perform image segmentation, a user only needs to frame a local clothing decoration by a rectangle to extract a clothing contour map, and the contour loss is introduced to realize retention of clothing shapes and limitation of style migration areas in combination with a convolutional neural network, so that the defects that the clothing shapes are deformed and cannot be fused with new styles are overcome. The invention adopts a simple user interactive method, can obtain a new clothes style with the picture style only fused with the local clothes, and realizes the user interactive image local clothes style migration.
The technical scheme adopted by the invention for solving the technical problems is as follows: a CNN-based user interactive image local clothing style migration method comprises the following steps:
step 1: taking a clothing image as a content image, taking a picture as a style image, inputting the picture into a CNN network for feature mapping to obtain content features and style features;
step 2: performing interactive image segmentation by using a GrabCut algorithm, framing the local clothing to be subjected to style migration in the content image in the step 1 by using a rectangle, and marking the local clothing as unknown; marking the region outside the local clothing as a background, calculating the probability that an unknown pixel in a rectangular frame belongs to the background or the target according to a Gaussian mixture model, thereby segmenting the image into the background and the target, extracting the target in the rectangular frame, namely the local clothing, and generating a local clothing contour map;
and step 3: converting the local clothing outline image in the step 2 into a binary image, and performing distance transformation by using an Euler distance formula to obtain a distance transformation matrix;
and 4, step 4: through the distance transformation in the step 3, the pixel value in the local clothing outline is 0, and the pixel value outside the local clothing outline is increased by utilizing power-level operation to increase the distance difference between the inside and the outside to form the outline characteristic;
and 5: respectively obtaining content loss, style loss and contour loss by respectively carrying out difference on the features of the random noise map, the content features and the style features obtained in the step 1 and the contour features obtained in the step 4;
step 6: and giving different weights to the three types of losses for addition, adding a regular term punishment weight, and finally updating the network weight by utilizing gradient descent to minimize the loss to generate a result graph.
The specific process of the step 1 is as follows:
step 1.1: inputting a clothing image as a content image into a trained VGG-19 network model to define a random noise image
Figure BDA0002567423350000021
The number of convolution kernels of the l-th layer is NlThe characteristic diagram size of the I layer is MlThe characteristics of the random noise figure at the l-th layer are expressed as a matrix
Figure BDA0002567423350000022
Represents the activation value of the noise figure at j on the ith convolution kernel of the I < th > layer of the CNN; defining content graph of input
Figure BDA0002567423350000023
Similarly, the characteristic of the content graph at the level l represents Pl，
Figure BDA0002567423350000024
The activation value of j on the ith convolution kernel of the I th layer of the CNN representing the content graph;
step 1.2: inputting a picture as a style diagram into a trained VGG-19 network model, calculating the inner product of different features from the same layer according to style features, defining by synthesizing the features of a plurality of layers of convolutional layers, introducing a Gram matrix,
Figure BDA0002567423350000025
and expressing the inner product of the feature map i and the feature map j of the layer I of the random noise map, wherein the formula is as follows:
Figure BDA0002567423350000026
where k represents the kth element of the feature map.
The specific process of the step 2 is as follows:
performing interactive image segmentation by using a GrabCut algorithm, framing the local clothing decoration needing style migration in the content image in the step 1 by using a rectangle, and marking the local clothing decoration as unknown; marking the region outside the local clothing as a background, calculating the probability that an unknown pixel in a rectangular frame belongs to the background or the target according to a Gaussian mixture model, thereby segmenting the image into the background and the target, extracting the target in the rectangular frame, namely the local clothing, and generating a local clothing contour map;
representing the gray value of the original gray map as z ═ z (z)1,z2,…,zn)znThe gray value of the nth pixel is represented, and the value of the pixel is represented by opacity α (α)1,α2,…,αn) Is represented by α∈ [0,1]The transparency value of 0 represents the background in the image, and the transparency value of 1 represents the foreground in the image; the algorithm models the foreground and background of a color image with Gaussian Mixture Models (GMMs), each of which is considered as a K-dimensional covariance, K ═ m1,k2,…,kn)，kn∈ {1,2, …, K } indicates to which Gaussian component each pixel belongs, the formula of the Gibbs energy function of the GrabCut algorithm is as follows:
E(α,k,θ,z)＝U(α,k,θ,z)+V(α,z) (2)
where E is Gibbs energy, U is a data item of an energy function, which represents a negative logarithm of a probability that a certain pixel belongs to an object or a background, V is a smoothing item of the energy function, θ is a histogram of gray values, { h (z; α), α ═ 0,1}, which describes a distribution of gray values z of the foreground and the background, and the data item U is defined as follows:
U(α,k,θ,z)＝∑nD(αn,kn,θ,zn) (3)
wherein the area term D is defined by the following formula:
D(αn,kn,θ,zn)＝-logp(zn|αn,kn,θ)-logπ(αn,kn) (4)
wherein the function p (-) is a Gaussian probability distribution function, pi (-) is a mixed weight coefficient of the Gaussian model sample occupying the whole, and the further derivation is as follows:
Figure BDA0002567423350000031
where Σ (α)n,kn) Is covariance matrix, det is determinant symbol; the parameter vector θ for GMM is represented as follows:
θ＝{π(a,k),μ(a,k),Σ(a,k),α＝0,1,k＝1…K} (6)
in the formula, pi is weight, the mean value of a mu Gaussian model, and sigma is covariance; the smooth term function V is defined as follows:
V(a,z)＝γ∑(m,n)∈C[αn≠αm]exp-β‖zm－zn‖2(7)
wherein the parameter γ is a weight of the degree of smoothing; m, n represents the neighborhood pixels in the picture C pixels; II zm－znThe parameter β is determined by the contrast of the image, if the contrast of the image is low, a large β value is selected to amplify the pixel difference value, and if the contrast of the image is high, a small β value is selected to reduce the pixel difference value.
The specific process of the step 3 is as follows:
converting the local clothing outline image into a binary image, calculating the distance between the inside and the outside of the outline by utilizing an Euler distance formula so as to obtain a distance transformation matrix, and defining the matrix as D, wherein the distance transformation formula is as follows:
D(p)＝Min(disf(pq))p∈O,q∈B (8)
Figure BDA0002567423350000032
wherein the defined pixel point p belongs to the image O in the local clothing outline and has the coordinate of (x)1,y1) Defining pixel point q as belonging to the out-of-contour image B, and its coordinate is (x)2,y2)；
The specific process of the step 4 is as follows:
the pixel value in the local clothing contour after the distance transformation is 0, the pixel value outside the local clothing contour is increased by utilizing power level operation to increase the distance difference between the inside and the outside to form contour characteristics, and the formula is as follows:
Figure BDA0002567423350000041
wherein n is at least 2.
The specific process of the step 5 is as follows:
step 5.1: random noise image
Figure BDA0002567423350000042
Content characteristics of
Figure BDA0002567423350000043
And content map
Figure BDA0002567423350000044
Content characteristics of
Figure BDA0002567423350000045
The difference is taken to obtain the content loss, and the formula is as follows:
Figure BDA0002567423350000046
step 5.2: random noise image
Figure BDA0002567423350000047
Style characteristics of
Figure BDA0002567423350000048
And style sheet
Figure BDA0002567423350000049
Style characteristics of
Figure BDA00025674233500000410
Making difference to obtain style loss, defining style loss E of layerlThe formula is as follows:
Figure BDA00025674233500000411
the style loss for all layers of the CNN is defined by the following formula:
Figure BDA00025674233500000412
wherein, wlRepresenting the weight lost by each layer style of CNN.
Step 5.3: mapping random noise
Figure BDA00025674233500000413
Defining the profile features after distance transformation as a matrix
Figure BDA00025674233500000414
Input local dress outline drawing
Figure BDA00025674233500000415
Defining the profile features after distance transformation as a matrix
Figure BDA00025674233500000416
Loss of contour LdThe formula is defined as follows:
Figure BDA00025674233500000417
the specific process of the step 6 is as follows:
step 6.1 α, β, gamma, r are four weight coefficients, LTVThe method is characterized in that a total variation regular term is introduced, the function of the total variation regular term is to inhibit noise generated in the style migration process and smooth the boundary of the local clothing outline, and the formula is as follows:
Ltotal＝αLc+βLs+γLd+rLTV(15)
Figure BDA00025674233500000418
Dx、Dyrespectively representing the lateral and longitudinal differences of the resulting image,
Figure BDA00025674233500000419
respectively, the number of elements corresponding to the difference result.
Step 6.2:
the loss function is minimized using gradient descent, the derivative of the content loss function being:
Figure BDA0002567423350000051
the derivative of the style loss function is:
Figure BDA0002567423350000052
step 6.3: and updating the network weight to minimize loss, and generating a new clothes style with the picture style only fused with partial clothes.
Has the advantages that:
the method can realize self-control of the clothing style, and the user interactive method is adopted, so that the clothing contour can be extracted by the user by using a rectangular frame to frame the clothing, the method is simple and convenient, the common user of a non-professional designer can select the clothing contour suitable for the user according to the trend, then a new design style is generated by utilizing the unique fashion sense and preference of the user, and finally a unique clothing style is formed, and the fashion requirement of the user is met. Meanwhile, the invention can provide a great deal of inspiration for professional designers, and can be quickly designed into draft according to the style given by the user, thereby improving the efficiency and the satisfaction degree of customers. The method has high efficiency and low cost, the generated design quality is high, and the pursuit of fashion is met.
Drawings
FIG. 1 is a flow chart of a method of the present invention;
FIG. 2 is a diagram of four contents inputted by the present invention, wherein (a), (b), (c), and (d) are the first, second, third, and fourth inputted contents respectively;
FIG. 3 is a partial clothing contour map extracted by Grabcut of the present invention, wherein (a), (b), (c), and (d) are the partial clothing contour maps extracted by the first, second, third, and fourth content maps, respectively;
FIG. 4 is two stylistic graphs input by the present invention, wherein (a) and (b) are the first and second tabbed graphs respectively;
fig. 5(e) and (f) are partial migration result graphs generated by the content fig. 2(a), the content fig. 2(b), and the genre chart 4(a), respectively, and fig. 5 (g) and (h) are partial migration result graphs generated by the content fig. 2(c), the content fig. 2(d), and the genre chart 4(b), respectively.
Detailed Description
The technical solutions in the embodiments of the present invention will be described clearly and completely with reference to the accompanying drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, rather than all embodiments, and all other embodiments obtained by a person skilled in the art based on the embodiments of the present invention belong to the protection scope of the present invention without creative efforts.
The invention discloses a CNN-based user interactive image local clothing style migration method, which comprises the following steps as shown in figure 1:
step 1: taking a clothing image as a content image, taking a picture as a style image, inputting the picture into a CNN network for feature mapping to obtain content features and style features;
step 2: performing interactive image segmentation by using a GrabCut algorithm, framing the local clothing to be subjected to style migration in the content image in the step 1 by using a rectangle, and marking the local clothing as unknown; marking the region outside the local clothing as a background, calculating the probability that an unknown pixel in a rectangular frame belongs to the background or the target according to a Gaussian mixture model, thereby segmenting the image into the background and the target, extracting the target in the rectangular frame, namely the local clothing, and generating a local clothing contour map;
and step 3: converting the local clothing outline image in the step 2 into a binary image, and performing distance transformation by using an Euler distance formula to obtain a distance transformation matrix;
and 4, step 4: through the distance transformation in the step 3, the pixel value in the local clothing outline is 0, and the pixel value outside the local clothing outline is increased by utilizing power-level operation to increase the distance difference between the inside and the outside to form the outline characteristic;
and 5: respectively obtaining content loss, style loss and contour loss by respectively carrying out difference on the features of the random noise map, the content features and the style features obtained in the step 1 and the contour features obtained in the step 4;
step 6: and giving different weights to the three types of losses for addition, adding a regular term punishment weight, and finally updating the network weight by utilizing gradient descent to minimize the loss to generate a result graph.
Specifically, the step 1 includes:
step 1.1: inputting the content maps, such as four content maps in the embodiment shown in FIGS. 2(a), (b), (c) and (d), into the trained VGG-19 network model to define a random noise image
Figure BDA0002567423350000061
The number of convolution kernels of the l-th layer is NlThe characteristic diagram size of the I layer is MlThe characteristics of the random noise figure at the l-th layer are expressed as a matrix
Figure BDA0002567423350000062
Represents the activation value of the noise figure at j on the ith convolution kernel of the I < th > layer of the CNN; defining content graph of input
Figure BDA0002567423350000063
Similarly, the characteristic of the content graph at the level l represents Pl，
Figure BDA0002567423350000064
The activation value of j on the ith convolution kernel of the I th layer of the CNN representing the content graph;
step 1.2: inputting the style diagrams, such as two style diagrams of the embodiment shown in fig. 4(a) and (b), into a trained VGG-19 network model, wherein the style characteristics need to calculate inner products between different characteristics from the same layer, and further integrating the characteristics of the multilayer convolutional layer for definition, introducing a Gram matrix,
Figure BDA0002567423350000065
showing random noise figure at lThe inner product of the layer feature map i and the feature map j is as follows:
Figure BDA0002567423350000066
where k represents the kth element of the feature map.
Step 2: performing interactive image segmentation by using a GrabCut algorithm, framing the local clothing decoration needing style migration in the content image in the step 1 by using a rectangle, and marking the local clothing decoration as unknown; marking the region outside the local clothing as a background, calculating the probability that an unknown pixel in a rectangular frame belongs to the background or the target according to a Gaussian mixture model, thereby segmenting the image into the background and the target, extracting the target in the rectangular frame, namely the local clothing, and generating a local clothing contour map; as shown in fig. 3(a) (b) (c) (d), four partial garment silhouettes of this embodiment;
representing the gray value of the original gray map as z ═ z (z)1,z2,…,zn)znThe gray value of the nth pixel is represented, and the value of the pixel is represented by opacity α (α)1,α2,…,αn) Is represented by α∈ [0,1]The transparency value of 0 represents the background in the image, and the transparency value of 1 represents the foreground in the image; the algorithm models the foreground and background of a color image with Gaussian Mixture Models (GMMs), each of which is considered as a K-dimensional covariance, K ═ m1,k2,…,kn)，kn∈ {1,2, …, K } indicates to which Gaussian component each pixel belongs, the formula of the Gibbs energy function of the GrabCut algorithm is as follows:
E(α,k,θ,z)＝U(α,k,θ,z)+V(α,z) (2)
where E is Gibbs energy, U is a data item of an energy function, which represents a negative logarithm of a probability that a certain pixel belongs to an object or a background, V is a smoothing item of the energy function, θ is a histogram of gray values, { h (z; α), α ═ 0,1}, which describes a distribution of gray values z of the foreground and the background, and the data item U is defined as follows:
U(α,k,θ,z)＝∑nD(αn,kn,θ,zn) (3)
wherein the area term D is defined by the following formula:
D(αn,kn,θ,zn)＝-logp(zn|αn,kn,θ)-logπ(αn,kn) (4)
wherein the function p (-) is a Gaussian probability distribution function, pi (-) is a mixed weight coefficient of the Gaussian model sample occupying the whole, and the further derivation is as follows:
Figure BDA0002567423350000071
where Σ (α)n,kn) Is covariance matrix, det is determinant symbol; the parameter vector θ for GMM is represented as follows:
θ＝{π(a,k),μ(a,k),Σ(a,k),α＝0,1,k＝1…K} (6)
in the formula, pi is weight, the mean value of a mu Gaussian model, and sigma is covariance; the smooth term function V is defined as follows:
V(a,z)＝γ∑(m,n)∈C[αn≠αm]exp-β‖zm－zn‖2 (7)
wherein the parameter γ is a weight of the degree of smoothing; m, n represents the neighborhood pixels in the picture C pixels; II zm－znThe parameter β is determined by the contrast of the image, if the contrast of the image is low, a large β value is selected to amplify the pixel difference value, and if the contrast of the image is high, a small β value is selected to reduce the pixel difference value.
Step 3, converting the local clothing outline image into a binary image, calculating the distance between the inside and the outside of the outline by using an Euler distance formula so as to obtain a distance transformation matrix, and defining the matrix as D, wherein the distance transformation formula is as follows:
D(p)＝Min(disf(pq))p∈O,q∈B (8)
Figure BDA0002567423350000081
wherein the defined pixel point p belongs to the image O in the local clothing outline and has the coordinate of (x)1,y1) Defining pixel point q as belonging to the out-of-contour image B, and its coordinate is (x)2,y2)
Step 4, the pixel value in the local clothing contour after the distance transformation is 0, the pixel value outside the local clothing contour is increased by utilizing power level operation to increase the distance difference between the inside and the outside to form the contour characteristic, and the formula is as follows:
Figure BDA0002567423350000082
wherein n is at least 2.
Step 5, specifically comprising:
step 5.1: random noise image
Figure BDA0002567423350000083
Content characteristics of
Figure BDA0002567423350000084
And content map
Figure BDA0002567423350000085
Content characteristics of
Figure BDA0002567423350000086
The difference is taken to obtain the content loss, and the formula is as follows:
Figure BDA0002567423350000087
step 5.2: random noise image
Figure BDA0002567423350000088
Style characteristics of
Figure BDA0002567423350000089
And styleDrawing (A)
Figure BDA00025674233500000810
Style characteristics of
Figure BDA00025674233500000811
Making difference to obtain style loss, defining style loss E of layerlThe formula is as follows:
Figure BDA00025674233500000812
the style loss for all layers of the CNN is defined by the following formula:
Figure BDA00025674233500000813
wherein, wlRepresenting the weight lost by each layer style of CNN.
Step 5.3: mapping random noise
Figure BDA00025674233500000814
Defining the profile features after distance transformation as a matrix
Figure BDA00025674233500000815
Input local dress outline drawing
Figure BDA00025674233500000816
Defining the profile features after distance transformation as a matrix
Figure BDA00025674233500000817
Loss of contour LdThe formula is defined as follows:
Figure BDA00025674233500000818
the step 6 specifically comprises the following steps:
step 6.1 α, β, gamma, r are four weight coefficients, LTVIs an introduced total variation regularization term which has the function of inhibiting the production in the style migration processThe noise generated smoothes the boundary of the local clothing outline, and the formula is as follows:
Ltotal＝αLc+βLs+γLd+rLTV(15)
Figure BDA00025674233500000819
Dx、Dyrespectively representing the lateral and longitudinal differences of the resulting image,
Figure BDA00025674233500000820
respectively, the number of elements corresponding to the difference result.
Step 6.2:
the loss function is minimized using gradient descent, the derivative of the content loss function being:
Figure BDA0002567423350000091
the derivative of the style loss function is:
Figure BDA0002567423350000092
step 6.3: and updating the network weight to minimize loss, and generating a new clothes style with the picture style only fused with partial clothes. As shown in fig. 5, (e) (f) are local migration result maps generated by the content map (a) and the content map (b) and the style map (a), respectively, and (g) (h) are local migration result maps generated by the content map (c) and the content map (d) and the style map (b), respectively.
Although illustrative embodiments of the present invention have been described above to facilitate the understanding of the present invention by those skilled in the art, it should be understood that the present invention is not limited to the scope of the embodiments, but various changes may be apparent to those skilled in the art, and it is intended that all inventive concepts utilizing the inventive concepts set forth herein be protected without departing from the spirit and scope of the present invention as defined and limited by the appended claims.

Claims (7)

1. A CNN-based user interactive image local clothing style migration method is characterized by comprising the following steps:
step 1: taking a clothing image as a content image, taking a picture as a style image, inputting the picture into a CNN network for feature mapping to obtain content features and style features;
step 2: performing interactive image segmentation by using a GrabCut algorithm, framing the local clothing to be subjected to style migration in the content image in the step 1 by using a rectangle, and marking the local clothing as unknown; marking the region outside the local clothing as a background, calculating the probability that an unknown pixel in a rectangular frame belongs to the background or the target according to a Gaussian mixture model, thereby segmenting the image into the background and the target, extracting the target in the rectangular frame, namely the local clothing, and generating a local clothing contour map;
and step 3: converting the local clothing outline image in the step 2 into a binary image, and performing distance transformation by using an Euler distance formula to obtain a distance transformation matrix;
and 4, step 4: through the distance transformation in the step 3, the pixel value in the local clothing outline is 0, and the pixel value outside the local clothing outline is increased by utilizing power-level operation to increase the distance difference between the inside and the outside to form the outline characteristic;
and 5: respectively obtaining content loss, style loss and contour loss by respectively carrying out difference on the features of the random noise map, the content features and the style features obtained in the step 1 and the contour features obtained in the step 4;
step 6: and giving different weights to the three types of losses for addition, adding a regular term punishment weight, and finally updating the network weight by utilizing gradient descent to minimize the loss to generate a result graph.
2. The CNN-based user-interactive image partial apparel style migration method of claim 1, wherein: the step 1 specifically comprises the following steps:
step 1.1: inputting a clothing drawing as a content drawing into the trained VGG-19 network model to defineRandom noise image
Figure FDA0002567423340000011
Letter l indicates the number of layers, the number of convolution kernels of the l-th layer being NlThe characteristic diagram size of the I layer is MlThe characteristics of the random noise figure at the l-th layer are expressed as a matrix
Figure FDA0002567423340000012
Figure FDA0002567423340000013
Represents the activation value of the noise figure at j on the ith convolution kernel of the I < th > layer of the CNN; defining content graph of input
Figure FDA0002567423340000014
Similarly, the characteristic of the content graph at the level l represents Pl，
Figure FDA0002567423340000015
The activation value of j on the ith convolution kernel of the I th layer of the CNN representing the content graph;
step 1.2: inputting a picture as a style diagram into a trained VGG-19 network model, calculating the inner product of different features from the same layer according to style features, defining by synthesizing the features of a plurality of layers of convolutional layers, introducing a Gram matrix,
Figure FDA0002567423340000016
Figure FDA0002567423340000017
and expressing the inner product of the feature map i and the feature map j of the layer I of the random noise map, wherein the formula is as follows:
Figure FDA0002567423340000018
where k represents the kth element of the feature map.
3. The CNN-based user-interactive image partial apparel style migration method of claim 1, wherein:
step 2, performing interactive image segmentation by using a GrabConut algorithm, and framing the local clothing to be subjected to style migration in the content image in the step 1 by using a rectangle, wherein the clothing is marked as unknown; marking the region outside the local clothing as a background, calculating the probability that an unknown pixel in a rectangular frame belongs to the background or the target according to a Gaussian mixture model, thereby segmenting the image into the background and the target, extracting the target in the rectangular frame, namely the local clothing, and generating a local clothing contour map;
representing the gray value of the original gray map as z ═ z (z)1,z2,…,zn),znThe gray value of the nth pixel is represented, and the value of the pixel is represented by opacity α (α)1,α2,…,αn) Is represented by α∈ [0,1]The transparency value of 0 represents the background in the image, and the transparency value of 1 represents the foreground in the image; the algorithm adopts Gaussian Mixture Model (GMM) to Model the foreground and the background of the color image, and each Gaussian Mixture Model is regarded as a K-dimensional covariance, K ═ K [ ((K) ])1,k2,…,kn)，kn∈ {1,2, …, K } indicates to which Gaussian component each pixel belongs, the formula of the Gibbs energy function of the GrabCut algorithm is as follows:
E(α,k,θ,z)＝U(α,k,θ,z)+V(α,z) (2)
where E is Gibbs energy, U is a data item of an energy function, which represents a negative logarithm of a probability that a certain pixel belongs to an object or a background, V is a smoothing item of the energy function, θ is a histogram of gray values, { h (z; α), α ═ 0,1}, which describes a distribution of gray values z of the foreground and the background, and the data item U is defined as follows:
U(α,k,θ,z)＝∑nD(αn,kn,θ,zn) (3)
wherein the area term D is defined by the following formula:
D(αn,kn,θ,zn)＝-logp(zn|αn,kn,θ)-logπ(αn,kn) (4)
wherein the function p (-) is a Gaussian probability distribution function, pi (-) is a mixed weight coefficient of the Gaussian model sample occupying the whole, and the further derivation is as follows:
Figure FDA0002567423340000021
where Σ (α)n,kn) Is covariance matrix, det is determinant symbol; the parameter vector θ for GMM is represented as follows:
θ＝{π(a,k),μ(a,k),Σ(a,k),α＝0,1,k＝1…K} (6)
in the formula, pi is weight, the mean value of a mu Gaussian model, and sigma is covariance; the smooth term function V is defined as follows:
V(a,z)＝γ∑(m,n)∈C[αn≠αm]exp-β‖zm－zn‖2(7)
wherein the parameter γ is a weight of the degree of smoothing; m, n represents the neighborhood pixels in the picture C pixels; II zm－znThe parameter β is determined by the contrast of the image, if the contrast of the image is low, a large β value is selected to amplify the pixel difference value, and if the contrast of the image is high, a small β value is selected to reduce the pixel difference value.
4. The CNN-based user-interactive image partial apparel style migration method of claim 1, wherein:
in the step 3, the local clothing outline image is converted into a binary image, the distance between the inside and the outside of the outline is calculated by using an Euler distance formula, so as to obtain a distance transformation matrix, the matrix is defined as D, and the distance transformation formula is as follows:
D(p)＝Min(disf(pq))p∈O,q∈B (8)
Figure FDA0002567423340000031
wherein the defined pixel point p belongs to the image O in the local clothing outline and has the coordinate of (x)1,y1) Defining pixel point q as belonging to the out-of-contour image B, and its coordinate is (x)2,y2)。
5. The CNN-based user-interactive image partial apparel style migration method of claim 1, wherein:
in step 4, the pixel value inside the local clothing contour after distance conversion is 0, and the pixel value outside the local clothing contour is increased by using power-level operation to increase the distance difference between the inside and the outside to form the contour characteristic, wherein the formula is as follows:
Figure FDA0002567423340000032
wherein n is at least 2.
6. The CNN-based user-interactive image partial apparel style migration method of claim 1, wherein: the step 5 specifically includes:
step 5.1: random noise image
Figure FDA0002567423340000033
Content characteristics of
Figure FDA0002567423340000034
And content map
Figure FDA0002567423340000035
Content characteristics of
Figure FDA0002567423340000036
The difference is taken to obtain the content loss, and the formula is as follows:
Figure FDA0002567423340000037
step 5.2: random noise image
Figure FDA0002567423340000038
Style characteristics of
Figure FDA0002567423340000039
And style sheet
Figure FDA00025674233400000310
Style characteristics of
Figure FDA00025674233400000311
Making difference to obtain style loss, defining style loss E of layerlThe formula is as follows:
Figure FDA00025674233400000312
the style loss for all layers of the CNN is defined by the following formula:
Figure FDA00025674233400000313
wherein, wlWeights representing the loss of each layer style of the CNN;
step 5.3: mapping random noise
Figure FDA00025674233400000314
Defining the profile features after distance transformation as a matrix
Figure FDA00025674233400000315
Input local dress outline drawing
Figure FDA00025674233400000316
Defining the profile features after distance transformation as a matrix
Figure FDA00025674233400000317
Loss of contour LdThe formula is defined as follows:
Figure FDA00025674233400000318
7. the CNN-based user-interactive image partial apparel style migration method of claim 1, wherein: the step 6 specifically includes:
step 6.1 α, β, gamma, r are four weight coefficients, LTVThe method is characterized in that a total variation regular term is introduced, the function of the total variation regular term is to inhibit noise generated in the style migration process and smooth the boundary of the local clothing outline, and the formula is as follows:
Ltotal＝αLc+βLs+γLd+rLTV(15)
Figure FDA0002567423340000041
Dx、Dyrespectively representing the lateral and longitudinal differences of the resulting image,
Figure FDA0002567423340000042
respectively representing the number of elements corresponding to the difference result;
step 6.2: the loss function is minimized using gradient descent, the derivative of the content loss function being:
Figure FDA0002567423340000043
the derivative of the style loss function is:
Figure FDA0002567423340000044
step 6.3: and updating the network weight to minimize loss, and generating a new clothes style with the picture style only fused with partial clothes.
CN202010628294.8A 2020-07-02 2020-07-02 CNN-based user interactive image local clothing style migration method Active CN111768335B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202010628294.8A CN111768335B (en) 2020-07-02 2020-07-02 CNN-based user interactive image local clothing style migration method

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202010628294.8A CN111768335B (en) 2020-07-02 2020-07-02 CNN-based user interactive image local clothing style migration method

Publications (2)

Publication Number Publication Date
CN111768335A true CN111768335A (en) 2020-10-13
CN111768335B CN111768335B (en) 2023-08-04

Family

ID=72723453

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202010628294.8A Active CN111768335B (en) 2020-07-02 2020-07-02 CNN-based user interactive image local clothing style migration method

Country Status (1)

Country Link
CN (1) CN111768335B (en)

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112508966A (en) * 2020-10-27 2021-03-16 北京科技大学 Interactive image segmentation method and system
CN113160033A (en) * 2020-12-28 2021-07-23 武汉纺织大学 Garment style migration system and method
CN114782653A (en) * 2022-06-23 2022-07-22 杭州彩连科技有限公司 Method and system for automatically expanding dress design layout
CN115205167A (en) * 2021-04-12 2022-10-18 北京字跳网络技术有限公司 Image processing method and device
WO2023151299A1 (en) * 2022-02-11 2023-08-17 华为云计算技术有限公司 Data generation method and apparatus, device, and storage medium

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20180068463A1 (en) * 2016-09-02 2018-03-08 Artomatix Ltd. Systems and Methods for Providing Convolutional Neural Network Based Image Synthesis Using Stable and Controllable Parametric Models, a Multiscale Synthesis Framework and Novel Network Architectures
CN109829537A (en) * 2019-01-30 2019-05-31 华侨大学 Style transfer method and equipment based on deep learning GAN network children's garment clothes
CN110111291A (en) * 2019-05-10 2019-08-09 衡阳师范学院 Based on part and global optimization blending image convolutional neural networks Style Transfer method
CN110490791A (en) * 2019-07-10 2019-11-22 西安理工大学 Dress ornament Graphic Arts generation method based on deep learning Style Transfer

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20180068463A1 (en) * 2016-09-02 2018-03-08 Artomatix Ltd. Systems and Methods for Providing Convolutional Neural Network Based Image Synthesis Using Stable and Controllable Parametric Models, a Multiscale Synthesis Framework and Novel Network Architectures
CN109829537A (en) * 2019-01-30 2019-05-31 华侨大学 Style transfer method and equipment based on deep learning GAN network children's garment clothes
CN110111291A (en) * 2019-05-10 2019-08-09 衡阳师范学院 Based on part and global optimization blending image convolutional neural networks Style Transfer method
CN110490791A (en) * 2019-07-10 2019-11-22 西安理工大学 Dress ornament Graphic Arts generation method based on deep learning Style Transfer

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112508966A (en) * 2020-10-27 2021-03-16 北京科技大学 Interactive image segmentation method and system
CN113160033A (en) * 2020-12-28 2021-07-23 武汉纺织大学 Garment style migration system and method
CN113160033B (en) * 2020-12-28 2023-04-28 武汉纺织大学 Clothing style migration system and method
CN115205167A (en) * 2021-04-12 2022-10-18 北京字跳网络技术有限公司 Image processing method and device
WO2023151299A1 (en) * 2022-02-11 2023-08-17 华为云计算技术有限公司 Data generation method and apparatus, device, and storage medium
CN114782653A (en) * 2022-06-23 2022-07-22 杭州彩连科技有限公司 Method and system for automatically expanding dress design layout

Also Published As

Publication number Publication date
CN111768335B (en) 2023-08-04

Similar Documents

Publication Publication Date Title
CN111768335B (en) CNN-based user interactive image local clothing style migration method
WO2020000814A1 (en) Computer-implemented method for generating composite image, apparatus for generating composite image, and computer-program product
CN111553837B (en) Artistic text image generation method based on neural style migration
Liao et al. Automatic caricature generation by analyzing facial features
CN111507334B (en) Instance segmentation method based on key points
CN111986125A (en) Method for multi-target task instance segmentation
Qin et al. Automatic skin and hair masking using fully convolutional networks
CN104715451A (en) Seamless image fusion method based on consistent optimization of color and transparency
CN108510500A (en) A kind of hair figure layer process method and system of the virtual figure image based on face complexion detection
CN107862664A (en) A kind of image non-photorealistic rendering method and system
CN111340720A (en) Color register woodcut style conversion algorithm based on semantic segmentation
CN108596992B (en) Rapid real-time lip gloss makeup method
Lu et al. Sketch simplification based on conditional random field and least squares generative adversarial networks
CN111932645A (en) Method for automatically generating ink and wash painting based on generation countermeasure network GAN
Pierre et al. Recent approaches for image colorization
CN113436058B (en) Character virtual clothes changing method, terminal equipment and storage medium
Fu et al. Fast accurate and automatic brushstroke extraction
CN114898021B (en) Intelligent cartoon method for music stage performance video
Kim et al. Low-light image enhancement using volume-based subspace analysis
CN108765384B (en) Significance detection method for joint manifold sequencing and improved convex hull
CN107818579B (en) Color texture feature extraction method based on quaternion Gabor filtering
CN112529771B (en) Portrait style migration method
CN114037644B (en) Artistic word image synthesis system and method based on generation countermeasure network
CN111627033B (en) Method, equipment and computer readable storage medium for dividing difficult sample instance
CN114332923A (en) Chinese annotation gesture recognition method and system based on improved YOLOv5

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant