WO2020034481A1 - 一种图像风格转换方及装置、设备、存储介质 - Google Patents
一种图像风格转换方及装置、设备、存储介质 Download PDFInfo
- Publication number
- WO2020034481A1 WO2020034481A1 PCT/CN2018/117293 CN2018117293W WO2020034481A1 WO 2020034481 A1 WO2020034481 A1 WO 2020034481A1 CN 2018117293 W CN2018117293 W CN 2018117293W WO 2020034481 A1 WO2020034481 A1 WO 2020034481A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- image
- gradient
- style
- pixel
- feature map
- Prior art date
Links
- 238000006243 chemical reaction Methods 0.000 title claims abstract description 154
- 238000000034 method Methods 0.000 title claims abstract description 112
- 238000012549 training Methods 0.000 claims abstract description 207
- 230000008569 process Effects 0.000 claims description 32
- 238000000605 extraction Methods 0.000 claims description 14
- 238000004590 computer program Methods 0.000 claims description 11
- 230000008859 change Effects 0.000 claims description 6
- 238000013527 convolutional neural network Methods 0.000 description 43
- 230000006870 function Effects 0.000 description 26
- 230000005284 excitation Effects 0.000 description 18
- 238000005070 sampling Methods 0.000 description 18
- 238000010586 diagram Methods 0.000 description 12
- 238000004422 calculation algorithm Methods 0.000 description 8
- 230000004913 activation Effects 0.000 description 7
- 238000005516 engineering process Methods 0.000 description 7
- 210000002569 neuron Anatomy 0.000 description 7
- 238000011176 pooling Methods 0.000 description 7
- 238000013528 artificial neural network Methods 0.000 description 5
- 238000004891 communication Methods 0.000 description 5
- 238000004364 calculation method Methods 0.000 description 4
- 239000000203 mixture Substances 0.000 description 4
- 230000009466 transformation Effects 0.000 description 4
- 238000013135 deep learning Methods 0.000 description 3
- 230000009286 beneficial effect Effects 0.000 description 2
- 238000006073 displacement reaction Methods 0.000 description 2
- 239000000284 extract Substances 0.000 description 2
- 238000007477 logistic regression Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000001537 neural effect Effects 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000005457 optimization Methods 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 230000004044 response Effects 0.000 description 2
- 238000012546 transfer Methods 0.000 description 2
- ORILYTVJVMAKLC-UHFFFAOYSA-N Adamantane Natural products C1C(C2)CC3CC1CC2C3 ORILYTVJVMAKLC-UHFFFAOYSA-N 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 239000003086 colorant Substances 0.000 description 1
- 230000008878 coupling Effects 0.000 description 1
- 238000010168 coupling process Methods 0.000 description 1
- 238000005859 coupling reaction Methods 0.000 description 1
- 230000010365 information processing Effects 0.000 description 1
- 230000005764 inhibitory process Effects 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 239000011159 matrix material Substances 0.000 description 1
- 238000003062 neural network model Methods 0.000 description 1
- 230000008447 perception Effects 0.000 description 1
- 238000004321 preservation Methods 0.000 description 1
- 230000000644 propagated effect Effects 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000012706 support-vector machine Methods 0.000 description 1
- 238000013519 translation Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/04—Context-preserving transformations, e.g. by using an importance map
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T11/00—2D [Two Dimensional] image generation
- G06T11/001—Texturing; Colouring; Generation of texture or colour
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/90—Determination of colour characteristics
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
- G06V10/443—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components by matching or filtering
- G06V10/449—Biologically inspired filters, e.g. difference of Gaussians [DoG] or Gabor filters
- G06V10/451—Biologically inspired filters, e.g. difference of Gaussians [DoG] or Gabor filters with interaction between the filter responses, e.g. cortical complex cells
- G06V10/454—Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/168—Feature extraction; Face representation
- G06V40/171—Local features and components; Facial parts ; Occluding parts, e.g. glasses; Geometrical relationships
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10024—Color image
Definitions
- the present application relates to image technology, and in particular, to an image style conversion method and device, device, and storage medium.
- Image style conversion based on deep learning is a new research issue in recent years. Although the problem of image style conversion has always existed, it was only in 2015 that German researcher Gatys used a neural network method to open the door to creating image art styles with deep learning. The current technology does not optimize the style conversion of face photos. For example, when the existing method is applied to a self-timer image, the common shortcomings are: the deformation of the face edge and the inconsistent face color caused by the image style conversion. .
- the embodiments of the present application provide an image style conversion method and device, device, and storage medium in order to solve at least one problem in the prior art.
- An embodiment of the present application provides an image style conversion method.
- the method includes: obtaining an initial image for style conversion; inputting a gradient of the initial image into an image style conversion model, and obtaining the image style conversion model from the image style conversion model.
- the feature map of the initial image on the gradient domain; the image style conversion model is obtained based on pixel-level loss and perceptual loss in the gradient domain; image reconstruction is performed according to the feature map of the initial image on the gradient domain to obtain Style image.
- the image style conversion model includes a pixel-level loss model and a perceptual loss model, wherein the pixel-level loss model is obtained by using a minimum pixel-level loss as a training target in a gradient domain, and the perceptual loss The model is obtained by training in the gradient domain with the minimum perceived loss as the training target.
- the training process of the pixel-level loss model and the perceptual loss model includes: inputting a gradient of a training sample into the pixel-level loss model, and obtaining a sample of the training sample from the pixel-level loss model. Output results; determining the gradient of the stylized reference image corresponding to the training sample; a first output feature map of the j-th convolution layer of the perceptual loss model according to the gradient of the reference image, and the output result according to the sample At the second output feature map of the j-th convolution layer of the perceptual loss model, the perceptual loss model is trained.
- the second output feature map of the convolution layer to train the perceptual loss model includes: training the perceptual loss model using the following formula: among them, Represents the gradient of the ith training sample, F W represents the pixel-level loss model, Represents the output of the gradient of the i-th training sample through the pixel-level loss model; Represents the gradient of the stylized reference image of the i-th training sample; ⁇ j () represents the output feature map of the j-th convolution layer when the perceptual loss model adopts the perceptual loss model, and C j H j W j does not denote the j-th The number of channels, height and width of the feature map corresponding to the layer convolution layer.
- the training process of the pixel-level loss model includes: taking a gradient of a training sample as an input of the pixel-level loss model, obtaining a sample output result from the pixel-level loss model; determining that the training sample corresponds to The gradient of the stylized reference image; training the pixel-level loss model according to the gradient of the reference image and sample output results.
- the pixel-level loss model includes a first set of convolutional layers, an upsampling layer, and a second set of convolutional layers, and the pixel-level loss is trained according to a gradient of the reference image and a sample output result.
- the model includes: inputting the gradient of the training sample to the first convolution layer set to obtain a sample feature map; inputting the sample feature map to the upsampling layer, and upsampling to the pixel size of the initial image ; Inputting the up-sampled sample feature map to the second set of convolutional layers to obtain a sample output result.
- the training the pixel-level loss model according to the gradient of the reference image and the sample output results includes: With the corresponding The absolute value of the difference trains the pixel-level loss model; wherein, Represents the gradient of the ith training sample, F W represents the pixel-level loss model, Represents the output of the gradient of the i-th training sample through the pixel-level loss model F W ; Gradient representing the stylized reference image of the i-th training sample.
- the With the corresponding Training the pixel-level loss model with the absolute value of the difference includes: training the pixel-level loss model with the following formula: among them, Represents the gradient of the ith training sample, F W represents the pixel-level loss model, Represents the output of the gradient of the i-th training sample through the pixel-level loss model F W ; Represents the gradient of the stylized reference image of the i-th training sample, and D represents the number of samples in the training sample set.
- performing image reconstruction based on the feature map of the initial image on the gradient domain to obtain a style image includes: satisfying structural similarity with the feature map of the initial image on the gradient domain.
- a conditional image is used as the style image; wherein the feature map on the gradient domain and the initial image meets the structural similarity condition, including: the degree of structural difference between the style image and the initial image is less than similar Or the degree of structural difference between the style image and the initial image is the smallest, wherein the degree of structural difference is that the style image on the gradient domain and the feature map of the initial image on the gradient domain are in at least one reference direction Trends.
- performing image reconstruction according to the feature map of the initial image on the gradient domain to obtain a style image includes: Perform image reconstruction to obtain a style image; of which: Represents the gradient of the initial image in the x direction, A feature map representing the gradient of the initial image in the x direction on the gradient domain after passing through the image style conversion model, Represents the gradient of the initial image in the y direction, A feature map representing the gradient of the initial image in the y direction on the gradient domain through the image style conversion model, Represents the gradient of the style image in the x direction, Represents the gradient of the style image in the y direction.
- performing image reconstruction according to the feature map of the initial image on the gradient domain to obtain a style image includes: according to the color information of the initial image and the initial image on the gradient domain.
- the feature map is reconstructed to obtain a style image.
- performing image reconstruction according to the color information of the initial image and a feature map of the initial image on a gradient domain to obtain a style image includes: comparing the initial image with a gradient domain. The image on which the feature map satisfies the structural similarity condition, and the image that satisfies the color similarity condition with the initial image is used as the style image.
- the method further includes: performing feature extraction on the initial image to obtain a face region in the initial image; and correspondingly, according to the color information of the initial image and the initial image Image reconstruction of the feature map on the gradient domain to obtain a style image, including: an image that will meet the structural similarity condition with the feature map on the gradient domain of the initial image, and An image in which a face region meets a color similarity condition is used as the style image.
- the image that will meet the structural similarity condition with the feature map of the initial image on the gradient domain, and the image that meets the color similarity condition with the initial image as the style image Including: according to Perform image reconstruction to obtain a style image; where: I represents the initial image, S represents the style image, Represents the gradient of the initial image in the x direction, A feature map representing the gradient of the initial image in the x direction on the gradient domain after passing through the image style conversion model, Represents the gradient of the initial image in the y direction, A feature map representing the gradient of the initial image in the y-direction on the gradient domain through the image style conversion model, Represents the gradient of the style image in the x direction, Represents the gradient of the style image in the y direction.
- the step of inputting the gradient of the initial image into an image style conversion model and obtaining a feature map of the initial image on the gradient domain from the image style conversion model includes: determining that the initial image is at least A gradient in a reference direction; a gradient in at least one reference direction is input to an image style conversion model, and a feature map of the initial image in the gradient domain in at least one reference direction is correspondingly obtained from the image style conversion model; correspondingly, according to Image reconstruction is performed on the feature maps in the gradient domain in at least one reference direction to obtain a style image.
- the at least one reference direction includes the x and y directions in a plane reference coordinate system, and correspondingly, the gradients of the initial image in the x and y directions are determined respectively;
- the gradient in the direction is input to the image style conversion model, and the feature map of the initial image in the gradient domain in the x and y directions is correspondingly obtained from the image style conversion model; correspondingly, according to the gradient in the x and y directions,
- the feature map on the domain is image reconstructed to obtain the style image.
- An embodiment of the present application provides an image style conversion device.
- the device includes: an obtaining unit for obtaining an initial image for style conversion; and an obtaining unit for inputting a gradient of the initial image into an image style conversion model.
- a feature map of the initial image on the gradient domain is obtained from the image style conversion model; the image style conversion model is obtained based on pixel-level loss and perceptual loss training in the gradient domain; a reconstruction unit is configured according to the The feature map of the initial image on the gradient domain is subjected to image reconstruction to obtain a style image.
- the image style conversion model includes a pixel-level loss model and a perceptual loss model, wherein the pixel-level loss model is obtained by using a minimum pixel-level loss as a training target in a gradient domain, and the perceptual loss model It is obtained by training in the gradient domain with the minimum perceived loss as the training target.
- the training unit includes: a first input module, configured to input a gradient of a training sample into the pixel-level loss model, and obtain a sample output result of the training sample from the pixel-level loss model; A determining module, configured to determine a gradient of a stylized reference image corresponding to the training sample; a first training module, configured to, according to the gradient of the reference image, in the j-th convolution layer of the perceptual loss model An output feature map, and training the perceptual loss model on a second output feature map of a j-th convolution layer of the perceptual loss model according to a sample output result.
- the first training module is configured to train the perceptual loss model using the following formula: among them, Represents the gradient of the ith training sample, F W represents the pixel-level loss model, Represents the output of the gradient of the i-th training sample through the pixel-level loss model; Represents the gradient of the stylized reference image of the i-th training sample; ⁇ j () represents the output feature map of the j-th convolution layer when the perceptual loss model adopts the perceptual loss model, and C j H j W j does not denote the j-th The number of channels, height and width of the feature map corresponding to the layer convolution layer.
- the training unit includes: a second determination module, configured to determine a gradient of training samples;
- a second input module is configured to use the gradient of the training sample as an input of the pixel-level loss model, and obtain a sample output result from the pixel-level loss model; a third determination module is configured to determine a corresponding value of the training sample.
- the pixel-level loss model includes a first set of convolutional layers, an upsampling layer, and a second set of convolutional layers
- the second training module includes: a first input sub-module for The gradient of the training sample is input to the first set of convolution layers to obtain a sample feature map; an upsampling submodule is used to input the sample feature map to the upsampling layer and upsampling to pixels of the initial image Size; a second input sub-module, configured to input the up-sampled sample feature map to the second set of convolutional layers to obtain a sample output result.
- the second training module is configured to: With the corresponding The absolute value of the difference trains the pixel-level loss model; wherein, Represents the gradient of the ith training sample, F W represents the pixel-level loss model, Represents the output of the gradient of the i-th training sample through the pixel-level loss model F W ; Gradient representing the stylized reference image of the i-th training sample.
- the second training module is configured to train the pixel-level loss model using the following formula: among them, Represents the gradient of the ith training sample, F W represents the pixel-level loss model, Represents the output of the gradient of the i-th training sample through the pixel-level loss model F W ; Represents the gradient of the stylized reference image of the i-th training sample, and D represents the number of samples in the training sample set.
- the reconstruction unit is configured to use, as the style image, an image that satisfies a structural similarity condition with a feature map on a gradient domain of the initial image; and
- the feature map of the image on the gradient domain satisfies the structural similarity condition, including: the degree of structural difference between the style image and the initial image is less than a similarity threshold, or the degree of structural difference between the style image and the initial image The smallest, wherein the degree of structural difference is the change trend of the style image on the gradient domain and the feature image on the gradient domain in at least one reference direction.
- the reconstruction unit is configured to: Perform image reconstruction to obtain a style image; of which: Represents the gradient of the initial image in the x direction, A feature map representing the gradient of the initial image in the x direction on the gradient domain after passing through the image style conversion model, Represents the gradient of the initial image in the y direction, A feature map representing the gradient of the initial image in the y-direction on the gradient domain through the image style conversion model, Represents the gradient of the style image in the x direction, Represents the gradient of the style image in the y direction.
- the reconstruction unit is configured to perform image reconstruction according to color information of the initial image and a feature map of the initial image on a gradient domain to obtain a style image.
- the reconstruction unit is configured to match the feature map of the initial image on the gradient domain with an image that satisfies the structural similarity condition, and the image that will meet the color similarity condition with the initial image. As the style image.
- the apparatus further includes: an extraction unit configured to perform feature extraction on the initial image to obtain a face region in the initial image; correspondingly, the reconstruction unit is configured to combine the The feature image of the initial image on the gradient domain satisfies the structural similarity condition, and the image that satisfies the color similarity condition with the face area in the initial image is used as the style image.
- the reconstruction unit is configured to Perform image reconstruction to obtain a style image; where: I represents the initial image, S represents the style image, Represents the gradient of the initial image in the x direction, A feature map representing the gradient of the initial image in the x direction on the gradient domain after passing through the image style conversion model, Represents the gradient of the initial image in the y direction, A feature map representing the gradient of the initial image in the y-direction on the gradient domain through the image style conversion model, Represents the gradient of the style image in the x direction, Represents the gradient of the style image in the y direction.
- the obtaining unit includes: a fourth determining module for determining a gradient of the initial image in at least one reference direction; an obtaining module for inputting a gradient in at least one reference direction to an image style A transformation model, corresponding to obtaining a feature map of the initial image in the gradient domain in at least one reference direction from the image style transformation model; correspondingly, the reconstruction unit is configured to: The feature map is reconstructed to obtain a style image.
- the at least one reference direction includes x and y directions in a plane reference coordinate system, and correspondingly, a determining unit is configured to determine gradients of the initial image in x and y directions, respectively;
- the obtaining unit is configured to input the gradients in the x and y directions to the image style conversion model, and correspondingly obtain a feature map of the initial image in the gradient domain in the x and y directions from the image style conversion model.
- the reconstruction unit is configured to perform image reconstruction according to the feature maps in the gradient domain in the x and y directions to obtain a style image.
- An embodiment of the present application provides a computer device including a memory and a processor.
- the memory stores a computer program executable on the processor, and the processor executes the steps in the image style conversion method when the program is executed.
- An embodiment of the present application provides a computer-readable storage medium on which a computer program is stored.
- the computer program is executed by a processor, the steps in the foregoing image style conversion method are implemented.
- An embodiment of the present application provides a computer program product.
- the computer program product includes computer-executable instructions. After the computer-executable instructions are executed, the steps in the image style conversion method can be implemented.
- an initial image to be style converted is obtained; a gradient of the initial image is input to an image style conversion model, and the image style conversion model is obtained from the image style conversion model.
- the image style conversion model based on pixel-level loss and perceptual loss training in the gradient domain can overcome the disadvantages of edge deformation and inconsistent color of the face in the related technology, and can achieve image style conversion.
- FIG. 1 is a schematic diagram of a composition structure of a network architecture according to an embodiment of the present application
- FIG. 2A is a schematic flowchart of an image style conversion method according to an embodiment of the present application.
- FIG. 2B is a schematic diagram of a download scenario according to an embodiment of the present application.
- FIG. 3A is a first schematic view of an implementation scenario according to an embodiment of the present application.
- FIG. 3B is a second schematic diagram of an implementation scenario according to an embodiment of the present application.
- FIG. 4A is a third schematic diagram of an implementation scenario according to an embodiment of the present application.
- FIG. 4B is a fourth schematic view of an implementation scenario according to an embodiment of the present application.
- FIG. 5A is a schematic structural diagram of a convolutional neural network model according to an embodiment of the present application.
- 5B is a schematic structural diagram of a pixel-level loss model according to an embodiment of the present application.
- FIG. 6 is a schematic structural diagram of a composition of an image style conversion device according to an embodiment of the present application.
- FIG. 7 is a schematic diagram of a hardware entity of a computer device according to an embodiment of the present application.
- the process of generating a style map using a neural network method is generally as follows: using a neural network model such as VGG16 model or VGG19, image feature extraction is performed on an original image (Content Image) and a style image (Style Image), that is, Content features are extracted from the original image, and style features are extracted from the style image.
- a loss function using the content features and style features, the loss value calculation is performed on a randomly initialized image and the redrawn image is fed back to generate a generated image.
- the generated image will be similar in content to the original image and in style. It will look similar to the style image.
- this algorithm needs to be trained every time it generates an image, and it takes a long time.
- training a network can transform any image into the corresponding style of the network, so each time an image is generated, the network is propagated forward only once, and the speed will be very fast.
- the fast transfer algorithm generally includes two networks: one is an Image Transform Network, and one is a Loss Network.
- the image conversion network is used to convert the image. Its parameters are changed, while the parameters of the loss network remain unchanged.
- the VGG-16 network trained in the ImageNet image library can be used as the loss network.
- the original image is converted by the image.
- the three results of the network, the style map and the original image, are through the loss network, extract the perceptual loss, and use the perceptual loss to train the image conversion network.
- a large number of images are used to train the image conversion network to obtain a model, and in the output phase, the model is applied to output to generate a map.
- the network obtained in this way is three orders of magnitude faster than the Gatys model in generating maps.
- the current technology does not optimize the style conversion of face photos.
- the edges of the face may deviate from the original The image, that is, the structural information of the output image is changed; 2)
- the skin color of the face may be inconsistent with the original skin color, that is, the color information of the output image is changed.
- the portrait of user A in the initial image is a round face.
- the portrait of user A in the output style image is an awl face;
- the skin of user B is fair.
- the skin of user B is darkened in the output style image. That is, how to better maintain the structural information and color information of the original initial image becomes a problem to be solved.
- an embodiment of the present application proposes a Convolutional Neural Networks (CNN) structure based entirely on image style conversion in the image gradient domain. Due to the edge preservation of the gradient domain learning, this The image style conversion network provided by the embodiment of the application can overcome the disadvantages of edge deformation of the previous method.
- a term called color confidence is introduced to maintain the fidelity of the skin color of the resulting image.
- the image reconstruction phase uses both the structure information and the color information of the original image, which can make the result more natural.
- perceptual loss is directly used in the gradient domain, so that the learned style information is more focused on strokes rather than colors, which makes it more suitable for the task of style conversion with human faces.
- the sampling operation refers to a subsampled operation or a down-sampled operation. If the sampling object is a continuous signal, the continuous signal is a discrete signal after the downsampling operation.
- the purpose of the downsampling operation may be to reduce the image for computational convenience.
- the principle of the downsampling operation For an image whose I size is M * N, downsampling it by s times, we get a resolution image with the size of (M / s) * (N / s). Of course, s should be M The common divisor of N and N only works. If you consider an image in the form of a matrix, you are turning the image in the original image s * s window into a pixel, and the value of this pixel is the average value of all pixels in the window.
- the upsampling operation is the reverse process of the downsampling operation. It is also called up-sampling or interpolating. For images, high-resolution images can be obtained by upsampling operations.
- the principle of the upsampling operation almost all image enlargement uses interpolation method, that is, based on the original image pixels, a new pixel is inserted between pixels using a suitable interpolation algorithm.
- Channel The word has two different meanings. The first is for sample images (images are used as training samples). Channels refer to color channels (number of channels). Examples of images are used below. Represents the channel of the sample image; the second is the dimension of the output space, such as the number of output channels in the convolution operation, or the number of convolution kernels in each convolution layer.
- Color channel which decomposes an image into one or more color components or color components.
- a single color channel only one numerical value is required for each pixel, which can only represent grayscale, and 0 is black.
- RGB red, green and blue
- the image is divided into three red, green, and blue color channels, which can represent color, and all 0s represent black.
- alpha 0 means full transparency.
- Convolutional neural network is a kind of multilayer supervised learning neural network.
- the convolutional layer and pool sampling layer of the hidden layer are the core modules to realize the feature extraction function of the convolutional neural network.
- the low hidden layer of the convolutional neural network is composed of the convolutional layer and the maximum pool sampling layer alternately, and the high layer is the hidden layer and the logistic regression classifier of the fully connected layer corresponding to the traditional multilayer perceptron.
- the input of the first fully connected layer is a feature image obtained by feature extraction from the convolutional layer and the sub-sampling layer.
- the final output layer is a classifier that can use logistic regression, Softmax regression or even support vector machines to classify the initial image.
- Each layer in CNN is composed of multiple maps, and each map is composed of multiple neural units.
- CNN generally uses convolutional layers and sampling layers alternately, that is, a convolutional layer is connected to a sampling layer, and a sampling layer is followed by a convolution; of course, multiple convolutional layers can be connected to a sampling layer, so that the convolutional layer Features are extracted, and then combined to form more abstract features, and finally to describe the feature of the image object, the CNN can be followed by a fully connected layer.
- the convolutional neural network structure includes a convolutional layer, a downsampling layer, and a fully connected layer.
- Each layer has multiple feature maps, each feature map extracts a feature of the input through a convolution filter, and each feature map has multiple neurons.
- Convolution layer The reason for using the convolution layer is an important feature of the convolution operation. Through the convolution operation, the original signal features can be enhanced and the noise can be reduced.
- the reason for using downsampling is that according to the principle of image local correlation, subsampling the image can reduce the amount of calculation while maintaining the image rotation invariance.
- Fully connected layer using softmax full connection, the activation value obtained is the image features extracted by the convolutional neural network.
- a neuron is the basic unit of a multilayer perceptron, and its function becomes activation transmission. That is, for a neuron, the input is part or all of the input of the convolutional neural network or part or all of the output of the previous layer. After calculation of the activation function, the result obtained is used as the output of the neuron.
- activation functions include sigmoid function, tanh function, and linear rectifier function (Rectified Linear Unit, ReLu).
- Pixel-wise Loss Assuming I est is the output of a convolutional neural network, and I HR is the original high-resolution image, then pixel-wise loss emphasizes that each image between I est and I HR The matching of the corresponding pixels is different from the perception result of the human eye. Generally speaking, images trained through pixel-wise loss are usually smooth and lack high-frequency information.
- I est represents the output of the convolutional neural network
- I HR represents the original high-resolution image
- I est and I HR are input into a differentiable function ⁇ , which avoids the need for the network
- the output image is pixel-wise consistent with the original high-resolution image.
- the VGG model and VGG model structure are simple and effective.
- the first few layers only use 3 ⁇ 3 convolution kernels to increase the network depth.
- the maximum number of neurons in each layer is reduced in turn by max pooling.
- the last three layers are two.
- "16" and "19" indicate the number of network layers in the network that need to be updated and require weights (that is, weight, parameters to be learned).
- the weights of the VGG16 model and the VGG19 model are both trained by ImageNet.
- Model parameters can generally be understood as configuration variables inside the model. Historical data or training samples can be used to estimate the values of model parameters, or model parameters are variables that can be automatically learned from historical data or training samples.
- model parameters have the following characteristics: model parameters are needed for model prediction; model parameter values can define model functions; model parameters are obtained by data estimation or data learning; model parameters are generally not manually set by practitioners; model parameters are usually Saved as part of the learning model; optimization algorithms are usually used to estimate model parameters, which is an efficient search of the possible values of the parameters.
- the weights and biases of network models are generally called model parameters.
- Model hyperparameters can generally be understood as the configuration outside the model, whose values cannot be estimated from the data. To some extent, the characteristics of model hyperparameters are: model hyperparameters are often used in the process of estimating model parameters; model hyperparameters are usually directly specified by practitioners; model hyperparameters can usually be set using heuristics; model hyperparameters It is usually adjusted for a given predictive modeling problem. In other words, model hyperparameters are used to determine some parameters of the model. Different hyperparameters are different for models. This model means that there are slight differences. For example, suppose that they are all CNN models. If the number of layers is different, the models are different, although they are all CNN models. In deep learning, the super parameters are: learning rate, number of iterations, number of layers, number of neurons in each layer, and so on.
- FIG. 1 is a schematic diagram of a composition structure of the network architecture according to the embodiment of the present application.
- the network architecture includes two or more electronic devices 11 to 1N and a server 31.
- the electronic devices 11 to 1N interact with the server 31 through the network 21.
- the electronic device may be various types of computer devices with information processing capabilities during the implementation process.
- the electronic device may include a mobile phone, a tablet computer, a desktop computer, a personal digital assistant, a navigator, a digital phone, a television, and the like.
- An embodiment of the present application proposes an image style conversion method, which can effectively solve the problem that the structural information of the output image changes compared to the initial image.
- the method is applied to an electronic device, and the functions implemented by the method can be processed by the electronic device.
- the programmer calls program code to implement.
- the program code can be stored in a computer storage medium. It can be seen that the electronic device includes at least a processor and a storage medium.
- FIG. 2A is a schematic flowchart of an image style conversion method according to an embodiment of the present application. As shown in FIG. 2A, the method includes:
- Step S201 Obtain an initial image for style conversion
- the image style conversion method provided in the embodiment of the present application may be embodied by a client (application) during the implementation process.
- the user downloads the client from the server 31 on the electronic device 12 on the user.
- the electronic device 12 sends a download request to the server 31.
- the download request is used to download the client, and the server 31 responds to the download request.
- the server 31 sends a download response to the electronic device 12, and the download response carries a client, such as an Android application package (APK) for the Android system, and then the user installs the downloaded client on his electronic device, and then the electronic device Running the client, that is, the electronic device can implement the image style conversion method provided in the embodiment of the present application.
- API Android application package
- step S201 is implemented on the electronic device side
- the implementation process may be as follows: When the user selects a picture from the album, the client receives the user's operation of selecting a picture, that is, the client determines the selected picture as a style to be performed. The converted initial image; or, the user takes a photo with the camera of the electronic device or an external camera, and the client receives the operation of the user to take the photo, that is, the client determines the captured photo as the initial image to be style converted.
- this step can also have other implementations.
- Step S202 input the gradient of the initial image into an image style conversion model, and obtain a feature map of the initial image on the gradient domain from the image style conversion model;
- the image style conversion model is trained and obtained based on pixel-level loss and perceptual loss in the gradient domain.
- the image style conversion model is obtained by using pixel-level loss and perceptual loss as training targets in the gradient domain;
- Step S203 Perform image reconstruction according to the feature map of the initial image on the gradient domain to obtain a style image.
- the style image is a reconstructed and stylized image.
- the trained image style conversion model can be local to the electronic device, or it can be on the server side.
- the trained image style conversion model may be when the electronic device installs a client, that is, the trained image style conversion model is installed.
- the electronic device obtains the initial image through step S201
- obtain a feature map ie, an output result
- the electronic device outputs the obtained style image to the user.
- the trained image style conversion model may also be located on the server side, as shown in FIG. 3B.
- the electronic device sends the initial image to the server, so that the server receives the initial image sent by the electronic device, and the server implements the steps.
- step S201 includes: the server receives the initial image sent by the electronic device, that is, the server obtains the initial image to be style converted, and the server obtains the step S202 The feature map of the initial image on the gradient domain, and finally the output style image is obtained through step S203.
- the server can also send the style image to An electronic device.
- the electronic device After receiving the style image, the electronic device outputs the style image to the user.
- the client after the client is installed on the electronic device, the user uploads the initial image of the user, receives the style image sent by the server, and outputs the style image to the user.
- steps S201 to S203 may be partially performed by an electronic device, or may be partially performed by a server.
- steps S201 and S202 may be performed by the electronic device. It executes locally, and then the electronic device sends the feature map of the initial image on the gradient domain to the server. After the server executes step S203, the style image is obtained, and then the style image is sent to the electronic device, and the electronic device outputs the style image.
- steps S201 and S202 may be performed by a server. The server sends a feature map of the initial image on the gradient domain to the electronic device. After the electronic device executes step S203, a style image is obtained, and then the style is The image is output to the user.
- the method further includes: training the image style conversion model, wherein the training target of the image style conversion model is to minimize the total loss L total , where L total is expressed by the following formula:
- L total ⁇ L feat + ⁇ L pixel ; wherein, L feat represents a perceived loss, L pixel represents a pixel-level loss, and the values of ⁇ and ⁇ are real numbers.
- the ratio of the ⁇ to the ⁇ is greater than 10 and less than the fifth power.
- the value of ⁇ is 10000, and the value of ⁇ is 1.
- the values of ⁇ and ⁇ can be set according to specific application scenarios, and the values of the ⁇ and ⁇ are not limited in the embodiments of the present application.
- the image style conversion model includes a pixel-level loss model and a perceptual loss model, wherein the pixel-level loss model is a pixel-level loss model obtained by using a minimum pixel-level loss as a training target in a gradient domain.
- the perceptual loss model is obtained by training in a gradient domain with a minimum perceptual loss as a training target.
- the pixel-level loss model is a pixel-level loss model
- the perceptual loss model is a training process when the perceptual loss model includes:
- Step S11 Determine the gradient of the training sample; assuming that i i represents the i-th training sample, determine that the gradient of the i-th training sample I i is
- Step S12 input the gradient of the training sample into the pixel-level loss model, and obtain a sample output result of the training sample from the pixel-level loss model; wherein, the gradient of the i-th training sample I i Input the pixel-level loss model F W , and obtain a sample output result of the training sample from the pixel-level loss model
- Step S13 Determine the gradient of the stylized reference image corresponding to the training sample; where the stylized reference image may be an unsatisfactory stylized reference image obtained by using an existing stylization algorithm, then suppose The stylized reference image corresponding to the training sample I i is Then the gradient of the reference image is
- Step S14 According to the gradient of the reference image, the first output feature map of the j-th convolution layer of the perceptual loss model, and the first output feature map of the j-th convolution layer of the perceptual loss model according to the sample output result.
- Two output feature maps train the perceptual loss model.
- the j-th convolutional layer may be any layer in a convolutional neural network model.
- the j-th convolutional layer may be a conv3-3 layer in VGG16.
- the pixel-level loss model includes a first set of convolutional layers, an upsampling layer, and a second set of convolutional layers, and the pixel-level loss is trained according to a gradient of the reference image and a sample output result.
- the model includes: inputting the gradient of the training sample to the first convolution layer set to obtain a sample feature map; inputting the sample feature map to the upsampling layer, and upsampling to the pixel size of the initial image ; Inputting the up-sampled sample feature map to the second set of convolutional layers to obtain a sample output result.
- the second output feature map of the convolution layer to train the perceptual loss model includes:
- the perceptual loss model is trained using the following formula:
- F W represents the pixel-level loss model
- ⁇ j () represents the output feature map of the j-th layer of the convolutional layer when the perceptual loss model uses a convolutional neural network model
- C j H j W j The number of channels, height, and width of the feature map corresponding to the j-th convolution layer.
- the j-th convolution layer is conv3-3.
- the training process when the pixel-level loss model is a pixel-level loss model includes:
- Step S21 Determine the gradient of the training sample
- Step S22 Use the gradient of the training sample as the input of the pixel-level loss model, and obtain the sample output result from the pixel-level loss model
- Step S23 determine the training sample corresponding The gradient of the stylized reference image
- step S24 training the pixel-level loss model according to the gradient of the reference image and the sample output result.
- training the pixel-level loss model according to the gradient of the reference image and the sample output results includes: With the corresponding The absolute value of the difference trains the pixel-level loss model; wherein, Represents the gradient of the ith training sample, F W represents the pixel-level loss model, Represents the output of the gradient of the i-th training sample through the pixel-level loss model F W ; Gradient representing the stylized reference image of the i-th training sample.
- the With the corresponding Training the pixel-level loss model with the absolute value of the difference includes: training the pixel-level loss model with the following formula:
- F W represents the pixel-level loss model
- D represents the number of samples in the training sample set.
- performing image reconstruction based on the feature map of the initial image on the gradient domain to obtain a style image includes: satisfying structural similarity with the feature map of the initial image on the gradient domain.
- a conditional image is used as the style image.
- the feature map on the gradient domain with the initial image satisfies a structural similarity condition, including: the degree of structural difference between the style image and the initial image is less than a similarity threshold or the style image and the The degree of structural difference of the initial image is the smallest, wherein the degree of structural difference is a change trend of a style image on the gradient domain and a feature map of the initial image on the gradient domain in at least one reference direction.
- the reference direction can be the x and y directions of the image in the plane reference coordinate system. Of course, there can be other directions, or only one direction can be used.
- the degree of difference can use the difference or the absolute value of the difference or various mathematical deformation operations based on the difference (such as the sum of the squares of the absolute values of the differences in the x and y directions, that is, (I represents the initial image, S represents the style image, and ⁇ ⁇ represents the absolute value symbol).
- performing image reconstruction according to the feature map of the initial image on the gradient domain to obtain a style image includes: Perform image reconstruction to obtain a style image; of which: Represents the gradient of the initial image in the x direction, A feature map representing the gradient of the initial image in the x-direction through the image style conversion model on the gradient domain, Represents the gradient of the initial image in the y direction, A feature map representing the gradient of the initial image in the y-direction on the gradient domain through the image style conversion model, Represents the gradient of the style image in the x direction, Represents the gradient of the style image in the y direction.
- performing image reconstruction according to the feature map of the initial image on the gradient domain to obtain a style image includes: according to the color information of the initial image and the initial image on the gradient domain. The feature map is reconstructed to obtain a style image.
- performing image reconstruction based on the color information of the initial image and a feature map of the initial image on the gradient domain to obtain a style image includes: comparing the feature image of the initial image on the gradient domain with the feature map. An image that satisfies a structural similarity condition, and an image that satisfies a color similarity condition with the initial image is used as the style image.
- the method further includes: performing feature extraction on the initial image to obtain a face region in the initial image; and correspondingly, according to the color information of the initial image and the initial image Image reconstruction of the feature map on the gradient domain to obtain a style image, including: an image that will meet the structural similarity condition with the feature map on the gradient domain of the initial image, and An image in which a face region meets a color similarity condition is used as the style image.
- the color similarity condition that is, the color similarity condition that the color information satisfies, that is, the degree of difference between the color of the style image and the initial image is less than the set value or the smallest, wherein the degree of color difference is the sampling point of the image to be processed and the target image
- the difference value of the color value is represented by "SI", where I represents the initial image and S represents the style image).
- a color similarity condition is set, where the color similarity condition may be the color of the entire initial image, or the color of the human face in the initial image. colour.
- the above two conditional structural similarity conditions and color similarity conditions can be used independently in theory, that is, only one condition is used to calculate the style image; two can also be used at the same time, and the corresponding coefficients (weights) are assigned at the same time. ), For example, the value of ⁇ is a real number.
- the image that will meet the structural similarity condition with the feature map of the initial image on the gradient domain, and the image that meets the color similarity condition with the initial image as the style image Including: according to Perform image reconstruction to obtain a style image; where: I represents the initial image, S represents the style image, Represents the gradient of the initial image in the x direction, A feature map representing the gradient of the initial image in the x-direction through the image style conversion model on the gradient domain Represents the gradient of the initial image in the y direction, A feature map representing the gradient of the initial image in the y-direction on the gradient domain through the image style conversion model, Represents the gradient of the style image in the x direction, Represents the gradient of the style image in the y direction.
- the step of inputting the gradient of the initial image into an image style conversion model and obtaining a feature map of the initial image on the gradient domain from the image style conversion model includes: Step S31, determining the initial image The gradient of the image in at least one reference direction; step S32, the gradient in the at least one reference direction is input to an image style conversion model, and the features of the initial image in the gradient domain in at least one reference direction are correspondingly obtained from the image style conversion model.
- image reconstruction is performed according to the feature map on the gradient domain in at least one reference direction to obtain a style image.
- the at least one reference direction includes x, y directions in a plane reference coordinate system, and correspondingly, determining the gradient of the initial image in at least one reference direction includes: determining the initial The gradient of the image in the x and y directions, respectively; said inputting the gradient in at least one reference direction to the image style conversion model, and correspondingly obtaining the initial image in the gradient domain in at least one reference direction from said image style conversion model
- the feature map includes: inputting gradients in the x and y directions to an image style conversion model, and obtaining a feature map of the initial image in the gradient domain in the x and y directions from the image style conversion model;
- the image reconstruction based on the feature map on the gradient domain in at least one reference direction to obtain a style image includes: performing image reconstruction based on the feature map on the gradient domain in the x and y directions to obtain a style. image.
- the first stage the structure of the convolutional neural network model provided in the embodiment of the present application is first described, and then in the second stage, the training of the provided convolutional neural network model is introduced.
- the process, followed by the third stage introduces the process of image reconstruction using the trained convolutional neural network, that is, the method of image style conversion on the initial image.
- the first stage the structure of the convolutional neural network model
- FIG. 5A is a schematic structural diagram of a convolutional neural network model according to an embodiment of the present application. As shown in FIG. 5A, the convolutional neural network network is composed of two parts:
- the first part is the convolutional neural network 51 (the first convolutional neural network) to be trained, which takes the gradient of the selfie image as input, followed by continuous convolutional layers and ReLu layers, and then uses an upsampling operation to feature the feature map (feature map) upsampling to the original image size, and finally calculating the pixel-wise loss (L- pixel ) with the gradient of the artistic style reference image;
- the gradient of the selfie image as input includes: taking the selfie image in the x direction gradient And selfie image gradient in y direction Respectively as the input of a convolutional neural network.
- each convolution filter of the convolutional layer repeatedly acts on the entire receptive field, convolving the input selfie image, and the result of the convolution constitutes the feature map of the input selfie image.
- Local features of the selfie image were extracted.
- a feature of convolutional neural networks is: max-pooling sampling, which is a non-linear downsampling method. From the mathematical formula of max-pooling, it can be seen that max-pooling is the extraction of feature points in the neighborhood. maximum. After obtaining image features through convolution, these features are used for classification. After obtaining the convolutional feature map of the image, the convolutional features are reduced in dimension by the maximum pool sampling method.
- the convolution features are divided into several disjoint regions, and the largest (or average) features of these regions are used to represent the convolution features after dimensionality reduction.
- the function of the maximum pool sampling method is reflected in two aspects: (1), it reduces the computational complexity from the upper hidden layer; (2), these pooling units have translation invariance, even if the image has a small displacement, extraction The characteristics obtained will remain unchanged. Due to the enhanced robustness to displacement, the maximum pool sampling method is an efficient sampling method that reduces the data dimension.
- the second part is a VGG-16 network 52 (second convolutional neural network) that has been trained in ImageNet to calculate the perceptual loss L feat .
- the output of the conv3-3 layer of VGG-16 is actually used to calculate the perceptual loss.
- the total objective function L total can be calculated using the following formula (3-1).
- L total ⁇ L feat + ⁇ L pixel (3-1); where the values of ⁇ and ⁇ are real numbers. For example, ⁇ and ⁇ can be set to integers during training.
- Image gradient is a method to describe the differences between image pixels. From a mathematical point of view, the image gradient refers to the first derivative of the pixel.
- the following formula can be used to represent the gradient in the x direction of the image And y-direction gradient Represented by the following formula (3-2) and formula (3-3):
- the second stage the training process of the first part of the convolutional neural network
- Equation (4-1) Represents the gradient or gradient representation of the i-th original image I i in the x direction, Represents the gradient or gradient representation in the y-direction. Represents the gradient of the original image, Represents the gradient of the original image I i in the x direction, Represents the gradient of the original image I i in the y direction.
- F W represents the first part of the convolutional neural network model, so Representing the result of the gradient of the i-th original image I i through a convolutional neural network, Represents the result of the gradient of the i-th original image I i in the x direction through a convolutional neural network network, Represents the result of the gradient of the i-th original image I i in the y direction through a convolutional neural network network.
- Represents the gradient of the stylized reference image of the i-th original image I i Represents the gradient of the stylized reference image of the i-th original image I i in the x direction, Represents the gradient in the y direction of the stylized reference image of the i-th original image I i .
- ⁇ j () represents the output feature map of the j-th convolution layer of the VGG-16 network
- C j , H j , and W j respectively represent the The number of channels, height and width of the feature map corresponding to the j-layer convolutional layer.
- the conv3-3 layer of VGG-16 is used.
- the total objective function is the sum of the perceptual loss L feat and the pixel-level loss L pixel ;
- L total ⁇ L feat + ⁇ L pixel (4-3);
- the values of ⁇ and ⁇ are real numbers.
- ⁇ and ⁇ can be set to integers during training.
- ⁇ and ⁇ were set to 10000 and 1, respectively.
- the adam optimization method was used to optimize the objective function formula 3.
- the learning rate was set to 10 -8 , the last 50K times, set the learning rate to 10 -9 . It should be noted that during the implementation process, those skilled in the art can make some modifications to formula (4-1) and formula (4-2).
- formula (4-1) as long as these modifications can represent pixel-level losses, for example, the formula (4-1) Modify it to something else, such as or Wait, modify the square of the absolute value in the formula (4-1) to an absolute value, or modify the square of the absolute value in the formula (4-1) to the square root of the absolute value.
- the third stage the image reconstruction process
- formula (5) When a new image is input, such as a new self-timer image, in order to obtain the corresponding style image, the following formula (5) is used to determine the output stylized image.
- I represents a new self-timer image, that is, an initial image
- S represents a style image corresponding to the new self-timer image.
- the embodiment of the present application implements a self-oriented image style conversion algorithm, which overcomes two important disadvantages of the previous style conversion method when applied to a human face: First, the deformation of the face edge Second, the inconsistency of human skin color.
- the neural network structure of the embodiment of the present application is entirely learned in the gradient domain. Compared with other image style conversion methods, this method can overcome the shortcomings of edge deformation and inconsistent color in the previous method in style conversion of selfie photos. It can beautify and enhance images while achieving image style conversion.
- the first part is the convolutional neural network 51 (first convolutional neural network) to be trained.
- the convolutional neural network shown in FIG. 5B may be adopted.
- FIG. 5B is a composition of the convolutional neural network model according to the embodiment of the present application. Schematic structure, as shown in Figure 5B. The structure of the model includes:
- Input layer (input) 501 The gradient of the selfie image in the x or y direction is used as input. It should be noted that h represents the high of the gradient of the selfie image in the x or y direction, and w represents the selfie image in the x or The width of the gradient in the y direction.
- h represents the high of the gradient of the selfie image in the x or y direction
- w represents the selfie image in the x or The width of the gradient in the y direction.
- the gradient of the selfie image I in the x direction is obtained
- the gradient of the selfie image I in the y direction is obtained followed by with Each color channel (or color component) as input.
- RGB Red Green Blue, Red Green Blue
- the output result is a feature map 502.
- the height of the feature map 502 is The width of the feature map 502 is The number of channels of the feature map 502 is c, where r is a coefficient, and the values of r and c are related to the model hyperparameters of the convolutional neural network model in the embodiment of the application. It includes the size of the convolution kernel, the stride of the convolution kernel, and the padding of the input feature map. Generally, the number of convolution kernels determines the number of channels c of the output feature map.
- the upsampling layer has 511 to 51C input and 521 to 52C output.
- the output feature map is disassembled according to the number of channels c, so that c feature maps 511 to 51C are obtained, and each of the feature maps 511 to 51C is sampled to the size of the initial image.
- the input layer 501 mentions the initial image, that is, the self-timer image, and the size of the self-timer image is h * w, so the sizes of the up-sampled images 521 to 52C output by the up-sampling layer are also h * w.
- the output corresponding to input 511 is 521
- the output corresponding to input 512 is 522, and so on
- the output corresponding to input 51C is 52C.
- Compositing layer 531 the input is 521 to 52C, and the output is 531; the upsampling images 521 to 52C are combined to obtain the feature map 531; the output layer, the input is 531, and the output is 541; the feature map 531 is convolved and excited, That is, input to conv8, ReLu8, and conv9, and finally get output 541, the size of output 541 is the size of the original image h * w.
- the convolutional neural network model shown in FIG. 5B can be used to replace the network part 53 in FIG. 5A.
- the convolution process before upsampling has 7 layers, which are conv1 to conv7
- the excitation process before upsampling also has 7 layers, which are ReLu1 to ReLu7.
- the 7-layer convolutional layer (conv1 to conv7) can be considered as the first set of convolutional layers of the pixel-level loss model.
- the 7-layer convolutional layer and 7-layer excitation layer (ReLu1 to ReLu7) can also be considered as pixels.
- the first convolutional layer set of the first-order loss model can be considered as the first-order loss model.
- the upsampling there are also two layers of convolutions, conv8 and conv9 respectively; after the upsampling, there is also a layer of excitation process, namely the excitation layer ReLu8.
- the two convolutional layers (conv8 and conv9) can be considered as the second convolutional layer set of the pixel-level loss model.
- the two-layer convolutional layer and the one-layer excitation layer (ReLu8) can also be considered as the pixel-level loss.
- the second set of convolutional layers of the model can be considered as the second convolutional layer set of the pixel-level loss model.
- the number of layers of the convolutional layer (the number of layers of the convolutional layer in the first set of convolutional layers) before the upsampling can be changed, for example, 5 layers, 9 layers, and 10 layers are used. Or dozens of layers.
- the number of layers of the excitation layer before the upsampling can also be changed. For example, 5 layers, 6 layers, 9 layers, and 15 layers are used. and many more.
- the convolution layer before the upsampling, the convolution layer will be followed by an excitation layer, that is, before the upsampling, a convolution layer and an excitation layer are alternated.
- the number of alternating layers and excitation layers can also vary, for example, two convolutional layers followed by one excitation layer, and then one convolutional layer followed by two excitation layers.
- the excitation function used by the excitation layer is ReLu.
- the excitation layer may also use other excitation functions, such as a sigmoid function.
- the pooling layer is not shown in the embodiment shown in FIG. 5B. In some embodiments, a pooling layer may also be added. After the upsampling, the number of layers of the convolutional layer (the number of layers of the convolutional layer in the second set of convolutional layers), and the order of the convolutional layer and the excitation layer can be changed.
- an embodiment of the present application provides an image style conversion device.
- the device includes each unit included, and each module included in each unit can be implemented by a processor in an electronic device. It is realized through specific logic circuits; in the implementation process, the processor may be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP), or a field programmable gate array (FPGA).
- CPU central processing unit
- MPU microprocessor
- DSP digital signal processor
- FPGA field programmable gate array
- FIG. 6 is a schematic structural diagram of an image style conversion device according to an embodiment of the present application. As shown in FIG. 6, the device 600 includes an obtaining unit 601, an obtaining unit 602, and a reconstruction unit 603, where:
- An obtaining unit 601 is configured to obtain an initial image for style conversion; an obtaining unit 602 is configured to input a gradient of the initial image into an image style conversion model, and obtain an on-gradient of the initial image from the image style conversion model. Feature map on the domain; the image style conversion model is obtained based on pixel-level loss and perceptual loss in the gradient domain; a reconstruction unit 603 is configured to perform image reconstruction based on the feature map of the initial image on the gradient domain To get style images.
- the apparatus further includes a training unit for training the image style conversion model, and the training target of the image style conversion model is to minimize the total loss L total , where L total is expressed by the following formula :
- L total ⁇ L feat + ⁇ L pixel ; wherein, L feat represents a perceived loss, L pixel represents a pixel-level loss, and the values of ⁇ and ⁇ are real numbers.
- a ratio of the ⁇ to the ⁇ is greater than 10 and less than a fifth power.
- the image style conversion model includes a pixel-level loss model and a perceptual loss model, wherein the perceptual loss model is a pixel-level loss model obtained by using the minimum pixel-level loss as a training target in the gradient domain, and the perceptual loss The model is obtained by training in the gradient domain with the minimum perceived loss as the training target.
- the training unit includes: a first input module, configured to input a gradient of a training sample into the pixel-level loss model, and obtain a sample output result of the training sample from the pixel-level loss model; A determining module, configured to determine a gradient of a stylized reference image corresponding to the training sample; a first training module, configured to, according to the gradient of the reference image, in the j-th convolution layer of the perceptual loss model An output feature map, and training the perceptual loss model on a second output feature map of a j-th convolution layer of the perceptual loss model according to a sample output result.
- the first training module is configured to train the perceptual loss model using the following formula:
- F W represents the pixel-level loss model
- ⁇ j () represents the output feature map of the j-th convolution layer when the perceptual loss model adopts the perceptual loss model
- C j H j W j does not denote the j-th The number of channels, height and width of the feature map corresponding to the layer convolution layer.
- the j-th convolution layer is conv3-3.
- the training unit further includes: a second determining module for determining a gradient of the training sample; and a second input module for using the gradient of the training sample as an input of the pixel-level loss model, A sample output result is obtained from the pixel-level loss model; a third determination module is configured to determine a gradient of a stylized reference image corresponding to the training sample; a second training module is configured to use the gradient and sample of the reference image The output trains the pixel-level loss model.
- the pixel-level loss model includes a first set of convolutional layers, an upsampling layer, and a second set of convolutional layers, and the pixel-level loss is trained according to a gradient of the reference image and a sample output result.
- the model includes: inputting the gradient of the training sample to a first set of convolutional layers to obtain a sample feature map; inputting the sample feature map to an upsampling layer and upsampling to the pixel size of the initial image; The sampled feature map of the sample is input to the second set of convolutional layers to obtain the sample output result.
- the second training module is configured to: With the corresponding The absolute value of the difference trains the pixel-level loss model; wherein, Represents the gradient of the ith training sample, F W represents the pixel-level loss model, Represents the output of the gradient of the i-th training sample through the pixel-level loss model F W ; Gradient representing the stylized reference image of the i-th training sample.
- the second training module is configured to train the pixel-level loss model using the following formula:
- F W represents the pixel-level loss model
- D represents the number of samples in the training sample set.
- the reconstruction unit is configured to use, as the style image, an image that meets a structural similarity condition with a feature map on the gradient domain of the initial image.
- the feature map on the gradient domain with the initial image satisfies a structural similarity condition, including: the degree of structural difference between the style image and the initial image is less than a similarity threshold; or The degree of structural difference between the style image and the initial image is the smallest, wherein the degree of structural difference is the change trend of the feature image on the gradient domain in the gradient domain and the feature map in the gradient domain in at least one reference direction.
- the reconstruction unit is configured to: Perform image reconstruction to obtain a style image; of which: Represents the gradient of the initial image in the x direction, A feature map representing the gradient of the initial image in the x direction on the gradient domain after passing through the image style conversion model, Represents the gradient of the initial image in the y direction, A feature map representing the gradient of the initial image in the y-direction on the gradient domain through the image style conversion model, Represents the gradient of the style image in the x direction, Represents the gradient of the style image in the y direction.
- the reconstruction unit is configured to perform image reconstruction according to color information of the initial image and a feature map of the initial image on a gradient domain to obtain a style image.
- a reconstruction unit is configured to use an image that satisfies a structural similarity condition with a feature map on the gradient domain of the initial image, and use an image that satisfies a color similarity condition with the initial image as the image. Style image.
- the apparatus further includes: an extraction unit configured to perform feature extraction on the initial image to obtain a face region in the initial image; correspondingly, the reconstruction unit is configured to combine the The feature image of the initial image on the gradient domain satisfies the structural similarity condition, and the image that satisfies the color similarity condition with the face area in the initial image is used as the style image.
- the reconstruction unit is configured to Perform image reconstruction to obtain a style image; where: I represents the initial image, S represents the style image, Represents the gradient of the initial image in the x direction, A feature map representing the gradient of the initial image in the x direction on the gradient domain after passing through the image style conversion model, Represents the gradient of the initial image in the y direction, A feature map on the gradient domain representing the gradient of the initial image in the y-direction through an image style conversion model, Represents the gradient of the style image in the x direction, Represents the gradient of the style image in the y direction.
- the obtaining unit includes: a fourth determining module for determining a gradient of the initial image in at least one reference direction; an obtaining module for inputting a gradient in at least one reference direction to an image style A transformation model, corresponding to obtaining a feature map of the initial image in the gradient domain in at least one reference direction from the image style transformation model; correspondingly, the reconstruction unit is configured to: The feature map is reconstructed to obtain a style image.
- the at least one reference direction includes x and y directions in a plane reference coordinate system, and correspondingly, a determining unit is configured to determine gradients of the initial image in x and y directions, respectively;
- the obtaining unit is configured to input the gradients in the x and y directions to the image style conversion model, and correspondingly obtain a feature map of the initial image in the gradient domain in the x and y directions from the image style conversion model.
- the reconstruction unit is configured to perform image reconstruction according to the feature maps in the gradient domain in the x and y directions to obtain a style image.
- the computer software product is stored in a storage medium and includes several instructions for A computer device (which may be an electronic device or a server, etc.) is caused to execute all or part of the methods described in the embodiments of the present application.
- the foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (Read Only Memory, ROM), a magnetic disk, or an optical disk, which can store program codes.
- ROM Read Only Memory
- ROM Read Only Memory
- magnetic disk or an optical disk
- an embodiments of the present application are not limited to any specific combination of hardware and software.
- an embodiment of the present application provides a computer device including a memory and a processor.
- the memory stores a computer program executable on the processor, and the processor implements the foregoing image style conversion method when the processor executes the program. Steps.
- An embodiment of the present application provides a computer-readable storage medium on which a computer program is stored, where the computer program is executed by a processor, and includes steps in the image style conversion method described above.
- An embodiment of the present application further provides a computer program product.
- the computer program product includes computer-executable instructions. After the computer-executable instructions are executed, the steps in the image style conversion method can be implemented.
- FIG. 7 is a schematic diagram of a hardware entity of a computer device according to an embodiment of the present application.
- the hardware entity of the computer device 700 includes a processor 701, a communication interface 702, and a memory 703, of which:
- the processor 701 generally controls the overall operation of the computer device 700.
- the communication interface 702 may enable a computer device to communicate with other terminals or servers through a network.
- the memory 703 is configured to store instructions and applications executable by the processor 701, and may also cache data to be processed or processed by each module in the processor 701 and the computer device 700 (for example, image data, audio data, voice communication data, and Video communication data), can be realized by flash memory (FLASH) or random access memory (Random Access Memory, RAM).
- an embodiment or “an embodiment” mentioned throughout the specification means that a particular feature, structure, or characteristic related to the embodiment is included in at least one embodiment of the present application.
- the appearances of "in one embodiment” or “in an embodiment” appearing throughout the specification are not necessarily referring to the same embodiment.
- the particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.
- the size of the sequence numbers of the above processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not deal with the embodiments of the present application.
- the implementation process constitutes any limitation.
- the above-mentioned serial numbers of the embodiments of the present application are merely for description, and do not represent the superiority or inferiority of the embodiments.
- the disclosed device and method may be implemented in other ways.
- the device embodiments described above are only schematic.
- the division of the unit is only a logical function division.
- there may be another division manner such as multiple units or components may be combined, or Can be integrated into another system, or some features can be ignored or not implemented.
- the displayed or discussed components are coupled, or directly coupled, or communicated with each other through some interfaces.
- the indirect coupling or communication connection of the device or unit may be electrical, mechanical, or other forms. of.
- each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above integration
- the unit can be implemented in the form of hardware, or in the form of hardware plus software functional units.
- the above-mentioned integrated unit of the present application is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.
- the computer software product is stored in a storage medium and includes several instructions for A computer device (which may be an electronic device or a server) is caused to execute all or part of the methods described in the embodiments of the present application.
- the foregoing storage media include: various types of media that can store program codes, such as a mobile storage device, a ROM, a magnetic disk, or an optical disc.
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Health & Medical Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Oral & Maxillofacial Surgery (AREA)
- Multimedia (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Human Computer Interaction (AREA)
- Computing Systems (AREA)
- Databases & Information Systems (AREA)
- Medical Informatics (AREA)
- Software Systems (AREA)
- Life Sciences & Earth Sciences (AREA)
- Biodiversity & Conservation Biology (AREA)
- Biomedical Technology (AREA)
- Molecular Biology (AREA)
- Image Analysis (AREA)
- Image Processing (AREA)
Abstract
Description
Claims (37)
- 一种图像风格转换方法,所述方法包括:获取待进行风格转换的初始图像;将所述初始图像的梯度输入到图像风格转换模型,从所述图像风格转换模型获得所述初始图像的在梯度域上的特征图;所述图像风格转换模型是在梯度域基于像素级损失和感知损失训练得到;根据所述初始图像的在梯度域上的特征图进行图像重构,得到风格图像。
- 根据权利要求1所述的方法,所述方法还包括:训练所述图像风格转换模型;其中,所述图像风格转换模型的训练目标为总的损失L total最小,其中,L total采用下式来表示:L total=αL feat+βL pixel;其中,所述L feat表示感知损失,所述L pixel表示像素级损失,所述α和所述β的取值均为实数。
- 根据权利要求1所述的方法,所述图像风格转换模型包括像素级损失模型和感知损失模型,其中所述像素级损失模型是通过在梯度域将像素级损失最小作为训练目标而得到的,所述感知损失模型是通过在梯度域训练将感知损失最小作为训练目标而得到的。
- 根据权利要求3所述的方法,所述像素级损失模型和所述感知损失模型的训练过程包括:将训练样本的梯度输入所述像素级损失模型,从所述像素级损失模型获得所述训练样本的样本输出结果;确定所述训练样本对应的风格化的参考图像的梯度;根据所述参考图像的梯度在所述感知损失模型的第j层卷积层的第一输出特征图,和根据样本输出结果在所述感知损失模型的第j层卷积层的第二输出特征图,训练所述感知损失模型。
- 根据权利要求3所述的方法,所述像素级损失模型的训练过程包括:将训练样本的梯度作为所述像素级损失模型的输入,从所述像素级损失模型获得样本输出结果;确定所述训练样本对应的风格化的参考图像的梯度;根据所述参考图像的梯度和样本输出结果训练所述像素级损失模型。
- 根据权利要求4所述的方法,所述像素级损失模型包括第一卷积层集合、上采样层和第二卷积层集合,所述根据所述参考图像的梯度和样本输出结果训练所述像素级损失模型包括:将所述训练样本的梯度输入到所述第一卷积层集合,得到样本特征图;将所述样本特征图输入到所述上采样层,上采样至所述初始图像的像素尺寸;将上采样后的样本特征图输入到所述第二卷积层集合,得到样本输出结果。
- 根据权利要求1至9任一项所述的方法,所述根据所述初始图像的在梯度域上的特征图进行图像重构,得到风格图像,包括:将与所述初始图像的在梯度域上的特征图满足结构相似度条件的图像,作为所述风格图像;其中,所述与所述初始图像的在梯度域上的特征图满足结构相似度条件,包括:所述风格图像与所述初始图像的结构差异程度小于相似度阈值,或者,所述风格图像与所述初始图像的结构差异程度最小,其中,结构差异程度为梯度域上的风格图像与所述初始图像的在梯度域上的特征图在至少一个参考方向的变化趋势。
- 根据权利要求1至9任一项所述的方法,所述根据所述初始图像的在梯度域上的特征图进行图像重构,得到风格图像,包括:根据所述初始图像的颜色信息和所述初始图像的在梯度域上的特征图进行图像重构,得到风格图像。
- 根据权利要求12所述的方法,所述根据所述初始图像的颜色信息和所述初始图像的在梯度域上的特征图进行图像重构,得到风格图像,包括:将与所述初始图像的在梯度域上的特征图满足结构相似度条件的图像,且将与所述初始图像满足颜色相似度条件的图像,作为所述风格图像。
- 根据权利要求12所述的方法,所述方法还包括:对所述初始图像进行特征提取,得到所述初始图像中的人脸区域;对应地,所述根据所述初始图像的颜色信息和所述初始图像的在梯度域上的特征图进行图像重构,得到风格图像,包括:将与所述初始图像的在梯度域上的特征图满足结构相似度条件的图像,且将与所述初始图像中的人脸区域满足颜色相似度条件的图像,作为所述风格图像。
- 根据权利要求1至9任一项所述的方法,所述将所述初始图像的梯度输入到图像风格转换模型,从所述图像风格转换模型获得初始图像的在梯度域上的特征图,包括:确定所述初始图像在至少一个参考方向的梯度;将在至少一个参考方向的梯度输入到图像风格转换模型,从所述图像风格转换模型对应获得初始图像在至少一个参考方向的在梯度域上的特征图;对应地,根据在至少一个参考方向的在梯度域上的特征图进行图像重构,得到风格图像。
- 根据权利要求16所述的方法,所述至少一个参考方向包括在平面参考坐标系中的x、y方向上,对应地,确定所述初始图像分别在x、y方向上的梯度;分别将在x、y方向上的梯度输入到图像风格转换模型,从所述图像风格转换模型对应获得初始图像在x、y方向上的在梯度域上的特征图;对应地,根据在x、y方向上的在梯度域上的特征图进行图像重构,得到风格图像。
- 一种图像风格转换装置,所述装置包括:获取单元,用于获取待进行风格转换的初始图像;获得单元,用于将所述初始图像的梯度输入到图像风格转换模型,从所述图像风格转换模型获得所述初始图像的在梯度域上的特征图;所述图像风格转换模型是在梯度域基于像素级损失和感知损失训练得到;重构单元,用于根据所述初始图像的在梯度域上的特征图进行图像重构,得到风格 图像。
- 根据权利要求18所述的装置,所述装置还包括:训练单元,用于训练所述图像风格转换模型,其中所述图像风格转换模型的训练目标为总的损失L total最小,其中,L total采用下式来表示:L total=αL feat+βL pixel;其中,所述L feat表示感知损失,所述L pixel表示像素级损失,所述α和所述β的取值均为实数。
- 根据权利要求18所述的装置,所述图像风格转换模型包括像素级损失模型和感知损失模型,其中所述像素级损失模型是通过在梯度域将像素级损失最小作为训练目标而得到,所述感知损失模型是通过在梯度域训练将感知损失最小作为训练目标而得到。
- 根据权利要求20所述的装置,所述训练单元包括:第一输入模块,用于将训练样本的梯度输入所述像素级损失模型,从所述像素级损失模型获得所述训练样本的样本输出结果;第一确定模块,用于确定所述训练样本对应的风格化的参考图像的梯度;第一训练模块,用于根据所述参考图像的梯度在所述感知损失模型的第j层卷积层的第一输出特征图,和根据样本输出结果在所述感知损失模型的第j层卷积层的第二输出特征图训练所述感知损失模型。
- 根据权利要求20所述的装置,所述训练单元包括:第二确定模块,用于确定训练样本的梯度;第二输入模块,用于将所述训练样本的梯度作为所述像素级损失模型的输入,从所述像素级损失模型获得样本输出结果;第三确定模块,用于确定所述训练样本对应的风格化的参考图像的梯度;第二训练模块,用于根据所述参考图像的梯度和样本输出结果训练所述像素级损失模型。
- 根据权利要求21所述的装置,所述像素级损失模型包括第一卷积层集合、上采样层和第二卷积层集合,所述第二训练模块包括:第一输入子模块,用于将所述训练样本的梯度输入到所述第一卷积层集合,得到样本特征图;上采样子模块,用于将所述样本特征图输入到所述上采样层,上采样至所述初始图像的像素尺寸;第二输入子模块,用于将上采样后的样本特征图输入到所述第二卷积层集合,得到 样本输出结果。
- 根据权利要求18至26任一项所述的装置,所述重构单元,用于将与所述初始图像的在梯度域上的特征图满足结构相似度条件的图像,作为所述风格图像;其中,所述与所述初始图像的在梯度域上的特征图满足结构相似度条件,包括:所述风格图像与所述初始图像的结构差异程度小于相似度阈值,或者,所述风格图像与所述初始图像的结构差异程度最小,其中,结构差异程度为梯度域上的风格图像与所述初始图像的在梯度域上的特征图在至少一个参考方向的变化趋势。
- 根据权利要求18至26任一项所述的装置,所述重构单元,用于根据所述初始图像的颜色信息和所述初始图像的在梯度域上的特征图进行图像重构,得到风格图像。
- 根据权利要求29所述的装置,所述重构单元,用于将与所述初始图像的在梯度域上的特征图满足结构相似度条件的图像,且将与所述初始图像满足颜色相似度条件的图像,作为所述风格图像。
- 根据权利要求29所述的装置,所述装置还包括:提取单元,用于对所述初始图像进行特征提取,得到所述初始图像中的人脸区域;对应地,所述重构单元,用于将与所述初始图像的在梯度域上的特征图满足结构相似度条件的图像,且将与所述初始图像中的人脸区域满足颜色相似度条件的图像,作为所述风格图像。
- 根据权利要求18至26任一项所述的装置,所述获得单元,包括:第四确定模块,用于确定所述初始图像在至少一个参考方向的梯度;获得模块,用于将在至少一个参考方向的梯度输入到图像风格转换模型,从所述图像风格转换模型对应获得初始图像在至少一个参考方向的在梯度域上的特征图;对应地,所述重构单元,用于根据在至少一个参考方向的在梯度域上的特征图进行图像重构,得到风格图像。
- 根据权利要求33所述的装置,所述至少一个参考方向包括在平面参考坐标系中的x、y方向上,对应地,确定单元,用于确定所述初始图像分别在x、y方向上的梯度;所述获得单元,用于分别将在x、y方向上的梯度输入到图像风格转换模型,从所述图像风格转换模型对应获得初始图像在x、y方向上的在梯度域上的特征图;对应地,所述重构单元,用于根据在x、y方向上的在梯度域上的特征图进行图像重构,得到风格图像。
- 一种计算机程序产品,所述计算机程序产品包括计算机可执行指令,该计算机可执行指令被执行后,能够实现权利要求1至17任一项所述图像风格转换方法中的步骤。
- 一种计算机存储介质,所述计算机存储介质上存储有计算机可执行指令,该计算机可执行指令被执行后,能够实现权利要求1至17任一项所述图像风格转换方法中的步骤。
- 一种计算机设备,所述计算机设备包括存储器和处理器,所述存储器上存储有计算机可执行指令,所述处理器执行所述程序时实现权利要求1至17任一项所述图像风格转换方法中的步骤。
Priority Applications (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2019569805A JP6874168B2 (ja) | 2018-08-13 | 2018-11-23 | 画像スタイル変換方法および装置、機器、ならびに記憶媒体 |
SG11202000062RA SG11202000062RA (en) | 2018-08-13 | 2018-11-23 | Image style transform methods and apparatuses, devices and storage media |
US16/726,885 US11200638B2 (en) | 2018-08-13 | 2019-12-25 | Image style transform methods and apparatuses, devices and storage media |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201810917979.7A CN109308679B (zh) | 2018-08-13 | 2018-08-13 | 一种图像风格转换方法及装置、设备、存储介质 |
CN201810917979.7 | 2018-08-13 |
Related Child Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US16/726,885 Continuation US11200638B2 (en) | 2018-08-13 | 2019-12-25 | Image style transform methods and apparatuses, devices and storage media |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2020034481A1 true WO2020034481A1 (zh) | 2020-02-20 |
Family
ID=65223859
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/CN2018/117293 WO2020034481A1 (zh) | 2018-08-13 | 2018-11-23 | 一种图像风格转换方及装置、设备、存储介质 |
Country Status (6)
Country | Link |
---|---|
US (1) | US11200638B2 (zh) |
JP (1) | JP6874168B2 (zh) |
CN (1) | CN109308679B (zh) |
SG (1) | SG11202000062RA (zh) |
TW (1) | TWI749356B (zh) |
WO (1) | WO2020034481A1 (zh) |
Cited By (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN111652846A (zh) * | 2020-04-30 | 2020-09-11 | 成都数之联科技有限公司 | 一种基于特征金字塔卷积神经网络的半导体缺陷识别方法 |
CN111932445A (zh) * | 2020-07-27 | 2020-11-13 | 广州市百果园信息技术有限公司 | 对风格迁移网络的压缩方法及风格迁移方法、装置和系统 |
CN112102154A (zh) * | 2020-08-20 | 2020-12-18 | 北京百度网讯科技有限公司 | 图像处理方法、装置、电子设备和存储介质 |
CN113496238A (zh) * | 2020-03-20 | 2021-10-12 | 北京京东叁佰陆拾度电子商务有限公司 | 模型训练方法、点云数据风格化方法、装置、设备及介质 |
CN114818803A (zh) * | 2022-04-25 | 2022-07-29 | 上海韶脑传感技术有限公司 | 基于神经元优化的单侧肢体患者运动想象脑电建模方法 |
TWI779824B (zh) * | 2021-09-10 | 2022-10-01 | 瑞昱半導體股份有限公司 | 卷積神經網路的圖像處理方法與系統 |
Families Citing this family (36)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2018033137A1 (zh) * | 2016-08-19 | 2018-02-22 | 北京市商汤科技开发有限公司 | 在视频图像中展示业务对象的方法、装置和电子设备 |
CN109308679B (zh) * | 2018-08-13 | 2022-08-30 | 深圳市商汤科技有限公司 | 一种图像风格转换方法及装置、设备、存储介质 |
CN109766895A (zh) * | 2019-01-03 | 2019-05-17 | 京东方科技集团股份有限公司 | 用于图像风格迁移的卷积神经网络的训练方法和图像风格迁移方法 |
CN111583165B (zh) * | 2019-02-19 | 2023-08-08 | 京东方科技集团股份有限公司 | 图像处理方法、装置、设备及存储介质 |
CN110070482B (zh) * | 2019-03-14 | 2023-05-02 | 北京字节跳动网络技术有限公司 | 图像处理方法、装置和计算机可读存储介质 |
EP3731154A1 (en) * | 2019-04-26 | 2020-10-28 | Naver Corporation | Training a convolutional neural network for image retrieval with a listwise ranking loss function |
CN111860823B (zh) * | 2019-04-30 | 2024-06-11 | 北京市商汤科技开发有限公司 | 神经网络训练、图像处理方法及装置、设备及存储介质 |
CN110232401B (zh) * | 2019-05-05 | 2023-08-04 | 平安科技(深圳)有限公司 | 基于图片转换的病灶判断方法、装置、计算机设备 |
CN110189246B (zh) * | 2019-05-15 | 2023-02-28 | 北京字节跳动网络技术有限公司 | 图像风格化生成方法、装置及电子设备 |
CN112561778B (zh) * | 2019-09-26 | 2024-07-02 | 北京字节跳动网络技术有限公司 | 图像风格化处理方法、装置、设备及存储介质 |
US11625576B2 (en) * | 2019-11-15 | 2023-04-11 | Shanghai United Imaging Intelligence Co., Ltd. | Systems and methods for image style transformation |
US11080833B2 (en) * | 2019-11-22 | 2021-08-03 | Adobe Inc. | Image manipulation using deep learning techniques in a patch matching operation |
KR102172644B1 (ko) * | 2020-01-13 | 2020-11-02 | (주)에스프레소미디어 | 스타일 변환 외부 연동 시스템, 그리고 스타일 변환 외부 연동 서버 |
CN111340905B (zh) * | 2020-02-13 | 2023-08-04 | 北京百度网讯科技有限公司 | 图像风格化方法、装置、设备和介质 |
CN111494946B (zh) * | 2020-04-23 | 2021-05-18 | 腾讯科技(深圳)有限公司 | 图像处理方法、装置、设备及计算机可读存储介质 |
CN113570508A (zh) * | 2020-04-29 | 2021-10-29 | 上海耕岩智能科技有限公司 | 图像修复方法及装置、存储介质、终端 |
CN111402143B (zh) * | 2020-06-03 | 2020-09-04 | 腾讯科技(深圳)有限公司 | 图像处理方法、装置、设备及计算机可读存储介质 |
CN111667401B (zh) * | 2020-06-08 | 2022-11-29 | 武汉理工大学 | 多层次渐变图像风格迁移方法及系统 |
CN111986075B (zh) * | 2020-08-12 | 2022-08-09 | 兰州交通大学 | 一种目标边缘清晰化的风格迁移方法 |
CN112070668B (zh) * | 2020-08-18 | 2024-07-12 | 中科南京人工智能创新研究院 | 一种基于深度学习和边缘增强的图像超分辨方法 |
US20220121931A1 (en) | 2020-10-16 | 2022-04-21 | Adobe Inc. | Direct regression encoder architecture and training |
CN112288622B (zh) * | 2020-10-29 | 2022-11-08 | 中山大学 | 一种基于多尺度生成对抗网络的伪装图像生成方法 |
CN112233041B (zh) * | 2020-11-05 | 2024-07-05 | Oppo广东移动通信有限公司 | 图像美颜处理方法、装置、存储介质与电子设备 |
US20220156987A1 (en) * | 2020-11-16 | 2022-05-19 | Disney Enterprises, Inc. | Adaptive convolutions in neural networks |
CN112348739B (zh) * | 2020-11-27 | 2021-09-28 | 广州博冠信息科技有限公司 | 图像处理方法、装置、设备及存储介质 |
CN112686269B (zh) * | 2021-01-18 | 2024-06-25 | 北京灵汐科技有限公司 | 池化方法、装置、设备和存储介质 |
KR102573822B1 (ko) * | 2021-02-04 | 2023-09-04 | (주)비케이 | 벡터 이미지의 화풍 변환 및 재생 방법 |
US11195080B1 (en) * | 2021-03-29 | 2021-12-07 | SambaNova Systems, Inc. | Lossless tiling in convolution networks—tiling configuration |
CN113240576B (zh) * | 2021-05-12 | 2024-04-30 | 北京达佳互联信息技术有限公司 | 风格迁移模型的训练方法、装置、电子设备及存储介质 |
CN113344772B (zh) * | 2021-05-21 | 2023-04-07 | 武汉大学 | 一种用于地图艺术化的迁移模型的训练方法和计算机设备 |
CN113256750B (zh) * | 2021-05-26 | 2023-06-23 | 武汉中科医疗科技工业技术研究院有限公司 | 医疗图像风格重建方法、装置、计算机设备和存储介质 |
CN113052786B (zh) * | 2021-05-31 | 2021-09-03 | 北京星天科技有限公司 | 一种声呐图像合成方法和装置 |
CN113763233B (zh) * | 2021-08-04 | 2024-06-21 | 深圳盈天下视觉科技有限公司 | 一种图像处理方法、服务器及拍照设备 |
US11989916B2 (en) * | 2021-10-11 | 2024-05-21 | Kyocera Document Solutions Inc. | Retro-to-modern grayscale image translation for preprocessing and data preparation of colorization |
CN114004905B (zh) * | 2021-10-25 | 2024-03-29 | 北京字节跳动网络技术有限公司 | 人物风格形象图的生成方法、装置、设备及存储介质 |
CN117974425A (zh) * | 2024-03-01 | 2024-05-03 | 艺咖(北京)科技有限公司 | 一种基于扩散模型的二维人脸重建方法及系统 |
Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN105719327A (zh) * | 2016-02-29 | 2016-06-29 | 北京中邮云天科技有限公司 | 一种艺术风格化图像处理方法 |
CN107171932A (zh) * | 2017-04-27 | 2017-09-15 | 腾讯科技(深圳)有限公司 | 一种图片风格转换方法、装置及系统 |
CN107277615A (zh) * | 2017-06-30 | 2017-10-20 | 北京奇虎科技有限公司 | 直播风格化处理方法、装置、计算设备及存储介质 |
CN107481185A (zh) * | 2017-08-24 | 2017-12-15 | 深圳市唯特视科技有限公司 | 一种基于视频图像优化的风格转换方法 |
CN107578367A (zh) * | 2017-04-25 | 2018-01-12 | 北京陌上花科技有限公司 | 一种风格化图像的生成方法及装置 |
Family Cites Families (23)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6064768A (en) * | 1996-07-29 | 2000-05-16 | Wisconsin Alumni Research Foundation | Multiscale feature detector using filter banks |
AUPS170902A0 (en) * | 2002-04-12 | 2002-05-16 | Canon Kabushiki Kaisha | Face detection and tracking in a video sequence |
US6862024B2 (en) * | 2002-04-17 | 2005-03-01 | Mitsubishi Electric Research Laboratories, Inc. | Enhancing textured range images using a 2D editor |
US8306366B2 (en) * | 2007-08-23 | 2012-11-06 | Samsung Electronics Co., Ltd. | Method and apparatus for extracting feature points from digital image |
US8009921B2 (en) * | 2008-02-19 | 2011-08-30 | Xerox Corporation | Context dependent intelligent thumbnail images |
US8705847B2 (en) * | 2011-09-30 | 2014-04-22 | Cyberlink Corp. | Method and system of two-dimensional to stereoscopic conversion |
CN102360490B (zh) * | 2011-09-30 | 2012-11-28 | 北京航空航天大学 | 基于颜色转换和编辑传播的图像季节特征增强方法 |
US9208539B2 (en) * | 2013-11-30 | 2015-12-08 | Sharp Laboratories Of America, Inc. | Image enhancement using semantic components |
CN106415594B (zh) * | 2014-06-16 | 2020-01-10 | 北京市商汤科技开发有限公司 | 用于面部验证的方法和系统 |
US20150371360A1 (en) * | 2014-06-20 | 2015-12-24 | Qualcomm Incorporated | Systems and methods for obtaining structural information from a digital image |
US9826149B2 (en) * | 2015-03-27 | 2017-11-21 | Intel Corporation | Machine learning of real-time image capture parameters |
CN106022221B (zh) * | 2016-05-09 | 2021-11-30 | 腾讯科技(深圳)有限公司 | 一种图像处理方法及处理系统 |
CN106780367B (zh) * | 2016-11-28 | 2019-11-15 | 上海大学 | 基于字典学习的hdr照片风格转移方法 |
US20180197317A1 (en) * | 2017-01-06 | 2018-07-12 | General Electric Company | Deep learning based acceleration for iterative tomographic reconstruction |
US10262238B2 (en) * | 2017-04-13 | 2019-04-16 | Facebook, Inc. | Panoramic camera systems |
US10565757B2 (en) * | 2017-06-09 | 2020-02-18 | Adobe Inc. | Multimodal style-transfer network for applying style features from multi-resolution style exemplars to input images |
AU2017101166A4 (en) * | 2017-08-25 | 2017-11-02 | Lai, Haodong MR | A Method For Real-Time Image Style Transfer Based On Conditional Generative Adversarial Networks |
TWM558943U (zh) | 2017-11-22 | 2018-04-21 | Aiwin Technology Co Ltd | 運用深度學習技術之智慧影像資訊及大數據分析系統 |
CN108280814B (zh) * | 2018-02-08 | 2021-08-31 | 重庆邮电大学 | 基于感知损失的光场图像角度超分辨率重建方法 |
US10783622B2 (en) * | 2018-04-25 | 2020-09-22 | Adobe Inc. | Training and utilizing an image exposure transformation neural network to generate a long-exposure image from a single short-exposure image |
CN109308679B (zh) * | 2018-08-13 | 2022-08-30 | 深圳市商汤科技有限公司 | 一种图像风格转换方法及装置、设备、存储介质 |
US10896534B1 (en) * | 2018-09-19 | 2021-01-19 | Snap Inc. | Avatar style transformation using neural networks |
US11310475B2 (en) * | 2019-08-05 | 2022-04-19 | City University Of Hong Kong | Video quality determination system and method |
-
2018
- 2018-08-13 CN CN201810917979.7A patent/CN109308679B/zh active Active
- 2018-11-23 WO PCT/CN2018/117293 patent/WO2020034481A1/zh active Application Filing
- 2018-11-23 JP JP2019569805A patent/JP6874168B2/ja active Active
- 2018-11-23 SG SG11202000062RA patent/SG11202000062RA/en unknown
-
2019
- 2019-08-13 TW TW108128796A patent/TWI749356B/zh active
- 2019-12-25 US US16/726,885 patent/US11200638B2/en active Active
Patent Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN105719327A (zh) * | 2016-02-29 | 2016-06-29 | 北京中邮云天科技有限公司 | 一种艺术风格化图像处理方法 |
CN107578367A (zh) * | 2017-04-25 | 2018-01-12 | 北京陌上花科技有限公司 | 一种风格化图像的生成方法及装置 |
CN107171932A (zh) * | 2017-04-27 | 2017-09-15 | 腾讯科技(深圳)有限公司 | 一种图片风格转换方法、装置及系统 |
CN107277615A (zh) * | 2017-06-30 | 2017-10-20 | 北京奇虎科技有限公司 | 直播风格化处理方法、装置、计算设备及存储介质 |
CN107481185A (zh) * | 2017-08-24 | 2017-12-15 | 深圳市唯特视科技有限公司 | 一种基于视频图像优化的风格转换方法 |
Cited By (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN113496238A (zh) * | 2020-03-20 | 2021-10-12 | 北京京东叁佰陆拾度电子商务有限公司 | 模型训练方法、点云数据风格化方法、装置、设备及介质 |
CN111652846A (zh) * | 2020-04-30 | 2020-09-11 | 成都数之联科技有限公司 | 一种基于特征金字塔卷积神经网络的半导体缺陷识别方法 |
CN111652846B (zh) * | 2020-04-30 | 2022-08-16 | 成都数之联科技股份有限公司 | 一种基于特征金字塔卷积神经网络的半导体缺陷识别方法 |
CN111932445A (zh) * | 2020-07-27 | 2020-11-13 | 广州市百果园信息技术有限公司 | 对风格迁移网络的压缩方法及风格迁移方法、装置和系统 |
CN112102154A (zh) * | 2020-08-20 | 2020-12-18 | 北京百度网讯科技有限公司 | 图像处理方法、装置、电子设备和存储介质 |
CN112102154B (zh) * | 2020-08-20 | 2024-04-26 | 北京百度网讯科技有限公司 | 图像处理方法、装置、电子设备和存储介质 |
TWI779824B (zh) * | 2021-09-10 | 2022-10-01 | 瑞昱半導體股份有限公司 | 卷積神經網路的圖像處理方法與系統 |
CN114818803A (zh) * | 2022-04-25 | 2022-07-29 | 上海韶脑传感技术有限公司 | 基于神经元优化的单侧肢体患者运动想象脑电建模方法 |
Also Published As
Publication number | Publication date |
---|---|
US11200638B2 (en) | 2021-12-14 |
SG11202000062RA (en) | 2020-03-30 |
CN109308679B (zh) | 2022-08-30 |
CN109308679A (zh) | 2019-02-05 |
TWI749356B (zh) | 2021-12-11 |
US20200134778A1 (en) | 2020-04-30 |
JP6874168B2 (ja) | 2021-05-19 |
JP2020533660A (ja) | 2020-11-19 |
TW202009800A (zh) | 2020-03-01 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
WO2020034481A1 (zh) | 一种图像风格转换方及装置、设备、存储介质 | |
US20210150678A1 (en) | Very high-resolution image in-painting with neural networks | |
CN109949255B (zh) | 图像重建方法及设备 | |
EP3678059B1 (en) | Image processing method, image processing apparatus, and a neural network training method | |
WO2021109876A1 (zh) | 图像处理方法、装置、设备及存储介质 | |
JP7417640B2 (ja) | リアルタイム映像超高解像度 | |
US11704844B2 (en) | View synthesis robust to unconstrained image data | |
Cao et al. | Image Super-Resolution via Adaptive $\ell _ {p}(0< p< 1) $ Regularization and Sparse Representation | |
WO2023000895A1 (zh) | 图像风格转换方法、装置、电子设备和存储介质 | |
US11915383B2 (en) | Methods and systems for high definition image manipulation with neural networks | |
CN113256504A (zh) | 图像处理的方法和电子设备 | |
Huang et al. | Hybrid image enhancement with progressive laplacian enhancing unit | |
Purkait et al. | Image upscaling using multiple dictionaries of natural image patches | |
Jiang et al. | Fast and high quality image denoising via malleable convolution | |
CN114830168A (zh) | 图像重建方法、电子设备和计算机可读存储介质 | |
CN110428422B (zh) | 超像素采样网络 | |
Mikaeli et al. | Single-image super-resolution via patch-based and group-based local smoothness modeling | |
Li et al. | Wavenhancer: Unifying wavelet and transformer for image enhancement | |
KR102548407B1 (ko) | 뉴럴 네트워크가 포함된 상미분 방정식을 이용하는 점진적 이미지 해상도 향상 방법 및 장치와 이를 이용하는 이미지 초해상도 방법 | |
Xu et al. | An edge guided coarse-to-fine generative network for image outpainting | |
CN109447900A (zh) | 一种图像超分辨率重建方法及装置 | |
Tojo et al. | Image denoising using multi scaling aided double decker convolutional neural network | |
Perla et al. | Low Light Image Illumination Adjustment Using Fusion of MIRNet and Deep Illumination Curves | |
Li et al. | Convolutional Neural Network Combined with Half‐Quadratic Splitting Method for Image Restoration | |
Li et al. | Incorporating multiscale contextual loss for image style transfer |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
ENP | Entry into the national phase |
Ref document number: 2019569805 Country of ref document: JP Kind code of ref document: A |
|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 18930498 Country of ref document: EP Kind code of ref document: A1 |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
32PN | Ep: public notification in the ep bulletin as address of the adressee cannot be established |
Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 24.06.2021) |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 18930498 Country of ref document: EP Kind code of ref document: A1 |