CN107767328B - Migration method and system of any style and content generated based on small amount of samples - Google Patents
Migration method and system of any style and content generated based on small amount of samples Download PDFInfo
- Publication number
- CN107767328B CN107767328B CN201710957685.2A CN201710957685A CN107767328B CN 107767328 B CN107767328 B CN 107767328B CN 201710957685 A CN201710957685 A CN 201710957685A CN 107767328 B CN107767328 B CN 107767328B
- Authority
- CN
- China
- Prior art keywords
- style
- content
- image
- target image
- target
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 36
- 238000013508 migration Methods 0.000 title claims description 22
- 230000005012 migration Effects 0.000 title claims description 22
- 238000000605 extraction Methods 0.000 claims abstract description 65
- 238000013528 artificial neural network Methods 0.000 claims abstract description 20
- 238000012549 training Methods 0.000 claims description 37
- 230000006870 function Effects 0.000 claims description 15
- 238000013527 convolutional neural network Methods 0.000 claims description 8
- 238000004364 calculation method Methods 0.000 claims description 6
- 238000003062 neural network model Methods 0.000 claims description 4
- 230000010354 integration Effects 0.000 claims description 3
- 239000000284 extract Substances 0.000 abstract description 4
- 241001465754 Metazoa Species 0.000 description 5
- 230000008569 process Effects 0.000 description 3
- 238000011160 research Methods 0.000 description 3
- 241000282414 Homo sapiens Species 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 238000005070 sampling Methods 0.000 description 2
- 241000282412 Homo Species 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
Images
Classifications
-
- G06T3/04—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
Abstract
The invention provides a method and a system for migrating any style and content based on a small amount of sample generation, which comprises the following steps: style feature extraction: for the style reference image, extracting style characteristics of the image by using a depth convolution neural network; a content feature extraction step: extracting content characteristics of the image by using a depth convolution neural network for the content reference image; and combining style and content characteristics: combining the extracted style features and the content features through a bilinear model to obtain the features of the target image; a target image generation step: and generating the target image by the characteristics of the target image through a deconvolution neural network. The method and the device separate styles and contents, realize the image generation of any target content in any target style, can migrate to unseen styles and unseen contents, and extract the characteristics of the unseen styles and contents. The invention only needs a small amount of reference images, and the style characteristic and the content characteristic can be learned from a small amount of data.
Description
Technical Field
The invention relates to a method in the field of computer vision and image processing, in particular to a method and a system for migrating any style and content generated based on a small number of samples.
Background
In the field of style migration, initial research efforts were directed to progressively transform a noisy image into an image with target style target content in an iterative fashion based on a trained existing network, such as a VGG network. However, this method requires many iterations and is inefficient. Much of the research work that follows is directed to learning a model that can generate images with a certain style directly from a content image. The loss functions used are mostly perceptual loss functions, so that the generated image is an image of the target genre and the target content. However, the deep neural network models used in these works do not separate styles and contents, and the two factors of styles and contents are still fused together, so these networks can only migrate styles appearing in the training set, if migration needs to be performed on other unseen styles, the models need to be retrained, and a large number of training images are used in the process of retraining the models.
The current deep neural network model generally needs a large amount of training data for learning, and human beings have the ability to learn a certain concept from a small amount of samples or even a single sample, for example, when a child is taught to recognize a certain animal, the child can recognize and learn the certain animal according to one or more images of the certain animal, and can recognize and judge the certain animal when the child sees a new image of the certain animal again. In particular, humans also have the ability to generate from a small number of samples, encounter a new image and concept, understand its structure and expressive information, and imagine its deformation in the mind. In addition, the human can automatically fuse the images according to certain styles and certain content images, and imagine the style-transferred images formed by combining the two images.
With the gradual development of the deep neural network in the field of image processing, some algorithms for target recognition based on a small number of samples appear at present, and good effects are obtained. Unlike the generation of images, object recognition based on a small number of samples focuses more on extracting features of an object, and because the number of samples is small, more representative and distinctive features need to be extracted. In the current research, no method for performing style migration by using a small amount of samples is available.
Through retrieval, the chinese patent publication No. CN106651766A discloses an image style migration method based on a deep convolutional neural network, which implements style migration through image input, loss function training, stylization, image enhancement, and image refinement, but has the disadvantage that the above patent can only implement style migration that occurs in a training set, and does not have a style that has already occurred in the training set, and needs retraining, which is time-consuming on one hand and requires a large number of training samples for training on the other hand.
Disclosure of Invention
Aiming at the defects in the prior art, the invention aims to provide a method and a system for transferring any style and content generated based on a small amount of samples, each trained module can be transferred to unseen styles and content, and for unseen styles and content, a small amount of style reference images and content reference images can be used for generation.
According to a first aspect of the present invention, there is provided a migration method of arbitrary styles and contents generated based on a small number of samples, comprising:
style feature extraction: for the style reference image, extracting style characteristics of the image by using a depth convolution neural network;
a content feature extraction step: extracting content characteristics of the image by using a depth convolution neural network for the content reference image;
and combining style and content characteristics: combining the style characteristics obtained in the style characteristic extraction step with the content characteristics obtained in the content characteristic extraction step through a bilinear model to obtain the characteristics of the target image;
a target image generation step: and performing deconvolution on the characteristics of the target image obtained in the step of combining the style and the content characteristics to generate the target image.
The migration method of any style and content generated based on a small amount of samples can carry out end-to-end training.
Preferably, the style feature extraction step, wherein: and combining a small number of style reference images in the style reference set on the image channel dimension by using a deep convolutional neural network model, inputting the combined style reference images into a style feature extraction step, and learning a common part in the reference images, namely the style of the images to obtain style features. The style reference set is an image of different contents in the same style, and provides information of the style to be generated for the style characteristic extraction step.
Preferably, the content feature extraction step, wherein: and combining a small number of content reference images in the content reference set on the image channel dimension by using a deep convolutional neural network model, inputting the combined images into a content feature extraction step, and learning a common part in the reference images, namely the content of the images to obtain content features. The content reference set is images of the same content in different styles, and provides information of the content to be generated for the content feature extraction step.
Preferably, the genre and content characteristics are combined, wherein: the style and content features are combined by a bilinear model. A bilinear model is a two-factor model that has the mathematical property that one factor is fixed and the output of the model is linear to the other factor. It is flexible to separate or combine two factors.
Preferably, the target image generating step specifically includes: and inputting the characteristics of the target image obtained in the step of combining the style and the content characteristics into a deconvolution neural network to generate the target image.
According to a second aspect of the present invention, there is provided a style migration system based on a small number of sample generations, comprising:
a style feature extraction module: for the style reference image, extracting style characteristics of the image by using a depth convolution neural network;
a content feature extraction module: extracting content characteristics of the image by using a depth convolution neural network for the content reference image;
a genre and content feature integration module: combining the style characteristics obtained by the style characteristic extraction module with the content characteristics obtained by the content characteristic extraction module through a bilinear model to obtain the characteristics of a target image;
a target image generation module: and performing deconvolution on the characteristics of the target image obtained by the style and content characteristic combination module to generate the target image.
Compared with the prior art, the invention has the following beneficial effects:
the method respectively extracts the style features and the content features from the style reference images and the content reference images through style feature extraction and content feature extraction, separates the style and the content by utilizing the conditional correlation between the style and the content, and learns the representation modes of the style and the content features, thereby migrating each learned module to any unseen style and content. And then combining the extracted style and content features by using a bilinear model to generate the features of the target image, and further generating the target image by using a target image generation module.
The method only uses a small number of reference images, and the style reference set and the content reference set used in the training process only have a small number of samples, so that the method can extract style and content characteristics from a small number of samples, and only needs to migrate a small number of samples for unseen style and content.
Drawings
Other features, objects and advantages of the invention will become more apparent upon reading of the detailed description of non-limiting embodiments with reference to the following drawings:
FIG. 1 is a flow chart of a method according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of a system according to an embodiment of the present invention.
Detailed Description
The present invention will be described in detail with reference to specific examples. The following examples will assist those skilled in the art in further understanding the invention, but are not intended to limit the invention in any way. It should be noted that variations and modifications can be made by persons skilled in the art without departing from the spirit of the invention. All falling within the scope of the present invention.
As shown in fig. 1, which is a flowchart of an embodiment of a migration method of an arbitrary genre and content generated based on a small number of samples according to the present invention, the method processes a genre reference image and a content reference image into a genre feature and a content feature through a genre feature extraction step and a content feature extraction step, combines the genre feature and the content feature into a feature of a target image using a genre and content feature combination step, and finally generates the target image using a target image generation step.
The method and the device can separate the styles from the content by utilizing the conditional correlation of the styles and the content, and realize the image generation of any target content in any target style. Because the feature representation mode is learned, the trained style and content feature extraction module can be transferred to unseen styles and unseen contents, and can extract the features of the unseen styles and contents. In addition, the invention only needs a small amount of reference images, and the style characteristic and the content characteristic can be learned from a small amount of data.
Specifically, with reference to fig. 1, the method comprises the steps of:
style feature extraction: for the style reference image, extracting style characteristics of the image by using a depth convolution neural network;
a content feature extraction step: extracting content characteristics of the image by using a depth convolution neural network for the content reference image;
and combining style and content characteristics: combining the style characteristics obtained in the style characteristic extraction step with the content characteristics obtained in the content characteristic extraction step through a bilinear model to obtain the characteristics of the target image;
a target image generation step: and performing deconvolution on the characteristics of the target image obtained in the step of combining the style and the content characteristics to generate the target image.
Corresponding to the above method, referring to fig. 2, a style migration system based on a small number of sample generation includes:
a style feature extraction module: for the style reference image, extracting style characteristics of the image by using a depth convolution neural network;
a content feature extraction module: extracting content characteristics of the image by using a depth convolution neural network for the content reference image;
a genre and content feature integration module: combining the style characteristics obtained by the style characteristic extraction module with the content characteristics obtained by the content characteristic extraction module through a bilinear model to obtain the characteristics of a target image;
a target image generation module: and performing deconvolution on the characteristics of the target image obtained by the style and content characteristic combination module to generate the target image.
Specific implementations of the above steps and modules are described in detail below to facilitate understanding of the technical solutions of the present invention.
In some embodiments of the present invention, the style feature extracting step includes: and combining a small number of style reference images in the style reference set on the image channel dimension by using a deep convolutional neural network model, inputting the combined style reference images into a style feature extraction module, and learning a common part in the reference images, namely the style of the images to obtain style features. The style reference set is an image of different contents in the same style, and provides the style information to be generated for the style characteristic extraction step.
In some embodiments of the present invention, the content feature extraction step includes: and combining a small number of content reference images in the content reference set on the image channel dimension by using a deep convolutional neural network model, inputting the combined content reference images into a content feature extraction module, and learning a common part in the reference images, namely the content of the images to obtain content features. The content reference set is images of the same content in different genres, and provides information of the content to be generated to the content feature extraction step.
In some embodiments of the present invention, the style and content features are combined, wherein: the style and content features are combined by a bilinear model. A bilinear model is a two-factor model that has the mathematical property that one factor is fixed and the output of the model is linear to the other factor. It is flexible to separate or combine two factors.
In some embodiments of the present invention, the target image generating step specifically includes: and inputting the characteristics of the target image obtained in the step of combining the style and the content characteristics into a deconvolution neural network to generate the target image.
In a preferred embodiment of the present invention, the target image generating step may specifically include the following operations:
in the training phase, in each iteration, a training target image set D is first definedtI.e. the set of target images that it is desired to generate; for each target image Iij∈DtProviding a referenceImages to determine a target style S to be generatediAnd target content CjRandomly selecting k style reference images from the training set for each target image to form a style reference set RsiRandomly selecting k content reference images to form a content reference set RcjThen is followed by<Style reference image, content reference image, target image>Training in a triple form, wherein a style reference image and a content reference image are respectively input into a style feature extraction step and a content feature extraction step to obtain style features and content features, the obtained style features and content features are combined by a style and content feature combination module to obtain features of a target image, and the features are input into a target image generation module to generate the target image;
in the character style migration task: the generated target image is compared with the real target image and the following L1 loss function is calculated:
wherein, IijIs a true style of SiThe content is CjThe target image of (a) is displayed,
is generated with a style SiThe content is CjA target image;
considering the imbalance of the samples in each training batch, adding a corresponding weight to each target image in a loss function, wherein the weight comprises two aspects, namely the weight related to the size and the thickness of the image character, namely taking the reciprocal of the number of pixel points covered by the character as the weight, and the calculation formula of the weight related to the size and the thickness of the image character is as follows:
wherein N isijIs a target image IijThe number of pixels covered by the middle character,
is a target image IijThe more pixel points are covered, the larger or thicker the character is, and the lower the corresponding weight is;
the other is weight related to the blackness of the image characters, and the calculation method is as follows: calculating the average pixel value of pixel points covered by the characters for each target image, and then setting the loss weight in the form of a softmax function:
wherein mean isijIs a target image IijThe middle character covers the average pixel value of the pixel,
is a target image IijThe lower the average pixel value, the darker the character, the lower the corresponding weight;
therefore, the training targets of the migration method based on the arbitrary style and content generated by a small number of samples are as follows:
where D is the set of all training samples, DsiIs to remove IijAll other styles are SiSet of training samples of (D)cjIs to remove IijAll styles except CjIs a parameter of the convolutional neural network used in the method and a parameter of the bilinear model,
is the generated target image.
As shown in FIG. 2, the migration system is composed of a style feature extraction module, a content feature extraction module, a style and content feature combination module and a target image generation module, and the whole system framework can be trained end to end.
In the system framework of the embodiment shown in fig. 2, the genre reference image and the content reference image are first combined in the image channel dimension and then input to the genre feature extraction module and the content feature extraction module, respectively. The style feature extraction module and the content feature extraction module are both composed of a series of downsampling modules consisting of convolutional layers + batchnorm layers + leakyrelu layers, wherein the sizes of convolution kernels of the other convolutional layers are 3 × 3 except that the size of a convolution kernel of the first convolutional layer is 5 × 5. The number of channels at the output of each layer is indicated in fig. 2.
In the system framework of the embodiment shown in fig. 2, the style and content feature combination module combines the style feature and the content feature using a bilinear model, and the formula of the combination is as follows:
Fij＝SiWCj
wherein S isiRepresenting a style feature, is an R-dimensional vector, CjRepresenting the content features, is a B-dimensional vector, W is an R × K × B-dimensional tensor, FijIs the characteristic of the target image, is a K-dimensional vector with the representative style of SiThe content is CjThe characteristic of the image of (1). And when the style image is combined, combining the style characteristics obtained from the style reference set and the content characteristics obtained from the content reference set according to the formula to obtain the characteristics of the target image.
In the system framework of the embodiment shown in fig. 2, the target image generation module inputs the features of the target image obtained by the style and content feature combination module into the deconvolution neural network to generate the target image. The structure of the target image generation module is symmetrical to the style feature extraction module and the content feature extraction module, and the target image generation module is composed of an up-sampling module consisting of a series of deconvolution layers, a batchnorm layer and a relu layer, and is used for up-sampling the target image features to enable the generated image to be the same as the real target image in size. Wherein the convolution kernel sizes of the deconvolution layers are 3 × 3, except for the convolution kernel size of the last deconvolution layer being 5 × 5. The number of channels at the output of each layer is indicated in fig. 2.
In the training phase, in each iteration, a training target image set D is first definedtI.e. the set of target images that it is desired to generate. For each target image Iij∈DtProviding a reference image to determine a target style S to be generatediAnd target content CjRandomly selecting k style reference images from the training set for each target image to form a style reference set RsiRandomly selecting k content reference images to form a content reference set Rcj. Then is provided with<Style reference image, content reference image, target image>And training in a triple form, wherein the style reference image and the content reference image are respectively input into a style feature extraction module and a content feature extraction module to obtain style features and content features. The obtained style characteristic and the content characteristic are combined by a style and content characteristic combining module to obtain the characteristic of the target image, and the characteristic is input into a target image generating module to generate the target image.
In the character style migration task, the generated target image is compared with the real target image, and the following L1 loss function is calculated:
wherein, IijIs a real image of the object and is,
is the generated target image.
Furthermore, because the size, thickness and blackness difference of characters in a batch of target images selected in each iteration is likely to be large, some target images contain larger and thicker characters, and the loss caused by more pixel points covered by the characters is larger than that of other images, the method can focus on learning and optimizing larger and thicker fonts. In addition, there are some images with darker characters, and the method also favors learning over them, and does not yield well for lighter characters. Considering the imbalance of the samples in each training batch, adding corresponding weight to each target image in a loss function, wherein the weight comprises two aspects, namely the weight related to the size and the thickness of the image character, namely taking the reciprocal of the number of pixel points covered by the character as the weight. The calculation formula of the weights for the size and thickness of the image character is as follows:
wherein N isijIs a target image IijThe number of pixels covered by the middle character,
is a target image IijWeight on image character size and thickness. The more pixel points covered, the larger or coarser the character, the lower the corresponding weight.
The other is weight related to the blackness of the image characters, and the calculation method is as follows: calculating the average pixel value of pixel points covered by the characters for each target image, and then setting the loss weight in the form of a softmax function:
wherein mean isijIs a target image IijThe middle character covers the average pixel value of the pixel,
is a target image IijWeight of blackness of the image character. The lower the average pixel value, the darker the character, and the lower the corresponding weight.
Therefore, the training targets of the migration method based on the arbitrary style and content generated by a small number of samples are as follows:
where D is the set of all training samples, DsiIs to remove IijAll other styles are SiSet of training samples of (D)cjIs to remove IijAll styles except CjIs a parameter of the convolutional neural network used in the method and a parameter of the bilinear model,
is the generated target image.
In summary, the style and the content of the image can be separated by using the conditional correlation between the style and the content, the style characteristic and the content characteristic can be respectively extracted from the style reference image and the content reference image by the style and content characteristic extraction module, then the style characteristic and the content characteristic are combined by the bilinear model, and the obtained characteristic of the target image is input to the target image generation module to generate the target image. By setting the weighting L1 loss function, the imbalance of the selected target image per iteration can be improved.
In addition, the training sample of the triple of the < style reference image, the content reference image and the target image > designed in the invention enables the style and content feature extraction module to learn the feature representation mode, so the style and content feature extraction module obtained by training can be transferred to the unseen style and content, and the features of the unseen style and content can also be extracted. In addition, the style reference set and the content reference set used in training only contain a small number of reference images, so the method can realize the function of generating the target image with unseen style and unseen content by using a small number of reference samples.
The foregoing description of specific embodiments of the present invention has been presented. It is to be understood that the present invention is not limited to the specific embodiments described above, and that various changes and modifications may be made by one skilled in the art within the scope of the appended claims without departing from the spirit of the invention.
Claims (3)
1. A migration method of any style and content generated based on a small number of samples is characterized by comprising the following steps:
style feature extraction: extracting style characteristics of the image from the style reference image by using a first depth convolution neural network;
a content feature extraction step: extracting content features of the image from the content reference image by using a second deep convolution neural network;
and combining style and content characteristics: combining the style characteristics obtained in the style characteristic extraction step with the content characteristics obtained in the content characteristic extraction step through a bilinear model to obtain the characteristics of the target image;
a target image generation step: performing deconvolution on the characteristics of the target image obtained in the step of combining the style and the content characteristics to generate a target image;
the style and content features combining step, wherein: the bilinear model is a two-factor model, has the mathematical property that one factor is fixed, and the output of the model is linear to the other factor, so that the two factors can be flexibly separated or combined;
the bilinear model is combined, and the formula is as follows:
Fij＝SiWCj
wherein S isiRepresenting a style feature, is an R-dimensional vector, CjRepresenting the content features, is a B-dimensional vector, W is an R × K × B-dimensional tensor, FijIs the characteristic of the target image, is a K-dimensional vector with the representative style of SiThe content is CjThe characteristics of the image of (a); when the image is combined, combining the style characteristics obtained by the style reference set and the content characteristics obtained by the content reference set according to the formula to obtain the characteristics of the target image;
the style feature extraction step, wherein: combining a small number of style reference images in a style reference set on the image channel dimension by using a first depth convolution neural network model, inputting the style reference images together into a style feature extraction step, and learning a common part in the reference images, namely the style of the images to obtain style features; the style reference set is an image with the same style and different contents, and provides information of the style to be generated for the style characteristic extraction step;
the content feature extraction step, wherein: combining a small number of content reference images in the content reference set on the image channel dimension by using a second deep convolution neural network model, inputting the combined content reference images into the content feature extraction step, and learning common parts in the reference images, namely the content of the images to obtain content features; the content reference set is images of the same content and different styles, and provides information of the content to be generated for the content feature extraction step;
the target image generating step, wherein: inputting the characteristics of the target image obtained in the step of combining the style and the content characteristics into a deconvolution neural network to generate a target image, and comparing the generated target image with a real target image through a loss function; the target image generation step specifically comprises the following steps:
in the training phase, in each iteration, a training target image set D is first definedtI.e. the set of target images that it is desired to generate; for each target image Iij∈DtProviding a reference image to determine a target style S to be generatediAnd target content CjRandomly selecting k style reference images from the training set for each target image to form a style reference set RsiRandomly selecting k content reference images to form a content reference set RcjThen is followed by<Style reference image, content reference image, target image>And training in a triple form, wherein the style reference image and the content reference image are respectively input into the style characteristic extraction step and the content characteristic extraction step to obtain style characteristics and content characteristics.
2. The method for migrating arbitrary styles and contents generated based on a small number of samples as claimed in claim 1, wherein said target image generation step further has the following features:
in the character style migration task, the generated target image is compared with the real target image, and the following L1 loss function is calculated:
wherein, IijIs a true style of SiThe content is CjThe target image of (a) is displayed,
is generated with a style SiThe content is CjA target image;
considering the imbalance of the samples in each training batch, a corresponding weight is added to each target image in the loss function, and the weight comprises two aspects: the first is the weight of the size and thickness of the image character, that is, the reciprocal of the number of pixel points covered by the character is taken as the weight, and the calculation formula of the weight of the size and thickness of the image character is as follows:
wherein N isijIs a target image IijThe number of pixels covered by the middle character,
is a target image IijThe more pixel points are covered, the larger or thicker the character is, and the lower the corresponding weight is;
the other is weight related to the blackness of the image characters, and the calculation method is as follows: calculating the average pixel value of pixel points covered by the characters for each target image, and then setting the loss weight in the form of a softmax function:
wherein mean isijIs a target image IijThe middle character covers the average pixel value of the pixel,
is a target image IijThe lower the average pixel value, the darker the character, the lower the corresponding weight;
the training targets of the migration method of any style and content generated based on a small amount of samples are as follows:
where D is the set of all training samples, DsiIs to remove IijAll other styles are SiSet of training samples of (D)cjIs to remove IijAll styles except CjIs a parameter of the convolutional neural network used in the method and a parameter of the bilinear model,
is the generated target image.
3. A migration system of arbitrary styles and contents based on small sample generation using the method of any one of claims 1-2, characterized by:
a style feature extraction module: for the style reference image, extracting style characteristics of the image by using a depth convolution neural network;
a content feature extraction module: extracting content characteristics of the image by using a depth convolution neural network for the content reference image;
a genre and content feature integration module: combining the style characteristics obtained by the style characteristic extraction module with the content characteristics obtained by the content characteristic extraction module through a bilinear model to obtain the characteristics of a target image;
a target image generation module: and performing deconvolution on the characteristics of the target image obtained by the style and content characteristic combination module to generate the target image.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710957685.2A CN107767328B (en) | 2017-10-13 | 2017-10-13 | Migration method and system of any style and content generated based on small amount of samples |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710957685.2A CN107767328B (en) | 2017-10-13 | 2017-10-13 | Migration method and system of any style and content generated based on small amount of samples |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107767328A CN107767328A (en) | 2018-03-06 |
| CN107767328B true CN107767328B (en) | 2021-12-17 |
Family
ID=61269684
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710957685.2A Active CN107767328B (en) | 2017-10-13 | 2017-10-13 | Migration method and system of any style and content generated based on small amount of samples |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107767328B (en) |
Families Citing this family (40)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108733439A (en) * | 2018-03-26 | 2018-11-02 | 西安万像电子科技有限公司 | Image processing method and device |
| CN108510466A (en) * | 2018-03-27 | 2018-09-07 | 百度在线网络技术（北京）有限公司 | Method and apparatus for verifying face |
| CN108596830B (en) * | 2018-04-28 | 2022-04-22 | 国信优易数据股份有限公司 | Image style migration model training method and image style migration method |
| CN108734653B (en) * | 2018-05-07 | 2022-05-13 | 商汤集团有限公司 | Image style conversion method and device |
| CN108805803B (en) * | 2018-06-13 | 2020-03-13 | 衡阳师范学院 | Portrait style migration method based on semantic segmentation and deep convolution neural network |
| CN109165376B (en) * | 2018-06-28 | 2023-07-18 | 西交利物浦大学 | Style character generation method based on small amount of samples |
| CN108961349A (en) * | 2018-06-29 | 2018-12-07 | 广东工业大学 | A kind of generation method, device, equipment and the storage medium of stylization image |
| CN109242775B (en) * | 2018-09-03 | 2023-05-30 | 科大讯飞股份有限公司 | Attribute information migration method, device, equipment and readable storage medium |
| CN109285112A (en) | 2018-09-25 | 2019-01-29 | 京东方科技集团股份有限公司 | Image processing method neural network based, image processing apparatus |
| CN109086753B (en) * | 2018-10-08 | 2022-05-10 | 新疆大学 | Traffic sign identification method and device based on two-channel convolutional neural network |
| CN109636712B (en) * | 2018-12-07 | 2022-03-01 | 北京达佳互联信息技术有限公司 | Image style migration and data storage method and device and electronic equipment |
| US11354791B2 (en) * | 2018-12-19 | 2022-06-07 | General Electric Company | Methods and system for transforming medical images into different styled images with deep neural networks |
| CN109726760B (en) * | 2018-12-29 | 2021-04-16 | 驭势科技（北京）有限公司 | Method and device for training picture synthesis model |
| CN109766895A (en) | 2019-01-03 | 2019-05-17 | 京东方科技集团股份有限公司 | The training method and image Style Transfer method of convolutional neural networks for image Style Transfer |
| CN109829849B (en) * | 2019-01-29 | 2023-01-31 | 达闼机器人股份有限公司 | Training data generation method and device and terminal |
| CN111583165B (en) | 2019-02-19 | 2023-08-08 | 京东方科技集团股份有限公司 | Image processing method, device, equipment and storage medium |
| CN110148424B (en) * | 2019-05-08 | 2021-05-25 | 北京达佳互联信息技术有限公司 | Voice processing method and device, electronic equipment and storage medium |
| WO2020238120A1 (en) * | 2019-05-30 | 2020-12-03 | Guangdong Oppo Mobile Telecommunications Corp., Ltd. | System and method for single-modal or multi-modal style transfer and system for random stylization using the same |
| CN112115452A (en) * | 2019-06-20 | 2020-12-22 | 北京京东尚科信息技术有限公司 | Method and apparatus for generating a captcha image |
| CN110427948A (en) * | 2019-07-29 | 2019-11-08 | 杭州云深弘视智能科技有限公司 | The generation method and its system of character sample |
| CN110555896B (en) * | 2019-09-05 | 2022-12-09 | 腾讯科技（深圳）有限公司 | Image generation method and device and storage medium |
| CN111179215B (en) * | 2019-11-29 | 2022-09-13 | 北京航空航天大学合肥创新研究院 | Method and system for analyzing internal structure of cell based on cell bright field picture |
| CN111127309B (en) * | 2019-12-12 | 2023-08-11 | 杭州格像科技有限公司 | Portrait style migration model training method, portrait style migration method and device |
| CN111340964B (en) * | 2020-03-05 | 2023-03-24 | 长春中国光学科学技术馆 | 3D model image construction method based on transfer learning |
| CN113496460B (en) * | 2020-04-03 | 2024-03-22 | 北京大学 | Neural style migration method and system based on feature adjustment |
| CN111738968A (en) * | 2020-06-09 | 2020-10-02 | 北京三快在线科技有限公司 | Training method and device of image generation model and image generation method and device |
| CN111932439A (en) * | 2020-06-28 | 2020-11-13 | 深圳市捷顺科技实业股份有限公司 | Method and related device for generating face image of mask |
| CN112381707B (en) * | 2020-11-02 | 2023-06-20 | 腾讯科技（深圳）有限公司 | Image generation method, device, equipment and storage medium |
| CN112766079B (en) * | 2020-12-31 | 2023-05-26 | 北京航空航天大学 | Unsupervised image-to-image translation method based on content style separation |
| CN112733946B (en) * | 2021-01-14 | 2023-09-19 | 北京市商汤科技开发有限公司 | Training sample generation method and device, electronic equipment and storage medium |
| CN112884758B (en) * | 2021-03-12 | 2023-01-10 | 国网四川省电力公司电力科学研究院 | Defect insulator sample generation method and system based on style migration method |
| CN113012038B (en) * | 2021-03-19 | 2023-11-28 | 深圳市兴海物联科技有限公司 | Image style migration processing method, mobile terminal and cloud server |
| CN113516582B (en) * | 2021-04-12 | 2023-08-18 | 浙江大学 | Network model training method, device, computer equipment and storage medium for image style migration |
| CN113160042B (en) * | 2021-05-21 | 2023-02-17 | 北京邮电大学 | Image style migration model training method and device and electronic equipment |
| WO2023283781A1 (en) * | 2021-07-12 | 2023-01-19 | 西门子股份公司 | Method for generating microstructure data of material |
| CN113792526B (en) * | 2021-09-09 | 2024-02-09 | 北京百度网讯科技有限公司 | Training method of character generation model, character generation method, device, equipment and medium |
| CN115965840A (en) * | 2021-10-11 | 2023-04-14 | 北京字节跳动网络技术有限公司 | Image style migration and model training method, device, equipment and medium |
| CN113642566B (en) * | 2021-10-15 | 2021-12-21 | 南通宝田包装科技有限公司 | Medicine package design method based on artificial intelligence and big data |
| CN113642262B (en) * | 2021-10-15 | 2021-12-21 | 南通宝田包装科技有限公司 | Toothpaste package appearance auxiliary design method based on artificial intelligence |
| CN115861312B (en) * | 2023-02-24 | 2023-05-26 | 季华实验室 | OLED dry film defect detection method based on style migration positive sample generation |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106952224A (en) * | 2017-03-30 | 2017-07-14 | 电子科技大学 | A kind of image style transfer method based on convolutional neural networks |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8467601B2 (en) * | 2010-09-15 | 2013-06-18 | Kyran Daisy | Systems, methods, and media for creating multiple layers from an image |
-
2017
- 2017-10-13 CN CN201710957685.2A patent/CN107767328B/en active Active
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106952224A (en) * | 2017-03-30 | 2017-07-14 | 电子科技大学 | A kind of image style transfer method based on convolutional neural networks |
Non-Patent Citations (2)
| Title |
|---|
| Lin TY等.Bilinear cnn models for fine-grained visualrecognition.《Proceedings of the IEEE International Conference on Computer》.2016, * |
| Separating style and content;J B Tenenbaum等;《Advances in Neural Information Processing Systems》;19970131;第662-668页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107767328A (en) | 2018-03-06 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107767328B (en) | Migration method and system of any style and content generated based on small amount of samples | |
| Zuo et al. | Learning iteration-wise generalized shrinkage–thresholding operators for blind deconvolution | |
| WO2018153322A1 (en) | Key point detection method, neural network training method, apparatus and electronic device | |
| CN107529650B (en) | Closed loop detection method and device and computer equipment | |
| CN111242841B (en) | Image background style migration method based on semantic segmentation and deep learning | |
| US11748619B2 (en) | Image feature learning device, image feature learning method, image feature extraction device, image feature extraction method, and program | |
| US20180068430A1 (en) | Method and system for estimating blur kernel size | |
| CN110826596A (en) | Semantic segmentation method based on multi-scale deformable convolution | |
| CN107464217B (en) | Image processing method and device | |
| CN111783779B (en) | Image processing method, apparatus and computer readable storage medium | |
| CN112966691A (en) | Multi-scale text detection method and device based on semantic segmentation and electronic equipment | |
| KR101888647B1 (en) | Apparatus for classifying image and method for using the same | |
| CN113674140B (en) | Physical countermeasure sample generation method and system | |
| CN114549913B (en) | Semantic segmentation method and device, computer equipment and storage medium | |
| CN111553462A (en) | Class activation mapping method | |
| CN112365514A (en) | Semantic segmentation method based on improved PSPNet | |
| CN112184577A (en) | Single image defogging method based on multi-scale self-attention generation countermeasure network | |
| CN112329801B (en) | Convolutional neural network non-local information construction method | |
| CN111583100A (en) | Image processing method, image processing device, electronic equipment and storage medium | |
| CN112801104B (en) | Image pixel level pseudo label determination method and system based on semantic segmentation | |
| CN114048822A (en) | Attention mechanism feature fusion segmentation method for image | |
| CN108268890A (en) | A kind of hyperspectral image classification method | |
| CN110852199A (en) | Foreground extraction method based on double-frame coding and decoding model | |
| CN114266894A (en) | Image segmentation method and device, electronic equipment and storage medium | |
| CN111046915B (en) | Method for generating style characters |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20181012 Address after: 200063 701, 85 Lane 2077 lane, Guangfu West Road, Putuo District, Shanghai. Applicant after: Wang Yanfeng Applicant after: Zhang Ya Address before: 200240 No. 800, Dongchuan Road, Shanghai, Minhang District Applicant before: Shanghai Jiao Tong University |
|
| TA01 | Transfer of patent application right | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20181112 Address after: Room 387, Building 333, Hongqiao Road, Xuhui District, Shanghai 200030 Applicant after: SHANGHAI MEDIA INTELLIGENCE Co.,Ltd. Address before: 200063 701, 85 Lane 2077 lane, Guangfu West Road, Putuo District, Shanghai. Applicant before: Wang Yanfeng Applicant before: Zhang Ya |
|
| TA01 | Transfer of patent application right | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| PE01 | Entry into force of the registration of the contract for pledge of patent right |
Denomination of invention: A Transfer Method and System for Arbitrary Style and Content Based on a Small Sample Generation Effective date of registration: 20230329 Granted publication date: 20211217 Pledgee: The Bank of Shanghai branch Caohejing Limited by Share Ltd. Pledgor: SHANGHAI MEDIA INTELLIGENCE Co.,Ltd. Registration number: Y2023310000098 |
|
| PE01 | Entry into force of the registration of the contract for pledge of patent right |