CN113327194A - Image style migration method, device, equipment and storage medium - Google Patents

Image style migration method, device, equipment and storage medium Download PDF

Info

Publication number
CN113327194A
CN113327194A CN202110739068.1A CN202110739068A CN113327194A CN 113327194 A CN113327194 A CN 113327194A CN 202110739068 A CN202110739068 A CN 202110739068A CN 113327194 A CN113327194 A CN 113327194A
Authority
CN
China
Prior art keywords
data
style
characteristic
feature
content
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202110739068.1A
Other languages
Chinese (zh)
Inventor
林天威
李甫
何栋梁
李鑫
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Baidu Netcom Science and Technology Co Ltd
Original Assignee
Beijing Baidu Netcom Science and Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Baidu Netcom Science and Technology Co Ltd filed Critical Beijing Baidu Netcom Science and Technology Co Ltd
Priority to CN202110739068.1A priority Critical patent/CN113327194A/en
Publication of CN113327194A publication Critical patent/CN113327194A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G06T3/04—
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/22—Matching criteria, e.g. proximity measures
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/25—Fusion techniques
    • G06F18/253—Fusion techniques of extracted features
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T11/00—2D [Two Dimensional] image generation
    • G06T11/001—Texturing; Colouring; Generation of texture or colour
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/46—Descriptors for shape, contour or point-related descriptors, e.g. scale invariant feature transform [SIFT] or bags of words [BoW]; Salient regional features
    • G06V10/462—Salient features, e.g. scale invariant feature transforms [SIFT]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/56—Extraction of image or video features relating to colour

Abstract

The disclosure provides an image style migration method, an image style migration device and a storage medium, relates to the field of artificial intelligence, particularly relates to computer vision and deep learning technologies, and can be particularly used for image processing scenes. The specific implementation scheme is as follows: determining content characteristic coded data of the content graph under at least one associated coding scale of the target coding scale and style characteristic coded data of the style graph under the corresponding associated coding scale; determining statistical characteristic data of the style sheet under the target coding scale according to each content characteristic coded data and each style characteristic coded data; and generating a style migration diagram according to the statistical characteristic data and the content characteristic coded data under the target coding scale. According to the technology disclosed by the invention, the stylization effect of the stylized migration of the image is improved.

Description

Image style migration method, device, equipment and storage medium
Technical Field
The present disclosure relates to the field of artificial intelligence, and more particularly to computer vision and deep learning techniques, particularly useful in image processing scenarios.
Background
The image style migration refers to a technology of combining semantic content information of an image with style information of colors, textures and the like of other images, so that the image has a new visual style while keeping original image content information.
Disclosure of Invention
The disclosure provides an image style migration method, an image style migration device, an image style migration apparatus and a storage medium.
According to an aspect of the present disclosure, there is provided an image style migration method, including:
determining content characteristic coded data of the content graph under at least one associated coding scale of the target coding scale and style characteristic coded data of the style graph under the corresponding associated coding scale;
determining statistical characteristic data of the style graph under the target coding scale according to the content characteristic coded data and the style characteristic coded data;
and generating a style migration diagram according to the statistical characteristic data and the content characteristic coded data under the target coding scale.
According to another aspect of the present disclosure, there is also provided an image style migration apparatus including:
the characteristic coded data determining module is used for determining content characteristic coded data of the content graph under at least one associated coding scale of the target coding scale and style characteristic coded data of the style graph under the corresponding associated coding scale;
the statistical characteristic data determining module is used for determining statistical characteristic data of the style graph under the target coding scale according to the content characteristic coded data and the style characteristic coded data;
and the style migration result generation module is used for generating a style migration diagram according to the statistical characteristic data and the content characteristic coded data under the target coding scale.
According to another aspect of the present disclosure, there is also provided an electronic device including:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform a method of image style migration provided by any embodiment of the present disclosure.
According to another aspect of the present disclosure, there is also provided a non-transitory computer readable storage medium storing computer instructions for causing a computer to perform an image style migration method provided by any of the embodiments of the present disclosure.
According to another aspect of the present disclosure, there is also provided a computer program product comprising a computer program which, when executed by a processor, implements an image style migration method provided by any of the embodiments of the present disclosure.
According to the technology disclosed by the invention, the stylization effect of the stylized migration of the picture is improved.
It should be understood that the statements in this section do not necessarily identify key or critical features of the embodiments of the present disclosure, nor do they limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description.
Drawings
The drawings are included to provide a better understanding of the present solution and are not to be construed as limiting the present disclosure. Wherein:
FIG. 1 is a flowchart of an image style migration method provided by an embodiment of the present disclosure;
FIG. 2 is a flow chart of another image style migration method provided by the embodiments of the present disclosure;
FIG. 3 is a flow chart of another image style migration method provided by the embodiments of the present disclosure;
FIG. 4A is a block diagram of an image style migration model provided by an embodiment of the present disclosure;
fig. 4B is a network structure diagram of a normalized network according to an embodiment of the present disclosure;
FIG. 4C is a comparison graph of image style migration results provided by embodiments of the present disclosure;
FIG. 5 is a block diagram of an image style migration apparatus provided in an embodiment of the present disclosure;
FIG. 6 is a block diagram of an electronic device for implementing an image style migration method of an embodiment of the present disclosure.
Detailed Description
Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, in which various details of the embodiments of the disclosure are included to assist understanding, and which are to be considered as merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the present disclosure. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.
The image style migration method and the image style migration device provided by the embodiment of the disclosure are suitable for migrating information such as textures and edges in a style graph to a content graph in an image processing scene, so that a new visual style application scene is formed on the basis of keeping the structure of the content graph. Each image style migration method provided by the present disclosure may be executed by an image style migration apparatus, which may be implemented by software and/or hardware and may be configured in an electronic device.
For ease of understanding, the present disclosure first explains the relevant contents of the image style migration method.
Referring to fig. 1, an image style migration method includes:
s101, determining content feature coded data of the content graph under at least one associated coding scale of the target coding scale and style feature coded data of the style graph under the corresponding associated coding scale.
The content graph is a to-be-processed picture, and the style graph is used for representing the style of the picture to be finally presented by the content graph. The content graph and the style sheet can be stored in the local electronic device in advance, or other storage devices or cloud terminals associated with the electronic device, so that the content graph and the style sheet can be searched and obtained when needed.
The target coding scale is a coding scale to be determined by the statistical characteristic data, and can be determined by a technician according to needs or empirical values. It should be noted that the present disclosure does not limit the number of target coding scales at all, and may be at least part of coding scales for performing feature coding processing on the content graph and the style graph.
Wherein the associated coding scale may comprise at least one of: the adjacent coding scale of the target coding scale, the coding scale of which the resolution of the output content characteristic coding data is smaller than the resolution corresponding to the target coding scale, and the target coding scale.
Determining the content feature encoding data of the content map under at least one associated encoding scale of the target encoding scale, and the style feature encoding data of the style map under the corresponding encoding scale may be: processing the content graph through a coding network provided with at least one coding convolution module to obtain content characteristic coded data under at least one coding scale output by each coding convolution module; processing the style graph through a coding network provided with at least one coding convolution module to obtain style characteristic coded data under at least one coding scale output by each coding convolution module; and acquiring content characteristic coded data and style characteristic coded data of at least one associated coding scale of each target coding scale for subsequent determination of statistical characteristic data under the target coding scale.
It should be noted that, the present disclosure does not limit the specific network structure of the coding network, and only needs that the coding network can output the feature encoding data (that is, the content feature encoding data or the style feature encoding data) with different encoding scales. The feature coded data with different coding scales are characterized by different meanings, for example, the feature coded data may include shallow semantic features such as color and edge, and may also include deep semantic features such as texture.
In an optional embodiment, at least one of the encoding convolution modules set by the encoding network may be configured in a cascade manner, and configured to perform feature encoding on input data in sequence, and perform subsequent processing using an output feature encoding result as input data of a next encoding convolution module set in the cascade manner.
In another optional embodiment, at least one of the encoding convolution modules provided by the encoding network may also be provided in parallel for sequentially performing signature encoding on the input data. The data input by the different coding convolution modules is a down-sampling result of the original image. When the content characteristic coded data is determined, the original image is a content graph; and when determining the style characteristic coded data, the original image is the style sheet.
In one specific implementation, the encoding network may be implemented using a VGG (Visual Geometry Group) network.
It is understood that, in order to ensure data consistency of the content feature encoded data of the content map and the style feature encoded data of the style map, in an alternative embodiment, the same encoding network may be adopted, or the same encoding network may be adopted to perform encoding processing on the content map and the style map respectively, so as to obtain the feature encoded data set in at least one encoding scale. The feature coded data group under the same coding scale comprises content feature coded data generated under the coding scale and style feature coded data generated under the coding scale.
In a specific implementation manner, feature coding can be performed on the content graph and the style graph under at least one coding scale to obtain content feature coded data and style feature coded data under each coding scale; aiming at each target coding scale, taking each content characteristic coded data of the content characteristic coded data corresponding to the target coding scale with the resolution smaller than the target coding scale as the content characteristic coded data under the associated coding scale of the target coding scale; and taking each style coding feature data of the style feature coding data corresponding to the target coding scale with the resolution smaller than the target coding scale as the style feature coding data under the associated coding scale of the target coding scale.
It can be understood that the content characteristic coded data and the style characteristic coded data of the associated coding scale of the target coding scale are determined by the method and serve as a determination basis of the subsequent statistical characteristic data, so that the richness and the comprehensiveness of semantic information used in the statistical characteristic data determination process are improved, and the accuracy of the statistical characteristic data determination result under the target coding scale is improved.
And S102, determining statistical characteristic data of the style sheet under the target coding scale according to the content characteristic coded data and the style characteristic coded data.
The statistical feature data can numerically quantify the stylized characteristics of the stylized graph from an overall perspective.
Optionally, the statistical feature data may include mean feature data for reflecting a central tendency of the style feature encoding data of the style sheet at feature values of the feature points.
Optionally, the statistical characteristic data may include variance characteristic data or standard deviation characteristic data, which is used to reflect fluctuation of the characteristic value of the style characteristic coded data of the style sheet at each characteristic point.
It can be understood that when the statistical characteristic data of the style sheet is determined according to the style characteristic encoded data, the content characteristic encoded data of the content sheet is introduced, so that the characteristics of the content structure of the content sheet can be considered in the process of generating the statistical characteristic data of the style sheet, and the situation that the generated statistical characteristic data does not have the pertinence of the content sheet due to the fact that the machine uses the style sheet to determine the self statistical characteristic data is avoided, so that the local stylization degree is not enough or the content image distortion occurs in the finally generated stylization result, and the stylization effect of the image is influenced.
When the statistical characteristic data of the style graph under the target coding scale is determined, the characteristic coding data under at least one associated coding scale is introduced, so that the richness and the comprehensiveness of the used data can be improved, the accuracy of the determination result of the statistical characteristic data is improved, and the stylization effect of the content graph is improved.
And S103, generating a style transition diagram according to the statistical characteristic data and the content characteristic coded data under the target coding scale.
Exemplarily, the style migration characteristic data under the target coding scale can be generated according to the statistical characteristic data and the content characteristic coded data under the target coding scale; and generating a style migration diagram according to the style migration characteristic data under at least one target coding scale.
According to the method and the device, the style migration diagram is carried out through style migration characteristic data under at least one target coding scale, so that semantic fusion information under associated coding scales corresponding to a plurality of target coding scales can be comprehensively considered in the process of generating the style migration diagram, introduction of noise is reduced, distortion of a content image structure and appearance of local dirty textures are avoided, and the stylization effect of the style migration diagram is improved.
In an optional embodiment, if the statistical feature data includes mean feature data, feature summation may be performed according to the mean feature data and the content feature encoding data of the content map to obtain style migration feature data.
For example, if the average feature data have different sizes, the content feature encoding data of the content graph and the average feature data may be resized in an upsampling or downsampling manner, so that the sizes of the content feature encoding data and the average feature data are consistent, and a foundation is laid for stylized migration of the content graph.
In another alternative embodiment, the statistical signature data may include mean signature data and standard deviation signature data; correspondingly, processing the content characteristic coded data of the content graph according to the standard deviation characteristic data to generate migration volume data; and performing characteristic summation on the migration volume data and the mean characteristic to obtain style migration characteristic data.
In yet another alternative embodiment, the statistical signature data may include mean signature data and variance signature data; correspondingly, standard deviation characteristic data are determined according to the variance characteristic data; processing the content characteristic coded data of the content graph according to the standard deviation characteristic data to generate migration volume data; and performing characteristic summation on the migration volume data and the mean characteristic to obtain style migration characteristic data.
For example, the style migration diagram is generated according to the style migration feature data under at least one target coding scale, and the style migration diagram may be generated by feature decoding of the style migration feature data under at least one target coding scale.
According to the method and the device, the content feature coded data of the target graph under at least one associated coding scale of the target coding scale and the style feature coded data of the style graph under the corresponding associated coding scale are introduced, the statistical feature data of the style graph under the target coding scale are generated, the characteristics of the content structure of the content graph can be considered in the statistical feature data generation process of the style graph, the generated statistical feature data have the pertinence of the content graph, and the situations that the local stylization degree is insufficient or the content image is distorted in the stylized result are avoided. In addition, the content characteristic coded data and the style characteristic coded data of at least one associated coding scale are adopted, and the richness and the comprehensiveness of semantic information carried by the used characteristic coded data are improved, so that the accuracy of the determination result of the statistical characteristic data is improved, and the effect of stylized migration results is improved.
On the basis of the above technical solutions, the present disclosure also provides an alternative embodiment of the method for implementing image style migration. In the embodiment, the generation process of the statistical characteristic data of the style sheet is optimized and improved.
Referring to fig. 2, an image style migration method includes:
s201, determining content feature coded data of the content graph under at least one associated coding scale of the target coding scale and style feature coded data of the style graph under the corresponding associated coding scale.
And S202, performing feature fusion on each content feature coded data to obtain content feature fusion data.
Because the resolution of the content feature encoded data under at least one associated encoding scale is different, that is, the size of each content feature encoded data is different, in order to facilitate data processing, it is necessary to firstly perform size normalization on each content feature encoded data, so as to perform feature fusion on the content feature encoded data after each size normalization.
In an optional embodiment, the content feature encoded data may be adjusted to a target size, and the adjusted content feature encoded data may be fused according to a channel to obtain content feature fused data.
The target size may be the size of any content feature encoded data, or may be set manually by a technician as needed or according to empirical values.
The larger the size of the content feature coded data is, the more shallow semantic information is carried; the smaller the size of the content feature coded data is, the more deep semantic information is carried. Therefore, the target size can be set according to actual requirements. Alternatively, the target size is determined by a number of experiments during the model training phase.
In an optional embodiment, each content feature coded data is used as content feature data to be adjusted, and if the size of the content feature data to be adjusted is smaller than a target size, the content feature data to be adjusted is subjected to up-sampling processing so as to adjust the content feature data to be adjusted to the target size; if the size of the content characteristic data to be adjusted is larger than the target size, downsampling the content characteristic data to be adjusted so as to adjust the content characteristic data to be adjusted to the target size; and if the size of the content characteristic data to be adjusted is the target size, not processing the content characteristic data to be adjusted.
It can be understood that the content feature fusion data is performed by adjusting the content feature coded data to the target size and fusing the adjusted content feature coded data according to the channels, so that the content feature fusion data of each channel can simultaneously carry semantic information of different dimensions, the richness and the comprehensiveness of the content feature fusion data are improved, the accuracy of the determined statistical feature data is improved, and the style migration effect of the content graph is improved.
After the content feature coded data are up-sampled, the data operation amount in the subsequent statistical feature data determination process is obviously increased, so that the calculation performance requirement on the electronic equipment executing the image style migration method is increased, and the calculation efficiency in the image style migration process is reduced to a certain extent. In order to avoid the above situation, in an alternative embodiment, the minimum size of the content feature encoded data at each associated encoding scale may also be directly used as the target size.
It can be understood that the minimum size of the content feature encoded data under the associated encoding scale is selected as a target size, and the size of the content feature encoded data under other associated encoding scales is adjusted subsequently in a downsampling mode, so that the size of the content feature fusion data is reduced, the data operation amount in the process of determining the statistical feature data is further reduced, the calculation performance requirement on the electronic equipment executing the image style migration method is reduced, and the calculation efficiency during the image style migration is improved to a certain extent.
And S203, performing feature fusion on each style feature coded data to obtain style feature fusion data.
Because the resolution of the style characteristic coded data under at least one associated coding scale is different, that is, the size of each style characteristic coded data is different, in order to facilitate data processing, it is necessary to firstly normalize the size of each style characteristic coded data, thereby performing feature fusion on each size-normalized style characteristic coded data.
In an optional embodiment, each style feature coded data may be adjusted to a target size, and the adjusted style feature coded data may be fused according to a channel to obtain style feature fused data.
The target size may be the size of any style feature encoded data, or may be set manually by a technician as needed or according to empirical values.
The bigger the size of the style characteristic coded data is, the more shallow semantic information is carried; the smaller the size of the style characteristic coded data is, the more deep semantic information is carried. Therefore, the target size can be set according to actual requirements. Alternatively, the target size is determined by a number of experiments during the model training phase.
In an optional embodiment, each style feature coded data is used as style feature data to be adjusted, and if the size of the style feature data to be adjusted is smaller than a target size, the style feature data to be adjusted is subjected to up-sampling processing so as to adjust the style feature data to be adjusted to the target size; if the size of the style characteristic data to be adjusted is larger than the target size, downsampling the style characteristic data to be adjusted so as to adjust the style characteristic data to be adjusted to the target size; and if the size of the style characteristic data to be adjusted is the target size, the style characteristic data to be adjusted is not processed.
It can be understood that the style feature fusion data is performed by adjusting the style feature coded data to a target size and fusing the adjusted style feature coded data according to the channels, so that the style feature fusion data of each channel can simultaneously carry semantic information of different dimensions, the richness and the comprehensiveness of the style feature fusion data are improved, the accuracy of the determined statistical feature data is improved, and the style migration effect of the content map is improved.
After the style characteristic coded data are up-sampled, the data operation amount in the subsequent statistical characteristic data determination process is obviously increased, so that the calculation performance requirement on the electronic equipment executing the image style migration method is increased, and the calculation efficiency in the image style migration process is reduced to a certain extent. In order to avoid the above situation, in an alternative embodiment, the minimum size of the style characteristic encoded data at each associated encoding scale may also be directly used as the target size.
It can be understood that the minimum size of the style feature coded data under the associated coding scale is selected as a target size, and the size of the style feature coded data under other associated coding scales is adjusted subsequently in a down-sampling mode, so that the size of the style feature fusion data is reduced, the data operation amount in the process of determining the statistical feature data is further reduced, the calculation performance requirement on the electronic equipment executing the image style migration method is reduced, and the calculation efficiency during the image style migration is improved to a certain extent.
It should be noted that S202 and S203 may be executed sequentially, may also be executed in parallel, or may be executed in a crossed manner, and the specific execution order of S202 and S203 is not limited in this disclosure.
It can be understood that, in order to simplify the network complexity, the same network structure is usually adopted to perform feature coding processing on the content graph and the style graph respectively, so as to obtain a feature coded data set, including content feature coded data and style feature coded data. In addition, in order to simplify the calculation, it is sufficient that only one target size is determined for the content feature encoded data or the target feature encoded data, and data processing is performed using the same target size when the content feature fusion data and the style feature fusion data are determined.
And S204, determining statistical characteristic data of the style sheet under the target coding scale according to the content characteristic fusion data and the style characteristic fusion data.
And S205, generating a style transition diagram according to the statistical characteristic data and the content characteristic coded data of the content diagram.
The method and the device have the advantages that the determination operation of the statistical characteristic data is refined into the characteristic fusion of the content characteristic coded data under each relevant coding scale to obtain the content characteristic fusion data, and the characteristic fusion of the style characteristic coded data of each dimension is carried out to obtain the style characteristic fusion data, so that the content characteristic fusion data and the style characteristic fusion data respectively contain semantic information under different coding scales, the richness and the comprehensiveness of the carried semantic information are improved, the statistical characteristic data is determined according to the content characteristic fusion data and the style characteristic fusion data, the accuracy of the determined statistical characteristic data is improved, and the stylized effect of the style migration diagram is improved.
On the basis of the above technical solutions, the present disclosure also provides another alternative embodiment of the method for implementing image style migration. In the embodiment, the specific generation mechanism of the statistical characteristic data is optimized and improved. In the detailed part of the embodiments of the present disclosure, reference may be made to the foregoing embodiments, which are not described herein again.
Referring to fig. 3, an image style migration method includes:
s301, determining content feature coded data of the content graph under at least one associated coding scale of the target coding scale and style feature coded data of the style graph corresponding to the associated coding scale.
And S302, performing feature fusion on the content feature coded data to obtain content feature fusion data.
And S303, performing feature fusion on each style feature coded data to obtain style feature fusion data.
And S304, determining similarity data according to the content feature fusion data and the style feature fusion data.
The similarity data is used for representing the relevance between each feature point of the content feature fusion data and each feature point of the style feature fusion data.
Exemplarily, the similarity between each feature point in the content feature fusion data and each feature point in the style feature fusion data is determined, and similarity data is constructed according to each similarity.
It can be understood that the similarity data is determined through the content feature fusion data and the style feature fusion data comprising the multi-scale semantic information, so that the determined similarity data can refer to richer semantic information, loss of important semantic information is avoided, and introduction of noise is reduced, thereby being beneficial to improving the accuracy of the statistical feature data determination result, further avoiding distortion of a content image structure and occurrence of a local dirty texture phenomenon or occurrence of a condition of weak local stylization degree, and laying a foundation for improving the stylization effect of the style migration diagram.
S305, generating mean characteristic data of the style sheet under the target coding scale according to the similarity data and the style characteristic coding data of the style sheet.
And S306, generating standard deviation characteristic data or variance characteristic data of the style sheet under the target coding scale according to the similarity data, the mean characteristic data and the style characteristic coded data under the target coding scale.
It can be understood that the standard deviation feature data or the variance feature data is determined by introducing the content feature coded data and the style feature coded data of at least one associated coding scale, so that the determined standard deviation feature data or the variance feature data can comprehensively consider the feature coded data of the style graph and the content graph, and the determined standard deviation feature data or the variance feature data is more targeted by the content graph. Meanwhile, by using the feature encoding data under at least one associated encoding scale, the standard deviation feature data or the variance feature data can give consideration to the features of shallow semantic information and deep semantic information, so that the final determination result is more accurate.
And S307, generating a style transition diagram according to the mean characteristic data under the target coding scale, the content characteristic coded data under the target coding scale, and the standard deviation characteristic data or the variance characteristic data.
For the selection of the content feature encoding data of the content map, reference may be made to the description of the foregoing embodiments, which is not repeated herein.
Illustratively, migration amount data under the target coding scale can be generated according to standard deviation characteristic data or variance characteristic data under the target coding scale and content characteristic coded data under the target coding scale; generating style migration characteristic data under the target coding scale according to the migration volume data and the mean characteristic data under the target coding scale; and generating a style migration diagram according to the style migration characteristic data under at least one target coding scale.
The migration quantity data is used for representing the characteristic values of different characteristic points of the content characteristic coded data of the content graph in the stylized migration process of the image and the differentiation change condition in the stylized migration process.
It can be understood that the generation process of the style migration result is refined into data according to standard deviation feature data or variance feature data and content feature encoding data of the content map to generate migration volume data, and the style migration result is generated according to the migration volume data and the mean feature data, so that the adaptive normalization of the content map can be realized, the stylization process of the content map is more locally targeted, and the stylized migration effect of the content map is improved.
The determination operation of the statistical characteristic data under the target coding scale is refined into fusion data according to the content characteristic and the style characteristic, and the similarity data is determined; generating mean characteristic data of the style sheet under a target coding scale according to the similarity data and the style characteristic coded data of the style sheet; according to the similarity data, the mean characteristic data and the style characteristic coded data of the style sheet, standard deviation characteristic data or variance characteristic data of the style sheet under the target coding scale are generated, and a determination mechanism of statistical characteristic data is perfected. Meanwhile, similarity data are determined based on the multi-scale content feature fusion data and the style feature fusion data, introduction of noise data is reduced, and therefore image stylized migration processing is performed based on the statistical feature data determined by the similarity data, distortion of a content graph structure and appearance of local dirty textures and the like can be avoided, and stylized migration effects are improved.
On the basis of the technical schemes, the disclosure also provides a preferred embodiment of the image style migration method.
Referring to fig. 4A, a block diagram of an image style migration model is shown, which includes: an encoding network 10, at least one normalizing network 20 and a decoding network 30.
Illustratively, the encoding network 10 may be implemented using a VGG network. The present disclosure does not set any limit to the specific network structure of the decoding network 30.
Using coding network 10 to respectively match content graph ICAnd chart ISCoding to obtain content feature coded data F with at least one dimensionC iAnd style feature encoded data Fs i(ii) a Where i is 1,2, …, n, n is the number of feature coding layers in the coding network. Encoding data F for content features for each normalized network 20C xAnd style feature encoded data Fs xPerforming self-adaptive normalization processing to obtain style migration characteristic data FCS x(ii) a Wherein x is 2,3, … n. It should be noted that x may select a part of data in 2-n according to actual requirements. Migrating feature data F for each style using decoding network 30CS xDecoding to obtain style transition diagram ICS。
Further, in the model training phase, the image style migration network further includes a loss function construction network 40 for generating a loss function, so as to optimally adjust the network parameters of the at least one normalization network 20 and the decoding network 30 through the generated loss function.
Illustratively, the loss function construction network 40 includes a feature extraction module 41 for respectively extracting the content graph ICStyle sheet ISAnd style migration diagram ICSRespectively extracting the features to obtain the content graph features FCStyle sheet feature FSAnd style migration graph feature FCS(ii) a From content graph feature FCAnd style migration graph feature FCSBuilding a content loss LC(ii) a From style sheet feature FSAnd style migration graph feature FCSBuilding a style loss LS(ii) a Generating including content loss LCAnd style loss LSTo optimize the network parameters of each of the normalized network 20 and the decoded network 30.
In an alternative embodiment, the feature extraction module 41 may be implemented using a VGG network.
Encoding data F with content characteristics output from the x-th layer characteristic encoding layerC xAnd style feature encoded data FS xThe adaptive normalization process is performed for example,
the data processing mechanism of the normalized network will be described in detail with reference to the structure diagram of the normalized network shown in fig. 4B.
Content feature encoding data F respectively output to layer 1 to layer x feature encoding layersC 1，FC 2,…,FC xSplicing and fusing according to channels to obtain content feature fusion data FC 1:x(ii) a Fusing data F for content featuresC 1:xAnd sequentially carrying out characteristic normalization processing and characteristic coding processing to obtain query data Q. Style characteristic coding data F respectively output to layer 1 to layer x characteristic coding layersS 1，FS 2,…,FS xObtaining style characteristic fusion data F after splicing and fusing according to channelsS 1:x(ii) a Fusing data F to style characteristicsS 1:xAnd sequentially carrying out feature normalization processing and feature coding processing to obtain keyword data K. And generating similarity data A according to the query data Q and the keyword data K. Style characteristic coding data F output to x layer characteristic coding layerS xAnd (5) carrying out coding processing to obtain result data V. Generating mean characteristic data M according to the similarity data A and the result data V; generating standard deviation feature data S according to the similarity data A, the result data V and the mean feature data M; according to the mean characteristic data M and the standard deviation characteristic data S, the content characteristic coded data F output by the x-th layer characteristic coded layerC xCarrying out self-adaptive normalization to obtain style migration characteristic data FCS x。
By way of example, the query data may be generated using the following formula:
Figure BDA0003142486210000131
wherein Q is query data; fC 1:xFusing data for the content features; norm () is a feature normalization function; f () is a preset encoding function.
Illustratively, the keyword data may be generated using the following formula:
Figure BDA0003142486210000132
wherein K is keyword data; fS 1:xFusing data for style features; norm () is a feature normalization function; g () is a preset encoding function. g () and f () may be the same or different, and are not limited by this disclosure.
Illustratively, the following formula may be employed to generate the resulting data:
Figure BDA0003142486210000133
wherein V is result data; fS xStyle characteristic encoding data output by the x layer characteristic encoding layer; h () is a preset encoding function. h () may be the same as or different from g (), f (), which is not limited by this disclosure.
For example, the similarity data may be generated using the following formula:
Figure BDA0003142486210000134
wherein A is similarity data; softmax () is an activation function that can be set by a technician as desired.
For example, the following equations may be used to generate the mean feature data and the standard deviation feature data:
Figure BDA0003142486210000141
Figure BDA0003142486210000142
wherein M is mean characteristic data, and S is standard deviation characteristic data.
For example, the following formula can be adopted to encode the content characteristics of the layer x characteristic outputSyndrome coded data FC xAnd performing adaptive normalization processing:
Figure BDA0003142486210000143
wherein, FCS xAnd migrating the feature data for the style corresponding to the x-th layer feature coding layer.
In a specific implementation manner, as can be seen from the result comparison diagram in fig. 4C, the stylized migration performed by using the technical solution of the embodiment of the present disclosure is more robust than the prior art that the stylized migration is performed based on AdaIN (Adaptive Instance Normalization), SANet (Structure-Aware Network), mask (manual Alignment for semantic Aligned Style conversion based on popular Alignment algorithm), Linear (Linear transformation for fast arbitrary Style conversion), MCCNet (Multi-Network Correlation Network), Avatar-Net (Multi-scale-Style zero transformation, Multi-Channel Correlation Network based on feature transformation), and zero-scale transformation of the local texture transformation based on the feature transformation, and the like, and the texture processing is performed without generating distortion, and the image processing is more robust than the prior art that the texture processing is performed based on the Adaptive Instance Normalization, and the texture processing is performed without generating strong distortion, the overall stylization effect is better.
On the basis of the above technical solutions, the present disclosure also provides an optional embodiment of an execution device for implementing the above image style migration methods, where the execution device may be implemented by software and/or hardware, and is specifically configured in an electronic device.
Referring to fig. 5, an image style migration apparatus 500 includes: a characteristic encoding data determining module 501, a statistical characteristic data determining module 502 and a lattice migration diagram generating module 503. Wherein the content of the first and second substances,
the feature coded data determining module 501 is configured to determine content feature coded data of the content map in at least one associated coding scale of the target coding scale, and style feature coded data of the style map in a corresponding associated coding scale;
a statistical characteristic data determining module 502, configured to determine statistical characteristic data of the style sheet in the target coding scale according to each content characteristic coded data and each style characteristic coded data;
and the style migration diagram generating module 503 is configured to generate a style migration result according to the statistical feature data and the content feature encoded data under the target encoding scale.
According to the method and the device, the content feature coded data of the target graph under at least one associated coding scale of the target coding scale and the style feature coded data of the style graph under the corresponding associated coding scale are introduced, the statistical feature data of the style graph under the target coding scale are generated, the characteristics of the content structure of the content graph can be considered in the statistical feature data generation process of the style graph, the generated statistical feature data have the pertinence of the content graph, and the situations that the local stylization degree is insufficient or the content image is distorted in the stylized result are avoided. In addition, the content characteristic coded data and the style characteristic coded data of at least one associated coding scale are adopted, and the richness and the comprehensiveness of semantic information carried by the used characteristic coded data are improved, so that the accuracy of the determination result of the statistical characteristic data is improved, and the effect of stylized migration results is improved.
In an alternative embodiment, the statistical characteristic data determination module 502 includes:
a content feature fusion data obtaining unit, configured to perform feature fusion on each content feature encoded data to obtain content feature fusion data;
the style feature fusion data obtaining unit is used for carrying out feature fusion on each style feature coded data to obtain style feature fusion data;
and the statistical characteristic data determining unit is used for determining statistical characteristic data of the style graph under the target coding scale according to the content characteristic fusion data and the style characteristic fusion data.
In an alternative embodiment, the statistical characteristic data determination unit comprises:
the similarity data determination subunit is used for determining similarity data according to the content feature fusion data and the style feature fusion data;
the mean characteristic data determining subunit is used for generating mean characteristic data of the style sheet under the target coding scale according to the similarity data and the style characteristic coding data of the style sheet;
the standard deviation characteristic data determining subunit is used for generating standard deviation characteristic data of the style sheet under the target coding scale according to the similarity data, the mean characteristic data and the style characteristic coded data of the style sheet; alternatively, the first and second electrodes may be,
and the variance characteristic data determining subunit is used for generating variance characteristic data of the style sheet under the target coding scale according to the similarity data, the mean characteristic data and the style characteristic coding data of the style sheet.
In an alternative embodiment, the content feature fusion data obtaining unit includes:
the content feature fusion data obtaining subunit is used for adjusting each content feature coded data to a target size and fusing each adjusted content feature coded data according to a channel to obtain content feature fusion data;
the style characteristic fusion data obtaining unit comprises:
and the style characteristic fusion data obtaining subunit is used for adjusting the style characteristic coded data to a target size, and fusing the adjusted style characteristic coded data according to the channels to obtain style characteristic fusion data.
In an optional embodiment, the content feature fusion data obtaining unit further includes:
a target size determining subunit, configured to use the minimum size of the content feature encoded data in each associated encoding scale as a target size;
alternatively, the first and second electrodes may be,
the style characteristic fusion data obtaining unit further comprises:
and the target size determining subunit is used for taking the minimum size of the style characteristic coded data under each associated coding scale as the target size. The size of the feature encoded data corresponding to the maximum layer number is set as a target size. In an alternative embodiment, the style migration diagram generation module 503 includes:
the style migration characteristic data generation unit is used for generating style migration characteristic data under the target coding scale according to the statistical characteristic data and the content characteristic coded data under the target coding scale;
and the style migration diagram generating unit is used for generating a style migration diagram according to the style migration characteristic data under at least one target coding scale.
In an alternative embodiment, the style migration feature data generation unit includes:
the migration volume data generation subunit is used for generating migration volume data under the target coding scale according to the standard deviation characteristic data or the variance characteristic data under the target coding scale and the content characteristic coded data under the target coding scale;
and the style migration result generation subunit is used for generating style migration characteristic data under the target coding scale according to the migration volume data under the target coding scale and the mean characteristic data.
In an alternative embodiment, the feature coded data determination module 501 includes:
the characteristic coding unit is used for carrying out characteristic coding on the content graph and the style graph under at least one coding scale to obtain content characteristic data and style characteristic coded data under each coding scale;
the content characteristic coded data determining unit is used for regarding each target coding scale, and regarding each content characteristic coded data of the content characteristic coded data corresponding to the target coding scale with the resolution smaller than the target coding scale as the content characteristic coded data under the relevant coding scale of the target coding scale;
and the style characteristic coded data determining unit is used for taking each style coded characteristic data of the style characteristic coded data corresponding to the target coding scale with the resolution smaller than the target coding scale as the style characteristic coded data under the relevant coding scale of the target coding scale. The image style migration device can execute the image style migration method provided by any embodiment of the disclosure, and has the corresponding functional modules and beneficial effects of executing the image style migration method.
In the technical scheme of the disclosure, the related content diagram and the style diagram are acquired, stored, applied and the like, which all accord with the regulations of related laws and regulations and do not violate the good customs of the public order.
The present disclosure also provides an electronic device, a readable storage medium, and a computer program product according to embodiments of the present disclosure.
FIG. 6 illustrates a schematic block diagram of an example electronic device 600 that can be used to implement embodiments of the present disclosure. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the disclosure described and/or claimed herein.
As shown in fig. 6, the apparatus 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a Read Only Memory (ROM)602 or a computer program loaded from a storage unit 608 into a Random Access Memory (RAM) 603. In the RAM603, various programs and data required for the operation of the device 600 can also be stored. The calculation unit 601, the ROM602, and the RAM603 are connected to each other via a bus 604. An input/output (I/O) interface 605 is also connected to bus 604.
A number of components in the device 600 are connected to the I/O interface 605, including: an input unit 606 such as a keyboard, a mouse, or the like; an output unit 607 such as various types of displays, speakers, and the like; a storage unit 608, such as a magnetic disk, optical disk, or the like; and a communication unit 609 such as a network card, modem, wireless communication transceiver, etc. The communication unit 609 allows the device 600 to exchange information/data with other devices via a computer network such as the internet and/or various telecommunication networks.
The computing unit 601 may be a variety of general and/or special purpose processing components having processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various dedicated Artificial Intelligence (AI) computing chips, various computing units running machine learning model algorithms, a Digital Signal Processor (DSP), and any suitable processor, controller, microcontroller, and so forth. The calculation unit 601 executes the respective methods and processes described above, such as the image style migration method. For example, in some embodiments, the image style migration method may be implemented as a computer software program tangibly embodied in a machine-readable medium, such as storage unit 608. In some embodiments, part or all of the computer program may be loaded and/or installed onto the device 600 via the ROM602 and/or the communication unit 609. When the computer program is loaded into RAM603 and executed by the computing unit 601, one or more steps of the image style migration method described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to perform the image style migration method by any other suitable means (e.g., by means of firmware).
Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuitry, Field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), Application Specific Standard Products (ASSPs), system on a chip (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, receiving data and instructions from, and transmitting data and instructions to, a storage system, at least one input device, and at least one output device.
Program code for implementing the methods of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the program codes, when executed by the processor or controller, cause the functions/operations specified in the flowchart and/or block diagram to be performed. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package partly on the machine and partly on a remote machine or entirely on the remote machine or server.
In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which a user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic, speech, or tactile input.
The systems and techniques described here can be implemented in a computing system that includes a back-end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), Wide Area Networks (WANs), and the Internet.
The computer system may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also called a cloud computing server or a cloud host, and is a host product in a cloud computing service system, so that the defects of high management difficulty and weak service expansibility in the traditional physical host and VPS service are overcome. The server may also be a server of a distributed system, or a server incorporating a blockchain.
Artificial intelligence is the subject of research that makes computers simulate some human mental processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.), both at the hardware level and at the software level. Artificial intelligence hardware technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing, and the like; the artificial intelligence software technology mainly comprises a computer vision technology, a voice recognition technology, a natural language processing technology, a machine learning/deep learning technology, a big data processing technology, a knowledge map technology and the like.
Cloud computing (cloud computing) refers to a technology system that accesses a flexibly extensible shared physical or virtual resource pool through a network, where resources may include servers, operating systems, networks, software, applications, storage devices, and the like, and may be deployed and managed in a self-service manner as needed. Through the cloud computing technology, high-efficiency and strong data processing capacity can be provided for technical application and model training of artificial intelligence, block chains and the like.
It should be understood that various forms of the flows shown above may be used, with steps reordered, added, or deleted. For example, the steps described in this disclosure may be performed in parallel or sequentially or in a different order, as long as the desired results of the technical solutions provided by this disclosure can be achieved, and are not limited herein.
The above detailed description should not be construed as limiting the scope of the disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may be made in accordance with design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present disclosure should be included in the scope of protection of the present disclosure.

Claims (19)

1. An image style migration method, comprising:
determining content characteristic coded data of the content graph under at least one associated coding scale of the target coding scale and style characteristic coded data of the style graph under the corresponding associated coding scale;
determining statistical characteristic data of the style sheet under the target coding scale according to each content characteristic coded data and each style characteristic coded data;
and generating a style migration diagram according to the statistical characteristic data and the content characteristic coded data under the target coding scale.
2. The method of claim 1, wherein said determining statistical feature data of the stylistic graph at the target coding scale from each of the content feature encoding data and each of the style feature encoding data comprises:
performing feature fusion on each content feature coded data to obtain content feature fusion data;
performing feature fusion on each style feature coded data to obtain style feature fusion data;
and determining statistical characteristic data of the style sheet under the target coding scale according to the content characteristic fusion data and the style characteristic fusion data.
3. The method of claim 2, wherein the determining statistical feature data of the style sheet at the target coding scale from the content feature fusion data and the style feature fusion data comprises:
determining similarity data according to the content feature fusion data and the style feature fusion data;
generating mean characteristic data of the stylistic graph under the target coding scale according to the similarity data and the stylistic characteristic coding data of the stylistic graph;
and generating standard deviation characteristic data or variance characteristic data of the style sheet under the target coding scale according to the similarity data, the mean characteristic data and the style characteristic coding data under the target coding scale.
4. The method of claim 2, wherein the performing feature fusion on each content feature encoded data to obtain content feature fusion data comprises:
adjusting the content feature coded data to a target size, and fusing the adjusted content feature coded data according to a channel to obtain content feature fusion data;
the performing feature fusion on each style feature coded data to obtain style feature fusion data includes:
and adjusting each style feature coded data to the target size, and fusing each adjusted style feature coded data according to a channel to obtain style feature fusion data.
5. The method of claim 4, further comprising:
and taking the minimum size of the content characteristic coded data or the style characteristic coded data under each associated coding scale as the target size.
6. The method of claim 3, wherein generating a style migration graph from the statistical and content feature encoding data at the target encoding scale comprises:
generating style migration characteristic data under the target coding scale according to the statistical characteristic data and the content characteristic coded data under the target coding scale;
and generating the style migration diagram according to style migration characteristic data under at least one target coding scale.
7. The method of claim 6, wherein the generating style migration feature data at the target encoding scale from the statistical feature data and the content feature encoding data at the target encoding scale comprises:
generating migration volume data under the target coding scale according to the standard deviation feature data or the variance feature data under the target coding scale and the content feature coded data under the target coding scale;
and generating style migration characteristic data under the target coding scale according to the migration volume data and the mean characteristic data under the target coding scale.
8. The method of any one of claims 1-7, wherein the determining content feature encoding data of the content map at least one associated encoding scale of the target encoding scale and style feature encoding data of the style map at the corresponding associated encoding scale comprises:
performing characteristic coding on the content graph and the style graph under at least one coding scale to obtain content characteristic coded data and style characteristic coded data under each coding scale;
regarding each target coding scale, taking each content feature coded data of which the resolution is smaller than that of the content feature coded data corresponding to the target coding scale as the content feature coded data under the associated coding scale of the target coding scale;
and taking each style coding feature data with the resolution smaller than the style feature coding data corresponding to the target coding scale as the style feature coding data under the associated coding scale of the target coding scale.
9. An image style migration apparatus comprising:
the characteristic coded data determining module is used for determining content characteristic coded data of the content graph under at least one associated coding scale of the target coding scale and style characteristic coded data of the style graph under the corresponding associated coding scale;
a statistical characteristic data determining module, configured to determine statistical characteristic data of the style sheet at the target coding scale according to each content characteristic coded data and each style characteristic coded data;
and the style transition diagram generating module is used for generating a style transition diagram according to the statistical characteristic data and the content characteristic coded data under the target coding scale.
10. The apparatus of claim 9, wherein the statistical feature data determination module comprises:
a content feature fusion data obtaining unit, configured to perform feature fusion on each content feature encoded data to obtain content feature fusion data;
the style feature fusion data obtaining unit is used for carrying out feature fusion on each style feature coded data to obtain style feature fusion data;
and the statistical characteristic data determining unit is used for determining the statistical characteristic data of the style graph under the target coding scale according to the content characteristic fusion data and the style characteristic fusion data.
11. The apparatus of claim 10, wherein the statistical feature data determination unit comprises:
the similarity data determination subunit is used for determining similarity data according to the content feature fusion data and the style feature fusion data;
the mean characteristic data determining subunit is used for generating mean characteristic data of the style sheet under the target coding scale according to the similarity data and the style characteristic coding data of the style sheet;
the standard deviation feature data determining subunit is used for generating standard deviation feature data of the style sheet under the target coding scale according to the similarity data, the mean feature data and style feature coded data of the style sheet; alternatively, the first and second electrodes may be,
and the variance characteristic data determining subunit is used for generating variance characteristic data of the style sheet under the target coding scale according to the similarity data, the mean characteristic data and style characteristic coded data of the style sheet.
12. The apparatus according to claim 10, wherein the content feature fusion data obtaining unit includes:
a content feature fusion data obtaining subunit, configured to adjust each content feature encoding data to a target size, and fuse each adjusted content feature encoding data according to a channel to obtain the content feature fusion data;
the style feature fusion data obtaining unit comprises:
and the style feature fusion data obtaining subunit is used for adjusting each style feature coded data to the target size, and fusing each adjusted style feature coded data according to a channel to obtain the style feature fusion data.
13. The apparatus of claim 12, the content feature fusion data obtaining unit, further comprising:
a target size determining subunit, configured to use a minimum size of the content feature encoded data at each associated encoding scale as the target size;
alternatively, the first and second electrodes may be,
the style feature fusion data obtaining unit further comprises:
and the target size determining subunit is used for taking the minimum size of the style characteristic coded data under each associated coding scale as the target size.
14. The apparatus of claim 11, wherein the style migration graph generation module comprises:
the style migration characteristic data generating unit is used for generating style migration characteristic data under the target coding scale according to the statistical characteristic data and the content characteristic coded data under the target coding scale;
and the style migration diagram generating unit is used for generating the style migration diagram according to style migration characteristic data under at least one target coding scale.
15. The apparatus of claim 14, wherein the style migration feature data generation unit comprises:
a migration volume data generating subunit, configured to generate migration volume data in the target coding scale according to the standard deviation feature data or the variance feature data in the target coding scale and the content feature encoded data in the target coding scale;
and the style migration result generation unit is used for generating style migration characteristic data under the target coding scale according to the migration volume data and the mean characteristic data under the target coding scale.
16. The apparatus according to any one of claims 9-15, wherein the feature encoding data determining module comprises:
the characteristic coding unit is used for carrying out characteristic coding on the content graph and the style graph under at least one coding scale to obtain content characteristic data and style characteristic coded data under each coding scale;
a content feature coded data determining unit, configured to, for each target coding scale, use each content feature coded data having a resolution smaller than the content feature coded data corresponding to the target coding scale as the content feature coded data in the associated coding scale of the target coding scale;
and the style characteristic coded data determining unit is used for taking each style coded characteristic data with the resolution smaller than the style characteristic coded data corresponding to the target coding scale as the style characteristic coded data under the relevant coding scale of the target coding scale.
17. An electronic device, comprising:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform an image style migration method as claimed in any one of claims 1-8.
18. A non-transitory computer readable storage medium having stored thereon computer instructions for causing a computer to execute an image style migration method according to any one of claims 1-8.
19. A computer program product comprising a computer program which, when executed by a processor, implements an image style migration method according to any one of claims 1-8.
CN202110739068.1A 2021-06-30 2021-06-30 Image style migration method, device, equipment and storage medium Pending CN113327194A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110739068.1A CN113327194A (en) 2021-06-30 2021-06-30 Image style migration method, device, equipment and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110739068.1A CN113327194A (en) 2021-06-30 2021-06-30 Image style migration method, device, equipment and storage medium

Publications (1)

Publication Number Publication Date
CN113327194A true CN113327194A (en) 2021-08-31

Family

ID=77425268

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110739068.1A Pending CN113327194A (en) 2021-06-30 2021-06-30 Image style migration method, device, equipment and storage medium

Country Status (1)

Country Link
CN (1) CN113327194A (en)

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20180350030A1 (en) * 2017-06-06 2018-12-06 Adobe Systems Incorporated Generating a stylized image or stylized animation by matching semantic features via an appearance guide, a segmentation guide, and a positional guide
CN110473141A (en) * 2019-08-02 2019-11-19 Oppo广东移动通信有限公司 Image processing method, device, storage medium and electronic equipment
CN111325681A (en) * 2020-01-20 2020-06-23 南京邮电大学 Image style migration method combining meta-learning mechanism and feature fusion
CN111932445A (en) * 2020-07-27 2020-11-13 广州市百果园信息技术有限公司 Compression method for style migration network and style migration method, device and system
CN112561792A (en) * 2020-12-25 2021-03-26 北京百度网讯科技有限公司 Image style migration method and device, electronic equipment and storage medium
CN112669308A (en) * 2021-01-06 2021-04-16 携程旅游信息技术（上海）有限公司 Image generation method, system, device and storage medium based on style migration
CN112883806A (en) * 2021-01-21 2021-06-01 杭州广电云网络科技有限公司 Video style migration method and device based on neural network, computer equipment and storage medium

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20180350030A1 (en) * 2017-06-06 2018-12-06 Adobe Systems Incorporated Generating a stylized image or stylized animation by matching semantic features via an appearance guide, a segmentation guide, and a positional guide
CN110473141A (en) * 2019-08-02 2019-11-19 Oppo广东移动通信有限公司 Image processing method, device, storage medium and electronic equipment
CN111325681A (en) * 2020-01-20 2020-06-23 南京邮电大学 Image style migration method combining meta-learning mechanism and feature fusion
CN111932445A (en) * 2020-07-27 2020-11-13 广州市百果园信息技术有限公司 Compression method for style migration network and style migration method, device and system
CN112561792A (en) * 2020-12-25 2021-03-26 北京百度网讯科技有限公司 Image style migration method and device, electronic equipment and storage medium
CN112669308A (en) * 2021-01-06 2021-04-16 携程旅游信息技术（上海）有限公司 Image generation method, system, device and storage medium based on style migration
CN112883806A (en) * 2021-01-21 2021-06-01 杭州广电云网络科技有限公司 Video style migration method and device based on neural network, computer equipment and storage medium

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
吴岳;宋建国;: "基于改进AdaIN的图像风格迁移", 软件导刊, no. 09, 15 September 2020 (2020-09-15) *
马赫;张涛;卢涵宇;: "基于CycleGAN的图像风格迁移", 电脑知识与技术, no. 27, 25 September 2020 (2020-09-25) *

Similar Documents

Publication Publication Date Title
CN114155543A (en) Neural network training method, document image understanding method, device and equipment
CN113792854A (en) Model training and word stock establishing method, device, equipment and storage medium
CN114612749B (en) Neural network model training method and device, electronic device and medium
CN114792355B (en) Virtual image generation method and device, electronic equipment and storage medium
CN113407850B (en) Method and device for determining and acquiring virtual image and electronic equipment
US20220343512A1 (en) Method and apparatus of processing image, electronic device, and storage medium
CN113657395A (en) Text recognition method, and training method and device of visual feature extraction model
CN113393371A (en) Image processing method and device and electronic equipment
CN114120413A (en) Model training method, image synthesis method, device, equipment and program product
CN113641829A (en) Method and device for training neural network of graph and complementing knowledge graph
CN116402914A (en) Method, device and product for determining stylized image generation model
CN113361535B (en) Image segmentation model training, image segmentation method and related device
CN115690443A (en) Feature extraction model training method, image classification method and related device
CN114707638A (en) Model training method, model training device, object recognition method, object recognition device, object recognition medium and product
CN114842541A (en) Model training and face recognition method, device, equipment and storage medium
CN113327194A (en) Image style migration method, device, equipment and storage medium
CN114119374A (en) Image processing method, device, equipment and storage medium
CN113610856A (en) Method and device for training image segmentation model and image segmentation
CN113344213A (en) Knowledge distillation method, knowledge distillation device, electronic equipment and computer readable storage medium
CN113903071A (en) Face recognition method and device, electronic equipment and storage medium
CN114330576A (en) Model processing method and device, and image recognition method and device
CN113032071A (en) Page element positioning method, page testing method, device, equipment and medium
CN114495236B (en) Image segmentation method, apparatus, device, medium, and program product
CN113361575B (en) Model training method and device and electronic equipment
CN113408592B (en) Feature point matching method, device, electronic equipment and computer readable storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination