WO2020114047A1

WO2020114047A1 - Image style transfer and data storage method and apparatus, and electronic device

Info

Publication number: WO2020114047A1
Application number: PCT/CN2019/108272
Authority: WO
Inventors: 许鸿民; 张文波; 郑文
Original assignee: 北京达佳互联信息技术有限公司
Priority date: 2018-12-07
Filing date: 2019-09-26
Publication date: 2020-06-11
Also published as: CN109636712B; CN109636712A

Abstract

An image style transfer and data storage method and apparatus, and an electronic device. The present invention is applicable in the technical field of image processing. The data storage method comprises: inputting images of a plurality of styles to a preset convolutional neural network model and performing training, obtaining intermediate feature maps and weights of output activation of each style filter bank with respect to the intermediate feature maps (S101); storing the intermediate feature maps and the weights of the output activation of each style filter bank with respect to the intermediate feature maps (S102). The data storage method stores, for a plurality of styles, the intermediate feature maps and the weights of the output activation of each style filter bank with respect to the intermediate feature maps, and the data volume of the weights is far less than that of the feature maps. Therefore, in comparison to storing one feature map for each style filter bank separately, the method can reduce the volume occupied in a storage space, thereby saving storage space.

Description

Image style transfer and data storage method, device and electronic equipment

This application requires the priority of the Chinese patent application submitted to the Chinese Patent Office on December 07, 2018, with the application number 201811496794.X and the invention titled "Image Style Migration and Data Storage Method, Device, and Electronic Equipment", all of which are approved by The reference is incorporated in this application.

Technical field

The present application relates to the field of image processing technology, and in particular, to image style transfer and data storage methods, devices, and electronic equipment.

Background technique

Style Transfer is a technology that has received much attention in the field of image processing and computer vision in recent years, and has practical application value. The style transfer can transform an Original Image into an artistically processed Stylized Image. The style elements come from the image of Style Image. For example, any image can be transformed into an image with the style of Van Gogh's famous oil painting "Starry Sky" through style transfer.

In the style transfer method based on the Style Bank framework, different styles are represented by the 128-dimensional Filter Bank in the middle, and each style has its own filter bank, which needs to be separately The feature map of the output activation of the oscilloscope group of each style type is stored, so when more and more styles, the stored data will grow linearly, taking up a lot of storage space.

Summary of the invention

The purpose of the embodiments of the present application is to provide an image style transfer and data storage method, device and electronic equipment, so as to reduce the storage space occupation. The specific technical solutions are as follows:

In a first aspect, an embodiment of the present application provides a data storage method. The method includes:

Input images of multiple styles into a preset convolutional neural network model for training to obtain an intermediate feature map and a group output of each of the style type filters to activate a weight value relative to the intermediate feature map;

The output value of each of the style-type filter banks activates the weight value relative to the intermediate feature map and the intermediate feature map.

Optionally, the data storage method according to an embodiment of the present application is based on a style separation framework.

Optionally, after the storing the output of each style-type filter bank activates the weight value relative to the intermediate feature map and the intermediate feature map, the method further includes:

Obtain the image to be styled, the target style type to be converted, and the intermediate feature map;

According to the target style type, obtain a target weight value of the output activation of the filter bank of the target style type relative to the intermediate feature map;

Adjust the weight of each feature in the intermediate feature map according to the target weight value to obtain the target filter bank output activation;

The target filter bank output is activated to process the to-be-style-transferred image to obtain the style-transferred image.

In a second aspect, an embodiment of the present application provides a method for transferring image styles. The method includes:

Obtain the image to be style transferred, the target style type to be converted and the preset intermediate feature map;

Optionally, the image style transfer method according to an embodiment of the present application is based on a style separation framework.

In a third aspect, an embodiment of the present application provides a data storage device. The device includes:

The weight value determination module is configured to input images of various styles into a preset convolutional neural network model for training to obtain an intermediate feature map and group output of each of the style type filters. Activation is relative to the intermediate features The weight value of the graph;

The data storage module is configured to store the weight value of the output activation of each style type filter bank relative to the intermediate feature map and the intermediate feature map.

Optionally, the data storage device of the embodiment of the present application is based on a style separation framework.

Optionally, the data storage device of the embodiment of the present application further includes:

The first parameter acquisition module is configured to acquire an image to be styled to be transferred, a target style type to be converted, and the intermediate feature map;

A target weight value determination module, configured to obtain the target weight value of the output activation of the filter bank of the target style type relative to the intermediate feature map according to the target style type;

The target filter bank determination module is configured to adjust the weight of each feature in the intermediate feature map according to the target weight value to obtain the target filter bank output activation;

The style transfer module is configured to process the to-be-style-transferred image through the output of the target filter bank to obtain the style-transferred image.

According to a fourth aspect, an embodiment of the present application provides an image style transfer device, the device including:

The second parameter acquisition module is configured to acquire the image to be styled, the target style type to be converted, and the preset intermediate feature map;

A weight value obtaining module configured to obtain the target weight value of the output activation of the filter bank of the target style type relative to the intermediate feature map according to the target style type;

The filter adjustment module is configured to adjust the weight of each feature in the intermediate feature map according to the target weight value to obtain the target filter bank output activation;

The image processing module is configured to process the to-be-style-transferred image through the target filter bank output activation to obtain the style-transferred image.

Optionally, the image style migration device according to the embodiment of the present application is based on a style separation framework.

According to a fifth aspect, an embodiment of the present application provides an electronic device, including:

processor;

Memory for storing processor executable instructions;

Wherein, the processor is configured to:

Optionally, the electronic device of the embodiment of the present application is based on a style separation framework.

Optionally, the above processor may also execute:

According to a sixth aspect, an embodiment of the present application provides an electronic device, including:

processor;

Memory for storing processor executable instructions;

Wherein, the processor is configured to:

According to a seventh aspect, an embodiment of the present application provides a non-transitory computer-readable storage medium. When instructions in the storage medium are executed by a processor of a mobile terminal, the mobile terminal can execute a data storage method. Characteristically, the method includes:

Optionally, after storing the weight values of the above style types and the preset filter bank, the method further includes:

In an eighth aspect, an embodiment of the present application provides a non-transitory computer-readable storage medium. When instructions in the storage medium are executed by a processor of a mobile terminal, the mobile terminal can perform an image style transfer method, It is characterized in that the method includes:

In a ninth aspect, an embodiment of the present application provides a computer program product. When the computer program product is executed on a processor, it implements:

Input images of multiple styles into a preset convolutional neural network model for training to obtain an intermediate feature map and a group output of each style type filter to activate a weight value relative to the intermediate feature map;

According to a tenth aspect, an embodiment of the present application provides a computer program product, which, when executed on a processor, implements:

The technical solution provided by the embodiments of the present application may include the following beneficial effects: In the embodiments of the present application, for multiple style types, the weight values and intermediate features of the output activation of the filter bank of each style type relative to the intermediate feature map are stored The data volume of the weight value is much smaller than the data volume of the feature map, so compared to storing a feature map for each style type filter bank separately, it can reduce the storage space occupation and save storage space. The general description and the detailed description below are only exemplary and explanatory, and do not limit the present disclosure.

BRIEF DESCRIPTION

In order to more clearly explain the embodiments of the present application or the technical solutions in the prior art, the following will briefly introduce the drawings required in the embodiments or the description of the prior art. Obviously, the drawings in the following description are only These are some embodiments of the present application. For those of ordinary skill in the art, without paying any creative work, other drawings can be obtained based on these drawings.

FIG. 1 is a schematic flowchart of a data storage method according to an embodiment of the present application;

2 is another schematic flowchart of a data storage method according to an embodiment of the present application;

3 is a schematic flowchart of a method for transferring image styles according to an embodiment of the present application;

4 is a schematic diagram of a data storage device according to an embodiment of the present application;

5 is another schematic diagram of a data storage device according to an embodiment of the present application;

6 is a schematic diagram of an image style migration device according to an embodiment of the present application;

7 is a first schematic diagram of an electronic device according to an embodiment of this application;

8 is a second schematic diagram of an electronic device according to an embodiment of the present application;

9 is a third schematic diagram of the electronic device of the embodiment of the present application;

10 is a fourth schematic diagram of the electronic device of the embodiment of the present application;

FIG. 11 is a schematic diagram of a preset convolutional neural network model according to an embodiment of the present application.

detailed description

Exemplary embodiments will be described in detail here, examples of which are shown in the drawings. When referring to the drawings below, unless otherwise indicated, the same numerals in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of devices and methods consistent with some aspects of the application as detailed in the appended claims.

First, the technical terms in the embodiments of the present application are explained:

Style transfer is a technology that has attracted the attention of many scholars in the field of image processing/computer vision in recent years and has practical application value. Style transfer can transform an original image into a stylized image after artistic processing. The style element comes from another image called style image. A typical example is to transform any image into an image with the style of Van Gogh's famous oil painting "Starry Sky".

Texture (Synthesis) is the predecessor of style transfer. The method of texture generation is to model the texture at the pixel level. The main purpose of this stage is to model the texture at the pixel level and migrate the given texture into the original image to generate the final texture image. After introducing the convolutional neural network into this classic problem. The field has made amazing progress. The quality of the final texture image is greatly improved, which is called stylized image in style transfer, so there is the field of style transfer. Style transfer is a method of texture modeling based on depth features. Therefore, the style transfer not only completes the more general texture generation, but also its technical characteristics are almost different from the texture generation.

The specific method of style transfer is to put the original image and the style image into a pre-trained convolutional neural network, such as VGG19 (Visual Geometry Group Network 19, Visual Geometry Group Network 19). By transforming the depth features of the original image into depth features similar to the style image, the purpose of transforming the original image into a style image is achieved.

Activations (output activation), also called activation functions or excitation functions, are functions that run on the neurons of the artificial neural network and are responsible for mapping the inputs of the neurons to the output.

In the related style migration scheme, there are generally three types of migration: single style migration, multi-style migration and arbitrary style migration. Classic methods for single style transfer include: FNS (Fast Neural Style). Typical methods of multi-style transfer include: multi-style method based on CIN (Conditional Instance Normalization), and Style Bank method. Arbitrary style transfer includes: NS (Neural Style) based on optimization methods, WCT (whitening and coloring transforms) methods based on brightness and color conversion, and methods based on exchanging depth features.

Among the multi-style migration methods are: CIN-based method and Style Bank method. Both of these methods are based on FNS) improvements. Therefore, they have a self-encoder structure in their network structure, namely the encoding network and the decoding network. In the CIN method, different styles are represented by different IN layer parameters. This means that each different style has its own different IN parameters. In the Style Bank method, different styles are represented by the middle 128-dimensional Filter Banks (filter bank). Similarly, each style has its own filter bank.

In the above two methods, whether it is the CIN method or the Style Bank method, the style is expressed by the parameters of the IN layer or the parameters of the filter bank, and the content is expressed by other parameters in the network. When the style is more and more, the stored data will grow linearly, which will take up a lot of storage space.

In view of this, the embodiments of the present application provide an efficient and interpretable data storage method. Referring to FIG. 1, the method includes:

S101, input images of various styles into a preset convolutional neural network model for training, and obtain intermediate feature maps (Intermediate Features) and output weights of the above-mentioned style type filter banks relative to the weights of the above intermediate feature maps value.

The data storage method in the embodiment of the present application may be implemented by a processing system, and the processing system is any system capable of implementing the data storage method in the embodiment of the present application. E.g:

The processing system may be a device, including: a processor, a memory, a communication interface, and a bus; the processor, the memory, and a communication interface are connected through a bus and complete communication with each other; the memory stores executable program code; the processor reads the memory by The executable program code stored in the program runs the program corresponding to the executable program code, and is used to execute the data storage method of the embodiment of the present application.

The processing system may also be an application program for executing the data storage method of the embodiment of the present application at runtime.

The processing system may also be a storage medium for storing executable code, and the executable code is used to execute the data storage method of the embodiments of the present application.

Through the related neural network training method, the weight value of the intermediate feature map and the output activation of each style type filter bank capable of style transfer to the intermediate feature map is obtained.

The preset convolutional neural network model may be as shown in FIG. 11 and includes an encoder (Encoder), a decoder (Decoder), and a style representation part. The style representation part is composed of style base (Style Basis) and style weights (Style Weights).

During the training process, the pictures received by the preset convolutional neural network include: a set of content images and a set of style images. For each input content image, randomly select a mini-batch size style image from the style image set and train with the content image. It should be noted that the random sampling of style sets can ultimately guarantee the traversal of style image sets. The specific training process can include:

Input the content image into the encoder to get the intermediate feature map, and then use T+1 training strategy, including T round for style training and 1 round for self-encoder training.

Among them, the style training is: after obtaining the intermediate feature map of the content image, the style weight and the style base are weighted to obtain the style representation. Operate the intermediate feature map of the content image according to the style representation to obtain a stylized intermediate feature map, and then use the decoder to decode the stylized intermediate feature map into a stylized image. Then, input the stylized image and the input content image into the loss network at the same time, calculate the loss (Loss), and optimize the network through the back propagation algorithm (BackPropagationAlgorithm), and adjust the relevant values of all parts of the network. When the subjective judgment of the final stylized image is qualified, the training process ends and the style weight corresponding to any style is finally determined. The training process corresponds to the solid line in Figure 11.

The training of the self-encoder is: after obtaining the intermediate feature map of the content image, directly use the decoder to decode to obtain the reconstructed content image. Then calculate the mean square error (Mean Square Error) of the reconstructed image and the input image. Then through the back propagation algorithm (BackPropagationAlgorithm) to optimize the network, and adjust the network encoder and decoder related values. When the subjective judgment of the stylized image is qualified, the training process ends. The training process corresponds to the dotted line in FIG. 11.

S102: Store the weight value of the output activation of each of the above-mentioned style-type filter banks relative to the above-mentioned intermediate feature map and the above-mentioned intermediate feature map.

The processing system stores the weight values of the output activation of each style type filter bank relative to the intermediate feature map and the intermediate feature map to a designated location for use in subsequent style transfer.

Wherein, the weight of the intermediate feature value may be a set of weight value arrays used to represent different feature graph styles, and different types of styles may be represented by different weight value arrays.

Optionally, the data storage method according to the embodiment of the present application may be based on a style separation framework, such as Style Bank. In the framework of style separation, the style is represented by the filter bank and Block-Wise (block dimension) coding, and the training of the decoding network is partially separated. In the style separation framework, for example, the filter bank in the Style Bank framework has better interpretation, but the utilization rate of the filter bank used in the style is actually low, and the exploration of redundancy between styles is feasible. Based on this, the parameters of the style can be shared, and all the styles that need to be expressed are represented by the same set of filters. For different styles, a set of weights representing different feature maps are used to represent each specific style. Therefore, the parameters of each style type can be expressed using the same set of filter banks. For each style itself, it only needs to be represented by a set of weight values.

In the embodiment of the present application, for multiple style types, the output value of the filter bank of each style type is stored to activate the weight value and the intermediate feature map relative to the intermediate feature map, and the data amount of the weight value is much smaller than the data of the feature map Quantitative, so compared to storing a feature map for each style type filter bank separately, it can reduce the amount of storage space occupied and save storage space. In addition, this method explores the association and redundancy between styles, so that the associations in styles can be expressed using shared parameters, thereby significantly reducing the parameters representing styles. Use a style-separated framework to express styles individually, and at the same time, make the styles expressed separately share parameters as much as possible. Thanks to the representation of the correlation between each style type, the parameters of the stored style are reduced. At the same time, a small number of channel weights are reserved to represent independent styles.

Optionally, referring to FIG. 2, after the storage of the output of each of the above-mentioned style type filter banks activates the weight value relative to the intermediate feature map and the intermediate feature map, the method further includes:

S103: Obtain a style transfer image to be converted, a target style type to be converted, and the above intermediate feature map.

When the user needs to perform the style transfer, the user will determine the image to be style transferred, that is, the style transfer image, and the user will input the style type into which the style transfer image is converted, that is, select the target style type to be converted. The processing system obtains the image to be style transferred, the target style type to be converted and the preset intermediate feature map.

S104: Acquire the target weight value of the output activation of the filter bank of the target style type relative to the intermediate feature map according to the target style type.

The processing system stores the weight value of the output activation of the filter bank of each style type relative to the intermediate feature map in advance, and the processing system determines the weight value of the output activation of the filter bank of the target style type relative to the intermediate feature map, that is, the target weight value And get the target weight value.

S105: Adjust the weight of each feature in the intermediate feature map according to the target weight value to obtain the target filter bank output activation.

The processing system adjusts the weight of each feature in the intermediate feature map according to the target weight value to obtain the weight-adjusted feature map, that is, the output of the target filter bank is activated.

S106: Activate and process the image to be style-transferred through the output of the target filter bank to obtain an image after style transfer.

The processing system uses the output of the target filter bank to activate the processing of the image to be styled, converts the style type of the image to be styled to the target style type, and obtains the image after style transfer.

In the embodiment of the present application, the target filter bank output activation required for conversion to the target style type is obtained through the target weight value and the intermediate feature map, and the target filter bank output activation is used to perform the style migration of the style migration image to realize the image Move to the specified style.

An embodiment of the present application also provides a method for transferring image styles. Referring to FIG. 3, the method includes:

S301: Obtain a style transfer image, a target style type to be converted, and a preset intermediate feature map.

The image style migration method in the embodiment of the present application may be implemented by a migration system, and the migration system is any system that can implement the image style migration method of the embodiment of the present application. E.g:

The migration system may be a device, including: a processor, a memory, a communication interface, and a bus; the processor, the memory, and a communication interface are connected through a bus and complete communication with each other; the memory stores executable program code; the processor reads the memory by The executable program code stored in the program runs the program corresponding to the executable program code, and is used to execute the image style transfer method of the embodiment of the present application.

The migration system may also be an application program for executing the image style migration method of the embodiment of the present application at runtime.

The migration system may also be a storage medium for storing executable code, and the executable code is used to execute the image style migration method of the embodiment of the present application.

When the user needs to perform the style transfer, the user will determine the image to be style transferred, that is, the style transfer image, and the user will input the style type into which the style transfer image is converted, that is, select the target style type to be converted. The migration system obtains the migration image to be styled, the target style type to be converted and the preset intermediate feature map. The preset intermediate feature map can be obtained by training a preset convolutional neural network model.

S302: Acquire the target weight value of the output activation of the filter bank of the target style type relative to the intermediate feature map according to the target style type.

The migration system stores the weight value of the output activation of the filter bank of each style type relative to the intermediate feature map in advance, and the migration system determines the weight value of the output activation of the filter bank of the target style type relative to the intermediate feature map, that is, the target weight value And get the target weight value.

Optionally, the step of pre-storing the weight values of each style type and the preset filter bank includes: inputting images of multiple style types into a preset convolutional neural network model for training to obtain an intermediate feature map and each of the above styles The group filter output of the type filter activates the weight value relative to the above-mentioned intermediate feature map; the weight value of the output activation of each style-type filter bank relative to the above-mentioned intermediate feature map and the above-mentioned intermediate feature map are stored.

S303: Adjust the weight of each feature in the intermediate feature map according to the target weight value to obtain the target filter bank output activation.

The migration system adjusts the weight of each feature in the intermediate feature map according to the target weight value to obtain the weight-adjusted feature map, that is, the output of the target filter bank is activated.

S304: Activate and process the to-be-style-transferred image through the above-mentioned target filter bank output activation to obtain the style-transferred image.

The migration system uses the target filter bank output to activate the processing of the image to be styled, converts the style type of the image to be styled to the target style type, and obtains the image after style transfer.

Optionally, the above image style transfer method is based on a style separation framework, such as Style Bank. In the framework of style separation, the style is represented by the filter bank and the encoding, and the training of the decoding network is partially separated. In the style separation framework, for example, the filter bank in the Style Bank framework has better interpretation, the utilization rate of the filter bank used by the style is actually low, and the exploration of redundancy between styles is feasible. Based on this, the parameters of the style can be shared, and all the styles that need to be expressed are represented by the same set of filters. For different styles, a set of weights representing different feature maps are used to represent each specific style. Therefore, the parameters of each style type can be expressed using the same set of filters. For each style itself, it only needs to be represented by a set of weight values.

In the embodiment of the present application, the target filter bank output activation required for conversion to the target style type is obtained through the target weight value and the preset intermediate feature map, and the target filter bank output activation is used to perform the style migration of the style migration image to achieve In order to transfer the image to the specified style. For multiple style types, the output of the filter bank that stores each style type activates the weight value and the intermediate feature map relative to the intermediate feature map, and the data amount of the weight value is much smaller than that of the feature map, so compared to A separate feature map is stored for each style of filter bank, which can reduce the amount of storage space occupied and save storage space. The association and redundancy between styles are explored, so that the associations in styles can be expressed in the form of shared parameters, so that the parameters representing styles are significantly reduced. Use a style-separated framework to express styles individually, and at the same time, make the styles expressed separately share parameters as much as possible. Thanks to the representation of the correlation between each style type, the parameters of the stored style are reduced. At the same time, a small number of channel weights are reserved to represent independent styles.

An embodiment of the present application further provides a data storage device. Referring to FIG. 4, the device includes:

The weight value determination module 401 is configured to input images of various styles into a preset convolutional neural network model for training to obtain an intermediate feature map and group output activation of each of the aforementioned style type filters relative to the aforementioned intermediate feature map Weight value

The data storage module 402 is configured to store the weight value of the output activation of each of the aforementioned style-type filter banks relative to the aforementioned intermediate feature map and the aforementioned intermediate feature map.

In the embodiment of the present application, for multiple style types, the output value of the filter bank of each style type is stored to activate the weight value and the intermediate feature map relative to the intermediate feature map, and the data amount of the weight value is much smaller than the data of the feature map Quantitative, so compared to storing a feature map for each style type filter bank separately, it can reduce the amount of storage space occupied and save storage space.

Optionally, referring to FIG. 5, the data storage device according to an embodiment of the present application further includes:

The first parameter obtaining module 403 is configured to obtain an image to be styled to be transferred, a target style type to be converted, and the above intermediate feature map;

The target weight value determination module 404 is configured to obtain the target weight value of the output activation of the filter bank of the target style type relative to the intermediate feature map according to the target style type;

The target filter bank determination module 405 is configured to adjust the weight of each feature in the intermediate feature map according to the target weight value to obtain the target filter bank output activation;

The style transfer module 406 is configured to process the image to be style transferred through the output of the target filter bank to obtain an image after the style transfer.

An embodiment of the present application further provides an image style transfer device. Referring to FIG. 6, the device includes:

The second parameter obtaining module 601 is configured to obtain the image to be style-transferred, the target style type to be converted and the preset intermediate feature map;

The weight value obtaining module 602 is configured to obtain the target weight value of the output activation of the filter bank of the target style type relative to the intermediate feature map according to the target style type;

The filter adjustment module 603 is configured to adjust the weight of each feature in the intermediate feature map according to the target weight value to obtain a target filter bank output activation;

The image processing module 604 is configured to process the image to be style-transferred through the output of the target filter bank to obtain an image after style transfer.

An embodiment of the present application also provides an electronic device, referring to FIG. 7, including:

Processor 701;

A memory 702 for storing executable instructions of the processor 701;

Wherein, the above processor 701 is configured as:

Input images of various styles into a preset convolutional neural network model for training to obtain the intermediate feature map and the group output of each of the above style type filters to activate the weight value relative to the above intermediate feature map;

The weight value of the output activation of each of the aforementioned style-type filter banks with respect to the aforementioned intermediate feature map and the aforementioned intermediate feature map are stored.

Optionally, the processor 701 may also execute:

Obtain the image to be style transferred, the target style type to be converted and the above intermediate feature map;

Obtain the target weight value of the output activation of the filter bank of the target style type relative to the intermediate feature map according to the target style type;

According to the above target weight value, adjust the weight of each feature in the above intermediate feature map to obtain the target filter bank output activation;

The above-mentioned target filter bank output activation is used to process the to-be-style-transferred image to obtain a style-transferred image.

Optionally, the electronic device in this embodiment of the present application may be the apparatus 800 shown in FIG. 8. For example, the device 800 may be a mobile phone, a computer, a digital broadcasting terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, and so on.

8, the device 800 may include one or more of the following components: a processing component 802, a memory 804, a power component 806, a multimedia component 808, an audio component 810, an input/output (I/O) interface 812, a sensor component 814,与通信组816.

The processing component 802 generally controls the overall operations of the device 800, such as operations associated with display, telephone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps in the above method. In addition, the processing component 802 may include one or more modules to facilitate interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate interaction between the multimedia component 808 and the processing component 802.

The memory 804 is configured to store various types of data to support operation at the device 800. Examples of these data include instructions for any application or method operating on the device 800, contact data, phone book data, messages, pictures, videos, and so on. The memory 804 may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read only memory (EEPROM), erasable and removable Programmable read only memory (EPROM), programmable read only memory (PROM), read only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

The power supply component 806 provides power to various components of the device 800. The power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device 800.

The multimedia component 808 includes a screen that provides an output interface between the device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, swipe, and gestures on the touch panel. The touch sensor may not only sense the boundary of the touch or sliding action, but also detect the duration and pressure related to the touch or sliding operation. In some embodiments, the multimedia component 808 includes a front camera and/or a rear camera. When the device 800 is in an operation mode, such as a shooting mode or a video mode, the front camera and/or the rear camera may receive external multimedia data. Each front camera and rear camera can be a fixed optical lens system or have focal length and optical zoom capabilities.

The audio component 810 is configured to output and/or input audio signals. For example, the audio component 810 includes a microphone (MIC). When the device 800 is in an operation mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal may be further stored in the memory 804 or transmitted via the communication component 816. In some embodiments, the audio component 810 further includes a speaker for outputting audio signals.

The I/O interface 812 provides an interface between the processing component 802 and a peripheral interface module. The peripheral interface module may be a keyboard, a click wheel, or a button. These buttons may include, but are not limited to: home button, volume button, start button, and lock button.

The sensor component 814 includes one or more sensors for providing the device 800 with status assessment in various aspects. For example, the sensor component 814 can detect the on/off state of the device 800, and the relative positioning of the components, for example, the component is the display and keypad of the device 800, and the sensor component 814 can also detect the position change of the device 800 or a component of the device 800 The presence or absence of user contact with the device 800, the orientation or acceleration/deceleration of the device 800, and the temperature change of the device 800. The sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor component 814 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor component 814 may further include an acceleration sensor, a gyro sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

The communication component 816 is configured to facilitate wired or wireless communication between the device 800 and other devices. The device 800 may access a wireless network based on a communication standard, such as WiFi, an operator network (such as 2G, 3G, 4G, or 5G), or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

In an exemplary embodiment, the apparatus 800 may be one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable A gate array (FPGA), controller, microcontroller, microprocessor or other electronic components are implemented to perform the above method.

In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions, such as a memory 804 including instructions, is also provided. The above instructions can be executed by the processor 820 of the device 800 to complete the above method. For example, the non-transitory computer-readable storage medium may be ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, optical data storage device, or the like.

Optionally, the electronic device in this embodiment of the present application may be the apparatus 900 shown in FIG. 9. For example, the device 900 may be provided as a server. 9, the device 900 includes a processing component 922, which further includes one or more processors, and memory resources represented by the memory 932, for storing instructions executable by the processing component 922, such as application programs. The application programs stored in the memory 932 may include one or more modules each corresponding to a set of instructions. In addition, the processing component 922 is configured to execute instructions to perform the above method.

The device 900 may also include a power component 926 configured to perform power management of the device 900, a wired or wireless network interface 950 configured to connect the device 900 to the network, and an input/output (I/O) interface 958. The device 900 can operate based on an operating system stored in the memory 932, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM or the like.

An embodiment of the present application also provides an electronic device, referring to FIG. 10, including:

Processor 1001;

A memory 1002 for storing executable instructions of the processor 1001;

Among them, the processor 1001 is configured as:

Optionally, the electronic device of the embodiment of the present application is based on a style separation framework. Optionally, the electronic device in the embodiment of the present application may be the apparatus shown in FIG. 8 or FIG. 9.

An embodiment of the present application provides a non-transitory computer-readable storage medium. When the instructions in the storage medium are executed by a processor of a mobile terminal, the mobile terminal can execute a data storage method. The method includes:

An embodiment of the present application further provides a non-transitory computer-readable storage medium. When the instructions in the storage medium are executed by the processor of the mobile terminal, the mobile terminal can perform a method for transferring image styles. The method includes:

An embodiment of the present application also provides a computer program product. When the computer program product is executed on a processor, a data storage method is implemented. The method includes:

An embodiment of the present application also provides a computer program product. When the computer program product is executed on a processor, a method for transferring image styles is implemented. The method includes:

After considering the description and practice of the application disclosed herein, those skilled in the art will easily think of other embodiments of the application. This application is intended to cover any variations, uses, or adaptive changes of this application, which follow the general principles of this application and include common general knowledge or customary technical means in the technical field not disclosed in this application . The description and examples are to be considered exemplary only, and the true scope and spirit of this application are pointed out by the following claims.

It should be understood that the present application is not limited to the precise structure that has been described above and shown in the drawings, and various modifications and changes can be made without departing from the scope thereof. The scope of this application is limited only by the appended claims.

The above are only the preferred embodiments of this application and are not intended to limit this application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this application should be included in this application Within the scope of protection.

Claims

A data storage method, characterized in that the method includes:

Input images of multiple styles into a preset convolutional neural network model for training to obtain an intermediate feature map and a group output of each of the style type filters to activate a weight value relative to the intermediate feature map;

The output value of each of the style-type filter banks activates the weight value relative to the intermediate feature map and the intermediate feature map.
The method according to claim 1, wherein the data storage method is based on a style separation framework.
The method according to claim 1, characterized in that after the output of the storage of each style-type filter bank activates the weight value relative to the intermediate feature map and the intermediate feature map, the method further include:

Obtain the image to be styled, the target style type to be converted, and the intermediate feature map;

According to the target style type, obtain a target weight value of the output activation of the filter bank of the target style type relative to the intermediate feature map;

Adjust the weight of each feature in the intermediate feature map according to the target weight value to obtain the target filter bank output activation;

The target filter bank output is activated to process the to-be-style-transferred image to obtain the style-transferred image.
An image style transfer method, characterized in that the method includes:

Obtain the image to be style transferred, the target style type to be converted and the preset intermediate feature map;

According to the target style type, obtain a target weight value of the output activation of the filter bank of the target style type relative to the intermediate feature map;

Adjust the weight of each feature in the intermediate feature map according to the target weight value to obtain the target filter bank output activation;

The target filter bank output is activated to process the to-be-style-transferred image to obtain the style-transferred image.
The method according to claim 4, wherein the image style transfer method is based on a style separation framework.
A data storage device, characterized in that the device includes:

The weight value determination module is configured to input images of various styles into a preset convolutional neural network model for training to obtain an intermediate feature map and group output of each of the style type filters. Activation is relative to the intermediate features The weight value of the graph;

The data storage module is configured to store the weight value of the output activation of each style type filter bank relative to the intermediate feature map and the intermediate feature map.
The device of claim 6, wherein the data storage device is based on a style separation framework.
The device according to claim 6, wherein the device further comprises:

The first parameter acquisition module is configured to acquire an image to be styled to be transferred, a target style type to be converted, and the intermediate feature map;

A target weight value determination module, configured to obtain the target weight value of the output activation of the filter bank of the target style type relative to the intermediate feature map according to the target style type;

The target filter bank determination module is configured to adjust the weight of each feature in the intermediate feature map according to the target weight value to obtain the target filter bank output activation;

The style transfer module is configured to process the to-be-style-transferred image through the output of the target filter bank to obtain the style-transferred image.
An image style transfer device, characterized in that the device includes:

The second parameter acquisition module is configured to acquire the image to be styled, the target style type to be converted, and the preset intermediate feature map;

A weight value obtaining module configured to obtain the target weight value of the output activation of the filter bank of the target style type relative to the intermediate feature map according to the target style type;

The filter adjustment module is configured to adjust the weight of each feature in the intermediate feature map according to the target weight value to obtain the target filter bank output activation;

The image processing module is configured to process the to-be-style-transferred image through the target filter bank output activation to obtain the style-transferred image.
The device according to claim 9, wherein the image style transfer means is based on a style separation framework.
An electronic device, characterized in that it includes:

processor;

Memory for storing processor executable instructions;

Wherein, the processor is configured to execute the data storage method according to any one of claims 1-3 at runtime.
An electronic device, characterized in that it includes:

processor;

Memory for storing processor executable instructions;

Wherein, the processor is configured to execute the style transfer method according to any one of claims 4-5 at runtime.
A non-transitory computer-readable storage medium, when instructions in the storage medium are executed by a processor of a mobile terminal, enabling the mobile terminal to execute a data storage method, characterized in that the method includes:

Input images of multiple styles into a preset convolutional neural network model for training to obtain an intermediate feature map and a group output of each of the style type filters to activate a weight value relative to the intermediate feature map;

The output value of each of the style-type filter banks activates the weight value relative to the intermediate feature map and the intermediate feature map.
A non-transitory computer-readable storage medium, when instructions in the storage medium are executed by a processor of a mobile terminal, enable the mobile terminal to perform a method for transferring image styles, characterized in that the method includes:

Obtain the image to be style transferred, the target style type to be converted and the preset intermediate feature map;

According to the target style type, obtain a target weight value of the output activation of the filter bank of the target style type relative to the intermediate feature map;

Adjust the weight of each feature in the intermediate feature map according to the target weight value to obtain the target filter bank output activation;

The target filter bank output is activated to process the to-be-style-transferred image to obtain the style-transferred image.