CN113112536A - Image processing model training method, image processing method and device - Google Patents
Image processing model training method, image processing method and device Download PDFInfo
- Publication number
- CN113112536A CN113112536A CN202110294804.7A CN202110294804A CN113112536A CN 113112536 A CN113112536 A CN 113112536A CN 202110294804 A CN202110294804 A CN 202110294804A CN 113112536 A CN113112536 A CN 113112536A
- Authority
- CN
- China
- Prior art keywords
- image
- neural network
- image processing
- model
- processing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000012545 processing Methods 0.000 title claims abstract description 125
- 238000000034 method Methods 0.000 title claims abstract description 74
- 238000012549 training Methods 0.000 title claims abstract description 45
- 238000003672 processing method Methods 0.000 title claims abstract description 18
- 238000013528 artificial neural network Methods 0.000 claims abstract description 91
- 238000003062 neural network model Methods 0.000 claims abstract description 79
- 230000036548 skin texture Effects 0.000 claims abstract description 28
- 238000010586 diagram Methods 0.000 claims description 58
- 230000008569 process Effects 0.000 claims description 18
- 238000004590 computer program Methods 0.000 claims description 14
- 238000004364 calculation method Methods 0.000 claims description 4
- 230000006870 function Effects 0.000 description 9
- 238000012986 modification Methods 0.000 description 5
- 230000004048 modification Effects 0.000 description 5
- 230000003287 optical effect Effects 0.000 description 5
- 238000004422 calculation algorithm Methods 0.000 description 4
- 239000000463 material Substances 0.000 description 4
- 238000012805 post-processing Methods 0.000 description 4
- 238000004891 communication Methods 0.000 description 3
- 239000013307 optical fiber Substances 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 230000003044 adaptive effect Effects 0.000 description 1
- 230000004075 alteration Effects 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000001364 causal effect Effects 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000005286 illumination Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/40—Analysis of texture
- G06T7/41—Analysis of texture based on statistical description of texture
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/50—Image enhancement or restoration by the use of more than one image, e.g. averaging, subtraction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20212—Image combination
- G06T2207/20221—Image fusion; Image merging
Abstract
The application relates to the technical field of image processing, and discloses an image processing model training method, an image processing method and an image processing device. The image processing model training method comprises the following steps: firstly, obtaining a sample image, processing skin texture in the sample image to obtain a reference image, then inputting the sample image into a neural network model to obtain a feature map output by a specified neural network layer, inputting a target feature map obtained by overlapping the feature map and a noise superposition map into a next neural network layer for continuous processing to obtain a predicted image of the neural network model on the sample image, and finally training the neural network model by using the loss value between the predicted image and the reference image to obtain an image processing model. Therefore, the skin texture of the image to be processed can be adaptively improved by utilizing the image processing model.
Description
Technical Field
The present application relates to the field of image processing technologies, and in particular, to an image processing model training method, an image processing method, and an image processing apparatus.
Background
The quality of the obtained images is different according to different devices for obtaining the images, and the obtained images need to be further processed according to the condition that the images with better skin texture effect need to be obtained.
In the prior art, skin noise can be acquired from a skin noise material, and then the skin noise is added to an image to improve the texture of the skin in the image; or generating skin noise through a post-processing algorithm, and improving the skin texture of the image by using the generated skin noise, but the skin noise material is limited and cannot adapt to the requirements of improving the skin texture in different scenes; after the skin noise is generated through the post-processing algorithm, noise logics adaptive to different scenes need to be designed, so that the images in different scenes cannot be self-adaptively improved in skin texture.
Disclosure of Invention
The embodiment of the application provides an image processing model training method, an image processing method and an image processing device, so that the skin texture of an image to be processed can be improved in a self-adaptive mode by using an image processing model.
In a first aspect, an embodiment of the present application provides an image processing model training method, including:
acquiring a sample image;
processing the skin texture in the sample image to obtain a reference image;
inputting the sample image into a neural network model to obtain a characteristic diagram output by a designated neural network layer in the neural network model;
overlapping the characteristic diagram and the noise characteristic diagram to obtain a target characteristic diagram;
taking the target characteristic graph as an output result of the appointed neural network layer, and inputting the output result to a next neural network layer for continuous processing to obtain a predicted image of the neural network model on the sample image;
obtaining a loss value according to the difference between the predicted image and the reference image;
and training the neural network model based on the loss value to obtain an image processing model.
Optionally, the neural network model includes at least one designated neural network layer, and each designated neural network layer is before a last convolutional layer of the neural network model.
Optionally, before the feature map and the noise feature map are subjected to superposition processing to obtain the target feature map, the method further includes:
randomly generating the noise characteristic diagram with the same size as the characteristic diagram.
Optionally, when there is one designated neural network layer, the designated neural network layer is a neural network layer before the last convolutional layer in the neural network model.
Optionally, the sample image is obtained from a sample image set, and the sample image set includes images acquired under different shooting conditions.
Optionally, the overlapping the feature map and the noise feature map to obtain a target feature map includes:
and carrying out weighted summation processing on the feature values of the same pixel position in the feature map and the noise feature map to obtain the target feature map.
In a second aspect, an embodiment of the present application provides an image processing method, including:
acquiring an image to be processed;
inputting the image to be processed into a pre-trained image processing model to process the skin texture of the image to be processed, and obtaining an optimized image of the image to be processed added with skin noise;
wherein the image processing model is trained according to the method of any one of the first aspect.
In a third aspect, an embodiment of the present application provides an image processing model training apparatus, including:
a first acquisition unit configured to perform acquisition of a sample image;
a first processing unit configured to perform processing on skin texture in the sample image, resulting in a reference image;
the output unit is configured to input the sample image into a neural network model to obtain a feature map output by a specified neural network layer in the neural network model;
the superposition processing unit is configured to perform superposition processing on the characteristic diagram and the noise characteristic diagram to obtain a target characteristic diagram;
the prediction unit is configured to input the target feature map as an output result of the designated neural network layer to a next neural network layer for continuous processing to obtain a predicted image of the neural network model on the sample image;
a calculation unit configured to perform deriving a loss value from a difference between the prediction image and the reference image;
and the training unit is configured to perform training on the neural network model based on the loss value to obtain an image processing model.
Optionally, the neural network model includes at least one designated neural network layer, and each designated neural network layer is before a last convolutional layer of the neural network model.
Optionally, the superposition processing unit is further configured to perform:
randomly generating the noise characteristic diagram with the same size as the characteristic diagram.
Optionally, when there is one designated neural network layer, the designated neural network layer is a neural network layer before the last convolutional layer in the neural network model.
Optionally, the sample image is obtained from a sample image set, and the sample image set includes images acquired under different shooting conditions.
Optionally, the superposition processing unit is configured to perform:
and carrying out weighted summation processing on the feature values of the same pixel position in the feature map and the noise feature map to obtain the target feature map.
In a fourth aspect, an embodiment of the present application provides an image processing apparatus, including:
a second acquisition unit configured to perform acquisition of an image to be processed;
the second processing unit is configured to input the image to be processed into a pre-trained image processing model to process the skin texture of the image to be processed, so as to obtain an optimized image of the image to be processed with skin noise;
wherein the image processing model is trained according to the image processing model training method of any one of the first aspect.
In a fifth aspect, an embodiment of the present application further provides an electronic device, including:
a processor;
a memory for storing the processor-executable instructions;
wherein the processor is configured to execute the instructions to implement any of the methods as provided in the first aspect of the application, or any of the methods as provided in the second aspect of the application.
In a sixth aspect, an embodiment of the present application further provides a computer-readable storage medium, in which instructions, when executed by a processor of an electronic device, enable the electronic device to perform any one of the methods as provided in the first aspect of the present application, or any one of the methods as provided in the second aspect of the present application.
In a seventh aspect, an embodiment of the present application provides a computer program product comprising a computer program that, when executed by a processor, performs any of the methods as provided in the first aspect of the present application, or any of the methods as provided in the second aspect of the present application.
The technical scheme provided by the embodiment of the application at least has the following beneficial effects:
the application provides an image processing model training method, which comprises the steps of firstly obtaining a sample image, processing skin textures in the sample image to obtain a reference image, then inputting the sample image into a neural network model to obtain a feature map output by a specified neural network layer, inputting a target feature map obtained by overlapping the feature map and a noise superposition map into a next neural network layer to obtain a predicted image of the neural network model on the sample image, and finally training the neural network model by using the loss value between the predicted image and the reference image to obtain the image processing model. According to the method, the image quality of the sample image is improved through a related method in advance, so that the sample image capable of training the neural network model is obtained, and then the generalization capability of the neural network model to the image under different shooting conditions can be improved through superimposing the noise map in the training process, so that the skin texture of the image to be processed can be improved in a self-adaptive mode through the image processing model.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the embodiments of the present application will be briefly described below, and it is obvious that the drawings described below are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
Fig. 1 is a schematic view of an application scenario of an image processing model training method according to an embodiment of the present application;
fig. 2 is a schematic flowchart of an image processing method according to an embodiment of the present application;
FIG. 3 is a schematic structural diagram of a neural network model provided in an embodiment of the present application;
FIG. 4 is a schematic structural diagram of a neural network model provided in an embodiment of the present application;
FIG. 5 is a schematic structural diagram of a neural network model provided in an embodiment of the present application;
FIG. 6 is a schematic flow chart illustrating adding noise information to a processing result of a specific neural network layer according to an embodiment of the present disclosure;
FIG. 7 is a block diagram illustrating an image processing model training apparatus in accordance with an exemplary embodiment;
FIG. 8 is a block diagram illustrating an image processing apparatus according to an exemplary embodiment;
FIG. 9 is a schematic diagram of an electronic device illustrating a method of training an image processing model according to an example embodiment.
Detailed Description
In order to make the technical solutions of the present application better understood by those of ordinary skill in the art, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.
It should be noted that the terms "first," "second," and the like in the description and claims of this application and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the application described herein are capable of operation in sequences other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.
Hereinafter, some terms in the embodiments of the present application are explained to facilitate understanding by those skilled in the art.
(1) In the embodiments of the present application, the term "plurality" means two or more, and other terms are similar thereto.
(2) "and/or" describes the association relationship of the associated objects, meaning that there may be three relationships, e.g., a and/or B, which may mean: a exists alone, A and B exist simultaneously, and B exists alone. The character "/" generally indicates that the former and latter associated objects are in an "or" relationship.
(3) A server serving the terminal, the contents of the service such as providing resources to the terminal, storing terminal data; the server is corresponding to the application program installed on the terminal and is matched with the application program on the terminal to run.
(4) The terminal device may refer to an APP (Application) of a software class, or may refer to a client. The system is provided with a visual display interface and can interact with a user; is corresponding to the server, and provides local service for the client. For software applications, except some applications that are only run locally, the software applications are generally installed on a common client terminal and need to be run in cooperation with a server terminal. After the internet has developed, more common applications include e-mail clients for e-mail receiving and sending, and instant messaging clients. For such applications, a corresponding server and a corresponding service program are required in the network to provide corresponding services, such as database services, configuration parameter services, and the like, so that a specific communication connection needs to be established between the client terminal and the server terminal to ensure the normal operation of the application program.
In a specific practice process, because the texture of the skin in the images acquired by different devices is poor, in the prior art, the texture of the skin in the images is improved by using a skin noise material, or the skin noise is generated by a post-processing algorithm and used for improving the texture of the skin in the images, but the skin noise material is limited and is not suitable for the situation that the texture of the skin is improved by the images in different scenes, and after the skin noise is generated by the post-processing algorithm, noise logics adapted to the different scenes are required to be designed to improve the texture of the skin in the images in the different scenes, so that the operation is complex and the improvement efficiency is low.
The method comprises the steps of firstly obtaining a sample image, processing skin textures in the sample image to obtain a reference image, then inputting the sample image into a neural network model to obtain a feature map output by a specified neural network layer, inputting a target feature map obtained by overlapping the feature map and a noise superposition map into a next neural network layer to obtain a predicted image of the neural network model on the sample image, and finally training the neural network model by utilizing the loss value between the predicted image and the reference image to obtain an image processing model. The image quality of the sample image is improved by a correlation method in advance, so that the sample image capable of training the neural network model is obtained, and then the generalization capability of the neural network model to the image under different shooting conditions can be improved by superposing the noise map in the training process, so that the skin texture of the image to be processed can be improved in a self-adaptive mode by utilizing the image processing model.
After introducing the design concept of the embodiment of the present application, some simple descriptions are provided below for application scenarios to which the technical solution of the embodiment of the present application can be applied, and it should be noted that the application scenarios described below are only used for describing the embodiment of the present application and are not limited. In specific implementation, the technical scheme provided by the embodiment of the application can be flexibly applied according to actual needs.
Fig. 1 is a schematic view of an application scenario of an image processing model training method according to an embodiment of the present application. The application scenario includes a plurality of terminal devices 101 (including terminal device 101-1, terminal device 101-2, … … terminal device 101-n), server 102. The terminal device 101 and the server 102 are connected via a wireless or wired network, and the terminal device 101 includes but is not limited to a desktop computer, a mobile phone, a mobile computer, a tablet computer, a media player, a smart wearable device, a smart television, and other electronic devices. The server 102 may be a server, a server cluster composed of several servers, or a cloud computing center. The server 102 may be an independent physical server, a server cluster or a distributed system formed by a plurality of physical servers, or a cloud server providing basic cloud computing services such as a cloud service, a cloud database, cloud computing, a cloud function, cloud storage, a network service, cloud communication, middleware service, a domain name service, a security service, a CDN, a big data and artificial intelligence platform, and the like.
The server 102 collects a sample image, processes skin texture in the sample image to obtain a reference image, inputs the sample image into the neural network model to obtain a feature map output by a specified neural network layer, inputs a target feature map obtained by overlapping the feature map and a noise superposition map into a next neural network layer for continuous processing to obtain a predicted image of the neural network model on the sample image, and finally trains the neural network model by using a loss value between the predicted image and the reference image to obtain an image processing model. The image processing model is stored in the server 102, so that when the user 1 sends the image to be processed to the server 102 through the terminal device 101-1, the server 102 can process the image to be processed by using the image processing model to obtain an optimized image of the image to be processed, the server 102 sends the optimized image to the terminal device 101-1, and the optimized image of the image to be processed is displayed through the terminal device 101-1.
Of course, the method provided in the embodiment of the present application is not limited to be used in the application scenario shown in fig. 1, and may also be used in other possible application scenarios, and the embodiment of the present application is not limited. The functions that can be implemented by each device in the application scenario shown in fig. 1 will be described in the following method embodiments, and will not be described in detail herein.
To further illustrate the technical solutions provided by the embodiments of the present application, the following detailed description is made with reference to the accompanying drawings and the detailed description. Although the embodiments of the present application provide the method operation steps as shown in the following embodiments or figures, more or less operation steps may be included in the method based on the conventional or non-inventive labor. In steps where no necessary causal relationship exists logically, the order of execution of the steps is not limited to that provided by the embodiments of the present application.
The following describes the technical solution provided in the embodiment of the present application with reference to the application scenario shown in fig. 1.
Referring to fig. 2, an embodiment of the present application provides an image processing method, including the following steps:
s201, acquiring an image to be processed.
S202, inputting the image to be processed into a pre-trained image processing model to process the skin texture of the image to be processed, and obtaining an optimized image of the image to be processed added with skin noise.
The image processing method and the device can adaptively optimize the image through the trained image processing model.
The sample images adopted by the image processing model in the training process are acquired from a sample image set, and the sample image set comprises images acquired under different shooting conditions.
By acquiring images under different shooting conditions, sample images of different scenes can be obtained, various sample images can be obtained, and an image processing model can be better trained by using the various sample images, so that the image to be processed can be adaptively optimized by using the trained image processing model.
In an embodiment of the present application, an image processing model is trained by:
the method comprises the steps of obtaining a sample image, processing skin textures in the sample image to obtain a reference image, and optionally, processing the reference image by improving skin texture after beautifying the sample image through a preset processing method.
And finally, training the neural network model by using the loss value between the predicted image and the reference image to obtain an image processing model.
For example, different noise information is previously superimposed on the sample image by combining different information such as the size of a human face and ambient illumination, so that a reference image obtained after the sample image is subjected to skin texture processing is obtained.
The specific method of the preset processing method for beautifying the original image is not limited, and the method can be adjusted according to the actual application scene.
By adding noise information into the appointed neural network layer of the image processing model, predicted images under more scenes can be obtained, and then the image processing model is trained better, so that the image to be processed can be optimized in a self-adaptive mode by utilizing the trained image processing model, and the image processing efficiency is improved. The random characteristic of skin shallow noise can be learned in the training process of the neural network model, and strip-shaped textures are prevented from being generated in the processing process of skin texture in the image.
In one embodiment of the present application, the neural network model includes at least one designated neural network layer, each designated neural network layer preceding the last convolutional layer of the neural network model.
Because at least one learnable convolutional layer is needed to process the added noise information after the noise information is added into the neural network model, the noise information can be better utilized, and because the skin noise belongs to the shallow feature, the noise can be preferably superposed on the shallow feature before the last convolutional layer in the neural network model, so that the generalization capability of the image processing model can be improved.
Optionally, when there is a designated neural network layer, the designated neural network layer is a preceding neural network layer of a last convolutional layer in the neural network model. Because the noise information is added into the neural network model, at least one learnable convolution layer is needed to process the added noise information, the noise information can be better utilized, and because the skin noise belongs to the shallow feature, the noise information is preferably added into the previous neural network layer of the last convolution layer in the neural network model, the processing capability of the image processing model can be effectively improved, and the processed skin texture is more vivid.
Here, the addition of the noise information to the neural network layer before the last convolutional layer in the neural network model is only an example, and the addition of the noise information to any neural network layer before the last convolutional layer in the neural network model is not limited herein, and may be adjusted according to an actual application scenario.
Exemplarily, as shown in fig. 3, assuming that the neural network model includes a neural network layer 1, a neural network layer 2, …, and a neural network layer n, where n is a positive integer, and the neural network layer n is a convolutional layer of the last layer in the neural network model, when the neural network model includes a specific neural network layer (i.e., the neural network layer n-1), the sample image is input into the neural network model, and after adding noise information 1 to a processing result of the sample image after passing through the neural network layer 1 to the neural network layer n-1, the processing result after adding noise information 1 is input into the neural network layer n, so that the neural network layer n outputs a predicted image of the sample image.
Exemplarily, as shown in fig. 4, still assuming that the neural network model includes a neural network layer 1, a neural network layer 2, …, and a neural network layer n, where n is a positive integer, and the neural network layer n is a last convolutional layer in the neural network model, when the neural network model includes two specified neural network layers (i.e. the neural network layer 2 and the neural network layer n-1), if the neural network layer 3 is a convolutional layer, the sample image is input into the neural network model, after the noise information 2 is added to the processing result of the sample image after passing through the neural network layer 1 to the neural network layer 2, the processing result after adding the noise information 2 is input into the neural network layer 3, the output result of the neural network layer 3 is continuously input into the neural network layer 4 to the neural network layer n-1, after the noise information 3 is added to the output result of the neural network layer n-1, as an input to the neural network layer n, the neural network layer n is caused to output a predicted image of the sample image.
Optionally, the noise information 1, the noise information 2, and the noise information 3 may be the same or different, and are not specifically limited herein, and may be adjusted according to an actual application scenario.
Illustratively, as shown in fig. 5, the neural network model is composed of an Encoder (Encoder) and a decoder (Generator), and it is assumed that the neural network model includes a specific neural network layer, which is located in the neural network model structure before the last convolutional layer in the decoder. And adding the noise information to an output result of the designated neural network layer, and taking the output result after the noise information is added as the input of the last convolution layer in the decoder, so that the last convolution layer in the decoder outputs a prediction image of the sample image.
In an embodiment of the present application, as shown in fig. 6, for the processing result of each designated neural network layer, noise information is added and then input to the next neural network layer for processing, which includes the following steps:
s601, obtaining a characteristic diagram output by the appointed neural network layer, and determining a noise characteristic diagram corresponding to the characteristic diagram output by the appointed neural network layer.
Here, the noise characteristic diagram may be randomly generated, and the size of the noise characteristic diagram is the same as that of the characteristic diagram, and different processing results may be obtained by processing the randomly generated noise characteristic diagram together with the characteristic diagram, so that the neural network model may be trained better.
S602, weighting and summing the characteristic value of the same pixel position in the characteristic diagram output by the appointed neural network layer and the noise characteristic diagram to obtain a target characteristic diagram;
s603, inputting the target characteristic diagram into a next neural network layer of the specified neural network layer.
Specifically, the method comprises the steps of determining a first weight of a noise feature map and determining a second weight of the feature map; and performing weighted summation processing on the noise characteristic diagram and the characteristic diagram based on the first weight and the second weight.
Optionally, the weighted summation process is performed by any one of the following ways:
firstly, adding a product result of a noise characteristic diagram and a first weight and the characteristic diagram;
adding the product result of the feature map and the second weight and the noise feature map;
and thirdly, adding the product of the noise characteristic diagram and the first weight and the product of the characteristic diagram and the second weight.
For example, assuming that the noise feature map is N, the feature map is X, the first weight of the noise feature map N is β, the second weight of the feature map X is α, and the superposition processing result is X', the weighting processing method may be any one of the following methods:
X’＝β*N+X (1)
X’＝N+X*α (2)
X’＝β*N+X*α (3)
the alpha and the beta are learnable parameters in the neural network model, the learning can be continuously updated when the neural network model is trained, and the parameters are stored for application of the model when the training of the neural network model is completed.
The input of the next neural network layer of the specified neural network layer in the neural network model is further determined by respectively determining the weight coefficients of the noise characteristic diagram and the characteristic diagram, so that the training of the neural network model is more accurate.
Fig. 7 is a block diagram illustrating an image processing model training apparatus according to an exemplary embodiment, and referring to fig. 7, the apparatus 700 includes: a first acquisition unit 701, a first processing unit 702, an output unit 703, a superimposition processing unit 704, a prediction unit 705, a calculation unit 706, and a training unit 707.
A first acquisition unit 701 configured to perform acquisition of a sample image;
a first processing unit 702 configured to perform processing on skin texture in the sample image, resulting in a reference image;
an output unit 703 configured to perform inputting the sample image into a neural network model, so as to obtain a feature map output by a specified neural network layer in the neural network model;
a superposition processing unit 704 configured to perform superposition processing on the feature map and the noise feature map to obtain a target feature map;
a prediction unit 705, configured to perform input of the target feature map as an output result of the specified neural network layer to a next neural network layer for continuous processing, so as to obtain a prediction image of the neural network model on the sample image;
a calculation unit 706 configured to perform deriving a loss value from a difference between the prediction image and the reference image;
a training unit 707 configured to perform training of the neural network model based on the loss values, resulting in an image processing model.
Optionally, the neural network model includes at least one designated neural network layer, and each designated neural network layer is before a last convolutional layer of the neural network model.
Optionally, the superposition processing unit 704 is further configured to perform:
randomly generating the noise characteristic diagram with the same size as the characteristic diagram.
Optionally, when there is one designated neural network layer, the designated neural network layer is a neural network layer before the last convolutional layer in the neural network model.
Optionally, the sample image is obtained from a sample image set, and the sample image set includes images acquired under different shooting conditions.
Optionally, the superposition processing unit 704 is configured to perform:
and carrying out weighted summation processing on the feature values of the same pixel position in the feature map and the noise feature map to obtain the target feature map.
Fig. 8 is a block diagram illustrating an image processing apparatus according to an exemplary embodiment, and referring to fig. 8, the apparatus 800 includes: a second acquisition unit 801 and a second processing unit 802.
A second acquisition unit 801 configured to perform acquisition of an image to be processed;
a second processing unit 802, configured to input the image to be processed into a pre-trained image processing model to process the skin texture of the image to be processed, so as to obtain an optimized image of the image to be processed with skin noise added;
wherein the image processing model is trained according to the image processing model training method of any one of the first aspect.
After the interface information switching method and apparatus according to the exemplary embodiment of the present application are introduced, an electronic device according to another exemplary embodiment of the present application is introduced next.
As will be appreciated by one skilled in the art, aspects of the present application may be embodied as a system, method or program product. Accordingly, various aspects of the present application may be embodied in the form of: an entirely hardware embodiment, an entirely software embodiment (including firmware, microcode, etc.) or an embodiment combining hardware and software aspects that may all generally be referred to herein as a "circuit," module "or" system.
In some possible implementations, an electronic device according to the present application may include at least one processor, and at least one memory. Wherein the memory stores program code which, when executed by the processor, causes the processor to perform the steps in the image scaling method according to various exemplary embodiments of the present application described above in the present specification. For example, the processor may perform steps as in the interface information switching method.
The electronic device 130 according to this embodiment of the present application is described below with reference to fig. 9. The electronic device 130 shown in fig. 9 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present application.
As shown in fig. 9, the electronic device 130 is represented in the form of a general electronic device. The components of the electronic device 130 may include, but are not limited to: the at least one processor 131, the at least one memory 132, and a bus 133 that connects the various system components (including the memory 132 and the processor 131).
Bus 133 represents one or more of any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a processor, or a local bus using any of a variety of bus architectures.
The memory 132 may include readable media in the form of volatile memory, such as Random Access Memory (RAM)1321 and/or cache memory 1322, and may further include Read Only Memory (ROM) 1323.
The electronic device 130 may also communicate with one or more external devices 134 (e.g., keyboard, pointing device, etc.), with one or more devices that enable a user to interact with the electronic device 130, and/or with any devices (e.g., router, modem, etc.) that enable the electronic device 130 to communicate with one or more other electronic devices. Such communication may occur via input/output (I/O) interfaces 135. Also, the electronic device 130 may communicate with one or more networks (e.g., a Local Area Network (LAN), a Wide Area Network (WAN), and/or a public network, such as the internet) via the network adapter 136. As shown, network adapter 136 communicates with other modules for electronic device 130 over bus 133. It should be understood that although not shown in the figures, other hardware and/or software modules may be used in conjunction with electronic device 130, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, among others.
In an exemplary embodiment, a computer-readable storage medium comprising instructions, such as memory 132 comprising instructions, executable by processor 131 of apparatus 700 or processor 131 of apparatus 800 to perform the above-described method is also provided. Alternatively, the storage medium may be a non-transitory computer readable storage medium, such as a ROM, a Random Access Memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like.
In an exemplary embodiment, there is also provided a computer program product comprising a computer program which, when executed by the processor 131, implements any of the image processing model training methods as provided herein, or any of the image processing methods as provided herein.
In an exemplary embodiment, aspects of an image processing training model method or an image processing method provided by the present application may also be implemented in the form of a program product including program code for causing a computer device to perform the steps of an interface information switching method according to various exemplary embodiments of the present application described above in this specification when the program product is run on the computer device.
The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
The program product for image scaling of embodiments of the present application may employ a portable compact disc read only memory (CD-ROM) and include program code, and may be run on an electronic device. However, the program product of the present application is not limited thereto, and in this document, a readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
A readable signal medium may include a propagated data signal with readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A readable signal medium may also be any readable medium that is not a readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
Program code embodied on a readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
Program code for carrying out operations of the present application may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the consumer electronic device, partly on the consumer electronic device, as a stand-alone software package, partly on the consumer electronic device and partly on a remote electronic device, or entirely on the remote electronic device or server. In the case of remote electronic devices, the remote electronic devices may be connected to the consumer electronic device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external electronic device (e.g., through the internet using an internet service provider).
It should be noted that although several units or sub-units of the apparatus are mentioned in the above detailed description, such division is merely exemplary and not mandatory. Indeed, the features and functions of two or more units described above may be embodied in one unit, according to embodiments of the application. Conversely, the features and functions of one unit described above may be further divided into embodiments by a plurality of units.
Further, while the operations of the methods of the present application are depicted in the drawings in a particular order, this does not require or imply that these operations must be performed in this particular order, or that all of the illustrated operations must be performed, to achieve desirable results. Additionally or alternatively, certain steps may be omitted, multiple steps combined into one step execution, and/or one step broken down into multiple step executions.
As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable image scaling apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable image scaling apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable image scaling apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable image scaling device to cause a series of operational steps to be performed on the computer or other programmable device to produce a computer implemented process such that the instructions which execute on the computer or other programmable device provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
While the preferred embodiments of the present application have been described, additional variations and modifications in those embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. Therefore, it is intended that the appended claims be interpreted as including preferred embodiments and all alterations and modifications as fall within the scope of the application.
It will be apparent to those skilled in the art that various changes and modifications may be made in the present application without departing from the spirit and scope of the application. Thus, if such modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is intended to include such modifications and variations as well.
Claims (10)
1. An image processing model training method, comprising:
acquiring a sample image;
processing the skin texture in the sample image to obtain a reference image;
inputting the sample image into a neural network model to obtain a characteristic diagram output by a designated neural network layer in the neural network model;
overlapping the characteristic diagram and the noise characteristic diagram to obtain a target characteristic diagram;
taking the target characteristic graph as an output result of the appointed neural network layer, and inputting the output result to a next neural network layer for continuous processing to obtain a predicted image of the neural network model on the sample image;
obtaining a loss value according to the difference between the predicted image and the reference image;
and training the neural network model based on the loss value to obtain an image processing model.
2. The method of claim 1, wherein the neural network model includes at least one of the designated neural network layers, each of the designated neural network layers preceding a last convolutional layer of the neural network model.
3. The method according to claim 1, wherein before the superimposing the feature map and the noise feature map to obtain the target feature map, the method further comprises:
randomly generating the noise characteristic diagram with the same size as the characteristic diagram.
4. The method of claim 2, wherein when there is one of the designated neural network layers, the designated neural network layer is a preceding neural network layer of a last convolutional layer in the neural network model.
5. An image processing method, comprising:
acquiring an image to be processed;
inputting the image to be processed into a pre-trained image processing model to process the skin texture of the image to be processed, and obtaining an optimized image of the image to be processed added with skin noise;
wherein the image processing model is trained according to the method of any one of claims 1-4.
6. An image processing model training apparatus, comprising:
a first acquisition unit configured to perform acquisition of a sample image;
a first processing unit configured to perform processing on skin texture in the sample image, resulting in a reference image;
the output unit is configured to input the sample image into a neural network model to obtain a feature map output by a specified neural network layer in the neural network model;
the superposition processing unit is configured to perform superposition processing on the characteristic diagram and the noise characteristic diagram to obtain a target characteristic diagram;
the prediction unit is configured to input the target feature map as an output result of the designated neural network layer to a next neural network layer for continuous processing to obtain a predicted image of the neural network model on the sample image;
a calculation unit configured to perform deriving a loss value from a difference between the prediction image and the reference image;
and the training unit is configured to perform training on the neural network model based on the loss value to obtain an image processing model.
7. An image processing apparatus characterized by comprising:
a second acquisition unit configured to perform acquisition of an image to be processed;
the second processing unit is configured to input the image to be processed into a pre-trained image processing model to process the skin texture of the image to be processed, so as to obtain an optimized image of the image to be processed with skin noise;
wherein the image processing model is trained according to the image processing model training method of any one of claims 1 to 4.
8. An electronic device, comprising:
a processor;
a memory for storing the processor-executable instructions;
wherein the processor is configured to execute the instructions to implement the image processing model training method of any one of claims 1 to 4 or the image processing method of any one of claim 5.
9. A computer-readable storage medium, wherein instructions in the computer-readable storage medium, when executed by a processor of an electronic device, enable the electronic device to perform the image processing model training method of any one of claims 1 to 4, or the image processing method of any one of claim 5.
10. A computer program product comprising a computer program, characterized in that the computer program, when being executed by a processor, implements the image processing model training method of any one of claims 1 to 4 or the image processing method of any one of claim 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110294804.7A CN113112536A (en) | 2021-03-19 | 2021-03-19 | Image processing model training method, image processing method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110294804.7A CN113112536A (en) | 2021-03-19 | 2021-03-19 | Image processing model training method, image processing method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113112536A true CN113112536A (en) | 2021-07-13 |
Family
ID=76711911
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110294804.7A Pending CN113112536A (en) | 2021-03-19 | 2021-03-19 | Image processing model training method, image processing method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113112536A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114511041A (en) * | 2022-04-01 | 2022-05-17 | 北京世纪好未来教育科技有限公司 | Model training method, image processing method, device, equipment and storage medium |
| CN115631103A (en) * | 2022-10-17 | 2023-01-20 | 北京百度网讯科技有限公司 | Training method and device of image generation model, and image generation method and device |
| CN116342984A (en) * | 2023-05-31 | 2023-06-27 | 之江实验室 | Model training method, image processing method and image processing device |
| CN116580269A (en) * | 2023-07-13 | 2023-08-11 | 荣耀终端有限公司 | Method for training model, method for processing image, electronic device and storage medium |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101739712A (en) * | 2010-01-25 | 2010-06-16 | 四川大学 | Video-based 3D human face expression cartoon driving method |
| CN105763747A (en) * | 2016-05-03 | 2016-07-13 | 成都索贝数码科技股份有限公司 | Mobile terminal for achieving high-quality real-time facial beautification |
| CN107437077A (en) * | 2017-08-04 | 2017-12-05 | 深圳市唯特视科技有限公司 | A kind of method that rotation face based on generation confrontation network represents study |
| CN107862659A (en) * | 2017-10-31 | 2018-03-30 | 广东欧珀移动通信有限公司 | Image processing method, device, computer equipment and computer-readable recording medium |
| CN109685743A (en) * | 2018-12-30 | 2019-04-26 | 陕西师范大学 | Image mixed noise removing method based on noise learning neural network model |
| CN110163827A (en) * | 2019-05-28 | 2019-08-23 | 腾讯科技（深圳）有限公司 | Training method, image de-noising method, device and the medium of image denoising model |
| CN110288547A (en) * | 2019-06-27 | 2019-09-27 | 北京字节跳动网络技术有限公司 | Method and apparatus for generating image denoising model |
| CN110706179A (en) * | 2019-09-30 | 2020-01-17 | 维沃移动通信有限公司 | Image processing method and electronic equipment |
| CN111199233A (en) * | 2019-12-30 | 2020-05-26 | 四川大学 | Improved deep learning pornographic image identification method |
| CN111401411A (en) * | 2020-02-28 | 2020-07-10 | 北京小米松果电子有限公司 | Method and device for acquiring sample image set |
-
2021
- 2021-03-19 CN CN202110294804.7A patent/CN113112536A/en active Pending
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101739712A (en) * | 2010-01-25 | 2010-06-16 | 四川大学 | Video-based 3D human face expression cartoon driving method |
| CN105763747A (en) * | 2016-05-03 | 2016-07-13 | 成都索贝数码科技股份有限公司 | Mobile terminal for achieving high-quality real-time facial beautification |
| CN107437077A (en) * | 2017-08-04 | 2017-12-05 | 深圳市唯特视科技有限公司 | A kind of method that rotation face based on generation confrontation network represents study |
| CN107862659A (en) * | 2017-10-31 | 2018-03-30 | 广东欧珀移动通信有限公司 | Image processing method, device, computer equipment and computer-readable recording medium |
| CN109685743A (en) * | 2018-12-30 | 2019-04-26 | 陕西师范大学 | Image mixed noise removing method based on noise learning neural network model |
| CN110163827A (en) * | 2019-05-28 | 2019-08-23 | 腾讯科技（深圳）有限公司 | Training method, image de-noising method, device and the medium of image denoising model |
| CN110288547A (en) * | 2019-06-27 | 2019-09-27 | 北京字节跳动网络技术有限公司 | Method and apparatus for generating image denoising model |
| CN110706179A (en) * | 2019-09-30 | 2020-01-17 | 维沃移动通信有限公司 | Image processing method and electronic equipment |
| CN111199233A (en) * | 2019-12-30 | 2020-05-26 | 四川大学 | Improved deep learning pornographic image identification method |
| CN111401411A (en) * | 2020-02-28 | 2020-07-10 | 北京小米松果电子有限公司 | Method and device for acquiring sample image set |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114511041A (en) * | 2022-04-01 | 2022-05-17 | 北京世纪好未来教育科技有限公司 | Model training method, image processing method, device, equipment and storage medium |
| CN115631103A (en) * | 2022-10-17 | 2023-01-20 | 北京百度网讯科技有限公司 | Training method and device of image generation model, and image generation method and device |
| CN115631103B (en) * | 2022-10-17 | 2023-09-05 | 北京百度网讯科技有限公司 | Training method and device for image generation model, and image generation method and device |
| CN116342984A (en) * | 2023-05-31 | 2023-06-27 | 之江实验室 | Model training method, image processing method and image processing device |
| CN116342984B (en) * | 2023-05-31 | 2023-08-08 | 之江实验室 | Model training method, image processing method and image processing device |
| CN116580269A (en) * | 2023-07-13 | 2023-08-11 | 荣耀终端有限公司 | Method for training model, method for processing image, electronic device and storage medium |
| CN116580269B (en) * | 2023-07-13 | 2023-09-19 | 荣耀终端有限公司 | Method for training model, method for processing image, electronic device and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113112536A (en) | Image processing model training method, image processing method and device | |
| US20210117776A1 (en) | Method, electronic device and computer readable medium for information processing for accelerating neural network training | |
| US20170255884A1 (en) | System and method for spatial and visual processing in cognitive architecture | |
| CN113627085B (en) | Transverse federal learning modeling optimization method, equipment and medium | |
| JP2020508504A (en) | Iterative multi-scale image generation using neural networks | |
| CN111626956B (en) | Image deblurring method and device | |
| CN113066034A (en) | Face image restoration method and device, restoration model, medium and equipment | |
| CN113538235A (en) | Training method and device of image processing model, electronic equipment and storage medium | |
| CN115272565A (en) | Head three-dimensional model reconstruction method and electronic equipment | |
| CN114792355A (en) | Virtual image generation method and device, electronic equipment and storage medium | |
| CN114863214A (en) | Image generation model training method, image generation device, image generation medium, and image generation device | |
| US20210027064A1 (en) | Parallel video processing neural networks | |
| CN110163401B (en) | Time sequence prediction method, data prediction method and device | |
| CN111475392A (en) | Method, apparatus, electronic device and computer readable medium for generating prediction information | |
| CN116309158A (en) | Training method, three-dimensional reconstruction method, device, equipment and medium of network model | |
| CN113808157B (en) | Image processing method and device and computer equipment | |
| CN111709784B (en) | Method, apparatus, device and medium for generating user retention time | |
| WO2022221205A1 (en) | Video super-resolution using deep neural networks | |
| CN112149426B (en) | Reading task processing method and related equipment | |
| CN113887319A (en) | Three-dimensional attitude determination method and device, electronic equipment and storage medium | |
| CN113554550A (en) | Training method and device of image processing model, electronic equipment and storage medium | |
| CN111353585A (en) | Structure searching method and device of neural network model | |
| CN114140363B (en) | Video deblurring method and device and video deblurring model training method and device | |
| CN115908962B (en) | Training method of neural network, pulse signal reconstruction image generation method and device | |
| CN116757254B (en) | Task processing method, electronic device and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |