CN116912602B

CN116912602B - Training method of image processing model, image processing method and electronic equipment

Info

Publication number: CN116912602B
Application number: CN202311166559.7A
Authority: CN
Inventors: 王敏刚
Original assignee: Honor Device Co Ltd
Current assignee: Honor Device Co Ltd
Priority date: 2023-09-11
Filing date: 2023-09-11
Publication date: 2023-12-15
Anticipated expiration: 2043-09-11
Also published as: CN116912602A

Abstract

The embodiment of the application provides a training method of an image processing model, an image processing method and electronic equipment, and relates to the technical field of image processing. According to the training method of the image processing model, iterative training is conducted through the first neural network, and the image processing model is obtained. Each group of sample images includes a first sample image and a second sample image, the second sample image being an image obtained by performing image quality improvement processing on the first sample image, the second sample image having higher image quality than the first sample image. The detail feature image serves as prior information to enhance the detail feature preserving function of the model. And in specific training, fusing a first sample image and a corresponding detail characteristic image in each group of sample images to be an input value, and iteratively training a first neural network by taking a corresponding second sample image as a target value until convergence to obtain an image processing model. The finally obtained image processing model not only has the image quality improving and processing functions, but also has the detail feature retaining functions.

Description

Training method of image processing model, image processing method and electronic equipment

Technical Field

The present application relates to the field of image processing technologies, and in particular, to a training method for an image processing model, an image processing method, and an electronic device.

Background

In the image processing technology, a Low-Level Low-dimensional image processing technology is commonly used. The low-dimensional image processing technique is an image quality improvement processing technique for processing a low-quality image due to factors such as blurring, a large amount of noise, and the like into a clear, low-noise, high-quality image. The image quality improvement processing technology is usually implemented by using some trained neural networks, and the neural networks with the image quality improvement processing function may be collectively referred to as a Low-Level model.

When the Low-Level model is used for carrying out reduction processing on the Low-dimensional images, elements such as noise and the like are processed, detail features originally existing in the images can be smeared wrongly, and the detail features which are required to be reserved in the images are lost.

Disclosure of Invention

The embodiment of the application provides a training method of an image processing model, the image processing method and electronic equipment, which are used for acquiring the image processing model which is more optimized relative to a Low-Level model.

In order to achieve the above purpose, the embodiment of the present application adopts the following technical scheme:

in a first aspect, a training method of an image processing model is provided, and iterative training is performed by using a first neural network to obtain the image processing model. Specifically, a plurality of sets of sample images are prepared first, each set of sample images including a first sample image and a second sample image, the second sample image being an image obtained by performing image quality improvement processing on the first sample image, the second sample image having an image quality higher than that of the first sample image, that is, the first sample image being different from the second sample image in that the second sample image includes an image quality higher than that of the first sample image, which is mainly used for training the image quality improvement processing function of the image processing model. In addition, the first sample image comprises detail features, and the detail features of the first sample image are required to be extracted when model training is carried out, so that detail feature images are obtained, and the detail feature images are used for training a detail feature retaining function of the model. The detail feature image serves as prior information to enhance the detail feature preserving function of the model. And in specific training, fusing a first sample image and a corresponding detail characteristic image in each group of sample images to be an input value, and iteratively training a first neural network by taking a corresponding second sample image as a target value until convergence to obtain an image processing model. Thus, the first neural network can learn the image quality improvement function through repeated iterative training, and can strengthen the detail feature preservation function through priori information. Then, the finally obtained image processing model has the image quality improvement processing function and the detail feature retaining function.

In one possible implementation of the first aspect, the step of extracting the detail features of the first sample image to obtain a detail feature image includes: inputting the first sample image into a pre-trained detailed feature extraction model to obtain a detailed feature image.

According to the embodiment, the pre-trained detail feature extraction model is obtained, the detail feature images of the first sample images are obtained through the pre-trained detail feature extraction model, the detail features of the images are extracted with high extraction efficiency, and the data size of the samples is expanded.

In one possible implementation of the first aspect, a training scheme of the detail feature extraction model is added. Specifically, a second neural network is first obtained, and the second neural network may be a neural network that has not been trained yet. Preparing a sample image for training the second neural network, based on what is to be trained is a function of the second neural network to extract image detail features. Then, the prepared samples need to include third sample images with detailed features and fourth sample images obtained after extracting the detailed features of each third sample image, and each third sample image corresponds to a fourth sample image. During training, each third sample image is taken as an input value, and the corresponding fourth sample image is taken as a target value, so that the second neural network is iteratively trained until convergence. At this time, the neural network has the function of extracting the detail features of the image, and can be used as a detail feature extraction model. The detail feature extraction model can be independently used in a scene where the detail features of the image need to be extracted, and can also be applied to a scene where the image processing model is trained in the embodiment of the application to assist in acquiring the detail feature images of each first sample image.

In one possible implementation of the first aspect, a solution for performing attention training for dark area detail features is added. Specifically, the electronic device iteratively trains the first neural network by taking the corresponding second sample image as a target value and fusing the first sample image and the corresponding detail characteristic image in each group of sample images as input values until converging, and before the step of obtaining an image processing model, the electronic device also needs to obtain a dark area brightness value threshold value of each first sample image. The dark area brightness value threshold is used for indicating each pixel area in the first sample image to distinguish a bright area and a dark area, wherein the pixel area with the brightness value larger than or equal to the dark area brightness value threshold can be divided into the bright area, and the pixel area with the brightness value smaller than the dark area brightness value threshold can be divided into the dark area. The detailed features of the dark areas in the image are relatively more easily smeared out, so the image processing model needs to pay more attention to the detailed features of the dark areas of the image.

In order to achieve the purpose, the detail characteristic image is segmented by utilizing the dark area brightness value threshold value, and the dark area detail characteristic image is obtained. The process of training by using the first neural network to obtain the image processing model is adjusted as follows: and fusing the first sample image and the dark area detail characteristic image to be input values, and iteratively training the first neural network by taking the second sample image as a target value until convergence to obtain an image processing model. The neural network is trained by taking the dark area detail features as priori information, so that the neural network can pay more attention to the dark area detail features in the iterative learning process, and then the function of retaining the dark area detail features of the trained image processing model is better.

In one possible implementation of the first aspect, a third neural network is utilized to obtain the dark region luminance value threshold. The third neural network may be a pre-trained neural network with a dark area brightness value threshold prediction function, or may be a neural network that has not been trained or has not had a dark area brightness value threshold prediction function. If the third neural network is a neural network with a dark area brightness value prediction function, the dark area brightness value threshold value of the first sample image can be obtained by inputting the first sample image into the third neural network. The dark field luminance value threshold may segment a dark field detail feature image from the detail feature image.

In one possible implementation of the first aspect, the third neural network may also be a neural network that has not been trained, and the third neural network and the first neural network may be jointly trained to enable the third neural network to output a more accurate dark region luminance value threshold. Specifically, the first sample graph is input into a third neural network to obtain a dark area brightness value threshold value, and then the detail characteristic image is segmented by the dark area brightness value threshold value to obtain a dark area detail characteristic image. And taking the first sample image and the dark area detail characteristic image as input values, taking the second sample image as a target value, performing iterative training on the first neural network for a plurality of times, updating a weight matrix of the third neural network and a weight matrix of the first neural network by using a prediction difference value obtained by each iterative training until the first neural network converges, and taking the converged first neural network as an image processing model.

The first neural network and the third neural network are combined to perform repeated iterative training, so that the third neural network can iteratively learn a more accurate dark region brightness value threshold, and meanwhile, the first neural network can pay more attention to dark region detail features according to dark region detail feature images segmented by the more accurate dark region brightness value threshold output by the third neural network, and further training is performed to obtain better detail feature retention capacity.

In one possible implementation of the first aspect, an iterative scheme for joint training of the first neural network and the third neural network is provided. Specifically, the first sample image is input into the third neural network to obtain a dark area brightness value threshold, and the dark area brightness value threshold at this time is not necessarily a brightness value capable of accurately distinguishing a dark area and a bright area, and the third neural network is trained along with multiple iterations. And then dividing the detail characteristic image by using the dark area brightness value threshold value to obtain the dark area detail characteristic image. And fusing the first sample image and the dark area detail characteristic image to be input values, and taking the second sample image as a target value to enter a first neural network to obtain a predicted image. And calculating a predicted difference value of the predicted image and the second sample image by using a loss function of the first neural network, and reversely modifying weight matrixes of the third neural network and the first neural network by using the predicted difference value.

In one possible implementation of the first aspect, the loss functions of the first neural network and the third neural network are the same.

The first neural network and the third neural network are trained in a combined mode, the same loss function is used, the two neural networks can be optimized and converged synchronously as much as possible, and the effect that the dark area detail characteristic recognition and preservation functions of the first neural network are better when the dark area brightness value threshold value of the first neural network is matched with the output comparison of the third neural network after training is achieved.

In one possible implementation of the first aspect, the loss function of the first neural network and the loss function of the third neural network are both average absolute error loss functions. The average absolute error loss function has higher accuracy.

In one possible implementation of the first aspect, the first neural network is a pre-trained Low-Level model, and the Low-Level model has an image quality improvement processing function.

Based on a pre-trained Low-Level model with an image quality improvement processing function, the detail feature image or the dark area detail feature image is used as priori information, and the Low-Level model with the image quality improvement processing function is trained, so that the detail feature or the dark area detail feature in the image quality improvement processing process of the Low-Level model can be focused, and the detail feature retaining function or the dark area detail feature retaining function of the Low-Level model can be improved.

In a second aspect, an image processing method is provided, where the image processing method provided in this embodiment applies an image processing model obtained by the training method of the image processing model provided in the first aspect to process an image. Specifically, a first image to be processed is input into a pre-trained image processing model, and detail features exist in the first image. And then, the image processing model sequentially processes the input first image by utilizing the finally determined weight matrix through each functional layer in the image processing model, wherein the image processing model comprises image quality improvement processing and detail feature preservation processing. After processing, a second image corresponding to the first image is obtained. The image quality improvement processing function and the detail feature preservation processing function based on the image processing model are used for obtaining a second image which is improved in image quality relative to the first image and reserves detail features in the first image.

In one possible implementation of the second aspect, the detail features include dark-area detail features; the dark region detail feature has a luminance value of a pixel region in the first image that is less than a dark region luminance value threshold.

Based on the fact that the dark area detail features are easier to smear, the dark area detail features can be paid more attention to when the image processing model is retrained. Then, when the first image is processed by using the trained image processing model, the obtained second image can identify and retain the detail characteristics of the dark area as far as possible relative to the first image. The pixel region where the dark region detail feature is located, i.e., the dark region, may be a region where the brightness value is less than the dark region brightness value threshold. And the dark area brightness value threshold may be a brightness value threshold for distinguishing dark areas from bright areas, which is determined by a neural network or user-definition.

In a third aspect, an electronic device is provided that includes a memory and a processor, the memory coupled to the processor;

the memory stores computer-executable instructions;

the processor causes the electronic device to perform the training method of the image processing model as in any one of the first aspects.

In one possible implementation, the electronic device is configured to:

acquiring a first neural network and a plurality of groups of sample images; each group of sample images comprises a first sample image and a second sample image, wherein the first sample image comprises detail characteristics, the second sample image is an image obtained by performing image quality improvement processing on the first sample image, and the image quality of the second sample image is higher than that of the first sample image;

extracting the detail features of the first sample image to obtain a detail feature image;

fusing the first sample image and the corresponding detail characteristic image in each group of sample images to be input values, and iteratively training the first neural network until convergence by taking the corresponding second sample image as a target value to obtain the image processing model; the image processing model has an image quality improving processing function and a detail feature retaining function.

In one possible implementation, the electronic device is configured to:

inputting the first sample image into a pre-trained detailed feature extraction model to obtain the detailed feature image.

In one possible implementation, the electronic device is configured to:

acquiring a second neural network and a plurality of third sample images;

extracting detail characteristics of the third sample image to obtain a fourth sample image corresponding to the third sample image;

and taking the third sample image as an input value, taking the fourth sample image as a target value, and iteratively training the second neural network until convergence to obtain the detail feature extraction model.

In one possible implementation, the electronic device is configured to:

acquiring a dark area brightness value threshold value of the first sample image;

dividing the detail characteristic image by using the dark area brightness value threshold value to obtain a dark area detail characteristic image;

and fusing the first sample image and the dark area detail characteristic image to be input values, and iteratively training the first neural network by taking the second sample image as a target value until convergence to obtain the image processing model.

In one possible implementation, the electronic device is configured to:

Acquiring a third neural network;

and inputting the first sample image into the third neural network to obtain the dark area brightness value threshold.

In one possible implementation, the electronic device is configured to:

and taking the first sample image and the detail characteristic image as input values, taking the second sample image as a target value, performing iterative training on the third neural network and the first neural network for a plurality of times, updating the weight matrix of the third neural network and the weight matrix of the first neural network by using a prediction difference value obtained by each iterative training until the first neural network converges, and taking the converged first neural network as the image processing model.

In one possible implementation, the electronic device is configured to:

inputting the first sample image into the third neural network to obtain the dark area brightness value threshold;

dividing the detail characteristic image by using the dark area brightness value threshold value to obtain the dark area detail characteristic image;

fusing the first sample image and the dark area detail characteristic image to be input values, and inputting the second sample image serving as a target value into the first neural network to obtain a predicted image;

And calculating a predicted difference value of the predicted image and the second sample image by using a loss function of the first neural network, and reversely modifying weight matrixes of the third neural network and the first neural network by using the predicted difference value.

In one possible embodiment, the loss functions of the first and third neural networks are the same.

In one possible implementation, the loss function of the first neural network and the loss function of the third neural network are both average absolute error loss functions.

In one possible implementation manner, the first neural network is a pre-trained Low-Level model, and the Low-Level model has an image quality improvement processing function.

In a fourth aspect, an electronic device is provided that includes a memory and a processor, the memory coupled to the processor;

the memory stores an image processing model and computer-executable instructions;

the processor is configured to execute computer-executable instructions stored in the memory, and the electronic device performs the image processing method according to any one of the second aspects.

In a fifth aspect, there is provided a computer readable storage medium having a computer program stored therein, which when run on a computer, causes the computer to perform the training method of the image processing model as in any of the first aspects or the image processing method as in any of the second aspects.

In a sixth aspect, there is provided a computer program product comprising instructions which, when run on a computer, cause the computer to perform the training method of the image processing model as in any of the first aspects or the image processing method as in any of the second aspects.

The technical effects of any one of the design manners of the second aspect to the sixth aspect may be referred to the technical effects of the different design manners of the first aspect, and will not be repeated here.

Drawings

FIG. 1 is a schematic diagram of a process for training a neural network;

FIG. 2 is a schematic diagram of comparison between the front and rear of image processing related to a model training method according to an embodiment of the present application;

FIG. 3 is a schematic flow chart of a model training method according to an embodiment of the present application;

FIG. 4 is a second flow chart of a model training method according to an embodiment of the present application;

FIG. 5 is a third flow chart of a model training method according to an embodiment of the present application;

fig. 6 is a schematic structural diagram of an electronic device according to an embodiment of the present application.

Detailed Description

Exemplary embodiments of the present application will now be described with reference to the accompanying drawings, in which various details of the embodiments of the present application are included to facilitate understanding, and are to be considered merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the application. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.

For the sake of easy understanding, a part of common technical knowledge related to the embodiments of the present application will be described.

Image processing scenes often use Low-dimensional (Low-Level) image processing techniques. The low-dimensional image processing technology processes a low-quality image into a high-quality image, wherein the low-quality image is an image with the characteristics of blurring, more noise, low resolution and the like, and the high-quality image is an image with the characteristics of high definition, less noise, high resolution and the like compared with the low-quality image. The low quality and high quality mentioned here can also be understood as the quality of the next picture. Then, the low-dimensional image processing technique can be interpreted as an image quality improvement processing technique relatively speaking. The image quality improvement processing technology is usually implemented by using some trained neural networks, and the neural networks with the image quality improvement processing function may be collectively referred to as a Low-Level model.

The Low-Level model has many image quality improvement processing scenes, such as noise reduction, super resolution, high dynamic range imaging (High Dynamic Range Imaging, HDR), and the like. Noise reduction refers to reducing noise in a digital image. Super-resolution, i.e., image super-resolution, refers to the recovery of a high-resolution image from a low-resolution image or image sequence. The image super-resolution technology is divided into super-resolution restoration and super-resolution reconstruction, and can be realized by interpolation and the like. High dynamic range imaging refers to a group of techniques used in computer graphics and cinematography to achieve a larger dynamic range of exposure (i.e., larger contrast) than conventional digital image techniques. The image quality improvement process also has other application scenarios, and will not be described in detail.

The essence of the Low-Level model is a trained neural network. Neural networks refer to networks formed by joining together a plurality of individual neural units. Wherein the output of one neural unit may be the input of another neural unit or units. The input of each neural unit may be connected to a local receptive field of a previous layer to extract features of the local receptive field, which may be an area composed of several neural units.

The neural network used to train the Low-Level model typically includes a deep neural network (Deep Neural Network, DNN) or a convolutional neural network (Convolutional Neuron Network, CNN), or the like. Among these, deep neural networks, also referred to as multi-layer neural networks, can be understood as neural networks having multiple hidden layers. The DNNs are divided according to the positions of different layers, and the neural networks inside the DNNs can be divided into three types: input layer, hidden layer and output layer. Typically the first layer is the input layer, the last layer is the output layer, and the intermediate layers are all hidden layers. The layers may be fully connected. That is, any one of the neurons of the i-th layer may be connected to any one of the neurons of the i+1-th layer.

A convolutional neural network is a deep neural network with a convolutional structure. The convolutional neural network comprises a feature extractor consisting of a convolutional layer and a sub-sampling layer, which can be regarded as a filter. The convolution layer refers to a neuron layer in the convolution neural network, which performs convolution processing on an input signal. In the convolutional layer of the convolutional neural network, one neuron may be connected with only a part of adjacent layer neurons. A convolutional layer typically contains a number of feature planes, each of which may be composed of a number of neural elements arranged in a rectangular pattern.

The process of training the neural network to obtain the Low-Level model mainly comprises the following steps: providing a sample image, acquiring a basic neural network, inputting the sample image into the neural network for iterative training, gradually optimizing a weight matrix in the neural network according to a loss function until the loss function converges, so that the weight matrix in the neural network is optimal, and the neural network can be used as a Low-Level model.

Performing model training typically requires multiple sets of sample images, each set comprising two classes of images, one class being a first sample image containing features to be classified and the other class being a second sample image containing target features. The feature to be classified herein refers to a feature that needs to be subjected to processing operations such as recognition, classification, labeling, and the like, for example, a feature having characteristics such as high noise and blurring. The target features are one or more kinds of features to be identified or processed, for example, features with low noise, clarity and the like. The image content of the first sample image and the second sample image are correlated.

Wherein the first sample image is taken as an input value and the second sample image is taken as a target value. The first sample image contains various basic elements possibly included in an image to be processed by the Low-Level model. The second sample image may be an image obtained by processing the first sample image according to the requirement of the user, where the second sample image may further include a target element that needs to be added by the user based on the first sample image. Alternatively, the first sample image may be an image obtained by processing the second sample image according to the requirement of the user, where the first sample image may remove target elements that are not required by the user based on the second sample image. That is, the element content of the first sample image and the second sample image are identical, and the feature attributes may be different.

For example, in the noise reduction scene, the first sample image may be an image containing noise, and the second sample image may be an image obtained by performing noise reduction processing on the first sample image. Alternatively, the second sample image is an image containing no noise, and the first sample image is an image obtained by adding noise to the first sample image. For another example, in the super-division scene, the first sample image is a low-resolution image, and the second sample image is an image obtained by performing super-division processing on the first sample image.

When preparing a sample image of the Low-Level model, a first sample image with Low dimension and Low quality can be acquired by a camera or a network channel, and then a corresponding second sample image is obtained by Low dimension image quality improvement. Of course, the second sample image with high-dimensional and high-quality can be acquired through the camera or the network channel, and then the corresponding first sample image can be obtained through the high-dimensional image degradation processing. The method for performing the low-dimensional image quality improvement process or the high-dimensional image quality reduction process may be various, for example, the method may be performed by using an image processing algorithm, or the method may be performed by the electronic device based on a manual input operation of a user, or the type of labeling may be performed, without limitation.

FIG. 1 is a schematic diagram of a training neural network, which can be applied to train the neural network to obtain a Low-Level model. The process of training the neural network to obtain the Low-Level model mainly comprises the following steps: initializing a weight matrix of the neural network, and inputting training sample data, namely a first sample image, into the neural network as an input value to be processed to obtain a processed image (which can be called a predicted value); the second sample image is taken as an output value (which can be called a target value or a real value), a difference value between a predicted value and the real value is calculated by using a loss function, and then a weight matrix is modified by using the back propagation of the difference value; and then inputting the training sample data into the neural network after modifying the weight matrix, and performing repeated iterative training until the difference value is minimum, namely the neural network converges, wherein the converged neural network is the Low-Level model with the image quality improving processing function.

When the neural network performs iterative training by using the input first sample image and the second sample image, an error back propagation algorithm is adopted to correct parameters of a weight matrix in the initial neural network in the training process, so that the reconstruction error loss of the neural network is smaller and smaller. Specifically, calculating the difference value between the predicted value and the target value obtained by each iterative training, and then reversely transmitting the difference value forward into the neural network to modify the weight matrix of the neural network, performing iterative training again by using the neural network with the weight matrix modified, and continuously updating the weight matrix of the neural network by using the difference value obtained by the iterative training again. Repeating the iterative training for a plurality of times and reversely modifying the weight matrix by the difference value of each iterative training until the obtained difference value of the predicted value and the target value is within the allowable loss range of the loss function, namely the loss function converges, so that the convergence of the neural network is realized by a plurality of iterative training modes, and the neural network with the image quality improvement processing function obtained after training is the Low-Level model. The back propagation algorithm is a back propagation motion that dominates the error loss, and is used to obtain the parameters of the optimal neural network model. Of course, the iterative training may be performed by other similar algorithms for iterative optimization, which is not limited.

As is clear from the above description, in the image quality improvement processing scene, processing of a target element such as noise is generally involved. When processing these target elements, there will typically be similar detail features to the target elements due to the region in the image where the target elements are located. The detail features described herein may include, without limitation, hair filament features of a person, hair features of a skin surface, scar texture, texture features of clothing, and the like. When the Low-Level model is used for restoring the Low-dimensional images, the detail characteristic elements originally existing in the images can be smeared by mistake while the target elements are processed, so that the detail characteristic elements which are originally reserved in the images are lost. Fig. 2 is a schematic diagram showing a comparison of the images before and after the image processing, wherein a in fig. 2 is an image after the processing, and b in fig. 2 is an original image. It can be seen that the P region in the original image has detail features (texture features of the garment shown in fig. 2), while in the processed image, the P' region loses detail features, that is, the detail features originally present in the image are easily and erroneously smeared when the Low-dimensional image is processed by using the Low-Level model.

In addition, the image may be divided into a dark area, which is an area having a relatively small luminance value, and a bright area, which is an area having a relatively large luminance value, according to the luminance value of each pixel in the image. As shown in fig. 2, Z1 may be a bright area and Z2 may be a dark area. Of course, the dark and bright areas are a relative concept, and the brightness threshold for bright and dark area division in different images may be different. As shown in fig. 2, the detail feature of the dark area in the image is more seriously smeared after the image quality improvement process. That is, the detail feature elements of the dark area are more easily smeared, so that the detail feature loss of the dark area is serious, and the image processing effect is poor. Reference herein to dark space detail features refers to detail features of dark spaces present in the image, such as hair filament features in areas where the brightness values are relatively small, hair features of the skin surface, scar texture, clothing texture features, etc.

Based on the above, the embodiment of the application provides an image processing method, which is to train a neural network by taking detailed characteristics of an image as priori information until convergence, so as to obtain an image processing model which is more optimized relative to a Low-Level model. The image processing model obtained by the embodiment of the application not only has the image quality improving and processing function of the Low-Level model, but also pays more attention to and retains the detail characteristics of the image while improving and processing the image quality, and optimizes the image quality improving and processing effect.

Illustratively, the above detailed features may include, but are not limited to: facial features of a person in the image, hair filament features, sweat features of the skin surface, scar texture, hair filaments, clothing texture, etc. Dark space detail features may include, but are not limited to: facial features of a person in the image, hair filament features, sweat features of the skin surface, scar texture, hair filaments, clothing texture, etc. Various detail features in the image can exist in a bright area or a dark area, and the detail features existing in the dark area are easier to smear during image quality improvement processing relative to the detail features existing in the bright area.

The image processing method provided by the embodiment of the application is applied to electronic equipment, a neural network can be operated in the electronic equipment, an image is input into the neural network for iterative training to obtain an image processing model, and the obtained image processing model can be used for image processing operation. The applied electronic devices may include personal computers (Personal Computer, PCs), tablet computers, notebook computers, portable computers (e.g., mobile phones), wearable electronic devices (e.g., smart watches), augmented Reality (Augmented Reality, AR) \virtual Reality (VR) devices, vehicle-mounted computers, etc., and the following embodiments do not limit the specific form of the electronic devices in particular.

The embodiment of the application mainly relates to three application scenes:

first application scenario: and training the neural network which is not trained by taking the detail characteristics of the image as priori information to obtain the image processing model with the image quality improving processing function and the detail retaining function. It should be noted that, the neural network is not trained, which means that the functional layer structure of the neural network has been determined, but the parameters of the weight matrix of the neural network have not been determined through repeated iterative training, and accordingly, the neural network has not been achieved through repeated iterative training to achieve a convergence state.

The second application scenario: and taking the detail characteristics of the image as priori information, and training a pre-trained Low-Level model to optimize so as to obtain the image processing model with the image quality improving and processing functions and detail retaining functions.

Third application scenario: and training the Low-Level model by taking the detail characteristics of the dark areas of the image as priori information to obtain the image processing model with the image quality improving and processing functions and the dark area detail retaining functions.

The following will describe the three application scenarios respectively.

As shown in fig. 3, one of the flow diagrams of the training method of the image processing model provided by the embodiment of the present application is shown in the first application scenario, that is, the detail feature of the image is used as prior information to train the basic neural network, so as to obtain the image processing model with the image quality improving and detail preserving functions.

First, a plurality of sets of sample images are prepared, each set of sample images including a first sample image and a second sample image associated with each other.

The method can acquire a preset number of first sample images, then perform image quality improvement processing on each first sample image to obtain corresponding second sample images, and take each first sample image and the corresponding second sample image as a group of sample images. The content of each first sample image can be as different as possible, so that the diversity of input features is increased, and the learning function of the model obtained through training can be improved.

The first sample image acquired by the electronic device may be a low quality image. It should be noted that detail features may also be included in the low quality image. The low quality image containing detail features here may be: an image containing clothing texture but high noise, an image containing skin scar features of a person but low image sharpness, and the like. The process of performing the image quality improvement process on the portion of the first sample image may include: the electronic equipment receives the label or drawing operation manually input by the user, and performs noise reduction processing, sharpness improvement and the like on the first sample image so as to obtain a corresponding second sample image. In the process of processing the first sample image to obtain the second sample image, the detail features originally existing in the image are kept as much as possible, so that the obtained second sample image also keeps as many detail features as possible.

A neural network is run within the electronic device, the neural network being a neural network that has not been trained. For ease of distinguishing descriptions, the neural network that has not been trained may be defined as a first neural network. The first neural network includes at least an input layer, an hidden layer, and an output layer. The hidden layers may include full connection layers, convolution layers, pooling layers, activation layers, batch normalization layers, etc., which play different roles in feature extraction.

The full connection layer is that all nodes of the layer are connected with the input nodes. Very few neural networks typically use fully connected layers. The common practice is that the full connection layer is used in the final stage of the deep neural network, is used for connecting the convolutional network or the cyclic neural network, and is used for synthesizing the information extracted by the previous network and mapping the information to the classification space. The convolution layer is a key component of the neural network, the convolution layer reserves space position information, and the correlation of space parts in natural images is fully utilized. Multiple convolution kernels in a convolution layer can be considered as uncorrelated feature extractors, which can express multiple features well. The pooling layer, also called downsampling layer, is mainly used to reduce the dimension of features and preserve valid information. Pooling has two main operations, namely maximum pooling and average pooling. The maximum pooling is to use the maximum value in the neighborhood as the sampling value, and the average pooling is to calculate the average value in the neighborhood as the sampling value. In addition to sampling, the pooling layer also ensures some translational and rotational invariance. The active layer introduces nonlinearity, and on the premise that the neural network can approach any function based on nonlinearity, the complex reality problem is solved by introducing nonlinearity, for example, common active functions include Sigmoid, reLU and the like. There is also a batch normalization layer (Batch Normalization) that converts the interlayer output into a standard format for neural networks, called normalization. Thus, the distribution of the output of the previous layer can be effectively reset, so that the output of the previous layer can be more effectively processed by the subsequent layer, and the learning speed and the accuracy of the whole neural network are improved. Of course, the neural network may also include other functional layers, which are not described in detail.

Another key step is that the electronic device needs to acquire the detail features of the first sample image as prior information for model training using the first neural network before training the first neural network. The detail features of the first sample image described herein may also include, but are not limited to: facial features of a person, scar texture, hair filaments, clothing texture, etc.

The electronic device acquires the detail features of the first sample image, which can be acquired through a detail feature extraction model. The detail feature extraction model may be a trained detail feature extraction model, and the electronic device may load a trained detail feature extraction model, and input the first sample image into the detail feature extraction model to obtain a detail feature image.

Of course, the electronic device may also run a new neural network, defined as a second neural network, and train the second neural network to obtain a detail feature extraction model with a detail feature extraction function. Specifically, a plurality of groups of third sample images containing detail features are prepared, and the detail features of each third sample image are extracted as corresponding fourth sample images. The step of training the second neural network to obtain the detailed feature extraction model may include:

and taking the third sample image as an input value, taking the fourth sample image as a target value, and iteratively training the second neural network until convergence to obtain a detail feature extraction model.

The third sample image is input into the second neural network, and the second neural network is iteratively trained in a reverse direction with the fourth sample image as a target value until the second neural network converges. After the second neural network converges, a detailed feature extraction model can be obtained. And inputting each first sample image into the trained detailed feature extraction model to obtain a detailed feature image.

The training method of the image processing model provided by the embodiment of the application uses the concept of the prior information auxiliary model training. In short, the prior information is the experience of artificial summarization, and the detail characteristic image and the first sample image are combined and input in the dimension of the image channel, so that the neural network focuses on. Specifically, the prior information refers to one of multiple types of information originally existing in the image, and the one type of information is information which a user expects a neural network to pay attention to. And extracting the information as priori information, combining the sample information as a new input value, and inputting the new input value into a neural network for training. The prior information is utilized to combine the input values to train the neural network, which essentially expands the input channels by the principle of information superposition, so that the neural network can focus on the part of information to be superposed. Then, the neural network can automatically learn the overlapped characteristics in a focus manner in the training process, and the model obtained by training the neural network correspondingly has the capability of better processing prior information.

For example, in the process of obtaining the image recognition model of the beagle by training the neural network, the information of the sharp ear can be used as prior information to train the neural network in consideration of the characteristic that the beagle has a longer sharp ear. For example, the neural network may be trained by fusing Ma Quan whole body images as new input values with the sharp ear images of the equine dogs as a priori information. Therefore, the neural network can focus on the priori information of the sharp ear, and the accuracy of the training-obtained identification model of the puppy is higher.

The neural network is trained by using the priori information, so that the neural network can be effectively prevented from being fitted with some unimportant features and sinking into local unimportant features when the features are automatically learned. By adding some manually selected prior information, the neural network learns the key features to obtain a model with more optimized functions.

After the preparation is completed, training of the first neural network can be started to obtain an image processing model. When the first neural network is trained, the detail features are used as priori information. Specifically, as shown in fig. 3, the detailed feature images of the first sample images are first used as prior information, fused with the first sample images to form new input values, and the new input values are input into the first neural network acquired in advance, and the second sample images are used as target values to perform iterative training. And sequentially inputting a plurality of groups of sample images into a first neural network for iterative training until the first neural network converges, and obtaining a required image processing model.

According to the training method of the image processing model, the detail characteristic image is added to the input end of the model to serve as priori information, the detailed characteristic image is fused with the original sample image to be input, and the model is trained in a mode of synchronously training an image quality improving processing function and a detail characteristic retaining function. And a training branch serving as priori information is added to the detail characteristics, so that model training is realized, and quality improvement training and detail retention training are realized at the same time. Thus, the Low-Level model with the image quality improving and processing functions can be obtained, and the detail feature retaining function of the Low-Level model is improved.

Fig. 4 is a second flow chart of a training method of an image processing model according to an embodiment of the present application, which shows the second application scenario described above, in which the Low-Level model is trained to be optimized by using detailed features of an image as prior information, so as to obtain the image processing model with an image quality improving and detail preserving function.

And running a Low-Level model in the electronic equipment, wherein the Low-Level model can perform basic image quality improvement processing operation.

In addition, the electronic device acquires a detail characteristic image of the first sample image as prior information of model training. The trained detail feature extraction model can be loaded in the electronic equipment, and the first sample image is input into the detail feature extraction model to obtain a detail feature image. Of course, the electronic device may also store a new neural network, defined as the second neural network. A plurality of groups of sample images containing detail features and sample images not containing detail features are prepared to train a second neural network, and a detail feature extraction model is obtained. And inputting each first sample image into the trained detailed feature extraction model to obtain a detailed feature image.

Then, the first sample image acquired by the electronic device may be a low-quality image including detail features, and the process of performing image quality improvement processing on the part of the first sample image to obtain a second sample image may include: the electronic equipment receives the label or drawing operation manually input by the user, and performs noise reduction processing, sharpness improvement and the like on the first sample image so as to obtain a corresponding second sample image. In the process of processing the first sample image to obtain the second sample image, the detail features of the image are kept as much as possible, so that the obtained second sample image also keeps as many detail features as possible.

As shown in fig. 4, the detail feature images of the first sample images are first used as prior information, fused with the first sample images to form new input values, a preloaded Low-Level model is input, and the second sample images are used as target values to perform repeated iterative training. And sequentially inputting a plurality of groups of sample images into the Low-Level model for iterative training until the Low-Level model converges, and obtaining a required image processing model.

According to the training method for the image processing model, provided by the embodiment of the application, the detail characteristic image is added at the input end of the model to serve as priori information, and the obtained image is fused with the original sample image to serve as a new input value to train the existing Low-Level model, so that the detail characteristic retaining function of the Low-Level model is optimized. Thus, the Low-Level model with the image quality improving and processing functions can be obtained, and the detail feature retaining function of the Low-Level model is improved.

Fig. 5 is a third flow chart of a training method of an image processing model according to an embodiment of the present application, which shows the third application scenario, that is, training a Low-Level model to obtain an image processing model with an image quality improving function and a dark area detail preserving function by using dark area detail features of an image as prior information. According to the training method of the image processing model, provided by the embodiment of the application, the dark area detail characteristics of the image are used as priori information, and the image processing model with the image quality improving processing function and the dark area detail retaining function is obtained through training.

And running a Low-Level model in the electronic equipment, wherein the Low-Level model can perform basic image quality improvement processing operation. The Low-Level model needs to be subjected to secondary optimization training to obtain an image processing model with the function of retaining image detail characteristics. The Low-Level model may include a denoising model, a deblurring model, a super-division model, etc., and the implementation structure may refer to LEDNet or NAFNet.

In addition, the electronic device may further store another neural network that has not been trained, defined as a third neural network, for learning a dark region luminance value threshold of the image to obtain a dark region luminance value threshold prediction model having a dark region luminance value threshold function of the predicted image. In a specific example, the functional layer structure of the third neural network may sequentially include: convolution layer (Conv), batch normalization layer (batch_norm, BN), activation function layer (Rectified Linear Unit, reLU), convolution layer, batch normalization layer, activation function layer, convolution layer, mathematical function layer (Sigmoid, mathematical function with Sigmoid curve), and the like. The convolution layer can be used for extracting features by establishing image local correlation, and the batch normalization layer can be used for realizing internal normalization of a feature network and reducing internal covariance drift. The activation function layer can perform nonlinear linearization on the linear features, and the expression capacity of the features is increased. And the mathematical function layer is used for converting the characteristics into coefficients with mathematical significance.

The third neural network and the Low-Level model can be synchronously trained, and the same loss function is used to ensure that the third neural network and the Low-Level model are converged simultaneously, so that a corresponding dark area brightness value threshold prediction model and an image processing model are respectively obtained. It should be noted that, the synchronous training herein refers to the associated iterative training, that is, each iterative training firstly trains the third neural network, then trains the Low-Level model, and then uses the error of the Low-Level model to reversely modify the weight matrix of the third neural network and the Low-Level model.

The same loss function mentioned here may be l1_loss, i.e. the mean absolute error (Mean Absolute Error, MAE) loss function, which refers to the average of the distances between the model predictor f (x) and the true value y. The L1_loss has stable gradient no matter what input value, does not cause gradient explosion problem, and has a solution with relatively robustness. Of course l2_loss, i.e. the mean square error (Mean Square Error, MSE) loss function, can also be used, being the average of the squares of the differences between the model predicted value f (x) and the sample true value y. Each point of the L2_loss is continuous and smooth, so that the derivation is convenient, and the solution is stable. When the predicted value and the true value differ greatly, l1_loss may be used. If the difference between the predicted value and the true value is small, for example, the absolute value difference is smaller than 1, L2 loss may be used.

In addition, a trained detailed feature extraction model can be loaded in the electronic equipment, and the detailed feature extraction model is used for extracting detailed features of images and is used as priori information of model training.

The electronic device may further provide a plurality of sets of sample images, including a low-quality first sample image including detail features, and the process of performing image quality enhancement processing on the first sample image to obtain a second sample image may include: the electronic equipment receives the label or drawing operation manually input by the user, and performs noise reduction processing, sharpness improvement and the like on the first sample image so as to obtain a corresponding second sample image. In the process of processing the first sample image to obtain the second sample image, the detail features of the image are kept as much as possible, so that the obtained second sample image also keeps as many detail features as possible. As shown in fig. 5, the model training process mainly includes:

step 1, obtaining a dark area brightness value threshold. The first sample image is input into a third neural network, and a dark region brightness value threshold is output. It should be noted that, the dark area luminance value threshold herein is only a predicted value that is initially output, and it is not necessarily able to accurately predict the dark area luminance value threshold of the first sample image. The electronic equipment optimizes the third neural network through repeated iterative training carried out on the combination of the third neural network and the Low-Level model, and improves the accuracy of the output dark area brightness value threshold value until the third neural network converges.

And step 2, acquiring a detail characteristic image. Inputting the first sample image into a detail characteristic extraction model, and outputting a detail characteristic image.

And step 3, acquiring a dark area detail characteristic image. And dividing the detail characteristic image by using the dark area brightness value threshold value to obtain the dark area detail characteristic image. Specifically, the dark area brightness value threshold obtained in the step 1 may be utilized to obtain the detail texture belonging to the dark area from the detail feature image, and then the non-dark area mask not belonging to the dark area in the detail feature image is set to 0, so as to obtain a dark area detail feature image only retaining the detail features of the dark area.

And 4, fusion training. And fusing the dark area detail characteristic image serving as priori information with the first sample image to obtain a new input value. And inputting the new input value obtained after fusion into a preloaded Low-Level model, calculating an error by taking the second sample image as a target value, and training a third neural network and the Low-Level model by utilizing the reverse iteration of the error.

Specifically, the electronic device may perform multiple iterative training on the third neural network and the first neural network with the first sample image and the detail feature image as input values and the second sample image as a target value, and update the weight matrix of the third neural network and the weight matrix of the first neural network with the predicted difference value obtained by each iterative training until the first neural network converges, and use the converged first neural network as the image processing model. The method comprises the steps of taking a first sample image and a detail characteristic image as input values, taking a second sample image as a target value, performing iterative training on a third neural network and a first neural network, and updating a weight matrix of the third neural network and a weight matrix of the first neural network by using a prediction difference value obtained by each iterative training, wherein the steps comprise:

Inputting the first sample image into a third neural network to obtain a dark area brightness value threshold;

dividing the detail characteristic image by using a dark area brightness value threshold value to obtain a dark area detail characteristic image;

fusing the first sample image and the dark area detail characteristic image to be input values, and inputting a second sample image serving as a target value into a first neural network to obtain a predicted image;

and calculating a predicted difference value of the predicted image and the first sample image by using a loss function of the first neural network, and reversely modifying weight matrixes of the third neural network and the first neural network by using the predicted difference value.

And sequentially inputting a plurality of groups of sample images into the Low-Level model for iterative training until the Low-Level model converges, and synchronously converging the third neural network based on the same loss function. Then, the required image processing model can be obtained after the Low-Level model is optimized and trained, and the dark area brightness value threshold prediction model with the image dark area brightness value threshold prediction function can also be obtained after the third neural network is optimized and trained. It should be noted that, the above step 2 may be performed only once for each set of sample images, and the steps 1, 3 and 4 may be repeatedly performed until convergence.

According to the training method for the image processing model, provided by the embodiment of the application, the existing Low-Level model is trained by adding the dark area detail characteristic image as priori information at the input end of the model and fusing the dark area detail characteristic image with the original sample image and then taking the fused dark area detail characteristic image as new input, so that the detail characteristic retaining function of the Low-Level model is optimized. Thus, the Low-Level model with the image quality improving and processing functions can be obtained, and the detail feature retaining function of the Low-Level model is improved.

In addition, the embodiment of the application also provides electronic equipment, which comprises a memory and a processor, wherein the memory is coupled with the processor;

the memory stores computer-executable instructions;

the processor causes the electronic device to perform the training method of the image processing model as provided by the foregoing embodiments.

the processor is configured to execute the computer-executable instructions stored in the memory, and the electronic device executes the image processing method provided in the foregoing embodiment.

The electronic device includes components for implementing basic functions in addition to main components such as a memory and a processor, which will be described in detail below with reference to fig. 6.

Fig. 6 is a schematic structural diagram of an electronic device 600 according to an embodiment of the present application. The electronic device 600 may include, among other things, a processor 610, an external memory interface 620, a memory 621, a universal serial bus (Universal Serial Bus, USB) interface 630, a charge management module 640, a power management module 641, a battery 642, an antenna 1, an antenna 2, a radio frequency module 650, a communication module 660, an audio module 670, a speaker 670A, a receiver 670B, a microphone 670C, an ear-piece interface 670D, a sensor module 680, keys 690, a motor 691, an indicator 692, a camera 693, a display 694, and a SIM card module 695. The sensor modules 680 may include pressure sensors 680A, gyroscope sensors 680B, barometric pressure sensors 680C, magnetic sensors 680D, acceleration sensors 680E, distance sensors 680F, proximity sensors 680G, fingerprint sensors 680H, temperature sensors 680J, touch sensors 680K, ambient light sensors 680L, bone conduction sensors 680M, and the like.

The illustrated structure of the embodiment of the present application does not constitute a limitation of the electronic device 600. More or fewer components than shown may be included, or certain components may be combined, or certain components may be split, or different arrangements of components. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

The processor 610 may include one or more processing units. For example, the processor 610 may include an application processor (Application Processor, AP), a modem processor, a graphics processor (Graphics Processing Unit, GPU), an image signal processor (Image Signal Processor, ISP), a controller, a memory, a video codec, a digital signal processor (Digital Signal Processor, DSP), a baseband processor, and/or a Neural network processor (Neural-Network Processing Unit, NPU), etc. Wherein the different processing units may be separate devices or may be integrated in one or more processors.

The controller may be a decision maker that directs the various components of the electronic device 600 to coordinate their operations in accordance with instructions. Is the neural and command center of the electronic device 600. The controller generates an operation control signal according to the instruction operation code and the time sequence signal to finish the control of instruction fetching and instruction execution.

A memory may also be provided in the processor 610 for storing instructions and data. In some embodiments, memory in the processor 610 is a cache memory that holds instructions or data that the processor 610 has just used or recycled. If the processor 610 needs to reuse the instruction or data, it may be called directly from memory. Repeated accesses are avoided, reducing the latency of the processor 610 and thus improving the efficiency of the system.

In some embodiments, the processor 610 may include an interface. The interfaces may include an integrated circuit (Inter-Integrated Circuit, I2C) interface, an integrated circuit built-in audio (Inter-Integrated Circuit Sound, I2S) interface, a pulse code modulation (Pulse Code Modulation, PCM) interface, a universal asynchronous receiver transmitter (universal asynchronous receiver/transmitter, UART) interface, a mobile industry processor interface (Mobile Industry Processor Interface, MIPI), a General-Purpose Input/Output (GPIO) interface, a SIM interface, and/or a USB interface, among others.

The I2C interface is a bi-directional synchronous Serial bus, comprising a Serial Data Line (SDL) and a Serial clock Line (Derail Clock Line, SCL). In some embodiments, the processor 610 may contain multiple sets of I2C buses. The processor 610 may be coupled to the touch sensor 680K, charger, flash, camera 693, etc., respectively, through different I2C bus interfaces. For example: processor 610 may couple touch sensor 680K through an I2C interface, causing processor 610 to communicate with touch sensor 680K through an I2C bus interface, implementing the touch functionality of electronic device 600.

The I2S interface may be used for audio communication. In some embodiments, the processor 610 may contain multiple sets of I2S buses. The processor 610 may be coupled to the audio module 670 via an I2S bus to enable communication between the processor 610 and the audio module 670. In some embodiments, the audio module 670 may communicate audio signals to the communication module 660 via the I2S interface to enable phone answering via a bluetooth headset.

PCM interfaces may also be used for audio communication to sample, quantize and encode analog signals. In some embodiments, the audio module 670 and the communication module 660 may be coupled by a PCM bus interface. In some embodiments, the audio module 670 may also transmit audio signals to the communication module 660 via the PCM interface to enable phone answering via the bluetooth headset. Both the I2S interface and the PCM interface may be used for audio communication, the sampling rates of the two interfaces being different.

The UART interface is a universal serial data bus for asynchronous communications. The bus is a bi-directional communication bus. It converts the data to be transmitted between serial communication and parallel communication. In some embodiments, a UART interface is typically used to connect the processor 610 with the communication module 660. For example: the processor 610 communicates with the bluetooth module through a UART interface to implement bluetooth functions. In some embodiments, the audio module 670 may transmit audio signals to the communication module 660 through a UART interface to implement a function of playing music through a bluetooth headset.

The MIPI interface may be used to connect the processor 610 with peripheral devices such as a display 694, a camera 693, and the like. The MIPI interfaces include camera serial interfaces (Camera Serial Interface, CSI), display serial interfaces (Display Serial Interface, DSI), and the like. In some embodiments, processor 610 and camera 693 communicate through a CSI interface to implement the photographing functions of electronic device 600. Processor 610 and display 694 communicate via a DSI interface to implement the display functionality of electronic device 600.

The GPIO interface may be configured by software. The GPIO interface may be configured as a control signal or as a data signal. In some embodiments, a GPIO interface may be used to connect the processor 610 with the camera 693, display 694, communication module 660, audio module 670, sensor module 680, and the like. The GPIO interface may also be configured as an I2C interface, an I2S interface, a UART interface, an MIPI interface, etc.

USB interface 630 may be a Mini USB interface, micro USB interface, USB Type C interface, etc. The USB interface 630 may be used to connect a charger to charge the electronic device 600, or may be used to transfer data between the electronic device 600 and a peripheral device. And can also be used for connecting with a headset, and playing audio through the headset. But also for connecting other electronic devices, such as AR devices, etc.

The interface connection relationship between the modules illustrated in the embodiment of the present invention is only schematically illustrated, and does not limit the structure of the electronic device 600. The electronic device 600 may employ different interfacing means, or a combination of interfacing means, in embodiments of the present invention.

The charge management module 640 is used to receive a charge input from a charger. The charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charge management module 640 may receive a charging input of a wired charger through the USB interface 630. In some wireless charging embodiments, the charge management module 640 may receive wireless charging input through a wireless charging coil of the electronic device 600. The charging management module 640 may also provide power to the electronic device through the power management module 641 while charging the battery 642.

The power management module 641 is used for connecting the battery 642, the charge management module 640 and the processor 610. The power management module 641 receives input from the battery 642 and/or the charge management module 640 and provides power to the processor 610, the internal memory 621, the external memory, the display 694, the camera 693, the wireless communication module 660, and the like. The power management module 641 may also be configured to monitor battery capacity, battery cycle times, battery health (leakage, impedance), and other parameters. In other embodiments, the power management module 641 may also be disposed in the processor 610. In other embodiments, the power management module 641 and the charge management module 640 may be disposed in the same device.

The wireless communication functions of the electronic device 600 may be implemented by the antenna 1, the antenna 2, the radio frequency module 650, the communication module 660, the modem, the baseband processor, and the like.

The antennas 1 and 2 are used for transmitting and receiving electromagnetic wave signals. Each antenna in the electronic device 600 may be used to cover a single or multiple communication bands. Different antennas may also be multiplexed to improve the utilization of the antennas. For example: the cellular network antennas may be multiplexed into wireless local area network diversity antennas. In some embodiments, the antenna may be used in conjunction with a tuning switch.

The radio frequency module 650 may provide a communication processing module for a solution including 2G/3G/4G/5G wireless communication applied to the electronic device 600. The radio frequency module 650 may include at least one filter, switch, power amplifier, low noise amplifier (Low Noise Amplifier, LNA), etc. The radio frequency module 650 receives electromagnetic waves from the antenna 1, filters, amplifies, and transmits the received electromagnetic waves to the modem for demodulation. The rf module 650 may amplify the signal modulated by the modem and convert the signal into electromagnetic waves through the antenna 1 to radiate the electromagnetic waves. In some embodiments, at least some of the functional modules of the radio frequency module 650 may be disposed in the processor 610. In some embodiments, at least some of the functional modules of the radio frequency module 650 may be disposed in the same device as at least some of the modules of the processor 610.

The modem may include a modulator and a demodulator. The modulator is used for modulating the low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used for demodulating the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low frequency baseband signal to the baseband processor for processing. The low frequency baseband signal is processed by the baseband processor and then transferred to the application processor. The application processor outputs sound signals through an audio device (not limited to speaker 670A, receiver 670B, etc.), or displays images or video through display 694. In some embodiments, the modem may be a stand-alone device. In some embodiments, the modem may be provided in the same device as the radio frequency module 650 or other functional module, independent of the processor 610.

The communication module 660 may provide a communication processing module that is applied to the electronic device 600 and includes solutions for wireless communication such as wireless local area network (Wireless Local Area Networks, WLAN) (e.g., wireless fidelity (Wireless Fidelity, wi-Fi) network), bluetooth (BT), global navigation satellite system (Global Navigation Satellite System, GNSS), frequency modulation (Frequency Fodulation, FM), near field wireless communication technology (Near Field Communication, NFC), inFrared technology (IR), and the like. The communication module 660 may be one or more devices that integrate at least one communication processing module. The communication module 660 receives electromagnetic waves via the antenna 2, modulates the electromagnetic wave signals, filters the electromagnetic wave signals, and transmits the processed signals to the processor 610. The communication module 660 may also receive a signal to be transmitted from the processor 610, frequency modulate it, amplify it, and convert it to electromagnetic waves for radiation via the antenna 2.

In some embodiments, antenna 1 and radio frequency module 650 of electronic device 600 are coupled, and antenna 2 and communication module 660 are coupled, such that electronic device 600 may communicate with a network and other devices via wireless communication techniques. Wireless communication techniques may include global system for mobile communications (Global System for Mobile communications, GSM), general packet radio service (General Packet Radio Service, GPRS), code division multiple access (Code Division Multiple Access, CDMA), wideband code division multiple access (Wideband Code Division Multiple Access, WCDMA), time division code division multiple access (Time-Division Code Division Multiple Access, TD-SCDMA), long term evolution (Long Term Evolution, LTE), BT, GNSS, WLAN, NFC, FM, and/or IR techniques, among others. The GNSS may include a global satellite positioning system (Satellite Based Augmentation Systems, SBAS), a global navigation satellite system (GLObal NavigAtion Satellite System, GLONASS), a Beidou satellite navigation system (BeiDou navigation Satellite system, BDS), a Quasi zenith satellite system (Quasi-Zenith Satellite System, QZSS) and/or a satellite-based augmentation system (Satellite Based Augmentation Systems, SBAS).

The electronic device 600 implements display functions via a GPU, a display screen 694, and an application processor, etc. The GPU is a microprocessor for image processing, and is connected to the display 694 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 610 may include one or more GPUs that execute program instructions to generate or change display information.

The display 694 is used to display images, video, and the like. The display 694 includes a display panel. The display panel may employ a liquid crystal display (Liquid Crystal Display, LCD), an Organic Light-Emitting Diode (OLED), an Active-matrix Organic Light-Emitting Diode (Active-matrix 6 Organic Light Emitting Diode, AMOLED), a flexible Light-Emitting Diode (Fle Light-Emitting Diode, FLED), miniled, micro-led, micro-OLED, quantum dot Light-Emitting Diode (Quantum dot Light Emitting Diodes, QLED), or the like. In some embodiments, the electronic device 600 may include 1 or N display screens 694, N being a positive integer greater than 1.

The electronic device 600 may implement shooting functions through an ISP, a camera 693, a video codec, a GPU, a display screen, an application processor, and the like.

The ISP is used to process the data fed back by the camera 693. For example, when photographing, the shutter is opened, light is transmitted to the camera photosensitive element through the lens, the optical signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing, so that the electrical signal is converted into an image visible to naked eyes. ISP can also optimize the noise, brightness and chromaticity of the image. The ISP can also optimize parameters such as exposure, color temperature and the like of a shooting scene. In some embodiments, the ISP may be provided in the camera 693.

The camera 693 is used to capture still images or video. The object generates an optical image through the lens and projects the optical image onto the photosensitive element. The photosensitive element may be a charge coupled device (Charge Coupled Device, CCD) or a complementary metal OXide Semiconductor (Complementary Metal-OXide-Semiconductor, CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, which is then transferred to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV, or the like format. In some embodiments, the electronic device 600 may include 1 or N cameras 693, N being a positive integer greater than 1.

The digital signal processor is used for processing digital signals, and can process other digital signals besides digital image signals. For example, when the electronic device 600 is selecting a frequency bin, the digital signal processor is used to fourier transform the frequency bin energy, or the like.

Video codecs are used to compress or decompress digital video. The electronic device 600 may support one or more video codecs. In this way, the electronic device 600 may play or record video in a variety of encoding formats, such as: moving Picture Experts Group (MPEG) 1, MPEG2, MPEG6, MPEG4, etc.

The NPU is a Neural-Network (NN) computing processor, and can rapidly process input information by referencing a biological Neural Network structure, for example, referencing a transmission mode between human brain neurons, and can also continuously perform self-learning. Applications such as intelligent awareness of the electronic device 600 may be implemented through the NPU, for example: image recognition, face recognition, speech recognition, text understanding, etc.

The external memory interface 620 may be used to connect an external memory card, such as a Micro SD card, to implement the memory function of the expansion electronic device 600. The external memory card communicates with the processor 610 through an external memory interface 620 to implement data storage functions. For example, files such as music, video, etc. are stored in an external memory card.

The internal memory 621 may be used to store computer-executable program code that includes instructions. The processor 610 executes instructions stored in the internal memory 621 to thereby perform various functional applications and data processing of the electronic device 600. The memory 621 may include a stored program area and a stored data area. The storage program area may store an application program (such as a sound playing function, an image playing function, etc.) required for at least one function of the operating system, etc. The storage data area may store data created during use of the electronic device 600 (e.g., audio data, phonebook, etc.), and so forth. In addition, the memory 621 may include high-speed random access memory, and may also include nonvolatile memory such as at least one magnetic disk storage device, flash memory device, other volatile solid-state memory device, universal flash memory (Universal Flash Storage, UFS), and the like.

Electronic device 600 may implement audio functions through audio module 670, speaker 670A, receiver 670B, microphone 670C, headphone interface 670D, and an application processor, among others. Such as music playing, recording, etc.

The audio module 670 is used to convert digital audio information to an analog audio signal output and also to convert an analog audio input to a digital audio signal. The audio module 670 may also be used to encode and decode audio signals. In some embodiments, the audio module 670 may be disposed in the processor 610, or some of the functional modules of the audio module 670 may be disposed in the processor 610.

Speaker 670A, also known as a "horn," is used to convert audio electrical signals into sound signals. The electronic device 600 may listen to music, or to hands-free conversations, through the speaker 670A.

A receiver 670B, also known as a "earpiece", is used to convert the audio electrical signal into a sound signal. When electronic device 600 is answering a telephone call or voice message, voice may be received by placing receiver 670B in close proximity to the human ear.

Microphone 670C, also known as a "microphone," is used to convert sound signals into electrical audio signals. When making a call or transmitting voice information, the user can sound near the microphone 670C through the mouth, inputting a sound signal to the microphone 670C. The electronic device 600 may be provided with at least one microphone 670C. In some embodiments, the electronic device 600 may be provided with two microphones 670C, and may implement a noise reduction function in addition to collecting sound signals. In some embodiments, the electronic device 600 may also be provided with three, four, or more microphones 670C to enable collection of sound signals, noise reduction, identification of sound sources, directional recording functions, etc.

The earphone interface 670D is used to connect a wired earphone. The earphone interface 670D may be a USB interface or a 3.5mm open mobile terminal platform (Open Mobile Terminal Platform, OMTP) standard interface, a american cellular telecommunications industry association (Cellular Telecommunications Industry Association of the USA, CTIA) standard interface.

The keys 690 include a power on key, a volume key, etc. The keys 690 may be mechanical keys. Or may be a touch key. The electronic device 600 receives key 690 inputs, generating key signal inputs related to user settings and function controls of the electronic device 600.

The motor 691 may generate a vibration alert. The motor 691 may be used for incoming call vibration alerting as well as for touch vibration feedback. For example, touch operations acting on different applications (e.g., photographing, audio playing, etc.) may correspond to different vibration feedback effects. Touch operations applied to different areas of the display 694 may also correspond to different vibration feedback effects. Different application scenarios (such as time reminding, receiving information, alarm clock, game, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect may also support customization.

The indicator 692 may be an indicator light, which may be used to indicate a state of charge, a change in power, a message, a missed call, a notification, or the like.

The SIM card module 695 is configured to implement a communication function of a SIM card, and the SIM card module 695 may include a SIM card interface, a SIM card circuit, and related auxiliary devices. The SIM card may be inserted into or removed from the SIM card interface to enable contact and separation with the electronic device 600. The electronic device 600 may support 1 or N SIM card interfaces, N being a positive integer greater than 1. The SIM card module 695 may support Nano SIM cards, micro SIM cards, etc. The same SIM card interface can be used to insert multiple cards simultaneously. The types of the plurality of cards may be the same or different. The SIM card interface may also be compatible with different types of SIM cards. The SIM card interface may also be compatible with external memory cards. The electronic device 600 interacts with the network through the SIM card to perform functions such as talking and data communication. In some embodiments, the electronic device 600 employs an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the electronic device 600 and cannot be separated from the electronic device 600.

The embodiment of the application also provides a computer readable storage medium, in which a computer program is stored, which when run on a computer, causes the computer to execute the training method of the image processing model or the image processing method provided by the embodiment.

The embodiments of the present application also provide a computer program product comprising instructions which, when run on a computer, enable the computer to perform the training method, or the image processing method, of the image processing model as provided in the embodiments described above.

The specific implementation manner of the electronic device, the computer readable storage medium, and the computer program product containing the instructions and the technical effects thereof provided in the embodiments of the present application can refer to the training method of the image processing model or the specific implementation process of the image processing method provided in the foregoing embodiments and the technical effects thereof, and are not repeated herein.

In some embodiments, it will be clearly understood by those skilled in the art from the foregoing description of the embodiments, for convenience and brevity of description, only the division of the above functional modules is illustrated, and in practical application, the above functional allocation may be implemented by different functional modules, that is, the internal structure of the apparatus is divided into different functional modules to implement all or part of the functions described above. The specific working processes of the above-described systems, devices and units may refer to the corresponding processes in the foregoing method embodiments, which are not described herein.

The functional units in the embodiments of the present application may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit. The integrated units may be implemented in hardware or in software functional units.

The integrated units, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the embodiments of the present application may be essentially or a part contributing to the prior art or all or part of the technical solution may be embodied in the form of a software product stored in a storage medium, including several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) or a processor to perform all or part of the steps of the methods of the embodiments of the present application. And the aforementioned storage medium includes: flash memory, removable hard disk, read-only memory, random access memory, magnetic or optical disk, and the like.

The foregoing is merely a specific implementation of the embodiment of the present application, but the protection scope of the embodiment of the present application is not limited thereto, and any changes or substitutions within the technical scope disclosed in the embodiment of the present application should be covered by the protection scope of the embodiment of the present application. Therefore, the protection scope of the embodiments of the present application shall be subject to the protection scope of the claims.

Claims

1. A method of training an image processing model, the method comprising:

acquiring a first neural network and a plurality of groups of sample images; each group of sample images comprises a first sample image and a second sample image, wherein the first sample image and the second sample image comprise dark area detail features, the second sample image is an image obtained by carrying out image quality improvement treatment on the first sample image, the image quality of the second sample image is higher than that of the first sample image, the dark area detail features refer to detail features of dark areas in the image, and the dark area detail features comprise facial features, hair filament features, sweat features on the surface of skin, scar texture features and clothes texture features of people in the image;

fusing the first sample image and the corresponding dark area detail characteristic image in each group of sample images to be input values, and iteratively training the first neural network until convergence by taking the corresponding second sample image as a target value to obtain the image processing model; the dark area detail characteristic image is used as priori information, the first neural network is trained to learn the image quality improvement processing function and learn and retain the dark area detail characteristic function, and the image processing model has the image quality improvement processing function and the detail characteristic retention function.

2. The method for training an image processing model according to claim 1, wherein the step of extracting detail features of the first sample image to obtain a detail feature image comprises:

3. The method of training an image processing model according to claim 2, wherein before the step of inputting the first sample image into a pre-trained detailed feature extraction model to obtain the detailed feature image, the method further comprises:

acquiring a second neural network and a plurality of third sample images;

4. A method of training an image processing model as claimed in claim 3, wherein the step of acquiring a dark area luminance value threshold of the first sample image comprises:

acquiring a third neural network;

5. The method for training an image processing model according to claim 4, wherein the step of inputting the first sample image into the third neural network to obtain the dark area luminance value threshold, fusing the first sample image in each set of sample images and the corresponding dark area detail feature image into an input value, and iteratively training the first neural network with the corresponding second sample image as a target value until convergence to obtain the image processing model comprises the steps of:

And taking the first sample image and the dark area detail characteristic image as input values, taking the second sample image as a target value, performing iterative training on the third neural network and the first neural network for a plurality of times, updating the weight matrix of the third neural network and the weight matrix of the first neural network by using a prediction difference value obtained by each iterative training until the first neural network converges, and taking the converged first neural network as the image processing model.

6. The method according to claim 5, wherein the steps of performing a plurality of iterative exercises on the third neural network and the first neural network with the first sample image and the dark area detail feature image as input values and the second sample image as target values, and updating the weight matrix of the third neural network and the weight matrix of the first neural network with a predicted difference value obtained by each iterative exercise, comprise:

7. The method of training an image processing model of claim 6 wherein the first neural network and the third neural network have the same loss function.

8. The method of training an image processing model of claim 7 wherein the loss function of the first neural network and the loss function of the third neural network are both average absolute error loss functions.

9. The training method of an image processing model according to any one of claims 1 to 8, wherein the first neural network is a Low-Level model trained in advance, and the Low-Level model has an image quality improvement processing function.

10. An image processing method, comprising:

Acquiring a first image; wherein the first image includes dark-area detail features;

inputting the first image into a pre-trained image processing model to obtain a second image; wherein the image quality of the second image is higher than that of the first image, the second image includes the dark area detail feature, and the image processing model is an image processing model obtained by the training method of the image processing model according to any one of claims 1 to 9.

11. The image processing method according to claim 10, wherein the detail features include dark-area detail features; the dark region detail feature has a luminance value of a pixel region in the first image that is less than a dark region luminance value threshold.

12. An electronic device comprising a memory and a processor, the memory coupled to the processor;

the memory stores computer-executable instructions;

the processor causes the electronic device to perform the training method of the image processing model according to any one of claims 1 to 9.

13. An electronic device comprising a memory and a processor, the memory coupled to the processor;

the processor is configured to execute computer-executable instructions stored in the memory, and the electronic device performs the image processing method according to claim 10 or 11.

14. A computer-readable storage medium, in which a computer program is stored which, when run on a computer, causes the computer to perform the training method of an image processing model according to any one of claims 1 to 9 or the image processing method according to claim 10 or 11.