CN110766153A - Neural network model training method and device and terminal equipment - Google Patents
Neural network model training method and device and terminal equipment Download PDFInfo
- Publication number
- CN110766153A CN110766153A CN201810847469.7A CN201810847469A CN110766153A CN 110766153 A CN110766153 A CN 110766153A CN 201810847469 A CN201810847469 A CN 201810847469A CN 110766153 A CN110766153 A CN 110766153A
- Authority
- CN
- China
- Prior art keywords
- image
- neural network
- network model
- noise
- initial image
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003062 neural network model Methods 0.000 title claims abstract description 132
- 238000012549 training Methods 0.000 title claims abstract description 105
- 238000000034 method Methods 0.000 title claims abstract description 74
- 238000012545 processing Methods 0.000 claims abstract description 66
- 238000005070 sampling Methods 0.000 claims abstract description 65
- 230000006870 function Effects 0.000 claims description 54
- 238000004590 computer program Methods 0.000 claims description 18
- 230000008569 process Effects 0.000 claims description 18
- 238000007781 pre-processing Methods 0.000 claims description 8
- 238000004379 similarity theory Methods 0.000 claims description 4
- 238000012935 Averaging Methods 0.000 claims description 3
- 230000000694 effects Effects 0.000 abstract description 47
- 238000010586 diagram Methods 0.000 description 8
- 230000005540 biological transmission Effects 0.000 description 6
- 238000013528 artificial neural network Methods 0.000 description 5
- 235000002566 Capsicum Nutrition 0.000 description 4
- 239000006002 Pepper Substances 0.000 description 4
- 241000722363 Piper Species 0.000 description 4
- 235000016761 Piper aduncum Nutrition 0.000 description 4
- 235000017804 Piper guineense Nutrition 0.000 description 4
- 235000008184 Piper nigrum Nutrition 0.000 description 4
- 150000003839 salts Chemical class 0.000 description 4
- 238000004422 calculation algorithm Methods 0.000 description 3
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 238000012544 monitoring process Methods 0.000 description 3
- 238000004891 communication Methods 0.000 description 2
- 238000013500 data storage Methods 0.000 description 2
- 230000007613 environmental effect Effects 0.000 description 2
- 238000003672 processing method Methods 0.000 description 2
- 230000009467 reduction Effects 0.000 description 2
- 230000002238 attenuated effect Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000013527 convolutional neural network Methods 0.000 description 1
- 230000003247 decreasing effect Effects 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000003203 everyday effect Effects 0.000 description 1
- 230000002349 favourable effect Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000008447 perception Effects 0.000 description 1
- 238000013441 quality evaluation Methods 0.000 description 1
- 238000004088 simulation Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Image Processing (AREA)
Abstract
The invention relates to the technical field of image processing, and provides a neural network model training method, a neural network model training device and terminal equipment. The method comprises the following steps: acquiring a neural network model for performing super-resolution processing on an image and an initial image for training the neural network model; adding noise to the initial image; carrying out downsampling processing on the initial image added with the noise to generate a sampling image corresponding to the initial image; and taking the initial image and the corresponding sampling image as training data to train the neural network model. According to the invention, the noise is added to the initial image, so that the generated sampling image is closer to the image containing the noise in practical application, the super-resolution effect of the neural network model in training is ensured to be consistent with the super-resolution effect of the neural network model in practical application, and the super-resolution processing effect of the trained neural network model on the image in practical application is improved.
Description
Technical Field
The invention relates to the technical field of image processing, in particular to a neural network model training method, a neural network model training device and terminal equipment.
Background
The super-resolution image processing method has very wide application. Thanks to the rapid development of networks and various social platforms today, people read, upload and download a very staggering number of pictures on the network every day. In this process, the picture needs to be transmitted from the remote server to the user side of the user.
In the image transmission process, in order to save transmission cost, the image can be subjected to dimension reduction processing at the input end and then subjected to dimension increasing processing at the output end. Therefore, the transmitted data volume can be reduced to a great extent, and the method is favorable for improving the transmission speed and relieving the network transmission pressure. More importantly, the image quality of the output end can be improved and the user experience can be improved under a certain transmission bandwidth.
In addition, the storage of these pictures and videos consumes a lot of storage resources, both for the server side and the client side. Super-resolution techniques can also help to relieve pressure on data storage. The pictures can be stored after dimension reduction, so that the size of the pictures can be reduced, and the storage pressure is relieved. When the pictures need to be viewed and applied, the pictures are quickly subjected to dimension-increasing processing through a super-resolution algorithm, picture details are supplemented, and high-dimensional pictures are obtained.
In addition to data transmission and storage, super-resolution techniques may also be applied in surveillance scenarios. Due to the consideration of the position and height of the monitoring camera and the cost of data storage, most of the existing monitoring images have lower resolution. These low resolution images undoubtedly increase the difficulty of further automatic recognition, detection, etc. These low resolution images can be restored to high resolution using super-resolution image processing methods. The improvement of the picture quality facilitates the further operation of the monitoring pictures according to the requirements.
In addition to the above scenarios, super-resolution techniques can also be used to retrofit older movies. In the past, many classical film and television works are limited by hardware devices such as cameras due to long shooting times, and the film image quality is poor. In particular, when these movies are projected on a modern high-definition screen, the defects of the image quality are particularly prominent. By utilizing the super-resolution technology, the size of the film can be increased, the details of the film can be supplemented, and the film watching experience can be improved.
At present, a neural network model is generally used to perform super-resolution processing on a low-resolution image to obtain a high-resolution image. However, the super-resolution effect of the current neural network model during training is good, and the super-resolution processing effect of the image during actual application is poor.
Disclosure of Invention
In view of this, embodiments of the present invention provide a neural network model training method, an apparatus, and a terminal device, so as to solve the problem that the current neural network model has a good super-resolution effect during training, but has a poor super-resolution effect on an image in practical application.
A first aspect of an embodiment of the present invention provides a neural network model training method, including:
acquiring a neural network model for performing super-resolution processing on an image and an initial image for training the neural network model;
adding noise to the initial image;
carrying out downsampling processing on the initial image added with the noise to generate a sampling image corresponding to the initial image;
and taking the initial image and the corresponding sampling image as training data to train the neural network model.
A second aspect of an embodiment of the present invention provides a neural network model training apparatus, including:
the acquisition module is used for acquiring a neural network model for performing super-resolution processing on an image and an initial image for training the neural network model;
the preprocessing module is used for adding noise to the initial image;
the sampling module is used for carrying out downsampling processing on the initial image added with the noise to generate a sampling image corresponding to the initial image;
and the training module is used for taking the initial image and the corresponding sampling image as training data to train the neural network model.
A third aspect of the embodiments of the present invention provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, where the processor implements the neural network model training method in the first aspect when executing the computer program.
A fourth aspect of embodiments of the present invention provides a computer-readable storage medium storing a computer program which, when executed by a processor, implements the neural network model training method of the first aspect.
Compared with the prior art, the embodiment of the invention has the following beneficial effects: the noise is added into the initial image used for training the neural network model, then the initial image after the noise is added is subjected to down-sampling processing to generate a corresponding sampling image, the initial image and the corresponding sampling image are used as training data, the neural network model is trained, and the effect of performing super-resolution processing on the image in practical application by the trained neural network model can be improved. According to the embodiment of the invention, the noise is added into the initial image, so that the generated sampling image is closer to the image containing the noise in practical application, the super-resolution effect of the neural network model in training is ensured to be consistent with the super-resolution effect of the neural network model in practical application, and the super-resolution processing effect of the trained neural network model on the image in practical application is improved.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the embodiments or the prior art descriptions will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without inventive exercise.
FIG. 1 is a flow chart of an implementation of a neural network model training method provided by an embodiment of the present invention;
FIG. 2 is a flowchart illustrating an implementation of establishing a loss function in a neural network model training method according to an embodiment of the present invention;
FIG. 3 is a diagram illustrating an example of an implementation of pre-processing an initial image according to an embodiment of the present invention;
FIG. 4 is a schematic diagram of the super-resolution effect of the trained neural network model according to the embodiment of the present invention;
FIG. 5 is a schematic diagram of a neural network model training apparatus according to an embodiment of the present invention;
fig. 6 is a schematic diagram of a terminal device according to an embodiment of the present invention.
Detailed Description
In the following description, for purposes of explanation and not limitation, specific details are set forth, such as particular system structures, techniques, etc. in order to provide a thorough understanding of the embodiments of the invention. It will be apparent, however, to one skilled in the art that the present invention may be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present invention with unnecessary detail.
In order to explain the technical means of the present invention, the following description will be given by way of specific examples.
Fig. 1 is a flowchart of an implementation of a neural network model training method according to an embodiment of the present invention, which is detailed as follows:
in S101, a neural network model for performing super-resolution processing on an image and an initial image for training the neural network model are acquired.
In this embodiment, the type of the neural network model for performing super-resolution processing on the image may be selected according to the pixel characteristics of the image to be processed, for example, the neural network model may be a convolutional neural network model (e.g., based on ResNet, FSRCNN, GoogleNet or other similar models), RNN or LSTM neural network, and the like, which is not limited herein.
The input to the neural network model is a low resolution image, with the goal of outputting a high quality, high resolution image (image C, size ma mb). For example, the input is image B, size a × B, the output is image C, size ma × mb, where m is the super-resolution coefficient. If m is 2, the image size is expanded to four times of the original size through super-resolution processing; if m is 3, the image size is enlarged to 9 times as large as the original size by the super resolution processing.
A neural network model for performing super-resolution processing on images can be established firstly, and then network parameters of the neural network model are trained through training data and a deep learning algorithm. The training process is to simulate the relationship between learning data with a complex network structure. Therefore, the quality of the training data largely determines the quality of the neural network model.
In this embodiment, the training data comprises at least one image pair of an initial image and a corresponding sampled image. Wherein the sample image is scaled from the initial image. In this step, an initial image for training the neural network model is first acquired, so that a corresponding sampling image is generated from the initial image in a subsequent step to form training data.
In this embodiment, the high resolution and the low resolution are relative and are not divided by the specific resolution value of the image. For convenience of description, the images are referred to as an initial image and a sample image, and are not limited to the images. For example, an image with a higher resolution in one image pair in the training data is referred to as an initial image, and an image with a lower resolution obtained by reducing the initial image is referred to as a sample image.
In S102, noise is added to the initial image.
In practical applications, an image to be processed by the neural network model is generally an image acquired by an image acquisition device such as a camera. The images have noise due to factors such as a lens, signal conversion, ambient light and the like, and even if the image acquisition device adopts an image processing means to eliminate the noise, part of noise information can be contained in the images, so that the effect of performing super-resolution processing on the images is influenced.
In this embodiment, in order to avoid the influence of noise on the image super-resolution processing, the training data of the neural network model may be processed, so that the image input to the neural network model during the training process also contains noise, which is close to the image containing noise in practical application. The neural network model trained by the training data has a better super-resolution processing effect on images in practical application.
In this step, noise is added to the high resolution image, wherein the noise may be image noise such as gaussian noise, poisson noise, multiplicative noise, salt and pepper noise, and is not limited herein. One kind of noise may be added, or a plurality of kinds of noise may be added.
As an embodiment of the present invention, S102 may include:
and adding noise to the initial image and carrying out fuzzy processing.
In this embodiment, in practical application, there may be blur in the image that needs to be processed by the neural network model, so noise may be added to the initial image, and the blur processing may be performed. The noise may be added to the high resolution image first and then the blurring process is performed, or the blurring process may be performed to the high resolution image first and then the noise is added, which is not limited herein.
In the embodiment, the initial image is subjected to fuzzy processing, fuzzy information is added into the training data, and the generated sampling image is further close to the image in practical application, so that the consistency of the super-resolution effect of the neural network model in training and the super-resolution effect of the neural network model in practical application is ensured, and the image super-resolution effect of the trained neural network model in practical application is further improved.
As an embodiment of the present invention, the noise is of at least one type, and S102 may include:
1) and adding various types of noise to the initial image respectively to obtain noise images corresponding to various types of noise.
In this embodiment, one or more types of noise may be added, for example, only gaussian noise may be added, three types of noise, i.e., gaussian noise, poisson noise, salt and pepper noise, or other one or more types of noise may be added. Noise and noise images correspond one-to-one. For example, adding gaussian noise to the initial image to obtain a noise image corresponding to the gaussian noise; adding the Poisson noise to the initial image to obtain a noise image corresponding to the Poisson noise; and adding the salt and pepper noise to the initial image to obtain a noise image corresponding to the salt and pepper noise.
2) And subtracting the pixel value of each pixel point in the noise image corresponding to each type of noise from the pixel value of the corresponding pixel point in the initial image to obtain the pixel difference value of each pixel point corresponding to each type of noise.
In this embodiment, the pixel values of the noise image corresponding to the noise of the same type and the corresponding pixel points in the initial image are subtracted, so as to obtain the pixel difference value of each pixel point corresponding to the noise of the same type. And respectively calculating the pixel difference value of each pixel point corresponding to each type of noise.
3) And acquiring preset weighted values of various noises, and weighting and averaging pixel difference values of the same pixel point corresponding to various noises to obtain a noise mean value corresponding to each pixel point.
For example, a class a noise, a class B noise and a class C noise are shared, and a pixel difference value of 50 pixel points corresponding to each class of noise is obtained through calculation, so that a noise mean value corresponding to an ith pixel point is equal to a weighted average value of the pixel difference value of the ith pixel point corresponding to the class a noise, the pixel difference value of the ith pixel point corresponding to the class B noise and the pixel difference value of the ith pixel point corresponding to the class C noise, and a preset weight value of each class of noise is a weight in a process of calculating the weighted average value.
4) And adding the noise mean value corresponding to each pixel point to the pixel value of the corresponding pixel point of the initial image.
In this embodiment, the noise mean value corresponding to each pixel point may be added to the pixel value of the corresponding pixel point in the initial image to obtain the pixel value corresponding to each pixel point after the noise mean value is added, so as to obtain the initial image after the noise is added.
In practical applications, an image captured by an image capturing device such as a camera may include a plurality of noises, and the noises included in the captured image may be different according to environmental parameters, capturing parameters, and the like of the captured image. The embodiment respectively calculates the pixel difference value of each pixel point corresponding to various types of noise, then calculates the noise mean value corresponding to each pixel point according to the preset weight value of various types of noise, adds noise to the initial image, can make the finally obtained sampling image more close to the image needing super-resolution processing through the adjustment of the preset weight value, and sets the preset weight value according to the acquisition parameters, the environmental parameters and the like of the image needing super-resolution processing, so that the training method provided by the embodiment can carry out targeted training on the super-resolution processing model of the acquired image under different environments, and the applicability of the training method is enhanced.
In S103, the initial image to which the noise is added is down-sampled to generate a sample image corresponding to the initial image.
In the present embodiment, the down-sampling process is an image sampling method for reducing an image. And performing downsampling processing on the initial image added with the noise to generate a corresponding sampled image.
As an embodiment of the present invention, the "down-sampling processing the initial image after adding the noise" in S103 may include:
and carrying out downsampling processing on the initial image added with the noise according to a neighbor sampling method.
The traditional training method adopts a Bicubic interpolation (Bicubic interpolation) sampling method to carry out down-sampling on the high-resolution image to obtain a corresponding low-resolution image. Each pixel point in the low-resolution image is actually obtained by fusing a plurality of pixel points in the high-resolution image, so that the information content contained in the low-resolution image is far more than the information content corresponding to the actual size of the low-resolution image, and information in a plurality of high-resolution images is hidden in the low-resolution image. In the training of the neural network model according to the high-resolution images and the corresponding low-resolution images, the neural network can mine information hidden in the low-resolution images and learn network parameters, so that the effect of performing super-resolution processing on the low-resolution images in the training data by the trained neural network model is good. In the images acquired by the image acquisition devices such as the camera and the like in practical application, a single pixel point has no hidden information so much, so that the effect of the trained neural network model on super-resolution processing of the images in practical application is much worse than that of training data.
In this embodiment, a neighboring sampling method (Nearest Neighbor) is used instead of the conventional bicubic interpolation sampling method to perform downsampling processing on the initial image after the noise is added. Compared with the sampling image obtained by a bicubic interpolation sampling method, the sampling image obtained by the adjacent sampling method contains too much hidden information, the information is lost in the sampling process, the obtained sampling image contains less hidden information, and the method is closer to the image acquired by an image acquisition device in practical application.
In the embodiment, the initial image added with the noise is subjected to down-sampling processing by adopting a neighbor sampling method, so that the generated sampling image and the image closer to the practical application can ensure the consistency of the super-resolution effect of the neural network model during training and the super-resolution effect of the neural network model in the practical application, and further the image super-resolution effect of the neural network model in the practical application after training is improved.
In S104, the initial image and the corresponding sampling image are used as training data to train the neural network model.
In this embodiment, one initial image and the corresponding sampled image are used as one image pair in the training data, and the training data may include one or more image pairs. Inputting a sampling image into the neural network model, comparing the image output by the neural network model with the initial image corresponding to the sampling image, and adjusting the network parameters according to the comparison result. The optimal network parameters of the neural network model can be determined through training.
According to the embodiment of the invention, the noise is added into the initial image used for training the neural network model, then the downsampling processing is carried out on the initial image added with the noise to generate the corresponding sampling image, the initial image and the corresponding sampling image are used as training data to train the neural network model, and the effect of carrying out super-resolution processing on the image in practical application of the trained neural network model can be improved. According to the embodiment of the invention, the noise is added into the initial image, so that the generated sampling image is closer to the image containing the noise in practical application, the super-resolution effect of the neural network model in training is ensured to be consistent with the super-resolution effect of the neural network model in practical application, and the super-resolution processing effect of the trained neural network model on the image in practical application is improved.
Optionally, when the trained neural network model is actually used to perform super-resolution processing on the image acquired by the image acquisition device, the image may be read from the memory of the image acquisition device and input to the neural network model for super-resolution processing, instead of reading the image from the album of the image acquisition device. Because the image in the memory of the image acquisition device is not compressed, and the information is completely stored, the image in the memory of the image acquisition device replaces the image in the album and is input into the neural network model, and the super-resolution processing effect can be improved.
As an embodiment of the present invention, as shown in fig. 2, "training the neural network model" in S104 may include:
in S201, a loss function is established according to the similarity relationship between the brightness, the contrast and the structure between two images in the structural similarity theory.
In this embodiment, the Structural Similarity Index (SSIM) is a method for measuring Similarity between two images, and provides a fully-referenced image quality evaluation Index, which can measure image Similarity from three aspects of brightness, contrast, and structure. The loss function (loss function) is used for estimating the degree of inconsistency between the predicted value and the true value of the model, and is a non-negative true value function, and the smaller the loss function is, the higher the accuracy of the model is.
As an embodiment of the present invention, the loss function is:
Lossnew(A,C)＝-l(A,C)-c(A,C)-s(A,C) (1)
wherein A represents a first image, C represents a second image, and l (A, C) is a brightness contrast function between the first image and the second image; c (A, C) is a contrast function between the first image and the second image; s (A, C) is a structural contrast function between the first image and the second image; wherein the first image is the initial image; and the second image is an image output by the neural network model after the sampling image corresponding to the initial image is input into the neural network model in the training process.
In this embodiment, the brightness contrast function is used to characterize the brightness similarity between two images. The contrast function is used for representing the degree of contrast similarity between the two images, and the structure contrast function is used for representing the degree of structure similarity between the two images. In this embodiment, the brightness contrast function, the contrast function, and the structure contrast function are added after being negated, so as to obtain the loss function. In the established loss function, the weights of the brightness, the contrast and the structure are all 1, so that the loss function can more comprehensively and uniformly reflect the similarity of the brightness, the contrast and the structure of the two images, and a neural network trained according to the loss function can realize a better super-resolution effect. It will be appreciated that the brightness, contrast and structural weights may also be adjusted.
As an embodiment of the present invention, the brightness contrast function is
The contrast function is
The structural contrast function is
Wherein, muAIs the pixel average value, mu, of the first imageCIs the pixel average, σ, of the second imageAIs the standard deviation, σ, of the pixels of the first imageCIs the standard deviation, σ, of the pixels of the second imageAσCPixel covariance for the first image and the second image; k1，K2And K3Are all constant.
In S202, the neural network model is trained according to the training data and the loss function.
In this embodiment, after the sampling image in the training data is input to the neural network model, a third image output after the neural network model performs super-resolution processing on the sampling image is obtained. And calculating a loss function value corresponding to the current neural network model according to the image information of the initial image, the image information of the third image and the loss function corresponding to the sampling image. And adjusting parameters of the neural network model according to the loss function value corresponding to the current neural network model. If the super-resolution effect of the adjusted neural network model meets the requirement, the training process of the neural network model can be finished, and if the super-resolution effect of the adjusted neural network model does not meet the requirement, the training data is input again for training.
In the embodiment, the idea of the structural similarity theory is integrated into the process of constructing the loss function, the training effect of the neural network model is improved by improving the loss function, and the image output by the trained neural network model can better accord with the super-resolution image of human vision, so that the super-resolution effect of the neural network model is improved.
Fig. 3 is a schematic diagram illustrating an implementation example of preprocessing an initial image according to an embodiment of the present invention. Fig. 3(a) shows an initial image, and fig. 3(b) shows an image obtained by adding noise to the initial image and performing blurring processing.
Fig. 4 is a schematic diagram illustrating a super-resolution effect of the trained neural network model according to the embodiment of the present invention. Fig. 4(a) and (c) are super-resolution images output by a neural network trained by a conventional training method, and fig. 4(b) and (d) are super-resolution images output by a neural network trained by the training method provided in the present embodiment. As can be seen from fig. 4, the neural network model trained by the training method provided by this embodiment has a better super-resolution effect. First, the resulting line segments from fig. 4(b) and (d) are more clear and sharp. Especially, when fig. 4(b) is compared with fig. 4(a), the size of the circle obtained in fig. 4(b) is much smaller than that obtained in the conventional training method, and the conclusion is more intuitively verified. Secondly, the noise on the background of fig. 4(b) and (d) is also greatly attenuated. These two points exactly coincide with the original intention of the concept of the present scheme, and it is proved that the training method of the present embodiment can achieve the purpose of good anticipation, improve the training effect of the neural network model, and enable the trained neural network model to achieve a good super-resolution effect, so that the human eye perception effect is improved.
The embodiment of the invention has the following advantages:
1. the down-sampling mode is changed from the traditional bicubic interpolation sampling method into the neighbor sampling method for the first time. By adopting a neighbor sampling method, the situation that the picture is really shot by the simulation camera is simulated, so that the network is helped to learn more information, and the more real picture is restored.
2. Noise and blur are added to the training data to simulate the hardware noise and blur that the actual captured image may be subjected to. Therefore, the trained neural network model has the functions of denoising and deblurring, and can restore the image details to the maximum extent according to the minimum information. In practical application, better super-resolution effect can be obtained.
3. In actual application, data is taken from the internal memory of the image acquisition device instead of being read from the photo album. Therefore, the image is not compressed, the information is completely stored, and the super-resolution effect of the neural network model is further improved.
According to the embodiment of the invention, the noise is added into the initial image used for training the neural network model, then the downsampling processing is carried out on the initial image added with the noise to generate the corresponding sampling image, the initial image and the corresponding sampling image are used as training data to train the neural network model, and the effect of carrying out super-resolution processing on the image in practical application of the trained neural network model can be improved. According to the embodiment of the invention, the noise is added into the initial image, so that the generated sampling image is closer to the image containing the noise in practical application, the super-resolution effect of the neural network model in training is ensured to be consistent with the super-resolution effect of the neural network model in practical application, and the super-resolution processing effect of the trained neural network model on the image in practical application is improved.
It should be understood that, the sequence numbers of the steps in the foregoing embodiments do not imply an execution sequence, and the execution sequence of each process should be determined by its function and inherent logic, and should not constitute any limitation to the implementation process of the embodiments of the present invention.
Fig. 5 is a schematic diagram of a neural network model training apparatus according to an embodiment of the present invention, corresponding to the neural network model training method described in the foregoing embodiment. For convenience of explanation, only the portions related to the present embodiment are shown.
Referring to fig. 5, the apparatus includes an acquisition module 51, a preprocessing module 52, a sampling module 53, and a training module 54.
An obtaining module 51, configured to obtain a neural network model for performing super-resolution processing on an image and an initial image for training the neural network model.
A pre-processing module 52 for adding noise to the initial image.
And the sampling module 53 is configured to perform downsampling processing on the initial image to which the noise is added, and generate a sampled image corresponding to the initial image.
And a training module 54, configured to train the neural network model by using the initial image and the corresponding sampling image as training data.
Optionally, the preprocessing module 52 is configured to:
and adding noise to the initial image and carrying out fuzzy processing.
Optionally, the noise is of at least one type, and the preprocessing module 52 is configured to:
adding various types of noise to the initial image respectively to obtain noise images corresponding to various types of noise;
subtracting the pixel value of each pixel point in the noise image corresponding to each type of noise from the pixel value of the corresponding pixel point in the initial image to obtain the pixel difference value of each pixel point corresponding to each type of noise;
acquiring preset weighted values of various noises, and weighting and averaging pixel difference values of the same pixel point corresponding to the various noises to obtain a noise mean value corresponding to each pixel point;
and adding the noise mean value corresponding to each pixel point to the pixel value of the corresponding pixel point of the initial image.
Optionally, the sampling module 53 is configured to:
and carrying out downsampling processing on the initial image added with the noise according to a neighbor sampling method.
Optionally, the training module 54 is configured to:
establishing a loss function according to the similarity relation between the brightness, the contrast and the structure of two images in the structural similarity theory;
and training the neural network model according to the training data and the loss function.
Optionally, the loss function is:
Lossnew(A,C)＝-l(A,C)-c(A,C)-s(A,C)
wherein A represents a first image, C represents a second image, and l (A, C) is a brightness contrast function between the first image and the second image; c (A, C) is a contrast function between the first image and the second image; s (A, C) is a structural contrast function between the first image and the second image; wherein the first image is the initial image; and the second image is an image output by the neural network model after the sampling image corresponding to the initial image is input into the neural network model in the training process.
Optionally, the brightness contrast function is
The contrast function is
The structural contrast function is
Wherein, muAIs the pixel average value, mu, of the first imageCIs the pixel average, σ, of the second imageAIs the standard deviation, σ, of the pixels of the first imageCIs the standard deviation, σ, of the pixels of the second imageAσCPixel covariance for the first image and the second image; k1，K2And K3Are all constant.
According to the embodiment of the invention, the noise is added into the initial image used for training the neural network model, then the downsampling processing is carried out on the initial image added with the noise to generate the corresponding sampling image, the initial image and the corresponding sampling image are used as training data to train the neural network model, and the effect of carrying out super-resolution processing on the image in practical application of the trained neural network model can be improved. According to the embodiment of the invention, the noise is added into the initial image, so that the generated sampling image is closer to the image containing the noise in practical application, the super-resolution effect of the neural network model in training is ensured to be consistent with the super-resolution effect of the neural network model in practical application, and the super-resolution processing effect of the trained neural network model on the image in practical application is improved.
Fig. 6 is a schematic diagram of a terminal device according to an embodiment of the present invention. As shown in fig. 6, the terminal device 6 of this embodiment includes: a processor 60, a memory 61 and a computer program 62, e.g. a program, stored in said memory 61 and executable on said processor 60. The processor 60, when executing the computer program 62, implements the steps in the various method embodiments described above, such as the steps 101 to 104 shown in fig. 1. Alternatively, the processor 60, when executing the computer program 62, implements the functions of the modules/units in the above-mentioned device embodiments, such as the functions of the modules 51 to 54 shown in fig. 5.
Illustratively, the computer program 62 may be partitioned into one or more modules/units that are stored in the memory 61 and executed by the processor 60 to implement the present invention. The one or more modules/units may be a series of computer program instruction segments capable of performing specific functions, which are used to describe the execution of the computer program 62 in the terminal device 6.
The terminal device 6 may be a desktop computer, a notebook, a palm computer, a cloud server, or other computing devices. The terminal device may include, but is not limited to, a processor 60, a memory 61. Those skilled in the art will appreciate that fig. 6 is merely an example of a terminal device 6, and does not constitute a limitation of the terminal device 6, and may include more or less components than those shown, or some components in combination, or different components, for example, the terminal device may also include an input-output device, a network access device, a bus, a display, etc.
The Processor 60 may be a Central Processing Unit (CPU), other general purpose Processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), an off-the-shelf Programmable Gate Array (FPGA) or other Programmable logic device, discrete Gate or transistor logic, discrete hardware components, etc. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
The memory 61 may be an internal storage unit of the terminal device 6, such as a hard disk or a memory of the terminal device 6. The memory 61 may also be an external storage device of the terminal device 6, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), and the like, which are provided on the terminal device 6. Further, the memory 61 may also include both an internal storage unit and an external storage device of the terminal device 6. The memory 61 is used for storing the computer program and other programs and data required by the terminal device. The memory 61 may also be used to temporarily store data that has been output or is to be output.
It will be apparent to those skilled in the art that, for convenience and brevity of description, only the above-mentioned division of the functional units and modules is illustrated, and in practical applications, the above-mentioned function distribution may be performed by different functional units and modules according to needs, that is, the internal structure of the apparatus is divided into different functional units or modules to perform all or part of the above-mentioned functions. Each functional unit and module in the embodiments may be integrated in one processing unit, or each unit may exist alone physically, or two or more units are integrated in one unit, and the integrated unit may be implemented in a form of hardware, or in a form of software functional unit. In addition, specific names of the functional units and modules are only for convenience of distinguishing from each other, and are not used for limiting the protection scope of the present application. The specific working processes of the units and modules in the system may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.
In the above embodiments, the descriptions of the respective embodiments have respective emphasis, and reference may be made to the related descriptions of other embodiments for parts that are not described or illustrated in a certain embodiment.
Those of ordinary skill in the art will appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.
In the embodiments provided in the present invention, it should be understood that the disclosed apparatus/terminal device and method may be implemented in other ways. For example, the above-described embodiments of the apparatus/terminal device are merely illustrative, and for example, the division of the modules or units is only one logical division, and there may be other divisions when actually implemented, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.
The integrated modules/units, if implemented in the form of software functional units and sold or used as separate products, may be stored in a computer readable storage medium. Based on such understanding, all or part of the flow of the method according to the embodiments of the present invention may also be implemented by a computer program, which may be stored in a computer-readable storage medium, and when the computer program is executed by a processor, the steps of the method embodiments may be implemented. Wherein the computer program comprises computer program code, which may be in the form of source code, object code, an executable file or some intermediate form, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, usb disk, removable hard disk, magnetic disk, optical disk, computer Memory, Read-Only Memory (ROM), Random Access Memory (RAM), electrical carrier wave signals, telecommunications signals, software distribution medium, and the like. It should be noted that the computer readable medium may contain other components which may be suitably increased or decreased as required by legislation and patent practice in jurisdictions, for example, in some jurisdictions, computer readable media which may not include electrical carrier signals and telecommunications signals in accordance with legislation and patent practice.
The above-mentioned embodiments are only used for illustrating the technical solutions of the present invention, and not for limiting the same; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; such modifications and substitutions do not substantially depart from the spirit and scope of the embodiments of the present invention, and are intended to be included within the scope of the present invention.
Claims (10)
1. A neural network model training method is characterized by comprising the following steps:
acquiring a neural network model for performing super-resolution processing on an image and an initial image for training the neural network model;
adding noise to the initial image;
carrying out downsampling processing on the initial image added with the noise to generate a sampling image corresponding to the initial image;
and taking the initial image and the corresponding sampling image as training data to train the neural network model.
2. The neural network model training method of claim 1, wherein the adding noise to the initial image comprises:
and adding noise to the initial image and carrying out fuzzy processing.
3. The neural network model training method of claim 1, wherein the noise is of at least one type, the adding noise to the initial image comprising:
adding various types of noise to the initial image respectively to obtain noise images corresponding to various types of noise;
subtracting the pixel value of each pixel point in the noise image corresponding to each type of noise from the pixel value of the corresponding pixel point in the initial image to obtain the pixel difference value of each pixel point corresponding to each type of noise;
acquiring preset weighted values of various noises, and weighting and averaging pixel difference values of the same pixel point corresponding to the various noises to obtain a noise mean value corresponding to each pixel point;
and adding the noise mean value corresponding to each pixel point to the pixel value of the corresponding pixel point of the initial image.
4. The neural network model training method of claim 1, wherein the downsampling process of the initial image after the noise is added comprises:
and carrying out downsampling processing on the initial image added with the noise according to a neighbor sampling method.
5. The neural network model training method of any one of claims 1 to 4, wherein the training of the neural network model comprises:
establishing a loss function according to the similarity relation between the brightness, the contrast and the structure of two images in the structural similarity theory;
and training the neural network model according to the training data and the loss function.
6. The neural network model training method of claim 5, wherein the loss function is:
Lossnew(A,C)＝-l(A,C)-c(A,C)-s(A,C)
wherein A represents a first image, C represents a second image, and l (A, C) is a brightness contrast function between the first image and the second image; c (A, C) is a contrast function between the first image and the second image; s (A, C) is a structural contrast function between the first image and the second image; wherein the first image is the initial image; and the second image is an image output by the neural network model after the sampling image corresponding to the initial image is input into the neural network model in the training process.
7. The neural network model training method of claim 6, wherein the brightness contrast function isThe contrast function is
The structural contrast function is
Wherein, muAIs the pixel average value, mu, of the first imageCIs the pixel average, σ, of the second imageAIs the standard deviation, σ, of the pixels of the first imageCIs the standard deviation, σ, of the pixels of the second imageAσCPixel covariance for the first image and the second image; k1，K2And K3Are all constant.
8. A neural network model training device, comprising:
the acquisition module is used for acquiring a neural network model for performing super-resolution processing on an image and an initial image for training the neural network model;
the preprocessing module is used for adding noise to the initial image;
the sampling module is used for carrying out downsampling processing on the initial image added with the noise to generate a sampling image corresponding to the initial image;
and the training module is used for taking the initial image and the corresponding sampling image as training data to train the neural network model.
9. A terminal device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that the processor implements the steps of the method according to any of claims 1 to 7 when executing the computer program.
10. A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, carries out the steps of the method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810847469.7A CN110766153A (en) | 2018-07-27 | 2018-07-27 | Neural network model training method and device and terminal equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810847469.7A CN110766153A (en) | 2018-07-27 | 2018-07-27 | Neural network model training method and device and terminal equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110766153A true CN110766153A (en) | 2020-02-07 |
Family
ID=69327845
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810847469.7A Pending CN110766153A (en) | 2018-07-27 | 2018-07-27 | Neural network model training method and device and terminal equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110766153A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113569737A (en) * | 2021-07-28 | 2021-10-29 | 合肥综合性国家科学中心人工智能研究院(安徽省人工智能实验室) | Notebook screen defect detection method and medium based on autonomous learning network model |
| CN114359144A (en) * | 2021-12-01 | 2022-04-15 | 阿里巴巴（中国）有限公司 | Image detection method and method for obtaining image detection model |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102360498A (en) * | 2011-10-27 | 2012-02-22 | 江苏省邮电规划设计院有限责任公司 | Reconstruction method for image super-resolution |
| CN105513019A (en) * | 2015-11-27 | 2016-04-20 | 西安电子科技大学 | Method and apparatus for improving image quality |
| CN106355561A (en) * | 2016-08-30 | 2017-01-25 | 天津大学 | TV (total variation) image noise removal method based on noise priori constraint |
| CN106570862A (en) * | 2016-10-25 | 2017-04-19 | 中国人民解放军信息工程大学 | Super-resolution reconstruction quality evaluation method and apparatus thereof |
| CN106663314A (en) * | 2014-09-19 | 2017-05-10 | 英特尔公司 | Real time skin smoothing image enhancement filter |
| CN106686281A (en) * | 2016-06-28 | 2017-05-17 | 南京理工大学 | Fusion circuit board noise suppression performance testing system |
-
2018
- 2018-07-27 CN CN201810847469.7A patent/CN110766153A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102360498A (en) * | 2011-10-27 | 2012-02-22 | 江苏省邮电规划设计院有限责任公司 | Reconstruction method for image super-resolution |
| CN106663314A (en) * | 2014-09-19 | 2017-05-10 | 英特尔公司 | Real time skin smoothing image enhancement filter |
| CN105513019A (en) * | 2015-11-27 | 2016-04-20 | 西安电子科技大学 | Method and apparatus for improving image quality |
| CN106686281A (en) * | 2016-06-28 | 2017-05-17 | 南京理工大学 | Fusion circuit board noise suppression performance testing system |
| CN106355561A (en) * | 2016-08-30 | 2017-01-25 | 天津大学 | TV (total variation) image noise removal method based on noise priori constraint |
| CN106570862A (en) * | 2016-10-25 | 2017-04-19 | 中国人民解放军信息工程大学 | Super-resolution reconstruction quality evaluation method and apparatus thereof |
Non-Patent Citations (3)
| Title |
|---|
| 一只飞鱼FY: "损失函数之SSIM", 《CSDN》 * |
| 戴丽等: "《典型高原湖泊流域生态安全评价与可持续发展战略研究》", 31 August 2008, 云南科技出版社 * |
| 高昭昭等: "基于卷积神经网络的单帧毫米波图像超分辨算法", 《电子信息对抗技术》 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113569737A (en) * | 2021-07-28 | 2021-10-29 | 合肥综合性国家科学中心人工智能研究院(安徽省人工智能实验室) | Notebook screen defect detection method and medium based on autonomous learning network model |
| CN114359144A (en) * | 2021-12-01 | 2022-04-15 | 阿里巴巴（中国）有限公司 | Image detection method and method for obtaining image detection model |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108520504B (en) | End-to-end blurred image blind restoration method based on generation countermeasure network | |
| Ignatov et al. | Dslr-quality photos on mobile devices with deep convolutional networks | |
| CN110675336A (en) | Low-illumination image enhancement method and device | |
| EP4105877A1 (en) | Image enhancement method and image enhancement apparatus | |
| CN111079764B (en) | Low-illumination license plate image recognition method and device based on deep learning | |
| JP7446457B2 (en) | Image optimization method and device, computer storage medium, computer program, and electronic equipment | |
| CN111275626A (en) | Video deblurring method, device and equipment based on ambiguity | |
| CN110136055B (en) | Super resolution method and device for image, storage medium and electronic device | |
| CN112602088B (en) | Method, system and computer readable medium for improving quality of low light images | |
| CN108600783B (en) | Frame rate adjusting method and device and terminal equipment | |
| CN112348747A (en) | Image enhancement method, device and storage medium | |
| CN111784570A (en) | Video image super-resolution reconstruction method and device | |
| WO2014070273A1 (en) | Recursive conditional means image denoising | |
| CN110675334A (en) | Image enhancement method and device | |
| US20220398698A1 (en) | Image processing model generation method, processing method, storage medium, and terminal | |
| CN115496668A (en) | Image processing method, image processing device, electronic equipment and storage medium | |
| CN111429371B (en) | Image processing method and device and terminal equipment | |
| CN113962859A (en) | Panorama generation method, device, equipment and medium | |
| CN110717864B (en) | Image enhancement method, device, terminal equipment and computer readable medium | |
| CN110766153A (en) | Neural network model training method and device and terminal equipment | |
| CN113658065A (en) | Image noise reduction method and device, computer readable medium and electronic equipment | |
| CN110458754B (en) | Image generation method and terminal equipment | |
| CN112614110A (en) | Method and device for evaluating image quality and terminal equipment | |
| WO2023215371A1 (en) | System and method for perceptually optimized image denoising and restoration | |
| CN111222446A (en) | Face recognition method, face recognition device and mobile terminal |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information | ||
| CB02 | Change of applicant information |
Address after: 516006 TCL technology building, No.17, Huifeng Third Road, Zhongkai high tech Zone, Huizhou City, Guangdong Province Applicant after: TCL Technology Group Co.,Ltd. Address before: 516006 Guangdong province Huizhou Zhongkai hi tech Development Zone No. nineteen District Applicant before: TCL Corp. |
|
| REG | Reference to a national code |
Ref country code: HK Ref legal event code: DE Ref document number: 40018676 Country of ref document: HK |
|
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200207 |