CN114897728A

CN114897728A - Image enhancement method and device, terminal equipment and storage medium

Info

Publication number: CN114897728A
Application number: CN202210507441.5A
Authority: CN
Inventors: 杨仁友; 秦浩; 郑凯健; 杨靓; 李日富
Original assignee: Southern Marine Science and Engineering Guangdong Laboratory Zhanjiang
Current assignee: Southern Marine Science and Engineering Guangdong Laboratory Zhanjiang
Priority date: 2022-05-10
Filing date: 2022-05-10
Publication date: 2022-08-12

Abstract

The invention discloses an image enhancement method, an image enhancement device, terminal equipment and a storage medium, wherein the image enhancement method comprises the following steps: acquiring image data to be enhanced; and inputting the image enhancement data to be enhanced into a pre-created image enhancement model for enhancement to obtain enhanced image data, wherein the image enhancement model is obtained based on a generative countermeasure network and combined with a preset enhanced image algorithm for training. The invention solves the problem of low real-time performance of image processing capability and improves the image enhancement efficiency.

Description

Image enhancement method and device, terminal equipment and storage medium

Technical Field

The present invention relates to the field of image processing technologies, and in particular, to an image enhancement method and apparatus, a terminal device, and a storage medium.

Background

In the fields of underwater archaeology, submarine exploration and the like which need to use underwater images with high imaging quality, the underwater image processing device has higher requirements on the image processing capacity of the underwater image processing device. The technology for enhancing the underwater image of the lightweight netting in real time is a key technology for improving the sensing distance, the feature extraction and the visual positioning capability of an underwater inspection robot.

However, in the process of processing the underwater image, if the traditional unsupervised image enhancement method is used, the problems of large calculation amount, complex operation and incapability of being arranged on a GPU exist when a large amount of image data is processed; if a method based on a convolutional neural network is simply used, an underwater image data set is required to have an underwater distorted image and a clear image, and images in water and air are required to be acquired at the same position and under the same parameter.

Therefore, the prior art has low real-time performance on image processing capacity and is not convenient to be arranged on the real-time image processing of the underwater robot; for collecting underwater image data sets, sharp images are not available in many cases.

Disclosure of Invention

The invention mainly aims to provide an image enhancement method, an image enhancement device, terminal equipment and a storage medium, and aims to improve the real-time processing capability of underwater images and the preprocessing capability of large data streams and improve the image enhancement efficiency.

To achieve the above object, the present invention provides an image enhancement method, comprising the steps of:

acquiring image data to be enhanced;

and inputting the image enhancement data to be enhanced into a pre-created image enhancement model for enhancement to obtain enhanced image data, wherein the image enhancement model is obtained based on a generative countermeasure network and combined with a preset enhanced image algorithm for training.

Optionally, the step of inputting the data to be enhanced into a pre-created image enhancement model for enhancement to obtain enhanced data further includes:

creating the image enhancement model specifically comprises:

acquiring an original data set;

establishing a generator;

establishing a discriminator;

constructing a generating type countermeasure network based on the generator and the discriminator;

and training the generative confrontation network based on the multi-scale homomorphic filtering label and the original data set to obtain the image enhancement model.

Optionally, the step of establishing a generator further includes:

inputting a first parameter to the generator, and initializing the generator;

inputting the original data set to the generator to obtain a corresponding pseudo-enhanced data set;

the step of training the generative confrontation network based on the multi-scale homomorphic filtering label and the original data set to obtain the image enhancement model comprises the following steps:

and training the generative countermeasure network based on the multi-scale homomorphic filtering label, the original data set and the pseudo enhancement data set to obtain the image enhancement model.

Optionally, the training the generative countermeasure network based on the multi-scale homomorphic filtering label, the original data set, and the pseudo-enhanced data set to obtain the image enhancement model includes:

enhancing the data in the original data set based on the multi-scale homomorphic filtering label to obtain a corresponding enhanced data set;

and training the generative confrontation network based on the original data set, the enhanced data set and the pseudo enhanced data set to obtain the trained generative confrontation network.

Optionally, the step of enhancing the data in the original data set based on the multi-scale homomorphic filtering tag to obtain a corresponding enhanced data set includes:

calculating a noise reduction item corresponding to the data in the original data set so as to reduce the noise of the data in the original data set;

calculating a spatial distance weight value and an adjacent pixel value weight value from a pixel point corresponding to the data in the original data set to a central point;

calculating to obtain a pixel weight sum based on the adjacent pixel value weight and the spatial distance weight value;

calculating to obtain a reflection image based on the pixel weight sum and the noise reduction term;

calculating to obtain a reflection response image based on the reflection image and a preset Gaussian function so as to carry out multi-scale space construction on the data in the original data set;

and calculating to obtain enhanced data corresponding to the data in the original data set based on the reflection response image and the noise reduction item, and so on to obtain the enhanced data set.

Optionally, the training the generative confrontation network based on the original data set, the enhanced data set, and the pseudo enhanced data set, and the step of obtaining the trained generative confrontation network includes:

combining the data in the original data set and the corresponding data in the enhanced data set to obtain an image pair data set;

inputting the enhanced data set and the pseudo enhanced data set into the discriminator so that the discriminator can discriminate the enhanced data set and the pseudo enhanced data set to obtain a discrimination result;

training and updating the discriminator by combining the discrimination result;

inputting the image pair data set to the generator for the generator to calculate the image pair data set to obtain a first calculation result;

training and updating the generator in combination with the first calculation result;

inputting an original data set to the generator, so that the generator can calculate the original data set to obtain a second calculation result;

taking the second result as the pseudo-enhancement data set; and returning to the execution step: inputting the enhanced data set and the pseudo enhanced data set into the discriminator so that the discriminator discriminates the enhanced data set and the pseudo enhanced data set to obtain a discrimination result;

and (4) according to the preset cycle times, carrying out the cycle until the cycle is finished, and terminating the training to obtain the trained image enhancement model.

Optionally, the step of inputting the image pair data set to the generator for the generator to calculate the image pair data set, and obtaining a first calculation result includes:

inputting the image pair data set to the generator, and calculating to obtain a countermeasure loss function based on the image pair data set and an expected value of the distribution function;

extracting output features of corresponding layers from the feature maps of a plurality of layers through a multi-layer image block network, and calculating to obtain features of the corresponding layers;

calculating maximum mutual information and the characteristics of the corresponding layers of the characteristic diagram based on a noise comparison estimation framework to obtain a noise comparison estimation repairing function;

calculating to obtain a total loss function based on the countermeasure loss function and the noise contrast estimation repair function;

calculating to obtain a second parameter based on a gradient descent algorithm and the total loss function;

taking the second parameter as the first parameter; and returning to the execution step: calculating to obtain a second parameter based on a gradient descent algorithm and the total loss function;

and performing parameter iteration according to preset iteration times by using the loop until iteration is finished, and taking the first parameter as the first calculation result.

An embodiment of the present invention further provides an image enhancement apparatus, where the image data enhancement apparatus includes:

the acquisition module is used for acquiring image data to be enhanced;

and the enhancement module is used for inputting the image enhancement data to be enhanced into a pre-created image enhancement model for enhancement to obtain enhanced image data, wherein the image enhancement model is obtained based on a generative countermeasure network and combined with a preset enhanced image algorithm for training.

An embodiment of the present invention further provides a terminal device, where the terminal device includes: a memory, a processor and an image enhancement program stored on the memory and executable on the processor, the image enhancement program being configured to implement the steps of the image enhancement method as described above.

An embodiment of the present invention further provides a storage medium, where an image enhancement program is stored, and the image enhancement program, when executed by a processor, implements the steps of the image enhancement method as described above.

The image enhancement method, the image enhancement device, the terminal equipment and the storage medium provided by the embodiment of the application acquire image data to be enhanced; and inputting the image enhancement data to be enhanced into a pre-created image enhancement model for enhancement to obtain enhanced image data, wherein the image enhancement model is obtained based on a generative countermeasure network and combined with a preset enhanced image algorithm for training. By means of the trained image enhancement model, the real-time processing capability of the underwater image and the preprocessing capability of the large data stream can be improved, and the image enhancement efficiency is improved. The embodiment starts from the problem of low image processing capability real-time performance, the generated confrontation network image enhancement based on the multi-scale homomorphic filtering label is a research object, the advantages and the disadvantages of the traditional image enhancement technology are deeply analyzed, a generated confrontation network image enhancement model based on the multi-scale homomorphic filtering label is designed, the effectiveness of the image enhancement method provided by the invention is verified on the image enhancement model, and finally the image enhancement real-time performance is obviously improved through the scheme of the invention.

Drawings

FIG. 1 is a schematic diagram of functional modules of a terminal device to which an image enhancement apparatus of the present invention belongs;

FIG. 2 is a flowchart illustrating a first exemplary embodiment of an image enhancement method according to the present invention;

fig. 3 is a schematic diagram of the overall data flow involved in the first exemplary embodiment of the image enhancement method of the present invention;

FIG. 4 is a flowchart illustrating a method for enhancing an image according to another exemplary embodiment of the present invention;

FIG. 5 is a schematic diagram of a training process of an image enhancement model according to an embodiment of the image enhancement method of the present invention;

FIG. 6 is a detailed flowchart of the step of step S51 in FIG. 5;

FIG. 7 is a detailed flowchart of the step of step S52 in FIG. 6;

fig. 8 is a detailed flowchart of the step of step S603 in fig. 6.

The implementation, functional features and advantages of the objects of the present invention will be further explained with reference to the accompanying drawings.

Detailed Description

It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

The technical terms related to the embodiment of the invention are as follows:

generating an antagonistic network, GAN, generic adaptive networks;

a Graphics processor, GPU, Graphics Processing Unit;

a Generator, Generator;

a Discriminator;

an Encoder, Encoder;

a decoder, decoder;

a multi-scale space;

homomorphic filtering, Homomorphic filtering;

multilayer perceptron, Multi-layer perceptron, MLP.

Among them, generating a countermeasure network (GAN, generic adaptive Networks) is a deep learning model, and is one of the most promising methods for unsupervised learning in complex distribution in recent years. Generating a confrontation network model through (at least) two modules in the framework: the mutual game learning of the Generative Model (or generator) and the discriminant Model (or discriminant) produces a fairly good output (in the embodiment of the present application, three modules are involved: encoder, generator and discriminant). In the original GAN theory, it is not required that the generation model and the discrimination model are both neural networks, but only that a function capable of fitting corresponding generation and discrimination is required. Deep neural networks are generally used as the generation model and the discriminant model in practice. An excellent GAN application requires a good training method, otherwise the output may be unsatisfactory due to the freedom of neural network models.

Goodfellow et al proposed a new framework in the general adaptive Networks to generate models by countermeasure process estimation in 10.2014. Two models were trained simultaneously in the framework: a generative model that captures the data distribution, and a discriminative model that estimates the probability that a sample is from the training data. The training procedure for generating the model is to maximize the probability of discriminant model errors.

Generally, machine-learned models can be roughly classified into two types, a Generative Model (Generative Model) and a Discriminative Model (Discriminative Model). The discriminant model requires input variables that are predicted by some model. Generative models are the random generation of observed data given some kind of implicit information. To take a simple example:

judging the model: setting a graph, and judging whether the animal in the graph is a cat or a dog;

generating a model: a new cat (not in the data set) is generated for a series of pictures of cats.

For discriminant models, the loss function is easily defined because the goal of the output is relatively simple. But the definition of the loss function is not so easy for generating the model. Therefore, the feedback part of the generated model is not handed to the discriminant model processing. This is to combine two major models in machine learning, namely Generative and discriminative, closely.

The basic principle of GAN is illustrated as follows, taking the generation of pictures as an example:

assume that there are two networks, G (Generator) and D (discriminator). Their functions are respectively:

g is a network of generated pictures which receives a random noise z from which the picture is generated, denoted G (z).

D is a discrimination network to discriminate whether a picture is "real". The input parameter is x, x represents a picture, and the output D (x) represents the probability that x is a real picture, if 1, 100% of the picture is real, and the output is 0, the picture cannot be real.

In the training process, the aim of generating the network G is to generate a real picture as much as possible to deceive the discrimination network D. And the aim of D is to separate the picture generated by G and the real picture as much as possible. Thus, G and D constitute a dynamic "gaming process". In the most ideal situation, the result of the final game is: g may generate enough pictures G (z) to be "spurious". For D, it is difficult to decide whether the picture generated by G is real or not.

This object is achieved: a generative model G is obtained which can be used to generate the picture. Goodfellow theoretically proves the convergence of the algorithm, and when the model converges, the generated data has the same distribution as the real data (the model effect is ensured).

Multi-scale space: the original signal is "embedded" in a series of signals obtained by a single parameter transformation, each signal obtained by the transformation corresponding to a parameter of the family of single parameters. An important requirement is that the coarser scale in the multi-scale representation should be a simplification of the finer scale, and that the coarser scale is smoothed from the finer scale image in some fixed manner. To satisfy this property, there are a variety of implementations. But, at a point, it is the gaussian function that is the only smoothing function available.

There are several ways to achieve multi-scale representation, for example, quarter trees or octant trees are used in the early days, and image pyramids. Pyramid is an image representation that combines a downsampling operation and a smoothing operation. One great benefit of the method is that the number of pixels of each layer from bottom to top is continuously reduced, which can greatly reduce the calculation amount; the disadvantage is that the pyramid quantization from bottom to top becomes coarser and faster. (it should be emphasized that the pyramid construction method and the wavelet pyramid construction method are similar, and the image of a certain layer is smoothed and then down-sampled, and the smoothing is performed for better representing the pixel points of the original image by the down-sampled pixel points, and is not the same as the smoothing in the multi-scale representation at all).

The above-mentioned quadtree or octadtree and pyramid representations, the steps taken in obtaining the multiscale are rather crude, with the "spacing" between scales being too large. While the "Scale-Space" representation referred to herein is another effective method of multi-Scale representation, where the Scale parameters are continuous and the number of spatial samples at all scales is the same (in practice, an image is obtained at one Scale and the Scale-Space samples are pixels of the image at that Scale).

The multi-scale space is formed by convolution of an original image and a two-dimensional Gaussian function, and continuously changing a parameter t to obtain continuously changing images, wherein information of the images is gradually reduced compared with the original image, detail information is gradually smoothed, but the number of pixels is kept unchanged, namely the resolution is unchanged. The image pyramid reduces the resolution by reducing the pixels of several lines at a time, so that the image information is reduced, and the two are different.

Homomorphic filtering: the method is a technology widely used for signal and image processing, original signals are converted into different domains which can use linear filters through nonlinear mapping, and the different domains are mapped back to the original domain after operation. Homomorphism is the property of keeping the relevant attributes unchanged, and homomorphic filtering has the advantage of converting originally complex operations into relatively simple operations with the same performance. This concept was introduced in the 1960 s by Thomas Stockham, Alan v. oppenheim and Ronald w.schafer at the institute of technology of massachusetts.

Homomorphic filtering removes multiplicative noise (multiplicative noise) to increase contrast and normalize brightness, thereby achieving image enhancement.

An image can be represented as the product of its illumination (luminance) and reflection (reflection) components, which, although they are inseparable in the time domain, can be linearly separated in the frequency domain via fourier transformation. Since the illumination can be regarded as illumination in the environment, the relative change is small, and the illumination can be regarded as a low-frequency component of an image; the high frequency component is considered when the reflectivity is relatively changed. By processing the influence of the illumination and the reflectivity on the gray value of the pixel respectively, the illumination of the image is more uniform by a high-pass filter (high-pass filter), so as to achieve the purpose of enhancing the detail characteristics of the shadow area.

MLP multilayer perceptron (Multi-layerpercpton): an Artificial Neural Network (ANN) is a forward structure that maps a set of input vectors to a set of output vectors. An MLP can be viewed as a directed graph, consisting of multiple layers of nodes, each layer being fully connected to the next. Except for the input nodes, each node is a neuron with a nonlinear activation function. The MLP is trained using a supervised learning approach of the BP back-propagation algorithm. MLP is the popularization of the sensor, and overcomes the weakness that the sensor cannot identify linear irreducible data.

Compared with a single-layer sensor, the MLP multi-layer sensor can have a plurality of hidden layers in the middle except for an input and output layer, and the simplest MLP only comprises one hidden layer, namely a three-layer structure. The multiple layers of perceptrons are fully connected. The bottom layer of the multilayer perceptron is an input layer, the middle layer is a hidden layer, and the last layer is an output layer. In the embodiment of the invention, in the process of processing the underwater image, if a traditional unsupervised image enhancement method is used, the problems of large calculated amount, complex operation and incapability of being arranged on a GPU exist when a large amount of image data is processed; if a method based on a convolutional neural network is used, an underwater image data set is required to have an underwater distorted image and a clear image, and images in the presence of water and images in the absence of water are required to be acquired at the same position and the same parameters.

Therefore, according to the scheme of the embodiment of the invention, from the problem of low image processing capacity real-time performance, an image enhancement model of a generating type countermeasure network based on a multi-scale homomorphic filtering label is designed by combining the generating type countermeasure network with a traditional unsupervised image enhancement method, so that the real-time processing capacity of underwater images and the preprocessing capacity of large data streams are improved, and the image enhancement efficiency is improved.

Specifically, referring to fig. 1, fig. 1 is a schematic diagram of functional modules of a terminal device to which the image enhancement apparatus of the present application belongs. The image enhancement device can be a device which is independent of the terminal equipment and can perform image processing and network model training, and the device can be borne on the terminal equipment in a hardware or software mode. The terminal device can be an intelligent mobile terminal with a data processing function, such as a mobile phone and a tablet personal computer, and can also be a fixed terminal device or a server with a data processing function.

In this embodiment, the terminal device to which the image enhancement apparatus belongs at least includes an output module 110, a processor 120, a memory 130, and a communication module 140.

The memory 130 stores an operating system and an image enhancement program, and the image enhancement device can enhance the acquired image data to be enhanced, the enhanced image data obtained by enhancing the image enhancement network model, and the acquired original data set; inputting the original data set to a generator to obtain a corresponding pseudo enhanced data set; enhancing the data in the original data set based on the multi-scale homomorphic filtering label to obtain a corresponding enhanced data set; combining the data in the original data set and the corresponding image data in the enhanced data set to obtain an image pair data set; inputting the image pair data set to the generator for the generator to calculate the image pair data set, resulting in a first calculation result; inputting an original data set to the generator, so that the generator can calculate the original data set, and storing information such as a second calculation result and the like obtained in the memory 130; the output module 110 may be a display screen or the like. The communication module 140 may include a WIFI module, a mobile communication module, a bluetooth module, and the like, and communicates with an external device or a server through the communication module 140.

Wherein the image enhancement program in the memory 130 when executed by the processor implements the steps of:

acquiring image data to be enhanced;

Further, the image enhancement program in the memory 130 when executed by the processor further implements the steps of:

creating the image enhancement model specifically comprises:

acquiring an original data set;

establishing a generator;

establishing a discriminator;

inputting a first parameter to the generator, and initializing the generator;

training and updating the discriminator by combining the discrimination result;

extracting output features of corresponding layers from the feature maps of the plurality of layers through a multi-layer image block network, and calculating to obtain the features of the corresponding layers;

An embodiment of the present invention provides an image enhancement method, and referring to fig. 2, fig. 2 is a flowchart illustrating a first embodiment of an image enhancement method according to the present invention.

In this embodiment, the image enhancement method includes:

step S1001: acquiring image data to be enhanced;

step S1002: and inputting the image enhancement data to be enhanced into a pre-created image enhancement model for enhancement to obtain enhanced image data, wherein the image enhancement model is obtained based on a generative countermeasure network and combined with a preset enhanced image algorithm for training.

The execution subject of the method of this embodiment may be an image enhancement device, or may also be an image enhancement terminal device or server, in this embodiment, the image enhancement device is taken as an example, and the image enhancement device may be integrated on a computer, a smart phone, a tablet computer, and other terminal devices with a data processing function, and is suitable for development of a computer visual image processing front end.

The embodiment mainly realizes the real-time performance of image enhancement, particularly image enhancement, improves the real-time processing of underwater images and the preprocessing capability of large data streams, and improves the image enhancement efficiency.

Particularly, underwater images with high imaging quality are required to be used in underwater archaeology, submarine exploration and the like, and the lightweight netting underwater image real-time enhancement technology is a key technology for improving the sensing distance of an underwater inspection robot and improving the capability of camera perception, feature extraction and visual positioning. Therefore, it is necessary to acquire image data to be enhanced, input the data to be enhanced into an image enhancement model created in advance for enhancement, and obtain enhanced image data, thereby obtaining an underwater image with high quality. The pre-created image enhancement model is obtained based on a generative countermeasure network and combined with a preset enhanced image algorithm for training.

The preset enhanced image algorithm can enhance the image data and greatly reduce the difficulty of collecting the data set, wherein the preset enhanced image algorithm can be an improved Multi-Scale Retinex algorithm; or non-physical model methods such as histogram equalization, gray world assumption, contrast-limited histogram equalization, multi-scale retina enhancement with color recovery, automatic white balance, color constancy, wavelet transformation, and the like; or the underwater image is restored through the inversion of the assumed conditions, and the image is restored through scene statistics and prior; some methods based on the underwater imaging optical properties, such as improving and restoring an image through an atmospheric turbulence model, restoring an image through designing a new underwater imaging model, restoring an image through considering the characteristics of the underwater imaging model, defogging based on an image and the like, are based on a physical model.

In this embodiment, the underwater image restored based on the improved Multi-Scale Retinex algorithm is used as a training label, and a neural network contextual unappered transformation Model is used to realize real-time enhancement of the underwater image of the lightweight mesh garment.

More specifically, in the embodiment, based on the problem of low image processing capability real-time performance, the generated confrontation network image enhancement based on the multi-scale homomorphic filtering label is a research object, advantages and disadvantages of the conventional image enhancement technology are deeply analyzed, and a generated confrontation network image enhancement model based on the multi-scale homomorphic filtering label is designed, and the image enhancement model is obtained based on the generated confrontation network training of the multi-scale homomorphic filtering label.

The embodiment obtains the image data to be enhanced; and inputting the image enhancement data to be enhanced into a pre-created image enhancement model for enhancement to obtain enhanced image data, wherein the image enhancement model is obtained based on a generative countermeasure network and combined with a preset enhanced image algorithm for training. By using the generative countermeasure network to enhance the low-illumination image, the dependence of the traditional method on the prior knowledge of the image can be abandoned; the mutual conversion from image to image is realized by adopting an unpaired mode through a generative countermeasure network, and one-to-one corresponding paired images are not needed; the generative countermeasure network can also arrange the image enhancement model on a GPU (graphics processing Unit), and the GPU is used for calculation to realize light weight processing of the underwater original image; by combining the generative countermeasure network with the traditional unsupervised enhanced image algorithm, the difficulty in collecting the data set is effectively reduced, the lightweight processing of the underwater original image is realized, and the real-time processing of the underwater image and the preprocessing capability of a large data stream are remarkably improved. The method is suitable for developing the computer vision image processing front end, and has important engineering value and theoretical guiding significance for improving the efficiency of image enhancement, image recognition, image restoration and the like under a large data set.

In this embodiment, an image restoration network model is used to enhance an image, and a framework of the image enhancement network model includes: the generative confrontation network comprises a generator and an encoder, the image enhancement model is obtained by training the generative confrontation network based on the multi-scale homomorphic filtering label, and the overall data flow of the network is as shown in fig. 3:

carrying out noise reduction, multi-scale space construction and image enhancement on an original data set by using a traditional image processing method (multi-scale homomorphic filtering label algorithm) to obtain a corresponding enhanced data set;

the generator is used for reconstructing the input original data set to obtain a pseudo enhanced data set, and providing the pseudo enhanced data set to the discriminator;

the generator is also used for calculating the input original data set and the corresponding enhanced data set to obtain a calculation result, training the generator by combining the calculation result, and providing the calculation result to the discriminator so that the discriminator can discriminate the calculation result;

the discriminator is used for discriminating the input enhanced data set and the pseudo enhanced data set to obtain a discrimination result, and the discriminator is trained by combining the discrimination result;

constructing and obtaining an image data processing end (image enhancement model) based on the generator after training and the discriminator after training;

the image data processing end is used for processing the input underwater original image data to obtain an underwater image enhancement data set.

Referring to fig. 4, fig. 4 is a flowchart illustrating an image enhancement method according to another exemplary embodiment of the present invention. Based on the embodiment shown in fig. 2, in this embodiment, before the step of inputting the data to be enhanced into the pre-created image enhancement model for enhancement in step S1002 to obtain the enhanced image data, the image enhancement method further includes:

creating the image enhancement model specifically comprises:

step S10: acquiring an original data set;

step S20: establishing a generator;

step S30: establishing a discriminator;

step S40: constructing a generating type countermeasure network based on the generator and the discriminator;

step S50: and training the generative confrontation network based on the multi-scale homomorphic filtering label and the original data set to obtain the image enhancement model.

In this embodiment, step S10 to step S50 are implemented before step S1001, and in other embodiments, step S10 to step S50 may also be implemented between step S1001 and step S1002.

Compared with the embodiment shown in fig. 2, the embodiment further includes a scheme for training the image enhancement model.

Specifically, the present embodiment collects a number of random image data in advance to form an original data set, for example, the original data set, where the original data set is used to train the image enhancement model;

then, a stereo image generator G (r) is formed by an encoder G (r) _dec (r) and decoder G _enc (r) the composition of the components (a),

in the formula (I), the compound is shown in the specification,

generating a pseudo-enhanced image for the original image data f generated under the generator G (f)The device is used for calculating the input image data to obtain a calculation result, training and updating the generator by combining the calculation result, and providing the calculation result to the discriminator so that the discriminator can discriminate the calculation result;

then establishing a discriminator, wherein the discriminator is used for discriminating input image data, when the input image data is enhanced image data, the discriminator outputs a high score (close to 1), when the input data is pseudo enhanced image data, the discriminator outputs a low score (close to 0), a discrimination result is obtained, and the discriminator is trained and updated by combining the discrimination result;

then, constructing and obtaining a generating type countermeasure network based on the generator and the discriminator;

and finally, enhancing the original data set based on the multi-scale homomorphic filtering label to obtain an enhanced data set, and training the generative confrontation type network to obtain the image enhancement model.

And then, enhancing the original data set through the trained image enhancement model.

In the embodiment, through the above scheme, an original data set is obtained; establishing a generator; establishing a discriminator; constructing a generating type countermeasure network based on the generator and the discriminator; and training the generative confrontation network based on the multi-scale homomorphic filtering label and the original data set to obtain the image enhancement model. The method has the advantages that the low-illumination image is enhanced by using the generative confrontation network, so that the dependence of the traditional method on the prior knowledge of the image can be eliminated; by introducing an enhanced image algorithm of multi-scale homomorphic filtering, the difficulty in collecting a data set can be effectively reduced; by combining the generative countermeasure network with the traditional unsupervised multi-scale homomorphic filtering label, the image enhancement model can be arranged on the GPU, and the GPU is used for calculation, so that the light weight processing of the underwater original image is realized. By means of the trained image enhancement model, the real-time processing capability of the underwater image and the preprocessing capability of the large data stream can be improved, and the image enhancement efficiency is improved.

Referring to fig. 5, fig. 5 is a schematic diagram of a training flow of an image enhancement model according to an embodiment of the image enhancement method of the present invention. Based on the above-mentioned embodiment shown in fig. 4, in this embodiment, step S20, after the step of establishing the generator, further includes:

step S21: inputting a first parameter to the generator, and initializing the generator;

step S22: inputting the original data set to the generator to obtain a corresponding pseudo-enhanced data set;

in the step S50, training the generative confrontation network based on the multi-scale homomorphic filtering label and the original data set to obtain the image enhancement model includes:

step S51: and training the generative countermeasure network based on the multi-scale homomorphic filtering label, the original data set and the pseudo enhancement data set to obtain the image enhancement model.

wherein, the first and the second end of the pipe are connected with each other,

the pseudo-enhancement image is generated for original image data f under a generator G (f), the generator is used for calculating input image data to obtain a calculation result, the generator is trained and updated by combining the calculation result, and the calculation result is provided for a discriminator to discriminate the calculation result;

a set of random vectors (theta) is input in a generator G (f) ₁ ,θ ₂ ,θ ₃ ,…,θ _k ) Initializing a generator, generating a set of pseudo-enhanced data sets from the original data set under G (f)；

Then establishing a discriminator, wherein the discriminator is used for discriminating input image data, when the input data is enhanced image data, the discriminator outputs a high score (close to 1), when the input data is pseudo-enhanced image data, the discriminator outputs a low score (close to 0), a discrimination result is obtained, and the discriminator is trained and updated by combining the discrimination result;

and finally, enhancing the original data set based on the multi-scale homomorphic filtering label to obtain an enhanced data set, inputting the enhanced data set and the pseudo enhanced data set into the generative confrontation network, and training the generative confrontation network by combining a calculation result to obtain the image enhancement model.

Further, the step of training the generative confrontation network based on the multi-scale homomorphic filtering label, the original data set and the pseudo-enhanced data set to obtain the image enhancement model includes:

firstly, combining data in the original data set and corresponding data in the enhanced data set to obtain an image pair data set;

then, inputting the enhanced data set and the pseudo enhanced data set to the discriminator so that the discriminator discriminates the enhanced data set and the pseudo enhanced data set to obtain a discrimination result;

then, combining the discrimination result to train and update the discriminator;

then, inputting the image pair data set to the generator, so that the generator can calculate the image pair data set to obtain a first calculation result;

then, training and updating the generator by combining the first calculation result;

then, inputting an original data set to the generator, so that the generator can calculate the original data set to obtain a second calculation result;

then, taking the second result as the pseudo-enhancement data set; and returning to the execution step: inputting the enhanced data set and the pseudo enhanced data set to the discriminator so that the discriminator discriminates the enhanced data set and the pseudo enhanced data set to obtain a discrimination result;

and finally, according to the preset cycle times, the training is terminated until the cycle is finished, and the trained image enhancement model is obtained.

The embodiment creates an image enhancement model through the above scheme, and specifically includes:

acquiring an original data set; establishing a generator; inputting a first parameter to the generator, and initializing the generator; inputting the original data set to the generator to obtain a corresponding pseudo-enhancement data set; establishing a discriminator; constructing a generating type countermeasure network based on the generator and the discriminator; and training the generative countermeasure network based on the multi-scale homomorphic filtering label, the original data set and the pseudo enhancement data set to obtain the image enhancement model. The generator in the generative countermeasure network is used for enhancing the low-illumination image, so that the dependence of the traditional method on the prior knowledge of the image can be avoided; by means of the trained image enhancement model, the real-time processing capability of the underwater image and the preprocessing capability of the large data stream can be improved, and the image enhancement efficiency is improved.

Referring to fig. 6, fig. 6 is a flowchart illustrating a detailed process of the step S51 in the embodiment shown in fig. 5, and the step S51 of training the generative confrontation network based on the multi-scale homomorphic filtering label, the original data set and the pseudo-enhanced data set to obtain the image enhancement model includes:

step S52: enhancing the data in the original data set based on the multi-scale homomorphic filtering label to obtain a corresponding enhanced data set;

step S53: and training the generative confrontation network based on the original data set, the enhanced data set and the pseudo enhanced data set to obtain the trained generative confrontation network.

Further, in step S53, the training of the generative confrontation network based on the original data set, the enhanced data set and the pseudo enhanced data set includes:

step S600: combining the data in the original data set and the corresponding data in the enhanced data set to obtain an image pair data set;

step S601: inputting the enhanced data set and the pseudo enhanced data set into the discriminator so that the discriminator can discriminate the enhanced data set and the pseudo enhanced data set to obtain a discrimination result;

step S602: training and updating the discriminator by combining the discrimination result;

step S603: inputting the image pair data set to the generator for the generator to calculate the image pair data set to obtain a first calculation result;

step S604: training and updating the generator in combination with the first calculation result;

step S605: inputting an original data set to the generator, so that the generator can calculate the original data set to obtain a second calculation result;

step S606: taking the second result as the pseudo-enhancement data set; and returns to execute step S601: inputting the enhanced data set and the pseudo enhanced data set into the discriminator so that the discriminator discriminates the enhanced data set and the pseudo enhanced data set to obtain a discrimination result;

step S607: and (4) according to the preset cycle times, carrying out the cycle until the cycle is finished, and terminating the training to obtain the trained image enhancement model.

Specifically, first, the original image data set F and its corresponding R (x, y) data set R of the homomorphic filtering enhanced image in the multi-scale space are set to { R ═ R { ₁ ,r ₂ ,r ₃ ,…r _n As an image pair dataset.

M＝{(f ₁ ,r ₁ ),(f ₂ ,r ₂ ),(f ₃ ,r ₃ ),…,(f _n ,r _n )}

Then, inputting the enhanced data set and the pseudo enhanced data set into the discriminator to be distinguished by the discriminator, marking the enhanced data set as 1 and distinguishing the enhanced data set as a high score (close to 1) when the input data is the enhanced data set, and then training and updating the discriminator by combining a high score result; when the input data is a pseudo-enhanced data set

Time, pseudo-enhanced data set

Marking as 0, judging the low score (close to 0), and then training and updating the judger by combining the high score result;

then, the optimization problem has been a very important field in machine learning and even deep learning. In particular, deep learning is adopted, and therefore, the embodiment adopts the Adam gradient descent algorithm, and the total function of the confrontation loss of the image pair data set is calculated, so that relatively low calculation amount is ensured. The magnitude of the parameter update in the generator does not change as the gradient magnitude scales; the boundary of the step length when updating the parameters is limited by the setting of the step length of the super parameter; no fixed objective function is required. Therefore, the image pair dataset is input to the generator for the generator to calculate a total loss function of the image pair dataset, and an Adam gradient descent algorithm is used to obtain updated parameters as a first calculation result;

then, training and updating the original parameters in the generator by combining the first calculation result;

then, inputting the original data set to the generator for the generator to calculate the original data set, encoding the input original image data into a low-dimensional vector by a decoder in the generator, wherein the low-dimensional vector contains main information of the original image data, for example, the element of the low-dimensional vector can represent the color, shape, size and the like of any underwater creature, and decoding the structural information of the low-dimensional vector on the image by the encoder in the generator to generate a new pseudo-enhanced data set as a second calculation result;

then, taking the second result as the pseudo-enhancement data set; and returning to the execution step: inputting the enhanced data set and the pseudo enhanced data set into a discriminator so that the discriminator can discriminate the enhanced data set and the pseudo enhanced data set to obtain a discrimination result;

and finally, according to actual requirements, circulating according to preset circulation times until the circulation is finished, and terminating the training to obtain the trained image enhancement model.

In this embodiment, by using the above scheme, an image pair data set is obtained by specifically combining data in the original data set and corresponding data in the enhanced data set; inputting the enhanced data set and the pseudo enhanced data set into the discriminator so that the discriminator can discriminate the enhanced data set and the pseudo enhanced data set to obtain a discrimination result; training and updating the discriminator by combining the discrimination result; inputting the image pair data set to the generator for the generator to calculate the image pair data set to obtain a first calculation result; training and updating the generator in combination with the first calculation result; inputting an original data set to the generator, so that the generator can calculate the original data set to obtain a second calculation result; taking the second result as the pseudo-enhancement data set; and returning to the execution step: inputting the enhanced data set and the pseudo enhanced data set to the discriminator so that the discriminator discriminates the enhanced data set and the pseudo enhanced data set to obtain a discrimination result; and (4) according to the preset cycle times, carrying out the cycle until the cycle is finished, and terminating the training to obtain the trained image enhancement model. Calculating a total loss function of an image pair data set, obtaining a difference value between a predicted value and a true value of an image enhancement model by using the total loss function and an Adam gradient descent algorithm, measuring the prediction quality of the image enhancement model by using the difference value between the predicted value and the true value, training and updating the image enhancement model, and obtaining the trained image enhancement model. The real-time processing of underwater images and the preprocessing capability of large data streams are improved.

Referring to fig. 7, fig. 7 is a detailed flowchart of the step of step S52 in fig. 6. Based on the above-described embodiment shown in fig. 6, step S52: based on the multi-scale homomorphic filtering label, enhancing the data in the original data set to obtain a corresponding enhanced data set, wherein the step of obtaining the corresponding enhanced data set comprises the following steps:

step S800: calculating a noise reduction item corresponding to the data in the original data set so as to reduce the noise of the data in the original data set;

specifically, the original data set is in a complex external environment in the processes of acquisition, transmission and reception, various interferences exist, and the influence of noise generally exists, so that the resolution of the image is reduced, the originally fine structure of the image is also damaged, and the noise reduction is a precondition for various feature identification and extraction for processing the digital image.

Processing original image data f (x, y) in the original data set, realizing noise reduction of the original image in the sliding window through a convolution network, and calculating an image convolution noise reduction term I according to the following formula _q (x,y)。

I _q (x,y)＝med{f(x-k,y-l),(k,l∈CW)}

Where CW is the convolution kernel size.

Step S801: calculating a spatial distance weight value and an adjacent pixel value weight value from a pixel point corresponding to the data in the original data set to a central point;

specifically, the weighted value of the spatial distance from the pixel point I to the central point is calculated by the following formula.

Then, the image neighboring pixel value weight s (ξ, x) is calculated by the following equation:

in the formula, ξ represents the spatial distance from the pixel point I to the central point.

Step S802: calculating to obtain a pixel weight sum based on the adjacent pixel value weight and the spatial distance weight value;

specifically, the image pixel weight sum w (ξ, x, k, l) is calculated by:

w(ξ,x,k,l)＝c(ξ,x)s(ξ,x)；

step S803: calculating to obtain a reflection image based on the pixel weight sum and the noise reduction term;

specifically, the reflection image L (x, y) is calculated by the following equation.

Step S804: calculating to obtain a reflection response image based on the reflection image and a preset Gaussian function so as to carry out multi-scale space construction on the data in the original data set;

specifically, as the original image data is blurred from a short distance to a long distance, that is, the original image data is subjected to a process of increasing the scale, the original image data needs to be subjected to multi-scale spatial construction to obtain the optimal scale of the object of interest; and the same key points exist under different scales, so that the key points can be detected under the input image data of different scales for matching.

The reflection image L (x, y) is calculated by the following equation.

Then, the reflection response image R (x, y, σ) is calculated by the following equation, and a multi-scale spatial configuration is performed.

R(x,y,σ)＝G(x,y,σ)*L(x,y)；

In the formula, G (x, y, σ) is a variable gaussian function, and is defined as follows:

where σ is a scale space factor, and determines the degree of image blur smoothing processing.

Step S805: and calculating to obtain enhanced data corresponding to the data in the original data set based on the reflection response image and the noise reduction item, and so on to obtain the enhanced data set.

Specifically, homomorphic filtering enhanced image r (x, y) in multi-scale space is calculated

Wherein N is the number of scales of the multi-scale space of the structure.

And finally, obtaining an enhanced image r (x, y) corresponding to the original image data relatively based on the multi-scale homomorphic filtering label, and forming an enhanced data set.

In this embodiment, by using the above scheme, a noise reduction item corresponding to data in the original data set is specifically calculated; calculating a spatial distance weight value and an adjacent pixel value weight value from a pixel point corresponding to the data in the original data set to a central point; calculating to obtain a pixel weight sum based on the adjacent pixel value weight and the spatial distance weight value; calculating to obtain a reflection image based on the pixel weight sum and the noise reduction item; calculating to obtain a reflection response image based on the reflection image and a preset Gaussian function; and calculating to obtain enhanced data corresponding to the data in the original data set based on the reflection response image and the noise reduction item, and so on to obtain the enhanced data set. Based on the multi-scale homomorphic filtering label, the noise reduction, the multi-scale space construction and the image enhancement are carried out on the original image data in the original data set, the collection difficulty of the original data set can be greatly reduced, the requirement that the underwater image data set has both an underwater distorted image and a clear image is not needed, and the requirement that the images in the presence of water and in the absence of water are collected at the same position and under the same parameter is also met, so that the real-time processing capability of the underwater image and the preprocessing capability of a large data stream are improved, and the image enhancement efficiency is improved.

Referring to fig. 8, fig. 8 is a detailed flowchart of the step of step S603 in fig. 6. Based on the embodiment shown in fig. 6 described above, step S603: inputting the image pair dataset to the generator for the generator to compute the image pair dataset, the step of obtaining a first computation result comprising:

step S700: inputting the image pair data set to the generator, and calculating to obtain a countermeasure loss function based on the image pair data set and an expected value of the distribution function;

specifically, the following formula is adopted to calculate the penalty function

In the formula (I), the compound is shown in the specification,

representing the expected value of the distribution function, D representing the discriminator, G representing the generator, f representing the data in the original dataset, and r representing the homomorphic filter enhanced image in the multi-scale space.

The generator calculates the input image pair data set to obtain a loss function of each image pair data, and then the difference between the forward calculation result of each iteration and the real image data value is obtained, so that the next training is guided to be carried out in the correct direction.

Step S701: extracting output features of corresponding layers from the feature maps of the plurality of layers through a multi-layer image block network, and calculating to obtain the features of the corresponding layers;

in particular, the features are raw materials of the machine learning system, and the influence on the final image enhancement model is undoubted. When the data is well characterized, the image enhancement model can achieve satisfactory accuracy.

Selecting a feature map of the common L layer of interest, passing it through a two-layer MLP network H _l The resulting signature was:

wherein the content of the first and second substances,

representing the output characteristic of the l-th layer, z _l Represents the L-th layer characteristics, L ∈ {1,2,3, …, L }, f represents the raw data in the raw data set, H _l A two-layer MLP network is shown.

Step S702: calculating maximum mutual information and the characteristics of the corresponding layers of the characteristic diagram based on a noise comparison estimation framework to obtain a noise comparison estimation repairing function;

in particular, the noise contrast estimation algorithm is a statistical model estimation method, which can be used to solve the complex computation problem in the generative countermeasure network.

The maximum mutual information is calculated by a Noise Contrast Estimation (NCE) framework, and a noise contrast estimation patch function is generated.

Where S represents the patch number per layer (S ∈ {1,2,3, …, S) _l }) in which S is _l Indicates that the l-th layer has S _l The position of the first and second sensors in space,

representing the probability of a positive sample being selected in the NCE,

the dimension representing the feature vector for the s-th patch in the l-th layer is C _l ，

G denotes a generator, F denotes an original data set, and H denotes a two-layer multi-level image block network.

At this time, the image is enhanced

Can be expressed as:

representing the output characteristics of the l < th > layer, and f representing the original data in the original data set.

Each enhanced image is obtained by calculating each image pair

An enhanced data set R is composed.

Step S703: calculating to obtain a total loss function based on the countermeasure loss function and the noise contrast estimation repair function;

specifically, if the parameters in the generator are adjusted to completely satisfy that the output error of any image pair data is zero, the error of any image pair data other than the current image pair data is generally made larger, and thus the loss function value as the sum of errors becomes larger. Therefore, after the weight is adjusted according to the error of any image pair data, the total loss function value of the image pair data set is calculated to judge that the image enhancement data master has trained to an acceptable state.

Then, the total loss function is calculated by the above loss function

In the formula, R represents an enhanced data set, and D represents a discriminator.

According to the calculation rate requirement, the following can be set: lambda [ alpha ] _F ＝1，λ _R 1 or λ _F ＝10，λ _R ＝0。

Step S704: calculating to obtain a second parameter based on a gradient descent algorithm and the total loss function;

specifically, in this embodiment, the total loss function is utilized by setting the number of iterations as needed

And the Adam gradient descent algorithm obtains a second parameter (theta) ₁ ,θ ₂ ,θ ₃ ,…,θ _k )。

By calculating the Adam gradient descent algorithm and the total loss function, the oscillation can be reduced, the direction is unchanged, the calculation efficiency and correct convergence are ensured, and the used memory is relatively small.

Step S705: taking the second parameter as the first parameter; and returning to the execution step: step S704, calculating to obtain a second parameter based on a gradient descent algorithm and the total loss function;

specifically, the second parameter is used as the first parameter, and the step S704 is executed again, and the next iteration is continued until the first parameter in the generator converges, so as to obtain the first parameter after the iteration is completed.

Step S706: and performing parameter iteration according to preset iteration times by using the loop until iteration is finished, and taking the first parameter as the first calculation result.

And sequentially circulating, performing parameter iteration according to preset iteration times until the iteration is finished, and training and updating the image enhancement model by taking the first parameter after the iteration is finished as a first calculation result.

In this embodiment, by the above scheme, specifically, the image pair data set is input to the generator, and a countermeasure loss function is calculated based on the image pair data set and an expected value of the distribution function; extracting output features of corresponding layers from the feature maps of the plurality of layers through a multi-layer image block network, and calculating to obtain the features of the corresponding layers; calculating maximum mutual information and the characteristics of the corresponding layers of the characteristic diagram based on a noise comparison estimation framework to obtain a noise comparison estimation repairing function; calculating to obtain a total loss function based on the countermeasure loss function and the noise contrast estimation repair function; calculating to obtain a second parameter based on a gradient descent algorithm and the total loss function; taking the second parameter as the first parameter; and returning to the execution step: calculating to obtain a second parameter based on a gradient descent algorithm and the total loss function; and performing parameter iteration according to preset iteration times by using the loop until iteration is finished, and taking the first parameter as the first calculation result. The total loss function is calculated through a noise comparison estimation algorithm, so that the complex calculation problem in the generative countermeasure network can be solved; through the gradient descent algorithm and the total loss function, the calculation efficiency and correct convergence are ensured, and the used memory is relatively small. The real-time processing of underwater images and the preprocessing capability of large data streams are improved, and the image enhancement efficiency is improved.

It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or system. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, method, article, or system that comprises the element.

The above-mentioned serial numbers of the embodiments of the present invention are merely for description and do not represent the merits of the embodiments.

Through the above description of the embodiments, those skilled in the art will clearly understand that the method of the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but in many cases, the former is a better implementation manner. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium (e.g., ROM/RAM, magnetic disk, optical disk) as described above and includes instructions for enabling a terminal device (e.g., a mobile phone, a computer, a server, or a network device) to execute the method according to the embodiments of the present invention.

The above description is only a preferred embodiment of the present invention, and not intended to limit the scope of the present invention, and all modifications of equivalent structures and equivalent processes, which are made by using the contents of the present specification and the accompanying drawings, or directly or indirectly applied to other related technical fields, are included in the scope of the present invention.

Claims

1. An image enhancement method, characterized in that it comprises the steps of:

acquiring image data to be enhanced;

2. The image enhancement method according to claim 1, wherein the step of inputting the data to be enhanced into a pre-created image enhancement model for enhancement to obtain enhanced data further comprises:

creating the image enhancement model specifically comprises:

acquiring an original data set;

establishing a generator;

establishing a discriminator;

3. The image enhancement method of claim 2, wherein the step of establishing a generator is further followed by:

inputting a first parameter to the generator, and initializing the generator;

4. The image enhancement method of claim 3, wherein the step of training the generative confrontation network based on the multi-scale homomorphic filtering label, the original data set, and the pseudo-enhancement data set to obtain the image enhancement model comprises:

5. The image enhancement method of claim 4, wherein the step of enhancing the data in the original data set based on the multi-scale homomorphic filtering label to obtain a corresponding enhanced data set comprises:

calculating to obtain a reflection image based on the pixel weight sum and the noise reduction item;

6. The image enhancement method of claim 4, wherein the training of the generative warfare network based on the original data set, the enhanced data set, and the pseudo-enhanced data set to obtain the trained generative warfare network comprises:

training and updating the discriminator by combining the discrimination result;

7. The image enhancement method of claim 6, wherein the step of inputting the image pair dataset to the generator for the generator to compute the image pair dataset comprises:

8. An image enhancement apparatus, characterized in that the image data enhancement apparatus comprises:

the acquisition module is used for acquiring image data to be enhanced;

9. A terminal device, characterized in that the terminal device comprises: memory, a processor and an image enhancement program stored on the memory and executable on the processor, the image enhancement program being configured to implement the steps of the image enhancement method of any one of claims 1 to 7.

10. A storage medium having stored thereon an image enhancement program which, when executed by a processor, implements the steps of the image enhancement method of any one of claims 1 to 7.