CN112085681B - Image enhancement method, system, device and storage medium based on deep learning - Google Patents
Image enhancement method, system, device and storage medium based on deep learning Download PDFInfo
- Publication number
- CN112085681B CN112085681B CN202010941865.3A CN202010941865A CN112085681B CN 112085681 B CN112085681 B CN 112085681B CN 202010941865 A CN202010941865 A CN 202010941865A CN 112085681 B CN112085681 B CN 112085681B
- Authority
- CN
- China
- Prior art keywords
- convolutional neural
- image
- neural network
- training
- network
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 55
- 238000013135 deep learning Methods 0.000 title claims abstract description 51
- 238000003860 storage Methods 0.000 title claims abstract description 26
- 238000013527 convolutional neural network Methods 0.000 claims abstract description 187
- 238000012549 training Methods 0.000 claims abstract description 84
- 230000006870 function Effects 0.000 claims description 52
- 238000013528 artificial neural network Methods 0.000 claims description 23
- 230000000694 effects Effects 0.000 abstract description 18
- 238000012545 processing Methods 0.000 description 14
- 241000282414 Homo sapiens Species 0.000 description 9
- 230000008859 change Effects 0.000 description 5
- 238000010586 diagram Methods 0.000 description 5
- 230000008447 perception Effects 0.000 description 5
- 230000009286 beneficial effect Effects 0.000 description 4
- 238000004458 analytical method Methods 0.000 description 3
- 238000013461 design Methods 0.000 description 3
- 230000006872 improvement Effects 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 230000008569 process Effects 0.000 description 3
- 230000003416 augmentation Effects 0.000 description 2
- 210000000887 face Anatomy 0.000 description 2
- 238000005111 flow chemistry technique Methods 0.000 description 2
- 238000003384 imaging method Methods 0.000 description 2
- 239000013307 optical fiber Substances 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 230000004913 activation Effects 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000003190 augmentative effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000005520 cutting process Methods 0.000 description 1
- 238000013434 data augmentation Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000009826 distribution Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000002708 enhancing effect Effects 0.000 description 1
- 238000005286 illumination Methods 0.000 description 1
- 238000002372 labelling Methods 0.000 description 1
- 238000010606 normalization Methods 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 238000011176 pooling Methods 0.000 description 1
- 230000003252 repetitive effect Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000016776 visual perception Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/80—Geometric correction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/40—Image enhancement or restoration using histogram techniques
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
- Y02T10/00—Road transport of goods or passengers
- Y02T10/10—Internal combustion engine [ICE] based vehicles
- Y02T10/40—Engine management systems
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Biomedical Technology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Life Sciences & Earth Sciences (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Image Analysis (AREA)
- Image Processing (AREA)
Abstract
The invention provides an image enhancement method, a system, equipment and a storage medium based on deep learning, wherein the method comprises the following steps: collecting sample images for model training; training a plurality of first convolutional neural networks connected in parallel based on the sample image, wherein each first convolutional neural network corresponds to at least one enhancement item; training a second convolutional neural network, which is connected in series after the plurality of first convolutional neural networks connected in parallel; combining the trained first convolutional neural network and the trained second convolutional neural network to obtain an image enhancement model; and inputting the image to be enhanced into the image enhancement model to obtain an enhanced image output by the image enhancement model. By adopting the invention, the image enhancement of a plurality of enhancement items can be realized simultaneously by adopting the image enhancement model based on deep learning, and the image enhancement effect is improved.
Description
Technical Field
The present invention relates to the field of data processing technologies, and in particular, to an image enhancement method, system, device, and storage medium based on deep learning.
Background
With the increasing complexity of camera imaging environments, based on the rapid development of deep learning and Artificial Intelligence (AI), the method can intelligently enhance the challenging imaging environments, and becomes an extremely important key part especially in important application scenes with high requirements on vehicles, human images, human faces and the like, thereby not only improving the solid foundation for subsequent analysis, but also enabling the captured images to better accord with the perception of human beings and improving the experience of customers.
The image enhancement method based on the traditional algorithm needs to design more independent modules, such as a demosaicing module, a denoising module, a white balance module and the like, and needs to carefully adjust each parameter based on scenes of various environments, so that a certain effect can be ensured, and the obtained enhanced image cannot necessarily reach a perfect state, and the subsequent analysis and the experience of customers are damaged.
Disclosure of Invention
In view of the problems in the prior art, an object of the present invention is to provide an image enhancement method, system, device and storage medium based on deep learning, which can simultaneously enhance images of multiple items and improve image optimization effect.
The embodiment of the invention provides an image enhancement method based on deep learning, which comprises the following steps:
collecting sample images for model training;
training a plurality of first convolutional neural networks connected in parallel based on the sample images, wherein each first convolutional neural network corresponds to at least one enhancement item;
training a second convolutional neural network, which is connected in series after the plurality of first convolutional neural networks connected in parallel;
combining the trained first convolutional neural network and the second convolutional neural network to obtain an image enhancement model;
and inputting the image to be enhanced into the image enhancement model to obtain an enhanced image output by the image enhancement model.
Optionally, training a plurality of first convolutional neural networks connected in parallel based on the sample image comprises the following steps:
training each of the first convolutional neural networks individually using the sample image as an input to the first convolutional neural network;
jointly training the plurality of first convolutional neural networks based on the sample images.
Optionally, jointly training the plurality of first convolutional neural networks comprises the following steps:
dividing the first convolutional neural network into a plurality of network groups;
performing joint training on a plurality of first convolutional neural networks in the same network group;
multiple network groups are jointly trained.
Optionally, the jointly training the plurality of first convolutional neural networks in the same network group includes the following steps:
inputting the sample images into each first convolutional neural network, and constructing a loss function of each first convolutional neural network based on an output image and a label image of each first convolutional neural network;
combining the loss functions of the first convolutional neural networks in the same network group to obtain the loss function of the network group;
iteratively training each first convolutional neural network in the network group based on the loss function of the network group;
the joint training of the plurality of network groups comprises the following steps:
inputting the sample images into each first convolution neural network, and constructing a loss function of each convolution neural network based on an output image and a label image of each first convolution neural network;
combining the loss functions of the first convolutional neural networks to obtain an overall loss function;
each convolutional neural network in each network group is iteratively trained based on the overall loss function.
Optionally, dividing the first convolutional neural network into a plurality of network groups, including the following steps: acquiring a label image of each first convolutional neural network;
comparing the color histogram of the label image of the first convolutional neural network with the color histogram of the corresponding sample image;
and grouping the first convolutional neural networks according to the comparison result of the color histogram.
Optionally, training the second convolutional neural network comprises the steps of:
combining label images based on each first convolution neural network to obtain a first multichannel image;
and taking the first multichannel image as an input image of a second convolutional neural network, and iteratively training the second convolutional neural network.
Optionally, after the first multichannel image is used as an input image of a second convolutional neural network and the second convolutional neural network is iteratively trained, the method further includes the following steps:
combining the output images of the first convolution neural networks to obtain a second multi-channel image;
and taking the second multichannel image as an input image of a second convolutional neural network, and iteratively training the second convolutional neural network.
By adopting the image enhancement method based on deep learning, the convolutional neural network based on deep learning replaces a manual design module in an image signal processing flow, a plurality of parallel first convolutional neural networks and a series second convolutional neural network are adopted to enhance the image, and image enhancement processing in multiple aspects is realized at the same time, so that the respective image enhancement effect of each convolutional neural network can be ensured, and the effect enhancement can be realized mutually, and the overall image enhancement effect is improved.
The embodiment of the invention also provides an image enhancement system based on deep learning, which is applied to the image enhancement method based on deep learning, and the system comprises:
the sample acquisition module is used for acquiring sample images for model training;
a first training module, configured to train, based on the sample image, a plurality of first convolutional neural networks connected in parallel, where each of the first convolutional neural networks corresponds to at least one enhancement item;
a second training module for training a second convolutional neural network, the second convolutional neural network being connected in series after the plurality of first convolutional neural networks connected in parallel;
the model combination module is used for combining the trained first convolutional neural network and the trained second convolutional neural network to obtain an image enhancement model;
and the image enhancement model is used for inputting the image to be enhanced into the image enhancement model to obtain an enhanced image output by the image enhancement model.
By adopting the image enhancement system based on deep learning, the convolutional neural network based on deep learning replaces a manual design module in an image signal processing flow, a plurality of parallel first convolutional neural networks and a series second convolutional neural network are respectively trained through the first training module and the second training module, the obtained image enhancement model can simultaneously realize image enhancement processing in multiple aspects, the respective image enhancement effect of each convolutional neural network can be ensured, the effect improvement can be mutually realized, and the integral image enhancement effect is improved.
The embodiment of the present invention further provides an image enhancement device based on deep learning, including: a processor;
a memory having stored therein executable instructions of the processor;
wherein the processor is configured to perform the steps of the deep learning based image enhancement method via execution of the executable instructions.
By adopting the image enhancement device based on the deep learning provided by the invention, the processor executes the image enhancement method based on the deep learning when executing the executable instructions, thereby obtaining the beneficial effects of the image enhancement method based on the deep learning.
An embodiment of the present invention further provides a computer-readable storage medium for storing a program, where the program implements the steps of the image enhancement method based on deep learning when executed.
By adopting the computer-readable storage medium provided by the invention, the stored program realizes the steps of the image enhancement method based on the deep learning when being executed, thereby the beneficial effects of the image enhancement method based on the deep learning can be obtained.
Drawings
Other features, objects and advantages of the present invention will become more apparent upon reading of the following detailed description of non-limiting embodiments thereof, with reference to the accompanying drawings.
FIG. 1 is a flowchart of an image enhancement method based on deep learning according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of a process after an image is input into an image enhancement model according to an embodiment of the present invention;
FIG. 3 is a flow chart of training of an image enhancement model according to an embodiment of the invention;
FIG. 4 is a schematic diagram of an image enhancement system based on deep learning according to an embodiment of the present invention;
FIG. 5 is a schematic structural diagram of an image enhancement apparatus based on deep learning according to an embodiment of the present invention;
fig. 6 is a schematic structural diagram of a computer storage medium according to an embodiment of the present invention.
Detailed Description
Example embodiments will now be described more fully with reference to the accompanying drawings. Example embodiments may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of example embodiments to those skilled in the art. The same reference numerals in the drawings denote the same or similar structures, and thus their repetitive description will be omitted.
As shown in fig. 1, in this embodiment, the image enhancement method based on deep learning includes the following steps:
s100: collecting sample images for model training;
the sample image can be a sample image for face recognition, a sample image for vehicle recognition of a user, and the like, a large number of original images (raw format images) in various different scenes can be collected as the sample image, for example, images in various challenging scenes are collected, the sample image is composed of original images of landscapes, traffic vehicles, figures, faces, and the like in various time periods, and the robustness of a network can be improved by using a large number of original image data;
s200: training a plurality of first convolutional neural networks connected in parallel based on the sample image, wherein each first convolutional neural network corresponds to at least one enhancement item;
s300: training a second convolutional neural network, which is connected in series after the plurality of first convolutional neural networks connected in parallel;
s400: combining the trained first convolutional neural network and the second convolutional neural network to obtain an image enhancement model;
s500: and inputting the image to be enhanced into the image enhancement model to obtain an enhanced image output by the image enhancement model.
By adopting the image enhancement method based on deep learning, a plurality of first convolutional neural networks connected in parallel are trained through the steps S100 and S200, a second convolutional neural network connected in series is trained through the step S300, the image enhancement model is obtained through the combination of the step S400, and the image to be enhanced is input into the image enhancement model through the step S500, so that the image enhancement in multiple aspects can be realized simultaneously, the respective image enhancement effect of each convolutional neural network can be ensured, the effect improvement can be realized mutually, and the overall image enhancement effect is improved.
As shown in fig. 2, a schematic diagram of the processing procedure after the image is input into the image enhancement model is shown, wherein the relationship of each convolutional neural network in the image enhancement model is shown. The first convolutional neural network and the enhancement item may have a one-to-one correspondence, for example, a first convolutional neural network is used for denoising, a first convolutional neural network is used for contrast enhancement, a first convolutional neural network is used for color enhancement, and the like. Therefore, the method can perform intelligent image enhancement processing by utilizing the deep convolution neural network based on the original image obtained by the camera photosensitive element under the condition of the challenging conditions of insufficient illumination, excessive noise, uneven exposure and the like, so that the processed image is objective and undistorted and can conform to the human perception system. Because each convolution neural network in the image enhancement model forms a complete neural network, when the image enhancement model is applied to image enhancement, the inference is carried out once without intermediate operation.
The step S100: after the sample images for model training are acquired, the number of the directly acquired sample images and related scenes are limited, so that the trained sample images can be appropriately augmented to increase the richness of data. The method comprises the steps of cutting the image at random positions, randomly turning the image horizontally and randomly turning the image vertically. The augmentations occur with independent probability and are cascaded, so that the diversity of training data can be greatly enriched, the robustness of the model is improved, and the phenomenon of overfitting is prevented.
After the sample image is obtained, label images of the sample image corresponding to the convolutional neural networks are further obtained. Specifically, the label image corresponding to the sample image may be obtained by labeling with tools such as Adobe and Photoshop, and the label image corresponding to the sample image is obtained for each of the first convolutional neural network and the second convolutional neural network. The label images of each first convolutional neural network are different from each other. For example, the label image of the first convolutional neural network with adjusted contrast is the label image obtained by adjusting the contrast of the sample image, the label image of the first convolutional neural network with enhanced color is the label image obtained by color enhancing the sample image, and the label image of the second convolutional neural network is the label image obtained by performing targeted adjustment on each enhanced item in the sample image.
As shown in fig. 3, in this embodiment, step S200: training a plurality of first convolutional neural networks connected in parallel based on the sample image comprises the following steps:
s210: training each of the first convolutional neural networks individually using the sample image as an input to the first convolutional neural network;
specifically, the sample image is input into each first convolution neural network to obtain an output image of each first convolution neural network, a loss function is constructed according to a label image and the output image of the first convolution neural network, and the first convolution neural network is iteratively trained according to the loss function;
s220: jointly training the plurality of first convolutional neural networks based on the sample images.
According to the invention, the parallel first convolution neural networks are distributed and jointly trained through the steps S210 and S220 respectively, so that the combined first convolution neural network can achieve a good image enhancement effect.
Specifically, in constructing each first convolutional neural network, the convolutional layer, the pooling layer, the activation function layer, and the batch normalization layer may adopt a hierarchical structure in the existing convolutional neural network. All data of the data layer may be further normalized to a data distribution of zero mean unit variance, which is normalized by the following formula:
wherein,
is the mean value, v is the variance, and the calculation formula is as follows:
wherein x is i Is the data to be normalized, x' i N is the number of data for the normalized data.
In the invention, when each convolutional neural network is trained, the size of the predicted output image is consistent with that of the label image, the predicted output image and the label image are sent to a loss function layer to calculate loss, and the loss is iteratively minimized, so that the training purpose is achieved. The loss function used may be any of various existing loss functions. For example, in this embodiment, the loss function may be a smooth L1 loss function, which is calculated as follows:
the smooth L1 loss function is adopted, so that the simplicity, the efficiency and the practicability are high. The invention is not limited thereto and other types of loss functions are possible.
As shown in fig. 3, in this embodiment, the step S220: jointly training the plurality of first convolutional neural networks, comprising the steps of:
s221: the first convolutional neural network is divided into a plurality of network groups, for example, the first convolutional neural network 1, the first convolutional neural network 2, and the first convolutional neural network 3 shown in fig. 2 may be divided into one group, and the first convolutional neural network 4 to the first convolutional neural network n shown in fig. 2 may be divided into one group;
s222: the method comprises the following steps that joint training is carried out on a plurality of first convolutional neural networks in the same network group, so that the first convolutional neural networks in the same network group can be combined to well enhance an image;
s223: and performing joint training on a plurality of network groups to realize that all the first convolution neural network combinations can well enhance the image.
In this embodiment, the step S221: dividing the first convolutional neural network into a plurality of network groups, comprising the steps of:
acquiring a label image of each first convolutional neural network;
comparing the color histogram of the label image of the first convolutional neural network with the color histogram of the corresponding sample image;
and grouping the first convolution neural network according to the comparison result of the color histogram, namely grouping according to the change condition of the first convolution neural network to the color histogram.
Specifically, the first convolutional neural network may be divided into two groups: group1 and Group2. The first convolutional neural network in Group1 is a network that does not change the color histogram of the image, e.g., a first convolutional neural network corresponding to denoising, a first convolutional neural network corresponding to super-resolution reconstruction, etc. Thus, the network in Group1 may focus primarily on the learning of image content, without concern for other aspects. The first convolutional neural network in Group2 is a network that can change the color histogram of the image, for example, the first convolutional neural network corresponding to color enhancement, the first convolutional neural network corresponding to contrast enhancement, and the like, and the label image of the network in Group2 mainly takes human subjective factors as criteria and better conforms to human visual perception. Therefore, the network in Group2 can focus on fitting the perception of human eyes, and can learn information more suitable for human subjective perception, such as hue, exposure, contrast, etc.
The network training purposes in the same network group have basic unity through division of the color histogram, so that the difficulty of training each network in each group is naturally reduced, and each network in the group has a mutual promotion effect on the basis of the unified purposes. In other alternative embodiments, the first convolutional neural network may also be grouped based on other factors, and the number of groups is not limited to two groups.
In this embodiment, the step S222: the joint training of a plurality of first convolutional neural networks in the same network group comprises the following steps:
inputting the sample images into each first convolutional neural network, and constructing a loss function of each first convolutional neural network based on an output image and a label image of each first convolutional neural network;
combining the loss functions of the first convolutional neural networks in the same network group to obtain the loss function of the network group, wherein the loss function may adopt smooth L1 loss function or other types of loss functions, and during the combination, the weight of the loss function of each first convolutional neural network in the loss function of the network group may be equal, equal to the reciprocal of the number of the first convolutional neural networks in the network group, or may be set as different weights, which all fall within the protection scope of the present invention;
each first convolutional neural network in the network set is iteratively trained based on the loss functions of the network set.
The step S223: performing joint training on a plurality of network groups, comprising the following steps:
inputting the sample images into each first convolutional neural network, and constructing a loss function of each convolutional neural network based on an output image and a label image of each first convolutional neural network;
combining the loss functions of the first convolutional neural networks to obtain an overall loss function, wherein the weight of the loss function of each first convolutional neural network in the overall loss function can be equal to the reciprocal of the number of all the first convolutional neural networks during combination, and can also be set as different weights, and the weights belong to the protection scope of the invention;
each convolutional neural network in each network group is iteratively trained based on the overall loss function.
Through the iterative training of step S222, the networks in each network group have substantially reached the purpose of designing the respective modules. In the iterative training in step S223, the iteration may use a small learning rate, which is only a process of adapting each network group, but cannot change the model parameters obtained by the training in step S222 too much.
As shown in fig. 3, in this embodiment, the step S300: training the second convolutional neural network comprises the following steps:
s310: combining label images based on each first convolution neural network to obtain a first multichannel image;
s320: and taking the first multichannel image as an input image of a second convolutional neural network, and iteratively training the second convolutional neural network, specifically, constructing a loss function according to an output image of the second convolutional neural network and a label image in iterative training, wherein the loss function may adopt a smooth L1 loss function or other types of loss functions.
The steps S310 and S320 first depend on the label image of the first convolutional neural network connected in parallel before, but not the predicted output image, to perform iterative training on the second convolutional neural network, so that the second convolutional neural network does not depend on the training result of the previous part and can not be interfered by the first convolutional neural network connected in parallel before.
In this embodiment, the step S320: after the first multichannel image is used as an input image of a second convolutional neural network and the second convolutional neural network is iteratively trained, the method further comprises the following steps:
s330: combining the output images of the first convolutional neural networks to obtain a second multichannel image;
s340: and taking the second multichannel image as an input image of a second convolutional neural network, and iteratively training the second convolutional neural network, specifically, constructing a loss function according to an output image and a label image of the second convolutional neural network in iterative training, wherein the loss function may adopt a smooth L1 loss function or other types of loss functions.
According to the invention, the previous part of label images are used as input for training through the steps S310 and S320, so that the training of the second convolutional neural network can be ensured in the early stage, the interference of the previous part of network is avoided, and the convergence of the current second convolutional neural network can be ensured and accelerated. And through step S330 and step S340, after the network of this part converges, the network parameters of the first convolutional neural network of the previous part of the robust are still needed in the later training period. Therefore, in the later training stage, the training of the second convolutional neural network is carried out based on the output image predicted by the first convolutional neural network, and the iterative training can be carried out at a very small learning rate, so that the whole image enhancement model has good compatibility before and after.
In the invention, the iterative training of each convolutional neural network can be performed by uniformly adopting SGD (Stochastic Gradient Descent) and using a learning rate change mode with linear Descent. The training platform of the convolutional neural network can adopt caffe or tensorflow, etc. The sample data augmentation and special loss functions after image acquisition in the image enhancement method are added into a training platform, and the method can be used for training each convolutional neural network after compiling and debugging.
By adopting the image enhancement method based on deep learning, the image enhancement effect can be greatly improved, the method can be applied to intelligent image enhancement in different scenes and different environments, a powerful basis is provided for subsequent analysis, and the perception degree of human eyes is met.
As shown in fig. 4, an embodiment of the present invention further provides a deep learning based image enhancement system, which is applied to the deep learning based image enhancement method, and the system includes:
the sample acquisition module M100 is used for acquiring sample images for model training;
a first training module M200, configured to train, based on the sample image, a plurality of first convolutional neural networks connected in parallel, where each first convolutional neural network corresponds to at least one enhancement item;
a second training module M300, configured to train a second convolutional neural network, where the second convolutional neural network is connected in series after the plurality of parallel first convolutional neural networks;
the model combination module M400 is used for combining the trained first convolutional neural network and the trained second convolutional neural network to obtain an image enhancement model;
and the image enhancement model M500 is used for inputting the image to be enhanced into the image enhancement model to obtain an enhanced image output by the image enhancement model.
By adopting the image enhancement system based on deep learning, a plurality of first convolutional neural networks connected in parallel are trained through the sample acquisition module M100 and the first training module M200, a second convolutional neural network connected in series is trained through the second training module M300, an image enhancement model is obtained through combination of the model combination module M400, and an image to be enhanced is input into the image enhancement model through the image enhancement model M500, so that image enhancement in multiple aspects can be realized simultaneously, the respective image enhancement effect of each convolutional neural network can be ensured, the effect improvement can be realized mutually, and the overall image enhancement effect is improved.
In the image enhancement system based on deep learning, the functions of the modules can be realized by adopting the specific implementation of the steps in the image enhancement method based on deep learning. For example, the sample acquisition module M100 may acquire a sample image by using the specific implementation manner of step S100, and may further perform an augmentation process on the sample image, the first training module M200 may train each first convolutional neural network by using the specific implementation manner of step S200, the second training module M300 may train each second convolutional neural network by using the specific implementation manner of step S300, and the image enhancement model M500 may implement the application of image enhancement by using the specific implementation manner of step S500.
The embodiment of the invention also provides image enhancement equipment based on deep learning, which comprises a processor; a memory having stored therein executable instructions of the processor; wherein the processor is configured to perform the steps of the deep learning based image enhancement method via execution of the executable instructions.
As will be appreciated by one skilled in the art, aspects of the present invention may be embodied as a system, method or program product. Thus, various aspects of the invention may be embodied in the form of: an entirely hardware embodiment, an entirely software embodiment (including firmware, microcode, etc.) or an embodiment combining hardware and software aspects that may all generally be referred to herein as a "circuit," module "or" system.
An electronic device 600 according to this embodiment of the invention is described below with reference to fig. 5. The electronic device 600 shown in fig. 5 is only an example and should not bring any limitation to the functions and the scope of use of the embodiments of the present invention.
As shown in fig. 5, the electronic device 600 is embodied in the form of a general purpose computing device. The components of the electronic device 600 may include, but are not limited to: at least one processing unit 610, at least one memory unit 620, a bus 630 that couples various system components including the memory unit 620 and the processing unit 610, a display unit 640, and the like.
Wherein the storage unit stores program code executable by the processing unit 610 to cause the processing unit 610 to perform steps according to various exemplary embodiments of the present invention described in the electronic prescription flow processing method section described above in this specification. For example, the processing unit 610 may perform the steps as shown in fig. 1.
The storage unit 620 may include readable media in the form of volatile storage units, such as a random access memory unit (RAM) 6201 and/or a cache storage unit 6202, and may further include a read-only memory unit (ROM) 6203.
The memory unit 620 may also include a program/utility 6204 having a set (at least one) of program modules 6205, such program modules 6205 including, but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which, or some combination thereof, may comprise an implementation of a network environment.
The electronic device 600 may also communicate with one or more external devices 700 (e.g., keyboard, pointing device, bluetooth device, etc.), with one or more devices that enable a user to interact with the electronic device 600, and/or with any devices (e.g., router, modem, etc.) that enable the electronic device 600 to communicate with one or more other computing devices. Such communication may occur via input/output (I/O) interface 650. Also, the electronic device 600 may communicate with one or more networks (e.g., a Local Area Network (LAN), a Wide Area Network (WAN), and/or a public network such as the Internet) via the network adapter 660. The network adapter 660 may communicate with other modules of the electronic device 600 via the bus 630. It should be appreciated that although not shown in the figures, other hardware and/or software modules may be used in conjunction with the electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, among others.
By adopting the image enhancement device based on deep learning provided by the invention, the processor executes the image enhancement method based on deep learning when executing the executable instruction, thereby obtaining the beneficial effects of the image enhancement method based on deep learning.
An embodiment of the present invention further provides a computer-readable storage medium for storing a program, where the program implements the steps of the image enhancement method based on deep learning when executed. In some possible embodiments, aspects of the present invention may also be implemented in the form of a program product comprising program code for causing a terminal device to perform the steps according to various exemplary embodiments of the present invention described in the above-mentioned electronic prescription flow processing method section of this specification, when the program product is run on the terminal device.
Referring to fig. 6, a program product 800 for implementing the above method according to an embodiment of the present invention is described, which may employ a portable compact disc read only memory (CD-ROM) and include program code, and may be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited in this regard and, in the present document, a readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
The computer readable storage medium may include a propagated data signal with readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A readable storage medium may also be any readable medium that is not a readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
Program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device, or entirely on the remote computing device or cluster. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computing device (e.g., through the internet using an internet service provider).
By adopting the computer-readable storage medium provided by the invention, the stored program realizes the steps of the image enhancement method based on the deep learning when being executed, thereby the beneficial effects of the image enhancement method based on the deep learning can be obtained.
The foregoing is a more detailed description of the invention in connection with specific preferred embodiments and it is not intended that the invention be limited to these specific details. For those skilled in the art to which the invention pertains, numerous simple deductions or substitutions may be made without departing from the spirit of the invention, which shall be deemed to belong to the scope of the invention.
Claims (8)
1. An image enhancement method based on deep learning is characterized by comprising the following steps:
collecting sample images for model training;
training a plurality of first convolutional neural networks connected in parallel based on the sample image, wherein each first convolutional neural network corresponds to at least one enhancement item;
training a second convolutional neural network, which is connected in series after the plurality of first convolutional neural networks connected in parallel;
combining the trained first convolutional neural network and the trained second convolutional neural network to obtain an image enhancement model;
inputting an image to be enhanced into the image enhancement model to obtain an enhanced image output by the image enhancement model;
training a plurality of first convolutional neural networks connected in parallel based on the sample image comprises the following steps:
training each of the first convolutional neural networks individually using the sample image as an input to the first convolutional neural network;
dividing the first convolutional neural network into a plurality of network groups;
performing joint training on a plurality of first convolutional neural networks in the same network group;
multiple network groups are jointly trained.
2. The deep learning based image enhancement method according to claim 1, wherein the jointly training a plurality of first convolutional neural networks in the same network group comprises the following steps:
inputting the sample images into each first convolutional neural network, and constructing a loss function of each first convolutional neural network based on an output image and a label image of each first convolutional neural network;
combining the loss functions of the first convolutional neural networks in the same network group to obtain the loss function of the network group;
iteratively training each first convolutional neural network in the network set based on the loss function of the network set;
the joint training of the plurality of network groups comprises the following steps:
inputting the sample images into each first convolution neural network, and constructing a loss function of each convolution neural network based on an output image and a label image of each first convolution neural network;
combining the loss functions of the first convolutional neural networks to obtain an overall loss function;
each convolutional neural network in each network group is iteratively trained based on the overall loss function.
3. The deep learning based image enhancement method according to claim 1, wherein the first convolutional neural network is divided into a plurality of network groups, and the method comprises the following steps:
acquiring a label image of each first convolutional neural network;
comparing the color histogram of the label image of the first convolutional neural network with the color histogram of the corresponding sample image;
and grouping the first convolutional neural network according to the comparison result of the color histogram.
4. The deep learning based image enhancement method of claim 1, wherein training the second convolutional neural network comprises the steps of:
combining label images based on each first convolution neural network to obtain a first multichannel image;
and taking the first multichannel image as an input image of a second convolutional neural network, and iteratively training the second convolutional neural network.
5. The image enhancement method based on deep learning of claim 4, wherein after the iterative training of the second convolutional neural network by using the first multichannel image as the input image of the second convolutional neural network, the method further comprises the following steps:
combining the output images of the first convolution neural networks to obtain a second multi-channel image;
and taking the second multichannel image as an input image of a second convolutional neural network, and iteratively training the second convolutional neural network.
6. An image enhancement system based on deep learning, which is applied to the image enhancement method based on deep learning of any one of claims 1 to 5, the system comprising:
the sample acquisition module is used for acquiring sample images for model training;
a first training module, configured to train, based on the sample image, a plurality of first convolutional neural networks connected in parallel, where each of the first convolutional neural networks corresponds to at least one enhancement item;
a second training module for training a second convolutional neural network, the second convolutional neural network being connected in series after the plurality of first convolutional neural networks connected in parallel;
the model combination module is used for combining the trained first convolutional neural network and the trained second convolutional neural network to obtain an image enhancement model;
and the image enhancement model is used for inputting the image to be enhanced into the image enhancement model to obtain an enhanced image output by the image enhancement model.
7. An image enhancement device based on deep learning, characterized by comprising:
a processor;
a memory having stored therein executable instructions of the processor;
wherein the processor is configured to perform the steps of the method of image enhancement based on deep learning of any one of claims 1 to 5 via execution of the executable instructions.
8. A computer-readable storage medium storing a program, wherein the program is configured to implement the steps of the deep learning based image enhancement method according to any one of claims 1 to 5 when executed.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010941865.3A CN112085681B (en) | 2020-09-09 | 2020-09-09 | Image enhancement method, system, device and storage medium based on deep learning |
| PCT/CN2021/082761 WO2022052445A1 (en) | 2020-09-09 | 2021-03-24 | Deep-learning-based image enhancement method, system and device, and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010941865.3A CN112085681B (en) | 2020-09-09 | 2020-09-09 | Image enhancement method, system, device and storage medium based on deep learning |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112085681A CN112085681A (en) | 2020-12-15 |
| CN112085681B true CN112085681B (en) | 2023-04-07 |
Family
ID=73732207
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010941865.3A Active CN112085681B (en) | 2020-09-09 | 2020-09-09 | Image enhancement method, system, device and storage medium based on deep learning |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN112085681B (en) |
| WO (1) | WO2022052445A1 (en) |
Families Citing this family (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112085681B (en) * | 2020-09-09 | 2023-04-07 | 苏州科达科技股份有限公司 | Image enhancement method, system, device and storage medium based on deep learning |
| CN113628116B (en) * | 2021-10-12 | 2022-02-11 | 腾讯科技（深圳）有限公司 | Training method and device for image processing network, computer equipment and storage medium |
| CN114677566B (en) * | 2022-04-08 | 2023-10-17 | 北京百度网讯科技有限公司 | Training method of deep learning model, object recognition method and device |
| CN114782830B (en) * | 2022-06-22 | 2022-09-06 | 天津大学 | Image deep learning feature enhancement method, system and device |
| CN115496131B (en) * | 2022-08-30 | 2023-06-13 | 北京华控智加科技有限公司 | Equipment health state classification method based on multiple pre-training neural networks |
| CN116363009B (en) * | 2023-03-31 | 2024-03-12 | 哈尔滨工业大学 | Method and system for enhancing rapid light-weight low-illumination image based on supervised learning |
| CN117173503B (en) * | 2023-08-16 | 2024-04-19 | 安徽派睿太赫兹医疗器械技术开发有限公司 | Fuzzy terahertz image recognition method and device based on deep learning and electronic equipment |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109583569A (en) * | 2018-11-30 | 2019-04-05 | 中控智慧科技股份有限公司 | A kind of multi-modal Feature fusion and device based on convolutional neural networks |
| CN111047532A (en) * | 2019-12-06 | 2020-04-21 | 广东启迪图卫科技股份有限公司 | Low-illumination video enhancement method based on 3D convolutional neural network |
| CN111415304A (en) * | 2020-02-26 | 2020-07-14 | 中国农业大学 | Underwater vision enhancement method and device based on cascade deep network |
Family Cites Families (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10140544B1 (en) * | 2018-04-02 | 2018-11-27 | 12 Sigma Technologies | Enhanced convolutional neural network for image segmentation |
| CN110909578A (en) * | 2018-09-18 | 2020-03-24 | 中国移动通信有限公司研究院 | Low-resolution image recognition method and device and storage medium |
| CN109753938B (en) * | 2019-01-10 | 2021-11-05 | 京东方科技集团股份有限公司 | Image recognition method and device, application of image recognition method and device, and training method of neural network |
| CN110675336A (en) * | 2019-08-29 | 2020-01-10 | 苏州千视通视觉科技股份有限公司 | Low-illumination image enhancement method and device |
| CN111079761B (en) * | 2019-11-05 | 2023-07-18 | 北京航空航天大学青岛研究院 | Image processing method, device and computer storage medium |
| CN112085681B (en) * | 2020-09-09 | 2023-04-07 | 苏州科达科技股份有限公司 | Image enhancement method, system, device and storage medium based on deep learning |
-
2020
- 2020-09-09 CN CN202010941865.3A patent/CN112085681B/en active Active
-
2021
- 2021-03-24 WO PCT/CN2021/082761 patent/WO2022052445A1/en active Application Filing
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109583569A (en) * | 2018-11-30 | 2019-04-05 | 中控智慧科技股份有限公司 | A kind of multi-modal Feature fusion and device based on convolutional neural networks |
| CN111047532A (en) * | 2019-12-06 | 2020-04-21 | 广东启迪图卫科技股份有限公司 | Low-illumination video enhancement method based on 3D convolutional neural network |
| CN111415304A (en) * | 2020-02-26 | 2020-07-14 | 中国农业大学 | Underwater vision enhancement method and device based on cascade deep network |
Also Published As
| Publication number | Publication date |
|---|---|
| WO2022052445A1 (en) | 2022-03-17 |
| CN112085681A (en) | 2020-12-15 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112085681B (en) | Image enhancement method, system, device and storage medium based on deep learning | |
| CN111079532B (en) | Video content description method based on text self-encoder | |
| CN110443222B (en) | Method and device for training face key point detection model | |
| CN109635918A (en) | The automatic training method of neural network and device based on cloud platform and preset model | |
| WO2022247005A1 (en) | Method and apparatus for identifying target object in image, electronic device and storage medium | |
| CN109416758A (en) | The method of neural network and neural metwork training | |
| CN106503723A (en) | A kind of video classification methods and device | |
| JP2008542911A (en) | Image comparison by metric embedding | |
| CN111292262B (en) | Image processing method, device, electronic equipment and storage medium | |
| CN113012068B (en) | Image denoising method, image denoising device, electronic equipment and computer-readable storage medium | |
| CN111292334B (en) | Panoramic image segmentation method and device and electronic equipment | |
| CN110838131A (en) | Method and device for realizing automatic cutout, electronic equipment and medium | |
| US20220156987A1 (en) | Adaptive convolutions in neural networks | |
| CN112419169B (en) | CNN medical CT image denoising method based on noise priori | |
| CN110807741B (en) | Training method of image processing network, image denoising method and device | |
| WO2021139351A1 (en) | Image segmentation method, apparatus, medium, and electronic device | |
| CN112416293A (en) | Neural network enhancement method, system and application thereof | |
| CN108831561A (en) | Generation method, device and the computer readable storage medium of influenza prediction model | |
| CN115526803A (en) | Non-uniform illumination image enhancement method, system, storage medium and device | |
| CN110580726B (en) | Dynamic convolution network-based face sketch generation model and method in natural scene | |
| CN117788946A (en) | Image processing method, device, electronic equipment and storage medium | |
| CN117392499A (en) | Method and device for detecting saliency target | |
| CN112183366A (en) | High-voltage power line bird nest detection method, system and machine readable medium | |
| CN116992941A (en) | Convolutional neural network pruning method and device based on feature similarity and feature compensation | |
| KR20210038027A (en) | Method for Training to Compress Neural Network and Method for Using Compressed Neural Network |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |