CN116363009B - Method and system for enhancing rapid light-weight low-illumination image based on supervised learning - Google Patents

Method and system for enhancing rapid light-weight low-illumination image based on supervised learning Download PDF

Info

Publication number
CN116363009B
CN116363009B CN202310343585.6A CN202310343585A CN116363009B CN 116363009 B CN116363009 B CN 116363009B CN 202310343585 A CN202310343585 A CN 202310343585A CN 116363009 B CN116363009 B CN 116363009B
Authority
CN
China
Prior art keywords
image
low
convolution
illumination
illumination image
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202310343585.6A
Other languages
Chinese (zh)
Other versions
CN116363009A (en
Inventor
成行
张雨
杨国辉
王越
王玉
漆保凌
王春晖
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Harbin Institute of Technology
Original Assignee
Harbin Institute of Technology
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Harbin Institute of Technology filed Critical Harbin Institute of Technology
Priority to CN202310343585.6A priority Critical patent/CN116363009B/en
Publication of CN116363009A publication Critical patent/CN116363009A/en
Application granted granted Critical
Publication of CN116363009B publication Critical patent/CN116363009B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0464—Convolutional networks [CNN, ConvNet]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/09—Supervised learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/40—Image enhancement or restoration by the use of histogram techniques
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/50—Image enhancement or restoration by the use of more than one image, e.g. averaging, subtraction
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/56—Extraction of image or video features relating to colour
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/774—Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/778—Active pattern-learning, e.g. online learning of image or video features
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10024—Color image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20112—Image segmentation details
    • G06T2207/20132—Image cropping

Abstract

A rapid light-weight low-illumination image enhancement method and system based on supervised learning belong to the technical field of digital image processing. The method aims to solve the problems of low processing speed and large dependence on the quantity of parameters in the existing low-illumination image enhancement method based on supervised learning. The invention firstly extracts the V-channel image I of the low-illumination image V Obtaining histogram information, then splicing the histogram information with an expected average brightness value, and extracting global information through a multi-layer perceptron; in pair I V Performing high-order curve adjustment to obtain I V′ The method comprises the steps of carrying out a first treatment on the surface of the The low-illumination image to be enhanced is simultaneously input into a local enhancement network module for processing, V channels corresponding to the input are firstly input into a first convolution unit after splicing operation, and the output of the first convolution unit is equal to I V′ After splicing, the output of the first convolution layer in the second convolution unit is spliced with the output of the second convolution unit and then is input into the third convolution unit, and the third convolution unit outputs an enhanced image after processing.

Description

Method and system for enhancing rapid light-weight low-illumination image based on supervised learning
Technical Field
The invention belongs to the technical field of digital image processing, and particularly relates to a method and a system for enhancing a low-illumination image.
Background
Images are one of the important sources of human information acquisition, however, when the ambient light is insufficient or the acquisition performance of equipment is limited, the acquired images are often low-illumination images, and the images have the problems of low brightness, low contrast, high noise, large color deviation and the like, so that the acquisition of useful information by people is seriously influenced. The low-illumination image enhancement methods commonly used at present can be divided into two main types, namely a traditional image enhancement method and a method based on deep learning:
conventional image enhancement methods mainly include histogram equalization, gamma correction, defogging or Retinex model-based methods, and other improved methods based on these methods. While these methods can significantly improve the brightness and contrast of the image, the subsequent separate denoising step or the variation-based joint iterative denoising process is time consuming and therefore unsuitable for real-time low-intensity image enhancement applications.
Image enhancement methods based on deep learning can be divided into two main categories, unsupervised learning and supervised learning. Generally, the method based on the unsupervised learning has faster processing speed and stronger robustness in various environments, but the method generally lacks a corresponding color correction and denoising method, and is more limited in processing subsequent tasks with high precision requirements. In contrast, the supervised learning-based method can generally solve various degradation problems existing in low-illumination images and remarkably improve the image quality, but most of the methods need complex network structures, are large in dependent parameter quantity and low in processing speed.
In addition, an image retouching method similar to the image enhancement method can be adopted for the problems of unsuitable brightness, poor contrast, color deviation and the like of the image. The method can work on a single pixel by basic repair operations and is therefore very fast and lightweight. But most of these methods do not take noise into account.
In summary, the conventional low-illumination image enhancement method mainly has the following problems: 1. the traditional low-illumination image enhancement method has the defects that the processing process is too long in time and cannot meet the requirement of low-illumination image real-time enhancement; 2. the existing low-illumination image enhancement method based on unsupervised learning cannot simultaneously solve the problems of low contrast, poor brightness, color distortion, serious noise and the like; 3. the existing low-illumination image enhancement method based on supervised learning is generally complex in network structure, large in parameter dependence and low in processing speed; 4. the image modification method is a lightweight method and processes very quickly, but most cannot handle noise problems. In summary, at present, the problems of low contrast, poor brightness, color distortion, serious noise and the like of a low-illumination image are rapidly, lightweight and effectively solved, and the method is still a great challenge in the field.
Disclosure of Invention
The invention aims to solve the problems of low processing speed and large dependence on parameters in the existing low-illumination image enhancement method based on supervised learning, and the problems of low contrast, poor brightness, color distortion and serious noise in the low-illumination image are difficult to effectively solve.
A rapid light-weight low-illumination image enhancement method based on supervised learning comprises the following steps:
inputting the low-illumination image I to be enhanced into an image enhancement network, and enhancing the low-illumination image by utilizing the image enhancement network to obtain a final enhanced image E; the low-illumination image I to be enhanced is a matrix of M x N x 3, M is the number of rows, N is the number of columns, and 3 is the three RGB color channels { R, G, B } of the image I;
the process of enhancing the low-illumination image by using the image enhancement network comprises the following steps:
step 111, extracting a V-channel image I of the low-illumination image I in the HSV color space V ；
Step 112, obtaining V-channel image I V The histogram information is spliced with an expected average brightness value mu and then is input into a global feature extraction module of the network, and the global feature extraction module extracts global information through a multi-layer perceptron:
{α 0，1…t }G(H(I V )，μ)
wherein alpha is 0 、α 1 、……、α t The μ is an average brightness value, which is an element in the output of the multi-layer perceptron and is used for controlling the image enhancement degree; h (I) V ) Representing acquired image I V An operation of a histogram; g (·) represents the global feature extraction module process;
representing elements in the output of the multi-layer perceptron as higher-order curve coefficients;
step 113, performing high-order curve adjustment on the input low-illumination image V channel image to obtain a global brightness adjustment scheme, and finally outputting I V Global brightness adjustment image I of (2) V ′：
Where k represents the number of iterations, for example:representing an input low-intensity image V-channel image I V When k=7, the number of the cells,α k-1 ∈{α 0,1…t }；
the low-illumination image to be enhanced is input into a local enhancement network module for processing; the local enhancement network module comprises three convolution units, namely a first convolution unit to a third convolution unit, wherein each convolution unit comprises a plurality of convolution layers;
the local enhancement network module firstly performs splicing operation on the V channels corresponding to the input and then inputs the V channels into a first convolution unit, and the output of the first convolution unit and the global brightness adjustment image I V The output of the first convolution layer in the second convolution unit and the final output of the second convolution unit are input into a third convolution unit after the splicing operation, the third convolution unit is connected with the output layer, and the output of the output layer is an enhanced image.
Further, the first to third convolution units each include two convolution layers.
Further, the convolution layers in the first to third convolution units are all the convolution layers of 3*3.
Further, the training process of the image enhancement network comprises the following steps:
taking the collected low-illumination image and the corresponding normal-illumination image as an image pair, and constructing a training data set, wherein the low-illumination image is taken as an input, and the corresponding normal-illumination image is taken as a reference image;
training the image enhancement network by using a training data set, wherein the specific training process comprises the following steps:
cutting and dicing the low-illumination image and the normal-illumination image in the image pair in the same mode, wherein the sizes of the diced image blocks are M 'and N'; training by using randomly cut low-illumination image blocks, wherein the input in the actual training is M '×N' ×3×n, and N is the number of the image blocks input in each training;
before inputting an image block into an image enhancement network, respectively obtaining corresponding V-channel images for each low-illumination image, and obtaining histogram information of the V-channel images; meanwhile, taking the average brightness value of the normal illumination image corresponding to the low illumination image as an expected average brightness value mu; extracting global information through a multi-layer perceptron; representing elements in the output of the multi-layer perceptron as higher-order curve coefficients;
since the training process inputs a plurality of image blocks at a time, for each image block, a V-channel image I is extracted V Then, the high-order curve coefficient of the whole image corresponding to the current image block is used for adjustment to obtain a global brightness adjustment image I V ′；
Training of the image enhancement network is achieved based on the image blocks in the training data set, and the trained image enhancement network is obtained. Further, the loss function of the image enhancement network in the training process is as follows:
L ALL ＝L 1 +L SSIM +L color +L brightness +L structure
wherein the first item L 1 Representing the mean absolute value error, the second term L SSIM To enhance SSIM loss between image and reference image, the third term L color For colour loss, fourth term L brightness For loss of brightness, the fifth item L structure Is a structural loss;
the color loss function L color The method comprises the following steps:
wherein T represents the processing of the image enhancement network; i (i ， j) 、Y (i ， j) The ith row and jth column pixels of the low-illumination image and the reference normal-illumination image respectively; m and n respectively represent m and n pixels for each row and each column of the input image;<·，·>representing cosine similarity of the two vectors;
the brightness loss function L brightness The method comprises the following steps:
wherein, c takes R, G, B, which corresponds to three color channels of red, green and blue in RGB color space; b (T (I) c (i，j) )))、b(Y c (i，j) ) Represented as pixels T (I c (i，j) )、Y c (i，j) The purpose of subtracting the minimum value is to eliminate the effect of the constant in b (·) for the centered image block;
the structure loss function L structure The method comprises the following steps:
wherein,is a gradient operator.
A rapid light-weight low-illumination image enhancement system based on supervised learning comprises an image acquisition unit and an image enhancement unit;
an image acquisition unit: the method comprises the steps of receiving a low-illumination image to be enhanced, and processing the image I into a matrix of M x N x 3, wherein M is the number of rows, N is the number of columns, and 3 is three RGB color channels { R, G, B } of the image I;
an image enhancement unit: the method comprises the steps of calling an image enhancement network to enhance a low-illumination image to obtain a final enhanced image E; the process of enhancing the low-illumination image by the image enhancement network comprises the following steps:
step 111, extracting a V-channel image I of the low-illumination image I in the HSV color space V ；
Step 112, obtaining V-channel image I V The histogram information is spliced with an expected average brightness value mu and then is input into a global feature extraction module of the network, and the global feature extraction module extracts global information through a multi-layer perceptron:
wherein alpha is 0 、α 1 、……、α t The μ is an average brightness value, which is an element in the output of the multi-layer perceptron and is used for controlling the image enhancement degree; h (I) V ) Representing acquired image I V An operation of a histogram; g (·) represents the global feature extraction module process;
representing elements in the output of the multi-layer perceptron as higher-order curve coefficients;
step 113, performing high-order curve adjustment on the input low-illumination image V channel image to obtain a global brightness adjustment scheme, and finally outputting I V Global brightness adjustment image I of (2) V ′：
Where k represents the number of iterations, for example:representing an input low-intensity image V-channel image I V When k=7, the number of the cells,α k-1 ∈{α 0,1…t }；
the low-illumination image to be enhanced is input into a local enhancement network module for processing; the local enhancement network module comprises three convolution units, namely a first convolution unit to a third convolution unit, wherein each convolution unit comprises a plurality of convolution layers;
the local enhancement network module firstly performs splicing operation on the V channels corresponding to the input and then inputs the V channels into a first convolution unit, and the output of the first convolution unit and the global brightness adjustment image I V ' after the splicing operation, the output of the first convolution layer in the second convolution unit and the final output of the second convolution unit are input into a third convolution unit, the third convolution unit is connected with the output layer and outputsThe output of the layer is an enhanced image.
Further, the first to third convolution units each include two convolution layers.
Further, the convolution layers in the first to third convolution units are all the convolution layers of 3*3.
A computer storage medium having stored therein at least one instruction that is loaded and executed by a processor to implement the fast light weight low light intensity image enhancement system based on supervised learning.
A fast light-weight low-light image enhancement device based on supervised learning, the device comprising a processor and a memory, the memory storing at least one instruction, the at least one instruction being loaded and executed by the processor to implement the fast light-weight low-light image enhancement system based on supervised learning.
The beneficial effects are that:
the invention not only can process with smaller parameter quantity by introducing the designed rapid lightweight image enhancement network and solving the typical low-illumination image degradation problem, but also can obviously reduce the network complexity, thereby having faster processing speed; and meanwhile, the brightness and contrast of the image, the color and structural information are restored and noise is removed. Compared with the current most advanced low-illumination image enhancement method, the method has excellent comprehensive performance.
Drawings
Fig. 1 is a schematic diagram of a low-light image enhancement flow.
Fig. 2 is a schematic diagram of a fast lightweight low-light image enhancement network architecture.
FIG. 3 is an original low-light image effect before enhancement in an embodiment.
Fig. 4 is an enhanced image effect in an embodiment.
Fig. 5 is a reference normal illuminance image effect in the embodiment.
Detailed Description
The invention provides a rapid and lightweight low-illumination image enhancement network, which comprises a global feature extraction module and a local enhancement network module, constructs a loss function based on relative information and is used for end-to-end training of low-illumination images so as to improve the overall performance of the images. The invention is further described in connection with the following detailed description.
The first embodiment is as follows: this embodiment will be described with reference to fig. 1.
The embodiment is a rapid light-weight low-illumination image enhancement method based on supervised learning, comprising the following steps:
step one, designing a rapid lightweight low-illumination image enhancement network;
as shown in fig. 2, the processing procedure of the fast lightweight low-light image enhancement network is as follows:
for the low-illumination image to be enhanced, firstly extracting a V-channel image I of an HSV color space of the low-illumination image to be enhanced V Obtaining histogram information of the V-channel image; then, the histogram information and the expected average brightness value are spliced and then input into a global feature extraction module of the network, wherein the global feature extraction module is realized by a 5-layer multi-layer perceptron; generating a global brightness adjustment scheme by a high-order curve adjustment method to obtain a global brightness adjusted image of the V-channel image;
the specific process comprises the following steps:
step 111, extracting a V-channel image I of the low-illumination image I in the HSV color space V ；
Step 112, obtaining V-channel image I V The histogram information is spliced with an expected average brightness value mu and then is input into a global feature extraction module of the network, and the global feature extraction module extracts global information through a five-layer multi-layer perceptron:
{α 0，1…t }G(H(I V )，μ)
wherein alpha is 0 、α 1 、……、α t The μ is an average brightness value, which is an element in the output of the multi-layer perceptron and is used for controlling the image enhancement degree; h (I) V ) Representing acquired image I V An operation of a histogram; g (·) represents the global feature extraction module process;
representing elements in the output of the multi-layer perceptron as higher-order curve coefficients;
the average luminance value μ is an expected average luminance value when the enhancement is actually performed, and is set according to the expectation.
Step 113, performing high-order curve adjustment on the input low-illumination image V channel image to obtain a global brightness adjustment scheme, and finally outputting I V Global brightness adjustment image I of (2) V ′：
Wherein k=0, 1, …, t;representing an input low-intensity image V-channel image I V ，/>Representing k iterative images;
the low-illumination image to be enhanced is input into a local enhancement network module for processing; the local enhancement network module firstly performs splicing operation on V channels input in an HSV color space corresponding to the input, and then inputs the V channels into a first convolution layer, wherein the first convolution layer is connected with a second convolution layer, and the first convolution layer and the second convolution layer are both convolution layers of 3*3;
output of the second convolution layer and global brightness adjustment image I V Inputting a third convolution layer after splicing operation, wherein the third convolution layer is connected with a fourth convolution layer, and the third convolution layer and the fourth convolution layer are both convolution layers of 3*3;
the output of the third convolution layer and the output of the fourth convolution layer are input into a fifth convolution layer after being spliced, the fifth convolution layer is connected with a sixth convolution layer, and the fifth convolution layer and the sixth convolution layer are both convolution layers of 3*3;
the sixth convolution unit is connected to the output layer and outputs an illumination image E.
And splicing and subsequent processing are carried out on the global brightness adjustment image output by the global feature extraction module and the output of the second convolution layer of the local enhancement network module, so that local enhancement can be effectively realized.
Step two, acquiring any number of image pairs formed by low-illumination images and normal-illumination images to construct a training data set, wherein the input of an image enhancement network is only n low-illumination images, and the corresponding normal-illumination images are used as reference images;
training the rapid lightweight low-illumination image enhancement network by using a training data set, wherein the training process comprises the following steps of:
step 311, cutting and dicing the low-illumination image and the normal-illumination image in the image pair in the same manner, wherein the size of the diced image block is M '×n', M ', N' can be arbitrarily set, and in some embodiments, the size of the image block is 100×100; training by using randomly cut low-illumination image blocks, wherein in the actual training, the input is M 'N' 3 x N, N is the number of the image blocks input in each training, and in some embodiments, 171 small blocks are randomly selected in each training;
what is to be described here is: before inputting an image block into a rapid lightweight low-illumination image enhancement network, respectively obtaining a corresponding V-channel image for each low-illumination image, and obtaining histogram information of the V-channel image; meanwhile, taking the average brightness value of the normal illumination image corresponding to the low illumination image as an expected average brightness value mu (the average brightness value mu in the training process is determined for the normal illumination image corresponding to each low illumination image); extracting global information through a multi-layer perceptron; representing elements in the output of the multi-layer perceptron as higher-order curve coefficients;
since the training process inputs a plurality of image blocks at a time, for each image block, a V-channel image I is extracted V Then using the higher-order curve coefficient of the whole image corresponding to the current image blockAdjusting to obtain a global brightness adjustment image I V ′；
For the first input of the training process, the global brightness adjustment image I V Splicing and subsequent processing are carried out on the output of the second convolution layer of the 'and local enhancement network module, so that local enhancement is realized, and a primarily enhanced image E' is obtained;
step 312, comparing the preliminary enhanced image E' with the normal illuminance reference image Y paired with the input low illuminance image, and constructing a loss function based on the relative information to enhance the generalization ability of the neural network, wherein the loss function is as follows:
L ALL ＝L 1 +L SSIM +L color +L brightness +L structure
wherein the first item L 1 Representing the mean absolute value error, the second term L SSIM To enhance SSIM loss between image and reference image, the third term L color For colour loss, fourth term L brightness For loss of brightness, the fifth item L structure Is a structural loss.
The color loss function L color The method comprises the following steps:
wherein T represents the processing of the fast lightweight low-illumination image enhancement network; i (i ， j) 、Y (i ， j) The ith row and jth column pixels of the low-illumination image and the reference normal-illumination image respectively; m and n respectively represent m and n pixels for each row and each column of the input image;<·，·>representing cosine similarity of the two vectors.
The brightness loss function L brightness The method comprises the following steps:
wherein, c takes R, G, B corresponding to red, green and blue under RGB color spaceColor channels; b (T (I) c (i，j) )))、b(Y c (i，j) ) Represented as pixels T (I c (i，j) )、Y c (i，j) The purpose of subtracting the minimum is to eliminate the effect of the constant in b (·) for the centered image block.
The structure loss function L structure The method comprises the following steps:
wherein,is a gradient operator.
Step 313, training the network using Adam random optimization algorithm, and setting the update rate to 0.0001.
Based on the image blocks in the training data set, training the image enhancement network according to the loss function to obtain a trained image enhancement network, and taking the output of the trained image enhancement network as an enhanced image E.
And fifthly, inputting the low-illumination image to be enhanced into a trained image enhancement network, and outputting the network as an enhanced image.
The low-illumination image I to be enhanced is a matrix of M x N x 3, M is the number of rows, N is the number of columns, and 3 is the three RGB color channels { R, G, B } of the image I; the final enhanced image E obtained by enhancing the image I is a matrix of m×n×3.
The second embodiment is as follows:
the embodiment is a rapid light-weight low-illumination image enhancement system based on supervised learning, which is characterized by comprising an image acquisition unit and an image enhancement unit;
an image acquisition unit: the method comprises the steps of receiving a low-illumination image to be enhanced, and processing the image I into a matrix of M x N x 3, wherein M is the number of rows, N is the number of columns, and 3 is three RGB color channels { R, G, B } of the image I;
an image enhancement unit: the method comprises the steps of calling an image enhancement network to enhance a low-illumination image to obtain a final enhanced image E; the process of enhancing the low-illumination image by the image enhancement network comprises the following steps:
step 111, extracting a V-channel image I of the low-illumination image I in the HSV color space V ；
Step 112, obtaining V-channel image I V The histogram information is spliced with an expected average brightness value mu and then is input into a global feature extraction module of the network, and the global feature extraction module extracts global information through a multi-layer perceptron:
{α 0，1…t }G(H(I V )，μ)
wherein alpha is 0 、α 1 、……、α t The μ is an average brightness value, which is an element in the output of the multi-layer perceptron and is used for controlling the image enhancement degree; h (I) V ) Representing acquired image I V An operation of a histogram; g (·) represents the global feature extraction module process;
representing elements in the output of the multi-layer perceptron as higher-order curve coefficients;
step 113, performing high-order curve adjustment on the input low-illumination image V channel image to obtain a global brightness adjustment scheme, and finally outputting I V Global brightness adjustment image I of (2) V ′：
Where k represents the number of iterations, for example:representing an input low-intensity image V-channel image I V When k=7, the number of the cells,α k-1 ∈{α 0,1…t }；
the low-illumination image to be enhanced is input into a local enhancement network module for processing; the local enhancement network module comprises three convolution units, namely a first convolution unit to a third convolution unit, wherein each convolution unit comprises a plurality of convolution layers;
the local enhancement network module firstly performs splicing operation on the V channels corresponding to the input and then inputs the V channels into a first convolution unit, and the output of the first convolution unit and the global brightness adjustment image I V′ And after the splicing operation is performed, the output of the first convolution layer in the second convolution unit and the final output of the second convolution unit are input into a third convolution unit, the third convolution unit is connected with the output layer, and the output of the output layer is an enhanced image.
Preferably, each of the first to third convolution units includes two convolution layers.
Preferably, the convolution layers in the first to third convolution units are all convolution layers of 3*3.
And a third specific embodiment:
the embodiment is a computer storage medium having at least one instruction stored therein, the at least one instruction being loaded and executed by a processor to implement the supervised learning based rapid lightweight low illumination image enhancement system.
It should be understood that the instructions comprise a computer program product, software, or computerized method corresponding to any of the methods described herein; the instructions may be used to program a computer system, or other electronic device. Computer storage media may include readable media having instructions stored thereon and may include, but is not limited to, magnetic storage media, optical storage media; magneto-optical storage media include read-only memory ROM, random-access memory RAM, erasable programmable memory (e.g., EPROM and EEPROM), and flash memory layers, or other types of media suitable for storing electronic instructions.
The specific embodiment IV is as follows:
the embodiment is a rapid light-weight low-illumination image enhancement device based on supervised learning, the device comprises a processor and a memory, and it should be understood that any device comprising the processor and the memory described by the invention can also comprise other units and modules for displaying, interacting, processing, controlling and other functions through signals or instructions;
the memory stores at least one instruction that is loaded and executed by the processor to implement the supervised learning based fast lightweight low illumination image enhancement system.
Examples
The low-light image enhancement experiment was performed according to the method of the first embodiment.
Embodiments of the present invention use pre-prepared datasets. The original low-luminance image before enhancement is shown in fig. 3, the effect after enhancement is shown in fig. 4, and the reference normal-luminance image is shown in fig. 5.
From the experimental results, it can be seen that: the rapid light-weight low-illumination image enhancement network based on supervised learning can remarkably enhance the brightness and contrast of images and retain the details and color information of the images. In addition, the network has simple structure, small dependent parameter quantity and high processing speed in experiments. In conclusion, the rapid light-weight low-illumination image enhancement method based on supervised learning has rapid, light-weight and remarkable low-illumination image enhancement effects.
The above examples of the present invention are only for describing the calculation model and calculation flow of the present invention in detail, and are not limiting of the embodiments of the present invention. Other variations and modifications of the above description will be apparent to those of ordinary skill in the art, and it is not intended to be exhaustive of all embodiments, all of which are within the scope of the invention.

Claims (10)

1. A rapid light-weight low-illumination image enhancement method based on supervised learning is characterized by comprising the following steps:
inputting the low-illumination image I to be enhanced into an image enhancement network, and enhancing the low-illumination image by utilizing the image enhancement network to obtain a final enhanced image E; the low-illumination image I to be enhanced is a matrix of M x N x 3, M is the number of rows, N is the number of columns, and 3 is the three RGB color channels { R, G, B } of the image I;
the process of enhancing the low-illumination image by using the image enhancement network comprises the following steps:
step 111, extracting a V-channel image I of the low-illumination image I in the HSV color space V ；
Step 112, obtaining V-channel image I V The histogram information is spliced with an expected average brightness value mu and then is input into a global feature extraction module of the network, and the global feature extraction module extracts global information through a multi-layer perceptron:
{α 0，1...t }＝G(H(I V )，μ)
wherein alpha is 0 、α 1 、……、α t The μ is an average brightness value, which is an element in the output of the multi-layer perceptron and is used for controlling the image enhancement degree; h (I) V ) Representing acquired image I V An operation of a histogram; g (·) represents the global feature extraction module process;
representing elements in the output of the multi-layer perceptron as higher-order curve coefficients;
step 113, performing high-order curve adjustment on the input low-illumination image V channel image to obtain a global brightness adjustment scheme, and finally outputting I V Global brightness adjustment image I of (2) V′ ：
Where k represents the number of iterations,representing an input low-intensity image V-channel image I V When k=7, _j->α k-1 ∈{α 0，1...t }；
The low-illumination image to be enhanced is input into a local enhancement network module for processing; the local enhancement network module comprises three convolution units, namely a first convolution unit to a third convolution unit, wherein each convolution unit comprises a plurality of convolution layers;
the local enhancement network module firstly performs splicing operation on the V channels corresponding to the input and then inputs the V channels into a first convolution unit, and the output of the first convolution unit and the global brightness adjustment image I V′ And after the splicing operation is performed, the output of the first convolution layer in the second convolution unit and the final output of the second convolution unit are input into a third convolution unit, the third convolution unit is connected with the output layer, and the output of the output layer is an enhanced image.
2. The method of claim 1, wherein the first to third convolution units each include two convolution layers.
3. The method of claim 2, wherein the convolution layers in the first to third convolution units are all convolution layers of 3*3.
4. A method for rapid lightweight low-light image enhancement based on supervised learning as set forth in claim 1, 2, or 3, wherein the training process of the image enhancement network comprises the steps of:
taking the collected low-illumination image and the corresponding normal-illumination image as an image pair, and constructing a training data set, wherein the low-illumination image is taken as an input, and the corresponding normal-illumination image is taken as a reference image;
training the image enhancement network by using a training data set, wherein the specific training process comprises the following steps:
cutting and dicing the low-illumination image and the normal-illumination image in the image pair in the same mode, wherein the sizes of the diced image blocks are M 'and N'; training by using randomly cut low-illumination image blocks, wherein the input in the actual training is M '×N' ×3×n, and N is the number of the image blocks input in each training;
before inputting an image block into an image enhancement network, respectively obtaining corresponding V-channel images for each low-illumination image, and obtaining histogram information of the V-channel images; meanwhile, taking the average brightness value of the normal illumination image corresponding to the low illumination image as an expected average brightness value mu; extracting global information through a multi-layer perceptron; representing elements in the output of the multi-layer perceptron as higher-order curve coefficients;
since the training process inputs a plurality of image blocks at a time, for each image block, a V-channel image I is extracted V Then, the high-order curve coefficient of the whole image corresponding to the current image block is used for adjustment to obtain a global brightness adjustment image I V′ ；
Training of the image enhancement network is achieved based on the image blocks in the training data set, and the trained image enhancement network is obtained.
5. The method for enhancing a rapid light-weight low-illumination image based on supervised learning as recited in claim 4, wherein the loss function of the image enhancement network in the training process is as follows:
L ALL ＝L 1 +L SSIM +L color +L brightness +L structure
wherein the first item L 1 Representing the mean absolute value error, the second term L SSIM To enhance SSIM loss between image and reference image, the third term L color For colour loss, fourth term L brightness For loss of brightness, the fifth item L structure Is a structural loss;
the color loss function L color The method comprises the following steps:
wherein T represents the processing of the image enhancement network; i (i，j) 、Y (i，j) The ith row and jth column pixels of the low-illumination image and the reference normal-illumination image respectively; m and n respectively represent m and n pixels for each row and each column of the input image;<·，·>representing cosine similarity of the two vectors;
the brightness loss function L brightness The method comprises the following steps:
wherein, c takes R, G, B, which corresponds to three color channels of red, green and blue in RGB color space; b (T (I) c (i，j) ))、b(Y c (i，j) ) Expressed in pixelsY c (i，j) The purpose of subtracting the minimum value is to eliminate the effect of the constant in b (·) for the centered image block;
the structure loss function L structure The method comprises the following steps:
wherein,is a gradient operator.
6. A rapid light-weight low-illumination image enhancement system based on supervised learning is characterized by comprising an image acquisition unit and an image enhancement unit;
an image acquisition unit: the method comprises the steps of receiving a low-illumination image to be enhanced, and processing the image I into a matrix of M x N x 3, wherein M is the number of rows, N is the number of columns, and 3 is three RGB color channels { R, G, B } of the image I;
an image enhancement unit: the method comprises the steps of calling an image enhancement network to enhance a low-illumination image to obtain a final enhanced image E; the process of enhancing the low-illumination image by the image enhancement network comprises the following steps:
step 111, extracting a V-channel image I of the low-illumination image I in the HSV color space V ；
Step 112, obtaining V-channel image I V The histogram information is spliced with an expected average brightness value mu and then is input into a global feature extraction module of the network, and the global feature extraction module extracts global information through a multi-layer perceptron:
{α 0，1...t }＝G(H(I V )，μ)
wherein alpha is 0 、α 1 、……、α t The μ is an average brightness value, which is an element in the output of the multi-layer perceptron and is used for controlling the image enhancement degree; h (I) V ) Representing acquired image I V An operation of a histogram; g (·) represents the global feature extraction module process;
representing elements in the output of the multi-layer perceptron as higher-order curve coefficients;
step 113, performing high-order curve adjustment on the input low-illumination image V channel image to obtain a global brightness adjustment scheme, and finally outputting I V Global brightness adjustment image I of (2) V′ ：
Where k represents the number of iterations,representing an input low-intensity image V-channel image I V When k=7, _j->α k-1 ∈{α 0，1...t }；
The low-illumination image to be enhanced is input into a local enhancement network module for processing; the local enhancement network module comprises three convolution units, namely a first convolution unit to a third convolution unit, wherein each convolution unit comprises a plurality of convolution layers;
the local enhancement network module firstly performs splicing operation on the V channels corresponding to the input and then inputs the V channels into a first convolution unit, and the output of the first convolution unit and the global brightness adjustment image I V′ And after the splicing operation is performed, the output of the first convolution layer in the second convolution unit and the final output of the second convolution unit are input into a third convolution unit, the third convolution unit is connected with the output layer, and the output of the output layer is an enhanced image.
7. A supervised learning based fast light weight low illumination image enhancement system according to claim 6, wherein the first to third convolution units each comprise two convolution layers.
8. The supervised learning based fast light weight low illumination image enhancement system according to claim 7, wherein the convolution layers in the first through third convolution units are each 3*3.
9. A computer storage medium having stored therein at least one instruction that is loaded and executed by a processor to implement a supervised learning based fast lightweight low illumination image enhancement system as recited in any of claims 6 to 8.
10. A supervised learning based fast lightweight low illumination image enhancement apparatus, the apparatus comprising a processor and a memory having at least one instruction stored therein, the at least one instruction being loaded and executed by the processor to implement a supervised learning based fast lightweight low illumination image enhancement system as recited in any of claims 6 to 8.
CN202310343585.6A 2023-03-31 2023-03-31 Method and system for enhancing rapid light-weight low-illumination image based on supervised learning Active CN116363009B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202310343585.6A CN116363009B (en) 2023-03-31 2023-03-31 Method and system for enhancing rapid light-weight low-illumination image based on supervised learning

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202310343585.6A CN116363009B (en) 2023-03-31 2023-03-31 Method and system for enhancing rapid light-weight low-illumination image based on supervised learning

Publications (2)

Publication Number Publication Date
CN116363009A CN116363009A (en) 2023-06-30
CN116363009B true CN116363009B (en) 2024-03-12

Family

ID=86915638

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202310343585.6A Active CN116363009B (en) 2023-03-31 2023-03-31 Method and system for enhancing rapid light-weight low-illumination image based on supervised learning

Country Status (1)

Country Link
CN (1) CN116363009B (en)

Citations (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109255758A (en) * 2018-07-13 2019-01-22 杭州电子科技大学 Image enchancing method based on full 1*1 convolutional neural networks
CN111402145A (en) * 2020-02-17 2020-07-10 哈尔滨工业大学 Self-supervision low-illumination image enhancement method based on deep learning
CN112308803A (en) * 2020-11-25 2021-02-02 哈尔滨工业大学 Self-supervision low-illumination image enhancement and denoising method based on deep learning
CN112465727A (en) * 2020-12-07 2021-03-09 北京邮电大学 Low-illumination image enhancement method without normal illumination reference based on HSV color space and Retinex theory
CN113191995A (en) * 2021-04-30 2021-07-30 东北大学 Video image automatic exposure correction method based on deep learning
WO2022052445A1 (en) * 2020-09-09 2022-03-17 苏州科达科技股份有限公司 Deep-learning-based image enhancement method, system and device, and storage medium
CN114881879A (en) * 2022-05-17 2022-08-09 燕山大学 Underwater image enhancement method based on brightness compensation residual error network
CN115035011A (en) * 2022-06-09 2022-09-09 重庆邮电大学 Low-illumination image enhancement method for self-adaptive RetinexNet under fusion strategy
CN115147311A (en) * 2022-08-01 2022-10-04 北京吉利学院 Image enhancement method based on HSV and AM-RetinexNet
CN115205147A (en) * 2022-07-13 2022-10-18 福州大学 Multi-scale optimization low-illumination image enhancement method based on Transformer

Patent Citations (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109255758A (en) * 2018-07-13 2019-01-22 杭州电子科技大学 Image enchancing method based on full 1*1 convolutional neural networks
CN111402145A (en) * 2020-02-17 2020-07-10 哈尔滨工业大学 Self-supervision low-illumination image enhancement method based on deep learning
WO2022052445A1 (en) * 2020-09-09 2022-03-17 苏州科达科技股份有限公司 Deep-learning-based image enhancement method, system and device, and storage medium
CN112308803A (en) * 2020-11-25 2021-02-02 哈尔滨工业大学 Self-supervision low-illumination image enhancement and denoising method based on deep learning
CN112465727A (en) * 2020-12-07 2021-03-09 北京邮电大学 Low-illumination image enhancement method without normal illumination reference based on HSV color space and Retinex theory
CN113191995A (en) * 2021-04-30 2021-07-30 东北大学 Video image automatic exposure correction method based on deep learning
CN114881879A (en) * 2022-05-17 2022-08-09 燕山大学 Underwater image enhancement method based on brightness compensation residual error network
CN115035011A (en) * 2022-06-09 2022-09-09 重庆邮电大学 Low-illumination image enhancement method for self-adaptive RetinexNet under fusion strategy
CN115205147A (en) * 2022-07-13 2022-10-18 福州大学 Multi-scale optimization low-illumination image enhancement method based on Transformer
CN115147311A (en) * 2022-08-01 2022-10-04 北京吉利学院 Image enhancement method based on HSV and AM-RetinexNet

Non-Patent Citations (4)

* Cited by examiner, † Cited by third party
Title
Zero-referencedeep curve estimation for low-light image enhancement;Chunle Guo.et al;《InProceedings of the IEEE/CVF Conference on Computer Vi-sion and Pattern Recognition》;第1780–1789页 *
基于深度学习的低光彩码图像增强;方路平等;《浙江工业大学学报》(第04期);全文 *
基于深度神经网络的图像复原与质量评估方法研究;张雨;《中国优秀硕士学位论文全文数据库 信息科技辑》(第01期);全文 *
基于灰度变换及改进Retinex的低照度图像增强;游达章等;《红外技术》;第45卷(第02期);全文 *

Also Published As

Publication number Publication date
CN116363009A (en) 2023-06-30

Similar Documents

Publication Publication Date Title
CN109360171B (en) Real-time deblurring method for video image based on neural network
CN110232661B (en) Low-illumination color image enhancement method based on Retinex and convolutional neural network
CN111127308B (en) Mirror image feature rearrangement restoration method for single sample face recognition under partial shielding
Varga et al. Fully automatic image colorization based on Convolutional Neural Network
CN112308803B (en) Self-supervision low-illumination image enhancement and denoising method based on deep learning
CN111402145B (en) Self-supervision low-illumination image enhancement method based on deep learning
CN110930327B (en) Video denoising method based on cascade depth residual error network
CN112381897B (en) Low-illumination image enhancement method based on self-coding network structure
CN111127331B (en) Image denoising method based on pixel-level global noise estimation coding and decoding network
CN107463932B (en) Method for extracting picture features by using binary bottleneck neural network
CN108734677B (en) Blind deblurring method and system based on deep learning
CN112465727A (en) Low-illumination image enhancement method without normal illumination reference based on HSV color space and Retinex theory
CN114820408A (en) Infrared and visible light image fusion method based on self-attention and convolutional neural network
CN114565539A (en) Image defogging method based on online knowledge distillation
CN107220934A (en) Image rebuilding method and device
CN116664454B (en) Underwater image enhancement method based on multi-scale color migration parameter prediction
CN116363009B (en) Method and system for enhancing rapid light-weight low-illumination image based on supervised learning
CN107729885A (en) A kind of face Enhancement Method based on the study of multiple residual error
CN111798381A (en) Image conversion method, image conversion device, computer equipment and storage medium
CN110796716A (en) Image coloring method based on multiple residual error networks and regularized transfer learning
CN116109538A (en) Image fusion method based on simple gate unit feature extraction
CN113240589A (en) Image defogging method and system based on multi-scale feature fusion
CN111899201B (en) Low-illumination image enhancement method based on conditional re-enhancement network
CN114511462B (en) Visual image enhancement method
CN112164012B (en) Method and system for realizing portrait color relief effect

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant