CN117173054A - Ultra-light low-resolution dim light face enhancement method and system - Google Patents
Ultra-light low-resolution dim light face enhancement method and system Download PDFInfo
- Publication number
- CN117173054A CN117173054A CN202311239595.1A CN202311239595A CN117173054A CN 117173054 A CN117173054 A CN 117173054A CN 202311239595 A CN202311239595 A CN 202311239595A CN 117173054 A CN117173054 A CN 117173054A
- Authority
- CN
- China
- Prior art keywords
- face
- enhancement
- resolution
- image
- face image
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 68
- 238000000605 extraction Methods 0.000 claims abstract description 62
- 238000012549 training Methods 0.000 claims abstract description 21
- PCHJSUWPFVWCPO-UHFFFAOYSA-N gold Chemical compound [Au] PCHJSUWPFVWCPO-UHFFFAOYSA-N 0.000 claims abstract description 18
- 238000012360 testing method Methods 0.000 claims abstract description 18
- 230000002708 enhancing effect Effects 0.000 claims abstract description 8
- 230000004927 fusion Effects 0.000 claims description 50
- 238000000926 separation method Methods 0.000 claims description 18
- 230000006870 function Effects 0.000 claims description 16
- 238000010586 diagram Methods 0.000 claims description 15
- 238000011156 evaluation Methods 0.000 claims description 14
- 230000004913 activation Effects 0.000 claims description 10
- 238000004422 calculation algorithm Methods 0.000 claims description 9
- 238000004590 computer program Methods 0.000 claims description 9
- 239000011159 matrix material Substances 0.000 claims description 9
- 238000013139 quantization Methods 0.000 claims description 8
- 238000010606 normalization Methods 0.000 claims description 5
- 238000009499 grossing Methods 0.000 claims description 3
- 238000004364 calculation method Methods 0.000 claims description 2
- 238000002474 experimental method Methods 0.000 abstract description 4
- 238000013527 convolutional neural network Methods 0.000 abstract 1
- 238000005070 sampling Methods 0.000 description 11
- 238000012545 processing Methods 0.000 description 7
- 102100030148 Integrator complex subunit 8 Human genes 0.000 description 3
- 101710092891 Integrator complex subunit 8 Proteins 0.000 description 3
- 238000004458 analytical method Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 2
- 238000013135 deep learning Methods 0.000 description 2
- 230000008451 emotion Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 239000003086 colorant Substances 0.000 description 1
- 230000000295 complement effect Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000013136 deep learning model Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000013213 extrapolation Methods 0.000 description 1
- 238000004451 qualitative analysis Methods 0.000 description 1
- 238000004445 quantitative analysis Methods 0.000 description 1
- 238000011084 recovery Methods 0.000 description 1
- 230000016776 visual perception Effects 0.000 description 1
Landscapes
- Image Processing (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses an ultra-light low-resolution dim light face enhancement method and system, comprising the following steps: extracting features of an input low-resolution dim light face image I to obtain a plurality of feature images, and fusing the feature images to obtain an enhanced feature image; performing depth feature extraction on the enhanced feature map through a plurality of constructed CNNs connected in series to obtain a depth feature map; fusing the depth feature image and the corresponding up-sampled face image I with low resolution and dim light to obtain a high-quality feature image, and further carrying out sub-pixel convolution to obtain a high-quality clear face image; training of a face enhancement network is completed by minimizing Euclidean distance between a high-quality clear face image and a real clear face gold standard corresponding to the high-quality clear face image; and enhancing the test data through the trained network, and evaluating the test data. The effectiveness of the method is proved by extensive experiments, and the method is applied to the enhancement of low-quality dim light face images.
Description
Technical Field
The invention relates to an image enhancement technology, in particular to an ultra-light weight low-resolution dim light face enhancement method and system.
Background
In general, high-quality clear face images can provide accurate and effective information for subsequent tasks such as visual perception, emotion analysis and the like. However, existing photosensitive devices may exhibit problems of low contrast, color imbalance, blurring, etc. in face images recorded in dim scenes. Thus, the performance of algorithms such as face recognition, face sparse representation, face emotion analysis and the like of downstream tasks can be severely reduced. To address this problem, low light image enhancement methods are widely used in a variety of mobile or edge devices that attempt to restore an image from a dim background to a sharp image with a High Dynamic Range (HDR). From a technical point of view, these methods can be broadly divided into two categories: conventional algorithms and algorithms based on deep learning. Where conventional algorithms empirically reconstruct a dim image to an HDR image by means of artificial priors and statistics, such learning methods often perform well in qualitative analysis and often perform poorly in quantitative analysis. In contrast, the deep learning method has a surprising performance in terms of both quantitative and qualitative performance, and in particular, in terms of texture recovery of images, the missing contours and details can be reconstructed by nonlinear extrapolation.
Although existing deep learning models perform well in reconstructing a high quality face image in a dim environment, the size of these models (most model sizes are greater than 1M) is often inadequate for existing application requirements. In particular, for embedded devices, the storage space needs to consider an operating system, a sensor read-write, and the like, and a common depth model cannot be executed on the devices with limited resources.
Disclosure of Invention
The invention aims to: an object of the present invention is to provide an ultra-lightweight low-resolution dim light face enhancement method that maximally ensures a faster execution speed of a model while focusing on a dim light face enhancement effect.
It is another object of the present invention to provide an ultra lightweight low resolution dim light face enhancement system.
The technical scheme is as follows: the invention discloses an ultra-light low-resolution dim light face enhancement method, which comprises the following steps of:
constructing a face enhancement network model for low-resolution dim light face enhancement, and training by using a face image data set; predicting the collected face image test sample by using the trained face enhancement network model to obtain a high-quality clear face image, and evaluating by using an evaluation index;
wherein the face enhances the network model, include: the device comprises a data enhancement module, a plurality of depth feature extraction modules and a high-quality feature fusion module which are connected in series, wherein the data enhancement module is used for extracting parallel convolution features of an input face image I with low resolution and dim light, and splicing and convolution fusion are carried out on a plurality of obtained feature images to obtain an enhanced feature image F'; performing depth feature extraction on the enhanced feature map F' through a plurality of built depth feature extraction modules connected in series to obtain a depth feature map F d The method comprises the steps of carrying out a first treatment on the surface of the Depth feature map F by high quality feature fusion module d The method comprises the following specific operations of: depth feature map F is obtained through an information fusion module d And the up-sampled low-resolution dim light face image I corresponding to the image is fused, and then a high-quality characteristic diagram F is further obtained through a convolution layer h Sub-pixel convolution is carried out on the high-quality feature map Fh to obtain a high-quality clear face image Y pred 。
Furthermore, before extracting the features of the low-resolution dim light face image I, normalization operation is required to be performed on the data samples in the face image dataset, so that the data range of the data samples is normalized to be between 0 and 1.
Further, the data enhancement module comprises n parallel convolution layers, the feature extraction is carried out on the face image I with low resolution and dim light by using the n parallel convolution layers to obtain n feature images, and then a feature image matrix F formed by the n feature images is obtained through a splicing operation s ＝{F 1 ，F 2 ，…，F n }，F 1 ，F 2 ，…，F n Feature graphs obtained by extracting features of the 1 st to nth convolution layers respectively;
then, the characteristic diagram matrix F is subjected to a convolution layer s Fusing to obtain an enhanced characteristic diagram F' which is:
F’＝CNN 3*3 (F s )
wherein CNN 3*3 Is process F s F' represents the enhanced feature map.
Further, the plurality of serially connected depth feature extraction modules comprise channel separation modules, m parallel convolution layers and convolution fusion modules which are sequentially connected, and the plurality of serially connected depth feature extraction modules are adopted to sequentially extract depth features of the enhanced feature map F' to obtain a depth feature map F d The method specifically comprises the following steps:
the first depth feature extraction module performs depth feature extraction on the enhanced feature map F': firstly, carrying out channel separation on the enhanced feature image F' through a channel separation module, and enhancing the features of face images of different channel domains; then, carrying out 3*3 convolution and activation on the m separated feature images through m parallel convolution layers respectively to realize depth feature extraction; finally, splicing the 3*3 convolved feature images to realize channel fusion, and using 1*1 convolution kernel fusion to obtain a fusion feature image;
the other serially connected depth feature extraction modules sequentially perform channel separation, depth feature extraction and convolution fusion on the output result of the previous depth feature extraction module to finally obtain a depth feature map F d ：
F d ＝CNN 1*1 (CON(CNN 3*3 (SP(F′)))；
Wherein F' is an enhanced feature map, SP is a channel separation operation, CNN 3*3 And CNN 1*1 All are convolution layers, CON represents a channel fusion layer.
Further, after the training of the face enhancement network model is completed, the trained face enhancement network model is further quantized through a dynamic quantization algorithm, and a quantized face enhancement network model is obtained; and predicting the collected face image test sample by using the quantized face enhancement network model.
Further, training the face enhancement network model by adopting the Euclidean distance, specifically:
high quality clear face image Y pred And the real clear face gold standard Y corresponding to the same gt The euclidean distance between them is:
wherein Y is pred Representing a network predicted face image, Y gt Representing a face image Y predicted from a network pred Corresponding real clear face gold standard, dis (Y pred ，Y gt ) Representing a face image Y pred And Y gt Is the Euclidean distance of Y i pred And Y i gt Respectively represent the images belonging to the face image Y pred And Y gt Is a pixel of (1);
representing an objective function of face enhancement by using a loss function, and completing training of a face enhancement network model;
the loss function is calculated by the Euclidean distance as:
L(Y pred ，Y gt )＝Dis(Y pred ，Y gt )；
wherein Dis (Y) pred ，Y gt ) High-quality clear face image Y representing model output pred And the real clear face gold standard Y corresponding to the same gt Euclidean distance between them.
Further, the evaluation indexes comprise peak signal-to-noise ratio PSNR and structural similarity SSIM;
the peak signal-to-noise ratio PSNR calculation method comprises the following steps:
wherein MAX I Representing a maximum value of the image point color;
the structural similarity SSIM calculating method comprises the following steps:
wherein,is a high-quality clear face image Y output by a model pred Average value of>Is with Y pred Corresponding real and clear face gold standard Y gt Average value of>Is Y pred Variance of->Is Y gt Variance of->Represents Y pred Standard deviation of>And Y gt Standard deviation of>Product of c 1 And c 2 Are smoothing parameters.
Based on the same inventive concept, the ultra-light weight low-resolution dim light face enhancement system of the invention comprises:
the model building and training unit builds a face enhancement network model for low-resolution dim light face enhancement and trains by using a face image data set;
the test evaluation module is used for predicting the collected face image test sample by using the trained face enhancement network model to obtain a high-quality clear face image, and evaluating by using an evaluation index;
wherein the face enhances the network model, include: the device comprises a data enhancement module, a plurality of depth feature extraction modules and a high-quality feature fusion module which are connected in series, wherein the data enhancement module is used for extracting parallel convolution features of an input face image I with low resolution and dim light, and splicing and convolution fusion are carried out on a plurality of obtained feature images to obtain an enhanced feature image F'; performing depth feature extraction on the enhanced feature map F' through a plurality of built depth feature extraction modules connected in series to obtain a depth feature map F d The method comprises the steps of carrying out a first treatment on the surface of the Depth feature map F by high quality feature fusion module d The method comprises the following specific operations of: depth feature map F is obtained through an information fusion module d And the up-sampled low-resolution dim light face image I corresponding to the image is fused, and then a high-quality characteristic diagram F is further obtained through a convolution layer h Sub-pixel convolution is carried out on the high-quality feature map Fh to obtain a high-quality clear face image Y pred 。
Based on the same inventive concept, the ultra-light low-resolution dim light face enhancement device of the present invention comprises a memory and a processor, wherein:
a memory for storing a computer program capable of running on the processor;
and the processor is used for executing the steps of the ultra-light weight low-resolution dim light face enhancement method when the computer program is run.
Based on the same inventive concept, a storage medium of the present invention stores a computer program thereon, which when executed by at least one processor implements the steps of an ultra-lightweight low-resolution dim light face enhancement method as described above.
The beneficial effects are that: compared with the prior art, the method of the invention recovers a high-quality face image by fusing the traditional arithmetic and the limited convolution block; specifically, features of an input image are extracted by adopting a plurality of parallel convolution layers, the characteristics of the input image can be captured by the plurality of parallel convolution layers from different view angles, and the feature images are mutually complemented and can guide a downstream convolution layer to extract required image features; then, the invention uses a plurality of serial depth feature extraction modules to deeply extract the features of the human face, and the depth feature extraction modules restore the lost color information of the input image by reconstructing the colors of the image by means of a channel separation technology; finally, the invention uses an up-sampling operation and an information fusion operation to fuse the extracted features and the input images, because the CNN network loses some beneficial input information along with the increase of the convolution layer number, and the extracted image features are complemented with the lost details through up-sampling and information fusion; the reconstruction capability of the model is greatly enhanced by using methods such as channel separation, up-sampling and the like; in order to further simplify the model, the invention uses a dynamic quantization method to implement INT8 quantization on the trained face enhancement network model, the size of the model is reduced by more than 40%, and a large number of experimental results prove that the method can accurately reconstruct a clear face image on limited computing resources.
The CNN network is used for face enhancement, the effectiveness of the method is proved through extensive experiments, and experiments are carried out on a real low-light face data set, so that the conclusion that the method provided by the invention has better performance can be obtained.
Drawings
FIG. 1 is a flow chart of the method of the present invention;
fig. 2 is a diagram of a low quality low light face enhancement network framework of the present invention.
Detailed Description
The invention will now be described in detail with reference to the drawings and specific examples.
The invention relates to an ultralightThe method comprises the steps of firstly constructing a face image data set, then constructing a face enhancement network model for low-resolution dim light face enhancement, and training by using the face image data set; and finally, predicting the collected face image test sample by using the trained face enhancement network model to obtain a high-quality clear face image, and evaluating by using an evaluation index. Wherein the face enhances the network model, include: the device comprises a data enhancement module, a depth feature extraction module and a high-quality feature fusion module, wherein the data enhancement module is used for extracting features of an input face image I with low resolution and dim light to obtain a plurality of feature images and fusing the feature images to obtain an enhanced feature image F'; performing depth feature extraction on the enhanced feature map F' through a plurality of built depth feature extraction modules connected in series to obtain a depth feature map F d The method comprises the steps of carrying out a first treatment on the surface of the High-quality feature fusion module pair depth feature map F d The method comprises the following specific operations of: depth feature map F is processed through an information fusion module d And the up-sampled low-resolution dim light face image I corresponding to the high-quality feature image Fh are fused, then a high-quality feature image Fh is further obtained through a convolution layer, and sub-pixel convolution (namely sub-pixel up-sampling) is carried out on the high-quality feature image Fh to obtain a high-quality clear face image Y pred The method comprises the steps of carrying out a first treatment on the surface of the For high-quality clear face image Y pred And the real clear face gold standard Y corresponding to the same gt And calculating the Euclidean distance.
After the training of the face enhancement network model is completed, in order to further reduce the parameter number and size of the model, a dynamic quantization algorithm can be used to implement the quantization operation of INT8 on the trained face enhancement model, so that the size of the model is effectively reduced.
As shown in fig. 1, the method specifically comprises the following steps:
s1, carrying out feature extraction on an input low-resolution dim light face image I by means of a data enhancement module to obtain a plurality of feature images, and fusing the plurality of feature images to obtain an enhanced feature image F';
the method comprises the following steps: constructing a face image dataset asS＝{(x 1 ，y 1 )，(x 2 ，y 2 )，...，(x L ，y L ) X, where x i Representing low resolution darkened face images I, y i Representation and x i Corresponding real and clear face gold standards, i=1, …, L representing the number of samples in the data set; controlling the data range in the face image dataset to be between 0 and 1 by using a maximum and minimum normalization method; feature extraction is carried out on the low-resolution dim light face image I which is obtained after normalization and is uniform in scale by using a plurality of parallel convolution layers, and then stitching is carried out, so that a feature map matrix F formed by a plurality of feature maps is obtained s ＝{F 1 ，F 2 ，…，F n -the feature map matrix comprises characteristics of different granularity of faces; then, a convolution layer is used for the characteristic diagram matrix F s Processing, and fusing the processing results together to obtain a reinforced characteristic diagram F'; the enhanced feature map F' is:
F′＝CNN(F s ) (1)；
wherein CNN is Process F s F' represents the enhanced feature map.
As shown in fig. 2, in this embodiment, the data enhancement module includes 6 parallel convolution layers, each convolution layer includes a convolution kernel of 3*3 and an activation function RELU, and the feature extraction is performed on the face image I of low-resolution dim light after unifying the scales by using the 6 parallel convolution layers, and then the feature extraction is performed on the face image I to obtain a feature map matrix F composed of 6 feature maps s ＝{F 1 ，F 2 ，…，F 6 }，F 1 ，F 2 ，…，F 6 Feature graphs obtained by extracting the features of the 1 st to 6 th convolution layers respectively; then feature map matrix F s And then the enhanced characteristic diagram F' is obtained through 1*1 convolution kernel processing.
S2, performing depth feature extraction on the enhanced feature map F' by means of a plurality of built depth feature extraction modules connected in series to obtain a depth feature map F d ；
The method comprises the following steps: enhancement of the step S1 by the first depth feature extraction moduleAnd (3) performing depth feature extraction on the feature map F'. Firstly, carrying out channel separation on the enhanced feature image F' through a channel separation module, and enhancing the features of face images of different channel domains; then, 3*3 convolution and activation are respectively carried out on the separated feature images, so that depth feature extraction is realized; finally, splicing the convolved feature images to realize channel fusion, and using a convolution kernel of 1*1 to obtain a fusion feature image; the rest serially connected depth feature extraction modules sequentially perform channel separation, depth feature extraction and convolution fusion on the output result (namely the fusion feature map) of the previous depth feature extraction module to finally obtain a depth feature map F d ：
F d ＝CNN 1*1 (CON(CNN 3*3 (SP(F′)))) (2)；
Wherein F' is an enhanced feature map, SP is a channel separation operation, CNN 3*3 Is a convolution layer with a convolution kernel of 3*3, CNN 1*1 Is a convolution layer with a convolution sum of 1*1, CON represents a channel fusion operation.
As shown in fig. 2, in this embodiment, 3 serially connected depth feature extraction modules are adopted to sequentially perform depth feature extraction on the enhanced feature map F' to obtain a depth feature map F d . Each depth feature extraction module includes 1 channel separation module, 4 parallel convolution layers, a splice module, and one 1*1 convolution layer, each of the 4 parallel convolution layers employing a 3*3 convolution kernel and a ReLU activation function.
S3, a high-quality feature fusion module is used for depth feature map F d Further processing to obtain a high-quality clear face image Y pred ；
The method comprises the following steps: the high-quality feature fusion module comprises an up-sampling module, an information fusion module, a convolution layer and a sub-pixel convolution (namely sub-pixel up-sampling), wherein the up-sampling module is bilinear up-sampling, and the convolution layer comprises a convolution kernel of 1*1, an activation function ReLU and a 3*3 convolution kernel. First, the depth feature map F obtained in step S2 d And the up-sampled low-resolution dim light face image I corresponding to the up-sampled low-resolution dim light face image I is fused in the channel domain dimension through an information fusion module; thereafter, the method comprisesAmplifying a channel domain by using a 1*1 convolution kernel and a 3*3 convolution kernel to obtain a high-quality characteristic diagram Fh; for high quality feature map F h Sub-pixel convolution is carried out to predict a high-quality clear face image Y pred ，Y pred The formula is:
Y pred ＝sub(CNN 1*1 (Up(I)+F d )) (3)；
wherein sub represents sub-pixel convolution, CNN 1*1 Representing a convolution operation, up represents bilinear upsampling.
S4, by minimizing high-quality clear face image Y pred And the real clear face gold standard Y corresponding to the same gt The Euclidean distance between the two faces is used for finishing the training of the face enhancement network;
the method comprises the following steps: the Euclidean distance is used for describing the distance between two face images, and the Euclidean distance is calculated as follows:
wherein Y is pred Representing a network predicted face image, Y gt Representing a face image Y predicted from a network pred Corresponding real clear face gold standard, dis (Y pred ，Y gt ) Representing a face image Y pred And Y gt Is the Euclidean distance, x i And y i Respectively represent the images belonging to the face image Y pred And Y gt The pixel point of (2) has a Euclidean distance of O (N 2 )。
Representing an objective function of face enhancement by using a loss function, and completing training of a face enhancement model;
the loss function is calculated by the Euclidean distance as:
L(Y pred ，Y gt )＝Dis(Y pred ，Y gt ) (5)；
wherein Dis (Y) pred ，Y gt ) And representing the Euclidean distance between the face image output by the model and the gold standard.
As shown in fig. 2, the face enhancement network model constructed according to the embodiment of the present invention includes: the system comprises a data enhancement module, three serially connected depth feature extraction modules and a high-quality feature fusion module, wherein the data enhancement module comprises 6 parallel convolution layers, a splicing module and a 1*1 convolution layer which are sequentially connected, and the 6 parallel convolution layers comprise a 3*3 convolution kernel and an activation function RELU: each depth feature extraction module comprises a channel separation module, 4 parallel convolution layers, a splicing operation and a 1*1 convolution layer which are connected in sequence; each of the 4 parallel convolution layers includes a convolution kernel of 3*3 and an activation function RELU: the high-quality feature fusion module comprises an up-sampling module, an information fusion module, a convolution layer and a sub-pixel convolution layer, wherein the up-sampling module outputs a depth feature map F with a depth feature extraction module d The corresponding low-resolution dim light face image I is up-sampled, and the information fusion module is used for up-sampling the image and the depth feature map F d The fusion is performed and the convolution layer includes a 1*1 convolution kernel, an activation function ReLU, and a 3*3 convolution kernel.
The face enhancement network model firstly uses 6 parallel convolution layers to extract features of an input image, then uses 3 serial depth feature extraction modules to reconstruct the features in depth, then uses an information fusion module to fuse the up-sampled input image and the depth extracted feature images, and finally uses one convolution layer and one sub-pixel convolution to generate a high-quality face image.
After the training of the face enhancement network model is completed, in order to further reduce the parameter number and size of the model, a dynamic quantization algorithm can be used to perform INT8 quantization operation on the trained face enhancement model, so that the size of the model is effectively reduced.
S5, predicting the collected face image test sample by using the trained face enhancement network model to obtain a high-quality clear face image, and evaluating by using an evaluation index;
the method comprises the following steps: and predicting the test sample, enhancing a low-resolution low-light face image, and evaluating the face enhancement result by taking PSNR and SSIM as evaluation indexes in order to evaluate the performance of the face enhancement result.
The PSNR represents the peak signal-to-noise ratio, which is determined by the minimum mean square error MSE.
Wherein Y is i pred And Y i gt Respectively represent high-quality clear face images Y belonging to model output pred Pixel sum and Y of (2) pred Corresponding real and clear face gold standard Y gt Is a pixel of (1); MAX (MAX) I The maximum value representing the color of the image point is typically 255, n represents the number of training set samples, and i represents the current sample.
SSIM represents structural similarity, which compares brightness, contrast, and structural characteristics between images.
c 1 ＝(k 1 L) 2
c 2 ＝(k 2 L) 2
Wherein,is a high-quality clear face image Y output by a model pred Average value of>Is with Y pred Corresponding real and clear face gold standard Y gt Average value of>Is Y pred Variance of->Is Y gt Is a variance of (2); l is the dynamic range of the pixel value, k 1 And k 2 All represent preset hyper-parameters, and the values are as follows: k (k) 1 ＝0.01，k 2 ＝0.03，/>Represents Y pred Standard deviation of>And Y gt Standard deviation of>Product of c 1 And c 2 Are smoothing parameters.
Enhancing the test data to obtain a high-quality clear face image, and evaluating by using the customized index; the effectiveness of the method is proved by extensive experiments, and the method is applied to low-quality dim light faces.
The method evaluates the complement result by using the two indexes, and selects two algorithms which are representative in the face enhancement field: the index results of SwinIR and ELAN are shown in Table 1, and the method (Ours) obtains the best results on the PSNR and SSIM, which shows that the enhancement result of the method is closer to the real result.
TABLE 1 Low light face enhancement index results
The invention discloses an ultra-light low-resolution dim light face enhancement system, which comprises: the model building and training unit is used for building a face enhancement network model for low-resolution dim light face enhancement and training by using a face image data set; the test evaluation module is used for predicting the collected face image test sample by using the trained face enhancement network model to obtain a high-quality clear face image, and evaluating by using an evaluation index.
The model construction and training unit comprises a data processing module, an encoding module and a training module, wherein the data processing module performs normalization processing on an input low-resolution dim light face image I and performs multi-channel feature extraction to obtain a feature map F' with enhanced multi-granularity; a plurality of depth feature extraction modules which are connected in series and constructed by the coding modules perform depth feature extraction on the enhanced feature map F' to obtain a depth feature map F d Depth profile F is then displayed d And the up-sampled low-resolution dim light face image I corresponding to the image is fused to obtain a high-quality characteristic diagram F h The high-quality characteristic image Fh is convolved by a convolution layer and sub-pixels to obtain a high-quality clear face image Y pred The method comprises the steps of carrying out a first treatment on the surface of the Training module by minimizing Y pred And the real clear face gold standard Y corresponding to the same gt The Euclidean distance between the two parts is used for completing the training of the face enhancement network model; and the test evaluation unit enhances the face image of the dim light according to the prediction result of the test sample and evaluates the face enhancement result.
Wherein the face enhances the network model, include: the method comprises the steps of firstly, carrying out feature extraction on an input face image I with low resolution and dim light through the data enhancement module to obtain a plurality of feature images, splicing the feature images, and then fusing the feature images through a 1*1 convolution layer to obtain an enhanced feature image F'; performing depth feature extraction on the enhanced feature map F' through a plurality of built depth feature extraction modules connected in series to obtain a depth feature map F d The method comprises the steps of carrying out a first treatment on the surface of the High-quality feature fusion module pair depth feature map F d The method comprises the following specific operations of: depth feature map F is obtained through an information fusion module d And the up-sampled low-resolution dim light face image I corresponding to the up-sampled low-resolution dim light face image I are fused, and a high-quality characteristic diagram F is further obtained through a convolution kernel of 1*1, an activation function ReLU and a 3*3 convolution kernel h For high quality feature map F h Sub-pixel convolution is carried out to obtain a high-quality clear face imageImage Y pred The method comprises the steps of carrying out a first treatment on the surface of the For high-quality clear face image Y pred And the real clear face gold standard Y corresponding to the same gt And calculating the Euclidean distance.
Based on the same inventive concept, the ultra-light low-resolution dim light face enhancement device of the present invention comprises a memory and a processor, wherein:
a memory for storing a computer program capable of running on the processor;
and the processor is used for executing the steps of the ultra-light weight low-resolution dim light face enhancement method when the computer program is run.
Based on the same inventive concept, a storage medium of the present invention stores a computer program thereon, which when executed by at least one processor implements the steps of the above-described ultra-lightweight low-resolution dim light face enhancement method.
Claims (10)
1. The ultra-light low-resolution dim light face enhancement method is characterized by comprising the following steps of:
constructing a face enhancement network model for low-resolution dim light face enhancement, and training by using a face image data set; predicting the collected face image test sample by using the trained face enhancement network model to obtain a high-quality clear face image, and evaluating by using an evaluation index;
wherein the face enhances the network model, include: the device comprises a data enhancement module, a plurality of depth feature extraction modules and a high-quality feature fusion module which are connected in series, wherein the data enhancement module is used for extracting parallel convolution features of an input face image I with low resolution and dim light, and splicing and convolution fusion are carried out on a plurality of obtained feature images to obtain an enhanced feature image F'; performing depth feature extraction on the enhanced feature map F' through a plurality of built depth feature extraction modules connected in series to obtain a depth feature map F d The method comprises the steps of carrying out a first treatment on the surface of the Depth feature map F by high quality feature fusion module d The method comprises the following specific operations of: by a fusion of informationThe combining module combines the depth characteristic map F d And the up-sampled low-resolution dim light face image I corresponding to the image is fused, and then a high-quality characteristic diagram F is further obtained through a convolution layer h For high quality feature map F h Sub-pixel convolution is carried out to obtain a high-quality clear face image Y pred 。
2. The ultra-lightweight low-resolution darkness face enhancement method according to claim 1, wherein before the feature extraction is performed on the low-resolution darkness face image I, a normalization operation is further performed on the data samples in the face image dataset, so that the data range of the data samples is normalized to be between 0 and 1.
3. The method for enhancing a face with ultra-light weight and low resolution and dim light as claimed in claim 1, wherein the data enhancement module comprises n parallel convolution layers, the feature extraction is performed on the face image I with low resolution and dim light by using the n parallel convolution layers to obtain n feature images, and a feature image matrix F composed of the n feature images is obtained by a stitching operation s ＝{F 1 ,F 2 ,...,F n }，F 1 ,F 2 ,...,F n Feature graphs obtained by extracting features of the 1 st to nth convolution layers respectively;
then, the characteristic diagram matrix F is subjected to a convolution layer s Fusing to obtain an enhanced characteristic diagram F' which is:
F'＝CNN 3*3 (F s )
wherein CNN 3*3 Is process F s F' represents the enhanced feature map.
4. The ultra-light low-resolution dim light face enhancement method according to claim 1, wherein the plurality of serially connected depth feature extraction modules comprise sequentially connected channel separation modules, m parallel convolution layers and convolution fusion modules, and the plurality of serially connected depth feature extraction modules are sequentially used for enhancingExtracting depth features from the feature map F' to obtain a depth feature map F d The method specifically comprises the following steps:
the first depth feature extraction module performs depth feature extraction on the enhanced feature map F': firstly, carrying out channel separation on the enhanced feature image F' through a channel separation module, and enhancing the features of face images of different channel domains; then, carrying out 3*3 convolution and activation on the m separated feature images through m parallel convolution layers respectively to realize depth feature extraction; finally, splicing the 3*3 convolved feature images to realize channel fusion, and using 1*1 convolution kernel fusion to obtain a fusion feature image;
the other serially connected depth feature extraction modules sequentially perform channel separation, depth feature extraction and convolution fusion on the output result of the previous depth feature extraction module to finally obtain a depth feature map F d ：
F d ＝CNN 1*1 (CON(CNN 3*3 (SP(F′)))；
Wherein F' is an enhanced feature map, SP is a channel separation operation, CNN 3*3 And CNN 1*1 All are convolution layers, CON represents a channel fusion layer.
5. The ultra-lightweight low-resolution dim light face enhancement method according to claim 1, wherein after the face enhancement network model is trained, the trained face enhancement network model is further quantized by a dynamic quantization algorithm to obtain a quantized face enhancement network model; and predicting the collected face image test sample by using the quantized face enhancement network model.
6. The ultra-lightweight low-resolution dim light face enhancement method according to claim 1, wherein the face enhancement network model is trained by using the euclidean distance, and specifically comprises:
high quality clear face image Y pred And the real clear face gold standard Y corresponding to the same gt The euclidean distance between them is:
wherein Y is pred Representing a network predicted face image, Y gt Representing a face image Y predicted from a network pred Corresponding real clear face gold standard, dis (Y pred ,Y gt ) Representing a face image Y pred And Y gt Is the Euclidean distance of Y i pr*d And Y i gt Respectively represent the images belonging to the face image Y pr*d And Y gt Is a pixel of (1);
representing an objective function of face enhancement by using a loss function, and completing training of a face enhancement network model;
the loss function is calculated by the Euclidean distance as:
L(Y pred ,Y gt )＝Dis(Y pref ,Y gt )；
wherein Dis (Y) pred ,Y gt ) High-quality clear face image Y representing model output pred And the real clear face gold standard Y corresponding to the same gt Euclidean distance between them.
7. The method for enhancing an ultra-lightweight low-resolution dim light face according to claim 1, wherein the evaluation indexes include peak signal-to-noise ratio PSNR and structural similarity SSIM;
the peak signal-to-noise ratio PSNR calculation method comprises the following steps:
wherein MAX I Representing a maximum value of the image point color;
the structural similarity SSIM calculating method comprises the following steps:
wherein,is a high-quality clear face image Y output by a model pr*d Average value of>Is with Y pr*d Corresponding real and clear face gold standard Y gt Average value of>Is Y pred Variance of->Is Y gt Variance of->Represents Y pred Standard deviation of>And Y gt Standard deviation of>Product of c 1 And c 2 Are smoothing parameters.
8. An ultra-lightweight low-resolution dim light face enhancement system, comprising:
the model building and training unit builds a face enhancement network model for low-resolution dim light face enhancement and trains by using a face image data set;
the test evaluation module is used for predicting the collected face image test sample by using the trained face enhancement network model to obtain a high-quality clear face image, and evaluating by using an evaluation index;
wherein the face enhances the network model, include: the device comprises a data enhancement module, a plurality of depth feature extraction modules and a high-quality feature fusion module which are connected in series, wherein the data enhancement module is used for extracting parallel convolution features of an input face image I with low resolution and dim light, and splicing and convolution fusion are carried out on a plurality of obtained feature images to obtain an enhanced feature image F'; performing depth feature extraction on the enhanced feature map F' through a plurality of built depth feature extraction modules connected in series to obtain a depth feature map F d The method comprises the steps of carrying out a first treatment on the surface of the Depth feature map F by high quality feature fusion module d The method comprises the following specific operations of: depth feature map F is obtained through an information fusion module d And the up-sampled low-resolution dim light face image I corresponding to the image is fused, and then a high-quality characteristic diagram F is further obtained through a convolution layer h For high quality feature map F h Sub-pixel convolution is carried out to obtain a high-quality clear face image Y pred 。
9. An ultra-lightweight low-resolution dim light face enhancement device, comprising a memory and a processor, wherein:
a memory for storing a computer program capable of running on the processor;
a processor for performing the steps of a method of ultra lightweight low resolution darkened-light face enhancement as in any one of claims 1-7, when said computer program is run.
10. A storage medium having stored thereon a computer program which, when executed by at least one processor, implements the steps of a method of ultra lightweight low resolution dim face enhancement according to any one of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311239595.1A CN117173054A (en) | 2023-09-25 | 2023-09-25 | Ultra-light low-resolution dim light face enhancement method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311239595.1A CN117173054A (en) | 2023-09-25 | 2023-09-25 | Ultra-light low-resolution dim light face enhancement method and system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117173054A true CN117173054A (en) | 2023-12-05 |
Family
ID=88935400
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311239595.1A Pending CN117173054A (en) | 2023-09-25 | 2023-09-25 | Ultra-light low-resolution dim light face enhancement method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117173054A (en) |
-
2023
- 2023-09-25 CN CN202311239595.1A patent/CN117173054A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP6980958B1 (en) | Rural area classification garbage identification method based on deep learning | |
| CN112233038B (en) | True image denoising method based on multi-scale fusion and edge enhancement | |
| CN110363716B (en) | High-quality reconstruction method for generating confrontation network composite degraded image based on conditions | |
| EP4105877A1 (en) | Image enhancement method and image enhancement apparatus | |
| CN111507914A (en) | Training method, repairing method, device, equipment and medium of face repairing model | |
| CN114463218B (en) | Video deblurring method based on event data driving | |
| CN114638768B (en) | Image rain removing method, system and equipment based on dynamic association learning network | |
| WO2022156621A1 (en) | Artificial intelligence-based image coloring method and apparatus, electronic device, computer readable storage medium, and computer program product | |
| CN116343052B (en) | Attention and multiscale-based dual-temporal remote sensing image change detection network | |
| CN114881871A (en) | Attention-fused single image rain removing method | |
| CN115272437A (en) | Image depth estimation method and device based on global and local features | |
| CN115359370A (en) | Remote sensing image cloud detection method and device, computer device and storage medium | |
| CN115731597A (en) | Automatic segmentation and restoration management platform and method for mask image of face mask | |
| CN114581789A (en) | Hyperspectral image classification method and system | |
| CN115082306A (en) | Image super-resolution method based on blueprint separable residual error network | |
| Chen et al. | Attention-based broad self-guided network for low-light image enhancement | |
| Liu et al. | WSDS-GAN: A weak-strong dual supervised learning method for underwater image enhancement | |
| CN116523985B (en) | Structure and texture feature guided double-encoder image restoration method | |
| CN115965844B (en) | Multi-focus image fusion method based on visual saliency priori knowledge | |
| Yu et al. | MagConv: Mask-guided convolution for image inpainting | |
| CN115937121A (en) | Non-reference image quality evaluation method and system based on multi-dimensional feature fusion | |
| CN117173054A (en) | Ultra-light low-resolution dim light face enhancement method and system | |
| US20220164934A1 (en) | Image processing method and apparatus, device, video processing method and storage medium | |
| Tran et al. | Encoder–decoder network with guided transmission map: Robustness and applicability | |
| CN115393491A (en) | Ink video generation method and device based on instance segmentation and reference frame |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |