CN115880177A - Full-resolution low-light image enhancement method for aggregating context and enhancing details - Google Patents

Abstract

The invention provides a full-resolution low-illumination image enhancement method for aggregating context and enhancing details, which comprises the following steps: carrying out data preprocessing, including data pairing, data random cutting and data enhancement processing, to obtain a training data set; designing a full-resolution low-illumination image enhancement network for aggregating context and enhancing details, wherein the network consists of a full-resolution detail extraction module, a frequency-space domain context information attention module and a feature aggregation and enhancement module; designing a loss function, and guiding the parameter optimization of the network designed in the step B; and B, training the full-resolution low-illumination image enhancement network for the aggregation context and the enhancement details in the step B by using the training data set obtained in the step A, and converging to Nash balance to obtain a trained full-resolution low-illumination image enhancement model for the aggregation context and the enhancement details. The invention can enhance the low-illumination image and solve the problems of low-illumination image detail loss, color distortion, insufficient brightness and the like.

Description

Full-resolution low-light image enhancement method for aggregating context and enhancing details

technical field

The invention belongs to the technical fields of image processing and computer vision, and in particular relates to a full-resolution low-illuminance image enhancement method for aggregating context and enhancing details.

Background technique

Low-light image enhancement is an important branch of image enhancement. Due to insufficient lighting, non-uniform lighting, backlight and other environmental reasons, as well as scene conditions such as interference during the imaging process of the camera, low-light images show degradation such as low brightness, high noise, and loss of color and detail information, which requires low-light The image enhancement algorithm processes it, and on the basis of retaining useful detail information, restores useful information such as color and removes noise, so as to obtain a normal illumination image that satisfies human visual perception experience or is more suitable for downstream task analysis and processing.

Low-light image enhancement technology has broad application prospects. It improves visibility for nighttime inspections or monitoring. The purpose of setting up cameras in public places is to better monitor and record, and the images captured by surveillance cameras at night are mostly dark and unclear in low light conditions, which cannot provide strong evidence for some events. After being processed by low-light image enhancement technology, the clearer the video, the more powerful the supporting role for the judgment and decision-making of the event. Low-light image enhancement technology can also improve the quality of human visual perception. Ambient lighting in captured images was less than ideal, but could be quickly brought to a satisfactory level by the technology without requiring set reshoots. At the same time, low-light image enhancement techniques can also improve the performance of downstream tasks. Many downstream technologies, such as face recognition and human body key point recognition, have relatively high requirements on the quality of the input image. The ability of the algorithm to correctly identify the position of the face and body depends on whether the input image is clear enough. In the case of dark or blurred input images, it is extremely challenging to identify human faces or body contours, and low-light image enhancement technology can improve image quality, thereby significantly improving the recognition and detection accuracy of the algorithm.

Early methods are mainly based on histogram equalization and Retinex theory. Histogram equalization achieves enhancement by expanding the dynamic range of image pixels. This method can improve the contrast, but due to the lack of local considerations, it is easy to cause excessive images. Exposure and underexposure; Retinex theory believes that the image can be described as the product of the reflection component R and the illumination component I, which requires prior knowledge. Poor prior knowledge will bring about serious color difference and unreal enhancement of noise amplification. In recent years, many deep learning methods have been proposed. Some methods combine Retinex theory with convolutional neural network, and directly classify reflections as enhanced images, resulting in loss of details and color deviation; Considering the characteristics; there are also some methods that independently solve certain aspects of the problems of insufficient illumination, large noise, and loss of color and detail information in low-illumination images, ignoring the correlation between the problems. However, low-light images contain low-level information such as details and noise, and high-level information such as color and scene, and the two types of features are not irrelevant. The processing of scenes and lighting helps restore details, and the restoration of details promotes the restoration of the overall scene.

Existing methods give the same status to low-level information such as details and noise and high-level information such as color and scene for independent processing, while low-light images themselves are a relatively delicate image processing task that requires more attention to detail and then fusion Into the processing of high-level information such as color and scene. Moreover, a connection should also be established between the two types of features, which are jointly enhanced and cannot be processed independently.

Contents of the invention

Aiming at the defects and deficiencies in the prior art, the purpose of the present invention is to provide a full-resolution low-light image enhancement method that aggregates context and enhances details. Performance of illumination image enhancement.

The present invention designs a full-resolution low-illuminance image enhancement method that aggregates context and enhances details. First, a full-resolution detail extraction module is designed to extract detail features, and then a frequency-space context information attention module is designed to extract colors in the frequency domain and space domain. , scene and other context features and use the attention module to learn the importance of features in the frequency domain and space domain, and finally design the feature aggregation and enhancement module to aggregate the detail features and context features and then synergistically enhance them.

It includes: data preprocessing, including data pairing, data random cropping, and data enhancement processing, to obtain a training data set; design a full-resolution low-light image enhancement network that aggregates context and enhances details, and the network is extracted from full-resolution details Module, frequency-space domain context information attention module, feature aggregation and enhancement module; design loss function to guide the parameter optimization of the network designed in step B; use the training data set obtained in step A to train the aggregation context and enhancement details in step B The full-resolution low-illumination image enhancement network converges to Nash equilibrium, and the trained full-resolution low-illumination image enhancement model with aggregated context and enhanced details is obtained; the low-illuminated image to be tested is input into the trained aggregated context and enhanced details. A high-resolution low-illumination image enhancement model that outputs an enhanced normal-illumination image. The invention can enhance the low-illuminance image, and solve the problems of low-illuminance image detail loss, color distortion, insufficient brightness and the like.

The technical scheme that the present invention solves its technical problem adopts is:

A full-resolution low-light image enhancement method that aggregates context and enhances details, characterized by:

Step A, data preprocessing, including data pairing, data random cutting, data enhancement processing, to obtain a training data set;

Step B. Design a full-resolution low-light image enhancement network that aggregates context and enhances details, including: a full-resolution detail extraction module, a frequency-spatial context information attention module, and a feature aggregation and enhancement module;

Step C, designing a loss function, which is used to guide the parameter optimization of the network designed in step B;

Step D. Use the training data set obtained in step A to train the full-resolution low-light image enhancement network for the aggregated context and enhanced details in step B, converge to Nash equilibrium, and obtain the trained aggregated context and enhanced details for the full-resolution low-light image Illuminance image enhancement model;

Step E: Input the low-illumination image to be tested into the trained full-resolution low-illumination image enhancement model that aggregates context and enhances details, and outputs an enhanced normal-illuminance image.

Further, the specific implementation steps of step A are as follows:

Step A1, pairing the low-illumination image and the corresponding label image;

Step A2. Randomly crop each low-illumination image with a size of h×w×3 into an image with a size of p×p×3, and use the same random cropping method for the corresponding label image, where h, w is the height and width of the low-light image and the label image, and p is the height and width of the cropped image;

Step A3. Randomly use one of the following 8 enhancement methods to perform data enhancement on the training paired image: keep the original image, flip vertically, rotate 90 degrees, rotate vertically after 90 degrees, rotate 180 degrees, rotate vertically after 180 degrees , rotate 270 degrees, rotate vertically after 270 degrees.

Further, the specific implementation steps of step B are as follows:

Step B1, build a full-resolution detail extraction module, which is composed of a shallow feature extraction submodule, a CBAM-based attention submodule, and a frequency domain transformation submodule, and use the designed network to extract detail features;

Step B2. Design the frequency-space context information attention module, which is composed of a multi-scale feature extraction sub-module and a frequency-space feature fusion sub-module, and use the designed network to extract context features;

Step B3, design feature aggregation and enhancement module, which is composed of feature aggregation convolution block and collaborative enhancement sub-module, aggregate the detailed features extracted in step B1 and the context features extracted in step B2, and jointly enhance the two types of features;

Step B4. Design a full-resolution low-light image enhancement network that aggregates context and enhances details, including a full-resolution detail extraction module, a frequency-spatial context information attention module, and a feature aggregation and enhancement module.

Further, the specific implementation steps of step B1 are as follows:

Step B11. Design the sub-module of shallow feature extraction. The input is the low-light image I. After 3×3 convolution to obtain the initial feature map F _ori , enter three branches. The first branch contains a 3×3 convolution, The second branch contains 2 serial 3×3 convolutions, and the third branch contains 3 serial 3×3 convolutions, and the processing results F _B1 , F _B2 , and F _B3 of the three branches are along the channel dimension After splicing, after a 3×3 convolution, the feature map F _low output by the shallow feature extraction sub-module is obtained; the specific formula is expressed as follows:

F _ori = Conv3(I)

F _B1 ＝Conv3(F _ori )

F _B2 ＝Conv3(Conv3(F _ori ))

F _B3 ＝Conv3(Conv3(Conv3(F _ori )))

F _low ＝Conv3(Concat(F _B1 ,F _B2 ,F _B3 ))

Among them, Conv3 is a 3×3 convolution, and Concat is a splicing operation along the channel dimension;

Step B12, build a CBAM-based attention sub-module, which is composed of serial channel-dimension attention Att _c and space-dimension attention Att _s , the input feature map is the feature map _Flow obtained in step B11, and CBAM-based The feature map output by the attention sub-module is F _spa ; the specific formula is expressed as follows:

F _spa ＝Att _s (Att _c (F _low ))

Among them, Att _c is the attention of the channel dimension, and Att _s is the attention of the spatial dimension;

Step B13, design the frequency domain transformation sub-module, the input feature map is the feature map F _spa obtained in step B12, use the Fourier transform function to convert the space domain into the frequency domain, and then go through 3×3 convolution, normalization layer, ReLU After activating the function, use the inverse Fourier transform function to convert the frequency domain into the space domain to obtain the output feature map F _fre ; the specific formula is expressed as follows:

F _fre = idft(ReLU(BN(Conv3(dft(F _spa )))))

Among them, dft is the Fourier transform, idft is the inverse Fourier transform, ReLU is the ReLU activation function, BN is the batch normalization layer, and Conv3 is a 3×3 convolution;

Step B14, build a full-resolution detail extraction module, which is composed of a shallow feature extraction submodule, a CBAM-based attention submodule and a frequency domain transformation submodule; assuming that the input is the low-illumination image I processed in step A, sequentially pass through The feature maps F _low , F _spa , and F _fre are obtained after processing by the shallow feature extraction submodule, the attention submodule, and the frequency domain transformation submodule.

Further, the specific implementation steps of step B2 are as follows:

H，W，C分别为特征F的高度、宽度和通道数，经过一个核大小为2×2、步长为2的平均池化层后，依次经过1×1卷积、ReLU激活函数、1×1卷积、ReLU激活函数进行降维，得到中间特征图

然后分成两个分支，上分支经过1×1卷积继续降维后，通过上采样层得到上分支的输出/>

a是降维后的通道数；另一个分支经过一个核大小为2×2、步长为2的平均池化层后，依次经过1×1卷积、ReLU激活函数、1×1卷积、ReLU激活函数进行降维，得到中间特征图/>

中间特征图F₁₂₁再依次经过上采样层、1×1卷积、ReLU激活函数、上采样层，得到下分支的输出/>

将F₁₁和F₁₂相加后，与F在通道维度上拼接，经过SE模块后再通过1×1卷积调整通道，得到多尺度特征提取子模块输出的特征图

具体公式表示如下：Step B21, design multi-scale feature extraction sub-module, input feature map is marked as F,

H, W, and C are the height, width, and number of channels of feature F respectively. After an average pooling layer with a kernel size of 2×2 and a step size of 2, it goes through 1×1 convolution, ReLU activation function, 1 ×1 convolution and ReLU activation function for dimensionality reduction to obtain intermediate feature maps

Then it is divided into two branches. After the upper branch continues to reduce the dimension through 1×1 convolution, the output of the upper branch is obtained through the upsampling layer>>

a is the number of channels after dimensionality reduction; the other branch passes through an average pooling layer with a kernel size of 2×2 and a step size of 2, followed by 1×1 convolution, ReLU activation function, 1×1 convolution, The ReLU activation function performs dimensionality reduction to obtain the intermediate feature map/>

The intermediate feature map F ₁₂₁ then passes through the upsampling layer, 1×1 convolution, ReLU activation function, and upsampling layer in sequence to obtain the output of the lower branch/>

After adding F ₁₁ and F ₁₂ , they are concatenated with F in the channel dimension, and after passing through the SE module, the channel is adjusted by 1×1 convolution to obtain the feature map output by the multi-scale feature extraction sub-module

The specific formula is expressed as follows:

F ₁ =ReLU(Conv1(ReLU(Conv1(Avgpooling(F)))))

F ₁₁ ＝Upsampling(Conv1(F ₁ ))

F ₁₂₁ =ReLU(Conv1(ReLU(Conv1(Avgpooling(F ₁ )))))

F ₁₂ ＝Upsampling(ReLU(Conv1(Upsampling(F ₁₂₁ ))))

F _m =Conv1(SE(Concat(F ₁₁ +F ₁₂ ,F)))

Among them, ReLU is an activation function, Conv1 is a 1×1 convolution, SE(·) is an SE module, Avgpooling is an average pooling layer with a kernel size of 2×2 and a step size of 2, and Upsampling is twice the nearest neighbor upsampling Layer, Concat is a splicing operation along the channel dimension;

Step B22, designing a frequency-space domain feature fusion sub-module, which is composed of serial connections of channel attention and spatial attention;

Step B23. Design the frequency-space context information attention module, which is composed of three multi-scale feature extraction sub-modules and frequency-space feature fusion sub-modules; the inputs of the three multi-scale feature extraction sub-modules are the three feature maps obtained in step B1 respectively F _low , F _spa , F _fre are respectively processed by the multi-scale feature extraction sub-module designed in step B21 to obtain feature maps F _{low_m} , F _{spa_m} , F _{fre_m} with context information, and then pass through the frequency-space domain designed in step B22 The feature fusion sub-module obtains the feature map F _f output by the frequency-space context information attention module.

Further, the specific implementation steps of step B22 are as follows:

三个特征图分别经过空间维度的全局平均池化后得到三个尺度为1×1×C的向量，然后将三个向量沿通道维度拼接，得到中间特征图/>

将F_c依次经过1×1卷积、ReLU激活函数、1×1卷积、ReLU激活函数、1×1卷积进行降维和升维，再通过Sigmoid激活函数得到通道维度上的权重/>

将F_W1沿通道维度分解为三个尺度为1×1×C的向量F_W10、F_W11、F_W12，分别与频空域特征融合子模块的输入特征图F_{low_m}、F_{spa_m}、F_{fre_m}相乘，得到通道注意力的输出特征图/>

具体公式表示如下：Step B221, design channel attention, the input is the feature map obtained in step B23

The three feature maps are respectively subjected to the global average pooling of the spatial dimension to obtain three vectors with a scale of 1×1×C, and then the three vectors are spliced along the channel dimension to obtain the intermediate feature map/>

_Fc undergoes 1×1 convolution, ReLU activation function, 1×1 convolution, ReLU activation function, 1×1 convolution to reduce and increase dimensionality, and then obtain the weight on the channel dimension through the Sigmoid activation function/>

Decompose F _W1 along the channel dimension into three vectors F _W10 , F _W11 , F _W12 with a scale of 1×1×C, and multiply them with the input feature maps F _{low_m} , F _{spa_m} , F _{fre_m} of the frequency-space domain feature fusion sub-module respectively , get the output feature map of the channel attention />

The specific formula is expressed as follows:

F _c ＝Concat(Avgpooling _s (F _{low_m} ),Avgpooling _s (F _{spa_m} ),Avgpooling _s (F _{fre_m} ))

FW ₁ ＝Sigmoid(Conv1(ReLU(Conv1(ReLU(Conv1(F _c ))))))

F _{low_c} = F _W10 × F _{low_m}

F _{spa_c} = F _W11 × F _{spa_m}

F _{fre_c} = F _W12 × F _{fre_m}

Among them, Concat is a splicing operation along the channel dimension, Avgpooling _s is the global average pooling of the spatial dimension, ReLU is the activation function, Conv1 is 1×1 convolution, and Sigmoid is the Sigmoid activation function;

将F_s依次经过核大小为2×2、步长为2的平均池化层、ReLU激活函数、上采样层后，通过Sigmoid激活函数得到空间维度上的权重/>

将F_W2分解为三个尺度为H×W×1的特征图F_W20、F_W21、F_W22，分别与空间注意力的输入特征图F_{low_c}、F_{spa_c}、F_{fre_c}相乘，得到空间注意力的输出特征图

具体公式表示如下：Step B222, design spatial attention, the input is the three feature maps F _{low_c} , F _{spa_c} , and F _{fre_c} obtained in step B221, and the three feature maps are respectively averaged in the channel dimension to obtain three scales of H×W×1 feature map, and then splicing the three feature maps along the channel dimension to obtain the intermediate feature map

After passing F _s through the average pooling layer with a kernel size of 2×2 and a step size of 2, the ReLU activation function, and the upsampling layer, the weight in the spatial dimension is obtained through the Sigmoid activation function>

Decompose F _W2 into three feature maps F _W20 , F _W21 , and F _W22 with a scale of H×W×1, and multiply them with the input feature maps F _{low_c} , F _{spa_c} , and F _{fre_c of} spatial attention to obtain spatial attention The output feature map of

The specific formula is expressed as follows:

F _s ＝Concat(Avgpooling _c (F _{low_c} ),Avgpooling _c (F _{spa_c} ),Avgpooling _c (F _{fre_c} ))

F _W2 ＝Sigmoid(Upsampling(ReLU(Avgpooling(F _s ))))

F _{low_s} = F _W20 × F _{low_c}

F _{spa_s} = F _W21 × F _{spa_c}

F _{fre_s} = F _W22 × F _{fre_c}

Among them, Concat is a splicing operation along the channel dimension, Avgpooling _c is the average pooling of the channel dimension, ReLU is the activation function, Sigmoid is the Sigmoid activation function, Avgpooling is the average pooling layer with a kernel size of 2×2 and a step size of 2, Upsampling is twice the nearest neighbor upsampling layer;

Step B223. Design the frequency-space domain feature fusion sub-module. The input feature maps are the feature maps F _{low_m} , F _{spa_m} , and F _{fre_m} obtained in step B23. The three feature maps first pass through the channel attention in step B221 to obtain the feature maps F _{low_c} , F _{spa_c} , F _{fre_c} , and then through the spatial attention in step B222, feature maps F _{low_s} , F _{spa_s} , F _{fre_s} are obtained, and the final output F _f is obtained after adding the three feature maps; the specific formula is as follows:

F _f =F _{low_s} +F _{spa_s} +F _{fre_s} .

Further, the specific implementation steps of step B3 are as follows:

Step B31, design a feature aggregation convolution block to realize the fusion of detail information and context information; the input feature map is the feature map F _fre obtained in step B1 and the feature map F _f obtained in step B2, and the two are spliced along the channel dimension and then passed through a 3×3 convolution to obtain the output feature map F _conv ; the specific formula is expressed as follows:

F _conv ＝Conv3(Concat(F _fre ,F _f ))

Among them, Conv3 is a 3×3 convolution, and Concat is a splicing operation along the channel dimension;

Step B32. Design a synergistic enhancement sub-module to synergistically enhance the fusion information of detail information and context information; the input feature map is the F _conv obtained in step B31, and the F _conv is sequentially passed through 1×1 convolution, ReLU6 activation function, and Dropout random After the deactivation layer, 1×1 convolution, and Dropout random deactivation layer, it is added to F _conv to obtain the intermediate feature map F _mid , and then through the LeakyReLU activation function, it is spliced with F _conv along the channel dimension and then passed through a 3×3 convolution product to obtain the output feature map F _co ; the specific formula is expressed as follows:

F _mid = Dropout(Conv1(Dropout(ReLU6(Conv1(F _conv )))))+F _conv

F _co ＝Conv3(Concat(LeakyReLU(F _mid ),F _conv ))

Among them, Conv1 is a 1×1 convolution, Conv3 is a 3×3 convolution, Concat is a splicing operation along the channel dimension, Dropout is a random inactivation layer, ReLU6 is a ReLU6 activation function, and LeakyReLU is a LeakyReLU activation function;

Step B33, design feature aggregation and enhancement module, which is composed of feature aggregation convolution block and collaborative enhancement sub-module, the input feature map is the feature map F _fre obtained in step B1 and the feature map F _f obtained in step B2, after feature aggregation convolution After blocks, the feature map F _conv is obtained, and then the feature map F _co is obtained after the collaborative enhancement sub-module.

Further, the specific implementation of step B4 is as follows:

Step B4. Design a full-resolution low-light image enhancement network that aggregates context and enhances details. It is composed of a full-resolution detail extraction module, a frequency-spatial context information attention module, and a feature aggregation and enhancement module; input a low-light image I, and pass After the full-resolution detail extraction module in step B1, three feature maps F _low , F _spa , and F _fre are obtained, and the feature map F _f is obtained after the frequency-spatial context information attention module, and then the feature map is obtained through the feature aggregation and enhancement module F _co , then F _co is spliced with the feature map F _low in step B1 along the channel dimension, and after 3×3 convolution, the final enhanced image I _out is obtained; the specific formula is expressed as follows:

I _out ＝Conv3(Concat(F _co ,F _low ))

Among them, Conv3 is a 3×3 convolution, and Concat is a splicing operation along the channel dimension.

Further, the specific implementation of step C is as follows:

Step C, design loss function, which is composed of L2 loss and VGG perception loss, the total target loss function of the network is as follows:

l＝ω ₁ ||I _out -G|| ² +ω ₂ ||Φ(I _out )-Φ(G)|| ₁

Among them, Φ( ) represents the operation of extracting the features of the Conv4-1 layer using the VGG-16 classification model pre-trained on the ImageNet dataset; I _out represents the enhanced image of the low-light image I, and G represents the label corresponding to the low-light image I Image, ||.|| ₁ represents L1 loss, ||.|| ² represents L2 loss, and ω ₁ and ω ₂ are weights.

Further, the specific implementation steps of step D are as follows:

Step D1, randomly divide the training data set obtained in step A into several batches, each batch contains N pairs of images;

Step D2, input a low-illumination image I, obtain an enhanced image I _out after the full-resolution low-illumination image enhancement network that aggregates context and enhances details in step B, and calculates the loss l using the formula in step C;

Step D3, using the backpropagation method to calculate the gradient of the parameters in the network according to the loss, and using the Adam optimization method to update the network parameters;

Step D4, repeat steps D1 to D3 in batches until the target loss function of the network converges to Nash equilibrium, save the network parameters, and obtain a full-resolution low-light image enhancement model that aggregates context and enhances details.

Compared with the prior art, the present invention and its preferred solution extract detailed features in full resolution, extract contextual features in frequency domain and spatial domain, and aggregate the two types of features together to enhance them together, which can better extract Two types of information, and the relationship between the two types of features is learned during the enhancement process. A full-resolution low-light image enhancement network that aggregates context and enhances details is designed, and the full-resolution detail extraction module, the frequency-spatial context information attention module, the feature aggregation and enhancement module to extract detail features, and the frequency-spatial context features are respectively set. 1. Aggregating and synergistically enhancing the two features, unlike other methods that independently solve the problems existing in low-light images, the present invention can better extract detail features and context features and achieve synergistic enhancement.

Description of drawings

Below in conjunction with accompanying drawing and specific embodiment the present invention is described in further detail:

Fig. 1 is a flow chart of the implementation of the method of the embodiment of the present invention.

Fig. 2 is a structural diagram of a full-resolution low-light image enhancement network that aggregates context and enhances details in an embodiment of the present invention.

Fig. 3 is a structural diagram of a multi-scale feature extraction sub-module in an embodiment of the present invention.

Fig. 4 is a structural diagram of an intermediate frequency spatial domain feature fusion submodule according to an embodiment of the present invention.

Fig. 5 is a structural diagram of the cooperative enhancement sub-module in the embodiment of the present invention.

Detailed ways

In order to make the features and advantages of this patent more obvious and easy to understand, the following special examples are described in detail as follows:

It should be pointed out that the following detailed description is exemplary and intended to provide further explanation to the present application. Unless otherwise specified, all technical and scientific terms used in this specification have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

It should be noted that the terminology used here is only for describing specific implementations, and is not intended to limit the exemplary implementations according to the present application. As used herein, unless the context clearly dictates otherwise, the singular is intended to include the plural, and it should also be understood that when the terms "comprising" and/or "comprising" are used in this specification, they mean There are features, steps, operations, means, components and/or combinations thereof.

Below in conjunction with accompanying drawing, present embodiment scheme is further described in detail:

The present invention provides a full-resolution low-light image enhancement method that aggregates context and enhances details, as shown in Figures 1-5, comprising the following steps:

Step A, data preprocessing, including data pairing, data random cutting, data enhancement processing, to obtain a training data set;

Step B. Design a full-resolution low-light image enhancement network that aggregates context and enhances details. The network consists of a full-resolution detail extraction module, a frequency-spatial context information attention module, and a feature aggregation and enhancement module;

Step C, design a loss function, and guide the parameter optimization of the network designed in step B;

Step D. Use the training data set obtained in step A to train the full-resolution low-light image enhancement network for the aggregated context and enhanced details in step B, converge to Nash equilibrium, and obtain the trained aggregated context and enhanced details for the full-resolution low-light image Illuminance image enhancement model;

Step E: Input the low-illumination image to be tested into the trained full-resolution low-illumination image enhancement model that aggregates context and enhances details, and outputs an enhanced normal-illuminance image.

Further, step A includes the following steps:

Step A1, pairing the low-illumination image and the corresponding label image;

Step A2. Randomly crop each low-illumination image with a size of h×w×3 into an image with a size of p×p×3, and use the same random cropping method for the corresponding label image, where h, w is the height and width of the low-light image and the label image, and p is the height and width of the cropped image;

Step A3. Randomly use one of the following 8 enhancement methods to perform data enhancement on the training paired image: keep the original image, flip vertically, rotate 90 degrees, rotate vertically after 90 degrees, rotate 180 degrees, rotate vertically after 180 degrees , rotate 270 degrees, rotate vertically after 270 degrees.

Further, step B includes the following steps:

Step B1, build a full-resolution detail extraction module, which is composed of a shallow feature extraction submodule, a CBAM-based attention submodule, and a frequency domain transformation submodule, and use the designed network to extract detail features;

Step B2. Design the frequency-space context information attention module, which is composed of a multi-scale feature extraction sub-module and a frequency-space feature fusion sub-module, and use the designed network to extract context features;

Step B3, design feature aggregation and enhancement module, which is composed of feature aggregation convolution block and collaborative enhancement sub-module, aggregate the detailed features extracted in step B1 and the context features extracted in step B2, and jointly enhance the two types of features;

Step B4. Design a full-resolution low-light image enhancement network that aggregates context and enhances details, including a full-resolution detail extraction module, a frequency-spatial context information attention module, and a feature aggregation and enhancement module.

Further, step B1 includes the following steps:

Step B11. Design the sub-module of shallow feature extraction. The input is the low-light image I. After 3×3 convolution to obtain the initial feature map F _ori , enter three branches. The first branch contains a 3×3 convolution, The second branch contains 2 serial 3×3 convolutions, and the third branch contains 3 serial 3×3 convolutions, and the processing results F _B1 , F _B2 , and F _B3 of the three branches are along the channel dimension After splicing, after a 3×3 convolution, the feature map F _low output by the shallow feature extraction sub-module is obtained. The specific formula is expressed as follows:

F _ori = Conv3(I)

F _B1 ＝Conv3(F _ori )

F _B2 ＝Conv3(Conv3(F _ori ))

F _B3 ＝Conv3(Conv3(Conv3(F _ori )))

F _low ＝Conv3(Concat(F _B1 ,F _B2 ,F _B3 ))

Among them, Conv3 is a 3×3 convolution, and Concat is a splicing operation along the channel dimension;

Step B12. Build a CBAM-based attention sub-module. This module is composed of serial attention Att _c in the channel dimension and attention Att _s in the space _dimension . The feature map output by the attention sub-module of CBAM is F _spa . The specific formula is expressed as follows:

F _spa ＝Att _s (Att _c (F _low ))

Among them, Att _c is the attention of the channel dimension, and Att _s is the attention of the spatial dimension.

Step B13, design the frequency domain transformation sub-module, the input feature map is the feature map F _spa obtained in step B12, use the Fourier transform function to convert the space domain into the frequency domain, and then go through 3×3 convolution, normalization layer, ReLU After activating the function, use the inverse Fourier transform function to convert the frequency domain into the space domain to obtain the output feature map F _fre . The specific formula is expressed as follows:

F _fre = idft(ReLU(BN(Conv3(dft(F _spa )))))

Among them, dft is the Fourier transform, idft is the inverse Fourier transform, ReLU is the ReLU activation function, BN is the batch normalization layer, and Conv3 is a 3×3 convolution.

Step B14, building a full-resolution detail extraction module, which is composed of a shallow feature extraction submodule, a CBAM-based attention submodule and a frequency domain transformation submodule. Assume that the input is the low-illumination image I processed in step A, and the feature maps F _low , F _spa , and F _fre are obtained after being processed by the shallow feature extraction sub-module, the attention sub-module, and the frequency-domain transformation sub-module in sequence.

Further, step B2 includes the following steps:

H，W，C分别为特征F的高度、宽度和通道数)经过一个核大小为2×2、步长为2的平均池化层后，依次经过1×1卷积、ReLU激活函数、1×1卷积、ReLU激活函数进行降维，得到中间特征图

然后分成两个分支，上分支经过1×1卷积继续降维后，通过上采样层得到上分支的输出/>

a是降维后的通道数。另一个分支经过一个核大小为2×2、步长为2的平均池化层后，依次经过1×1卷积、ReLU激活函数、1×1卷积、ReLU激活函数进行降维，得到中间特征图/>

中间特征图F₁₂₁再依次经过上采样层、1×1卷积、ReLU激活函数、上采样层，得到下分支的输出/>

将F₁₁和F₁₂相加后，与F在通道维度上拼接，经过SE模块后再通过1×1卷积调整通道，得到多尺度特征提取子模块输出的特征图

具体公式表示如下：Step B21, designing a multi-scale feature extraction sub-module. The input feature map of this module is denoted as F,

H, W, and C are the height, width, and number of channels of the feature F respectively) After passing through an average pooling layer with a kernel size of 2×2 and a step size of 2, it goes through 1×1 convolution, ReLU activation function, 1 ×1 convolution and ReLU activation function for dimensionality reduction to obtain intermediate feature maps

Then it is divided into two branches. After the upper branch continues to reduce the dimension through 1×1 convolution, the output of the upper branch is obtained through the upsampling layer>>

a is the number of channels after dimensionality reduction. After the other branch passes through an average pooling layer with a kernel size of 2×2 and a step size of 2, it undergoes 1×1 convolution, ReLU activation function, 1×1 convolution, and ReLU activation function for dimensionality reduction to obtain the middle Feature map />

The intermediate feature map F ₁₂₁ then passes through the upsampling layer, 1×1 convolution, ReLU activation function, and upsampling layer in sequence to obtain the output of the lower branch/>

After adding F ₁₁ and F ₁₂ , they are concatenated with F in the channel dimension, and after passing through the SE module, the channel is adjusted by 1×1 convolution to obtain the feature map output by the multi-scale feature extraction sub-module

The specific formula is expressed as follows:

F ₁ =ReLU(Conv1(ReLU(Conv1(Avgpooling(F)))))

F ₁₁ ＝Upsampling(Conv1(F ₁ ))

F ₁₂₁ =ReLU(Conv1(ReLU(Conv1(Avgpooling(F ₁ )))))

F ₁₂ ＝Upsampling(EeLU(Conv1(Upsampling(F ₁₂₁ ))))

F _m =Conv1(SE(Concat(F ₁₁ +F ₁₂ ,F)))

Among them, ReLU is an activation function, Conv1 is a 1×1 convolution, SE(·) is an SE module, Avgpooling is an average pooling layer with a kernel size of 2×2 and a step size of 2, and Upsampling is twice the nearest neighbor upsampling Layer, Concat is a splicing operation along the channel dimension;

Step B22, designing a frequency-space domain feature fusion sub-module, which is composed of serial connections of channel attention and spatial attention;

Step B23, designing the frequency-space context information attention module, which is composed of three multi-scale feature extraction sub-modules and frequency-space feature fusion sub-module. The inputs of the three multi-scale feature extraction sub-modules are the three feature maps F _low , F _spa , and F _fre obtained in step B1 respectively. Feature maps F _{low_m} , F _{spa_m} , F _{fre_m} , and then through the frequency-space domain feature fusion sub-module designed in step B22, the feature map F _f output by the frequency-space domain context information attention module is obtained.

Further, step B22 includes the following steps:

三个特征图分别经过空间维度的全局平均池化后得到三个尺度为1×1×C的向量，然后将三个向量沿通道维度拼接，得到中间特征图/>

将F_c依次经过1×1卷积、ReLU激活函数、1×1卷积、ReLU激活函数、1×1卷积进行降维和升维，再通过Sigmoid激活函数得到通道维度上的权重/>

将F_W1沿通道维度分解为三个尺度为1×1×C的向量F_W10、F_W11、F_W12，分别与频空域特征融合子模块的输入特征图F_{low_m}、F_{spa_m}、F_{fre_m}相乘，得到通道注意力的输出特征图/>

具体公式表示如下：Step B221, design channel attention, the input is the feature map obtained in step B23

The three feature maps are respectively subjected to the global average pooling of the spatial dimension to obtain three vectors with a scale of 1×1×C, and then the three vectors are spliced along the channel dimension to obtain the intermediate feature map/>

_Fc undergoes 1×1 convolution, ReLU activation function, 1×1 convolution, ReLU activation function, 1×1 convolution to reduce and increase dimensionality, and then obtain the weight on the channel dimension through the Sigmoid activation function/>

Decompose F _W1 along the channel dimension into three vectors F _W10 , F _W11 , F _W12 with a scale of 1×1×C, and multiply them with the input feature maps F _{low_m} , F _{spa_m} , F _{fre_m} of the frequency-space domain feature fusion sub-module respectively , get the output feature map of the channel attention />

The specific formula is expressed as follows:

F _c ＝Concat(Avgpooling _s (F _{low_m} ),Avgpooling _s (F _{spa_m} ),Avgpooling _s (F _{fre_m} ))

F _W1 ＝Sigmoid(Conv1(ReLU(Conv1(ReLU(Conv1(F _c ))))))

F _{low_c} = F _W10 × F _{low_m}

F _{spa_c} = F _W11 × F _{spa_m}

F _{fre_c} = F _W12 × F _{fre_m}

Among them, Concat is a splicing operation along the channel dimension, Avgpooling _s is the global average pooling of the spatial dimension, ReLU is the activation function, Conv1 is 1×1 convolution, and Sigmoid is the Sigmoid activation function;

将F_s依次经过核大小为2×2、步长为2的平均池化层、ReLU激活函数、上采样层后，通过Sigmoid激活函数得到空间维度上的权重/>

将F_W2分解为三个尺度为H×W×1的特征图F_W20、F_W21、F_W22，分别与空间注意力的输入特征图F_{low_c}、F_{spa_c}、F_{fre_c}相乘，得到空间注意力的输出特征图

具体公式表示如下：Step B222, design spatial attention, the input is the three feature maps F _{low_c} , F _{spa_c} , and F _{fre_c} obtained in step B221, and the three feature maps are respectively averaged in the channel dimension to obtain three scales of H×W×1 feature map, and then splicing the three feature maps along the channel dimension to obtain the intermediate feature map

After passing F _s through the average pooling layer with a kernel size of 2×2 and a step size of 2, the ReLU activation function, and the upsampling layer, the weight in the spatial dimension is obtained through the Sigmoid activation function>

Decompose F _W2 into three feature maps F _W20 , F _W21 , and F _W22 with a scale of H×W×1, and multiply them with the input feature maps F _{low_c} , F _{spa_c} , and F _{fre_c of} spatial attention to obtain spatial attention The output feature map of

The specific formula is expressed as follows:

F _s ＝Concat(Avgpooling _c (F _{low_} c),Avgpooling _c (F _{spa_c} ),Avgpooling _c (F _{fre_c} ))

F _W2 ＝Sigmoid(Upsampling(ReLU(Avgpooling(F _s ))))

F _{low_s} = F _W20 × F _{low_c}

F _{spa_s} = F _W21 × F _{spa_c}

F _{fre_s} = F _W22 × F _{fre_c}

Among them, Concat is a splicing operation along the channel dimension, Avgpooling _c is the average pooling of the channel dimension, ReLU is the activation function, Sigmoid is the Sigmoid activation function, Avgpooling is the average pooling layer with a kernel size of 2×2 and a step size of 2, Upsampling is twice the nearest neighbor upsampling layer;

Step B223. Design the frequency-space domain feature fusion sub-module. The input feature maps are the feature maps F _{low_m} , F _{spa_m} , and F _{fre_m} obtained in step B23. The three feature maps first pass through the channel attention in step B221 to obtain the feature maps F _{low_c} , F _{spa_c} , F _{fre_c} , and then through the spatial attention in step B222, feature maps F _{low_s} , F _{spa_s} , F _{fre_s} are obtained, and the final output F _f is obtained after adding the three feature maps. The specific formula is expressed as follows:

F _f =F _{low_s} +F _{spa_s} +F _{fre_s}

Further, step B3 is implemented as follows:

Step B31, designing a feature aggregation convolution block to realize the fusion of detail information and context information. The input feature map is the feature map F _fre obtained in step B1 and the feature map F _f obtained in step B2. The two are concatenated along the channel dimension and then undergo a 3×3 convolution to obtain the output feature map F _conv . The specific formula is expressed as follows:

F _conv ＝Conv3(Concat(F _fre ,F _f ))

Among them, Conv3 is a 3×3 convolution, and Concat is a splicing operation along the channel dimension;

Step B32, designing a synergistic enhancement sub-module to synergistically enhance the fusion information of the detail information and the context information. The input feature map is the F _conv obtained in step B31, and the F _conv is sequentially passed through 1×1 convolution, ReLU6 activation function, Dropout random inactivation layer, 1×1 convolution, Dropout random inactivation layer, and then compared with F _conv plus, get the intermediate feature map F _mid , and then use the LeakyReLU activation function to concatenate with F _conv along the channel dimension and then go through a 3×3 convolution to get the output feature map F _co . The specific formula is expressed as follows:

F _mid = Dropout(Conv1(Dropout(ReLU6(Conv1(F _conv )))))+F _conv

F _co ＝Conv3(Concat(LeakyReLU(F _mid ),F _conv ))

Among them, Conv1 is a 1×1 convolution, Conv3 is a 3×3 convolution, Concat is a splicing operation along the channel dimension, Dropout is a random inactivation layer, ReLU6 is a ReLU6 activation function, and LeakyReLU is a LeakyReLU activation function;

Step B33, design feature aggregation and enhancement module, which is composed of feature aggregation convolution block and collaborative enhancement sub-module, the input feature map is the feature map F _fre obtained in step B1 and the feature map F _f obtained in step B2, after feature aggregation convolution After blocks, the feature map F _conv is obtained, and then the feature map F _co is obtained after the collaborative enhancement sub-module.

Further, step B4 is implemented as follows:

Step B4. Design a full-resolution low-light image enhancement network that aggregates context and enhances details, and is composed of a full-resolution detail extraction module, a frequency-spatial context information attention module, and a feature aggregation and enhancement module. Input the low-light image I, and get three feature maps F _low , F _spa , F _fre after the full-resolution detail extraction module in step B1, get the feature map F _f after the frequency-space domain context information attention module, and finally pass the feature The aggregation and enhancement module obtains the feature map F _co , and then F _co is concatenated with the feature map _Flow in step B1 along the channel dimension, and then undergoes 3×3 convolution to obtain the final enhanced image I _out . The specific formula is expressed as follows:

I _out ＝Conv3(Concat(F _co ,F _low ))

Among them, Conv3 is a 3×3 convolution, and Concat is a splicing operation along the channel dimension.

Further, step C is implemented as follows:

Step C, design loss function, which is composed of L2 loss and VGG perception loss, the total target loss function of the network is as follows:

l＝ω ₁ ||I _out -G|| ² +ω ₂ ||Φ(I _out )-Φ(G)|| ₁

Among them, Φ( ) represents the operation of extracting the features of Conv4-1 layer using the VGG-16 classification model pre-trained on the ImageNet dataset. I _out represents the enhanced image of the low-light image I, G represents the label image corresponding to the low-light image I, ||.|| ₁ represents the L1 loss, ||.|| ² represents the L2 loss, and ω ₁ and ω ₂ are weights.

Further, step D is implemented as follows:

Step D1, randomly divide the training data set obtained in step A into several batches, each batch contains N pairs of images;

Step D2, input a low-illumination image I, obtain an enhanced image I _out after the full-resolution low-illumination image enhancement network that aggregates context and enhances details in step B, and calculates the loss l using the formula in step C;

Step D3. Calculate the gradient of the parameters in the network by using the backpropagation method according to the loss, and update the network parameters by using the Adam optimization method.

Step D4, repeat steps D1 to D3 in batches until the target loss function of the network converges to Nash equilibrium, save the network parameters, and obtain a full-resolution low-light image enhancement model that aggregates context and enhances details.

Those skilled in the art should understand that the embodiments of the present application may be provided as methods, systems, or computer program products. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.

The present application is described with reference to flowcharts and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present application. It should be understood that each procedure and/or block in the flowchart and/or block diagram, and a combination of procedures and/or blocks in the flowchart and/or block diagram can be realized by computer program instructions. These computer program instructions may be provided to a general purpose computer, special purpose computer, embedded processor, or processor of other programmable data processing equipment to produce a machine such that the instructions executed by the processor of the computer or other programmable data processing equipment produce a An apparatus for realizing the functions specified in one or more procedures of the flowchart and/or one or more blocks of the block diagram.

These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing apparatus to operate in a specific manner, such that the instructions stored in the computer-readable memory produce an article of manufacture comprising instruction means, the instructions The device realizes the function specified in one or more procedures of the flowchart and/or one or more blocks of the block diagram.

These computer program instructions can also be loaded onto a computer or other programmable data processing device, causing a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process, thereby The instructions provide steps for implementing the functions specified in the flow chart or blocks of the flowchart and/or the block or blocks of the block diagrams.

The above is only a preferred embodiment of the present invention, and is not intended to limit the present invention to other forms. Any skilled person who is familiar with this profession may use the technical content disclosed above to change or modify the equivalent of equivalent changes. Example. However, any simple modifications, equivalent changes and modifications made to the above embodiments according to the technical essence of the present invention without departing from the content of the technical solution of the present invention still belong to the protection scope of the technical solution of the present invention.

This patent is not limited to the above-mentioned optimal implementation mode, anyone can draw other various forms of full-resolution low-illuminance image enhancement methods that aggregate context and enhance details under the inspiration of this patent. All equivalent changes and modifications should fall within the scope of this patent.

Claims (10)

1. A full-resolution low-light image enhancement method that aggregates context and enhances details, characterized in that: Step A, data preprocessing, including data pairing, data random cutting, data enhancement processing, to obtain a training data set; Step B. Design a full-resolution low-light image enhancement network that aggregates context and enhances details, including: a full-resolution detail extraction module, a frequency-spatial context information attention module, and a feature aggregation and enhancement module; Step C, designing a loss function, which is used to guide the parameter optimization of the network designed in step B; Step D. Use the training data set obtained in step A to train the full-resolution low-light image enhancement network for the aggregated context and enhanced details in step B, converge to Nash equilibrium, and obtain the trained aggregated context and enhanced details for the full-resolution low-light image Illuminance image enhancement model; Step E. Input the low-illuminance image to be tested into the trained full-resolution low-illumination image enhancement model that aggregates context and enhances details, and outputs an enhanced normal-illuminance image. 2. the full-resolution low-illuminance image enhancement method of aggregation context and enhanced detail according to claim 1, is characterized in that, the specific implementation steps of step A are as follows: Step A1, pairing the low-illumination image and the corresponding label image; Step A2. Randomly crop each low-illumination image with a size of h×w×3 into an image with a size of p×p×3, and use the same random cropping method for the corresponding label image, where h, w is the height and width of the low-light image and the label image, and p is the height and width of the cropped image; Step A3. Randomly use one of the following 8 enhancement methods to perform data enhancement on the training paired image: keep the original image, flip vertically, rotate 90 degrees, rotate vertically after 90 degrees, rotate 180 degrees, rotate vertically after 180 degrees , Rotate 270 degrees, rotate vertically after 270 degrees. 3. the full-resolution low-illuminance image enhancement method of aggregation context and enhanced detail according to claim 1, is characterized in that, the specific implementation steps of step B are as follows: Step B1, build a full-resolution detail extraction module, which is composed of a shallow feature extraction submodule, a CBAM-based attention submodule, and a frequency domain transformation submodule, and use the designed network to extract detail features; Step B2. Design the frequency-space context information attention module, which is composed of a multi-scale feature extraction sub-module and a frequency-space feature fusion sub-module, and use the designed network to extract context features; Step B3, design feature aggregation and enhancement module, which is composed of feature aggregation convolution block and collaborative enhancement sub-module, aggregate the detailed features extracted in step B1 and the context features extracted in step B2, and jointly enhance the two types of features; Step B4. Design a full-resolution low-light image enhancement network that aggregates context and enhances details, including a full-resolution detail extraction module, a frequency-spatial context information attention module, and a feature aggregation and enhancement module. 4. the full-resolution low-illuminance image enhancement method of aggregation context and enhanced detail according to claim 3, is characterized in that, the specific implementation steps of step B1 are as follows: Step B11. Design the sub-module of shallow feature extraction. The input is the low-light image I. After 3×3 convolution to obtain the initial feature map F _ori , enter three branches. The first branch contains a 3×3 convolution, The second branch contains 2 serial 3×3 convolutions, and the third branch contains 3 serial 3×3 convolutions, and the processing results F _B1 , F _B2 , and F _B3 of the three branches are along the channel dimension After splicing, after a 3×3 convolution, the feature map F _low output by the shallow feature extraction sub-module is obtained; the specific formula is expressed as follows: F _ori = Conv3(I) F _B1 ＝Conv3(F _ori ) F _B2 ＝Conv3(Conv3(F _ori )) F _B3 ＝Conv3(Conv3(Conv3(F _ori ))) F _low ＝Conv3(Concat(F _B1 ,F _B2 ,F _B3 )) Among them, Conv3 is a 3×3 convolution, and Concat is a splicing operation along the channel dimension; Step B12, build a CBAM-based attention sub-module, which is composed of serial channel-dimension attention Att _c and space-dimension attention Att _s , the input feature map is the feature map _Flow obtained in step B11, and CBAM-based The feature map output by the attention sub-module is F _spa ; the specific formula is expressed as follows: F _spa ＝Att _s (Att _c (F _low )) Among them, Att _c is the attention of the channel dimension, and Att _s is the attention of the spatial dimension; Step B13, design the frequency domain transformation sub-module, the input feature map is the feature map F _spa obtained in step B12, use the Fourier transform function to convert the space domain into the frequency domain, and then go through 3×3 convolution, normalization layer, ReLU After activating the function, use the inverse Fourier transform function to convert the frequency domain into the space domain to obtain the output feature map F _fre ; the specific formula is expressed as follows: F _fre = idft(ReLU(BN(Conv3(dft(F _spa ))))) Among them, dft is the Fourier transform, idft is the inverse Fourier transform, ReLU is the ReLU activation function, BN is the batch normalization layer, and Conv3 is a 3×3 convolution; Step B14, build a full-resolution detail extraction module, which is composed of a shallow feature extraction submodule, a CBAM-based attention submodule and a frequency domain transformation submodule; assuming that the input is the low-illumination image I processed in step A, sequentially pass through The feature maps F _low , F _spa , and F _fre are obtained after processing by the shallow feature extraction submodule, the attention submodule, and the frequency domain transformation submodule. 5. the full-resolution low-illuminance image enhancement method of aggregation context and enhanced detail according to claim 4, is characterized in that, the specific implementation steps of step B2 are as follows: Step B21, design multi-scale feature extraction sub-module, input feature map is marked as F,

H, W, and C are the height, width, and number of channels of feature F respectively. After an average pooling layer with a kernel size of 2×2 and a step size of 2, it goes through 1×1 convolution, ReLU activation function, 1 ×1 convolution and ReLU activation function for dimensionality reduction to obtain intermediate feature maps/>

Then it is divided into two branches. After the upper branch continues to reduce the dimension through 1×1 convolution, the output of the upper branch is obtained through the upsampling layer.

a is the number of channels after dimensionality reduction; the other branch passes through an average pooling layer with a kernel size of 2×2 and a step size of 2, followed by 1×1 convolution, ReLU activation function, 1×1 convolution, The ReLU activation function performs dimensionality reduction to obtain the intermediate feature map/>

The intermediate feature map F ₁₂₁ then passes through the upsampling layer, 1×1 convolution, ReLU activation function, and upsampling layer in sequence to obtain the output of the lower branch/>

After adding F ₁₁ and F ₁₂ , they are concatenated with F in the channel dimension, and after the SE module, the channel is adjusted by 1×1 convolution to obtain the feature map output by the multi-scale feature extraction sub-module/>

The specific formula is expressed as follows: F ₁ =ReLU(Conv1(ReLU(Conv1(Avgpooling(F))))) F ₁₁ ＝Upsampling(Conv1(F ₁ )) F ₁₂₁ =ReLU(Conv1(ReLU(Conv1(Avgpooling(F ₁ ))))) F ₁₂ ＝Upsampling(ReLU(Conv1(Upsampling(F ₁₂₁ )))) Fm＝Conv1(SE(Concat(F ₁₁ +F ₁₂ ,F))) Among them, ReLU is an activation function, Conv1 is a 1×1 convolution, SE(·) is an SE module, Avgpooling is an average pooling layer with a kernel size of 2×2 and a step size of 2, and Upsampling is twice the nearest neighbor upsampling Layer, Concat is a splicing operation along the channel dimension; Step B22, designing a frequency-space domain feature fusion sub-module, which is composed of serial connections of channel attention and spatial attention; Step B23. Design the frequency-space context information attention module, which is composed of three multi-scale feature extraction sub-modules and frequency-space feature fusion sub-modules; the inputs of the three multi-scale feature extraction sub-modules are the three feature maps obtained in step B1 respectively F _low , F _spa , F _fre are respectively processed by the multi-scale feature extraction sub-module designed in step B21 to obtain feature maps F _{low_m} , F _{spa_m} , F _{fre_m} with context information, and then pass through the frequency-space domain designed in step B22 The feature fusion sub-module obtains the feature map F _f output by the frequency-space context information attention module. 6. the full-resolution low-illuminance image enhancement method of aggregation context and enhanced detail according to claim 5, is characterized in that, the specific implementation steps of step B22 are as follows: Step B221, design channel attention, the input is the feature map obtained in step B23

The three feature maps are respectively subjected to the global average pooling of the spatial dimension to obtain three vectors with a scale of 1×1×C, and then the three vectors are spliced along the channel dimension to obtain the intermediate feature map/>

_Fc undergoes 1×1 convolution, ReLU activation function, 1×1 convolution, ReLU activation function, 1×1 convolution to reduce and increase dimensionality, and then obtain the weight on the channel dimension through the Sigmoid activation function/>

Decompose F _W1 along the channel dimension into three vectors F _W10 , F _W11 , F _W12 with a scale of 1×1×C, and multiply them with the input feature maps F _{low_m} , F _{spa_m} , F _{fre_m} of the frequency-space domain feature fusion sub-module respectively , get the output feature map of the channel attention />

The specific formula is expressed as follows: F _c ＝Concat(Avgpooling _s (F _{low_m} ),Avgpooling _s (F _{spa_m} ),Avgpooling _s (F _{fre_m} )) FW ₁ ＝Sigmoid(Conv1(ReLU(Conv1(ReLU(Conv1(F _c )))))) F _{low_c} = F _W10 × F _{low_m} F _{spa_c} = F _W11 × F _{spa_m} F _{fre_c} = F _W12 × F _{fre_m} Among them, Concat is a splicing operation along the channel dimension, Avgpooling _s is the global average pooling of the spatial dimension, ReLU is the activation function, Conv1 is 1×1 convolution, and Sigmoid is the Sigmoid activation function; Step B222, design spatial attention, the input is the three feature maps F _{low_c} , F _{spa_c} , and F _{fre_c} obtained in step B221, and the three feature maps are respectively averaged in the channel dimension to obtain three scales of H×W×1 feature map, and then splicing the three feature maps along the channel dimension to obtain the intermediate feature map

After passing F _s through the average pooling layer with a kernel size of 2×2 and a step size of 2, the ReLU activation function, and the upsampling layer, the weight in the spatial dimension is obtained through the Sigmoid activation function>

Decompose F _W2 into three feature maps F _W20 , F _W21 , and F _W22 with a scale of H×W×1, and multiply them with the input feature maps F _{low_c} , F _{spa_c} , and F _{fre_c of} spatial attention to obtain spatial attention The output feature map of

The specific formula is expressed as follows: F _s ＝Concat(Avgpooling _c (F _{low_c} ),Avgpooling _c (Fs _{pa_c} ),Avgpooling _c (F _{fre_c} )) F _W2 ＝Sigmoid(Upsampling(ReLU(Avgpooling(F _s )))) F _{low_s} = F _W20 × F _{low_c} F _{spa_s} = F _W21 × F _{spa_c} F _{fre_s} = F _W22 × F _{fre_c} Among them, Concat is a splicing operation along the channel dimension, Avgpoolingc is the average pooling of the channel dimension, ReLU is the activation function, Sigmoid is the Sigmoid activation function, Avgpooling is the average pooling layer with a kernel size of 2×2 and a step size of 2, Upsampling is twice the nearest neighbor upsampling layer; Step B223. Design the frequency-space domain feature fusion sub-module. The input feature maps are the feature maps F _{low_m} , F _{spa_m} , and F _{fre_m} obtained in step B23. The three feature maps first pass through the channel attention in step B221 to obtain the feature maps F _{low_c} , F _{spa_c} , F _{fre_c} , and then through the spatial attention in step B222, feature maps F _{low_s} , F _{spa_s} , F _{fre_s} are obtained, and the final output F _f is obtained after adding the three feature maps; the specific formula is as follows: F _f =F _{low_s} +F _{spa_s} +F _{fre_s} . 7. The full-resolution low-illuminance image enhancement method of aggregation context and enhanced details according to claim 6, characterized in that, the specific implementation steps of step B3 are as follows: Step B31, design a feature aggregation convolution block to realize the fusion of detail information and context information; the input feature map is the feature map F _fre obtained in step B1 and the feature map F _f obtained in step B2, and the two are spliced along the channel dimension and then passed through a 3×3 convolution to obtain the output feature map F _conv ; the specific formula is expressed as follows: F _conv ＝Conv3(Concat(F _fre ,F _f )) Among them, Conv3 is a 3×3 convolution, and Concat is a splicing operation along the channel dimension; Step B32. Design a synergistic enhancement sub-module to synergistically enhance the fusion information of detail information and context information; the input feature map is the F _conv obtained in step B31, and the F _conv is sequentially passed through 1×1 convolution, ReLU6 activation function, and Dropout random After the deactivation layer, 1×1 convolution, and Dropout random deactivation layer, it is added to F _conv to obtain the intermediate feature map F _mid , and then through the LeakyReLU activation function, it is spliced with F _conv along the channel dimension and then passed through a 3×3 volume product to obtain the output feature map F _co ; the specific formula is expressed as follows: F _mid = Dropout(Conv1Dropout(ReLU6(Conv1(F _conv )))))+F _conv F _co ＝Conv3(Concat(LeakyReLU(F _mid ),F _conv )) Among them, Conv1 is a 1×1 convolution, Conv3 is a 3×3 convolution, Concat is a splicing operation along the channel dimension, Dropout is a random inactivation layer, ReLU6 is a ReLU6 activation function, and LeakyReLU is a LeakyReLU activation function; Step B33, design feature aggregation and enhancement module, which is composed of feature aggregation convolution block and collaborative enhancement sub-module, the input feature map is the feature map F _fre obtained in step B1 and the feature map F _f obtained in step B2, after feature aggregation convolution After blocks, the feature map F _conv is obtained, and then the feature map F _co is obtained after the collaborative enhancement sub-module. 8. The full-resolution low-light image enhancement method of aggregating context and enhancing details according to claim 7, characterized in that the specific implementation of step B4 is as follows: Step B4. Design a full-resolution low-light image enhancement network that aggregates context and enhances details. It is composed of a full-resolution detail extraction module, a frequency-spatial context information attention module, and a feature aggregation and enhancement module; input a low-light image I, and pass After the full-resolution detail extraction module in step B1, three feature maps F _low , F _spa , and F _fre are obtained, and the feature map F _f is obtained after the frequency-spatial context information attention module, and then the feature map is obtained through the feature aggregation and enhancement module F _co , and then splicing F _co and the feature map F _low in step B1 along the channel dimension, and then undergoing 3×3 convolution to obtain the final enhanced image I _out ; the specific formula is expressed as follows: I _out ＝Conv3(Concat(F _co ,F _low )) Among them, Conv3 is a 3×3 convolution, and Concat is a splicing operation along the channel dimension. 9. The full-resolution low-light image enhancement method of aggregating context and enhancing details according to claim 1, characterized in that the specific implementation of step C is as follows: Step C, design loss function, which is composed of L2 loss and VGG perception loss, the total target loss function of the network is as follows: l＝ω ₁ ||I _out -G|| ² +ω ₂ ||Φ(I _out )-Φ(G)|| ₁ Among them, Φ( ) represents the operation of extracting the features of the Conv4-1 layer using the VGG-16 classification model pre-trained on the ImageNet dataset; I _out represents the enhanced image of the low-light image I, and G represents the label corresponding to the low-light image I Image, ||.|| ₁ represents L1 loss, ||.|| ² represents L2 loss, and ω ₁ and ω ₂ are weights. 10. The full-resolution low-illuminance image enhancement method of aggregating context and enhancing details according to claim 1, wherein the specific implementation steps of step D are as follows: Step D1, randomly divide the training data set obtained in step A into several batches, each batch contains N pairs of images; Step D2, input a low-illumination image I, obtain an enhanced image I _out after the full-resolution low-illumination image enhancement network that aggregates context and enhances details in step B, and calculates the loss l using the formula in step C; Step D3, using the backpropagation method to calculate the gradient of the parameters in the network according to the loss, and using the Adam optimization method to update the network parameters; Step D4: Repeat step D1 to step D3 in batches until the target loss function of the network converges to Nash equilibrium, save the network parameters, and obtain a full-resolution low-light image enhancement model that aggregates context and enhances details.

Images