CN115082315B - Demosaicing method applicable to low-illumination small-pixel CFA sampling and edge computing equipment - Google Patents

Abstract

The invention provides a demosaicing method applicable to low-illumination small-pixel CFA sampling and edge computing equipment, which comprises the following steps of: after Gaussian noise is added to the original data set and the brightness is reduced, the RGB image is processed into a mosaic image through the CFA with 75% transparent elements, and parameters of a training target neural network are set. Step 2: and building a neural network model taking the UNet++ network as a main framework. Step 3: according to the neural network model, training a corresponding network model in two stages with the aim of minimizing respective loss functions. Step 4: and processing the image to be processed by using the trained neural network model to obtain a demosaiced image. The invention optimizes the network topology structure on the basis of ensuring the image restoration effect of the small-pixel color filtering whole-row mosaic, reduces the parameter as much as possible, accelerates the training time, and can be applied to network deployment on edge computing equipment.

Description

Low-light small-pixel CFA sampling and de-mosaicing methods for edge computing devices

Technical Field

The present invention belongs to the field of digital image processing, and more specifically, to the collaborative design of software and hardware of a visual processing neural network.

Background Art

In computer vision, many deep learning methods have been used to complete the de-mosaicing task. However, these methods only focus on the optimal effect at the algorithm level, so the number of parameters is huge, and they rely on high-cost, low-energy-efficiency graphics processing units or remote computing centers. Under tight energy and cost budgets, the training time is long, and these algorithms are difficult to deploy on portable or mobile real-time systems. On the other hand, most of these algorithms are based on the mosaic image restoration after sampling the color filter array of regular pixel size based on the classic Bayer pattern, and are not suitable for de-mosaicing of mosaic images obtained through low-light small-pixel color filter arrays.

Therefore, existing technical solutions mainly focus on the de-mosaicing effect at the algorithm level, without considering the calculation time of the algorithm and whether it is suitable for deployment on edge computing devices. In addition, these solutions are suitable for color filter arrays with regular-sized pixels in a conventional Bayer pattern, and are not suitable for small pixels in low light conditions.

Summary of the invention

In view of the shortcomings of the prior art, the present invention proposes a de-mosaicing method based on low-light small-pixel CFA (color filter array) sampling and applicable to edge computing devices. This method optimizes the network topology, minimizes the number of parameters, and accelerates the training time on the basis of ensuring the restoration effect of the entire mosaic image of small-pixel color filtering. At the same time, it can be suitable for network deployment on edge computing devices.

The technical solution of the present invention is as follows:

A demosaicing method for low-light small-pixel CFA sampling and edge computing devices, comprising the following steps:

Step 1: First, add Gaussian noise to the original data set and reduce the brightness. Then, process the RGB image into a mosaic image through CFA with 75% transparent elements. Then, perform data preprocessing (such as cropping each image to 256*256 and then cropping it again into 4 128*128 images) to form a training set and set the parameters of the training target neural network.

Step 2: Build a neural network model with UNet++ network as the main framework.

Step 3: According to the neural network model, the corresponding network model is trained in two stages with the goal of minimizing the respective loss functions.

Step 4: Using the trained neural network model, the image to be processed that has been sampled using a color filter array with 75% transparent elements and to which Gaussian noise has been added and brightness has been reduced is demosaiced to obtain a demosaiced image.

Furthermore, in step 1, the method of adding Gaussian noise to the image and reducing the brightness is:

By calculating the variance v of a single black photo image and using this value as a benchmark, the Gaussian noise with a mean of 0 and a variance of v is multiplexed onto the three channels of the dataset to be processed according to the following formula (1):

Y＝C(A·X+N(0,B·v)) (1)

Where, Y: processed low-light noise image, C: image pixel truncation function, A: light reduction factor, X: original image to be processed, N(a,b): Gaussian normal distribution noise generation function with mean a and variance b, B: Gaussian noise variance fine-tuning factor.

In the above, using Gaussian smoothing instead of convolutional structure can ensure that semantics at different levels can be connected across layers.

Furthermore, in step 1, forming a training set after data preprocessing means sampling and preprocessing the data set used for training according to the mode of the color filter array to obtain the data set used for neural network training, including:

For the image to which Gaussian noise has been added and the brightness has been reduced, sampling is performed according to the data sampling method of the color filter array with 75% transparent elements. The image containing only B channel sampling pixels, the image containing only R channel sampling pixels, the image G1 containing only G channel sampling pixels, the image G2 containing only G channel sampling pixels, and the image containing only transparent channel sampling pixels are respectively divided into multiple mosaic image blocks, among which the size of the image containing only transparent channel sampling is 3 times that of other images. After two downsamplings, a transparent channel image block with the same size as other image blocks is obtained.

Furthermore, the neural network model in step 2 includes a feature extraction module and an image reconstruction module.

The feature extraction module includes three Gaussian smoothing and feature extraction, six cross-layer connections and one upsampling structure. In the feature extraction module, there are three Gaussian smoothings in total, and the relationship between the three Gaussian smoothings is as follows: the input image is firstly smoothed once to obtain a picture that has been smoothed once, and then the picture that has been smoothed once is horizontally sent to the feature extraction module; at the same time, the picture that has been smoothed once is also smoothed downward once again to extract a higher level of semantics, and a picture that has been smoothed twice is obtained, and then the picture that has been smoothed twice is also horizontally sent to the feature extraction module; at the same time, the picture that has been smoothed twice is also smoothed downward once again to extract a higher level of semantics, and a picture that has been smoothed three times is obtained, and then the picture that has been smoothed three times is also horizontally sent to the feature extraction module.

In the feature extraction module, the images that have been Gaussian smoothed 0, 1, 2, and 3 times are first sent to the feature extraction modules of each layer, and the convolution kernel size in different feature extraction modules varies according to the number of smoothing times. There are different convolution kernel sizes in the feature extraction modules of each layer, among which the convolution kernel size after 0 Gaussian smoothing is 3*3, the convolution kernel size after 1 Gaussian smoothing is 3*3, the convolution kernel size after 2 Gaussian smoothing is 5*5, and the convolution kernel size after 3 Gaussian smoothing is 7*7. The role of these convolution structures is to further expand the receptive field, so that the network can obtain a wider range of image information in the upper layer of the involved triangular structure. After the convolution structure of each layer are three densely connected units, each of which consists of three depth-separable convolutions and one PReLU. In each densely connected unit, each depth-separable convolution structure uses a dense connection structure, that is, the input and output of the first layer of depth-separable convolution will be sent together to the input of the second layer of depth-separable convolution, and similarly, the input and output of the second layer of depth-separable convolution will be sent together to the input layer of the third layer of depth-separable convolution. At the same time, in the feature reconstruction module, the number of input and output channels of each densely connected unit.

The image reconstruction module consists of a 3×3 convolution, a 1×1 convolution structure and a PReLU, and is used to reconstruct the feature map after feature extraction into a mosaic-free and noise-free image.

Furthermore, the feature extraction module embeds a densely connected unit with a residual connection structure in a feature extraction module connected with a residual connection structure. The residual structure consists of a convolution structure and three densely connected units. The convolution structure is used to preliminarily extract corresponding horizontal features of different layers while further expanding the receptive field. Each densely connected unit contains three depth-wise separable convolutions and six PReLU activation layers.

Further, the step 3 comprises:

3.1 Data selection,ImageNet is selected as the data set for training the network. Before being used for training, the image is first cropped into a 256*256 resolution image based on the center, and then the network is cropped into a 128*128 resolution image for network training.

3.2 Optimizer selection: Adam optimization is used, the initial learning rate is set to 0.001, the learning rate is reduced to half of the original value every 10 cycles, the mini-batch size is 16, and other hyperparameters are set by default.

3.3 Loss function design. Within 10 cycles of training (e≤10), the SSIM indicator for measuring local similarity is used as the basis for loss function design. After 10 cycles (e>10), the MSE for measuring global features is used as the basis for loss function design. The total number of experimental cycles is set to 150 cycles.

It can be seen from the above technical solutions that the solution of the present invention is a neural network demosaicing method suitable for edge computing devices that is matched with a color filter array dedicated to low-light small-pixel cameras, and its advantages are:

1. De-noising and demosaicing of images sampled with a color filter array with 75% transparent elements suitable for low light conditions.

2. Based on the original UNet++ network structure, the original convolution structure is replaced with a Gaussian smoothing structure in the downsampling stage of the UNet++ network, ensuring that semantics at different levels can be connected across layers and feature maps with different semantics can be used for image restoration at the same time.

3. Different loss functions are used in different training cycles for neural network training, which makes full use of prior knowledge, ensuring both the local effect of image restoration and the training convergence speed of the network.

4. On the basis of ensuring the image restoration effect, the network topology is modified and the traditional convolution in the upsampling stage is replaced by the depthwise separable convolution to reduce the number of network parameters. After the network training is completed, the appropriate network scale can be selected by pruning according to the actual situation, which is convenient for porting to edge computing devices.

5. This method is more efficient and takes less training time.

The neural network algorithm designed in the present invention is specifically used for denoising and demosaicing of RAW images obtained by sampling the color filter array under low-light and small-pixel conditions. On the basis of ensuring good image restoration quality, the parameters of the neural network are reduced by modifying the topological structure and other methods, making the neural network suitable for edge computing devices.

BRIEF DESCRIPTION OF THE DRAWINGS

Figure 1 is a diagram showing the processing effect of adding Gaussian noise and reducing brightness;

Fig. 2 is a color filter array;

Fig. 3 Dataset preprocessing process;

Figure 4 is a network topology;

Figure 5 is the feature extraction module.

DETAILED DESCRIPTION

The technical implementation of the present invention is further described in detail below with reference to the accompanying drawings:

The solution proposed by the present invention is a demosaicing method for a color filter array with 75% transparent elements. This method mainly solves the problem of different sizes of feature maps, and realizes the demosaicing effect based on the high sensitivity of the color filter array and the characteristics of being suitable for small cameras. The method mainly includes the following steps:

Step 1: First, after adding Gaussian noise to the original data set and reducing the brightness, the RGB image is processed into a mosaic image through CFA with 75% transparent elements. After data preprocessing, the training set is constructed, and the parameters of the training target neural network are set.

Step 2: Build a neural network model with UNet++ network as the main framework.

Step 3: According to the neural network model, the corresponding network model is trained in two stages with the goal of minimizing the respective loss functions.

Step 4: Using the trained neural network model, the image to be processed, which has been added with Gaussian noise and has reduced brightness, and sampled according to a color filter array with 75% transparent elements, is processed to obtain a demosaiced image.

The following examples describe in detail the specific implementation of each step:

Step 1: First, add Gaussian noise to the original image, reduce the brightness, and then process the RGB image into a mosaic image through a CFA with 75% transparent elements. After data preprocessing, form a training set and set the parameters of the training target neural network. Specifically, it includes:

Step 1.1, add Gaussian noise of appropriate intensity to the original image to reduce the brightness:

By calculating the variance v of a single black photo image and using this value as a benchmark, Gaussian noise with a mean of 0 and a variance of v is multiplexed onto the three channels of the data set to be processed according to the following formula (1).

Y＝C(A·X+N(0,B·v)) (1)

in,

Y: processed low-light noise image;

C: image pixel truncation function;

A: The light reduction multiple;

X: original image to be processed;

N(a,b): noise generating function that obeys the Gaussian normal distribution with mean a and variance b;

B: Gaussian noise variance fine-tuning multiple.

This process is designed to simulate the intensity of noise such as photon readout noise generated by electronic devices during image transmission from the sensor. By further comparing the processing effect of reducing brightness by adding noise after fine-tuning the intensity, a low-light image with Gaussian noise with a mean of 0 and a variance of B·V is obtained, as shown in Figure 1.

According to the above results, the parameters of A=0.2, B=2 and 3 are selected to process the image.

Step 1.2, process the RGB image into a mosaic image through CFA with 75% transparent elements, and form a training set after data preprocessing:

Take the minimum repeating 4*4 unit as an example. There are 16 elements in the 4*4 unit. The upper left corner is sampled according to the Bayer mode, that is, the main diagonal samples the green value, the upper right corner samples the red value, and the lower left corner samples the blue value. The rest are transparent samples, that is, these parts only sample the brightness value of the original image (here calculated according to Gray = R*0.299+G*0.587+B*0.114). The specific sampling method can refer to the following paper: G _ANG L _UO , "A _{NOVEL COLOR FILTER ARRAY WITH} 75％ TRANSPARENT ELEMENTS," PROC. SPIE 6502, DIGITAL PHOTOGRAPHY III, 65020T (20 FEBRUARY 2007); DOI: 10.1117/12.702950.

The data sampling method of the color filter array involved in the present invention (as shown in FIG2 ) is analyzed, and the original image is sampled according to the method shown in FIG2 .

First, the R and G channels of the pixels whose horizontal coordinates modulo 4 are 0 and whose vertical coordinates modulo 4 are 1 in the original image are set to 0, so as to obtain an image containing only B channel sampling pixels; the B and G channels of the pixels whose horizontal coordinates modulo 4 are 1 and whose vertical coordinates modulo 4 are 0 in the original image are set to 0, so as to obtain an image containing only R channel sampling pixels; the R and G channels of the pixels whose horizontal coordinates modulo 4 are 0 and whose vertical coordinates modulo 4 are 0 in the original image are set to 0, so as to obtain an image G1 containing only G channel sampling pixels; the R and G channels of the pixels whose horizontal coordinates modulo 4 are 1 and whose vertical coordinates modulo 4 are 1 in the original image are set to 0, so as to obtain an image G2 containing only G channel sampling pixels; the pixel values in the original image except the pixels at the above coordinates are calculated according to the formula Gray=R*0.299+G*0.587+B*0.114, and then sampled to obtain an image containing only transparent channel sampling pixels.

Then, the image containing only B channel sampling pixels, the image containing only R channel sampling pixels, the image G1 containing only G channel sampling pixels, the image G2 containing only G channel sampling pixels, and the image containing only transparent channel sampling pixels are respectively divided into a plurality of mosaic image blocks. The size of the image containing only transparent channel sampling pixels is three times that of other images, and after two downsamplings, a transparent channel image block with the same size as other image blocks is obtained. This process is shown in FIG3.

In the above manner, the data set used for training is sampled and preprocessed according to the mode of color filtering array with 75% transparent elements to obtain the data set used for neural network training in this design.

Step 1.3, set the parameters for training the target neural network.

Specifically, the optimizer during training is set to Adam, the initial learning rate is set to 0.001, and the learning rate is reduced by half every 10 epochs during training. The mini-batch size during training is set to 16. The rest of the hyperparameters are set by default.

Step 2: Build a neural network model with UNet++ as the main framework

The main framework of the model is the UNet++ structure, and the network features are as follows:

The overall model of this method is shown in Figure 4. The neural network model includes a feature extraction part and an image reconstruction module.

2.1 Feature extraction part:

The feature extraction part includes three Gaussian smoothing and feature extraction, six cross-layer connections, and one upsampling structure. The processing process is as follows:

In the feature extraction module, there are three Gaussian smoothings in total. The relationship between these three Gaussian smoothings is as follows: the input image is first Gaussian smoothed once to obtain a picture that has been smoothed once, and then the picture that has been smoothed once is horizontally sent to the feature extraction module; at the same time, the picture that has been smoothed once is downwardly smoothed once again to extract higher-level semantics, and obtain a picture that has been smoothed twice, and then, the picture that has been smoothed twice is also horizontally sent to the feature extraction module; at the same time, the picture that has been smoothed twice is downwardly smoothed once again to extract higher-level semantics, and obtain a picture that has been smoothed thrice, and then, the picture that has been smoothed thrice is also horizontally sent to the feature extraction module.

In the feature extraction module, the images that have been Gaussian smoothed 0, 1, 2, and 3 times are first sent to the feature extraction modules of each layer, and the convolution kernel size in different feature extraction modules varies according to the number of smoothing times. There are different convolution kernel sizes in the feature extraction modules of each layer, among which the convolution kernel size after 0 Gaussian smoothing is 3*3, the convolution kernel size after 1 Gaussian smoothing is 3*3, the convolution kernel size after 2 Gaussian smoothing is 5*5, and the convolution kernel size after 3 Gaussian smoothing is 7*7. The purpose of these convolution structures is to further expand the receptive field, so that the network can obtain a wider range of image information in the upper layer of the involved triangular structure. After the convolution structure of each layer are three densely connected units, each of which consists of three depth-separable convolutions and one PReLU. In each densely connected unit, each depth-separable convolution structure uses a dense connection structure, that is, the input and output of the first layer of depth-separable convolution will be sent together to the input of the second layer of depth-separable convolution, and the input and output of the second layer of depth-separable convolution will be sent together to the input layer of the third layer of depth-separable convolution. At the same time, in the feature reconstruction module, the number of input and output channels of each densely connected unit is the same.

In the above feature extraction module, after the input image is Gaussian smoothed three times, the network obtains three different levels of image blur levels. The advantage of this action is that it can obtain feature maps of different semantic levels without changing the size of the intermediate feature map, which is convenient for subsequent cross-layer connection. After Gaussian smoothing, the image is sent to the feature extraction modules of each layer as shown in Figure 5.

As shown in FIG5 , the feature extraction module of the neural network model described in the present method embeds a dense connection unit with a residual connection structure in the feature extraction module connected with a residual connection structure. The residual structure consists of a convolution structure and three dense connection units, wherein the convolution structure is used to preliminarily extract the corresponding horizontal features of different layers while further expanding the receptive field. Each dense connection unit contains three depth-separable convolutions and six PReLU activation layers. The residual structure consists of a convolution structure and three dense connection units, wherein the convolution structure is used to preliminarily extract the corresponding horizontal features of different layers while further expanding the receptive field. In the present neural network model, the sizes of the convolution kernels of layers 0, 1, 2, and 3 are 3×3, 3×3, 5×5, and 7×7, respectively. The feature extraction module also contains three dense connection units, each of which contains three depth-separable convolutions and six PReLU activation layers.

The input and output channel numbers of the feature extraction module of the network model included in this method are consistent. The channel number parameters of the middle layer are shown in Table 1.

Table 1

2.2 Image reconstruction part:

After the input image passes through the above feature extraction part, the feature map will be reconstructed into a mosaic-free and noise-free image in the purple part in Figure 4. This rightward and upward process is the image reconstruction part. In this part, the features of each layer will be connected with the features of each layer of the previous and lower levels by 1×1 convolution (as shown by the white circle in the figure); each image reconstruction operation in this part consists of a 3×3 convolution, a 1×1 convolution structure and a PReLU; the dotted part in the figure represents the connection between layers of different semantic levels, which is realized by 1×1 convolution; the orange arrow at the top of the structure represents the upsampling realized by the 3×3 transposed convolution structure. The final output image between layers will be obtained at the top layer of the structure, which is the output of L＝1, L＝2, L＝3 in the figure. In the image reconstruction part, in the process before the transposed convolution operation, the image size remains unchanged. After the transposed convolution, the length and width of the image are each doubled. The input image size, convolution kernel size, and output image size of the intermediate process are shown in Table 2.

Table 2

Step 3: According to the neural network model, the corresponding network model is trained in two stages with the goal of minimizing the respective loss functions.

3.1 Data Selection,This method selects ImageNet as the dataset for training the network. Before using it for training, the image is firstly cropped into 256*256 resolution images based on the center, and then the network is cropped into 128*128 resolution images for network training.

3.2 Optimizer selection. This method uses Adam optimization. The initial learning rate is set to 0.001. The learning rate is reduced to half of the original value every 10 cycles. The mini-batch size is 16, and other hyperparameters are set by default.

3.3 Loss function design. Within 10 cycles of training (e≤10), the SSIM indicator for measuring local similarity is used as the basis for loss function design. After 10 cycles (e>10), the MSE for measuring global features is used as the basis for loss function design. The total number of experimental cycles is set to 150 cycles.

The SSIM loss function L _SSIM is as follows:

The MSE loss function L _MSE is as follows:

The overall loss function is as follows:

Step 4: Using the trained neural network model, the image to be processed, which has been added with Gaussian noise and has reduced brightness, and sampled according to a color filter array with 75% transparent elements, is processed to obtain a de-mosaiced image.

It can be seen from the above embodiments that this method, as a de-mosaicing algorithm for a color filter array with 75% transparent elements, is suitable for small-pixel color filter arrays under low-light conditions. This algorithm mainly solves the problem of different sizes of feature maps, and achieves a de-mosaicing effect based on the higher photosensitivity of the color filter array and its suitability for small cameras.

In addition, this method realizes the idea of collaborative design of software and hardware. On the basis of ensuring good image restoration quality, it reduces the number of parameters from the perspective of topological structure by modifying the topological structure and other methods while ensuring an acceptable de-mosaicing effect. This reduces the number of parameters of the neural network and makes the neural network suitable for edge computing devices.

Claims (5)

1. A demosaicing method applicable to low-light small-pixel CFA sampling and edge computing equipment, comprising the steps of:

Step 1: firstly, after Gaussian noise is added to an original data set and the brightness is reduced, an RGB image is processed into a mosaic image through a color filter array mode (CFA) with 75% transparent elements, data preprocessing is carried out to form a training set, and parameters of a training target neural network are set;

Step 2: building a neural network model taking a UNet++ network as a main framework, wherein the neural network model comprises a feature extraction part and an image reconstruction module:

The characteristic extraction part comprises a three-time Gaussian smoothing and characteristic extraction module, a cross-layer connection structure and a one-time up-sampling structure; the feature extraction module is embedded with a dense connection unit with a residual connection structure in a feature extraction module connected by the residual connection structure, wherein the residual connection structure consists of a convolution structure and three dense connection units, the convolution structure is used for preliminarily extracting corresponding horizontal features of different layers and further expanding a receptive field, and each dense connection unit comprises three depth separable convolutions and six PReLU activation layers;

The image reconstruction module consists of a 3×3 convolution product, a1×1 convolution structure and a PReLU, and is used for reconstructing the feature map after feature extraction into a mosaic-free and noise-free image;

step 3: according to the neural network model, training a corresponding network model in two stages with the aim of minimizing respective loss functions, including:

3.1 data selection, namely selecting an ImageNet as a data set of a training network, firstly cutting a picture into 256-by-256 resolution pictures by taking a center as a reference before the training, and then cutting the network into 128-by-128 resolution pictures and then using the network for network training;

3.2, selecting by an optimizer, adopting Adam optimization, setting an initial learning rate to be 0.001, reducing the learning rate to be half of the original learning rate every 10 periods, and setting the small batch size to be 16, wherein other super parameters adopt default settings;

3.3 designing a loss function, wherein e is less than or equal to 10 in 10 training periods, SSIM indexes for measuring local similarity are adopted as the basis of the loss function design, and after 10 training periods, e is more than 10, MSE for measuring global characteristics is adopted as the basis of the loss function design, and the total experimental period number is set to 150;

Step 4: and processing the image to be processed which is sampled in a color filter array mode with 75% transparent elements by utilizing the trained neural network model, wherein Gaussian noise is added, the brightness is reduced, and a demosaiced image is obtained.

2. The method for demosaicing applicable to a low-light small-pixel CFA sampling and edge computing device according to claim 1, wherein in the step 1, the method for adding gaussian noise to an original data set and reducing brightness is as follows:

by calculating the variance v of the image of the single Zhang Hei photo and taking the value as a reference, gaussian noise subject to a mean of 0 and distributed with the variance v is multiplexed onto three channels of the original dataset according to the following formula (1)

Y＝C((A·X+N(0,B·v)) (1)

Wherein Y is a processed low-light noise image, C is an image pixel cut-off function, A is an illumination reduction multiple, X is an original image to be processed, N (a, B) is a Gaussian normal distribution noise generation function with a mean value of a and a variance of B, and B is a Gaussian noise variance fine adjustment multiple.

3. The method for demosaicing as recited in claim 1, wherein in step 1, the processing of the RGB image into the mosaic image by the CFA having 75% transparent elements means:

The original image is sampled in a mode of a color filter array with 75% transparent elements, and an image only containing B channel sampling pixels, an image only containing R channel sampling pixels, an image only containing G channel sampling pixels G2 and an image only containing transparent channel sampling pixels are respectively divided into a plurality of mosaic image blocks, wherein the size of the image only containing transparent channel sampling is 3 times that of other images, and after downsampling is carried out twice, the transparent channel image blocks with the same size as the other image blocks are obtained.

4. The low-light small-pixel CFA sampling and edge computing device-adapted demosaicing method of claim 1, wherein the sizes of the convolution kernels of the 0, 1,2, 3 layers in the neural network model are 3 x 3, 5 x 5, 7 x 7, respectively.

5. The low-light small-pixel CFA sampling and edge computing device-adapted demosaicing method of claim 1, wherein in step 3.3:

The SSIM loss function L _SSIM is as follows:

Wherein, Representing the structural similarity of two images, where x and y represent the two images used to calculate the structural similarity, where mu _x is the average of x, mu _y is the average of y,Is the variance of the x-value,Is the variance of y, σ _xy is the covariance of x and y, and C ₁＝(k₁L)²,C₂＝(k₂L)² is a constant used to maintain stability; l is the dynamic range of pixel values, k ₁＝0.01,k₂ =0.03; the size of the sliding window in the SSIM calculation process in this design is set to 11;

M in the loss function L _SSIM represents that the matrix abstracted from the two images has M rows in total, and N represents that the matrix abstracted from the two images has N columns in total;

After the number of epochs trained exceeds 10, the loss function takes the form of MSE loss function L _MSE, which is specified as follows:

The overall loss function is as follows:

the superscript 1 for the loss function in the above equation indicates that the layer loss functions L ₁、L₂、L₃ of the network that introduce deep supervision need to be considered together in the training process, and therefore the above SSIM loss functions representing layers 1, 2, and 3, the sameRepresenting layer 1, 2, 3 MSE loss functions.