CN110246105B - Video denoising method based on actual camera noise modeling - Google Patents

Abstract

The invention discloses a video denoising method based on actual camera noise modeling. The specific steps are as follows: (1) Explore the physical causes of the main noise in the imaging process and establish a mathematical model of noise distribution; (2) Based on the established noise model, perform model parameter calibration to generate a realistic noise video training set; (3) Design a video denoising and enhancement neural network, and combine spatial and temporal information to suppress and weaken the noise; (4) Train and optimize the neural network, and use synthetic and actual video datasets to verify the practicability of the method. The denoising method of the present invention is suitable for low-light video denoising, and has very important application requirements in the fields of national defense, military, security monitoring, scientific research and environmental protection.

Description

A video denoising method based on actual camera noise modeling

Technical Field

The present invention relates to the fields of computational photography and deep learning, and in particular to the field of low-light video denoising based on actual camera noise modeling.

Background Art

Under extremely low light conditions, a large amount of noise can significantly reduce the quality of the image, so low-light video imaging is a challenging problem. A large number of video denoising or video enhancement algorithms have been proposed to solve this problem. However, most of the noise models in these algorithms are simple independent and identically distributed assumptions, including additive Gaussian white noise, Poisson noise, or a mixture of Gaussian noise and Poisson noise. In reality, the noise in the video is very complex, especially in low light conditions, and some commonly ignored factors, such as dynamic graphic noise, noise channel correlation, truncation effect, etc., will become major problems.

Methods based on deep learning have made significant progress in many image processing tasks. Deep learning builds a neural network model with many hidden layers and a large amount of training data to learn more useful features of noise, thereby ultimately improving the effect of image denoising. Through layer-by-layer feature transformation, the feature representation of the sample in the original space is transformed into another new feature space, making image reconstruction easier.

Therefore, how to accurately establish a mathematical model for the noise distribution of the video and use neural networks to restore high-quality low-light video is a current research direction.

Summary of the invention

In view of the shortcomings of the above existing video denoising methods, the purpose of the present invention is to propose a video denoising method based on actual camera noise modeling.

To achieve the above object, the technical solution adopted by the present invention is as follows:

A low-light video denoising method based on actual camera noise modeling,

Step 1, establish a mathematical model of actual noise in a weak light environment, which includes dynamic stripe noise, the connection between noise channels, and the truncation effect;

Step 2, using the noise of the actual camera to calibrate the parameters of the mathematical model in step 1, and generate a noise video training set that conforms to the actual situation;

Step 3: construct a video denoising and enhancement neural network to suppress and weaken the noise by combining spatial and temporal information;

Step 4: Use the noise video training set generated in step 2 and the actually collected low-light video data set to train and optimize the neural network.

The present invention takes into account the physical causes of the main noise in actual camera imaging. First, a mathematical model of noise distribution is established. Based on this model, data that is more in line with realistic situations is generated to train a video denoising and enhancement network based on long short-term memory (LSTM). The designed network input can be low-light video of any number of frames, and the network outputs high-quality video frames frame by frame.

The advantages of the present invention are as follows: (1) the proposed actual noise model is based on the physical imaging process and the hardware characteristics of the sensor, and mainly considers three types of non-negligible noise under low-light conditions: dynamic stripe noise, noise channel correlation and truncation effect, so that the model can well handle complex noise in actual situations, especially videos under low-light imaging;

(2) A noise model estimation method is proposed to synthesize a realistic noisy video training set without requiring too many real-world training datasets, which is very attractive for video denoising algorithms that have difficulty in simultaneously acquiring paired noisy and clean video frames.

(3) Designing a long short-term memory (LSTM) video denoising and enhancement neural network, the network structure of the present invention can memorize and utilize information of up to 20 previous frames to process and restore the information of the current frame image;

(4) The low-light video denoising method of the present invention has very important application prospects in the fields of national defense, military, security monitoring, scientific research and environmental protection.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a flow chart of a denoising method according to an embodiment of the present invention.

FIG. 2 is a schematic diagram of noise sources during the imaging process according to an embodiment of the present invention.

FIG. 3 is a 2D lookup table in an embodiment of the present invention, wherein (a) is a truncation effect mean correction, and (b) is a truncation effect variance correction.

FIG4 is a schematic diagram of a network architecture designed in an embodiment of the present invention.

DETAILED DESCRIPTION

The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

1, a video denoising method for actual camera noise modeling in this embodiment, the specific steps are as follows:

Step 1, explore the physical causes of the main noise in the imaging process and establish a mathematical model of noise distribution. In the low-light imaging process, the high-sensitivity camera setting makes some tiny noise become important noise components in low light. Figure 2 shows the common noise sources in the entire imaging process. Based on this physical imaging process, a basic noise model mainly including three noise sources is established. The entire model assumes that the camera obeys a globally consistent linear response curve with a gain of K. The measured value y ⁱ is obtained by the following formula:

^yi ＝K( ^Si + ^Di + ^Ri ) (1)

表示散粒噪声，

是泊松分布，

是像素i处的光电子数量，

表示暗电流，N_d是每个像素值的暗电流电子数，

代表读出噪声，

是高斯分布符号，

高斯分布的方差。Where i is the pixel index, ^yi is the obtained pixel value,

represents the shot noise,

is a Poisson distribution,

is the number of photoelectrons at pixel i,

represents the dark current, _Nd is the number of dark current electrons per pixel value,

represents the read noise,

is the Gaussian distribution symbol,

Variance of the Gaussian distribution.

In addition to the above widely known noise categories, the present invention proposes some special noise sources during low-light imaging.

Dynamic stripe noise: Compared with traditional fixed pattern noise, dynamic stripe noise is a continuous change and cannot be eliminated by direct subtraction. Usually in videos, dynamic stripe noise appears in the form of row stripes. This phenomenon exists not only in rolling shutter cameras, but also in global shutter cameras, but the characteristics are slightly different. After exploring the imaging process of imaging chips, two physical causes are used to explain it, circuit fluctuations and asynchronous triggers. Both causes slightly affect the global row gain, and finally K in formula (1) is replaced by the row gain ^Kr :

是电路波动引起的扰动，符合颜色(1/f)高斯分布。

是异步触发器引起的扰动，符合高斯白噪声分布。λ是电路波动与异步触发器之间的权重衡量。因为

和

都遵循均值为0的高斯分布，K^r的期望恰好是K，为了简化参数的标定，公式(2)简化为：where K is the globally consistent system gain.

It is the disturbance caused by circuit fluctuation and conforms to the color (1/f) Gaussian distribution.

is the disturbance caused by the asynchronous trigger, which conforms to the Gaussian white noise distribution. λ is the weight between the circuit fluctuation and the asynchronous trigger.

and

All follow a Gaussian distribution with a mean of 0. The expectation of K ^r is exactly K. To simplify the calibration of parameters, formula (2) is simplified to:

K ^r ＝Kβ ^r (3)

在全局快门相机中，相当于

β^r分别符合颜色或白色高斯分布

β ^r is the correction parameter for dynamic fringe noise, which is equivalent to

In a global shutter camera, this is equivalent to

β ^r conforms to the color or white Gaussian distribution respectively

是与通道有关的动态条纹噪声的矫正参数，K_c是与通道有关的全局系统增益。Noise channel relationship: In this invention, the noise relationship between channels is simulated by exploring the physical characteristics of the color sensor, while considering pixel uniformity and channel differences. Usually, rolling shutter and global shutter cameras acquire three-channel images by covering a color array filter on a silicon sensor. In practice, the noise of the three channels is not consistent, mainly for two reasons: the three channels have different system gains and the three channels have obvious DPN fluctuations. Based on this, modifying equation (3) K ^r yields:

is the correction parameter for the channel-dependent dynamic fringe noise, and _Kc is the channel-dependent global system gain.

可以表示如下：Truncation effect: Without considering the read noise, digital sensors are usually positive values. However, the read noise follows a Gaussian distribution with a mean of 0, so negative values exist. Especially in low light environments, many signals are even weaker than the read noise, which leads to many negative values. These negative values will be truncated to 0 before the result is output. In mathematical expression, the truncation operation

It can be expressed as follows:

Based on the above analysis, the actual noise model for a weak light environment proposed by the present invention is finally in the form of:

对三通道是相同的。For global shutter cameras,

It is the same for three channels.

Step 2, based on the established noise model, calibrate the model parameters to generate a noise video training set that conforms to reality. The noise model of the present invention can calibrate the parameters of the actual camera noise. In order to facilitate reasoning, truncation operation is not considered first. The present invention proposes to correct the bias of expectation and variance caused by truncation based on a 2D lookup table, as shown in Figure 3.

暗场情况下获取视频帧，可得

利用每行的像素均值除以全局的像素均值可得

即

得到

后，动态图案噪声可以通过除以每行的像素值来移除。推导得到动态图案噪声校正后的算式：yⁱ＝K_c(Dⁱ+Rⁱ)|_c∈{r,g,b}。根据相机的类型，卷帘快门相机或者全局快门相机，确定动态图案噪声的分布特性，为颜色噪声或者高斯白噪声。Step 21, Calibration

Acquire video frames in dark field, and we can get

By dividing the pixel mean of each row by the global pixel mean, we can get

Right now

get

After that, the dynamic pattern noise can be removed by dividing by the pixel value of each row. The formula after dynamic pattern noise correction is derived: ^yi = _Kc ( ^Di + ^Ri ) | _c∈{r,g,b} . Depending on the type of camera, rolling shutter camera or global shutter camera, the distribution characteristics of dynamic pattern noise are determined, which is color noise or Gaussian white noise.

读出噪声

yⁱ校正后的期望和方差可表示为：Step 22, calibrate K _c . Dark current

Read Noise

The corrected expectation and variance of ^yi can be expressed as:

E[y']＝K _c N _d

Substituting N _d with N _d = E[y']/K _c , we get the following formula:

可得最后的公式：Based on the change of dark current with exposure time, the constant can be eliminated by using dark field video frames with different exposures.

The final formula is:

ΔD[y']＝K _c ΔE[y'] (8)

t₁，t₂代表不同的曝光时间。实际中，E[y']可等效于mean(y')，D[y']可等效于var[y']。拍摄一系列不同曝光时间的暗场视频，可计算出一组点(ΔE[y']，ΔD[y'])，所以K_c可由这些点进行线性拟合得出。in

t ₁ , t ₂ represent different exposure times. In practice, E[y'] is equivalent to mean(y'), and D[y'] is equivalent to var[y']. By shooting a series of dark-field videos with different exposure times, a set of points (ΔE[y'], ΔD[y']) can be calculated, so K _c can be obtained by linear fitting of these points.

校准得到K_c后，N_d和

可由公式(6)计算得到。Step 23, calibrate _Nd and

After calibration, K _c is obtained, N _d and

It can be calculated by formula (6).

方差

与真实的均值mean(x)、方差var(x)对应的2D表格。通过查找表格的方法，可得真实数据的值。Step 24, truncation error correction by table lookup. The mean mean(x) and variance var(x) after truncation are very different from the mean mean(x) and variance var(x) without truncation. In actual calculations, it is difficult to calculate the impact of truncation without prior knowledge of pixel x. In the present invention, all random variables in truncation can be divided into two parts, a Poisson distribution part and a Gaussian distribution part with a mean of zero. Matlab is used to generate a series of pixels x with different expectations and variances for a large amount of data, and the mean after truncation is produced.

variance

A 2D table corresponding to the true mean(x) and variance var(x). By looking up the table, the true data value can be obtained.

Step 25, training data synthesis. According to formula (5) and the above camera parameter calibration steps, a noise training data set can be synthesized from a clean video sequence. The present invention first derives the expected number of photoelectrons _Ne ,

Among them, S _pixel is the area of a single pixel, C _lum2radiant is the transfer constant from light flux to radiation intensity, E _p is the energy of a single photon, and Q _e is the equivalent system quantum efficiency of the camera. By adjusting the average value of the image to the expected number of photoelectrons E[N _e ], the expected number of photons can be derived from the image pixel value. Finally, according to the Monte Carlo simulation formula, the noise training data set is synthesized according to formula (5).

Step 3, design a video denoising and enhancement neural network, and combine spatial and temporal information to suppress and weaken noise. The present invention proposes a video denoising and enhancement network based on LSTM, the input is a noisy video shot under low light by an actual camera, and the output is a bright and clear video frame. Figure 4 is a network architecture designed in this embodiment. In order to adaptively extract short-term dependencies and long-term dependencies from the video at the same time, a spatiotemporal joint memory unit ST-LSTM is used in the network. ST-LSTM can model spatial and temporal representations in a unified memory unit, and transfer the memory vertically across layers and horizontally across states. The network architecture of the present invention includes two convolutional layers and four ST-LSTM layers. First, the convolutional layer extracts the features of the input frame and then transfers the features to the ST-LSTM layer. Jump connections are added to the spatial association. The last layer merges the reconstruction information learned by the previous layer into the sRGB space. The convolution kernel size used in the first and fourth ST-LSTM layers is 3x3, the convolution kernel size used in the second and third ST-LSTM layers is 5x5, and the number of features in each layer is 64. Zero padding is used to ensure size consistency between input and output.

Step 4: Train and optimize the neural network, and use the noise training data set synthesized in step 25 and the actual low-light video data set collected to verify the practicability of the method. The network is trained by minimizing the loss function (Formula (10)) between the frame I output by the network and the corresponding actual real frame I ^* .

和

损失的加权平均值，

距离和

都计算像素强度的一致性，前者使输出平滑，而后者使输出更加细致。为了进一步提高感知质量，引入感知损失

其是通过预先训练的视觉感知组(VGG)网络提取高级特征进行网络输出和真实值的约束。此外，在损失函数中加入了总变分稳压器

作为平滑的正则化项。在这里，α,β,γ,δ是训练过程的超参数，N是训练的视频帧数。本发明的训练过程中，α＝5,β＝1,γ＝0.06,δ＝2×10^-6，N＝8。The basic loss function is defined as

and

The weighted average of the losses,

Distance and

Both calculate the consistency of pixel intensity, the former makes the output smoother, while the latter makes the output more detailed. In order to further improve the perceptual quality, perceptual loss is introduced

It extracts high-level features from a pre-trained Visual Group (VGG) network to constrain network output and true values. In addition, a total variation regulator is added to the loss function.

As a smooth regularization term. Here, α, β, γ, δ are hyperparameters of the training process, and N is the number of video frames for training. In the training process of the present invention, α＝5, β＝1, γ＝0.06, δ＝2×10 ^-6 , N＝8.

The deep learning network structure implemented by the network of the present invention is Pytorch, the optimizer used in the training process is Adma, and the learning rate is 1× ^10-6 . The training set contains a large number of clean videos, and about 900 sequences are selected, which are very rich in mobile scenes. Considering that each camera has a unique set of noise parameters, the network is trained with different training data for different cameras.

Claims (4)

1. A video denoising method based on actual camera noise modeling is characterized by comprising the following steps:

step 1, establishing an actual noise mathematical model of a weak light environment, wherein the model comprises dynamic stripe noise, inter-noise channel relation and truncation influence, and the actual noise mathematical model specifically comprises the following steps:

where i is the pixel index, y ⁱ Is the value of the pixel that was acquired,

is a truncation operation, K _c Is the global system gain associated with the channel,

is a fluctuation factor of the dynamic streak noise; s ⁱ Indicates shot noise, and>

is a poisson distribution,. Sup.>

Is the number of photoelectrons at pixel i; d ⁱ Represents dark current, is greater than or equal to>

N _d Is the number of dark current electrons, R, per pixel value ⁱ Represents a readout noise, <' > or>

Is a Gaussian distribution sign>

The variance of Gaussian distribution, c ∈ { r, g, b } is a color three channel;

step 2, calibrating parameters of the mathematical model in the step 1 by using noise of an actual camera to generate a noise video training set which accords with an actual situation;

step 3, constructing a video denoising and enhancing neural network, and combining spatial and temporal information to suppress and weaken noise;

and 4, training and optimizing the neural network by using the noise video training set generated in the step 2 and the actually acquired low-light video data set.

2. The method as claimed in claim 1, wherein in step 2, a series of pixels x with different mean and variance are generated, and then the mean value after truncation operation is made

Variance->

Two-dimensional tables corresponding to the true mean (x), variance var (x), by looking up the tablesAnd the grid method can obtain the value of the real data.

3. The method for denoising video based on actual camera noise modeling according to claim 1, wherein in step 3, the input of the video denoising and enhancing neural network is the noise video shot under the weak light of the actual camera, and the output is a bright and clear video frame.

4. The method as claimed in claim 1, wherein in step 4, the frame I outputted by the minimization network and the corresponding actual real frame I are outputted ^* Training the network with a loss function as follows:

wherein,

is a final loss function>

Is an absolute value error function>

Mean square error function->

A perception-loss function, <' > or>

The total variation regularization function, α, β, γ, and δ are all hyper-parameters. />

Images