CN112529854B

CN112529854B - Noise estimation method, device, storage medium and equipment

Info

Publication number: CN112529854B
Application number: CN202011375395.5A
Authority: CN
Inventors: 李琤; 宋风龙; 黄亦斌
Original assignee: Huawei Technologies Co Ltd
Current assignee: Huawei Technologies Co Ltd
Priority date: 2020-11-30
Filing date: 2020-11-30
Publication date: 2024-04-09
Anticipated expiration: 2040-11-30
Also published as: CN112529854A

Abstract

The application relates to the technical field of artificial intelligence and discloses a noise estimation method, a device, a storage medium and equipment, wherein the method comprises the following steps: firstly, dividing N frames of images to be estimated into a plurality of image blocks with fixed sizes; performing DCT on each image block to obtain a low-frequency component, an intermediate-frequency component and a high-frequency component; then calculating the pixel estimation value of the low frequency component, calculating the optimal sub-threshold value corresponding to the medium frequency component, correcting the power spectrum density of the high frequency component, correcting the ISP (Internet service provider) path of the image signal processor of the high frequency component to obtain an initial noise estimation value, and further determining the final noise estimation result of the N frames of images to be estimated according to the pixel estimation value and the initial noise estimation value of the low frequency component of the N frames of images. Therefore, the power spectrum density and ISP channel correction and other processes are respectively carried out on the low-frequency component, the medium-frequency component and the high-frequency component, so that the accuracy of the noise estimation result of the real hand image and the video can be improved.

Description

Noise estimation method, device, storage medium and equipment

Technical Field

The present disclosure relates to the field of artificial intelligence technologies, and in particular, to a noise estimation method, apparatus, storage medium, and device.

Background

With the rapid development of mobile internet, internet of things and artificial intelligence (artificial intelligence, AI) technology, photographing and video quality of terminal devices such as smart phones have advanced rapidly. But is limited by the hardware capabilities of the end device optical sensor and the hardware area and power consumption constraints of the image signal processor (image signal processor, ISP), the image and video quality is still not high enough. Therefore, image and video denoising is still a very urgent need, so that the imaging quality is greatly improved, and the accuracy of later computer vision processing is improved. In order to realize effective image and video denoising, accurate noise estimation is important, and the actual denoising effect is greatly affected.

Currently, there are generally two methods for noise estimation: an estimation method based on poisson-Gaussian Noise (PG Noise) model, the method is to make discrete wavelet transform (discrete wavelet transform, DWT) on the image by assuming that the original data Noise is composed of poisson Noise related to the signal and Gaussian Noise irrelevant to the signal, in order to separate the image signal from the Noise, divide the image at the low frequency of the DWT, obtain the signal intensity estimation value by using the low frequency coefficient of each divided area, obtain the Noise intensity estimation value by using the high frequency coefficient, finally fit the relation between all the signals and the Noise intensity to obtain the final Noise estimation model, but the accuracy of the estimation method is seriously influenced by the image content, the robustness is insufficient, and the assumption of P-G Noise model is not satisfied for the dark area and overexposure position of the original data, meanwhile, the model does not consider the camera shooting parameters and cannot estimate the whole scene Noise; yet another common Noise estimation method is an estimation method based on a Noise Flow (Noise Flow) model, which converts a gaussian white Noise (white gaussian Noise, WGN) distribution into a true Noise distribution by data training, and takes a minimized negative maximum likelihood loss (negative likelihood loss, NLL) as a training target. And some shooting parameters (such as ISO, camera model and the like) are introduced as priori information to guide the network to generalize on different scenes, but the method has the defects that a large number of Jacobian ranks, matrix inversions and other operators are contained in the calculation process, support and acceleration operations of hardware (such as a graphic processing unit (graphics processing unit, GPU), a network processing unit (network processing unit, NPU) and the like) cannot be obtained, and the noise estimation module also needs about 160GMAC calculation amount, which is far beyond the calculation capacity of the current mobile phone. Therefore, there is a need to design a full-scene, high-quality and efficient noise estimation AI model for noise of real images and videos, so as to improve the accuracy of the noise estimation results of real hand images and videos.

Disclosure of Invention

The embodiment of the application provides a noise estimation method, a device, a storage medium and equipment, which are beneficial to overcoming the defects of the existing noise estimation method, reducing estimation errors and improving the accuracy of a noise estimation result of a real hand image and a video.

In a first aspect, the present application provides a noise estimation method, including: when noise estimation is carried out, firstly, dividing N frames of images to be estimated into a plurality of image blocks with fixed sizes; discrete Cosine Transform (DCT) is carried out on each image block to obtain a low-frequency component, an intermediate-frequency component and a high-frequency component; n is a positive integer greater than 1, then calculating a pixel estimation value of a low-frequency component, calculating an optimal sub-threshold corresponding to a medium-frequency component, correcting the power spectral density of a high-frequency component, correcting an ISP (Internet service provider) path of an image signal processor of the high-frequency component to obtain an initial noise estimation value, and further determining a final noise estimation result of the N frames of images to be estimated according to the pixel estimation value and the initial noise estimation value of the low-frequency component of the N frames of images.

Compared with the prior art, in the embodiment of the application, when noise estimation is carried out on videos and images, signals and noise are adaptively separated through DCT coefficients, the number of sample blocks for estimation is increased by utilizing multi-frame information, and meanwhile, color noise and ISP channels are corrected, so that errors of noise estimation can be reduced, and in particular, in dark areas and overexposure areas, the calculation mode is simple and efficient, and the accuracy of noise estimation results of real hand images and videos is improved.

In a possible implementation manner, after determining a final noise estimation result of the N frame image to be estimated according to the pixel estimation value and the initial noise estimation value of the low frequency component of the N frame image, the method further includes: performing model fitting by using the sensitivity ISO value, the pixel estimation value and the initial noise estimation value of the camera to obtain a relational expression of a fitting model; and calculating parameters of the fitting model by using a least square method to obtain a pixel estimated value and a full scene noise intensity curved surface corresponding to the initial noise estimated value, and taking the pixel estimated value and the full scene noise intensity curved surface as a final fitting result.

In a possible implementation, calculating a pixel estimate of a low frequency component includes: dividing a pixel value range into K intervals according to the low-frequency component; wherein K is a positive integer greater than 0; and calculating the average value of all DCT low-frequency components in each interval to obtain the pixel estimation value of the area.

In a possible implementation, correcting the power spectral density of the high frequency component includes: and correcting the power spectral density of the high-frequency component through collaborative filtering.

In one possible implementation, the image signal processor ISP path correction for the high frequency component results in an initial noise estimate, comprising: and correcting the high-frequency component by using the ISP channel parameter, and calculating the median of the corrected high-frequency component as an initial noise estimated value.

In a second aspect, the present application further provides a noise estimation apparatus, including: a dividing unit for dividing the N frame image to be estimated into a plurality of image blocks of fixed size; discrete cosine transform is carried out on each image block to obtain a low-frequency component, an intermediate-frequency component and a high-frequency component; wherein N is a positive integer greater than 1; a first calculation unit for calculating a pixel estimation value of the low frequency component; the second calculation unit is used for calculating an optimal sub-threshold corresponding to the intermediate frequency component; the correction unit is used for correcting the power spectral density of the high-frequency component and correcting an ISP (Internet service provider) channel of an image signal processor of the high-frequency component to obtain an initial noise estimation value; and the determining unit is used for determining a final noise estimation result of the N frame images to be estimated according to the pixel estimation value of the low-frequency component of the N frame images and the initial noise estimation value.

In a possible implementation manner, the apparatus further includes: the fitting unit is used for performing model fitting by using the sensitivity ISO value, the pixel estimated value and the initial noise estimated value of the camera to obtain a relational expression of a fitting model; the obtaining unit is used for calculating parameters of the fitting model by using a least square method to obtain a pixel estimated value and a full scene noise intensity curved surface corresponding to the initial noise estimated value, and the full scene noise intensity curved surface is used as a final fitting result.

In a possible implementation manner, the first computing unit includes: a dividing subunit, configured to divide the pixel value range into K intervals according to the low-frequency component; wherein K is a positive integer greater than 0; and the obtaining subunit is used for calculating the average value of all DCT low-frequency components in each interval to obtain the pixel estimated value of the area.

In a possible implementation manner, the correction unit is specifically configured to: and correcting the power spectral density of the high-frequency component through collaborative filtering.

In a possible implementation manner, the correction unit is specifically further configured to: and correcting the high-frequency component by using the ISP channel parameter, and calculating the median of the corrected high-frequency component as an initial noise estimated value.

In a third aspect, the present application also provides a noise estimation apparatus, including: a memory, a processor;

a memory for storing instructions; a processor for executing instructions in memory, performing the method of the first aspect and any one of its possible implementations.

In a fourth aspect, the present application also provides a computer readable storage medium comprising instructions which, when run on a computer, cause the computer to perform the method of the first aspect and any one of its possible implementations.

From the above technical solutions, the embodiments of the present application have the following advantages:

in the embodiment of the application, when noise estimation is performed, firstly, an N-frame image to be estimated is divided into a plurality of image blocks with fixed sizes; discrete Cosine Transform (DCT) is carried out on each image block to obtain a low-frequency component, an intermediate-frequency component and a high-frequency component; n is a positive integer greater than 1, then calculating a pixel estimation value of a low-frequency component, calculating an optimal sub-threshold corresponding to a medium-frequency component, correcting the power spectral density of a high-frequency component, correcting an ISP (Internet service provider) path of an image signal processor of the high-frequency component to obtain an initial noise estimation value, and further determining a final noise estimation result of the N frames of images to be estimated according to the pixel estimation value and the initial noise estimation value of the low-frequency component of the N frames of images. Therefore, when the noise estimation is performed on the video and the image, the embodiment of the application adaptively separates the signal and the noise through the DCT coefficient, utilizes multi-frame information to increase the number of sample blocks used for estimation, and simultaneously corrects the color noise and the ISP channel, so that the error of noise estimation can be reduced, and the calculation mode is simple and efficient, especially in a dark area and an overexposure area, and is beneficial to improving the accuracy of the noise estimation result of the real hand image and the video.

Drawings

FIG. 1 is a schematic structural diagram of an artificial intelligence main body framework according to an embodiment of the present application;

fig. 2 is one of application scenario diagrams in the embodiment of the present application;

FIG. 3 is a second application scenario diagram according to an embodiment of the present application;

FIG. 4 is a third application scenario diagram according to an embodiment of the present disclosure;

fig. 5 is a flowchart of a noise estimation method according to an embodiment of the present application;

FIG. 6 is a schematic diagram illustrating a method for estimating noise associated with the present embodiment;

FIG. 7 is a schematic diagram comparing the actual calibration noise provided by the embodiment of the present application;

FIG. 8 is a schematic diagram illustrating a comparison of the visual effects of the denoising network according to an embodiment of the present application;

FIG. 9 is a second comparison diagram of the denoising network visual effect provided in the embodiment of the present application;

FIG. 10 is a third comparative schematic diagram of a denoising network visual effect provided according to an embodiment of the present application;

FIG. 11 is a schematic diagram of a noise data synthesis effect provided in an embodiment of the present application;

fig. 12 is a block diagram of a noise estimation device according to an embodiment of the present application;

fig. 13 is a schematic structural diagram of a noise estimation apparatus according to an embodiment of the present application.

Detailed Description

The embodiment of the application provides a noise estimation method, a device, a storage medium and equipment, which can reduce noise estimation errors and improve the accuracy of noise estimation results of real hand images and videos.

Embodiments of the present application are described below with reference to the accompanying drawings. As one of ordinary skill in the art can appreciate, with the development of technology and the appearance of new scenes, the technical solutions provided in the embodiments of the present application are applicable to similar technical problems.

Referring to fig. 1, a schematic structural diagram of an artificial intelligence main body framework is shown in fig. 1, and the artificial intelligence main body framework is described below from two dimensions of "intelligent information chain" (horizontal axis) and "IT value chain" (vertical axis). Where the "intelligent information chain" reflects a list of processes from the acquisition of data to the processing. For example, there may be general procedures of intelligent information awareness, intelligent information representation and formation, intelligent reasoning, intelligent decision making, intelligent execution and output. In this process, the data undergoes a "data-information-knowledge-wisdom" gel process. The "IT value chain" reflects the value that artificial intelligence brings to the information technology industry from the underlying infrastructure of personal intelligence, information (provisioning and processing technology implementation), to the industrial ecological process of the system.

(1) Infrastructure of

The infrastructure provides computing capability support for the artificial intelligence system, realizes communication with the outside world, and realizes support through the base platform. Communicating with the outside through the sensor; the computing power is provided by a smart chip (CPU, NPU, GPU, ASIC, FPGA and other hardware acceleration chips); the basic platform comprises a distributed computing framework, a network and other relevant platform guarantees and supports, and can comprise cloud storage, computing, interconnection and interworking networks and the like. For example, the sensor and external communication obtains data that is provided to a smart chip in a distributed computing system provided by the base platform for computation.

(2) Data

The data of the upper layer of the infrastructure is used to represent the data source in the field of artificial intelligence. The data relate to graphics, images, voice and text, and also relate to the internet of things data of the traditional equipment, including service data of the existing system and sensing data such as force, displacement, liquid level, temperature, humidity and the like.

(3) Data processing

Data processing typically includes data training, machine learning, deep learning, searching, reasoning, decision making, and the like.

Wherein machine learning and deep learning can perform symbolized and formalized intelligent information modeling, extraction, preprocessing, training and the like on data.

Reasoning refers to the process of simulating human intelligent reasoning modes in a computer or an intelligent system, and carrying out machine thinking and problem solving by using formal information according to a reasoning control strategy, and typical functions are searching and matching.

Decision making refers to the process of making decisions after intelligent information is inferred, and generally provides functions of classification, sequencing, prediction and the like.

(4) General capability

After the data has been processed, some general-purpose capabilities can be formed based on the result of the data processing, such as algorithms or a general-purpose system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

(5) Intelligent product and industry application

The intelligent product and industry application refers to products and applications of an artificial intelligent system in various fields, is encapsulation of an artificial intelligent overall solution, and realizes land application by making intelligent information decisions, and the application fields mainly comprise: intelligent terminal, intelligent traffic, intelligent medical treatment, automatic driving, safe city, etc.

The method and the device can be applied to the field of image processing in the field of artificial intelligence, and an application scene of landing to a product is described below.

The noise estimation process applied to the computing equipment such as the terminal equipment, the cloud product and the like is as follows:

the noise estimation method provided by the embodiment of the application can be applied to image processing processes in computing equipment such as terminal equipment and cloud products, and particularly can be applied to cameras and the like on the terminal equipment. Referring to fig. 2, fig. 3, and fig. 4, which are schematic application scenarios of the embodiments of the present application, as shown in fig. 2, the terminal device is provided with an AI system for implementing an image processing function, such as a metagraph camera installed in a mobile phone. The non-parameter estimation model flow chart based on the power spectrum density can be used for firstly cutting a group of multi-frame images to be estimated, carrying out DCT conversion on each image block, dividing DCT coefficients into high, medium, low frequency and high frequency components, and then processing the DCT high, medium and low frequency coefficients in parallel. The separated high-frequency component passes through a power spectrum correction and ISP (Internet service provider) channel correction module to obtain standard deviation statistics of noise, and the low-frequency component passes through a pixel value interval division module to finally obtain DCT (discrete cosine transform) block positions and pixel values of each interval. Meanwhile, the intermediate frequency component judges whether the DCT block is used as an estimated sampling point through the self-adaptive threshold frequency dividing module, and the influence of texture and structure information is removed on the basis of reserving the sampling quantity as much as possible, so that the offset of an estimation result is reduced. Finally, a one-dimensional curve of pixel values and noise standard deviation is drawn. Next, as shown in fig. 3, after noise estimation is performed for different shooting parameters, the relation between the estimated noise curve and the shooting parameters is further fitted by using ISO values corresponding to each group of images, so as to obtain a final noise estimation result of the whole scene, and the final noise estimation result is used as a final fitting result. Furthermore, as shown in fig. 4, on the one hand, for the denoising network, the estimated noise model can be directly used as the network input, and the network is introduced to enable the network to obtain the prior information of the noise, so that the generalization of the model is improved; on the other hand, based on the noise model after fitting, a noisy image can be generated by using the clean image, training data is supplemented, and the network effect is improved. In fig. 4, a noisy video of N ISO scenes is input, noise estimation is performed on a multi-frame image, then N noise curves are output, and then a Minimum Square Error (MSE) can be used to fit to obtain a curved surface of the noise intensity of the full scene.

The terminal equipment can be a mobile phone, a tablet, a notebook computer, an intelligent wearable device and the like, and the terminal equipment can perform noise estimation processing on each acquired multi-frame image. It should be understood that the embodiments of the present application may also be applied to other scenarios where noise estimation is required, and no one-to-one enumeration is given here for other application scenarios.

Based on the application scenario, the embodiment of the application provides a noise estimation method, which can be applied to weak computing equipment such as terminal equipment or high-performance computing equipment such as cloud products. As shown in fig. 5, the method includes:

s501: dividing N frames of images to be estimated into a plurality of image blocks with fixed sizes; discrete cosine transform is carried out on each image block to obtain a low-frequency component, an intermediate-frequency component and a high-frequency component; wherein N is a positive integer greater than 1.

In this embodiment, the N-frame image to be estimated may be image data (such as a scenery image captured by a user) captured by the terminal device through an image capturing device (such as a camera), or may be image data stored previously obtained from inside the terminal device. The specific acquisition mode and the specific source of the N frames of images to be estimated are not limited, and the N frames of images to be estimated can be selected according to actual conditions.

Further, after acquiring the N frames of images to be estimated, the terminal device may divide the N frames of images into a plurality of image blocks with fixed sizes; and discrete cosine transforming (iscrete cosine transformation, DCT) is performed on each image block to obtain a low frequency component, an intermediate frequency component, and a high frequency component. An alternative implementation may perform a spatial loop operation on the image sequence to be estimated (defined herein as I), assuming that patch_size is [ n, n ], truncating the image blocks in step size n_step, e.g., dividing h×w×f I into n×n×1 image blocks, where the first image block contains pixels with corresponding I coordinates of [0:n-1, 0]; if the image sequence to be estimated is the original Raw domain image sequence, the image is required to be subjected to a shuffle operation, and each color component is processed respectively. Next, the divided image block may be subjected to discrete cosine transform, and the transformed DCT coefficient position is denoted as (i, j), and the frequency component, the intermediate frequency component, and the high frequency component are divided by the value of i+j to perform the subsequent steps S502 to S505.

Specifically, first, a sequence of images to be estimated may be defined as I _{f∈[0,1,…,F]} The resolution of each frame image is [ H, W ]]And F frames are shared. Then, the image is diced, the size of the image block is [ n ] _patch ,n _patch ]Slice interval is n _slide Wherein the (I, j) th image block contains the corresponding I of the pixel _f The middle coordinates are [ i ] (n _slide -1):i*(n _slide -1)+n _patch -1,j*(n _slide -1):j*(n _slide -1)+n _patch -1]. Then, DCT is carried out on the image block to obtain a low-frequency component D _L Intermediate frequency component D _M And height ofFrequency component D _H By DCT coefficient position (i _dct ,j _dct ) Division D _L ，D _M And D _H For example n _patch =8, then i will _dct +j _dct Set to low frequency component D =1 _L Will 1<i _dct +j _dct Less than or equal to 8 as intermediate frequency component D _M Will be 8<i _dct +j _dct Less than or equal to 16 is set as a high-frequency component D _H To perform the following steps S502-S505.

S502: a pixel estimate of the low frequency component is calculated.

In this embodiment, after the low-frequency component is obtained in step S501, the pixel value range may be further divided into K intervals according to the low-frequency component; wherein K is a positive integer greater than 0, and calculates the average value of all DCT low frequency components in each interval to obtain the pixel estimation value of the area for executing the subsequent steps.

Specifically, it can be based on the low frequency component D _L Dividing the pixel value range into K intervals, i.e. D _L Dividing into K segments, and setting the pixel value range as [0,1]Then the kth _i The pixel value ranges of the intervals correspond toBy calculating the average value of all DCT low frequency components per interval (i.e. calculating +.>Average value of (d) to obtain a pixel estimate of the region, which is defined herein as +. >And gets its location information +.>For processing the medium and high frequency components.

S503: and calculating an optimal sub-threshold corresponding to the intermediate frequency component.

In the present embodiment, after the intermediate frequency component is obtained in step S501For the kth _i Each interval is provided with its shareThe DCT blocks are used for calculating the average value of intermediate frequency components of the DCT blocks, sorting the intermediate frequency components according to ascending order of the values, taking the value of the mth block as a division threshold value delta p, iterating through a preset algorithm to finally obtain an optimal division threshold value p, and conforming to the value ++>Is->The DCT blocks perform subsequent noise value estimation, and the preset algorithm is specifically as follows:

Adaptive-Threshold Frequency Component Division

Input:D＝Set of N×N DCT blocks

Output:Chosen D _m for noise estimation

1:for each pixel value bins:

2:Initialize p＝0.8andΔp＝0.05

3:Compute

4:Compute

5:whileand/>and/>

6:Set p＝p-△p

7:Update

8:while end

9:

10:for end。

s504: and correcting the power spectral density of the high-frequency component and correcting an ISP (Internet service provider) path of an image signal processor of the high-frequency component to obtain an initial noise estimated value.

Note that, the actual noise is not white noise, but has spatial correlation, and the frequency domain shows a change in power spectral density, so that an offset may be caused to the estimation result. Thus, an alternative implementation manner is that after the high-frequency component is obtained in step S501, the power spectral density of the high-frequency component may be corrected by collaborative filtering according to the power spectral density stability inconsistency of the color noise in the space domain and the time domain. Wherein, the collaborative filtering can adopt a wiener filter.

In particular, for the kth _i Interval, getCoordinates of->Obtain->All of the high frequency components of (a)Calculate->The ratio of the spatial standard deviation to the time domain standard deviation is used as a wiener filter coefficient to carry out cooperative filtering so as to correct the power spectrum density of the DCT block. The specific formula for calculating the wiener filter coefficients is as follows:

wherein,represents the kth _i The number of DCT blocks whose interval is used for noise estimation; f represents the number of frames of the image;

then, utilizeFor high frequency components->Correcting; finally, calculating the high-frequency standard deviation of all frames to obtain the noise standard deviation after power spectrum correction>

Further, taking into consideration the influence of other processing modules in the ISP on the noise intensity and the characteristics, obtaining the noise standard deviation after the power spectrum correctionThereafter, the ISP channel parameter may be further used to correct the ISP channel parameter, and calculate the median of the corrected high frequency component as the initial noise estimation value, so as to perform the subsequent step S505.

Specifically, LSC Metric and AWB Gain may be extracted from the ISP path, both remodelled reshape to the input sizeMultiplying and utilizing the reciprocal and noise standard deviation after power spectrum correction>Dot multiplication is performed to obtain a corrected result +. >And then each interval k can be calculated _i Is defined herein as +.>

S505: and determining a final noise estimation result of the N frame images to be estimated according to the pixel estimation value of the low-frequency component of the N frame images and the initial noise estimation value.

In the present embodiment, in the pixel estimation value of the low frequency component of the N-frame image obtained by the above stepsAnd initial noise estimate +.>After that, it is further possible to add +_for each frame belonging to the same section>And->Averaging is performed as a final noise estimation result, and a relation curve (shown in a graph on the far right side of fig. 2) between the two is drawn as a final result of an estimation model. However, since the estimation result is a discrete value, no +.>And->Further using linear interpolation estimation to determine the final noise estimation result of the N frames of images to be estimated.

Further, in an alternative implementation manner, the relation between the non-parametric noise estimation result and the shooting parameters can be modeled, so that the full scene noise fitting is realized, and the method can be used for supporting the noise estimation of the mobile phone end real-time 4K video. Specifically, after the final noise estimation result of the N frames of images is obtained, model fitting may be performed by using the sensitivity ISO value, the pixel estimation value and the initial noise estimation value of the camera, to obtain a relational expression of the fitting model, where the specific relational expression is as follows:

λ _a ＝a ₁ ×ISO ² +a ₂ ×ISO+a ₃

λb＝b ₁ ×ISO ² +b ₂ ×ISO+b ₃

Then, the least square method can be utilized to calculate the parameters of the fitting model to obtain the estimated value of the pixelAnd initial noise estimate +.>The corresponding full scene noise intensity surface (as shown in the right-most diagram of fig. 3), which is defined herein as σ (Y, iso|β) as the final fitting result. Wherein, beta represents a model parameter, and the specific value is beta= [ a ] _i ,b _i ]。

Therefore, on the design of the noise estimation integral model, a non-parametric estimation model is adopted, the problem of failure of the poisson-Gaussian model in a dark area and an overexposure area is avoided, the relation between a pixel value and noise intensity is directly estimated, and estimation errors are remarkably reduced. In addition, as multi-frame images are introduced, estimated sampling points are increased, and estimation accuracy is improved; and meanwhile, the power spectrum of the image block is corrected by utilizing the correlation difference of the color noise in space and time, so that the estimation effect of the color noise is greatly improved. In addition, the application designs a self-adaptive threshold frequency dividing module, which utilizes the intermediate frequency information of the image to effectively eliminate the interference caused by structures, textures and the like in the image, and reduces the estimation error. And when ISP is corrected, correction matrixes such as LSC, AWB and the like are introduced to correct the influence of each link in the mobile phone on the noise characteristics, so that the method is simple and effective. And finally, modeling by using the estimation result and the shooting parameters of the camera, fitting the model parameters, and supporting the full shooting parameters and accurate parameter estimation under the scene. Compared with other related noise estimation methods, the scheme of the application has better technical effects, and is specifically as follows:

(1) Compared with other related noise estimation methods, the method has better performance and more accurate estimation result.

As shown in fig. 6, for Raw data acquired by the Sensor (i.e., data in columns where the Sensor in fig. 6 is located) and Raw data fused by long, medium and short exposure (i.e., data in columns where the HDR in fig. 6 is located), the estimation method provided in the present application exceeds the related Noise estimation methods such as White Noise, PG Noise, noise Flow, and DCT on the average error of the division map (precision-quantile plot error, Q-Q error) and the relative entropy (kullback-leibler divergence, KLD) (i.e., KL-diversity in fig. 6). That is, the smaller the values of the column in which the Sensor and the column in which the HDR are in fig. 6 are, the smaller the error of the estimation is, and the higher the quality of the estimation result is. Meanwhile, the estimation method provided by the application can obtain excellent estimation results on different ISP positions and different ISO scenes for noise with different characteristics and distribution, and detailed description is omitted here.

(2) The difference is very small compared to the real calibration noise.

As shown in fig. 7, when comparing the noise estimation results of the full Raw domain positions, including Sensor Raw and HDR, ATR, BLC, LSC, AWB, GCD output Raw of different exposures, the noise estimation results of the present application represented by discrete points in the graph are obtained, and the dashed line represents the statistical calibration result, so that the difference between the estimated value and the true value can be seen to be very small, thereby indicating that the present application can effectively reflect the real noise level.

(3) The denoising network visual effect is better.

For a large model network, a noise variance diagram is calculated according to a specific image pixel value and a noise model, the noise variance diagram is input into a denoising network together with an input image, and compared with a blind denoising mode, the comparison result of the performance and the calculated amount of blind denoising is shown in fig. 8, compared with other related estimation methods, the estimation method provided by the application can improve the peak signal-to-noise ratio (PSNR) by about 0.46dB, and meanwhile, only 3.34GMAC calculated amount is introduced. For a 4K image with the size of 2160×4096, the actual operation time is only 0.8ms under the condition of GPU V100 hardware, and algorithm instantaneity is hardly affected. Meanwhile, texture and structure information in the result can be obviously improved, and color noise and artifacts (artifacts) are effectively restrained.

For a lightweight network, in order to meet the computational power constraint of a weak computational power terminal device, obtaining a small-computational-power network with good denoising effect is a final target. The validity of the noise estimation method in the present application can be verified on Video JDD (joint denoising and demosaicing) of 15 GMAC. The JDD network is introduced in the same manner as the large model, and the comparison result of the performance and the calculation amount is shown in fig. 9, and it can be seen from fig. 9 that, compared with other related estimation methods, the estimation method provided in the application can raise the PSNR by about 0.55dB, and only 0.68GMAC is added for additional calculation. In addition, as shown in fig. 10 (in the two diagrams of fig. 10, the left side of each diagram represents a blind denoising mode, the middle represents a mode of introducing ISO information, and the right side represents a method of the application), under the 5lux illumination condition, compared with the mode of blind denoising and introducing ISO information, the estimation method provided by the application can enable the JDD network to recover more detail textures, less color noise, more stable high-frequency details and stronger weak contrast textures.

(4) The noise data synthesis effect is good.

Based on the scheme of the application, through adding random noise on the clean image, the noise data of the Mate20pro mobile phone can be synthesized, as shown in fig. 11, the left graph in fig. 11 is real noise, the right graph is noise estimated through the application, and the noise data is almost indiscriminate and strong in consistency, so that the noise data synthesis effect of the application is good. Therefore, the scheme of the application can be utilized to effectively amplify the training data set, synthesize noise data, greatly reduce the cost overhead of data acquisition, effectively make up the deficiency of real training data and be used for improving the accuracy of subsequent image processing operation.

In summary, in the noise estimation method provided in this embodiment, when noise estimation is performed, first, N frames of images to be estimated are divided into a plurality of image blocks with a fixed size; discrete Cosine Transform (DCT) is carried out on each image block to obtain a low-frequency component, an intermediate-frequency component and a high-frequency component; n is a positive integer greater than 1, then calculating a pixel estimation value of a low-frequency component, calculating an optimal sub-threshold corresponding to a medium-frequency component, correcting the power spectral density of a high-frequency component, correcting an ISP (Internet service provider) path of an image signal processor of the high-frequency component to obtain an initial noise estimation value, and further determining a final noise estimation result of the N frames of images to be estimated according to the pixel estimation value and the initial noise estimation value of the low-frequency component of the N frames of images. Therefore, when the noise estimation is performed on the video and the image, the embodiment of the application adaptively separates the signal and the noise through the DCT coefficient, utilizes multi-frame information to increase the number of sample blocks used for estimation, and simultaneously corrects the color noise and the ISP channel, so that the error of noise estimation can be reduced, and the calculation mode is simple and efficient, especially in a dark area and an overexposure area, and is beneficial to improving the accuracy of the noise estimation result of the real hand image and the video.

In order to better understand the above noise estimation method provided in the embodiments of the present application, the above noise estimation method will be described by taking RAW domain image data with a resolution of [1080,1920] and a total frame number f=7 as an example.

Specifically, first, the image data may be subjected to a shuffle operation to obtain a size of [540,960,28 ]]Setting n _patch ＝n _slide =8. Then, set i _dct +j _dct ＝1As low frequency component D _L ，1<i _dct +j _dct Less than or equal to 8 is an intermediate frequency component D _M ，8<i _dct +j _dct Less than or equal to 16 is a high-frequency component D _H . Next, D is carried out _L Dividing into k=500 segments; meanwhile, dividing the threshold value delta p=0.005, and obtaining the optimal sub-threshold value of each interval through a preset algorithmAnd, for the kth _i Interval, get M _p Coordinates of->Obtaining M _p Is +.>Calculate->The ratio of the spatial standard deviation to the temporal standard deviation modifies the power spectral density of the DCT block. Further, utilize->For high frequency components->Correcting; still further, the high frequency standard deviation of all frames is calculated to obtain the noise standard deviation after the power spectrum correction>Further, LSC Metric and AWBGain are extracted, and both are reshaped to [540/n ] _patch ,960/n _patch ]LSC Metric and +/for each corresponding spatial position>Dot product, AWBGain is then associated with +. >Dot multiplication; calculating each interval k after correction _i Is used as the median of the high frequency component of the noise estimate +.>Finally, k in the 7 frames _i Zone>And->And respectively averaging to obtain a final estimation result. On the basis of this, the sensitivity ISO value, pixel estimation value of the camera can be utilized>And initial noise estimate +.>Performing model fitting to obtain a relational expression of a fitting model, and solving parameters in the fitting model by using actual data through a least square method to finally obtain sigma (Y, ISO|beta) (shown in the rightmost graph of fig. 3), wherein beta represents model parameters, and the specific value is beta= [ a ] _i ,b _i ]。

In order to facilitate better implementation of the above-described aspects of the embodiments of the present application, the following further provides related devices for implementing the above-described aspects. Referring to fig. 12, a noise estimation apparatus 1200 is provided in an embodiment of the present application. The apparatus 1200 may include: a dividing unit 1201, a first calculating unit 1202, a second calculating unit 1203, a correcting unit 1204, and a determining unit 1205. Wherein the dividing unit 1201 is for supporting the apparatus 1200 to perform S501 in the embodiment shown in fig. 5. The first calculation unit 1202 is configured to support the apparatus 1200 to execute S502 in the embodiment shown in fig. 5. The second calculating unit 1203 is configured to support the apparatus 1200 to perform S503 in the embodiment shown in fig. 5. The correction unit 1204 is for supporting the apparatus 1200 to perform S504 in the embodiment shown in fig. 5. The determination unit 1205 is used to support the apparatus 1200 to execute S505 in the embodiment shown in fig. 5. In particular, the method comprises the steps of,

A dividing unit 1201 for dividing the N frame image to be estimated into a plurality of image blocks of a fixed size; performing discrete cosine transform on each image block to obtain a low-frequency component, an intermediate-frequency component and a high-frequency component; the N is a positive integer greater than 1;

a first calculation unit 1202 for calculating a pixel estimation value of a low frequency component;

a second calculating unit 1203, configured to calculate an optimal sub-threshold corresponding to the intermediate frequency component;

a correction unit 1204, configured to correct a power spectral density of the high-frequency component, and correct an ISP path of an image signal processor of the high-frequency component, to obtain an initial noise estimation value;

a determining unit 1205 is configured to determine a final noise estimation result of the N frame image to be estimated according to the pixel estimation value and the initial noise estimation value of the low frequency component of the N frame image.

In an implementation manner of this embodiment, the apparatus further includes:

the fitting unit is used for performing model fitting by using the sensitivity ISO value, the pixel estimated value and the initial noise estimated value of the camera to obtain a relational expression of a fitting model;

the obtaining unit is used for calculating parameters of the fitting model by using a least square method to obtain a full scene noise intensity curved surface corresponding to the pixel estimated value and the initial noise estimated value, and the full scene noise intensity curved surface is used as a final fitting result.

In one implementation of the present embodiment, the first computing unit 1202 includes:

a dividing subunit, configured to divide the pixel value range into K intervals according to the low-frequency component; wherein K is a positive integer greater than 0;

and the obtaining subunit is used for calculating the average value of all DCT low-frequency components in each interval to obtain the pixel estimated value of the area.

In one implementation of this embodiment, the correction unit 1204 is specifically configured to:

the power spectral density of the high frequency component is modified by collaborative filtering.

In one implementation of this embodiment, the correction unit 1204 is specifically further configured to:

and correcting the high-frequency component by using the ISP channel parameter, and calculating the median of the corrected high-frequency component as an initial noise estimated value.

In summary, in the noise estimation device provided in this embodiment, when noise estimation is performed, first, N frames of images to be estimated are divided into a plurality of image blocks with a fixed size; discrete Cosine Transform (DCT) is carried out on each image block to obtain a low-frequency component, an intermediate-frequency component and a high-frequency component; n is a positive integer greater than 1, then calculating a pixel estimation value of a low-frequency component, calculating an optimal sub-threshold corresponding to a medium-frequency component, correcting the power spectral density of a high-frequency component, correcting an ISP (Internet service provider) path of an image signal processor of the high-frequency component to obtain an initial noise estimation value, and further determining a final noise estimation result of the N frames of images to be estimated according to the pixel estimation value and the initial noise estimation value of the low-frequency component of the N frames of images. Therefore, when the noise estimation is performed on the video and the image, the embodiment of the application adaptively separates the signal and the noise through the DCT coefficient, utilizes multi-frame information to increase the number of sample blocks used for estimation, and simultaneously corrects the color noise and the ISP channel, so that the error of noise estimation can be reduced, and the calculation mode is simple and efficient, especially in a dark area and an overexposure area, and is beneficial to improving the accuracy of the noise estimation result of the real hand image and the video.

Referring to fig. 13, an embodiment of the present application provides a noise estimation device 1300, comprising a memory 1301, a processor 1302 and a communication interface 1303,

a memory 1301 for storing instructions;

a processor 1302, configured to execute the instructions in the memory 1301 and perform the noise estimation method applied to the embodiment shown in fig. 5;

a communication interface 1303 for performing communication.

The memory 1301, the processor 1302, and the communication interface 1303 are connected to each other through a bus 1304; bus 1304 may be a peripheral component interconnect standard (peripheral component interconnect, PCI) bus, or an extended industry standard architecture (extended industry standard architecture, EISA) bus, among others. The buses may be divided into address buses, data buses, control buses, etc. For ease of illustration, only one thick line is shown in fig. 13, but not only one bus or one type of bus.

In a specific embodiment, the processor 1302 is configured to divide the N frame image to be estimated into a plurality of image blocks with a fixed size when performing noise estimation; discrete Cosine Transform (DCT) is carried out on each image block to obtain a low-frequency component, an intermediate-frequency component and a high-frequency component; n is a positive integer greater than 1, then calculating a pixel estimation value of a low-frequency component, calculating an optimal sub-threshold corresponding to a medium-frequency component, correcting the power spectral density of a high-frequency component, correcting an ISP (Internet service provider) path of an image signal processor of the high-frequency component to obtain an initial noise estimation value, and further determining a final noise estimation result of the N frames of images to be estimated according to the pixel estimation value and the initial noise estimation value of the low-frequency component of the N frames of images. For details of the processing procedure of the processor 1302, please refer to the detailed descriptions of S501, S502, S503, S504 and S505 in the embodiment shown in fig. 5, which are not described herein.

The memory 1301 may be random-access memory (RAM), flash memory (flash), read-only memory (ROM), erasable programmable read-only memory (erasable programmable read only memory, EPROM), electrically erasable programmable read-only memory (electrically erasable programmable read only memory, EEPROM), registers (registers), hard disk, a removable disk, a CD-ROM, or any other form of storage medium known to those skilled in the art.

The processor 1302 may be, for example, a central processing unit (central processing unit, CPU), a general purpose processor, a digital signal processor (digital signal processor, DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (field programmable gate array, FPGA) or other programmable logic device, a transistor logic device, a hardware component, or any combination thereof. Which may implement or perform the various exemplary logic blocks, modules and circuits described in connection with the disclosure of embodiments of the present application. A processor may also be a combination that performs computing functions, e.g., including one or more microprocessors, a combination of a DSP and a microprocessor, and so forth.

The communication interface 1303 may be, for example, an interface card, an ethernet (ethernet) interface, or an asynchronous transfer mode (asynchronous transfer mode, ATM) interface.

Embodiments of the present application also provide a computer-readable storage medium comprising instructions that, when executed on a computer, cause the computer to perform the above-described noise estimation method.

The terms first, second and the like in the description and in the claims of the present application and in the above-described figures, are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the terms so used are interchangeable under appropriate circumstances and are merely illustrative of the manner in which the embodiments of the application described herein have been described for objects of the same nature. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of elements is not necessarily limited to those elements, but may include other elements not expressly listed or inherent to such process, method, article, or apparatus.

It will be clear to those skilled in the art that, for convenience and brevity of description, specific working procedures of the above-described systems, apparatuses and units may refer to corresponding procedures in the foregoing method embodiments, which are not repeated herein.

In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods may be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative, e.g., the division of the units is merely a logical function division, and there may be additional divisions when actually implemented, e.g., multiple units or components may be combined or integrated into another system, or some features may be omitted or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be an indirect coupling or communication connection via some interfaces, devices or units, which may be in electrical, mechanical or other form.

The units described as separate units may or may not be physically separate, and units shown as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit. The integrated units may be implemented in hardware or in software functional units.

The integrated units, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application may be embodied in essence or a part contributing to the prior art or all or part of the technical solution in the form of a software product stored in a storage medium, including several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a magnetic disk, or an optical disk, or other various media capable of storing program codes.

The above embodiments are merely for illustrating the technical solution of the present application, and not for limiting the same; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the corresponding technical solutions.

Claims

1. A method of noise estimation, the method comprising:

dividing N frames of images to be estimated into a plurality of image blocks with fixed sizes; performing discrete cosine transform on each image block to obtain a low-frequency component, an intermediate-frequency component and a high-frequency component; the N is a positive integer greater than 1;

calculating a pixel estimation value of the low frequency component;

calculating an optimal sub-threshold corresponding to the intermediate frequency component;

correcting the power spectral density of the high-frequency component and correcting an ISP (Internet service provider) channel of an image signal processor of the high-frequency component to obtain an initial noise estimated value;

and determining a final noise estimation result of the N frame images to be estimated according to the pixel estimation value of the low-frequency component of the N frame images and the initial noise estimation value.

2. The method according to claim 1, wherein after determining a final noise estimation result of the N-frame image to be estimated from the pixel estimation value of the low-frequency component of the N-frame image and the initial noise estimation value, the method further comprises:

performing model fitting by using the sensitivity ISO value of the camera, the pixel estimated value and the initial noise estimated value to obtain a relational expression of a fitting model;

And calculating parameters of the fitting model by using a least square method to obtain a full scene noise intensity curved surface corresponding to the pixel estimated value and the initial noise estimated value, and taking the full scene noise intensity curved surface as a final fitting result.

3. The method of claim 1, wherein said calculating a pixel estimate of said low frequency component comprises:

dividing a pixel value range into K intervals according to the low-frequency component; the K is a positive integer greater than 0;

and calculating the average value of all DCT low-frequency components in each interval to obtain the pixel estimation value of the interval.

4. The method of claim 1, wherein said modifying the power spectral density of the high frequency component comprises:

and correcting the power spectral density of the high-frequency component through collaborative filtering.

5. The method of claim 1, wherein said image signal processor ISP path correction for said high frequency component results in an initial noise estimate comprising:

6. A noise estimation device, the device comprising:

A dividing unit for dividing the N frame image to be estimated into a plurality of image blocks of fixed size; performing discrete cosine transform on each image block to obtain a low-frequency component, an intermediate-frequency component and a high-frequency component; the N is a positive integer greater than 1;

a first calculation unit configured to calculate a pixel estimation value of the low frequency component;

the second calculation unit is used for calculating an optimal sub-threshold value corresponding to the intermediate frequency component;

a correction unit, configured to correct a power spectral density of the high-frequency component, and correct an ISP path of an image signal processor of the high-frequency component, so as to obtain an initial noise estimation value;

and the determining unit is used for determining a final noise estimation result of the N frame images to be estimated according to the pixel estimation value of the low-frequency component of the N frame images and the initial noise estimation value.

7. The apparatus of claim 6, wherein the apparatus further comprises:

the fitting unit is used for performing model fitting by using the sensitivity ISO value of the camera, the pixel estimated value and the initial noise estimated value to obtain a relational expression of a fitting model;

8. The apparatus of claim 6, wherein the first computing unit comprises:

a dividing subunit, configured to divide the pixel value range into K intervals according to the low-frequency component; the K is a positive integer greater than 0;

and the obtaining subunit is used for calculating the average value of all DCT low-frequency components in each interval to obtain the pixel estimated value of the interval.

9. The apparatus according to claim 6, wherein the correction unit is specifically configured to:

10. The apparatus according to claim 6, wherein the correction unit is further specifically configured to:

11. A noise estimation device, the device comprising a memory, a processor;

the memory is used for storing instructions;

the processor being configured to execute the instructions in the memory and to perform the method of any of claims 1-5.

12. A computer readable storage medium comprising instructions which, when run on a computer, cause the computer to perform the method of any of the preceding claims 1-5.