CN111145103B - Monte Carlo denoising method based on detail retention neural network model - Google Patents
Monte Carlo denoising method based on detail retention neural network model Download PDFInfo
- Publication number
- CN111145103B CN111145103B CN201911199715.3A CN201911199715A CN111145103B CN 111145103 B CN111145103 B CN 111145103B CN 201911199715 A CN201911199715 A CN 201911199715A CN 111145103 B CN111145103 B CN 111145103B
- Authority
- CN
- China
- Prior art keywords
- color
- denoising
- specular reflection
- layer
- buffer area
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 43
- 238000003062 neural network model Methods 0.000 title claims abstract description 12
- 230000014759 maintenance of location Effects 0.000 title abstract description 5
- 238000005286 illumination Methods 0.000 claims abstract description 52
- 238000012549 training Methods 0.000 claims abstract description 30
- 230000005540 biological transmission Effects 0.000 claims abstract description 22
- 230000006870 function Effects 0.000 claims abstract description 21
- 238000009877 rendering Methods 0.000 claims abstract description 17
- 238000013528 artificial neural network Methods 0.000 claims abstract description 16
- 238000001914 filtration Methods 0.000 claims abstract description 10
- 238000007781 pre-processing Methods 0.000 claims abstract description 8
- 230000008447 perception Effects 0.000 claims abstract description 6
- 230000005855 radiation Effects 0.000 claims description 12
- 208000037170 Delayed Emergence from Anesthesia Diseases 0.000 claims description 10
- 230000008569 process Effects 0.000 claims description 9
- 230000009466 transformation Effects 0.000 claims description 8
- 238000004364 calculation method Methods 0.000 claims description 7
- 238000012795 verification Methods 0.000 claims description 7
- 238000000605 extraction Methods 0.000 claims description 6
- 239000011159 matrix material Substances 0.000 claims description 6
- 238000013527 convolutional neural network Methods 0.000 claims description 5
- 238000005070 sampling Methods 0.000 claims description 4
- 239000013598 vector Substances 0.000 claims description 4
- 230000004913 activation Effects 0.000 claims description 3
- ORILYTVJVMAKLC-UHFFFAOYSA-N Adamantane Natural products C1C(C2)CC3CC1CC2C3 ORILYTVJVMAKLC-UHFFFAOYSA-N 0.000 claims description 2
- 238000001228 spectrum Methods 0.000 claims description 2
- 230000003044 adaptive effect Effects 0.000 abstract description 2
- 230000000694 effects Effects 0.000 description 11
- 238000010586 diagram Methods 0.000 description 3
- 238000010801 machine learning Methods 0.000 description 3
- 238000000844 transformation Methods 0.000 description 3
- 238000011423 initialization method Methods 0.000 description 2
- PEDCQBHIVMGVHV-UHFFFAOYSA-N Glycerine Chemical compound OCC(O)CO PEDCQBHIVMGVHV-UHFFFAOYSA-N 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 230000015556 catabolic process Effects 0.000 description 1
- 238000006731 degradation reaction Methods 0.000 description 1
- 238000012417 linear regression Methods 0.000 description 1
- 239000000463 material Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/70—Denoising; Smoothing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/90—Determination of colour characteristics
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y04—INFORMATION OR COMMUNICATION TECHNOLOGIES HAVING AN IMPACT ON OTHER TECHNOLOGY AREAS
- Y04S—SYSTEMS INTEGRATING TECHNOLOGIES RELATED TO POWER NETWORK OPERATION, COMMUNICATION OR INFORMATION TECHNOLOGIES FOR IMPROVING THE ELECTRICAL POWER GENERATION, TRANSMISSION, DISTRIBUTION, MANAGEMENT OR USAGE, i.e. SMART GRIDS
- Y04S10/00—Systems supporting electrical power generation, transmission or distribution
- Y04S10/50—Systems or methods supporting the power network operation or management, involving a certain degree of interaction with the load-side end user applications
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Image Generation (AREA)
Abstract
The invention discloses a Monte Carlo denoising method based on a detail retention neural network model, which comprises the steps of firstly enabling an output channel of a renderer to comprise a color buffer area and an auxiliary characteristic buffer area, increasing illumination transmission covariance in the auxiliary characteristic buffer area, rendering a large amount of noise data and corresponding reference data to form a training data set, then preprocessing the noise data of the training set, inputting the preprocessed noise data into a neural network comprising a characteristic extractor and a kernel predictor for filtering, and outputting a filtering kernel; denoising noise input by using a filter core output by a neural network, calculating an error between a denoising result and reference data by using a SMAPE (simple wavelet transform and adaptive wavelet transform) and a perception loss function, and training and optimizing parameters of the network; and denoising the data which are in any condition of meeting the network input structure by using the trained network. The invention can keep better geometric details and illumination details of the denoising result.
Description
Technical Field
The invention belongs to the technical field of computer graphic rendering, and particularly relates to a Monte Carlo denoising method based on a detail retention neural network model.
Background
Regarding the monte carlo denoising method, it can be classified into a monte carlo denoising method based on an image space and a monte carlo denoising method based on machine learning. There have been many studies on the image space-based monte carlo denoising method. Roussell et al propose a zero-order linear regression model-based method (F. Roussell, M. Manzi, and M. Zwicker. robust differentiating using feature and color information. computer Graphics Forum, 32(7): 121-. These methods produce good denoising effect by selecting good filter kernel weight parameters, but this makes it limited by an explicit filter, resulting in a filter kernel that is not flexible enough. Then Bitterli proposes a method based on a first order model (B.Bitterli, F.Rousselle, B.Moon, J.A.Igleis-Guiti. n, D.Adler, K.Mitchell, W.Jaross, and J.Nov. k.nonlinear weighted first-order regression for differentiating Monte Carlo reproducing. computer Graphics Forum,35(4): 107-117, 2016.), and Moon proposes a method based on a high order model (B.Moon, S.McDonaven, K.Mitchell, and M.Gros.Adaptive polymeric reproducing. ACM.graphics. graph, page 10,2014.). These methods are less constrained and they make more use of domain data by exploiting the relationship between the assist feature buffer and the color buffer. However, the method based on the first-order model has a poor denoising effect on low-frequency noise, and the method based on the high-order model is prone to the problem of overfitting.
Many relevant works have also emerged in recent years with respect to machine learning-based monte carlo denoising methods. Kalatari first proposed to handle the Monte Carlo denoising problem with neural network methods (n.k. kalatari, s.bako, and p.sen. a machine learning approach for filtering Monte Carlo noise. acm Transformations On Graphics (TOG) (Proceedings of sigwrah 2015),34(4),2015.), they learned the relationship between noise input and ideal filter parameters by using multi-layer perceptual neural networks, and then used the trained models to denoise other new scenes. Bako proposes a convolutional neural network model-based method (s.bako, t.vogels, b.mcwilliams, m.meyer, j.nov a k, a.harville, p.sen, t.derose, and f.rousselle. kernel compressing connecting networks for differentiating Monte Carlo reproducing. ACM Transformations On Graphics (TOG) (Proceedings of SIGRAPH 2017),36(4), July 2017.), abbreviated as KPCN, which outputs a convolutional kernel through a nine-layer convolutional neural network for filtering and denoising noise input; they also propose to divide the input into diffuse and specular components and process them separately with separate convolutional neural networks. The experimental result shows that the KPCN has better effect than the previous Monte Carlo denoising method. Vogels improved on KPCN (t.vogels, f.rousselle, b.mcwilliams, G). 
A. Harville, D.Adler, M.Meyer, and J.Nov k.denoising with kernel prediction and asymmetry loss functions, ACM transformations on Graphics (Proceedings of SIGGRAGH 2018, 37(4):124: 1-124: 15,2018.), which further enhance denoising by combining KPCN, source-aware encoder, and an asymmetric loss function. However, these methods often lose a part of geometric details or illumination details during denoising, or excessively blur some sharp features.
Disclosure of Invention
The invention aims to provide a Monte Carlo denoising method based on a detail-preserving neural network model, which adds an illumination transmission covariance in a path space in an input auxiliary feature buffer area by using a neural network structure comprising a feature extractor and a kernel predictor, and trains a network by increasing a perception loss in a loss function, so that a denoising result can keep better geometric details and illumination details.
The technical solution for realizing the purpose of the invention is as follows: a Monte Carlo denoising method based on a detail-preserving neural network model comprises the following steps:
step one, enabling an output channel of a renderer to comprise a color buffer area and an auxiliary feature buffer area, increasing an illumination transmission covariance in the auxiliary feature buffer area, and rendering a large amount of noise data and corresponding reference data to form a training data set;
Preprocessing noise data of the training data set, converting diffuse reflection color into radiation illumination color, converting specular reflection color into a logarithmic domain, and normalizing data in the auxiliary feature buffer area to obtain preprocessed noise data;
inputting the preprocessed noise data into a neural network comprising a feature extractor and a kernel predictor for filtering, and outputting a filtering kernel;
performing convolution on the radiation illumination color in the diffuse reflection assembly and the logarithmic domain specular reflection color in the specular reflection assembly by using a filter core output by the neural network to obtain a denoised radiation illumination color and a denoised logarithmic domain specular reflection color;
step five, calculating the error between the denoising result and the reference data by using the SMAPE and a perception loss function, training and optimizing parameters of the network, and if the verification result is converged or reaches the maximum iteration number after training, performing the step six, otherwise, returning to the step three;
and step six, denoising the data which are in accordance with the network input structure by using the trained network.
Compared with the prior art, the invention has the following remarkable advantages: (1) the denoising result can keep better illumination details: due to the fact that the illumination propagation covariance is added to the auxiliary feature buffer area, the neural network can learn frequency information of illumination propagation from the auxiliary feature buffer area, and the capability of keeping illumination details is improved, as shown in fig. 5. In fig. 5, the denoising results with and without increasing the illumination propagation covariance are compared, and the error between the denoising result and the corresponding reference data is calculated (the error calculation adopts RelMSE and DSSIM, the lower the value is, the better the effect is). As can be seen from the results in fig. 5, in the case of increasing the covariance of illumination propagation, the error between the denoising result and the reference data is lower, the illumination detail can be better maintained, and a better shape or a smoother can be maintained for some special details formed by complex illumination propagation. (2) The network structure provided promotes the whole denoising effect: the color buffer area and the auxiliary characteristic buffer area are respectively sent to the independent characteristic extractor, so that the network can better respectively extract high-dimensional characteristic information from the color buffer area and the auxiliary characteristic buffer area, network parameters of the color buffer area and the auxiliary characteristic buffer area are separated, and mutual influence cannot be caused during training; the training degradation problem is avoided by using a shallow kernel predictor. Thereby improving the denoising effect, as shown in fig. 6. In fig. 6, the denoising results of KPCN and our network are shown, except for the difference of network structures, the training data and parameter settings are the same, and in fig. 6, the error between the denoising result and the corresponding reference data is also calculated (the error calculation adopts RelMSE and DSSIM, the lower the value is, the better the effect is). As can be seen from FIG. 6, the error between the denoising result of the network and the reference data is lower, the denoising effect at some details is better, and the KPCN has some aliasing. (3) The denoising result can keep better geometric details: the network is trained by adding a perception loss function, so that the retention capability of geometric details is improved, as shown in FIG. 7. In fig. 7, the denoising results of the training with increased perceptual loss and the training without increased perceptual loss are compared, and the error between the denoising result and the corresponding reference data is calculated (the error calculation adopts RelMSE and DSSIM, the lower the value is, the better the effect is illustrated). From the results in fig. 7, it can be seen that without increasing the perceptual loss, some geometric details are blurred, and in the case of training with increasing the perceptual loss, the error between the denoising result and the reference data is lower, and better and sharper geometric details can be maintained.
The present invention is described in further detail below with reference to the attached drawing figures.
Drawings
FIG. 1 is a schematic flow chart of a Monte Carlo denoising method based on a detail-preserving neural network model according to the present invention.
FIG. 2 is a detailed flow chart of step one of the method of the present invention.
FIG. 3 is a detailed flow chart of step two of the method of the present invention.
FIG. 4 is a diagram of the entire denoising frame and the corresponding network structure in the method of the present invention, wherein: the diagram (a) is the whole denoising process framework, and comprises the processes of the third step, the fourth step, the fifth step and the sixth step; the diagram (b) is the structure of the residual error network designed in step three; FIG. (c) shows the shallow core predictor network structure involved in step three.
FIG. 5 is a graph illustrating the effect of using the illumination transmission covariance on the denoising result in the present invention.
FIG. 6 shows the influence of the network structure on the denoising result in the present invention.
FIG. 7 is a graph illustrating the effect of the use of the perceptual loss function on the denoising result in the present invention.
Detailed Description
As shown in fig. 1, the monte carlo denoising method based on the detail-preserving neural network model of the present invention includes the following steps:
step one, an output channel of the renderer comprises a color buffer area and an auxiliary feature buffer area, illumination transmission covariance is added in the auxiliary feature buffer area, a large amount of noise data and corresponding reference data (such as more than 200 pairs) are rendered, and a training data set is formed. As shown in fig. 2, the forming of the training data set includes the following specific steps:
1.1, the rendering result of the renderer is divided into a diffuse reflection color and a specular reflection color (i.e., in the monte carlo rendering process, if the first collision surface is made of pure diffuse reflection material, the color obtained by calculating the path is used as the diffuse reflection color, and finally the diffuse reflection color is subtracted from the complete color obtained by the monte carlo rendering calculation to obtain the specular reflection color), and a corresponding color variance is calculated (i.e., in the monte carlo rendering process, the variance of the sample sampled by each pixel is calculated, and the brightness of the sample is converted into a single channel) to be used as a color buffer (wherein the diffuse reflection color and the specular reflection color are respectively 3 channels, and the color variance is 1 channel, i.e., the color buffer totally contains 8 channels).
1.2 calculating the illumination transmission covariance in the path space in the rendering pipeline, namely calculating the covariance matrix of the local illumination field to approximately represent the Fourier spectrum, wherein the local illumination field is a four-dimensional function which has two dimensions in space and angle; the covariance matrix sigma has the formula
Where Ω is the four-dimensional domain mentioned above, f is the local illumination field function over the four-dimensional domain, e i Is the ith canonical basis vector on Ω, x · y is the dot product between vectors x and y; then, the determinant of the matrix is calculated and the root number is taken as: 
Eta is the calculation result of the covariance of the illumination transmission, the higher the value is, the higher the information frequency at the position is, and the lower the value is, the lower the information frequency at the position is.
1.3 the illumination transmission covariance and the normal, depth, albedo and corresponding feature variance obtained in the rendering pipeline (i.e. in the monte carlo rendering process, the variance of the sample sampled by each pixel is calculated, and the brightness is calculated and converted into a single channel) are used together as an auxiliary feature buffer (the normal and albedo are 3 channels, the depth and illumination transmission covariance are 1 channel, and the feature variance is 1 channel, i.e. the auxiliary feature buffer contains 12 channels altogether).
1.4 the resolution of the rendering is set to 1280 × 720, a large amount of noise data (more than 200) is rendered using a sampling rate of 128spp, while reference data corresponding to the noise data is rendered using a sampling rate of 8192spp, from which a training data set is constructed.
Step two, preprocessing the noise data of the training data set, converting diffuse reflection color into radiation illumination color, converting specular reflection color into a log domain, and normalizing the data in the auxiliary feature buffer area to obtain preprocessed noise data, as shown in fig. 3, the process is as follows:
2.1 extraction of albedo from diffuse reflectance color: dividing the diffuse reflection color by the sum of the albedo and a minimum number to obtain the radiance color, wherein the formula is
Wherein c is i Representing the color of the irradiance, c d Representing diffuse reflectance color, a representing albedo, ε is a very small number to prevent the denominator from being zero, where ε takes the value of 0.000001;
2.2 logarithmic transformation of the mirror reflection color by the specific formula
Wherein
Representing the color of the specular reflection in the logarithmic domain, c s Representing a specular reflection color;
2.3 linearly scaling the depth and the illumination transmission covariance obtained in the steps 1.2 and 1.3 to the range of [0,1 ];
2.4 calculating the radiance color, the log domain specular reflection color, the normal, the albedo, the depth, the gradient of the illumination transmission covariance in the x and y directions, and merging with the corresponding buffer (for example, the radiance color buffer after merging contains the radiance color and the corresponding gradient and variance, the gradient and color are 3 channels each, the variance is 1 channel, so 7 channels after merging);
2.5 combining the auxiliary feature buffer area preprocessed in the steps 2.3 and 2.4 with the diffuse reflection color preprocessed in the steps 2.1 and 2.4 and the specular reflection color preprocessed in the steps 2.2 and 2.4 to obtain a diffuse reflection assembly and a specular reflection assembly respectively, wherein each assembly comprises an initial color (diffuse reflection or specular reflection color), a preprocessing color (radiant illumination color or logarithmic domain specular reflection color), a normal, albedo, depth, illumination transmission covariance and corresponding gradient and variance, and 30 channels are provided in total. The diffuse reflection assembly and the specular reflection assembly are preprocessed noise data.
Inputting the preprocessed noise data into a neural network comprising a feature extractor and a kernel predictor for filtering, and outputting a filtering kernel, wherein the method comprises the following specific steps:
3.1 respectively sending the diffuse reflection assembly and the specular reflection assembly into two neural networks comprising a feature extractor and a kernel predictor;
3.2 in the feature extraction part of fig. 4 (a), the color component (i.e. the initial color, the pre-processed color and the corresponding gradient and variance in the component, which are 10 channels) and the auxiliary feature component (i.e. the normal, the albedo, the depth, the illumination transmission covariance and the corresponding gradient and variance in the component, which are 20 channels) are sent to two feature extractors respectively for feature extraction, where the feature extractor uses a residual network, and the convolutional layer parameters used in the network are: the number of input channels of the first layer is 10 and 20 (corresponding to the color buffer area and the feature buffer area), the number of input channels of the other layers is 100, the number of output channels is 100, and the sizes of convolution kernels are 3 × 3. As in fig. 4 (b), the first layer of the residual network is a convolutional layer; eight residual blocks are arranged next, each residual block is of a two-layer network structure, each layer of the residual block comprises a Relu active layer and a convolution layer, and the output of the convolution layer and the input of the residual block are subjected to jump-in connection at the tail of the residual block, namely, addition is carried out; outputting the extracted high-dimensional features through a Relu activation layer and a convolution layer, and splicing the two extracted high-dimensional features;
3.3 the kernel prediction part of fig. 4 (a), the stitched high-dimensional features are input into a shallow kernel predictor, fig. 4 (c), which uses four convolutional neural networks, the first three layers each including a convolutional layer and a Relu activation layer, and the last layer having only one convolutional layer. The convolution kernel size of each layer is 3 multiplied by 3, the number of input channels of the first layer of convolution layer is 200, and the number of output channels is 100; the number of input and output channels of the second and third layers of convolution layers is 100; the number of input channels of the fourth layer is 100, and the number of output channels is 441, that is, the filter kernel of 21 × 21 size is finally output.
And step four, as a kernel prediction part in (a) in fig. 4, convolving the radiance color in the diffuse reflection assembly and the logarithmic domain specular reflection color in the specular reflection assembly by using the filter kernel output by the neural network to obtain the denoised radiance color and the denoised logarithmic domain specular reflection color.
Step five, calculating the error between the denoising result and the reference data by using the SMAPE and the perception loss function, training and optimizing parameters of the network, and if the verification result is converged or reaches the maximum iteration number after training, performing the step six, otherwise, returning to the step three, wherein the training and parameter optimizing steps are as follows:
5.1 respectively calculating SMAPE and sensing loss among the diffuse reflection denoising result, the specular reflection denoising result and corresponding reference data, wherein the formula of the SMAPE loss function is
Wherein
Representing the result of denoising, c r Representing the reference color, epsilon is a very small number to prevent the denominator from being zero, where epsilon takes 0.000001. The perceptual loss function is formulated as
Where whd represents the width, height, and number of channels of the denoising result, respectively, and phi is the high-dimensional feature extractor, where pre-trained VGG-19 is used as the feature extractor. The diffuse reflection loss is calculated by the error between the denoised radiance and the reference radiance; the specular reflection loss is calculated as the error between the denoised specular reflection and the reference specular reflection in the logarithmic transformed domain;
5.2 using ADAM optimizer to carry out training optimization on corresponding network parameters according to the calculated loss function, wherein the training hyper-parameters are set as: learning rate 10 -4 (ii) a Taking the Batchsize as 10; patchsize takes 128 x 128; the network parameter initialization method uses the Xavier initialization method. And (3) performing verification (namely taking a part of data in the training data set to perform denoising processes of the second step, the third step and the fourth step and outputting the denoising result and the error before the denoising result and the corresponding reference data) and batch replacement every 500 times of iteration, and if the verification result shows that convergence is achieved (namely the errors of the verification results do not have large floating any more for a plurality of times, namely convergence is achieved), or the iteration number reaches the maximum value (generally, the iteration number is set to be more than 100000 times), performing the sixth step, otherwise, returning to the third step.
Step six, using a trained (namely trained to be convergent) network to denoise any data (namely noise data rendered according to the step one) conforming to the network input structure, and reconstructing to obtain a complete denoised color image, wherein the method comprises the following substeps:
6.1 preprocessing input data and denoising by using a trained neural network, wherein the flow is the same as the processing flow of the noise data of the training set in the second, third and fourth steps, so that a radiation illumination denoising result and a logarithmic domain specular reflection denoising result of the input data are obtained;
6.2 as the reconstruction part of (a) in fig. 4, the radiation illumination denoising result and the logarithm domain specular reflection denoising result are subjected to inverse transformation of the steps 2.1 and 2.2, and the denoised radiation illumination is multiplied by the albedo to restore the complete diffuse reflection color, wherein the formula is
Wherein
Representing the denoised radiance color,
representing the de-noised diffuse reflectance color, a representing the albedo, and e being the fraction used in the calculation of the radiance in step 2.1. The logarithm domain specular reflection denoising result is subjected to exponential transformation to restore the specular reflection color, and the formula is
Wherein
Representing the denoised logarithmic domain specular reflection color, 
Representing the de-noised specular reflection color;
and 6.3, adding the restored diffuse reflection denoising result and the mirror reflection denoising result to obtain a final denoising result, namely a complete 3-channel denoising color image.
The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents and improvements made within the spirit and principle of the present invention are intended to be included within the scope of the present invention.
Claims (4)
1. A Monte Carlo denoising method based on a detail-preserving neural network model is characterized by comprising the following steps:
step one, enabling an output channel of a renderer to comprise a color buffer area and an auxiliary feature buffer area, increasing an illumination transmission covariance in the auxiliary feature buffer area, and rendering a large amount of noise data and corresponding reference data to form a training data set;
preprocessing noise data of the training data set, converting diffuse reflection color into radiation illumination color, converting specular reflection color into a logarithmic domain, and normalizing data in the auxiliary feature buffer area to obtain preprocessed noise data;
inputting the preprocessed noise data into a neural network comprising a feature extractor and a kernel predictor for filtering, and outputting a filtering kernel, wherein the step three is as follows:
3.1 respectively sending the diffuse reflection assembly and the specular reflection assembly into two neural networks comprising a feature extractor and a kernel predictor;
3.2 in the feature extraction part, a color component and an auxiliary feature component are respectively sent into two feature extractors for feature extraction, wherein the color component comprises 10 channels including an initial color, a preprocessing color and a corresponding gradient and variance, and the auxiliary feature component comprises 20 channels including a normal, an albedo, a depth, an illumination transmission covariance and a corresponding gradient and variance; the feature extractor uses a residual error network, and convolution layer parameters used in the network are as follows: the number of input channels of the first layer corresponds to a color buffer area and a characteristic buffer area which are respectively 10 and 20, the number of input channels of the other layers is 100, the number of output channels is 100, and the sizes of convolution kernels are all 3 multiplied by 3; the first layer of the residual error network is a convolution layer; then eight residual blocks are arranged, each residual block is of a two-layer network structure, each layer of the residual block comprises a Relu active layer and a convolution layer, and the output of the convolution layer and the input of the residual block are in jump-in connection at the tail end of the residual block, namely, addition is carried out; outputting the extracted high-dimensional features through a Relu activation layer and a convolution layer, and splicing the two extracted high-dimensional features;
3.3, inputting the spliced high-dimensional features into a shallow kernel predictor, wherein the shallow kernel predictor uses four layers of convolutional neural networks, each layer of the first three layers comprises a convolutional layer and a Relu active layer, the last layer only comprises one convolutional layer, the size of a convolutional kernel of each layer is 3 multiplied by 3, the number of input channels of the convolutional layer of the first layer is 200, and the number of output channels is 100; the number of input and output channels of the second layer and the third layer of convolution layers is 100; the number of input channels of the fourth layer is 100, the number of output channels is 441, that is, a filter kernel with the size of 21 × 21 is finally output;
performing convolution on the radiation illumination color in the diffuse reflection assembly and the logarithmic domain specular reflection color in the specular reflection assembly by using a filter core output by the neural network to obtain a denoised radiation illumination color and a denoised logarithmic domain specular reflection color;
step five, calculating the error between the denoising result and the reference data by using the SMAPE and the perception loss function, training and optimizing parameters of the network, and if the verification result is converged or reaches the maximum iteration number after training, performing step six, otherwise, returning to the step three, wherein the method specifically comprises the following steps:
5.1 respectively calculating SMAPE and sensing loss among the diffuse reflection denoising result, the specular reflection denoising result and corresponding reference data, wherein the formula of the SMAPE loss function is 
Wherein
Representing the denoising result, c r Representing a reference color, epsilon is a very small number to prevent the denominator from being zero; the perceptual loss function is formulated as
Whd, wherein phi is a high-dimensional feature extractor, and a pre-trained VGG-19 is used as the feature extractor; the diffuse reflection loss is calculated by the error between the denoised radiance and the reference radiance; the specular reflection loss is calculated as the error between the denoised specular reflection and the reference specular reflection in the logarithmic transformed domain;
5.2, using an ADAM optimizer to train and optimize corresponding network parameters according to the calculated loss function, verifying and batch replacing after iteration, if the verification result shows that convergence is achieved, performing the sixth step, and if not, returning to the third step;
and step six, denoising the data which are in accordance with the network input structure by using the trained network.
2. The Monte Carlo denoising method based on the detail-preserving neural network model as claimed in claim 1, wherein the specific steps of forming the training data set in the step one are as follows:
1.1, dividing a rendering result of a renderer into diffuse reflection color and specular reflection color, and calculating corresponding color variance, namely in the process of Monte Carlo rendering, calculating the variance of a sample sampled by each pixel, and calculating the brightness of the sample converted into a single channel serving as a color buffer area, wherein the diffuse reflection color and the specular reflection color are respectively 3 channels, the color variance is 1 channel, and the color buffer area contains 8 channels;
1.2 calculating the illumination transmission covariance in the path space in the rendering pipeline, namely calculating the covariance matrix of the local illumination field to approximately represent the Fourier spectrum, wherein the local illumination field is a four-dimensional function which has two dimensions in space and angle; the covariance matrix sigma has the formula
Where Ω is a four-dimensional function, f is the local illumination field function over the four-dimensional function, e i Is the ith canonical basis vector on Ω, x · y is the dot product between vectors x and y; then, the determinant of the matrix is calculated and the root number is taken as:
eta is the illumination transmission covariance calculation result;
1.3, the illumination transmission covariance, a normal line, a depth, an albedo and a corresponding feature variance obtained in a rendering pipeline are used as an auxiliary feature buffer area, the normal line and the albedo are 3 channels, the depth and the illumination transmission covariance are 1 channel, the feature variance is 1 channel, namely the auxiliary feature buffer area contains 12 channels together;
1.4 the resolution of the rendering is set to 1280 × 720, a large amount of noise data is rendered using a sampling rate of 128spp, while reference data corresponding to the noise data is rendered using a sampling rate of 8192spp, from which a training data set is constructed.
3. The Monte Carlo denoising method based on the detail-preserving neural network model as claimed in claim 2, wherein the noise data preprocessed in step two is processed as follows:
2.1 extraction of albedo from diffuse reflectance color: dividing the diffuse reflection color by the sum of the albedo and a minimum number to obtain the radiance color, wherein the formula is
Wherein c is i Representing the color of the irradiance, c d Representing diffuse reflectance color, a representing albedo, epsilon being a very small number to prevent the denominator from being zero;
2.2 logarithmic transformation of the mirror reflection color by the specific formula
Wherein
Representing the color of the specular reflection in the logarithmic domain, c s Representing a specular reflection color;
2.3 linearly scaling the depth and the illumination transmission covariance obtained in the steps 1.2 and 1.3 to the range of [0,1 ];
2.4 calculating the gradient of the radiance color, the logarithm domain specular reflection color, the normal, the albedo, the depth and the illumination transmission covariance in the directions of X and y, and merging the gradient with the corresponding buffer area;
2.5 combining the auxiliary feature buffer area preprocessed in the steps 2.3 and 2.4 with the diffuse reflection color preprocessed in the steps 2.1 and 2.4 and the specular reflection color preprocessed in the steps 2.2 and 2.4 to obtain a diffuse reflection assembly and a specular reflection assembly respectively, wherein each assembly comprises an initial color, a preprocessed color, a normal line, an albedo, a depth, an illumination transmission covariance and a corresponding gradient and variance, and the diffuse reflection assembly and the specular reflection assembly are preprocessed noise data.
4. The Monte Carlo denoising method based on the detail-preserving neural network model as claimed in claim 1, wherein the step of denoising the data conforming to any network input structure in the sixth step is as follows:
6.1, preprocessing noise data in the data and denoising by using a trained neural network so as to obtain a radiation illumination denoising result and a logarithmic domain specular reflection denoising result of the data;
6.2 denoising result and logarithm domain of irradianceThe denoising result of the specular reflection is inversely transformed, and the denoised radiation illumination is multiplied by the albedo to restore the complete diffuse reflection color, wherein the formula is
Wherein
Representing the denoised radiance color,
representing the denoised diffuse reflection color, a representing the albedo, epsilon being the minimum number used in calculating the radiance, and carrying out exponential transformation on the denoising result of the logarithmic domain specular reflection to restore the specular reflection color, wherein the formula is
Wherein
Representing the denoised logarithmic domain specular reflection color,
representing the de-noised specular reflection color;
and 6.3, adding the restored diffuse reflection denoising result and the mirror reflection denoising result to obtain a final denoising result, namely a complete 3-channel denoising color image.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911199715.3A CN111145103B (en) | 2019-11-29 | 2019-11-29 | Monte Carlo denoising method based on detail retention neural network model |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911199715.3A CN111145103B (en) | 2019-11-29 | 2019-11-29 | Monte Carlo denoising method based on detail retention neural network model |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111145103A CN111145103A (en) | 2020-05-12 |
| CN111145103B true CN111145103B (en) | 2022-08-12 |
Family
ID=70517340
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911199715.3A Active CN111145103B (en) | 2019-11-29 | 2019-11-29 | Monte Carlo denoising method based on detail retention neural network model |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111145103B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113158563B (en) * | 2021-04-16 | 2022-11-18 | 华南理工大学 | One-dimensional photonic crystal reverse design method based on neural network and Monte Carlo |
| CN113628126B (en) * | 2021-06-29 | 2022-03-01 | 光线云(杭州)科技有限公司 | Real-time Monte Carlo path tracking noise reduction method and device based on importance characteristic graph sharing and computer equipment |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10475165B2 (en) * | 2017-04-06 | 2019-11-12 | Disney Enterprises, Inc. | Kernel-predicting convolutional neural networks for denoising |
| CN108765319B (en) * | 2018-05-09 | 2020-08-14 | 大连理工大学 | Image denoising method based on generation countermeasure network |
-
2019
- 2019-11-29 CN CN201911199715.3A patent/CN111145103B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN111145103A (en) | 2020-05-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109308679B (en) | Image style conversion method and device, equipment and storage medium | |
| US10796414B2 (en) | Kernel-predicting convolutional neural networks for denoising | |
| CN114140353B (en) | Swin-Transformer image denoising method and system based on channel attention | |
| CN109859147B (en) | Real image denoising method based on generation of antagonistic network noise modeling | |
| US11645835B2 (en) | Hypercomplex deep learning methods, architectures, and apparatus for multimodal small, medium, and large-scale data representation, analysis, and applications | |
| Matakos et al. | Accelerated edge-preserving image restoration without boundary artifacts | |
| CN110276726B (en) | Image deblurring method based on multichannel network prior information guidance | |
| Yin et al. | Highly accurate image reconstruction for multimodal noise suppression using semisupervised learning on big data | |
| CN112541864A (en) | Image restoration method based on multi-scale generation type confrontation network model | |
| CN111986075B (en) | Style migration method for target edge clarification | |
| CN106204483B (en) | Multiplicative noise minimizing technology based on non local self-adapting dictionary | |
| CN109410149B (en) | CNN denoising method based on parallel feature extraction | |
| CN108765330B (en) | Image denoising method and device based on global and local prior joint constraint | |
| CN111145103B (en) | Monte Carlo denoising method based on detail retention neural network model | |
| CN112150354B (en) | Single image super-resolution method combining contour enhancement and denoising statistical prior | |
| Kim et al. | Deeply aggregated alternating minimization for image restoration | |
| CN110349112B (en) | Two-stage image denoising method based on self-adaptive singular value threshold | |
| CN111127354A (en) | Single-image rain removing method based on multi-scale dictionary learning | |
| CN114820352A (en) | Hyperspectral image denoising method and device and storage medium | |
| CN113052775A (en) | Image shadow removing method and device | |
| CN112270650B (en) | Image processing method, system, medium, and apparatus based on sparse autoencoder | |
| CN117788629A (en) | Image generation method, device and storage medium with style personalization | |
| CN111667401B (en) | Multi-level gradient image style migration method and system | |
| CN113496460B (en) | Neural style migration method and system based on feature adjustment | |
| CN114862699B (en) | Face repairing method, device and storage medium based on generation countermeasure network |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |