CN113628126B - Real-time Monte Carlo path tracking noise reduction method and device based on importance characteristic graph sharing and computer equipment - Google Patents
Real-time Monte Carlo path tracking noise reduction method and device based on importance characteristic graph sharing and computer equipment Download PDFInfo
- Publication number
- CN113628126B CN113628126B CN202110762590.1A CN202110762590A CN113628126B CN 113628126 B CN113628126 B CN 113628126B CN 202110762590 A CN202110762590 A CN 202110762590A CN 113628126 B CN113628126 B CN 113628126B
- Authority
- CN
- China
- Prior art keywords
- importance
- monte carlo
- noise reduction
- path tracking
- carlo path
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 82
- 230000009467 reduction Effects 0.000 title claims abstract description 54
- 238000001914 filtration Methods 0.000 claims abstract description 50
- 238000012545 processing Methods 0.000 claims abstract description 35
- 238000013528 artificial neural network Methods 0.000 claims abstract description 25
- 239000000463 material Substances 0.000 claims abstract description 14
- 239000003086 colorant Substances 0.000 claims abstract description 8
- 238000009877 rendering Methods 0.000 claims description 28
- 230000008569 process Effects 0.000 claims description 22
- 230000004927 fusion Effects 0.000 claims description 16
- 238000004590 computer program Methods 0.000 claims description 12
- 238000007781 pre-processing Methods 0.000 claims description 12
- 238000010586 diagram Methods 0.000 claims description 10
- 238000010606 normalization Methods 0.000 claims description 7
- 238000004364 calculation method Methods 0.000 claims description 5
- 238000006243 chemical reaction Methods 0.000 claims description 4
- 238000013135 deep learning Methods 0.000 abstract description 3
- 230000006870 function Effects 0.000 description 9
- 238000013507 mapping Methods 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 3
- 238000009825 accumulation Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 2
- 230000002146 bilateral effect Effects 0.000 description 2
- 238000010276 construction Methods 0.000 description 2
- 238000012360 testing method Methods 0.000 description 2
- 230000000007 visual effect Effects 0.000 description 2
- 238000007792 addition Methods 0.000 description 1
- 238000004422 calculation algorithm Methods 0.000 description 1
- 239000002131 composite material Substances 0.000 description 1
- 230000001186 cumulative effect Effects 0.000 description 1
- 238000005265 energy consumption Methods 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 238000005286 illumination Methods 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012805 post-processing Methods 0.000 description 1
- 238000011946 reduction process Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/70—Denoising; Smoothing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/90—Determination of colour characteristics
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Biomedical Technology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Life Sciences & Earth Sciences (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Image Generation (AREA)
Abstract
The invention discloses a Monte Carlo path tracking noise reduction method, a Monte Carlo path tracking noise reduction device and an importance characteristic graph sharing method, and a Monte Carlo path tracking noise reduction device, wherein the method comprises the following steps: performing preliminary filtering pretreatment on a real-time drawing frame generated by Monte Carlo path tracking; splicing the color features corresponding to each preprocessed drawing frame with auxiliary features corresponding to scene geometric data which does not contain noise to serve as input features; the method comprises the steps that a plurality of importance feature maps of input features are obtained through deep neural network prediction, a multi-channel filter with a specific window size on each pixel is built through a cross-pixel importance value sharing mode aiming at each importance feature map, and the multi-channel filters with different window sizes built on the basis of the importance feature maps are used for multiple filtering of the pixels; and fusing the multiple filtering processing results and then superposing the colors of the basic materials to obtain a final noise reduction processing result. The filter is constructed by using the importance value predicted by the deep learning network to reduce noise of the drawn frame, so that the efficiency is high and the cost is low.
Description
Technical Field
The invention belongs to the field of real-time drawing, and particularly relates to a Monte Carlo path tracking noise reduction method and device based on importance characteristic graph sharing and computer equipment.
Background
Monte Carlo path tracing is a technology for drawing photo-level realistic pictures, and is characterized in that various complex physical-based visual effects such as soft shadows, global illumination and the like are generated by using only one set of uniform drawing pipeline, and the drawing pipeline does not need to perform additional large-scale customization on a certain visual effect, so that the Monte Carlo path tracing is attractive to off-line drawing application and real-time drawing application.
One of the core steps of the monte carlo path tracing technique is to calculate a monte carlo integral, which requires a large number of samples to be converged, and if a small number of samples are used, the convergence is poor, i.e. the rendered picture has significant noise, so that a large number of samples per pixel are required to obtain a composite picture in which the noise is difficult to distinguish by naked eyes. This process is very time consuming, and a picture may take many hours to draw, which obviously does not meet the time requirement for real-time drawing.
One solution that can be used in practical applications is to use a monte carlo ray tracing technique to generate a two-dimensional picture with noise using a small number of samples, and then perform a noise reduction algorithm in the two-dimensional picture space to reduce the noise contained in the picture, thereby approximating the result of monte carlo integration under a large number of samples. The method is called Monte Carlo noise reduction method, which has been a hot problem for research in industry and academia for its high application value, and has produced a large number of noise reduction methods based on technical backgrounds such as filters, regression models, neural networks, etc., but most of these methods are based on off-line rendering application.
Real-time rendering applications have more stringent time constraints than offline rendering applications, typically requiring sixty pictures to be rendered per second. Under this time constraint, the conventional hardware device only allows one Monte Carlo sample to be calculated for each pixel, and therefore the result of the rendering contains more serious noise. That is, compared to the offline noise reduction method, the real-time noise reduction method faces a more challenging task, i.e., reducing more severe noise in a shorter time. The traditional real-time noise reduction method has poor self-adaptive capacity, so that the quality of a reconstructed picture is low.
Meng et al also propose a real-time noise reduction method combining a neural network with a three-dimensional bilateral grid, but the related operation overhead of the three-dimensional bilateral grid is large, so that the overall noise reduction method is insufficient in capacity, and the quality of a reconstructed picture is poor.
Therefore, a method for denoising in real-time monte carlo path tracking is urgently needed to satisfy real-time rendering application.
Disclosure of Invention
Based on the above, the present invention provides a monte carlo path tracking denoising method, apparatus and computer device based on importance characteristic map sharing, which construct a multi-channel filter weight by sharing importance value predicted by using a neural network to filter a monte carlo path tracking rendering frame, so as to implement denoising.
In a first aspect, an embodiment of the present invention provides a monte carlo path tracking noise reduction method based on importance feature map sharing, including the following steps:
performing preliminary filtering on a real-time drawing frame generated by Monte Carlo path tracking to realize the pretreatment of the drawing frame;
splicing the color features corresponding to each preprocessed drawing frame with auxiliary features corresponding to scene geometric data which does not contain noise to serve as input features;
the method comprises the steps that a plurality of importance characteristic graphs of input characteristics are obtained through deep neural network prediction, a multi-channel filter with a specific window size on each pixel is built in a cross-pixel importance value sharing mode aiming at each importance characteristic graph, and the multi-channel filters with different window sizes built on the basis of the importance characteristic graphs are used for multiple filtering processing of real-time drawn frames;
and fusing the multiple filtering processing results and then superposing the colors of the basic materials to obtain a final noise reduction processing result.
In a second aspect, an embodiment provides an apparatus for performing monte carlo path tracking noise reduction based on importance feature map sharing, including:
the preprocessing module is used for carrying out preliminary filtering on the Monte Carlo path tracking drawing frame to realize preprocessing of the drawing frame;
the splicing module is used for splicing the color features corresponding to each preprocessed drawing frame with the auxiliary features corresponding to the scene geometric data which does not contain noise as input features;
the prediction module is used for predicting a plurality of importance characteristic graphs of the input characteristics by utilizing the deep neural network;
the filtering module is used for constructing a multi-channel filter with a specific window size on each pixel in a cross-pixel importance value sharing mode aiming at each importance characteristic diagram, and the multi-channel filters with different window sizes constructed based on the importance characteristic diagrams are used for multiple filtering processing of real-time drawn frames;
and the fusion module is used for fusing the multiple filtering processing results and then superposing the colors of the basic materials to obtain a final noise reduction processing result.
In a third aspect, an embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, where the processor, when executing the computer program, implements the method for monte carlo path tracking noise reduction based on importance feature map sharing according to the first aspect.
In a fourth aspect, an embodiment provides a computer storage medium, on which a computer program is stored, and the computer program is processed and executed to implement the steps of the method for monte carlo path tracking noise reduction based on importance feature map sharing according to the first aspect.
The technical scheme provided by the embodiment has the beneficial effects that at least:
the importance value of each pixel in the noise reduction process is predicted by utilizing the super-strong expression capacity of the deep neural network, and the multichannel filter weight is constructed by sharing the importance value, so that the throughput of the neural network, the video memory use in the reconstruction process and the bandwidth can be reduced by the efficient and compact representation of the middle of the filter weight; meanwhile, the whole image denoising and reconstruction process is carried out based on a multi-channel filter, the execution operation is simple, and the parallelism is high; moreover, the method can be conveniently deployed and integrated into the existing path tracking renderer, is simple and efficient, and can reconstruct an image with higher quality with high efficiency, low throughput and low video memory overhead.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to these drawings without creative efforts.
FIG. 1 is a flowchart of a Monte Carlo path tracing noise reduction method based on importance feature map sharing according to an embodiment;
FIG. 2 is a flow chart of pixel importance prediction and filter-based image reconstruction according to an embodiment;
FIG. 3 is a flow diagram illustrating the construction of a filter according to an embodiment;
fig. 4 is a schematic structural diagram of an apparatus for monte carlo path tracking noise reduction based on importance map sharing according to an embodiment.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention will be further described in detail with reference to the accompanying drawings and examples. It should be understood that the detailed description and specific examples, while indicating the scope of the invention, are intended for purposes of illustration only and are not intended to limit the scope of the invention.
The real-time drawing frame is obtained through the Monte Carlo path tracking technology, and due to the fact that the sample number is insufficient and certain noise exists, the noise reduction processing is needed, and a high-quality drawing image can be obtained. The invention provides a Monte Carlo path tracking noise reduction method and device based on importance characteristic graph sharing, which are used for realizing high-efficiency and low-energy-consumption real-time noise reduction processing on a real-time drawing frame generated by Monte Carlo path tracking.
Fig. 1 is a flowchart of a monte carlo path tracing noise reduction method based on importance feature map sharing according to an embodiment. Fig. 2 is a flowchart of pixel importance prediction and image reconstruction based on filter filtering according to an embodiment. As shown in fig. 1 and fig. 2, the monte carlo path tracking noise reduction method based on importance feature map sharing according to the embodiment includes the following steps:
In an embodiment, the preliminary filtering of the noise is performed in a time sequence accumulation manner of a real-time rendered frame of the noise, and includes: and carrying out re-projection on the adjacent historical drawing frames of the current drawing frame by adopting camera parameters and noiseless scene geometric data so as to accumulate the adjacent historical drawing frames to the current drawing frame, so as to realize preliminary filtering of the current drawing frame and preliminarily reduce the noise in the current drawing frame.
In an embodiment, the cumulative formula used for preliminary filtering is an exponential moving average:
wherein r isi-1Rendering results of the past rendering frames adjacent to the current rendering frame, riA drawing result representing the current drawing frame,
and a represents a drawing result of the current drawing frame after time sequence accumulation, and a represents a coefficient of a moving average, and has a value range of 0 to 1, and preferably, a value of 0.2 can be adopted.
The drawing frame drawn by the monte carlo path tracking technology is represented by a high dynamic representation range color, the data size is large, and prediction calculation of the importance value is not facilitated, so that the method provided by the embodiment further comprises the following steps: and after the real-time drawing frame generated by the Monte Carlo path tracking is primarily filtered, color value space conversion from a high dynamic representation range to a low dynamic representation range is carried out on the primary filtering result so as to realize the preprocessing of the drawing frame. In the embodiment, a color space mapping method is adopted to realize color numerical value space conversion of the drawing frame.
After the preliminary filtering preprocessing or the preprocessing of preliminary filtering and color value space conversion, the color features corresponding to the rendering frame can be obtained and used for calculating the importance value of the rendering pixel.
And 2, splicing the color features corresponding to each preprocessed drawing frame with auxiliary features corresponding to scene geometric data which does not contain noise to serve as input features.
In an embodiment, the rendering frame mainly provides color features, and in order to improve the comprehensiveness and accuracy of the pixel importance value of the rendering frame, other information of the rendering process needs to be referred to when the pixel importance value is calculated. Therefore, the auxiliary features corresponding to the scene geometric data not containing noise are included, the color features corresponding to each preprocessed rendering frame and the auxiliary features corresponding to the scene geometric data not containing noise are spliced to be used as input features, and the pixel importance value is calculated based on the input features.
In an embodiment, the scene geometry data without noise is a result of a first intersection calculation of a principal ray and a scene in monte carlo path tracking, and includes a camera view depth value, an object surface normal vector, and an object base material color. The auxiliary features are spliced together along the dimension of an image channel as input features in the form of a two-dimensional image and a color feature map of a drawing image.
the deep neural network has super mapping expression capability, and based on the super mapping expression capability, in the embodiment, the deep neural network is used for predicting the importance of the predicted pixels. In order to meet the real-time requirement, the capacity of the deep neural network is required to be as small as possible on the premise of ensuring the quality, and specifically, a full convolution neural network based on a RepVGG-Block module can be adopted, and the number of network layers is six. The full convolution neural network predicts an importance value of a pixel corresponding to a scalar representation value for each pixel, and the output of the full convolution neural network is a single-channel two-dimensional importance characteristic diagram consisting of the importance values.
In the embodiment, a low-quality drawing frame containing noise obtained by Monte Carlo path tracking drawing and a high-quality drawing frame after denoising form a sample pair, and the network parameters of the deep neural network are optimized by using the sample pair through a supervised learning process. And when a large number of sample pairs are used for determining network parameters through supervised learning, the prediction calculation of the pixel importance values of other input features of the deep learning network determined by the network parameters is used. After the input features corresponding to each real-time rendering frame are subjected to mapping calculation through a depth neural network determined by parameters, n importance feature maps are obtained, and 1 multi-channel filter is established by sharing importance values for each importance feature map so as to realize multi-aspect noise reduction processing on the rendering frames.
In one possible embodiment, the constructing a multi-channel filter with a specific window size on each pixel by sharing across pixel importance values for each importance feature map includes: performing image unfolding operation in a sliding window mode, and unfolding each single-channel importance characteristic graph into a multi-channel importance characteristic graph; and taking the normalization processing result of the importance value of each position in the multichannel importance characteristic diagram along the channel dimension as the filter weight so as to construct the multichannel filter. In this embodiment, a process of cross-pixel importance value sharing is implemented by performing image unfolding from a single channel to multiple channels and performing normalization processing on the importance value at each position of the multiple channel image along the channel dimension.
And aiming at image expansion operation, when each single-channel importance characteristic graph is expanded, all importance values in the sliding window are sequentially stored into a plurality of image channels at the central pixel position of the window, so that a multi-channel importance characteristic graph is obtained.
The aim is to expand the significance signature of a single H × W × 1 channel of size into a significance signature of multiple channels of size H × W × (k × k), where H, W denotes the width and height of the image and k denotes the size of the expansion window, also equal to the window size of the constructed filter. The specific unfolding process is as follows: adopting a sliding window, and sequentially storing all importance values in the sliding window into a k × k image channel at the central pixel position of the window, wherein it should be noted that the whole image expansion operation only comprises a memory access operation, and when k is 3, the specific image expansion operation is the left half shown in fig. 3.
For the normalization operation, the importance value of each position in the importance feature map of H × W × (k × k) multiple channels may be normalized, preferably using a SoftMax function as shown below:
wherein, wiThe importance value stored at one pixel location,
representing numbers after normalizationThe value, which is the filter weight, the normalization process corresponds to the right half of fig. 3, when each pixel position stores the normalization result as a filter weight.
The weight normalization ensures that each filtering core is energy-conserving, which is beneficial to not only optimizing weight parameters of a weight function, but also avoiding color drift of a result after filtering. The process of filtering the rendered frame containing noise by each filter in the construction of filter weights using the normalized result is represented as:
wherein R isiRepresenting the filtered result.
And 4, fusing the multiple filtering processing results, and then superposing the colors of the basic materials to obtain a final noise reduction processing result.
And predicting a plurality of importance characteristic graphs by using the shared weight function, constructing a plurality of filters according to the importance characteristic graphs, and fusing the filtering and denoising processing results for a plurality of times to obtain a fusion result.
In the embodiment, M importance characteristic graphs are predicted by utilizing a shared weight function, and each importance characteristic graph corresponds to a filter window k with different sizesmThe importance value of each importance value feature map is, for example, M ═ 6, and the filter window size corresponding to each importance value feature map is kmM 2+1, m {1, …,6}, i.e., the minimum filter window is 3 and the maximum filter window is 13. Then, a filter with a corresponding size is constructed in the step 3, and the noise drawing frame is independently filtered to obtain M filtered pictures { R }1,…,RMAnd finally, filtering the picture { R }1,…,RMCarrying out weighted fusion in the following mode to obtain a fusion result:
wherein alpha iskIs a weighted fusion weight coefficient, and satisfies
In the embodiment, in the process of supervising learning for optimizing the network parameters of the deep neural network, the fusion weight adopted for fusing the multiple filtering processing results is also optimized, and the sum of all the fusion weights is 1. It can be understood that the process of fusing the results of the multiple filtering processes is used as a part of the supervised learning process, that is, the whole supervised learning process includes: the method comprises the following steps of predicting an importance characteristic diagram part of input characteristics by using a deep neural network, constructing a multi-channel filter and filtering processing part according to the importance characteristic diagram, and fusing a plurality of filtering processing results, wherein the whole purpose of supervised learning is as follows: and learning the low-quality rendering frame containing noise obtained by tracing and rendering the Monte Carlo path to realize noise reduction, and updating the network parameters of the deep learning network and the fusion weight of the fusion part according to the difference between the denoising result and the denoised high-quality rendering frame serving as the label.
In order to prevent the high-frequency material information from being blurred during filtering, the filtering object is to remove the irradiance color in the noise picture of the base material color. Therefore, on the basis of obtaining the fusion result, the base material color needs to be added to the irradiance color image after noise reduction, and optionally, the base material color and the fusion result may be multiplied to obtain the final noise reduction processing result.
In the embodiment, with the peak signal-to-noise ratio (PSNR) and objective index based on perceptual Structural Similarity (SSIM) as evaluation criteria, the test results of the method provided by the embodiment and the existing method (NBGD) on the monte carlo path tracking noise image of a small number of samples in the Sponza test scenario are shown in table 1. It can be seen that the noise reduction picture with higher quality score can be reconstructed in shorter time by the embodiment.
TABLE 1
| Time consuming | Peak signal to noise ratio | Structural similarity | |
| Examples provide methods | 12.89ms | 34.60 | 0.983 |
| Existing methods | 13.97ms | 33.41 | 0.975 |
The monte carlo path tracking and denoising method based on importance characteristic graph sharing, which is provided by the embodiment, includes a process of constructing a filter based on a simple and compact expression form of filter weight with high efficiency, low throughput and low video memory overhead, and can effectively learn the importance degree of each pixel in a noise image in a denoising process, further construct filtering kernels with various sizes in an efficient manner to filter the noise image, and finally reconstruct a monte carlo path tracking and rendering result close to a high sampling number, so that the rendering time for obtaining a high-quality monte carlo path tracking image is greatly shortened.
The monte carlo path tracking noise reduction method based on importance characteristic graph sharing provided by the embodiment comprises a compact and dense intermediate representation method of filter weights and a high-efficiency, low-bandwidth and low-throughput filter reconstruction method; the whole method only needs one forward operation during execution, wherein the reconstruction of the filter and the fusion part of the multiple filters execute common algebraic operation in parallel, so that the whole noise reduction frame can be used as a post-processing module and conveniently integrated into the existing various path tracking renderers, and a high-quality Monte Carlo path tracking image can be obtained in real time (namely drawing 60 frames per second) only by a small amount of samples, such as one sample per pixel, and the method is simple and efficient.
Fig. 4 is a schematic structural diagram of an apparatus for monte carlo path tracking noise reduction based on importance map sharing according to an embodiment. As shown in fig. 4, an embodiment of the monte carlo path tracking noise reduction apparatus 400 includes:
the preprocessing module 401 is configured to perform preliminary filtering on the monte carlo path tracking rendering frame to implement preprocessing on the rendering frame;
a splicing module 402, configured to splice a color feature corresponding to each preprocessed rendering frame with an auxiliary feature corresponding to scene geometric data that does not include noise as an input feature;
a prediction module 403, configured to obtain multiple importance feature maps of the input features by using deep neural network prediction;
a filtering module 404, which constructs a multi-channel filter with a specific window size on each pixel by sharing the cross-pixel importance value for each importance feature map, and uses the multi-channel filters with different window sizes constructed based on the plurality of importance feature maps for multiple filtering processes of real-time rendering frames;
and the fusion module 405 is configured to fuse the multiple filtering processing results and then superimpose the colors of the basic material to obtain a final denoising processing result.
It should be noted that, in the monte carlo path tracking noise reduction apparatus based on importance feature map sharing according to the embodiment, when performing noise reduction processing, the division of each function module is described as an example, and the function distribution may be completed by different function modules according to needs, that is, the internal structure of the terminal or the server is divided into different function modules to complete all or part of the functions described above. In addition, the monte carlo path tracking noise reduction device based on the importance feature map sharing and the monte carlo path tracking noise reduction method based on the importance feature map sharing provided by the embodiment belong to the same concept, and the specific implementation process is detailed in the embodiment of the monte carlo path tracking noise reduction method based on the importance feature map sharing, and is not described herein again.
An embodiment further provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, where the processor implements the above method for reducing noise based on monte carlo path tracking shared by importance feature maps when executing the computer program, and the method includes the following steps:
step 2, splicing the color features corresponding to each preprocessed drawing frame with auxiliary features corresponding to scene geometric data which does not contain noise as input features;
and 4, fusing the multiple filtering processing results, and then superposing the colors of the basic materials to obtain a final noise reduction processing result.
Embodiments also provide a computer storage medium having stored thereon a computer program that, when being processed and executed, performs the steps of the above-described method for monte carlo path tracing noise reduction based on importance feature map sharing.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by hardware instructions of a computer program, which can be stored in a non-volatile computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. Any reference to memory, storage, database, or other medium used in the embodiments provided herein may include non-volatile and/or volatile memory, among others.
The above-mentioned embodiments are intended to illustrate the technical solutions and advantages of the present invention, and it should be understood that the above-mentioned embodiments are only the most preferred embodiments of the present invention, and are not intended to limit the present invention, and any modifications, additions, equivalents, etc. made within the scope of the principles of the present invention should be included in the scope of the present invention.
Claims (10)
1. A Monte Carlo path tracking noise reduction method based on importance characteristic graph sharing is characterized by comprising the following steps:
performing preliminary filtering on a real-time drawing frame generated by Monte Carlo path tracking to realize the pretreatment of the drawing frame;
splicing the color features corresponding to each preprocessed drawing frame with auxiliary features corresponding to scene geometric data which does not contain noise to serve as input features;
the method comprises the steps that a plurality of importance characteristic graphs of input characteristics are obtained through deep neural network prediction, a multi-channel filter with a specific window size on each pixel is built in a cross-pixel importance value sharing mode aiming at each importance characteristic graph, and the multi-channel filters with different window sizes built on the basis of the importance characteristic graphs are used for multiple filtering processing of real-time drawn frames;
and fusing the multiple filtering processing results and then superposing the colors of the basic materials to obtain a final noise reduction processing result.
2. The method for Monte Carlo path tracing noise reduction based on importance feature graph sharing of claim 1, wherein the method further comprises: and after the real-time drawing frame generated by the Monte Carlo path tracking is primarily filtered, color value space conversion from a high dynamic representation range to a low dynamic representation range is carried out on the primary filtering result so as to realize the preprocessing of the drawing frame.
3. The method of claim 1 or 2, wherein the preliminary filtering of the monte carlo path trace rendering frame comprises: and carrying out re-projection on the adjacent historical drawing frames of the current drawing frame by adopting camera parameters and noiseless scene geometric data so as to accumulate the adjacent historical drawing frames to the current drawing frame and realize preliminary filtering of the current drawing frame.
4. The method as claimed in claim 1 or 2, wherein the scene geometry data not containing noise is a result of first intersection calculation of a principal ray and a scene in the monte carlo path tracking, and includes a camera view depth value, an object surface normal vector, and an object basic material color.
5. The method for Monte Carlo path tracing noise reduction based on importance feature map sharing of claim 1 or 2, wherein the constructing a multi-channel filter with a specific window size on each pixel by sharing across pixel importance values for each importance feature map comprises: performing image unfolding operation in a sliding window mode, and unfolding each single-channel importance characteristic graph into a multi-channel importance characteristic graph; and taking the normalization processing result of the importance value of each position in the multichannel importance characteristic diagram along the channel dimension as the filter weight so as to construct the multichannel filter.
6. The method of claim 5, wherein when each single-channel importance map is expanded, all importance values in a sliding window are sequentially stored in a plurality of image channels at a central pixel position of the window to obtain a multi-channel importance map.
7. The method for Monte Carlo path tracking noise reduction based on importance feature map sharing of claim 1 or 2, wherein a low-quality rendered frame containing noise obtained by Monte Carlo path tracking rendering and a high-quality rendered frame after noise removal form a sample pair, and network parameters of the deep neural network are optimized by using the sample pair through a supervised learning process;
in the process of supervised learning for optimizing the network parameters of the deep neural network, fusion weights adopted for fusing the multiple filtering processing results are also optimized, and the sum of all the fusion weights is 1.
8. An importance feature map sharing-based Monte Carlo path tracking noise reduction device, comprising:
the preprocessing module is used for carrying out preliminary filtering on the Monte Carlo path tracking drawing frame to realize preprocessing of the drawing frame;
the splicing module is used for splicing the color features corresponding to each preprocessed drawing frame with the auxiliary features corresponding to the scene geometric data which does not contain noise as input features;
the prediction module is used for predicting a plurality of importance characteristic graphs of the input characteristics by utilizing the deep neural network;
the filtering module is used for constructing a multi-channel filter with a specific window size on each pixel in a cross-pixel importance value sharing mode, and the multi-channel filters with different window sizes constructed on the basis of a plurality of importance characteristic graphs are used for multiple times of filtering processing of real-time drawn frames;
and the fusion module is used for fusing the multiple filtering processing results and then superposing the colors of the basic materials to obtain a final noise reduction processing result.
9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor when executing the computer program implements the method for monte carlo path tracing noise reduction based on importance feature map sharing of any one of claims 1 to 8.
10. A computer storage medium having a computer program stored thereon, wherein the computer program is configured to, when executed, implement the method of any of claims 1 to 8 for monte carlo path tracing noise reduction based on importance feature map sharing.
Applications Claiming Priority (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110723508 | 2021-06-29 | ||
| CN2021107235084 | 2021-06-29 |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113628126A CN113628126A (en) | 2021-11-09 |
| CN113628126B true CN113628126B (en) | 2022-03-01 |
Family
ID=78379114
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110762590.1A Active CN113628126B (en) | 2021-06-29 | 2021-07-06 | Real-time Monte Carlo path tracking noise reduction method and device based on importance characteristic graph sharing and computer equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113628126B (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105787892A (en) * | 2016-02-22 | 2016-07-20 | 浙江传媒学院 | Monte Carlo noise removal method based on machine learning |
| CN110728636A (en) * | 2019-09-17 | 2020-01-24 | 杭州群核信息技术有限公司 | Monte Carlo rendering image denoising model, method and device based on generative confrontation network |
| CN111145103A (en) * | 2019-11-29 | 2020-05-12 | 南京理工大学 | Monte Carlo denoising method based on detail retention neural network model |
| CN111583135A (en) * | 2020-04-24 | 2020-08-25 | 华南理工大学 | Nuclear prediction neural network Monte Carlo rendering image denoising method |
| CN112150631A (en) * | 2020-09-23 | 2020-12-29 | 浙江大学 | Real-time energy consumption optimization drawing method and device based on neural network |
-
2021
- 2021-07-06 CN CN202110762590.1A patent/CN113628126B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105787892A (en) * | 2016-02-22 | 2016-07-20 | 浙江传媒学院 | Monte Carlo noise removal method based on machine learning |
| CN110728636A (en) * | 2019-09-17 | 2020-01-24 | 杭州群核信息技术有限公司 | Monte Carlo rendering image denoising model, method and device based on generative confrontation network |
| CN111145103A (en) * | 2019-11-29 | 2020-05-12 | 南京理工大学 | Monte Carlo denoising method based on detail retention neural network model |
| CN111583135A (en) * | 2020-04-24 | 2020-08-25 | 华南理工大学 | Nuclear prediction neural network Monte Carlo rendering image denoising method |
| CN112150631A (en) * | 2020-09-23 | 2020-12-29 | 浙江大学 | Real-time energy consumption optimization drawing method and device based on neural network |
Non-Patent Citations (5)
| Title |
|---|
| a survey on deep learning-based monte carlo denoising;Yuchi Huo等;《Computational Visual Media》;20210329;第169-185页 * |
| Nonlinearly Weighted First-order Regression for Denoising Monte Carlo Renderings;Benedikt Bitterli等;《Eurographics Symposium on Rendering 2016》;20161231;第1-11页 * |
| Supplemental Material: Interactive Reconstruction of Monte Carlo Image Sequences using a Recurrent Denoising Autoencoder;CHAKRAVARTY R.ALLA CHAITANYA等;《ACM Transactions on Graphics》;20170731;第36卷(第4期);第1-2页 * |
| 基于时间与空间滤波的实时路径追踪去噪算法研究;贺卫东;《研究与开发》;20200331;第12-16页 * |
| 基于真实感图像合成的去噪算法研究;吴熙;《中国优秀博硕士学位论文全文数据库(硕士) 信息科技辑》;20180615;I138-1655 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113628126A (en) | 2021-11-09 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111105352B (en) | Super-resolution image reconstruction method, system, computer equipment and storage medium | |
| CN111583135B (en) | Nuclear prediction neural network Monte Carlo rendering image denoising method | |
| CN111242844B (en) | Image processing method, device, server and storage medium | |
| CN111445418A (en) | Image defogging method and device and computer equipment | |
| EP2528042B1 (en) | Method and device for the re-meshing of 3D polygon models | |
| CN115345866B (en) | Building extraction method in remote sensing image, electronic equipment and storage medium | |
| CN110852199A (en) | Foreground extraction method based on double-frame coding and decoding model | |
| CN113744136A (en) | Image super-resolution reconstruction method and system based on channel constraint multi-feature fusion | |
| CN113870124A (en) | Dual-network mutual excitation learning shadow removing method based on weak supervision | |
| CN112288788A (en) | Monocular image depth estimation method | |
| CN114663662B (en) | Hyper-parameter searching method, device, computer equipment and storage medium | |
| CN116912405A (en) | Three-dimensional reconstruction method and system based on improved MVSNet | |
| CN109871790B (en) | Video decoloring method based on hybrid neural network model | |
| Zhang et al. | Dynamic multi-scale network for dual-pixel images defocus deblurring with transformer | |
| CN115689964A (en) | Image enhancement method and device, electronic equipment and storage medium | |
| CN110580726A (en) | Dynamic convolution network-based face sketch generation model and method in natural scene | |
| Wang et al. | Agcyclegan: Attention-guided cyclegan for single underwater image restoration | |
| CN113628126B (en) | Real-time Monte Carlo path tracking noise reduction method and device based on importance characteristic graph sharing and computer equipment | |
| Ponomaryov et al. | Fuzzy color video filtering technique for sequences corrupted by additive Gaussian noise | |
| CN105513120A (en) | Adaptive rendering method based on weight local regression | |
| Xing et al. | Progressive path tracing with bilateral-filtering-based denoising | |
| KR102057395B1 (en) | Video generation method using video extrapolation based on machine learning | |
| CN114022371A (en) | Defogging device and defogging method based on space and channel attention residual error network | |
| CN113160081A (en) | Depth face image restoration method based on perception deblurring | |
| CN105957031A (en) | Projection filtering type fast spectral de-noising method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |