KR20210098656A - A rendering method based on machine learning driven adaptive sampling - Google Patents

Abstract

Disclosed is a rendering method based on driven adaptive sampling using machine learning. According to an embodiment of the present invention, the rendering method operated by a rendering system comprises the following steps of: training a rendering network in which a first network and a second network are combined using training data; reconstructing a radiance field by making the radiance field of an image to learn based on the trained first network and the second network; and performing rendering to generate an unbiased image using the reconstructed radiance field.

Description

A RENDERING METHOD BASED ON MACHINE LEARNING DRIVEN ADAPTIVE SAMPLING based rendering method using machine learning

The description below relates to rendering techniques.

Global illumination is very important for rendering realistic images. Rendering high-quality images with classic path tracing takes a lot of time, but there are many ways to improve performance.

One of the widely used methods is a filtering method that performs denoising of a noise input. The filtering method is very fast, but mathematically it cannot converge to the ground truth. Furthermore, another widely used method is unbiased path guidance, which first reconstructs the light field to sample a method similar to that of path tracing. Although slower than filtering, it is powerful for high-quality rendering, such as creating data sets for design, physics simulation, and deep learning.

The above methods may benefit from efficiently reconstructing the Light Field or Radiance Field.

It is possible to provide a rendering method and system based on adaptive sampling using machine learning.

A rendering method performed by a rendering system includes: training a rendering network in which a first network and a second network are combined using training data; reconstructing the radiance field by learning the radiance field of the image based on the trained first network and the second network; and performing rendering to generate an unbiased image using the reconstructed radiance field.

The trained first network and the second network are an R network and a Q network based on deep reinforcement learning, and the step of reconstructing the radiance field includes performing adaptive sampling and refinement on the radiance field of the image. In addition, the radiance field may mean a combination of an image space of a pixel processed in a tile and a directional space parameterizing an incident hemisphere of each pixel.

The step of reconstructing the radiance field comprises uniformly dividing the radiance field and applying a deep reinforcement learning-based Q-network to divide the directional space of the radiance field in a field block with a resolution higher than or equal to a preset standard. The method may include obtaining a quality value by performing an operation of refining or increasing the sample of the field block by a preset multiple.

The step of reconstructing the radiance field may include obtaining a refined hierarchical structure of the radiance field block by evaluating and specifying the radiance field through iteration of the operation a plurality of times.

The step of reconstructing the radiance field comprises reconstructing the radiance field in 4D space using an R network consisting of an image part and a directional part for processing an image space and a directional space, wherein the image part in the R network is , a geometric feature and a radiance feature are included in the image space, the geometric feature includes a feature including a normal, a position, and a depth, the radiance feature represents an incident radiance feature of a plurality of different blocks or directions, the A plurality of feature maps are obtained as a result of outputting the image portion of the R network, and the direction portion in the R network is individually convolved with the direction space CNN of each pixel after rearranging the feature maps in the order of the direction space ( can convolve).

The training includes constructing a rendering network in which the first network and the second network are combined, the first network is an R network, and the second network is Q based on deep reinforcement learning. It may be a network.

The training includes outputting a reconstruction result by training the first network using training data including a feature and a reference, and using a reward generated using the output reconstruction result and a feature of the training data. to train a second network, wherein the first network may be an R network, and the second network may be a deep reinforcement learning-based Q network.

The rendering system includes: a training unit for training using training data in a rendering network in which the first network and the second network are combined; a reconstruction unit configured to reconstruct the radiance field by learning the radiance field of the image based on the trained first network and the second network; and a rendering performing unit that performs rendering to generate an unbiased image by using the reconstructed radiance field.

The trained first network and the second network are an R network and a deep reinforcement learning-based Q network, and the reconstruction unit includes performing adaptive sampling and refinement on a radiance field of an image, and the radiance ( Radiance) field may mean a combination of an image space of a pixel processed in a tile and a directional space parameterizing an incident hemisphere of each pixel.

The reconstruction unit uniformly divides the radiance field, and refines the field block to a resolution higher than or equal to a preset standard in a field block in which the directional space of the radiance field is divided by applying a Q-network based on deep reinforcement learning or the field block A quality value may be obtained by performing an operation of increasing the samples of .

The reconstruction unit may obtain a refined hierarchical structure of the radiance field block by evaluating and specifying the radiance field through a plurality of iterations of performing the operation.

wherein the reconstruction unit reconstructs a radiance field in 4D space using an R network composed of an image part and a directional part for processing an image space and a directional space, wherein the image part in the R network includes a geometry in the image space a feature and a radiance feature are included, the geometric feature includes a feature comprising a normal, a position and a depth, the radiance feature represents an incident radiance feature of a plurality of different blocks or directions, and the image portion of the R network is A plurality of feature maps are obtained as a result of the output, and the direction part in the R network can be individually convolved with the direction space CNN of each pixel after rearranging the feature maps in the order of the direction space.

The training unit may include configuring a rendering network in which the first network and the second network are combined, the first network may be an R network, and the second network may be a deep reinforcement learning-based Q network. there is.

The training unit outputs a reconstruction result by training the first network using training data including a feature and a reference, and uses a reward generated using the output reconstruction result and a feature of the training data. and training two networks, wherein the first network may be an R network, and the second network may be a deep reinforcement learning-based Q network.

It is possible to provide a rendering method and system based on adaptive sampling using machine learning. Specifically, by training a rendering network (a Q network and an R network based on deep reinforcement learning), sampling and refinement of the radiance sample through the trained rendering network, and then reconstructing the radiance field in 4D space, the unbiased results can be obtained.

1 is a diagram for explaining an operation of a rendering system according to an embodiment.
2 is a diagram for explaining an operation of training training data in a rendering system according to an embodiment.
3 is a diagram for describing an R network of a rendering system according to an embodiment.
4 is a diagram for describing a Q network of a rendering system according to an embodiment.
5 is a diagram for explaining refining and resampling of a radiance field in a rendering system according to an embodiment.
6 is a block diagram illustrating a configuration of a rendering system according to an embodiment.
7 is a flowchart illustrating an adaptive sampling-based rendering method using machine learning in a rendering system according to an embodiment.
8 is an example for explaining reconstruction of a radiance field in a rendering system according to an embodiment.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings.

1 is a diagram for explaining an operation of a rendering system according to an embodiment.

A 4D incident radiance field can be defined as a combination of image (the space of pixels or shaded points) and directional space (the space of the incident hemisphere centered on the mean normal distribution of the group of shaded points).

The rendering system can partition the image space (pixels) into tiles and the directional space into bins called radiance field blocks. GPUs can efficiently perform network inference on inputs that are placed and ordered in single instruction multiple data (SIMD) operations. In this case, the pixel tile may represent one pixel without loss of generality. The radiance field block B ^j is defined by the boundary domain of the directional space and can be expressed as follows.

와

는 각각 B ^j 의 고도 및 방위각 경계이다. B ^j 는 방향 공간의 경계만 정의하는 반면, B ^j 는 때때로 특정 픽셀

의 방향 공간에 있는 j번째 파티션인

또는 타일의 모든 픽셀이 공유하는 방향 공간의 j번째 파티션인

를 간략하게 나타내는 데 사용될 수 있다. here,

Wow

are the elevation and azimuth boundaries of ^{B j , respectively.} B ^j only defines the bounds of directional space, whereas B ^j is sometimes a specific pixel

The jth partition in the direction space of

or the jth partition of the directional space shared by all pixels in the tile.

can be used to briefly represent

과 코사인 가중 천정각

을 절반으로 재귀적으로 분할하여 다양한 크기의 노드를 가진 래디언스 필드 계층으로 방향 공간을 적응적으로 분할할 수 있다. The rendering system is azimuthed by the Q network.

and cosine weighted zenith angle

We can adaptively partition the directional space into layers of radiance fields with nodes of various sizes by recursively dividing in half.

The rendering system may reconstruct the incident radiance field per tile instead of per pixel as the input of the network. The layer is thus built on the hemisphere of the tile's local frame, which can be defined from the average normal pixels of the pixels in the tile. After reconstructing the radiance field from the average local frame, we can project the result onto individual frames of pixels. In such a setup, there may be a special case where the hemisphere of an individual pixel has a particular direction of incidence (eg unexposed domains) that is not covered in the hemisphere of the average local frame. Reconfiguring a single domain using a network can degrade runtime performance. The rendering system according to the embodiment may allocate a uniform PDF to each unexposed domain (private domain) for unbiased sampling.

The rendering system may reconstruct the radiance field using the R network. In CNN, the radiance field reconstruction of the incident radiance block B ^{j for N is}

는 네트워크를 통과함에 따라 출력된 결과(예를 들면, 픽셀

에서의 B ^j 의 평균 입사 래디언스)이고, X는 B ^j 의 도메인에 있는 입사 래디언스 샘플이고,

는 보조 특징(예를 들면, 위치, 법선, 깊이)을 나타내고,

는 N의 훈련 가능한 무게 및 바이어스(bias)이다. 훈련 과정에서 네트워크는 오프라인 훈련 데이터가 공급될 수 있다.

를 최적화하기 위해 오프라인 훈련 데이터를 제공받아 결과와 GT 사이의 오류를 최소화할 수 있다.am. here,

is the output output as it traverses the network (e.g., pixels

And the average incident radiance) of B ^j in, X is the radiance incident on the sample in the domain B ^j,

denotes auxiliary features (e.g., position, normal, depth),

is the trainable weight and bias of N. During the training process, the network may be supplied with offline training data.

It is possible to minimize the error between the result and the GT by receiving the offline training data to optimize it.

Referring to FIG. 5 , the rendering system may refine and resample the radiance field. In the embodiment, an operation for subdividing the radiance field block into smaller blocks or increasing the number of samples may be repeatedly performed by treating the adaptive sampling as a dynamic field. The trained Q network can evaluate the value of each operation at runtime to guide the adaptive process.

The rendering system may perform a training process and a runtime rendering process. The GT data is the average value of the incident radiance field with directional resolution of 256X256, which can be used to train the R network and the Q network. In the training process, we can train the R network at first, and then use the trained R network to train the Q network. In order to minimize the loss of the Q network, the compensation may be evaluated according to the difference between the reconstruction result output from the R network and the GT.

After all network training is finished, the trained network can be used for runtime rendering. The rendering system can split the image into tiles and use the same radiance field hierarchy for every pixel in the tile. The rendering system can iteratively use the Q network to predict the value of the motion until a given stopping criterion is reached, and guide refinement or sampling of the radiance field layer. We can then use the R network to reconstruct the leaf nodes of the incident radiance field layer. If the application is filtered, the product of the BRDF and the reconstructed incident radiance field can be evaluated directly as a preview. If the desired result is an unbiased image, BRDF can be used to create an incident radiance field to generate a PDF for final rendering (guided path tracing).

Referring to FIG. 1 , (a) an image space may be divided into tiles. (b) The directional space shared by pixels within each tile is initialized to a uniform block, and arbitrary samples can be spread across each block. (c) The Q network can be used to evaluate the quality value of refinement (block division) or resampling (double the sample of a block) of the directional space. (d) The directional space can be specified as a layer of radiance fields by taking an optimal action. An R network can be used to reconstruct the incident radiance field.

3 is a diagram for describing an R network of a rendering system according to an embodiment.

는 각각의 픽셀

, 전체 픽셀 세트를 나타내는 이미지 방향 네트워크의 예이다. j와

는 각각 래디언스 필드 블록 B ^j 및 래디언스 필드 블록의 전체 세트를 각각 나타낸다.

는 이미지 공간 보조 특징의 특징맵이다.

은 B ¹ 에서 B ¹⁶ 까지의 래디언스 특징의 특쟁맵이고, F_d는 래디언스 필드 블록에 대응하는 특징맵의 방향을 나타내고, 최종 출력 결과는 예측된 입사 래디언스이다. 이미지 방향 네트워크의 첫 번째 부분은 이미지 공간에서 동작하고, 두 번째 부분은 각각의 개별 픽셀

의 방향 공간에서 동작한다. 재배열 동작은 첫 번째 부분의 출력을 방향 공간 입력으로 재구성될 수 있다. Referring to Figure 3, the subscripts i and

is each pixel

, is an example of an image orientation network representing the entire set of pixels. j and

denotes a radiance field block B ^j and a full set of radiance field blocks, respectively.

is the feature map of the image space auxiliary feature.

is the characteristic map of the radiance features from B ¹ to B ¹⁶ _{, F d} represents the direction of the feature map corresponding to the radiance field block, and the final output result is the predicted incident radiance. The first part of the image orientation network operates in image space, and the second part is for each individual pixel.

operates in the directional space of The rearrangement operation may reconstruct the output of the first part as a directional space input.

The rendering system presents the R network used for reconstruction of the radiance field. Technically it has some challenges and includes certain differences compared to image spatial filtering. First, the samples are scattered in multiple directions, so the input is sparse in individual blocks of radiance fields. Second, direct convolution in the 4D light source space requires high memory and computation. To solve this problem, we can search and compare four different CNNs of image, orientation, image orientation, and orientation image. Image networks use image space auxiliary features and perform image space convolution, whereas directional networks operate with features and convolutions in directional space. Image orientation network and orientation image network perform 4D light source convolution by connecting image and orientation spatial convolutions in different orders. In conclusion, the image direction network finally used as the R-network in the embodiment can achieve the lowest error in consideration of the set time budget.

The strategy chosen for reconstruction of the radiance field in the rendering system is the image orientation network. First, convolve the image space auxiliary features with incident radiance field to learn the directional feature map, and then rearrange these feature maps into the directional space to explore the coherence of the directional space. )can do.

과 관련된 특징맵을 나타내기 위해

를 사용할 수 있다. 이미지 부분은 일부 이미지 공간 보조 특징맵

와 래디언스 특징맵

(래디언스의 평균, 분산 및 기울기)을 입력으로 사용할 수 있다. 출력 결과는 방향 공간 특징맵

이다.

는 이미지 공간에 설정된 전체 픽셀을 나타내고, 위첨자 j는 j번째 래디언스 필드 블록 B ^j 를 가리킨다. 예를 들면,

는 첫 번째 래디언스 필드 블록 B¹의 이미지 공간에 정의된 특징맵을 나타낸다. 수작업으로 설계한 특징에 의존하는 대신, 방향 부분은 이미지 부분으로부터 학습된 방향 특징맵

을 입력으로 취하여 모든 래디언스 필드 블록의 래디언스 예측을 동시에 컨벌브(convolve)할 수 있다.The image direction network may be divided into an image portion and a direction portion. Direction block i and pixel tiles

To show a feature map related to

can be used The image part is some image space auxiliary feature map

and radiance feature map

(mean, variance and slope of radiance) can be used as input. The output result is a directional space feature map.

am.

denotes the entire pixel set in image space, and the superscript j denotes the j-th radiance field block B ^j . For example,

denotes the feature map defined in the image space of the first radiance field block B ^{1 .} Instead of relying on hand-designed features, the directional part is a directional feature map learned from the image part.

can be taken as input to convolve the radiance prediction of all radiance field blocks simultaneously.

과 래디언스 특징맵

이 포함될 수 있다. 이미지 공간 보조 특징은 대부분 기하학적 특징, 예를 들면 표면 법선(채널 3개), 위치(채널 3개), 깊이(채널 1개)이다. 래디언스 특징은 희박한 샘플, 예를 들면, 각 래디언스 블록 B ^j 의 평균적인 입사 래디언스

에서 나온 것이다. 또한, 각 특징의 픽셀당 분산과 이미지 공간 그래디언트(gradients)를 제공하여 훈련을 용이하게 하고 중요한 세부 사항을 강조한다. 예를 들면, 기하학적 특징은 다음과 같이 총 26개의 채널이 있다.The initial input to the network is an image space auxiliary feature map.

and radiance feature map

may be included. Image space auxiliary features are mostly geometric features, eg surface normals (3 channels), position (3 channels), and depth (1 channel). The radiance characteristic is the average incident radiance of a sparse sample, e.g., each radiance block B ^{j .}

it comes from It also provides per-pixel variance and image space gradients of each feature to facilitate training and highlight important details. For example, the geometric feature has a total of 26 channels as follows.

- 3 channels for the mean normal normal, 1 channel for the mean variance of the normal, and 6 channels for the slope of the mean normal.

- 3 channels for average position, 1 channel for average variance of positions, 6 channels for gradient of average position

- 1 channel for average depth, 1 channel for depth dispersion, 2 channels for average depth gradients

- 2 channels for gradients of the average radiance of all blocks

The radiance feature has a total of 4 channels and is composed as follows.

- 3 channels for radiance

- 1 channel for average variance of radiance

All of these features can be accumulated as a feature map as tensors to be combined in image space. The radiance characteristic for a radiance field block may be initialized to 0 if there are no samples in the block.

을 추출한다.The first part of the image-oriented network contains 6 canonical convolutional hidden layers with 100 kernels of size 5x5. A convolutional layer can be used to prevent overfitting compared to a fully connected layer. Such a structure can be effective in the image spatial filtering problem. In each layer, some trainable kernel is applied to the output of the previous layer to generate a linear convolution from nearby pixels. The result of the convolutional output is added as a trainable bias, and it can be applied to the element-by-element nonlinear activation function as the output of the current layer. The embodiment uses the rectified ReLU activation function for all networks. The six layers integrate the input tensors so that for each block of radiance fields, a 4X4 directional spatial feature is created for each block of radiance fields.

to extract

The second part of the network can integrate the feature maps to produce the final output result. To perform convolution in directional space, we can rearrange the output of the image part to rearrange the directional space as an input tensor representing 12 directional features for a block of radiance fields of 4X4. This allows you to test different features and use 12 features to balance accuracy and runtime performance.

이며, 여기서

는 래디언스 필드 블록의 집합을 나타내고

는 픽셀을 나타낸다. 각 픽셀마다 래디언스 필드 블록이 다르기 때문에 서로 다른 픽셀에 걸쳐 있는 이러한 래디언스 필드 블록은 4D 래디언스 필드로 간주될 수 있다. 이러한 결과를 MC 추정을 위한 PDF로 처리함으로써, 편향되지 않은(unbiased) 렌더링을 수행할 수 있다. 또한, 이미지 방향 네트워크 이외의 이미지 네트워크, 방향 네트워크, 방향 이미지 네트워크가 사용될 수 있다. Note that the rearrangement operation only rearranges the features to enable the end-to-end training of the whole network in a completely different way. The network can then connect the inputs individually using 50 kernels of 3X3 and 3 standard convolutional hidden layers using the final output layer. The output of the network is the predicted or filtered incident radiance of blocks of 16 radiance fields at each pixel.

and where

represents the set of radiance field blocks and

represents a pixel. Since each pixel has a different block of radiance fields, these blocks of radiance fields that span different pixels can be considered 4D radiance fields. By processing these results as PDFs for MC estimation, unbiased rendering can be performed. Also, an image network, a directional network, and a directional image network other than the image directional network may be used.

To evaluate these reconstruction networks, they can be trained using the same training set and then compared with the same sample setup. 8 shows the results of reconstruction of the radiance field block and the means of all blocks. Stairs scene rendered at 64 samples per pixel, 4X4 radiance field resolution, and 500X500 image resolution. The top row is the reconstruction result of one block of radiance field that can be used for route guidance, and the bottom row is the average of all the radiance field blocks, i.e. the individual blocks containing the block shown in the first row, so that the surrounding occlusion or preliminaries are integrated. can be used for viewing.

In the stair scene outside the training set of Figure 8, 64 samples per pixel, 4X4 radiance field resolution, 500X500 image resolution may be used. The individual block results are in radiance field space and can be used to composite PDFs for path guidance. The average result is an image space result that can be used for radiance caching or previewing. As pointed out in the results, orientation networks and image networks create artifacts or overly blurry detail because they utilize information only in image space or orientation space. However, image orientation network and orientation image network utilize both space and show improved results. Nevertheless, since directional image networks output results directly in image space, whereas image directional networks output results from blocks of radiance fields, smoother equal samples per pixel (SPP) can be derived as directional image results.

In embodiments, the rendering system may select the image orientation network as the final reconstruction network. Hereinafter, a proposed reinforcement learning-based adaptive sampling method that gradually improves the overall rendering quality in consideration of the selected image direction reconstruction network will be described.

The rendering system may perform sampling of the radiance field through reinforcement learning. The R network can take as input a radiance field hierarchy with a fixed number of nodes (i.e. 4X4 nodes). However, the strategy of adaptively refining the layer can improve its adaptability to different scenarios. In an embodiment, a DRL-based Q network may be used to adaptively induce sampling and refinement of the radiance field layer.

When building a hierarchy, two factors are important. The first factor is the hierarchical structure, i.e. how to discretize the radiance field. Higher grid resolution can effectively capture high-frequency lighting features, but with overhead. The second factor is the adaptive sample distribution, which places more samples (i.e., greater sample density) into the noisy region (blocks or nodes) to reduce reconstruction errors. In embodiments, a large amount of offline data may be used to train the DNN to guide the process. We decompose the refinement process into a series of minimal operations and choose to adapt DRL, an unsupervised learning technique, for training the Q network to guide the process and minimize reconstruction errors.

The adaptive sampling and segmentation approach can be considered as a dynamic process and the problem can be solved by DRL. Specifically, deep Q learning can be applied to select the next best sampling or segmentation operation for a block of radiance fields in a given state. Q-learning can typically involve agents located in specific environments. Given s, the agent performs an action a, which leads to a new state s' and a reward r'. In this framework, neural networks can predict long-term values of behaviors in different states after learning from rewards. In the adaptive sampling scenario according to the embodiment, it is possible to train a neural network that receives global radiance field information (eg, geometry information, radiance sample, and radiance field layer) as an input state and predicts the quality value of the next operation as an output. First, the operation, quality value, state, and network structure can be defined as follows.

Two different operations can be used for the radiance field block in an epoch. First, resampling the block by doubling the sample density per block (number of samples per block), then subdividing the radiance field block into 4X4 new blocks by equally splitting each axis, adding some new samples to average per block Keep the number of samples. Maintaining the sample density per block is to prevent degradation of reconstruction quality due to sparse samples. These two actions affect the reconstruction result in different ways. By increasing the number of samples, the dispersion of the radiance feature can be reduced to suppress noise. Refining increases the resolution of the grid, allowing it to capture high-frequency details.

A quality value Q and motion compensation r are defined for each radiance field block. In the case ^{of the radiance field block B j} in the pixel (or tile), the quality value of the operation a acting in the state s ^{j may be defined by Equation 1 below.}

Equation 1:

은 각각 0과 1 사이의 붕괴 파라미터로, 근시안적 사고와 원시안적 사고를 유발한다. 직관적으로 주어진 래디언스 필드 계층과 샘플 분포로 표현되는 상태 s ^j 를 고려할 때, 식은 동작 a에 기초하는 동작의 최대 품질 가치를 찾으려 할 뿐만 아니라 다음 가능한 동작 a'을 다시 고려함으로써 추가 단계를 통해 시도한다. 이러한 식을 여러 단계로 평가하는 것은 상당한 오버헤드를 수반하며 행동의 장기적인 영향을 학습하는 측면에서 어려움을 야기한다. 이에, 두 가지 추가 단계를 고려하여 동작의 Q값을 추정하여 동작의 조합 수를 제한하여 추정 문제를 단순화할 수 있다. 수학식 2를 다음과 같이 계산한다.where s ^j and s ' ^j correspond to the state before and after the action a occurs, a' is the next possible action, and r(s ^j , a) represents the compensation of the action,

is a decay parameter between 0 and 1, respectively, causing myopic thinking and farsighted thinking. Given the intuitively given layer of radiance field and the state s ^j represented by the distribution of samples, the equation not only tries to find the maximum quality value of the action based on action a, but also tries through additional steps by reconsidering the next possible action a' . Evaluating these equations in multiple steps involves significant overhead and creates difficulties in terms of learning the long-term effects of behavior. Accordingly, it is possible to simplify the estimation problem by estimating the Q value of the motion in consideration of two additional steps and limiting the number of combinations of motions. Equation 2 is calculated as follows.

Equation 2:

The reward r(s ^j , a) can be defined as

를 훈련 및 추론하고 Q 값을

로 대체할 수 있다. actionvalue 함수를 사용하여, Q'(s ^j , a)와 그 예측값 Q^' _pre (s ^j , a) 사이의 제곱 오차를 학습 과정에서의 손실로 사용할 수 있다. 런타임에서 적응형 샘플링 동작을 위해 Q ^j (s ^j , a)를 복구할 수 있다.where E ^j (s ^j ) is the reconstruction error of block B ^j , that is, the integration of the absolute difference between the reconstructed radiance in the domain of ^{B j and the ground-truth radiance.} Also, the scalar range of r(s ^j , a) depends on the integration domain (region of image space) of B ^j called dom( j ). Since the network is created independent of the scalar range, its robustness is improved, so it is a normalized replacement reward.

train and infer the value of Q

can be replaced with Using the actionvalue function, ^{the squared error between Q'(s j} , a) and its predicted value Q ^' _pre (s ^j , a) can be used as a loss in the learning process. ^{We can recover Q j} (s ^j , a) for adaptive sampling operation at runtime.

The full learning algorithm is described in Algorithm 1, given the definition, quality value and state of the behavior.

The agent may perform an operation (eg resampling or refinement) and observe the state and reward of the ^{radiance field block B j as an input of the learning operation.} First, the reproduction memory D and the state s are initialized. Replay memory is used to store a set of states, actions, and rewards. Based on regenerative memory, instead of only observing short-term effects, we can delay the update of the Q network to learn long-term rewards.

의 확률은 알고리즘이 가장 가치 있는 동작을 취하지 않고 두 동작에서 임의의 동작을 선택한다는 것이 존재한다. 동작을 취하면, 네트워크의 지연된 업데이트에 대한 보상 r과 새로운 상태 s'이 재생 메모리 D에 기록된다. 반복이 끝나면 메모리에서 무작위로

한 쌍을 선택하여 손실을 계산하고 Q네트워크를 훈련시킨다. 응답 메모리를 통해 학습 단계는 경험을 획득하는 과정과 분리되어 순차적 학습 샘플의 의존성을 줄이고 컨버전스(convergence)를 높이는 데 도움이 된다.For each learning iteration, the algorithm first calculates the Q-values of the two actions and performs the most valuable action. When it comes to increasing the robustness of the algorithm and exploring a wide range,

The probability exists that the algorithm chooses a random action from the two actions without taking the most valuable action. When an action is taken, the compensation r for the delayed update of the network and the new state s' are written to the replay memory D. At the end of the iteration, randomly from memory

Choose a pair, compute the loss, and train the Q network. With response memory, the learning phase is separated from the process of acquiring experience, which helps to reduce dependence on sequential learning samples and increase convergence.

4 is a diagram for describing a Q network of a rendering system according to an embodiment.

, 래디언스 필드 블록 B ^j 의 래디언스 특징맵

, 새로 도입된 계층 특징맵

, 주변 방향-공간 정보를 캡처하기 위해 B ^j 의 상위 래디언스 필드 블록의 래디언스 특징을 포함하는 부모 래디언스 특징맵

등이 포함된다. 이러한 특징은 모두 이미지 공간에 정렬되어 있다. G와 R의 의미는 R 네트워크에서 설명한 바와 같이 의미가 동일하다.The Q network can take as input information about pixel i and the radiance field block B ^j , and predict the compensation of the two operations it can perform on ^{B j .} Q networks use image space convolution and fewer hidden layers for performance compared to image-directed R networks. During the adaptive segmentation operation, hierarchical structure information can be used to access information from nearby radiance field blocks. Specifically, two hidden convolutional layers with 50 kernels of 3X3 first convolve ^{the features of the radiance field block B j} in image space, and then the fully connected layer can be used to predict the final output result. . The input is an image space auxiliary feature map.

, the radiance feature map of the radiance field block B ^j

, the newly introduced hierarchical feature map

, the parent radiance feature map containing the radiance features of the parent radiance field block of ^{B j} to capture the surrounding direction-spatial information.

etc. are included. All of these features are aligned in image space. The meanings of G and R are the same as described in the R network.

The newly used layer feature describes the current layer state within the scope ^{of B j .} In other words, it describes the number of samples and the maximum level of the ^{radiance field block B j .} Hierarchical levels represent the granularity of nodes. For example, the root node is level 0 and its descendants are level 1. The maximum level of the layer B ^j may be currently defined as the highest hierarchical level of the radiance field blocks in the domain of B ^j. This feature indicates how, given the central pixel i, the surrounding pixels ^{refine B j .} If the adjacent pixels have the ^{maximum hierarchical level within B j} , the center pixel must also see the deep level of ^{B j .}

The R network uses multiple (eg, 7) image space convolutional layers to efficiently remove noise, whereas the Q network uses fewer image space convolutional layers for two reasons. First, since DRL requires more data and time for training, the number of parameters needs to be limited to a reasonable size. Second, the performance of the Q network needs to be considered because it is frequently evaluated during the adaptive sampling operation. Thus, we can test the training loss of multiple layers and select three layers.

The rendering system will be described in terms of how to train and use the R network and the Q network. As shown in Figure 1, adaptive sampling and rendering involves three steps given a trained network. First, a trained Q network can be used to guide the adaptive sampling process and refine field blocks. A layer of radiance field blocks may be created. Second, we can use the trained R network to reconstruct the radiance coming from the layer. Finally, we apply the reconstruction result to the final rendering.

Referring to FIG. 2 , it is a diagram for explaining an operation of training training data.

Given that the Q network depends on the R network, we can train the R network first. For example, assume that all variables in the network are initialized and updated with a learning rate of 0.0001. Also, initializer and optimizer are available in TensorFlow. In the embodiment, it is assumed that the data set includes 20 scenes. To enrich the training data set, about 300 scenes can be generated by randomly changing the views, lighting, materials, and textures of the scenes. We can also randomly select five values: samples per block (SPB) and radiance field resolution per scene to tune the R network to different ranges. The SPB configuration may be selected from (1, 2, 4, 8, 16). Resolution radiance field block may be selected from (16 ^1, 16 ^2, 16 ^3, 16 ^4, 16 ^5).

Once the R network converges, it can be used to train the Q network. In other words, we can compute the quality value of the same training data set. For example, nine examples of the training data set are available. The Q network can be trained via the classical probabilistic RL algorithm (Algorithm 1), which simulates the real data distribution via replay memory and can avoid data inconsistency. However, overhead may occur in the probabilistic training process. To accelerate deep reinforcement training, we add a deterministic step to pretrain the Q network by increasing the resolution or sample density with different configurations and then fine-tuning the pretrained Q network using a probabilistic algorithm (Algorithm 1). Can be used.

The rendering system may perform adaptive sampling by applying the trained Q network. The pseudo code for adaptive sampling is shown as Algorithm 2.

Divide the image space into 25x25 tiles, each tile can be processed in parallel. Since Q networks convolve the features of surrounding pixels, they are relatively efficient when their inputs are batched. Then we can combine all the tile T pixels together using the pixel's average Q value to determine its behavior. Accordingly, pixels of the same tile _{share the same hierarchical structure H T} and can be fused at the same time. Although pixels of the same tile share the same hierarchical structure H _T , that is, the same division in the radiance field, each pixel has a different reconstructed incident radiance field value than the other pixels in providing a custom reconstruction of each pixel in the tile. has A pseudocode subscript T is used to indicate that a block or action is shared by the pixels of the tile T .

)이다. 래디언스 필드를 깊이 1(4X4 = 16 래디언스 필드 블록)로 균일하게 분할하여 계층을 초기화하며, 각 블록에서 하나의 샘플로 균일하게 샘플을 추출한다. 그런 다음 Q 네트워크를 사용하여 H _T 에서 정제 또는 리샘플링 블록의 품질값을 예측한다.The quality value of tile Q _T ^j is simply the sum of the quality values of all pixels (

)am. The layer is initialized by uniformly dividing the radiance field into depth 1 (4X4 = 16 blocks of radiance field), and samples are uniformly sampled from each block with one sample. Then, the Q network is used to predict the quality value of the refinement or resampling block in _{H T .}

The iteration of the adaptive sampling operation is as follows. Using the heap associated with the action and Q value, we can select the higher (ie, most valuable) action a ^j _T and perform the action on the tile T . When the refinement operation is selected, _{a new block B T} ^sub(j) is used ^{to replace B j} with a new quality value. If resampling is selected, only the quality value of ^{B j is updated after performing the operation.}

이다. 둘째, 최대 샘플 밀도(블록 당 샘플)는 16이다. 셋째,

에 대한 새로운 동작의 품질값이 추정된 irraiance의

보다 작을 때, 예상 효과가 작을 것이기 때문에

에 대한 새로운 샘플을 분배하지 않는다. In order to limit the practical range of the training data, which is useful for limiting memory and computational cost, we can set some rules. First, the maximum number of iterations per tile is set to 100, and the maximum depth of the layer is set to 5. In other words, the smallest block is the size of the radiance field.

am. Second, the maximum sample density (samples per block) is 16. third,

The quality value of the new motion for

Since the expected effect will be small when smaller than

Do not distribute new samples for

The rendering system may reconstruct the radiance field block of _{layer H T} using the image direction R network. You can simply use the reconstructed incident radiance field to evaluate and integrate the result and BRDF of incident radiance to generate a quick preview. To render an unbiased image, the reconstructed radiance field and BRDF are processed into two PDFs to generate sampling directions. These two samplers can then be combined via multiple importance sampling (MIS).

Specifically, MIS samples the BRDF and two independent estimators from the reconstructed radiance field, and then the results of the two estimators are weighted and summed to the final rendering result, taking into account the BRDF and the incident radiance.

BRDF samples can be analytically extracted from the cumulative distribution function. Reconstructed radiance field samples can be generated by first selecting a block of radiance fields from a discrete PDF and then proportionally sampling points in the block according to cosine weighting terms. Currently, the focus is on reconstructing a high-quality incident radiance field at the first bounce shadow point. Because multi-bounce vertices are sparse, they are not compatible with the input format of the network. For multi-bounce vertices, one can switch to standard multi-importance sampling or adapt to other photon guidance methods if lighting is complex. For example, you can spread out a million photons simply to guide the sample direction. Specifically, by dividing the unit square of the hemisphere into distinct regions, the photon contribution of each region can be accumulated for path guidance.

6 is a block diagram illustrating a configuration of a rendering system according to an embodiment, and FIG. 7 is a flowchart illustrating an adaptive sampling-based rendering method using machine learning in the rendering system according to an embodiment.

The processor included in the rendering system 100 may include a training unit 610 , a reconstruction unit 620 , and a rendering performer 630 . The processor and its components may control the rendering system to perform steps 710 to 730 included in the adaptive sampling-based rendering method using machine learning of FIG. 7 . In this case, the processor and components of the processor may be implemented to execute instructions according to the code of the operating system and the code of at least one program included in the memory. Here, the components of the processor may be expressions of different functions performed by the processor according to a control command provided by the program code stored in the rendering system 100 .

The processor may load the program code stored in the file of the program for the rendering method based on adaptive sampling using machine learning in the system into the memory. For example, when a program is executed in the rendering system 100 , the processor may control the rendering system to load the program code from the program file into the memory according to the control of the operating system.

In step 710 , the training unit 610 may train a rendering network in which the first network and the second network are combined using training data. The training unit 610 may configure a rendering network in which the first network and the second network are combined. In this case, the first network and the second network are R networks and Q networks based on deep reinforcement learning. The training unit 610 outputs a reconstruction result by training the first network using training data including features and references, and uses the compensation generated using the output reconstruction result and the features of the training data to provide a second You can train the network.

In operation 720 , the reconstruction unit 620 may reconstruct the radiance field by learning the radiance field of the image based on the trained first network and the second network. The reconstruction unit 620 may perform adaptive sampling and refinement on the radiance field of the image. The reconstruction unit 620 uniformly divides the radiance field, and in the field block in which the directional space of the radiance field is divided by applying a Q-network based on deep reinforcement learning, refines the field block to a resolution higher than or equal to a preset standard or a field block A quality value may be obtained by performing an operation of increasing the samples of n by a preset multiple (eg, doubling). The reconstruction unit 620 may obtain a refined hierarchical structure of the radiance field by evaluating and specifying the radiance field through a plurality of operations. The reconstruction unit 620 may reconstruct the radiance field by using an R network composed of an image part and a directional part for processing the image space and the directional space.

In operation 730 , the rendering performer 630 may perform rendering to generate an unbiased image using the reconstructed radiance field.

The device described above may be implemented as a hardware component, a software component, and/or a combination of the hardware component and the software component. For example, devices and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA). , a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions, may be implemented using one or more general purpose or special purpose computers. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For convenience of understanding, although one processing device is sometimes described as being used, one of ordinary skill in the art will recognize that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that can include For example, the processing device may include a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as parallel processors.

The software may comprise a computer program, code, instructions, or a combination of one or more thereof, which configures a processing device to operate as desired or is independently or collectively processed You can command the device. The software and/or data may be any kind of machine, component, physical device, virtual equipment, computer storage medium or device, to be interpreted by or to provide instructions or data to the processing device. may be embodied in The software may be distributed over networked computer systems, and stored or executed in a distributed manner. Software and data may be stored in one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and carry out program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like.

As described above, although the embodiments have been described with reference to the limited embodiments and drawings, various modifications and variations are possible from the above description by those skilled in the art. For example, the described techniques are performed in a different order than the described method, and/or the described components of the system, structure, apparatus, circuit, etc. are combined or combined in a different form than the described method, or other components Or substituted or substituted by equivalents may achieve an appropriate result.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (14)

A rendering method performed by a rendering system, comprising:
training a rendering network in which the first network and the second network are combined using the training data;
reconstructing the radiance field by learning the radiance field of the image based on the trained first network and the second network; and
performing rendering to generate an unbiased image using the reconstructed radiance field;
Rendering method including . According to claim 1,
The trained first network and the second network are R networks and Q networks based on deep reinforcement learning,
Reconstructing the radiance field comprises:
performing adaptive sampling and refinement on the radiance field of the image;
including,
The radiance field means a combination of an image space of a pixel processed in a tile and a directional space parameterizing an incident hemisphere of each pixel. 3. The method of claim 2,
Reconstructing the radiance field comprises:
In a field block in which the radiance field is uniformly divided and the directional space of the radiance field is divided by applying a Q-network based on deep reinforcement learning, the field block is refined to a resolution higher than or equal to a preset standard, or a sample of the field block is obtained. Acquiring a quality value by performing an operation to increase by a set multiple
Rendering method including . 4. The method of claim 3,
Reconstructing the radiance field comprises:
Evaluating and specifying the radiance field through repeating the operation a plurality of times to obtain a refined hierarchical structure of the radiance field block
Rendering method including . According to claim 1,
Reconstructing the radiance field comprises:
Reconstructing the radiance field in 4D space using an R network consisting of image parts and orientation parts for processing image space and orientation space.
including,
The image part in the R network includes a geometry feature and a radiance feature in the image space,
wherein the geometric features include features including normals, positions, and depths;
The radiance feature represents an incident radiance feature of a plurality of different blocks or directions, and a plurality of feature maps are obtained as a result of outputting the image portion of the R network,
The direction part in the R network rearranges the feature map in the order of direction space, and then individually convolves with the direction space CNN of each pixel. According to claim 1,
The training step is
configuring a rendering network in which the first network and the second network are combined
including,
The first network is an R network, and the second network is a deep reinforcement learning-based Q network. According to claim 1,
The training step is
As the first network is trained using training data including features and references, the reconstruction result is output, and the second network is trained using the reward generated using the output reconstruction result and the features of the training data. step to let
including,
The first network is an R network, and the second network is a deep reinforcement learning-based Q network. In the rendering system,
a training unit that trains a rendering network in which the first network and the second network are combined using training data;
a reconstruction unit configured to reconstruct the radiance field by learning the radiance field of the image based on the trained first network and the second network; and
A rendering performing unit that performs rendering to generate an unbiased image using the reconstructed radiance field
Rendering system that includes. 9. The method of claim 8,
The trained first network and the second network are R networks and Q networks based on deep reinforcement learning,
The reconstruction unit,
performing adaptive sampling and refinement on the radiance field of the image;
The radiance field means a combination of an image space of a pixel processed in a tile and a directional space parameterizing an incident hemisphere of each pixel. 10. The method of claim 9,
The reconstruction unit,
In a field block in which the radiance field is uniformly divided and the directional space of the radiance field is divided by applying a Q-network based on deep reinforcement learning, the field block is refined to a resolution higher than or equal to a preset standard, or a sample of the field block is obtained. To obtain a quality value by performing an operation to increase by a set multiple
Rendering system, characterized in that. 11. The method of claim 10,
The reconstruction unit,
To obtain a hierarchical structure of a refined radiance field block by evaluating and specifying the radiance field through a plurality of iterations of performing the operation
Rendering system, characterized in that. 9. The method of claim 8,
The reconstruction unit,
reconstructing the radiance field in 4D space using an R network consisting of image parts and orientation parts for processing image space and orientation space;
The image part in the R network includes a geometry feature and a radiance feature in the image space,
wherein the geometric features include features including normals, positions, and depths;
The radiance feature represents an incident radiance feature of a plurality of different blocks or directions, and a plurality of feature maps are obtained as a result of outputting the image portion of the R network,
The directional part in the R network rearranges the feature maps in the order of directional space and then individually convolves with the directional space CNN of each pixel. 9. The method of claim 8,
The training department
and configuring a rendering network in which the first network and the second network are combined,
The first network is an R network, and the second network is a deep reinforcement learning-based Q network. 9. The method of claim 8,
The training department
As the first network is trained using training data including features and references, the reconstruction result is output, and the second network is trained using the reward generated using the output reconstruction result and the features of the training data. including making
The first network is an R network, and the second network is a deep reinforcement learning-based Q network.

Images