KR102555165B1 - Method and System for Light Field Synthesis from a Monocular Video using Neural Radiance Field - Google Patents

Abstract

A method and system for synthesizing a light field based on neural radiance in monocular video are presented. In the neural radiance-based light field synthesis method in monocular video proposed in the present invention, the step of pre-processing data for learning by using the RGBD image of the input image to improve the data for the visible region through the pre-processing unit, through the modeling unit Performing NeRF (Nueral radiance field)-based scene modeling using a color image and depth image, expanding data using the color image and depth image input as the input through a learning unit, and depth for the depth image through a learning unit and learning to reduce an error of NeRF-based scene modeling in an area in which the value is continuously changed and no texture exists.

Description

Method and System for Light Field Synthesis from a Monocular Video using Neural Radiance Field}

The present invention relates to a method and system for synthesizing a light field based on neural radiance in monocular video.

Research on synthesizing virtual viewpoint images from one or more input images has received a lot of interest and research in that there are various application directions in the immersive media environment along with contents such as VR/AR, 3D display, and hologram.

In general, virtual view generation uses relationships between pixels of images from input images and camera parameters, or uses additional information such as depth information and point clouds for estimating insufficient geometric information to express a scene. Therefore, existing systems for constructing immersive media use a method of collecting and processing data by using a plurality of cameras and arranging them to suit a specific purpose. As a result, there are several limitations in that it has various problems, such as an increase in cost due to the configuration of multiple cameras, a synchronization problem between multi-view images, and an increase in the amount of calculation required for estimating the relationship between multi-view images.

In order to solve this problem, hardware that simultaneously acquires multi-viewpoint images has been proposed, and typical examples include a binocular camera and a light field camera. In addition, researches that can generate images from various viewpoints that meet the existing purposes while minimizing the number of cameras are being pursued due to the rapid growth of deep learning architectures from the development of graphics processing units (GPUs) and large-scale data. has been carried out

A number of virtual viewpoint studies using deep learning enable the creation of virtual viewpoints using a deep learning framework, from multi-viewpoint based virtual viewpoint creation to even monocular images. In detail, studies that show high performance in terms of speed or image quality compared to previous scene representation methods by modeling scenes by learning by combining learning-based networks with existing scene representation methods or by proposing new scene representation methods. was also performed It can be seen that the new scene representation method can be robust to generalization and error compared to traditional computer vision techniques that estimate and optimize the relationship between input images.

[1] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng, "NeRF: Representing scenes as neural radiance fields for view synthesis," in Proc. of European Conference on Computer Vision, pp. 405-421, Springer, 2020. [2] K. Deng, A. Liu, J.-Y. Zhu, and D. Ramanan, "Depth-supervised NeRF: Fewer views and faster training for free," in Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12882-12891, 2022.

A technical problem to be achieved by the present invention is a method and system for synthesizing a light field based on neural radiance in a monocular video that synthesizes a light field using virtual view image synthesis techniques by using forward moving video frames as a multi-view image. is providing Lightfield is effective in evaluating the necessity of virtual viewpoint synthesis research and the evaluation of synthesized results in that it can build a immersive media environment with the advantage of having structured multi-viewpoint images. In the present invention, a method of generating a light field through two scene expression methods is proposed.

In one aspect, the neural radiance-based light field synthesis method in monocular video proposed in the present invention preprocesses data for learning by using RGBD images of input images to enhance visible region data through a preprocessor. Step, performing NeRF (Nueral radiance field)-based scene modeling using a color image and depth image through a modeling unit, expanding data using the color image and depth image input as the input through a learning unit, and through a learning unit and learning to reduce an error of NeRF-based scene modeling in an area in which a change in a depth value for a depth image is continuously made and no texture exists.

The step of pre-processing the data for learning by using the RGBD image of the input image to improve the data for the visible region through the pre-processing unit obtains a virtual viewpoint image for the visible region using the DIBR (Depth Image Based Rendering) method After creating a masking to determine whether the corresponding rays are in the visible region, they are input to the NeRF to be learned later, and the rays of the masked part are excluded from learning.

In the step of performing NeRF (Nueral radiance field)-based scene modeling using a color image and a depth image through the modeling unit, the sampling rate for the visible region is increased and data is estimated from the input according to the video direction in the case of the occlusion region, A light field dataset is created by acquiring large-scale data with a wide baseline between SAIs (Sub-aperture images) of the light field from the rendering-based input image.

The step of expanding data using the color image and the depth image input through the learning unit generates viewpoints surrounding the input viewpoint using a depth-based rendering method, and warps the scope with DIBR to have a wide baseline. It expands the data by selecting a certain portion of SAIs without generating all SAIs.

In another aspect, the neural radiance-based light field synthesis system in the monocular video proposed in the present invention preprocesses data for learning by utilizing the RGBD image of the input image to improve data for the visible region. A modeling unit that performs scene modeling based on a neutral radiance field (NeRF) using a color image and a depth image, expands data using the input color image and depth image, and changes the depth value for the depth image and a learning unit that learns to reduce an error of NeRF-based scene modeling in an area that is continuous and does not have a texture.

In a monocular video according to embodiments of the present invention, by using the frames of a video moving in a forward direction as a multi-view image through a neural radiance-based light field synthesis method and system, a light field is generated using virtual view image synthesis techniques. can be synthesized. Lightfield is effective in evaluating the necessity of virtual viewpoint synthesis research and the evaluation of synthesized results in that it can build a immersive media environment with the advantage of having structured multi-viewpoint images. In the present invention, a method of generating a light field through two scene expression methods is proposed.

1 is a diagram illustrating an entire framework for neural radiance-based light field synthesis in monocular video according to an embodiment of the present invention.
2 is a flowchart illustrating a method of synthesizing a light field based on neural radiance in a monocular video according to an embodiment of the present invention.
3 is a diagram illustrating a color image and a depth image received as inputs during a real environment experiment according to an embodiment of the present invention.
4 is a diagram showing the configuration of a neural radiance-based light field synthesis system in monocular video according to an embodiment of the present invention.
5 is a diagram showing an example of SAI of a synthesized result according to an embodiment of the present invention.
6 is a diagram showing a light field generated using a network according to an embodiment of the present invention according to an embodiment of the present invention.
7 is a diagram showing KITTI data broad baseline qualitative results according to an embodiment of the present invention.

The present invention proposes an easily obtainable light field data generation method by synthesizing a light field having a wide baseline, and proposes a network for generating a light field from a monocular video through a large-scale light field video acquired through the method. The proposed network is a framework that combines the characteristics of two scene representation methods by combining multiplane-images (MPIs) and depth estimation, which are highly useful in existing virtual view synthesis.

According to an embodiment of the present invention, a light field image video composed of 8x8 SAIs is generated using GTAV, a video game based on realistic rendering.

According to an embodiment of the present invention, a light field dataset is created by acquiring data having a wide baseline between sub-aperture images (SAIs) of a light field from a photorealistic rendering-based video game on a large scale.

In addition, by combining multi-plane based and DIBR (Depth image-based rendering) techniques, a high-quality light field is synthesized while improving the rendering range, and the existing light field is synthesized through NeRF (Neural radiance field)-based scene modeling method. Unlike the synthesis method, a light field with a wide baseline is synthesized. Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

,

)으로 방출하는 빛, 각 점 (x, y, z)마다 빛이 얼마나 통과할 것인지를 결정하는 밀도를 출력으로 하는 연속적인 5D 함수로 표현하였다.NeRF means expressing a scene itself as a network based on information input from various viewpoints. In the NeRF modeling process, the network uses a single 5D coordinate as input and models the scene using a fully connected network that outputs light that varies depending on the volume density and light at a point in space. . In the case of synthesizing a virtual viewpoint using a trained network, the viewpoint is synthesized using the output value of 5D coordinates along the camera light, and the color and density of the sampled points are synthesized into an image using a traditional volume rendering technique. In the case of NeRF, which proposed such a scene representation method for the first time, a static scene is mapped to a point (x, y, z) and direction (

,

), expressed as a continuous 5D function whose output is the density that determines how much light will pass through each point (x, y, z).

r(t) = o + td (1)

One pixel in the input image progresses along the camera projection center o and direction d. Therefore, in order to render an image at a specific point of view, 3D points in space are sampled by following camera lights in the scene, and the color and density of each point are obtained by using the 2D point direction corresponding to these points as input to the neural network, and then the traditional volume A 2D image is created by accumulating color and density using a volume rendering method.

,

)로 5개의 차원으로만 이루어져 있기에 MLP(Multi-Layer Perceptron)에 충분한 정보를 제공하지 못하고 손실된다는 점을 보완하고자 위치 인코딩(positional encoding)으로 입력차원을 증가시켜 고차원 데이터로 매핑하여 입력으로 사용하였다. 이러한 위치 인코딩은 고주파 영역의 처리 성능을 향상된 방향으로 학습되도록 유도하며 주기함수를 통해 다음과 같이 표현할 수 있다.Since these processes are all differentiable, gradient descent can be used in the process of optimizing the model while reducing the difference between the observed image and the corresponding rendered result. In the case of NeRF, when training is performed using only this ray-based representation method, the following technique is proposed, paying attention to the characteristic that high-frequency image information is restored in a lost state or detailed expressions are removed. First, the dimensions of position and direction in space are (x, y, z), (

,

) is composed of only five dimensions, so to compensate for the fact that it does not provide enough information to MLP (Multi-Layer Perceptron) and is lost, the input dimension is increased by positional encoding, and it is mapped to high-dimensional data and used as input. . This positional encoding induces the processing performance of the high-frequency region to be learned in an improved direction, and can be expressed as follows through a periodic function.

γ(x) = [sin(x), cos(x), ..., sin(2 ^L-1 x), cos(2 ^L-1 x)] (2)

The above expression is simply expressed as a power of 2 from 1 to 2 ^L-1 as the sine function and cosine function of each dimension of a point x in the 3D position space. Here, L corresponds to a hyperparameter, and in NeRF, location information and each information are embedded using a periodic function using specific values. However, it is difficult to simply model in general road scenes. In the case of NeRF, since the input images are all large amounts of data acquired at similar distances, the geometric content of the scene through multi-view is limited to be easily learned through the network. Accordingly, the form of the image used as an input is limited, and a scene with only one object shows high performance and excellent performance in viewpoint synthesis. However, since it is difficult to acquire these images in a general situation, researches are being conducted to reduce the number of input images or to learn with general scenes.

In the present invention, we propose a NeRF-based modeling technique for a scene in which occluded areas frequently appear, assuming a general road condition away from existing data. For such a complex scene, the existing NeRF fails to converge sufficiently to produce a high-resolution result, and fails to accurately estimate the geometry because it is short in terms of the required number of samples per camera ray. To solve this problem, the number of input images required in the existing model is reduced by additionally providing depth information other than color information. This proceeds similarly to the method proposed in [2]. However, in [2], relationships between images are identified in the posture information estimation process and matching 3D points between images are used for learning. However, in the present invention, the entire depth information image is used to train the network. This enables the MLP to learn information that changes according to the movement of the camera, and thus provides a great direction for estimating the 3D geometric relationship, which is helpful for convergence of learning. Next, by separately modeling the parts that are difficult to learn with only a single MLP by using RGBD images, the rays are expanded according to the number of appropriate samples required in the sampling process.

According to an embodiment of the present invention, a video game based on realistic rendering is used to acquire data for learning the proposed light field synthesis network. The present invention is different from the prior art in that a depth image is additionally obtained.

According to the prior art, it was necessary to acquire a large amount of data to train the CNN, but in the present invention, only enough data to compose several scenes is needed, so the data is generated by modifying accordingly.

According to an embodiment of the present invention, data to be used for learning is acquired in that it is a game based on a virtual environment modeling the real world, and this can be confirmed through operation in real environment data. In the case of virtual data, the detailed dataset acquisition method is as follows.

According to an embodiment of the present invention, Script Hook V library is used to handle data in a virtual environment. Using the library, a multi-camera array was constructed in the virtual environment, and color images and depth images were extracted by rendering each camera to obtain depth images corresponding to one light field image. KITTI data was used for the data of the real environment, and the data obtained through the lidar sensor was used to process depth information images to be used for learning.

In the present invention, the baseline is expanded beyond existing techniques through Novel view synthesis, which operates on more general data, away from data looking at a single object from the front or 360 degrees, which was mainly handled by the existing NeRF-based method. It aims to create a light field.

1 is a diagram showing an overall framework for neural radiance-based light field synthesis in monocular video according to an embodiment of the present invention.

1 shows the overall configuration of the algorithm to be proposed in the present invention.

The proposed neural radiance-based light field synthesis method expresses a scene itself as a network based on information 110 input from various viewpoints.

First, the NeRF method of modeling 121 by considering a color image as supervision is generally followed. However, as the technique was created assuming dense inputs of various angles, depth information is additionally used as a supervision to generate high-quality results from limited inputs as video is used as an input (122). This is used for the purpose of supplementing the geometric interpretation of the scene by using the characteristics of multi-view when input is received only with RGB images, but when video is used as an input. Second, in the part of the road where the depth value changes continuously and there is almost no texture, a problem in the generation of the neural radiance field is identified and solved to solve the part of the road that adversely affects learning. A plan to do so is proposed (130). As described above, the technique proposed by the present invention aims to expand to data in a more general environment.

2 is a flowchart illustrating a method of synthesizing a light field based on neural radiance in a monocular video according to an embodiment of the present invention.

In the proposed monocular video, the neural radiance-based light field synthesis method uses the RGBD image of the input image to enhance the data for the visible region, preprocessing the data for learning (210), and the color image and the depth image. Performing NeRF (Nueral radiance field)-based scene modeling (220), expanding data using the input color image and depth image (230), and changing the depth value for the depth image continuously and learning to reduce an error of NeRF-based scene modeling in an area where the texture is not present (240).

In step 210, data for learning is preprocessed by utilizing the RGBD image of the input image in order to improve the data for the visible region.

In the case of road driving data, which is the target of the present invention, there are several limitations in solving using the existing NeRF technique. Unlike the existing input, it has difficulty in overall learning in that the camera moves and the scene is not fixed. In particular, the network has difficulty in estimating depth or 3D structure in road portions where textures are difficult to distinguish and large features do not exist. In addition, the fact that the road part appears in the whole part of the image but continuously connects pixels at various depths makes the problem even more difficult.

NeRF according to the prior art designed a network centered on data at the same distance, so in the case of data, geometric estimation can be performed through many inputs, but it seems difficult to obtain from a small number of frames in a straight line direction. Therefore, input RGBD images are used to improve the data for the road part, which is mostly visible. The biggest problem in synthesizing viewpoints with the existing DIBR method is that it is difficult to obtain information about the occluded area. However, in the case of roads, there are almost no occluded areas, and most of them are visible, so DIBR is used to create virtual viewpoint data. As a result, a virtual viewpoint image for the visible region is acquired using the DIBR method, and masking for determining whether the corresponding rays are visible regions is created. This is input to the neural radiance field to be learned later, and the rays of the masked part are excluded from learning.

In step 220, scene modeling based on a neutral radiance field (NeRF) using a color image and a depth image is performed.

The overall framework according to the embodiment of the present invention proceeds similarly to the pipeline of the existing NeRF proposed for NeRF. However, depth information, which was not considered in the existing method, was additionally used, and various data for samples of the visible region can be obtained from a small number of inputs through a preprocessing process.

3 is a diagram illustrating a color image and a depth image received as inputs during a real environment experiment according to an embodiment of the present invention.

FIG. 3(a) is a diagram illustrating a color image, and FIG. 3(b) is a diagram illustrating a depth image.

NeRF (Nueral radiance field)-based scene modeling using color and depth images consequently increases the sampling rate for the visible region and estimates the occluded region from the input according to the video direction. Since the increase in the amount of computation is large to generate and train all 8 × 8 SAI (Sub-aperture images), a certain SAI was selected and proceeded.

In step 230, data is expanded using the color image and the depth image input as the input.

In the present invention, we propose a NeRF-based network that synthesizes a light field with a wide baseline from an input monocular video. In addition, the performance of the synthesized light field is verified by generating data to be used in the proposed network. The proposed method is divided into two stages. The first expands the data by using the input color image and depth image. Using depth-based rendering method, views around the input point are created. This helps the MLP make geometric inferences smoothly from sparse inputs generated by straight-line driving. In addition, the range of warping with DIBR was adjusted to have a wider baseline compared to the existing technique, and data was augmented by selecting a certain portion of SAI without generating the entire SAI.

In step 240, the depth value of the depth image is continuously changed and learning is performed to reduce errors of NeRF-based scene modeling in an area where no texture exists.

The overall process of learning according to an embodiment of the present invention generates a virtual view including the visible region and models NeRF through input RGBD. In addition, techniques such as positional embedding and hierarchical volume sampling for smooth learning of NeRF proposed in [1] were used. The network is trained end-to-end, and only the L2 function was used as the loss function used in the training process. The network proposed in the present invention uses the ADAM optimizer, and the learning rate starts with 0.0005 and proceeds with learning by decreasing the learning rate. The experiment was learned for a period of about 1 day through NVIDIA RTX A6000 48GB GPU.

4 is a diagram showing the configuration of a neural radiance-based light field synthesis system in monocular video according to an embodiment of the present invention.

The light field synthesis system 400 according to this embodiment may include a processor 410, a bus 420, a network interface 430, a memory 440, and a database 450. The memory 440 may include an operating system 441 and a lightfield composition routine 442 . The processor 410 may include a pre-processing unit 411 , a modeling unit 412 and a learning unit 413 . In other embodiments the lightfield synthesis system 400 may include more components than those of FIG. 4 . However, there is no need to clearly show most of the prior art components. For example, the lightfield synthesis system 400 may include other components such as a display or transceiver.

The memory 440 is a computer-readable recording medium, and may include a random access memory (RAM), a read only memory (ROM), and a permanent mass storage device such as a disk drive. Also, program codes for the operating system 441 and the light field synthesis routine 442 may be stored in the memory 440 . These software components may be loaded from a computer-readable recording medium separate from the memory 440 using a drive mechanism (not shown). The separate computer-readable recording medium may include a computer-readable recording medium (not shown) such as a floppy drive, a disk, a tape, a DVD/CD-ROM drive, and a memory card. In another embodiment, software components may be loaded into the memory 440 through the network interface 430 rather than a computer-readable recording medium.

Bus 420 may enable communication and data transfer between components of lightfield synthesis system 400 . Bus 420 may be configured using a high-speed serial bus, a parallel bus, a storage area network (SAN), and/or other suitable communication technology.

The network interface 430 may be a computer hardware component for connecting the lightfield synthesis system 400 to a computer network. The network interface 430 may connect the lightfield synthesis system 400 to a computer network through a wireless or wired connection.

The database 450 may serve to store and maintain all information required for light field synthesis. 4 shows that the database 450 is built and included inside the light field synthesis system 400, but is not limited thereto, and may be omitted depending on the system implementation method or environment, or all or part of the database It is also possible to exist as an external database built on a separate and different system.

The processor 410 may be configured to process commands of a computer program by performing basic arithmetic, logic, and input/output operations of the light field synthesis system 400 . Instructions may be provided to processor 410 by memory 440 or network interface 430 and via bus 420 . The processor 410 may be configured to execute program codes for the pre-processing unit 411 , the modeling unit 412 and the learning unit 413 . These program codes may be stored in a recording device such as memory 440 .

The pre-processing unit 411, the modeling unit 412, and the learning unit 413 may be configured to perform steps 210 to 240 of FIG.

The light field synthesis system 400 may include a pre-processing unit 411 , a modeling unit 412 and a learning unit 413 .

The pre-processing unit 411 pre-processes data for learning by utilizing the RGBD image of the input image in order to improve the data for the visible region.

The pre-processing unit 411 obtains the virtual viewpoint image for the visible region using the DIBR (Depth Image Based Rendering) method, generates masking to determine whether the corresponding rays are visible regions, and inputs them to the NeRF to be learned later, which is masked The ray of the part is excluded from learning.

The modeling unit 412 performs scene modeling based on a neutral radiance field (NeRF) using a color image and a depth image.

The modeling unit 412 increases the sampling rate for the visible region and, in the case of the occlusion region, estimates data from the input according to the video direction, and baselines between SAIs (Sub-aperture images) of the light field from the rendering-based input image By acquiring this wide data on a large scale, a light field dataset is created.

The learning unit 413 expands data by using the color image and the depth image input as the input, and errors in NeRF-based scene modeling in an area where depth values for the depth image are continuously changed and no texture exists. learn to reduce

The learning unit 413 creates viewpoints around an input viewpoint using a depth-based rendering method, adjusts the range of DIBR warping to have a wide baseline, and selects a certain portion of SAIs without generating all SAIs. expand the data.

5 is a diagram showing a light field generated using a network according to an embodiment of the present invention.

Figure 5 (a) is G.T., Figure 5 (b) is Srinivasan, Figure 5 (c) is NeRF, Figure 5d) shows the results according to an embodiment of the present invention.

View synthesis technology generally uses supervised learning-based methods, so GT images are required for quantitative evaluation. Quantitative evaluation is performed using light field video data generated directly through the road driving light field dataset in the present invention. The proposed method creates SAI by querying light rays from a point not used for learning. Next, for verification in the real environment, results are generated using KITTI, a road driving dataset. Since the KITTI dataset generated data while driving on the road based on a binocular camera, quantitative evaluation was impossible, and comparative analysis was performed only with qualitative evaluation. The data used in the experiment was performed through the 480Х640 spatial resolution used in the learning process, and the KITTI data was also resized and used in the experiment.

5 shows a light field generated using the proposed network. Since the neural radiance-based method aims at optimization according to the scene, it was confirmed that there is almost no difference in performance between the real environment and the virtual environment.

A qualitative evaluation was performed on the framework proposed in the present invention. 6 shows SAI generated by each method. In the case of FIG. 5 (a), it is an example of SAI corresponding to the Srinivasan technique based on estimating depth information from a monocular, and in the case of FIG. 5 (b), it can be seen that the overall geometric structure was not accurately estimated. In addition, in the case of FIG. 5(c), when there are geometrically complex or various objects, a depth value cannot be accurately estimated, and many floating artifacts can be seen. In the case of qualitative evaluation, it was conducted in both virtual and real environments. When comparing the results generated by NeRF and the proposed framework, it can be confirmed that the proposed method relatively preserves the geometric shape of the GT as a whole and expresses it accurately in the road part.

6 is a diagram showing an example of SAI of a synthesized result according to an embodiment of the present invention.

7 is a diagram showing KITTI data broad baseline qualitative results according to an embodiment of the present invention.

Figure 6 shows the SAI of a portion of the light field. 6(a) shows virtual environment data, and FIG. 6(b) shows KITTI data. As confirmed in the comparison, even when the baseline increases, the overall geometric structure is maintained and high quality is maintained.

The devices described above may be implemented as hardware components, software components, and/or a combination of hardware components and software components. For example, devices and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA) , a programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions. A processing device may run an operating system (OS) and one or more software applications running on the operating system. A processing device may also access, store, manipulate, process, and generate data in response to execution of software. For convenience of understanding, there are cases in which one processing device is used, but those skilled in the art will understand that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that it can include. For example, a processing device may include a plurality of processors or a processor and a controller. Other processing configurations are also possible, such as parallel processors.

Software may include a computer program, code, instructions, or a combination of one or more of the foregoing, which configures a processing device to operate as desired or processes independently or collectively. You can command the device. Software and/or data may be any tangible machine, component, physical device, virtual equipment, computer storage medium or device, intended to be interpreted by or provide instructions or data to a processing device. can be embodied in Software may be distributed on networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer readable media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer readable medium. The computer readable medium may include program instructions, data files, data structures, etc. alone or in combination. Program commands recorded on the medium may be specially designed and configured for the embodiment or may be known and usable to those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. - includes hardware devices specially configured to store and execute program instructions, such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include high-level language codes that can be executed by a computer using an interpreter, as well as machine language codes such as those produced by a compiler.

As described above, although the embodiments have been described with limited examples and drawings, those skilled in the art can make various modifications and variations from the above description. For example, the described techniques may be performed in an order different from the method described, and/or components of the described system, structure, device, circuit, etc. may be combined or combined in a different form than the method described, or other components may be used. Or even if it is replaced or substituted by equivalents, appropriate results can be achieved.

Therefore, other implementations, other embodiments, and equivalents of the claims are within the scope of the following claims.

Claims (8)

pre-processing data related to the RGBD image of the input image for learning by utilizing the RGBD image of the input image in order to enhance the data of the visible region through a pre-processing unit;
Performing scene modeling based on a NeRF (Nueral radiance field) using a color image and a depth image of the RGBD image of the preprocessed input image through a modeling unit;
expanding data by using a color image and a depth image of the RGBD image of the preprocessed input image through a learning unit; and
Learning to reduce an error of NeRF-based scene modeling in an area in which a change in a depth value for a data-extended depth image is continuously made through a learning unit and a texture does not exist.
A light field synthesis method comprising a. According to claim 1,
The step of pre-processing data on the RGBD image of the input image for learning by utilizing the RGBD image of the input image to enhance the data for the visible region through the pre-processing unit,
DIBR (Depth Image Based Rendering) method acquires the virtual viewpoint image for the visible region, creates masking to determine whether the corresponding rays are visible regions, and inputs them to NeRF to be learned later, and the rays of the masked part are used in learning excluded
Light field synthesis method. According to claim 1,
The step of performing NeRF (Nueral radiance field) based scene modeling using a color image and a depth image of the RGBD image of the preprocessed input image through the modeling unit,
By increasing the sampling rate for the visible region and estimating data from the input along the video direction in the case of the occlusion region, large-scale data with a wide baseline between SAIs (Sub-aperture images) of the light field from the rendering-based input image Acquisition to create a light field dataset
Light field synthesis method. According to claim 1,
The step of expanding data using a color image and a depth image of the RGBD image of the preprocessed input image through the learning unit,
Using the depth-based rendering method, viewpoints around the input viewpoint are created, the range of warping with DIBR is adjusted to have a wide baseline, and data is expanded by selecting a certain portion of SAIs without generating all SAIs.
Light field synthesis method. a pre-processing unit for pre-processing RGBD image data of the input image for learning by utilizing the RGBD image of the input image to improve data for the visible region;
a modeling unit that performs scene modeling based on NeRF (Nueral radiance field) using a color image and a depth image of the RGBD image of the preprocessed input image; and
Data is expanded using a color image and a depth image of the RGBD image of the preprocessed input image coming into the input, and the depth value of the data-extended depth image is continuously changed through the learning unit, and no texture is present. A learning unit that learns to reduce the error of NeRF-based scene modeling in an area not covered by
A light field synthesis system comprising a. According to claim 5,
The pre-processing unit,
DIBR (Depth Image Based Rendering) method acquires the virtual viewpoint image for the visible region, creates masking to determine whether the corresponding rays are visible regions, and inputs them to NeRF to be learned later, and the rays of the masked part are used in learning excluded
Lightfield synthesis system. According to claim 5,
The modeling unit,
By increasing the sampling rate for the visible region and estimating data from the input along the video direction in the case of the occlusion region, large-scale data with a wide baseline between SAIs (Sub-aperture images) of the light field from the rendering-based input image Acquisition to create a light field dataset
Lightfield synthesis system. According to claim 5,
The learning unit,
Using the depth-based rendering method, viewpoints around the input viewpoint are created, the range of warping with DIBR is adjusted to have a wide baseline, and data is expanded by selecting a certain portion of SAIs without generating all SAIs.
Lightfield synthesis system.

Images