NZ752912B2 - A rendering process and system - Google Patents
A rendering process and system Download PDFInfo
- Publication number
- NZ752912B2 NZ752912B2 NZ752912A NZ75291216A NZ752912B2 NZ 752912 B2 NZ752912 B2 NZ 752912B2 NZ 752912 A NZ752912 A NZ 752912A NZ 75291216 A NZ75291216 A NZ 75291216A NZ 752912 B2 NZ752912 B2 NZ 752912B2
- Authority
- NZ
- New Zealand
- Prior art keywords
- maps
- rendering
- rendering process
- threshold
- radiance
- Prior art date
Links
- 238000009877 rendering Methods 0.000 title claims abstract description 82
- 238000000034 method Methods 0.000 title claims abstract description 58
- 239000002131 composite material Substances 0.000 claims abstract description 10
- 230000003190 augmentative Effects 0.000 abstract description 7
- 230000003993 interaction Effects 0.000 abstract description 6
- 238000001514 detection method Methods 0.000 description 15
- 235000019587 texture Nutrition 0.000 description 10
- 238000010586 diagram Methods 0.000 description 8
- 238000005286 illumination Methods 0.000 description 8
- 239000000463 material Substances 0.000 description 8
- 230000000007 visual effect Effects 0.000 description 8
- 239000000203 mixture Substances 0.000 description 7
- 238000001914 filtration Methods 0.000 description 6
- 238000005070 sampling Methods 0.000 description 6
- 239000007787 solid Substances 0.000 description 6
- UIIMBOGNXHQVGW-UHFFFAOYSA-M buffer Substances [Na+].OC([O-])=O UIIMBOGNXHQVGW-UHFFFAOYSA-M 0.000 description 5
- 235000019592 roughness Nutrition 0.000 description 5
- 230000000694 effects Effects 0.000 description 4
- 210000003128 Head Anatomy 0.000 description 3
- 238000009826 distribution Methods 0.000 description 2
- 102100004209 LMAN1 Human genes 0.000 description 1
- 101700082720 LMAN1 Proteins 0.000 description 1
- 206010052143 Ocular discomfort Diseases 0.000 description 1
- 101700064519 PSTN Proteins 0.000 description 1
- 238000007630 basic procedure Methods 0.000 description 1
- 230000002457 bidirectional Effects 0.000 description 1
- 238000005266 casting Methods 0.000 description 1
- 230000001413 cellular Effects 0.000 description 1
- 238000004590 computer program Methods 0.000 description 1
- 230000001276 controlling effect Effects 0.000 description 1
- 230000000875 corresponding Effects 0.000 description 1
- 238000005315 distribution function Methods 0.000 description 1
- 238000004134 energy conservation Methods 0.000 description 1
- 239000000284 extract Substances 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 239000003550 marker Substances 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000006011 modification reaction Methods 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 239000010979 ruby Substances 0.000 description 1
- 229910001750 ruby Inorganic materials 0.000 description 1
- 238000003860 storage Methods 0.000 description 1
- 230000003746 surface roughness Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T15/00—3D [Three Dimensional] image rendering
- G06T15/10—Geometric effects
- G06T15/20—Perspective computation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T15/00—3D [Three Dimensional] image rendering
- G06T15/50—Lighting effects
- G06T15/503—Blending, e.g. for anti-aliasing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T15/00—3D [Three Dimensional] image rendering
- G06T15/50—Lighting effects
- G06T15/506—Illumination models
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T15/00—3D [Three Dimensional] image rendering
- G06T15/50—Lighting effects
- G06T15/55—Radiosity
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T15/00—3D [Three Dimensional] image rendering
- G06T15/50—Lighting effects
- G06T15/60—Shadow generation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20172—Image enhancement details
- G06T2207/20208—High dynamic range [HDR] image processing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/007—Dynamic range modification
- G06T5/009—Global, i.e. based on properties of the image as a whole
-
- H04N5/23238—
Abstract
rendering process and system that may be used to composite virtual objects in panoramic video to provide a virtual reality and augmented reality experience. The process includes receiving low dynamic range (LDR) video data, e.g. 360° video; generating radiance maps, such as diffuse and specular maps, from the LDR data; inverse tone mapping the LDR data of the maps to generate high dynamic range (HDR) data for the maps; and receiving at least one virtual object and applying image based lighting (IBL) to the virtual object using the HDR data of the maps. A perceptually based threshold is also applied to radiance maps to detect prominent pixels, and using the prominent pixels as salient lights for image based shadowing (IBS) associated the virtual object. Objects are composited 360° video in real time using IBL and IBS without precomputation allowing user interaction with the objects. ps, from the LDR data; inverse tone mapping the LDR data of the maps to generate high dynamic range (HDR) data for the maps; and receiving at least one virtual object and applying image based lighting (IBL) to the virtual object using the HDR data of the maps. A perceptually based threshold is also applied to radiance maps to detect prominent pixels, and using the prominent pixels as salient lights for image based shadowing (IBS) associated the virtual object. Objects are composited 360° video in real time using IBL and IBS without precomputation allowing user interaction with the objects.
Description
A RENDERING PROCESS AND SYSTEM
FIELD
The present invention relates to a rendering process and system that may be used to
composite objects with panoramic video and provide a mixed reality experience for a user.
BACKGROUND
Stereographic head mounted displays (HMDs), such as the Oculus Rift, HTC Vive, and Sony
PlayStation VR, and accompanying controllers are now available for use with a computer to
provide a computer system that delivers a virtual reality (VR) experience for a user. By
adding content to the rendered video presented to the user on the HMD, an augmented reality
(AR) experience can also be delivered. Combining virtual reality and augmented reality
allows a user to interact with augmented objects in a virtual environment. The environment
may include computer generated images or a captured real-world video to provide an
immersive video viewing experience. The computer system can then be considered to be
delivering a mixed reality experience to a user wearing the HMD and operating the
controllers.
Equipment, such as catadioptric omnidirectional camera or multiple-camera rigs, is now
available to capture 360 panoramic video (i.e. video with a full spherical 4π steradian field
of view) and when displayed on a HMD provides a user with a completely omnidirectional
and immersive video viewing experience. Many 360 panoramic videos are available on
video sharing and streaming web sites, such as YouTube and Facebook.
Whilst 360 videos provide for impressive natural and real world visuals, the current virtual
reality computer systems only allow interaction limited to the head motion of the user.
A significant technical challenge is to provide a rendering process and system that can
seamlessly composite virtual objects into live video, particularly 360 video (360-video). To
provide an impressive immersive mixed reality experience, the process or system needs to
deliver both high visual quality for seamless composition of objects and performance for
real-time rendering. Image based lighting (IBL) techniques can be used to illuminate 3D
objects, but the low dynamic range (LDR) data captured using conventional 360 video
camera equipment does not provide the sufficient dynamic range for IBL. High performance
rendering is also particularly important to mitigate against visual discomfort when using a
HMD. Also in adapting a live video stream, or even a synthetic computer generated video,
and providing interaction with composited virtual objects, a rendering process that requires
precomputation cannot be used.
Accordingly, it is desired to address the above or at least provide a useful alternative.
SUMMARY
An embodiment of the present invention provides a rendering process applying a
perceptually based threshold for image based shadowing (IBS). Salient lights may be
detected from LDR video data and used to provide realistic shadows for real-time differential
rendering.
An embodiment of the present invention also provides a rendering process including:
receiving low dynamic range (LDR) video data;
generating radiance maps from said LDR data;
inverse tone mapping LDR data of said maps to generate high dynamic range (HDR)
data for said maps; and
receiving at least one virtual object and applying image based lighting to said at least
one virtual object using said maps, including applying a dynamic perceptual based threshold
to detect prominent pixels; and converting said prominent pixels into directional lights for
image based shadowing associated with at least one virtual object.
The maps may be diffuse and specular maps.
Advantageously, the rendering process may also use the LDR video data as a background
for differential rendering to composite said at least one virtual object into the background.
The prominent pixels for salient lights may detected from LDR video data, HDR video, or
inverse tone mapped LDR video data to provide realistic shadows for real-time differential
rendering.
An embodiment of the invention also provides a rendering computer system including:
a head mounted display;
a graphics processing unit (GPU);
a computer processing unit (CPU); and
processing modules including GPU shaders to provide a pipeline to perform the
rendering process.
DRAWINGS
Embodiments of the present invention are described herein, by way of example only, with
reference to the accompanying drawings wherein:
Figure 1 is a block diagram of an embodiment of a rendering system according to the
present invention;
Figure 2 is a diagram of an embodiment of a rendering process according to the present
invention;
Figure 3 is an image comparison of IBL rendering using diffuse radiance maps of
various resolutions;
Figure 4 is a glossy specular reflection image with various roughness parameters;
Figure 5 is images rendered using real-time IBL integrated with various shading
models in different 360-videos;
Figure 6 is images illustrating light detection and includes input environment maps
(a), light detection masks (b), rendered results with only IBL (c) and rendered results with
IBS (d), and dots on the masks indicate the positions of the detected lights;
Figure 7 is a flow diagram of the light detection process for image based shadowing
(IBS);
Figure 8 is a diagram of view changes in 360-video using a HMD;
Figure 9 is images produced by the compositing steps of real-time differential
rendering of the process and includes (a) 360 background (user’s view), (b) background,
objects and local scene, (c) local scene without objects, (d) object matte (stencil buffer), (e)
difference being image (b) subtracted from image (c), and (f) the final composite;
Figure 10 is a flow diagram of differential rendering and composition of the process;
Figure 11 is a diagram of centre point location for stereoscopic rendering;
Figure 12 is diagram of image rotation for each eye to match the centre point for
stereoscopic rendering; and
Figure 13 are diagrams of different image rotation based on object position for
stereoscopic rendering.
DESCRIPTION
A rendering computer system 100, as shown in Figure 1, is an exemplary embodiment of a
mixed reality system that executes a rendering process 200, as shown in Figure 2. The
rendering computer system 100 is herein referred to as MR360. The rendering process
provides a processing pipeline to allow users of the system 100 to interact with 360 videos
containing three dimensional (3D) virtual objects seamlessly augmented into a panoramic
360-video. The rendering process can also be used to seamlessly augment the objects into a
computer generated or synthetic video.
The rendering computer system 100 includes a computer 102, such as an Intel architecture
computer produced by Lenovo Corporation, IBM Corporation or Apple Inc. The data
processing executed by the computer system 102 is defined and controlled by computer
program instruction code and data of software components or modules 150 stored on non-
volatile (e.g. hard disk) storage 104 of the computer 102. The modules 150 include a game
engine 170 and GPU shaders that can be integrated into the game engine or run separately.
The processes performed by the modules 150 can, alternatively, be performed by firmware
stored in read only memory (ROM) or at least in part by dedicated hardware circuits of the
computer 102, such as application specific integrated circuits (ASICs) and/or field
programmable gate arrays (FPGAs). These integrated circuits can be part of a display adapter
114 and execute processes of the rendering pipeline.
The computer 102 includes random access memory (RAM) 106, at least one microprocessor
(CPU) 108, and external interfaces 110, 112, 114 that are all connected by a system bus 116.
The external interfaces include at least universal serial bus (USB) interfaces 110, a network
interface connector (NIC) 112, and the display adapter 114. Other interfaces, such as
Bluetooth and Wifi interfaces can also be included. The USB interfaces 110 are connected
to input/output devices, such as VR hand controllers, a keyboard and/or mouse 118. The
display adapter 114 is connected to a Head Mounted Display (HMD) 122, such as the Oculus
Rift, HTC Vive, or Sony PlayStation VR. The display adapter 114 includes a graphics
processing unit (GPU) to support and execute the rendering pipeline. The NIC 112 enables
the computer 102 to connect to a communications network 120. The network 120 may
include one or a combination of existing networks, such as a LAN, private WAN, the PSTN,
the Internet, mobile cellular telephone networks, etc. The computer 102 includes an
operating system (OS) 124, such as Microsoft Windows, Mac OSX or Linux. The modules
150 can run on the OS 124, and include program code written using languages such as C,
C++, Ruby or C# or the language supported by the framework provided by the game engine
170.
For example, the system 100 can have a computer 102 with an Intel Xeon 3.5 GHz CPU
108, 8 GB memory 106, and an Nvidia Geforce GTX 970 GPU 114. The HMD 122 can be
a HTC Vive VR headset (HTC headset) supporting display at 90 FPS with HTC Vive hand-
held controllers (HTC controller) 118 for user interactions. The MR360 rendering pipeline
200 can be implemented by GPU shaders and/or one or more software programs that run on
the CPU 108 directly. The pipeline 200 is integrated into a 3D game engine 170 such as
Unreal Engine 4 (UE4) or Unity with a light detection process 212 of the pipeline 200 for
image based shadowing (IBS) running on the CPU 108 concurrently.
MR360, as described herein, provides realistic illumination of 3D virtual objects using a
LDR 360-video as the light source as well as the backdrop for real world backgrounds, and
presents high visual quality and rendering speed.
The system 100 uses perceptually based thresholding for real time image based shadowing
(IBS). Salient lights are detected from the 360-video providing optimal realistic shadows for
real-time differential rendering.
MR360 does not require any pre-computation, and therefore allows a rich user interaction
that is beyond the scope of conventional 360-videos; e.g. manipulating dynamic objects
within a live 360-video stream in varying lighting conditions.
Compared with full 3D rendering, MR 360 provides better rendering performance because
the complex background can be pre-rendered, such as a previously stored 360-video or
synthetic video, and in real-time rendering the pipeline focuses on rendering the augmented
virtual objects and thereby allows the computing power of the GPU 114 to focus on those
objects.
REAL-TIME IBL USING 360-VIDEO
An image based lighting (IBL) process 202 of the pipeline supports photorealistic rendering
of virtual objects using real-world lighting.
Perceptually Based IBL
MR360 receives LDR 360-video 201 as an input stream and dynamic real-world radiance
maps 204 are generated to illuminate virtual objects. Since lighting and materials are
reciprocal in illumination, radiance maps are reconstructed to represent illumination of
diffuse, glossy, and mirror-like specular materials.
For diffuse illumination, a diffuse radiance map is generated per frame. When using a
perceptually optimised radiance map, reducing the resolution for a tiny radiance map (e.g.
32 x 16 pixels), reduces the computation cost to generate the diffuse map per frame while
maintaining the perceptible visual quality for IBL as shown in Figure 3. Once the diffuse
radiance map for each frame is generated, the diffuse lighting for any point on a composite
object’s surface consists of a single texture lookup by using the pixel position to look up a
texture value from the map.
Mirror-like specular reflection can be achieved by environment mapping, but glossy specular
reflection is usually computationally expensive. MR360 approximates glossy specular
reflection with mipmaps in a similar manner to the diffuse illumination. The sampling radius
around the specular direction depends on a surface roughness parameter related to the glossy
lobe. In a radius around the specular direction, MR360 samples a mipmap chain of specular
radiance maps at an appropriate level, for example, a higher resolution mipmap level is
sampled for lower roughness. Glossy specular lighting is approximated using a fixed number
of texture lookups per rendering fragment. The pipeline 200 uses 18 samples for inside, and
18 samples for outside the primary glossy lobe.
For the GPU 114 every 360-video frame is convolved by a GPU shader 150, which outputs
204 a diffuse radiance map and mipmaps for the specular radiance map to multiple render
targets. The radiance maps 204 are then applied to IBL 202 for virtual objects 206 by
sampling the radiance map render targets corresponding to proper material properties as
shown in Figure 4 and described further below. The maps 204 are used to effectively change
material properties of the rendered objects during IBL processing 202 rather than actually
varying the material properties of the objects themselves.
Inverse Tone Mapping from LDR to HDR
IBL 202 requires HDR radiance maps as the input for realistic illumination. Missing radiance
values in LDR radiance maps cannot produce believable lighting in IBL. Inverse tone
mapping can boost the radiance values of LDR radiance maps to provide believable IBL
results targeting the human visual system. The MR60 pipeline applies inverse tone mapping
208 such that the HDR colour is calculated from the LDR colour as:
⟨ , , ⟩ = ∙ ⟨ , , ⟩ (1)
= 10 ∙ + 1.8 (2)
The input luminance is calculated as:
= 0.3 ∙ + 0.59 ∙ + 0.11 ∙ (3)
The tone-mapping process 208 is independent of varying frame properties, and the same
transform is applied to each pixel individually. As such it is easily and efficiently
implemented on the GPU 114.
While GPU instructions are being queued, a separate thread loads and decodes the next video
frame. This is passed to the GPU using a pixel buffer object for asynchronous memory
transfer. As the target display refresh rate is much higher than typical video framerate
(minimum 75 frames per second for HMD, typical 25 frames per second for video) one frame
is used for display as the next is being loaded, giving smooth video performance.
The basic procedure code is described below for the processes 202, 204, 208. The diffuse
radiance map is upscaled by texture filtering such as bilinear texture filtering. For the HMD
the display process is executed twice, once for each eye.
Input : LDR 360 video
Input : 3D mesh data
Output : LDR framebuffer texture for display
while program running do
if time elapsed > 1 / video FPS then
/* tonemap next frame to HDR texture */
for pixel ← video frame do
pixel colour *= 10 * (pixel luminance) + 1.8;
/* render diffuse radiance map */
for outpixel ← diffuse map do
accumulator = 0;
weight = 0;
for inpixel ← low resolution specular map do
inpixel area = cos (inpixel latitude);
inpixel weight = inpixel area * angle between (inpixel, outpixel);
accumulator += inpixel weight * environment colour (inpixel direction);
weight += inpixel weight;
outpixel colour = accumulator / weight
reset elapsed time;
draw specular environment to output buffer as background;
rasterize virtual objects into output buffer using IBL shader;
As mentioned, object lighting is performed in a GPU fragment shader 150. Specular and
diffuse lighting consist of texture lookups into the specular and diffuse radiance maps.
Glossy specular lighting uses a somewhat more complicated system of sampling in a fixed
pattern from lower-resolution mipmap levels of the specular map. Samples are taken in
concentric rings around the specular direction and which mipmap level to sample from is
determined by taking the distance between samples in a ring, and choosing the level with an
equivalent distance between pixels. Samples are weighted, and the radius of the glossy lobe
is defined to be the distance at which the weighting is 0:5. To reduce discretization artifacts,
hardware trilinear filtering is enabled.
IBL in different materials
The IBL process 202 supports real-time rendering for various materials covering diffuse,
glossy, and mirror-like specular reflection, and Figure 5 shows the IBL produced using
different UE4 shading models for two different 360-videos augmented with the same
objects. The reconstructed LDR 360-videos using inverse tone-mapping provide believable
IBL when viewed independently as shown in Figure 5.
REAL-TIME IBS USING 360-VIDEO
Light Detection
Real-time IBL provides realistic illumination but has limitations for casting realistic
shadows. Studio artists in visual effects studios often add artificial lights, such as a
directional light, on top of the radiance map. The process is automated in MR360 100 for
real-time image based shadowing (IBS) 210.
Given a radiance map (step 702), MR360 detects prominent patches of pixels and convert
them into directional lights 212, as shown in Figure 7. Standard shadow mapping can then
be used to cast shadows from detected lights in dynamic scenes. To allow for dynamic
radiance maps, IBS needs to runs in real-time. To achieve this, the MR360 pipeline uses a
thresholding approach. Given a threshold value, if a pixel’s luminance (after inverse tone-
mapping) is above the threshold, it is considered part of a light source. This produces a mask,
as shown in Figure 6(b) for which pixels belong to light sources. In order to produce a
minimal set of discrete directional lights, a breadth-first search on this mask is performed to
determine connectivity between pixels to find the area of lights. The threshold luminance is
subtracted from each pixel before computing the light properties, and clamp the luminance
of the pixels in the radiance map to the threshold luminance in order to ensure energy
conservation (708).
For each detected light, the pipeline determines its properties or parameters (710) from the
threshold luminance Y and the pixels that belong to it in terms of their luminance Y ,
radiance Lp, solid angle Ωp and spherical coordinates 〈 θp, ∅p〉 . Each pixel’s irradiance is
determined from the amount of its radiance over the luminance threshold:
= Ω ∙ − ∙ min 1, (4)
The light’s irradiance is then = . The light’s position in spherical coordinates
is determined by a weighted mean where Y(E) is as for equation 3:
〈 , ∅ 〉= ∑ ∙ 〈 , ∅ 〉 (5)
This process can run in real-time once the following is addressed. Firstly, the actual threshold
needs to be determined, and changes based on the radiance map. For example, a radiance
map with a strong dominant light requires a high threshold value, but a radiance map with
diffuse light sources requires a lower threshold value. For dynamic radiance maps used by
MR360, the threshold value needs to change in real-time based on the current frame.
To calculate the dynamic threshold Y , it is determined statistically in terms of the
distribution of luminance values. When the pipeline runs, the mean μ and variance of
each frame’s luminance is determined and the threshold is generated based on these
statistical metrics to ensure real-time performance. The threshold value used is:
+ 2 (6)
A threshold value produces noisy patch areas, rather than a minimal number of contiguous
patches but applying a small box blur (706) to the luminance data before thresholding
successfully deals with the noise, and produces much cleaner patches. The light detection
can produce a small number of strong lights together with a number of very weak lights
whose luminance was only slightly over the threshold, and to address this, the lights are
sorted in order of descending strength, and then only the first n lights are kept for the smallest
value of n such that their combined irradiance is at least 90% of that from all detected lights
(714).
In some scenarios a strong ground reflection may be erroneously detected as a light,
potentially at the expense of a real light. MR360 is primarily directed to provide lighting
from the upper hemisphere to cast shadows, so only the upper half of the frame to is
processed to avoid strong ground reflections (704).
Due to the video 201 being LDR, lights can be clipped to plain white. While inverse tone-
mapping 208 significantly improves this scenario, there is no a priori way to determine how
bright the light was before clipping and this presents an issue when it comes to matching the
strength of shadows visible in the video. Small light patches are more likely to be lights
proper, while larger patches may be clouds or reflections. MR360 applies (712) a greater
brightness increase to lights of smaller solid angles and reduces the brightness of those with
very large solid angles. The solid angle of the light sources is determined, and this is used to
control shadow filtering (using for example percentage closer filtering) to achieve
perceptually acceptable artificial soft shadow penumbrae that vary dynamically with the
environment. IBS 210 is achieved by applying the detected light parameters to a dynamic
shadowing procedure of the game engine 170.
Perceptually Based Thresholding
While thresholding allows for real-time light detection, it has two potential issues. Firstly,
the threshold value needs to be manually set as described above. Secondly, the threshold
acts as a global value across the image, which may suggest that it is prone to noise. MR360
uses experimentally derived perceptually based thresholding which addresses both of these
issues. The experiment produces a perceptually acceptable statistical rule for a threshold
value.
In order to determine a threshold value, an optimization was performed between
thresholding and a known method for light detection in LDR images that is considered
perceptually acceptable to humans. For example, a known method is described in
“Automatic scene inference for 3d object compositing”, K. Karsch, K. Sunkavalli, S. Hadap,
N. Carr, H. Jin, R. Fonte, M. Sittig, and D. Forsyth, ACM Trans. Graph., 33(3):32:1–32:15,
June 2014. This method is able detect lights in images by using machine learning that has
been trained using a manually labelled database of LDR images. A threshold value can then
be determined which best matches the results produced by the known method. This in turn
produces a statistical rule for the threshold value using statistical metrics. In order to find a
general rule which is consistent among various lighting conditions, three broad categories of
radiance maps are used: single hard light, single soft light, and multiple light. For each case,
the known light detection method is run, which produces a binary mask of the light pixels.
From this, a binary search is used to find an optimal threshold value. The binary search
adjusts the threshold value such that a weighted sum of the active luminance values in the
upper hemisphere of the radiance map is similar to that produced by a mask of the known
method, where the error metric is:
Ω ∙ − Ω ∙
Y is the pixel luminance and pixels and pixels respectively denote the sets of pixels
p t k
provided by the threshold mask and mask of the known method. From this it was determined
a threshold value of approximately 2 standard deviations above the mean captures the
necessary lighting conditions, as specified by equation 6. The use of this threshold results in
the pipeline producing images with lighting that is considered perceptually acceptable to
humans.
DIFFERENTIAL RENDERING FOR 360-VIDEO
In typical 360-video viewing scenarios using a HMD 122, a user’s head rotations tracked by
the sensor of the HMD 122 will be mapped to a virtual camera centred at the spherical
coordinates of the 360-video. Then, the virtual camera will project a part of 360-video into
the HMD screen as shown in Figure 8. It provides the immersive illusion that the users view
is navigating the surrounding 360 panoramic environment. In a general capturing setup for
360 panoramic videos, the camera is located at the origin of the spherical coordinates
mapping the panoramic scene. If the camera is in a fixed position, or camera and virtual
objects move together while maintaining relative position between them, the virtual objects
will be in the same position at the camera coordinates in every frame of the 360-video. Since
the HMD viewer’s position will be in the camera origin when viewing the 360-video, the
position of the virtual objects will be the same in every frame when rotating the head, and
therefore sophisticated camera tracking is not actually required for differential rendering
with 360-video as the backdrop.
In MR360, the differential rendering and composition 214, as shown in Figure 10, is applied
in a real-time context to seamlessly composite virtual objects into the 360-video background.
This involves processing a frame (step 1002 and Figure 9(a)) and generating two renders of
a local scene: one with objects (step 1004 and Figure 9(b)), and one without (step 1006 and
Figure 9(c)). The local scene is a piece of geometry representing the compositing surface as
shown in Figure 9c. The first render (1004 and Figure 9(b)) is generated by adding the
objects and using the IBL and IBS processing described above. The second render of only
the local scene (Figure 9(c)) is generated directly from the input background frame (Figure
9(a)). The object matte (Figure 9(d)) is used as a mask to remove the virtual objects from
the first render before the difference render is generated. The per-pixel difference between
the first and second renders is then generated to produce the difference render (step 1008
and Figure 9(e)) which extracts the effect that the objects had on the local scene, such as
shadows and reflections as shown in Figure 9(e). The difference render is then applied to the
background frame (Figure 9(a)) and the virtual object previously removed using the mask
(Figure 9(d)) is returned to the frame to produce the final composite image of the frame (step
1010 and Figure 9(f)). This composition method captures shadows and screen-space
reflections in real-time and applies the global illumination effects such as shadows and
reflections onto the 360 background (1010) as depicted in Figure 9(f).
Image Based Lighting in UE4
The node based material editing interface of the UE4 engine 170 is used to build the IBL
rendering process using GPU shaders. Render-to-texture shaders compute a diffuse radiance
map and mipmaps for specular maps; and four levels can be used to cover different
roughnesses. Low resolution radiance maps can be computed for perceptually optimised
IBL to provide a high frame rate.
The IBL output data is applied to the virtual objects using UE4’s post-processing pipeline.
Since UE4 170 uses deferred shading, per-pixel properties (e.g. diffuse colour, surface
normal, roughness, etc.) can be accessed in the post-processing stage via G-buffers. The IBL
output consists of the diffuse component (diffuse radiance map sampled by the surface
normal), and the rough specular component (a mipmap of specular radiance maps in different
roughness, sampled by the surface reflection vector). Shading the IBL result in a post-
process has the benefit of easy integration with pre-made assets and projects in UE4.
Image Based Shadowing in UE4
To facilitate using the detected lights to control UE4 scene properties and to make the
implementation of breadth-first search for light detection easier, the system 100 executes the
light detection process 212 entirely on the CPU 108. To achieve real-time performance, the
process 212 resamples the input frame to a fixed resolution of 600 x 150 (top half only)
before performing light detection.
The Media Framework of the UE4 engine 170 is modified to get access to the raw video
frame before it is uploaded to the GPU 114. Light detection 212 is run whenever the video
decoder thread delivers a new frame to its target, a Media Texture object. As such, the light
detection just has to keep up with the video frame rate and does not affect rendering
performance. New properties added to the Media Texture are used to control the light
detection and to export the results such that they can be used by a Blueprint visual
programming interface in UE4 to update the directional lights in the scene. The detected
light’s spherical coordinates are used to set the direction of the UE4 light. The detected
irradiance is used to set the intensity and colour of the light.
UE4’s dynamic shadowing procedure for directional lights is implemented with cascaded
shadow maps and percentage closer filtering (PCF). By adjusting the shadow parameters
exposed by UE4, its PCF is used to achieve the effect of soft shadows. Controlling the
dynamic shadow distance parameter, provides control of the spatial resolution of the shadow
map. A lower spatial resolution causes the soft penumbrae produced by PCF to increase in
size.
The pipeline linearly maps and clamps light solid angles of [0.1sr; 0.6sr] to shadow distances
of [4000cm, 20000cm] and linearly maps and clamps the same range of solid angles to
cascade distribution exponents of in order to increase shadow detail for smaller lights. The
pipeline also reduces the intensity for larger lights which helps to reduce artifacts from a
lower shadow resolution.
Differential Rendering in UE4
Differential rendering is modified for high performance real-time mixed reality rendering in
the game engine 170 of the system 100. Diffuse and specular local scene reflectance are
separately addressed for differential rendering. A diffuse local scene captures shadows and
screen-space ambient occlusion (SSAO) from the virtual objects, and a specular local scene
captures screen-space reflections (SSR). To meet a high frame rate, the pipeline directly uses
the background video as the diffuse colour of the local scene. This grants consistent
composition without manual tweaking, and it has the added benefit of automatically adapting
to spatial colour variation in the compositing region of the video.
As discussed, the process 214 requires two renders as shown in Figure 10. The first is of the
entire scene including objects and local scene (1004), and the second is of the local scene
without objects (1006). If the pipeline only uses a diffuse bidirectional reflectance
distribution function (BRDF) for the local scene, the Lambertian colour value can be easily
computed in a post process. This allows the difference image (1008) to be generated in the
compositing stage, without taking a second render of the scene (local scene without objects).
Stereoscopic Mixed Reality Rendering
To compose stereoscope renderings of 3D virtual objects into a monoscopic 360-video
background, stereo 360-video with stereo VR rendering is ideal but 360 stereo-video is
expensive and impractical. Mono 360-video with stereo VR rendering, as discussed above
is relatively simple approach but has parallax artifact. With input mono 360-video 201 the
stereo VR rendering discussed above can be adjusted to obviate the parallax artifact in
MR360.
This is achieved by adjusting the 360-video rotation angle use convergence angle of the
rendered virtual object. In mono 360-video and mono VR rendering, the center points (black
line) are determined as shown in Figure 11. For a stereo VR scene, the 360-video is rotated
for each eye to match the center points (in AR it is a gaze point or marker position for the
virtual object) as shown in Figure 12. The pipeline 200 adjusts the process by adjusting the
rotation depending on the distance of the virtual object from the centre as shown in Figure
13. A closer object will give rise to more rotation, as shown in Figure 13(a), and far away
object will generate less rotation (less parallax btw fore and background, as shown in Figure
13(b)). This is similar to simulating convergence of eyes using rotations.
Synthetic Video
Whilst MR360 allows 360 video to be adapted in real-time, MR360 can also be used with
pre-rendered synthetic or offline generated video as the background or lighting source,
which may be required for visual special effects (VFX) films and games. This is achieved
by first pre-rendering a full 3D synthetic background scene using an omnidirectional virtual
camera and saving the result as a 360 video. This 360 video is then used as the lighting
source and back drop in the MR360 pipeline during the composition stage and in particular
during differential rendering, the 3D virtual objects are rendered onto the 360 video
background scene.
Many modifications will be apparent to those skilled in the art without departing from the
scope of the present invention as herein described with reference to the accompanying
drawings.
Claims (18)
1. A rendering process including: receiving low dynamic range (LDR) video data; 5 generating radiance maps from said LDR data; inverse tone mapping LDR data of said maps to generate high dynamic range (HDR) data for said maps; and receiving at least one virtual object and applying image based lighting to said at least one virtual object using said HDR data of said maps, including applying a dynamic threshold 10 to said radiance maps to detect prominent pixels, and converting said prominent pixels into directional lights for image based shadowing associated with said at least one virtual object.
2. The rendering process as claimed in claim 1, wherein said maps include a diffuse radiance map and a specular radiance map.
3. The rendering process as claimed in claim 1 or 2, wherein the LDR video data is used as a background for differential rendering to composite said at least one virtual object into the background. 20
4. The rendering process as claimed in claim 3, wherein said differential rendering includes generating a first render of a local scene with said at least one object, generating a second render of said local scene without said at least one object, generating a differential render from the first and second renders, and generating a composite render with said at least one object and with lighting and shadowing adjusted based on said differential render.
5. The rendering process as claimed in claim 3 or 4, wherein said video data is monoscopic and the process includes stereoscopic rendering said at least one virtual object into said background. 30
6. The rendering process as claimed in claim 5, wherein said stereoscopic rendering includes rotating a rendered scene relative to a centre gaze point based on the distance of said at least one virtual object to said gaze point.
7. The rendering process as claimed in claim 1, wherein said threshold may be applied to the LDR video data, HDR video data, or inverse tone mapped LDR video data.
8. The rendering process as claimed in claim 1 or 7, wherein said threshold is applied to the upper half of frame of said radiance map.
9. The rendering process as claimed in claim 1, 7 or 8, wherein said threshold is 10 determined for each frame using statistical metrics to produce a perceptually acceptable render.
10. The rendering process as claimed in claim 9, wherein said threshold is determined for each frame of said radiance maps using the mean μ and variance of the luminance of 15 the pixels of each frame, and the threshold is + 2 .
11. The rendering process as claimed claim 10, wherein a blur is applied to the pixels before applying said threshold. 20
12. The rendering process as claimed in claim 11, including selecting said pixels above said threshold and representing a high percentage of the radiance of the respective frame as the detected salient lights for image based shadowing.
13. A rendering process including applying a perceptually based threshold to radiance 25 maps to detect prominent pixels, and using said prominent pixels as lights for image based shadowing associated with at least one virtual object, wherein the prominent pixels are detected from LDR video data, HDR video data, or inverse tone mapped LDR video data and used to provide shadows for real-time differential rendering, and the radiance maps are generated from the video data.
14. The rendering process as claimed in claim 13, wherein said threshold is applied to the upper half of frame of said radiance map.
15. The rendering process as claimed in claim 13 or 14, wherein said threshold is determined for each frame using statistical metrics to produce a perceptually acceptable 5 render.
16. The rendering process as claimed in claim 15, wherein said threshold is determined for each frame of said radiance maps using the mean μ and variance of the luminance of the pixels of each frame, and the threshold is + 2 .
17. The rendering process as claimed in any one of the preceding claims, wherein said video data represents a panoramic video or 360 video.
18. A rendering computer system including: 15 a display; a graphics processing unit (GPU); a computer processing unit (CPU); and processing modules including GPU shaders to provide a pipeline to perform the rendering process as claimed in any one of the preceding claims.
Applications Claiming Priority (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
AU2016904448 | 2016-10-31 | ||
AU2016904448A AU2016904448A0 (en) | 2016-10-31 | A rendering process and system | |
PCT/AU2016/051137 WO2018076038A1 (en) | 2016-10-31 | 2016-11-22 | A rendering process and system |
Publications (2)
Publication Number | Publication Date |
---|---|
NZ752912A NZ752912A (en) | 2021-10-29 |
NZ752912B2 true NZ752912B2 (en) | 2022-02-01 |
Family
ID=
Similar Documents
Publication | Publication Date | Title |
---|---|---|
AU2016427777B2 (en) | A rendering process and system | |
Rhee et al. | Mr360: Mixed reality rendering for 360 panoramic videos | |
US11210838B2 (en) | Fusing, texturing, and rendering views of dynamic three-dimensional models | |
US10839591B2 (en) | Stereoscopic rendering using raymarching and a virtual view broadcaster for such rendering | |
US9854176B2 (en) | Dynamic lighting capture and reconstruction | |
Fischer et al. | Stylized augmented reality for improved immersion | |
Supan et al. | Image based shadowing in real-time augmented reality | |
US20070139408A1 (en) | Reflective image objects | |
US10171785B2 (en) | Color balancing based on reference points | |
US9183654B2 (en) | Live editing and integrated control of image-based lighting of 3D models | |
US11823321B2 (en) | Denoising techniques suitable for recurrent blurs | |
CN112262413A (en) | Real-time synthesis in mixed reality | |
Schwandt et al. | A single camera image based approach for glossy reflections in mixed reality applications | |
US8314797B1 (en) | Method and apparatus for irradiance computation in 3-D computer graphics | |
Ma et al. | Neural compositing for real-time augmented reality rendering in low-frequency lighting environments | |
US11302068B2 (en) | Area light shadows in computer generated scenes | |
Fischer et al. | Enhanced visual realism by incorporating camera image effects | |
NZ752912B2 (en) | A rendering process and system | |
Dai | Stylized rendering for virtual furniture layout | |
Schwandt | High-Quality Illumination of Virtual Objects Based on an Environment Estimation in Mixed Reality Applications | |
JP7190780B1 (en) | Image processing program, image processing apparatus, and image processing method | |
Schwandt et al. | Illumination in Mixed Reality | |
JP2007264633A (en) | Surround visual field system, method for synthesizing surround visual field relating to input stream, and surround visual field controller | |
CN115375824A (en) | Interactive panoramic space ray tracking method based on RGBD panoramic video | |
Hoffman et al. | Real-time photorealistic terrain lighting |