JP7123041B2 - Fast generation of ray-traced reflections of virtual objects in real-world environments - Google Patents

Description

The present invention relates to a new and improved method of performing ray tracing with reduced computational complexity.

Augmented Reality (AR) is a live view of a physical real-world environment whose elements are augmented by computer-generated graphics. Information about the augmented object is overlaid on the real world. Augmentation is traditionally real-time and visual background with environmental elements. With advanced AR technology, information about the real world around the user is interactive and digitally manipulable. Because AR brings elements of the digital world into the human perceived real world, users can experience and interact with augmented reality, depending on the level of realism of the augmented objects and their integration with the real environment. do.

In the prior art, graphics used in AR were based on traditional raster technology. However, raster graphics are very mediocre in favor of speed due to the trade-off between quality and speed. The visual background lacks the necessary elements of realism such as reflections, refractions, color bleeding, caustics, and the like. In the prior art, high-quality computer-generated realism is seen in the motion picture industry, made possible by the use of ray tracing computer-generated graphics.

Ray tracing is a computer graphics technique for generating images by tracing rays and simulating the effects of their encounters with virtual objects. The idea behind ray tracing is to find a mathematical solution for computing the intersection of rays with various types of geometry by solving for point-to-point visibility. This technique makes it possible to produce a high visual realism. In low-end raytracing, the light source is the only lighting object in the scene.

More realistic imaging is achieved by high-end ray tracing called path-tracking, based on global illumination. Global illumination considers not only the light coming directly from the light source, but also the subsequent cases where light is returned from other surfaces in the scene, whether reflected or not.

Path tracing, called Monte Carlo ray tracing, renders a 3D scene by randomly tracing a sample of possible light paths. By repeatedly sampling any given pixel in the image, the mean of the samples will eventually converge to the correct solution of the rendering equation, making it one of the most physically accurate 3D graphics rendering methods in existence. Become. Path tracking can produce images that are true to reality and indistinguishable from photographs (eg, Avatar movies). The visual quality is higher than raytracing, but the computational cost is much higher.

The most time consuming tasks in prior art ray tracing are traversing acceleration structures and intersection testing between rays and polygons. Every single ray traverses over the acceleration structure (eg, K-tree or BVH-tree) and searches for candidate polygons to intersect. These traverses become large time consuming actions, typically requiring 60% to 70% of the imaging time. All candidate polygons involved in the search must then undergo a line-triangle intersection test to determine the earliest hit along the ray's path.

A layout of a prior art ray tracing method is depicted in FIG. First, the acceleration structure 10 needs to be constructed. Construction is done as a preprocessing step and requires much more time than generating a single image. In general, build time depends on the size of the scene. The larger the scene, the longer the build time. Any large change in the scene requires reconstruction of the acceleration structure. Memory size is typically doubled by acceleration structures. The tracing of rays 12 is based on extensive traversal of the accelerating structure 11 as each ray traverses over the structure to search for intersections between rays and various scene objects. The resulting intersections are brightened, textured, shaded 13 and collapsed into image pixels.

There are two major drawbacks associated with the use of acceleration structures in prior art ray tracing. (i) they need to be repeatedly reconstructed for scene changes; (ii) traversing these structures is time consuming; Both disadvantages conflict with the real-time requirements of AR.

Therefore, the main objective of the present invention is to accelerate the performance of global illumination ray tracing to real-time and make it suitable for AR.

Another object of the present invention is to reduce the complexity of ray tracing.

Another object of the invention is to reduce the power consumption of ray tracing.

Another object of the present invention is to enable global illumination raytracing at the processing level of consumer computing devices.

Some embodiments of the invention apply to both Augmented Reality (AR) and Virtual Reality (VR). AR is a live view of a physical real-world environment whose elements are augmented by computer-generated graphics. VR replaces the real world with a simulated one. Augmentation is traditionally real-time and visual background with environmental elements. User experience and interaction are directly affected by the level of realism in AR and VR.

In the prior art, the imaging of augmented objects occurs with conventional raster graphics due to their high speed. With advanced AR technology (eg, adding computer vision and object recognition), information about the real world around the user becomes interactive and digitally manipulable. Information about the environment and its objects are overlaid on the real world.

However, typical raster graphics technology is mediocre in image quality and visual context with real-world environments. The desired high quality of computer-generated realism can be seen in the film industry today and is made possible by the use of global illumination ray tracing, or path tracing. Unfortunately, it is not suitable for AR due to its very high computational complexity, which results in long production times and requires expensive computational firmware.

The present invention teaches an innovative way to achieve real-time route tracking that reduces computational complexity and power consumption and is suitable for the processing level of consumer computing devices. Aspects of the present invention allow for concentrating on selected objects to generate photorealistic images of the objects that are realistically overlaid on the preset environment.

In embodiments of the present invention, the prior art acceleration structure is replaced by a new and novel device, the Dynamic Alignment Structure (DAS). DAS is a means for performing intersections between secondary rays and scene geometry in large ray families, providing speed and computational complexity.

FIG. 2 illustrates the path tracing stages of the present invention. The main difference compared to the prior art (Fig. 1) is the absence of acceleration structures. These structures are replaced by DAS devices 21 . There is no preprocessing for reconstruction and no acceleration structure traversal.

A DAS is an aligned projection of rays used to carry secondary rays associated with existing hitpoints. Instead of individually firing secondary rays for each single hitpoint (or in small batches of rays) as in the prior art, Applicants do it in batches to reduce costs.

The present invention is herein described, by way of non-limiting example only, with reference to the accompanying drawings, in which like designations represent like elements. It is appreciated that these drawings merely provide information on typical embodiments of the invention and are therefore not to be considered limiting in scope.

1 is a prior art block diagram of ray tracing; FIG. 1 is a block diagram of the ray tracing of the present invention; FIG. Prior art, secondary rays that produce global illumination at the point of ray/object intersection. This is the basic mechanism of DAS. Multiple DAS projections emitted in randomly varying directions. Generate hitpoints by continuous DAS projections. Opening scene consisting of two triangles and two primary HIPs. Initial segment of the DAS beam directed towards the HIP. The main segment of the DAS beam carries secondary beams emitted from the HIP. DAS renders HIP data only. DAS renders the scene geometry. Objects along the initial segment of the DAS ray are discarded. It is a flow chart of DAS. Various cases of secondary rays carried by a single DAS projection. An augmented object that stands on a semi-reflective real desk. Direct imaging of extended objects. Primary rays are emitted from the camera and slanted in various directions. Direct imaging of extended objects. Secondary rays are generated by multiple DAS firings of perspective projection. Direct imaging of extended objects. Secondary rays are generated by multiple DAS firings of parallel projection to preserve the rendering of larger data. Fig. 10 is a flow chart for generating a direct image of an augmented object; Reflection imaging of extended objects. A primary ray is launched into the reflective area and is slanted in multiple iterations. Reflection imaging of extended objects. The secondary rays carried by the DAS projection are launched through the cluster of primary HIPs to the object. Reflection imaging of extended objects. Multiple DAS projections are randomly tilted. Reflection imaging of extended objects. The contribution of secondary rays to the aggregate light energy follows a BRDF function. 4 is a flow chart of reflectance imaging. Color bleeding effect of extended objects for primary rays in the environment. Color bleeding effect of extended objects of secondary rays in its environment. Fig. 10 is a flow chart for creating a color bleeding effect; Collect the sampled light values at the pixel origin. Hardware for AR and VR.

The principles and operation of apparatus according to the present invention may be understood with reference to the drawings and accompanying description, wherein similar components appearing in different drawings are indicated with the same reference numerals. The drawings and descriptions are conceptual only. In practice, a single component may implement one or more functions, or each function may be implemented by multiple components and devices. It will be readily appreciated that the components of the present invention as generally described and illustrated in the drawings can be arranged and designed in a wide variety of different configurations. Consequently, the following more detailed description of embodiments of the apparatus, systems, and methods of the invention, as depicted in the drawings, is intended to limit the scope of the invention as claimed. rather, it is merely representative of an embodiment of the present invention.

Throughout the discussion of the specification, terms such as "process," "calculate," "calculate," "determine," "generate," or "create," etc., will be apparent from the discussion below, unless otherwise stated. The use of the term refers to data represented as physical quantities, e.g. It is understood to refer to the actions and/or processes of a computer or computing system, or processor or similar electronic computing device, that manipulates and/or transforms other data that are similarly represented as physical quantities in a display device. be.

Embodiments of the present invention use terms such as processor, computer, apparatus, system, subsystem, module, unit, and device (in singular or plural forms) for performing the operations herein. Sometimes. It may be specifically constructed for the desired purposes, or it may comprise a general-purpose computer selectively activated or reconfigured by a computer program stored in the computer. Several technical terms specifically related to this disclosure are defined herein.

Computer graphics pipeline refers to computer 3D rendering, the most common form of 3D polygon rendering, distinct from ray tracing and ray casting. Specifically, in raycasting, when a ray is launched at a point where the camera is, and the ray hits a surface, the color and illumination of the point on the surface hit by the ray is computed. In 3D polygon rendering the opposite happens, the area in the camera's field of view is calculated, and rays are created from all parts of all surfaces in the camera's field of view and back to the camera. Graphics pipelines are typically used for real-time rendering.

Rendering a projection is a 3D computer graphics process that automatically transforms a 3D wireframe model into a 2D image for rendering on a computer. The projection can be perspective, parallel, reverse, or other shapes.

A render target is a feature of modern graphics processing units (GPUs) that allows rendering a 3D scene to an intermediate memory buffer or render target texture (RTT) instead of a framebuffer or backbuffer. This RTT can then be manipulated by a pixel shader to perform lookups or apply effects to the final image.

Primary rays are the first generation of rays in ray tracing that are cast into the scene from the camera or eye to resolve visibility, i.e. to find out if a primary ray intersects a surface.

Secondary rays in ray tracing originate from primary rays at their ray-polygon intersections. They are used to calculate things like shadows, reflections, refractions, etc. Applicants herein also use the term collectively for all successive generations, such as the third ray, fourth generation, and so on.

Global illumination is a general term for a group of algorithms used in 3D computer graphics that are meant to add more realistic lighting to a 3D scene, not only the light that comes directly from the light source (direct lighting), but also We also consider the subsequent case (indirect lighting) where rays from the same source are reflected by other surfaces in the scene, whether reflective or not.

Color bleeding in computer graphics is the phenomenon of objects or surfaces being colored by the reflection of indirect light from nearby surfaces. This is the visibility effect that appears when the scene is rendered with full global illumination.

Fast structures such as grids, octrees, binary space partitioning trees (BSP trees), kd-trees, BVH (bounding volume hierarchy) are used in ray tracing to solve visibility, and simple Allows rendering time improvements in speed and efficiency compared to raytracing.

GPGPU (General Purpose Computing in Graphics Processing Units) is graphics processing, which typically handles computations for computer graphics only, to perform computations in applications traditionally handled by a central processing unit (CPU). The second is the use of units (GPUs).

Preset scenes in AR replace the real-time world. Preprocessed environment scene for inclusion of augmented objects.

Objects can represent primitives (polygons, triangles, solids, etc.) or complex objects made up of primitives.

The hitpoint is the point where the ray intersects the object. Also called HIP.

Visibility—Given a set of obstacles in Euclidean space, two points in the space are said to be visible to each other if the line segment connecting them does not intersect any obstacles.

Scene—A collection of 3D models and light sources in world space in which a camera may be placed to describe a scene for 3D rendering. Scene model elements include geometric primitives: points or vertices, lines or edges, polygons or faces.

Clipping, in the context of computer graphics, is a method of selectively enabling or disabling rendering operations within a defined region of interest.

The processes/devices and displays presented herein are not inherently related to any particular computer or other apparatus unless specifically stated otherwise. Various general-purpose systems may be used with programs in accordance with the teachings herein, or it may prove convenient to construct a more specialized apparatus to perform the desired method. good. The desired structure for a variety of these systems will appear in the description below. Additionally, embodiments of the present invention are not described with reference to any particular programming language. It will be appreciated that a variety of programming languages may be used to implement the teachings of the invention as described herein.

Diverting from the prior art, aspects of the present invention teach how to implement ray tracing in both reduced computational complexity and increased speed. One aspect of the invention relates to path tracking, which is high quality ray tracing based on global illumination. Its superior performance stems from a different technical approach to resolving intersections between rays and scene objects. It is based on the Dynamic Alignment Structure (DAS), which is a projection of collimated rays used to carry secondary rays emitted from existing hitpoints. The DAS mechanism can be implemented by either the GPU (Graphics Processing Unit) graphics pipeline or the CPU (Central Processing Unit). The mechanism can resolve ray-triangle intersections by using conventional graphics mechanisms, replacing the expensive traversal of acceleration structures in the prior art.

DAS Mechanism In one embodiment, the DAS mechanism is applied to path tracking based on global illumination. Global illumination (or indirect illumination) considers not only light coming directly from the light source, but also light reflected by surfaces in the scene, whether specular, diffuse, or semi-reflective. FIG. 3 depicts a sampling of diffuse interreflections from the surrounding environment at a given surface point. In order to achieve global illumination at the diffuse surface, the sampling ray needs to emanate from the hit point (HIP) 33 . A HIP is the result of a previous encounter between a ray (first or second order) and a triangle. Sampling is done by firing rays in random directions within the boundaries of hemisphere 31 . The hemisphere is oriented so that its north pole is aligned with the surface normal.

The basic mechanism of DAS is illustrated in FIG. 4, showing that DAS is associated with perspective projection, but other projections such as parallel or reverse are possible as well. The DAS structure includes projections of rays that pass through clusters of HIPs, eg, 403, 405, 408, and target objects. A DAS beam incident on the HIP is used as a carrier for secondary beams. For example, a DAS ray incidentally incident on HIP 408 carries secondary ray 406 . This ray is within the hemisphere 407 of the HIP. A DAS ray incident on a HIP or triangle may consist of a starting point 400 and have two intersections, the first with the HIP 408 and the second with the scene object (triangle) 409 . Additional secondary rays associated with the same HIP 408 can be independently generated by additional DAS structures, carrying additional secondary rays in other directions within the same hemisphere.

According to one embodiment of the invention, DAS projection can be implemented by a CPU software graphics pipeline, but the most efficient device is a GPU hardware graphics pipeline. This well-known computer graphics pipeline is the process of turning a 3D model into what a computer displays. The graphics pipeline consists of two subsystems, geometry and rasterization. First, all objects within the DAS frustum are transformed by the geometry subsystem according to the camera view. Then, in the raster subsystem, the ray/triangle intersection points are selected by the Z-buffering mechanism. For example, DAS ray 402 in FIG. 4 originates from projection origin 400 and intersects two objects, 408 and 409 . Which of these two objects is selected depends on the API directive (Direct3D or OpenGL) that controls Z-buffering.

An exemplary cluster of three existing HIPs 405, 408, and 403 with their underlying triangles is shown. The HIP 405, 408 secondary rays are driven by the DAS structure. As an example, carrier beam 402 is incident on HIP 408 . From the HIP encounter onwards, it becomes the secondary ray 406 associated with the HIP and seeks intersection 409 . DAS is only relevant for HIPs whose hemispheres are oriented in the direction of the projection, such as 405 and 408, but not 403. The DAS method is expressed mathematically as follows.

・Ｌ_ｃは同じ入力からの一連のマッピングのセットであるので、任意の入力頂点のための複数のターゲット頂点が存在することに留意すべきである。
全ての頂点ｖ∈Ｖ_ｄｃを、

における全ての可能なジオメトリｇ∈Ｇに投影する代わりに、

における全てのクラスタｃ∈Ｃ_ｄに全ての可能なジオメトリｇ∈Ｇを投影する。
・Ｒ^３においては、高速マッピング（プロジェクション）を並行して達成するために、従来の３Ｄグラフィックスパイプライン（ラスタハードウェア）を利用することができる。
スループット／オーバーフィットにおけるＣ_ｄ／Ｌ_ｃを最適化して、
・クラスタ当たりの平均の最大頂点数（スループット）
・全ての頂点にフィットする（オーバーフィットする）ジオメトリの最小（離散）プロジェクション数
・前処理／実行時間制約
を有する。
Ｌ_ｃは、物理的シナリオをシミュレートするために各ｖ∈Ｖ_ｄｃのための可能な分布セグメントを表す擬似ランダム出力を有するように選択される。 Let T be a d-level tree graph and V be its top vertex of a geometry G in space.
Define Vd, which is the vertex in V at level _d .
Let C _d be the partition of V _d into clusters.
Extend T to d+1 levels by finding V _d+1 .
Choose a cluster c ε C _d with vertices at V _dc and define L _c , a set of mappings from V _dc to V _{d+1 c} such that V d+1 _c is the projection of the vertices at V _dc on top of G .

• Note that there are multiple target vertices for any input vertex, since L _c is a set of successive mappings from the same input.
Let all vertices v∈V _dc be

Instead of projecting onto all possible geometries gεG in

Project all possible geometries gεG onto all clusters cεC _d in .
• In R3, a conventional ^3D graphics pipeline (raster hardware) can be utilized to achieve fast mapping (projection) in parallel.
Optimizing C _d /L _c in throughput/overfit,
・Average maximum number of vertices per cluster (throughput)
- Minimum number of (discrete) projections of geometry that fits (overfits) all vertices - Has pre-processing/run-time constraints.
L _c is chosen to have pseudo-random outputs representing possible distribution segments for each vεV _dc to simulate a physical scenario.

Multiple DAS projections are projected onto the scene, or portions thereof, in slightly different directions, and each direction can be taken randomly. As a result, multiple neighborhood samples can be taken at each HIP for global illumination. This is illustrated in FIG. HIP 507 is the primary hitpoint created by primary ray 508 emanating from image pixel 500 . Three subsequent DAS projections—501, 502, and 503 visit HIP 507. Each DAS carries one secondary ray for HIP 507 . Each of the three secondary rays achieves a different sample from the environment, ie from surfaces 504, 505 and 506 respectively.

There may be various ways to exploit the HIP produced by continuous DAS projection. According to one embodiment shown in FIG. 6, all newly generated HIPs contribute their data. In this example, four sequential DAS projections are used. Assuming 601 is the primary HIP previously generated by the primary firing from image pixel 600, its first subsequent HIP 602 is the product of the first DAS projection. A second DAS projection with a different orientation produces HIP children 603 and 604 . A third DAS projection produces HIPs 605 , 606 and 607 . A single child HIP 608 is then generated as a result of the fourth DAS projection. Light samplings from all HIPs must be averaged and converged to the correct solution of the rendering equation for image pixel 600, eg 608 and 605 converge to 603, 603 converge to 602, then converges to 601. Finally, primary HIP 601 converges the aggregate sample values of all its children 602, 604, and 607, and the result enters pixel 600 as a partial contribution to pixels among the other primary HIPs. A correct rendering equation for all convergence will ultimately produce a physically accurate image.

Secondary rays are meant to intersect scene objects, as shown in Figures 7a-7c. FIG. 7a shows an initial scene consisting of two triangles 711,712 and two primary HIPs 713,714. In Figure 7b, a DAS projection 721 is emitted towards the HIP. Since HIP 714 faces the opposite direction, it is excluded from the current DAS. The HIP 713, positively aligned with the projection, activates secondary rays. Further, as shown in FIG. 7c, the carrier beam associated with HIP 713 is split into two parts, initial segment 731 and main segment 732 . Initial segment 731 extends from the DAS origin to HIP 713 . Its function is to identify the HIP and its depth. Once the depth Z _HIP is found, the main segment from the HIP to intersection 733 acts as a carrier for secondary rays searching for the intersection. It hits the object at 733, which creates a secondary HIP.

According to one embodiment of the present invention, DAS projection takes advantage of the Z-buffering mechanism of the GPU, as illustrated in FIGS. 8a and 8b. The Z-buffering mechanism must always discard objects before the HIP and only start looking for objects after the HIP. It is based on selective use of the GPU's Z-buffering mechanism, eg, the function glDepthMask of the OpenGL graphics library. This is done in two separate rendering passes. In the first pass, the HIP is rendered as the only object in the scene, ignoring geometric data and generating a HIP depth mask. In the second pass, the HIP depth mask is used to render the scene geometry. The first pass is shown in Figure 8a. Carrier beam 812, which overlays HIP 811, is split into two segments and processed in two passes. The initial segment rendered during the first pass extends from camera 813 to the HIP. The HIP depth value Z _HIP is registered in the HIP depth mask 810 . The depth value is retained during the second pass to exclude all objects en route to the HIP for later use. In the second pass (Fig. 8b) the geometric data is rendered starting at depth Z _HIP (811), eg triangle 825 is ignored. The main segment, the carrier of the secondary ray, hits the triangle at 823 . The results of the second pass are stored in render target 820 . Rays that miss the HIP are completely discarded and considered as the initial segment as a whole. Once the render target is complete, the correct ray/triangle intersection 823 is found by inspecting the render target at the u,v coordinates of the DAS carrier ray. Intersecting triangles implement important data such as color, light, normals, materials, and so on.

The DAS flowchart in FIG. 8c summarizes how to create and use the DAS mechanism. A DAS projection targets an object (eg, an augmented object) or sub-scene and passes through a cluster of HIPs to generate secondary rays for the HIP. DAS is issued twice. The first is fired at the HIP data only, ignoring the scene's geometric data and generating a HIP depth mask (831). A second time, the same DAS projection is issued (832). This time the geometry data of the scene is rendered, ignoring the HIP data. A depth mask 810 is used at the starting point of the secondary rays. Secondary rays are propelled in the DAS projection and seek intersections with the geometric data. The render result, the render target, and the 2D projection of the 3D subscene are essentially the set at all intersections between the secondary rays and the geometric data of the scene. An intersection point directly associated with a particular HIP can be found by searching for coordinates u', v' in the render target that match the coordinates u, v of the HIP. The color and light values at the intersection point are fed back to the HIP to provide global illumination samples (833). Finally, the intersection is stored in the HIP repository as the next generation HIP (834).

Various cases of secondary rays are shown in FIG. 9, all carried by a single DAS projection. Ray 900 consists of two segments. The initial segment extending from camera 909 to HIP 903 discards triangle 906 while the main segment encounters triangle 907 at intersection 905 . A secondary segment of ray 902 does not hit the object. Ray 901 does not encounter a primary HIP, so it is considered the initial segment in its entirety, ignoring triangle 908 .

The DAS mechanism of the present invention can be implemented in AR, among other fields. One of its embodiments enables local path tracking, focused on rendering one or more objects in a scene and full integration between augmented objects and the real environment. FIG. 10 shows an example of an augmented object, a Buddha statue 101, standing on a real desk with a semi-reflective surface 107. FIG. It is not only the image of the object 101 but also its reflection 102 that is needed to produce the realistic appearance of the image. Effects that augmented objects may have on their real environment result in reflections, shadows, and color bleeding that modify the preset environment. On the other hand, environmental influences on augmented objects may result in illumination and reflection of the object itself.

According to one embodiment of the invention, the image of the object and the reflection of the object in the environment are generated by two separate tasks and the combined result is supplied to the image pixels.

Direct Imaging of Extended Objects A basic image of an extended object can be reconstructed from just the primary HIP covering the surface of the object. However, for global lighting effects in an image such as the environment reflected at an object, secondary rays emanating from the object into its environment are required. The rendering task of extension object 110 is shown in FIG. For simplicity it is illustrated in a 2D drawing. A camera 113 emits a primary ray 114 onto the augmented object and searches for intersections with the object. It is meant that these intersections become HIPs and are used as starting points for secondary rays for global illumination.

The primary ray firings are repeated, each with a slight change in direction, to achieve multisampling in the image pixels. The change of direction is made randomly to prevent unwanted patterns in the image. Multisampling contributes to anti-aliased image quality. In FIG. 11, three primary launches 115, 116 and 117 are shown.

A faithful and unified appearance of augmented objects in the scene is achieved with global illumination. The environment associated with global illumination is sampled by secondary rays emitted from the primary HIP towards relevant parts of the scene. If the object is reflective, the relevant part of the scene is that part of the visibility from the camera due to the reflection on the object.

For example, such a relevant portion can become sub-scene 123 of FIG.

Secondary rays are generated by the DAS structure for either perspective projection as in FIG. 12a or parallel projection as in FIG. 12b. In FIG. 12a, the DAS projection passes through the primary HIP (eg 128) and targets subscene 123. In FIG.

Since all successive DAS projections target the same sub-scene 123, this sub-scene can be clipped from the full scene, selectively allowing rendering operations within a reduced area, thus allowing the rendering process to a minimum.

Each of the multiple DAS projections are randomly generated in different directions from slightly different viewpoints to generate multiple secondary rays at each HIP. The use of randomness prevents unwanted patterns from appearing in the image. Secondary rays sample the global illumination for HIP (Fig. 3, 31) and integrate between the object and the environment. The sampled illumination affects the image depending on the object's material and its level of specularity or diffuseness, e.g. It just gives the amount of background lighting and, in the diffuse case, creates the object's reaction to the environment.

The higher the number of DAS projections, the better the global illumination coverage. However, a large number of projections may degrade performance. Therefore, there is a trade-off between image quality and performance.

A method for generating direct images of augmented objects is summarized in the flow chart of FIG. 12c. First, multiple primary projections are emitted 1231 from the camera (eye, point of view) to the augmented object to generate a cluster of primary HIPs. The portion of the scene targeted by the secondary ray should then be defined, possibly cropped as a subscene (1232), and the reference point for the DAS projection set according to the selected subscene. (1236). Secondary rays generated by the multiple DAS projections are then launched 1233 into the relevant subscenes. The result of the DAS projection is the determined subscene render target texture. A search for the intersection between the secondary ray and the determined subscene is done by matching the coordinates of the associated primary hitpoint and render target texture (1237).

Each primary HIP is provided with the light value of the corresponding intersection point between its secondary ray and the triangle it encounters (1234). The above procedure can be repeated if multiple subscenes are required. And finally, the intersection is added to the HIP repository as a new generation HIP (1235). The processed samples of color and light values, aggregated from all primary hitpoints, are converged onto the pixels of the image to create a full image of the augmented object affected by the 3D scene.

Reflection of Augmented Objects Reflection of an image of an object in an environment item is accomplished by following rays from the camera to surfaces in the scene and bouncing back toward the augmented object. Reflections on glossy surfaces or tiles enhance the photorealistic effect of 3D renderings. The degree of reflection depends on the reflectance of the surface (BRDF of the material).

First, reflective or semi-reflective surfaces (or items) in the real scene that may reflect augmented objects need to be identified. Applicant then directs a primary ray onto the surface or portion thereof where the object is intended to be reflected, producing a primary HIP. From these HIPs, Applicants emit secondary rays that target and sample the augmented object. This method of generating reflections is illustrated in Figures 13a and 13b. A primary HIP covering the intended reflective area is created by primary rays emitted from camera 133 through image screen 130 toward reflective area 134 . The location and boundaries of the reflective area 134 on the surface 132 are determined according to the camera position, the distance and size of the augmented object 110, and consideration of the principal direction 131 by Snell's law. Due to the multisampling in the image pixels, the primary firing is repeated multiple times. Each successive time, the primary projection deviates slightly from the main direction at random so that each pixel of the image gets multiple samples. The surface of the reflective area 134 becomes covered by a dense array of primary HIPs. The randomness of multisampling prevents unwanted patterns in the resulting image.

FIG. 13b illustrates the generation of reflected images by secondary rays. The reflection of augmented object 110 at surface 132 is reconstructed from sampled data in the primary HIP collected by firing secondary rays at the object. Applicants use geometric point 136, which is the reflection of camera 133 at surface 132, as a reference point for multiple DAS projections. Each projection originates from a different point that randomly deviates from the reference point 136 .

The DAS 135 shown in FIG. 13b begins with a reference point 136 oriented along an axis 139 pointing toward the center of the extension object 110 . The DAS carries secondary rays 138 that all start at the primary HIP (eg 137) and target the extended object.

To perform spectral sampling of the BRDF function in HIP, multiple DAS projections are randomly deviated from the reference DAS projections (starting at the reference point and having the axis of the projections directed toward the center of the extended object). The tilt from the reference DAS is done randomly and slightly off from the reference point 142 and central axis 145 as shown in FIG. 14a. Three DAS projections are shown.

Assuming that the reference DAS starts exactly at the reference point 142 and its axis 145 is centered, two other DAS projections start at nearby points 141 and 143 and their axes 144 and 146 are centered 145 deviate from As an example, Applicants have chosen HIP 140 from which three secondary beams 144, 145, and 146 are emitted, each carried by a different DAS.

The relationship between the deviation of a DAS secondary ray from the reference DAS and its contribution to the concentrated light energy is shown in Figure 14b. It is strongly related to the BRDF function 147 of the material 132 of the surface. Each of the three secondary rays 144, 145, and 146 emanate from the same HIP in different directions and are bounded by the hemispheres of FIG. As a result, the sampled data contributes to the aggregated light energy according to the BRDF function. Assuming the secondary ray 145 travels in the exact Snell direction, then it has the largest contribution at the peak of the BRDF function 147 . Secondary rays 144 and 145 have smaller contributions depending on the BRDF value at the distance from the peak.

A method for generating a reflection image of an augmented object is summarized in the flowchart of FIG. 14c. First, the area in the real scene where the augmented object should reflect is determined (1431). A plurality of primary projections are then emitted from the camera into the reflective area to generate clusters of primary HIPs (1432). Next, the position of the reflected camera as a reference point for the DAS projection and the central axis towards the augmented object need to be calculated (1433). Secondary rays generated by the DAS are then emitted toward the object. Multiple DAS projections are randomly tilted to deviate from the reference DAS (1434). The light values sampled at the intersection points are then fed 1435 to the origin of their respective HIPs. Finally, the intersection is added to the HIP repository as a new generation HIP (1436). These HIPs can serve to generate further generations of secondary rays.

Color bleeding is the phenomenon in which an object or surface is colored by the reflection of indirect light from nearby surfaces. It is a global lighting algorithm in the sense that the illumination reaching a surface does not just come directly from the light source, but also from other surfaces that reflect the light. Color bleeding is viewpoint independent and useful for all viewpoints. Color bleeding effects in AR or VR occur in the immediate vicinity of augmented objects. An example of creating a bleeding effect according to one embodiment of the present invention is illustrated in FIG. 15a. An augmented object 154 standing on the substrate 152 will create a color bleeding effect on the real substrate 152 . First, Applicants define the boundaries of the color-bleeding patch around the center of the dilated object where the color-bleeding appears. The size of the patch depends on the materials involved, the distance and the amount of light. Applicant then emits a primary ray from camera 153 onto patch 155 in the absence of the augmented object. A cluster of primary HIPs covering the patch is created. The primary shot is repeated multiple times, each time slightly off the main direction 151 . The primary direction 156 is from the camera towards the center of the standing position of the object.

Figure 15b illustrates the use of secondary rays. The color bleeding effect is reconstructed by sampling the object with secondary rays emitted from the primary HIP towards the object. Secondary rays are generated by DAS projection. DAS projection obtains the shape of the inverse projection 156, unlike reflection or direct imaging of extended objects. Each time, slightly off the main direction, multiple DAS projections are made. Assuming the correct rendering equations are used, samples of the object's surface are taken from the substrate, allowing calculation of the amount of energy at the substrate.

A method for producing color bleeding is summarized in the flow chart of FIG. 15c. First, the location and size of the color bleed patch in the scene are defined (1531). A plurality of primary projections are then emitted 1532 from the camera onto the color bleeding patches to generate clusters of primary HIPs. Next, the reference point at the center of the DAS projection is calculated (1533), and similarly the required shape of the inverse projection is calculated (1534). Secondary rays are then emitted (1535) by a plurality of DAS projections, each randomly off center of the DAS projection, and the light values at the intersections are fed (1536) to the primary HIP. In color bleeding, this is the only generation of HIP.

All sample values in the light value collection HIP need to be processed by the correct rendering equations for physically accurate results. Parameters considered are surface materials, scene geometry, hemispherical active area, and others. For a given image pixel, all HIPs generated by the primary firing from the pixel and the light contributions of all their secondary successors are aggregated, processed and converged to the source pixel for the image. need to be As shown in FIG. 16, the sampling from the object and from its environment are converged to image pixels 164 . A pixel receives input from the primary HIP 165 in the surface from the extension object and then collects values from its successive generations. The pixels also receive input from the reflective HIP 161 and its successive generations. The processing results of 165 and 161 are weighted and aggregated into image pixels 164 .

Implementation The heart of the invention is the DAS mechanism. When implemented in path tracing, it generates secondary rays and locates their intersection with scene objects, but precludes the use of prior art acceleration structures. A DAS mechanism based on a traditional raster graphics pipeline can be implemented by either a GPU hardware pipeline or a CPU software pipeline. The GPU's parallel structure makes the graphics pipeline more efficient than a general purpose CPU. A GPU is a special-purpose electronic circuit designed to accelerate the graphics pipeline. Where CPUs are made up of a few cores focused on sequential serial processing, GPUs are equipped with thousands of tiny cores designed for multitasking. There are two main types of graphics processors: integrated and discrete. DAS can be utilized either by a separate component in the system (discrete GPU) or by an embedded GPU in the CPU die (integrated GPU). Integrated GPUs are used in embedded systems, mobile phones, personal computers, workstations, and game consoles.

As detailed above, the computing task of creating augmented objects and their visual backgrounds within a preset scene is largely based on the graphics pipeline. For these tasks, the use of GPUs is of great advantage. There is also the additional task of collecting global illumination sampled values, processing these values according to the rendering equations, and converging the results to image pixels. Acquisition tasks associated with conventional processing can be implemented by either the CPU or the GPGPU. There are also additional tasks associated with the user's visual device 171, as shown in FIG. For augmented reality, they are wearable computer glasses that add information alongside or to what the wearer sees. Typically, this is accomplished via an optical head-mounted display (OHMD) or transparent head-up display (HUD) or built-in wireless glasses with an AR overlay that has the ability to reflect the projected digital image, through which the user can allow you to see. For virtual reality, viewing device 171 may represent a virtual reality headset that provides virtual reality for the wearer. VR headsets are widely used in computer games, but they are also used in other applications including simulators and trainers. They include stereoscopic head-mounted displays (providing separate images for each eye), stereo sound, and head motion tracking sensors. In any event, component 171 must typically be interfaced to the computing platform by API software implemented by the CPU.

As a result, embodiments of the present invention call for a combined CPU and GPU implementation, as generally illustrated in FIG. A GPU may represent discrete graphics, integrated graphics, or a combination of both (integrated graphics working together with discrete graphics).

Integrated graphics means that the GPU is integrated into the CPU die and shares memory with the processor. Since integrated GPUs rely on system RAM, they do not have the computing power of their discrete counterparts, they are contained on their own cards and have their own memory, VRAM. Integrated GPUs have lower memory bandwidth from system RAM compared to discrete graphics cards between the VRAM and the CPU core. This bandwidth, called the memory bus, can limit performance. Also, since the GPU is very memory intensive, integrated processing may have minimal or no dedicated video memory, so if competing with the CPU for relatively slow system RAM There is Discrete graphics chips outperform integrated GPUs to enable the highest graphics performance.

On the other hand, graphics cores in multi-core chips can work well with CPU cores to exchange big data, so sharing the same RAM memory can also be beneficial. Pure graphics tasks of imaging objects, reflections, and color bleeding result in big data of light values that need to be collected and calculated for rendering equations by the CPU core.

However, despite the performance advantages of discrete GPUs, their better power efficiency, affordability, portability, and versatility make them ideal for applications such as augmented reality, virtual reality, and computer games. It is desirable to use an integrated GPU for implementation of the invention. Integrated GPUs as building blocks of multi-core CPU chips are used in embedded systems, mobile phones, tablets, and game consoles.

In addition to using discrete or integrated GPUs, there is also the alternative of using hybrid systems with discrete and integrated GPUs that cooperate and alternate, depending on the task.

Claims (14)

A method for creating augmented objects in a three-dimensional scene utilizing a graphics pipeline mechanism for ray tracing, comprising:
a) emitting a primary rendering projection onto the extension object to generate a cluster of primary hitpoints;
b) determining the sub-scene space targeted by the secondary ray;
c) setting a reference point for the secondary rendering projection;
d) on each occasion,
1) emit a secondary rendering projection from the vicinity of the reference point through the cluster of primary hit points to generate a render target texture for the determined subscene;
2) searching for intersections of said secondary rays with said subscene;
3) preserving the intersection points for the next generation of secondary rays;
4) sampling light values at said intersection;
5) repeating feeding back the sampled values to the primary hitpoints multiple times, each time tilting the secondary rendering projection;
e) processing aggregate values at the primary hitpoints;
f) converging the processing results to image pixels. 2. The method of claim 1, wherein searching for intersections between the secondary rays and the determined subscene is done by matching coordinates of the primary hit points and the render target texture. 2. The method of claim 1, wherein the stored secondary ray intersections can serve as clusters of primary hitpoints for the next generation of secondary rays. 2. The processed color and light value samples aggregated from all the primary hitpoints are converged onto the image pixels to create a full image of the augmented object affected by the three-dimensional scene. The method described in . A system for ray tracing augmented objects in a three-dimensional scene using a graphics pipeline, the system comprising:
at least one graphics processor having memory;
at least one general purpose processor having memory;
a geometric database of the three-dimensional scene;
with render target memory and, at runtime,
a) the graphics processor emits a primary rendering projection onto the augmented object to generate clusters of primary hitpoints;
b) the sub-scene space is determined to be targeted by the secondary ray;
c) a reference point is set for the secondary rendering projection;
d) the secondary rendering projection is repeated multiple times, each time tilted, each time
1) A secondary ray is generated using the secondary rendering projection, the secondary rendering projection emanating from the vicinity of the reference point through the cluster of primary hit points to render a render target texture of the subscene. generate and
2) searching for intersections of said secondary rays with said subscene;
3) the intersection points are preserved for subsequent generations of secondary rays;
4) light values are sampled at said intersection points;
5) the sampled light values are fed back to the primary hitpoints;
e) the sampled light values are aggregated and processed for the primary hitpoints;
f) A system in which the processed sampled light values are converged into image pixels. 6. The system of claim 5, wherein the graphics processor is a discrete GPU with a hardware graphics pipeline. 6. The system of claim 5, wherein the graphics processor is an integrated GPU with a hardware graphics pipeline. A computer-based method for generating secondary rays in ray tracing from non-primary rays, the method comprising:
emitting additional rays into a three-dimensional scene after the primary rays are emitted, said scene comprising:
a cluster of existing hitpoints previously generated by the primary ray, and
a geometry object containing geometric data of said scene;
At intersections between said additional rays and existing hitpoints, secondary rays arise from said additional rays, each resulting secondary ray comprising:
Occurs at an existing hitpoint of the intersection,
become associated with the existing hitpoints of said intersection,
always retains its original firing direction,
used to search for intersections with geometric objects positioned in front of its associated existing hitpoint ,
A method wherein an intersection between a secondary ray and a geometric object occurs during a second projection of said additional ray . 9. The method of claim 8, wherein the intersection between an additional ray and an existing hitpoint occurs by emitting a first projection of an additional ray onto the cluster of existing hitpoints. 9. The method of claim 8, wherein the projected additional rays are used as carriers for secondary rays. 9. The method of claim 8, wherein the association between a secondary ray resulting from an additional ray and an existing hitpoint means that the result of intersection of said ray with a geometric object is relative to said existing hitpoint. described method. A computer-based method for rapidly generating ray-traced reflections of an augmented object into an image of a real-world environment utilizing a graphics pipeline, the method comprising:
a. identifying an area for creating a reflection of the augmented object;
b. emitting projections of primary rays from a camera into the identified regions to generate clusters of primary hitpoints;
c. projecting a secondary ray onto the augmented object through the cluster of primary hitpoints, thereby resulting in a render target texture;
d. identifying and sampling points of intersection between secondary rays and dilation objects from the render target texture;
e. and providing the sampled light values at the intersection points to their respective primary hit points ;
The method of claim 1, wherein the projection of the primary ray is repeated multiple times, each time the projection of the primary ray being emitted such that its direction randomly deviates from the direction of another time . 13. The method of claim 12 , wherein the position and boundaries of the region for creating the reflection are determined according to camera position, distance and size of the augmented object, and consideration of the main direction. 13. The method of claim 12 , wherein the secondary ray projection utilizes a Z-buffering mechanism of a GPU.

Images