EP1716543A1 - Device for the photorealistic representation of dynamic complex three-dimensional scenes by means of ray tracing - Google Patents

Abstract

The invention relates to a device for the photorealistic representation of dynamic complex three-dimensional scenes by means of ray tracing. Said device comprises at least one programmable ray tracing processor in which are implemented: special traversing commands; and/or vector arithmetic commands; and/or commands for establishing ray tracing acceleration structures; and/or at least one decision unit (mailbox) which prevents objects or triangles that have already been intersected by a beam during launching thereof when performing ray tracing from being intersected several times by said beam. The inventive device is organized such that several threads can be processed in parallel and several threads can automatically be processed in a synchronous manner, the device being provided with an n-level cache hierarchy and/or virtual memory management and/or a direct link to the main memory.

Description

 Device for the photorealistic representation of dynamic complex three-dimensional scenes using the ray tracing method

The invention relates to a device with which dynamic, complex three-dimensional scenes with high refresh rates can be represented on a two-dimensional display using real-time ray tracing hardware architecture. Dynamic scenes are scenes in which, in addition to the camera position, the geometry of the objects to be displayed can change from frame to frame. The invention is characterized above all by the fact that it supports a hierarchy structure of objects, that is to say the main scene can consist of several objects, each of which is composed of further objects, this nesting being able to be continued as desired. The objects located on the individual hierarchy levels can be moved individually as well as in a group. This makes it possible to create complex scenes with high dynamics and to keep the representation of the scene in memory small by using the same object at several points in the scene.

To implement these nested object levels according to the invention, the hardware implementation of the known ray tracing pipeline is expanded by a hardware-implemented transformation unit which transforms the beams into the objects. This unit is only available once for optimal utilization of the hardware resources and is used in addition to the object space transformation also for calculating the point of intersection of the beam with a triangle, for creating primary beams and for creating secondary beams.

Through the use of special processors designed for ray tracing, the invention allows the user to fully program the system by using a novel processor architecture consisting of the combination of a standard processor core with one or more special ray tracing commands. The use of these ray tracing processors allows the programming of various ray tracing processes. Primitive objects in the scene can be designed to be programmable, so that, in contrast to today's graphics cards, the use of spline surfaces is also possible by using a special one Intersection calculation algorithm of a beam with the spline surface is programmed. As is common in today's rasterization hardware, a wide variety of shading models can be programmed for surfaces.

To output the image data on a display, the invention can be combined by using common frame buffers and Z buffers with rasterization hardware corresponding to the prior art.

State of the art

The prior art regarding the representation of three-dimensional scenes can currently be divided into two main sectors, the rasterization method and the

Ray tracing method (see Computer Graphics / Addison-Wesley ISBN 0201848406).

The known rasterization method, which is mainly used in computer graphics cards, is based on the principle of projecting every geometry of the scene onto a frame buffer and Z buffer. For this purpose, the color and brightness values of the pixels are saved in the frame buffer and the geometric depth values in the Z buffer, but only if the previous geometric value in the Z buffer is larger (further away from the viewer) than the new one. This ensures that nearer objects overwrite further ones and that only the really visible objects are shown in the frame buffer after the procedure has been completed.

However, this method has the decisive disadvantage that complex scenes with millions of objects cannot be displayed in real time with the hardware known to date, since it is generally necessary to project all triangles (objects) of the scene. In addition, a frame buffer and Z buffer are required, on which many billions of write operations have to be carried out per second, with most pixels being overwritten several times per frame for image construction. The overwriting of the pixel further away from the viewer by pixels of closer objects has the consequence that already calculated data is discarded, as a result of which optimal system performance cannot be realized.

Shadows can be calculated using today's rasterization hardware using complex techniques, but problems arise regarding complex scenes Accuracy. Reflections on curved surfaces as well as the calculation of light refractions cannot be realized physically correctly with this technology.

A second process, the ray tracing process, is an improvement in performance, which is known for its photorealistic images, but also for its computational complexity. The basic idea of ray tracing is closely related to physical light distribution models (see Computer Graphics / Addison-Wesley ISBN 0201848406).

In a real environment, light is emitted from light sources and distributed in the scene according to physical laws. The image of the surroundings can be captured with a camera. Ray tracing goes the opposite way and traces the light from the camera, which represents the viewer's position, back to its source. For this purpose, a virtual beam is fired in the direction illuminating the pixel for each pixel of the image. This shooting of the beam is called ray casting. If the beam hits an object, the color of the pixel is calculated from the color of the object hit, the surface normal and the light sources visible from the point of impact. The visible light sources are determined by tracking the secondary rays that are fired from each light source to the point of impact. If these shadow rays hit an object between the light source and the point of impact, the point lies in the shadow with respect to the light source.

In addition to the shadow calculation described, this method also enables the calculation of reflections and the refractions of the light by calculating reflection rays or refracted secondary rays. Furthermore, scenes of almost any size can be handled and displayed. The reason for this lies in the use of an acceleration structure. This is a special procedure with a corresponding data structure, which makes it possible to "shoot" or traverse the virtual beam quickly through the scene. On the way, some objects are selected that are possible candidate candidates, so that the point of impact is found quickly. Theoretical Studies have shown that the complexity of the ray tracing process grows logarithmically with the size of the scene, which means that squaring the number of objects in the scene means only twice the computing effort.

Typical acceleration structures include the regular grid, the kD tree, the octree and the bounding volume hierarchy (see Computer Graphics / Addison-Wesley ISBN 0201848406). All of these processes are based on the idea that Divide space into many subspaces and save the geometry there for each such subspace. The traversing process then traces the beam from subspace to subspace and always intersects it with exactly the objects that are in the subspace. The four methods differ only in the arrangement of the subspaces. In the regular grid, the room is divided into cubic parts of the same size. In this connection, reference is made to the illustration in FIG. 7 for illustration purposes. The three other methods are based on a recursive division of the space. With the kD tree method, the starting space is recursively divided at any point along the axis. For this purpose, reference is made to FIG. 8 for illustration. This division of space is stored in a recursive data structure (a binary tree). The third method, called Octree, is also recursive, except that the subspaces under consideration are always divided into 8 rectangular subspaces of the same size. Reference is made to FIG. 9 for this purpose. The bounding volume hierarchy divides the space into n arbitrary volumes, which may even overlap, which is not permitted with the other methods.

In contrast to the rasterization process, there is currently no pure hardware solution that implements the ray tracing process, but only software-based systems that require a comparatively large amount of computing power and computing time. To illustrate the extent of the calculations over time, it should be noted that depending on the complexity of the image and the software used with the PC hardware currently corresponding to the state of the art, a computing time of a few seconds to several hours is required in order to produce a single still image using this method to create. The calculation of moving images requires a lot of computing time and / or the availability of special mainframes.

The Chair of Computer Graphics at Saarland University has developed a software-based real-time ray tracing system that is used on a cluster of over 20 computers.

US Pat. No. 6,597,359 B1 describes a hardware solution for the ray tracing method, which is, however, limited to static scenes.

US Pat. No. 5,933,146 also describes a hardware solution for the ray tracing method, also restricted to static scenes.

The paper "SaarCOR - A Hardware Architecture for Ray-Tracing" from the Chair of Computer Graphics at the University of Saarland describes a hardware architecture for ray-tracing but again limited to static scenes. The paper "A Simple and Practical Method for Interactive Ray-Tracing of Dynamic Scenes" from the Chair of Computer Graphics at Saarland University describes a software approach to support dynamic scenes in a ray tracer. However, the software method described uses only one level of objects, cannot carry out nesting in several stages.

The described prior art currently offers neither software nor hardware solutions with which complex dynamic scenes can be displayed in real time. In the known rasterization methods, the performance limitation lies in the number of objects to be displayed.

Ray tracing systems can represent many triangles, but due to the pre-calculations required, they are limited in that the position can only be changed to a limited extent. Scenes from a few billion triangles require a lot of computing power and memory and can only be handled on fast and complex mainframes or cluster solutions.

Therefore, software-based dynamic real-time ray tracing systems cannot be implemented with the available personal computer hardware. For reasons of cost, the cluster solution described should remain limited to special applications.

In contrast, the present invention is based on the object of proposing a device with which ray tracing methods can be carried out more quickly — preferably also in real time — in dynamic complex three-dimensional scenes in such a way that a photorealistic representation results.

This object is achieved by a device according to claim 1, in that this device has at least one programmable ray tracing processor, in which the following are implemented: • special traversing commands and / or • vector arithmetic commands and / or • commands for creating ray tracing acceleration structures and / or • at least one decision unit (mailbox) with which it is suppressed that when the ray tracing method is executed when shooting a beam, objects or triangles already cut with the beam are cut several times with the beam.

The structure of the device is organized in such a way that several threads can be processed in parallel and several threads can be processed automatically and synchronously. Furthermore, the device has an n-level cache hierarchy and / or a virtual memory management and / or a direct connection to the main memory.

This device can preferably be implemented as FPGA and / or in ASIC technology and / or another logic-based semiconductor technology or in discrete integrated logic, or in the combination of these technologies.

For a more detailed discussion of the decision unit, reference is made to FIG. 2, from which it can be seen that the list unit is expanded by a mailbox. This mailbox prevents a triangle or object from being cut multiple times with the beam when a beam is shot by remembering objects or triangles already cut with the beam. As a result, not so many beam-object or beam-triangle intersection calculations need to be carried out, which speeds up the calculation. The mailbox can be seen as a kind of intersection calculation cache, which, in contrast to a storage cache, does not prevent storage requests for storage, but rather intersection calculations. Standard caching methods such as 4-way caches can be used to implement the mailbox.

Claim 2 relates to a device for the photorealistic representation of dynamic complex three-dimensional scenes by means of the ray tracing method, which has at least one special traversing unit and at least one list unit and at least one decision unit (mailbox) with which is suppressed that at Execution of the ray tracing method when shooting a beam, objects or triangles already cut with the beam are cut several times with the beam, and at least one intersection calculation unit and at least one unit for creating acceleration structures and at least one transformation unit and / or at least a unit for solving linear systems of equations and that several beams or threads can be processed in parallel and several beams or threads can be processed automatically synchronously and any number of levels of dynamic objects in dynamic objects n can be realized and that the device has an n-level cache hierarchy and / or a virtual memory management and / or a direct connection to the main memory.

In the embodiment according to claim 3, the difference from the embodiment according to claim 2 is that the at least one transformation unit and / or the at least one unit for solving linear systems of equations and / or the at least one intersection calculation unit in claim 2 Claim 3 was supplemented by a ray tracing processor, which was already explained in claim 1.

This device, based on the ray tracing method, for the photorealistic representation of three-dimensional moving scenes, in which the software-defined acceleration structures and methods are implemented in corresponding hardware structures, is primarily intended for real-time use.

In order to realize any disordered dynamics in a scene, the acceleration structure has to be recalculated for each image in the sequence. In the case of large scenes, this means an enormous amount of computing effort since the entire geometry of the scene has to be "touched". The advantage of the logarithmic complexity in the scene size disappears.

A solution to this problem described in the paper "A Simple and Practical Method for Interactive Ray Tracing" is to allow the scene to be divided into objects and only to move these objects as a whole. Two acceleration structures are required.

A top-level acceleration structure over the objects of the scene and a bottom-level acceleration structure for each of the objects. The objects are positioned in the scene in the form of instances of objects.

The difference between an object and the instance is that an instance of an object consists of an object and a transformation. The transformation is an affine function that moves the object anywhere in the scene. Affine transformations also allow objects to be scaled, rotated and sheared. In the following, for the sake of simplicity, the term object is also used for instances of objects if there is no possibility of confusion.

A beam is first traversed through the top-level acceleration structure (track beam through the scene) until a possible hit object (the object hit by the beam) is found. Now transform the beam into the local coordinate system of the object and continue traversing in the bottom-level acceleration structure of the object until a meeting point with a primitive object is found. Primitive objects are objects that have no other structure in themselves. For ray tracers, these are usually triangles and spheres. This method works very well in practice, but only as long as the number of objects does not become too large, since the top-level acceleration structure has to be rebuilt in every picture. It is necessary to rebuild this acceleration structure if the objects have been moved in it.

The invention now represents a hardware solution that supports the above division of the scene into objects recursively. This means that it is not restricted to objects that consist of primitive objects, but also allows these objects to be composed again of objects, which in turn can consist of objects, etc. Fig. 1 shows how a tree consists of several levels of objects can be created. First, a leaf is modeled as a level 1 object. This leaf is now instantiated several times and placed on a branch, which creates another object, but now an object of level 2. These small branches can now be instantiated several times again to form a larger branch or tree as an object of level 3 etc. It is It should be noted that there are several levels of objects in objects and that the representation of the scene is small due to the repeated use of the same geometries.

The method for ray casting used in the invention according to claim 2 looks as follows:

The beam is traversed through the acceleration structure of the top level until a possible hit object is found. If the object is a primitive object, the intersection of the beam with the object is calculated. If the object is not a primitive object, the beam is transformed into the local coordinate system of the object and continues traversing there recursively.

An essential part of the process is the transformation of the beam into the local coordinate system of the object, which in principle reverses the positioning of the object through the affine transformation. This means that the transformed ray no longer sees the object transformed. This transformation step requires a fairly complex affine transformation of the beam starting point and the beam direction, although the complex hardware unit required for this can also be used for further tasks. It turns out that the transformation unit can also be used to calculate the intersection with many types of primitive objects, to calculate primary rays and to calculate many types of secondary rays.

A similar camera transformation matrix is used to calculate the primary beams as in the known rasterization methods. First of all Pre-primary rays of the shape R = ((0,0,0), (x, y, 1)), i.e. rays with the starting point (0,0,0) and the direction (x, y, 1), where x and y represent the coordinates of the pixel to which a primary beam is to be calculated. There is an affine transformation for each camera position and orientation, which transforms the beam R in such a way that it corresponds exactly to the direction of incidence of the pixel (x, y) of the camera.

In order to calculate the intersection with a primitive object, the beam is transformed into a space in which the primitive object is normalized. In the case of a triangle as a primitive object, the beam is transformed into a space, for example, such that the triangle has the shape Δ _norm = ((1, 0.0), (0.0.0), (0.1, 0) ) Has. For illustration, reference is made to FIG. 10. This transformation can be done through an affine transformation. In contrast to the general case, the subsequent intersection calculation with the standard triangle is very easy to solve in hardware. If the transformation is chosen in such a way that the triangle normal is transformed to the vector (0,0,1) in the triangle space, the scalar product of the ray and triangle normal can be calculated very easily in the triangle space, since the scalar product from the beam direction (xt.yt.zt ) and the triangle normal (0,0,1) is even 0 * xt + 0 * yt + 1 * zt = zt.

The transformation can also be selected so that only 9 floating point numbers are required for its representation by mapping the triangle normal to a suitable normal in the norm triangle space. However, this prevents the possibility of calculating the dot product in the standard triangular space.

It is clearly evident that this standard object transformation can also be used for other types of objects, such as spheres, planes, cuboids, cylinders and many other geometric structures, only one different intersection calculation unit has to be created in each case.

A great advantage here is the fact that every type of primitive object has the same representation in memory, namely an affine transformation that transforms into the object's normal space. This simplifies the design of the memory interface of a hardware solution. The transformation into the object norm space is called the norm space transformation.

Shadow rays and reflections can be efficiently calculated by the transformation unit by calculating suitable transformations and suitable rays. It is also possible to use the transformation unit to transform normals (vectors that are perpendicular to a surface). This normal transformation is necessary because some shading models require the normal of the geometry at the point of impact. However, this normal must exist in the world coordinate system, which is not necessarily the case with the above method. Rather, for the first time, the normal is only available in the local coordinate system of the object hit. From there it must be transformed back into the world coordinate system.

However, the transformation unit also has a disadvantage. Since the affine transformations, which can be saved as matrices, have to be pre-calculated for triangles as well as for the objects of the scene, it is not easily possible to change the position of the triangle corner points efficiently from frame to frame. However, this is possible in vertex shaders on today's graphics cards. Vertex shaders are programmable special units that are optimized to calculate movements of points in space.

To make this possible, you have to move away from the precalculation of the data. As a result, it is then necessary to solve a system of linear equations with three unknowns to calculate the intersection with a triangle. This explicit solving requires more floating point operations, but is required in conjunction with vertex shaders. As a result, it may make sense to replace the above transformation unit with a unit that solves a linear system of equations. Among other things, this unit can be used to cut with triangles or to transform rays into the local coordinate system of an object.

The method for ray casting used in the invention according to claim 1 is structured analogously to the method described according to claim 2, with the difference that the transformation and the intersection calculation are implemented by suitable instructions for the ray tracing processor. Here, in particular, there is the possibility of using alternative methods for object-beam intersection calculation. For example, the use of the Plücker ray triangle test enables the efficient use of vertex shaders.

A problem with detailed scenes is unwanted aliasing effects, which arise especially when the object density in one direction is very high. Then it can happen that a beam hits a black triangle, for example, and suddenly a white triangle is hit with a minimal movement of the camera. Such Effects lead to temporal and local noise in the picture. The reason is that ray tracing generally uses infinitely narrow rays and does not take into account that the light that affects a pixel spreads out in a pyramid shape and the beam expands with distance. All objects that are in this pyramid of rays should actually be used to calculate the pixel color, which is not possible in a real-time system. This is remedied by a new, simplified form of cone tracing. Instead of looking at an arbitrarily narrow beam, the opening angle of the beam is also assessed. Depending on the distance to the camera, the corresponding beam width can be calculated. If you encounter a sub-area during traversal that is largely covered by the beam, it may not make sense to continue traversing. At this point, a simplified geometry of the volume interior can advantageously be used for the calculation. It can then be ignored that there are perhaps a million triangles in the volume. These triangles may only form the wall of a mountain range, which can also be approximated by a colored plane due to the size of the beam.

However, if the triangles model a hole-like structure such as the Eiffel Tower, the approximation should rather be the color of the constructive grid elements and a transparency value.

Such simplified geometry presentations can advantageously be supported in the acceleration structure. Figure 6 shows the concept using the example of an octree. The simplified geometry is shown for the volume with a bold outline, to which a node belongs in the acceleration structure. The beam overlaps almost the entire volume of the node, so the simplified geometry is used to calculate the intersection.

An alternative method is to save the objects of the scene with different levels of detail. This means that the objects are modeled with different resolutions or numbers of triangles and that depending on the distance of the objects from the camera, detailed or simplified objects are still used.

A disadvantage of the hard-wired, hardware-based ray tracing pipeline described above could be that it is complex to make it programmable. Compared to a software ray tracing approach, the hardware-based pipeline looks very rigid and special. This is remedied by the development of a CPU specially adapted to the ray tracing process. This special ray tracing processor consists of a standard CPU, such as a RISC processor, whose instruction set is expanded to include special instructions. In particular, a traversal command that traverses the beam through an acceleration structure is important. Some parts of the method also require complex arithmetic operations, which mainly take place in three-dimensional space. It therefore makes sense to equip the CPU with a vector arithmetic unit, similar to the SSE2 instruction sets currently used.

A further optimization of the CPU can be achieved by taking advantage of the fact that the method can be parallelized. It is therefore possible to run several threads (program sequences) on one CPU very effectively, which significantly increases the utilization and effectiveness of the CPU. This is especially true with regard to storage latency. If one thread makes a memory request, another thread can be executed during the request. An exemplary structure of such a CPU can be seen in FIG. 5.

Since dynamic scenes have to be recalculated for acceleration structures for every frame, it is necessary to expand the instruction set of the CPU with special instructions for creating acceleration structures. When creating acceleration structures, it often has to be decided whether an object is in a given subspace into which it should be sorted or not. A special unit that optimizes the creation of acceleration structures can accelerate the required calculation by making a very easy-to-calculate preliminary decision. Boxes are often used as subspaces, the 6 boundary surfaces of which are perpendicular to the x, y or z axis. Such a box can be characterized by its corner points, whereby the definition of just two points is even sufficient. The determination / decision whether the triangle is in this box can in many cases be made by simply comparing the coordinates of the points. For example, if the triangle lies in the X direction far to the left of the box, the X coordinates of the 3 corner points of the triangle are all smaller than the smallest X coordinate of the corner points of the box (see Fig. 13). Many other constellations can also be decided in this way, for example whether the triangle is completely in the box. If no decision is possible, complex mathematical formulas such as the SAP (Separating Axis Theorem) must be used. The decision whether a box overlaps with another box can also be decided using the corner point comparisons. For further optimization, a decision unit is used which prevents objects or triangles already cut with the beam from being cut several times with the beam when a beam is fired. This is done by adding a mailbox to the list unit as shown in Figure 2. This mailbox prevents a triangle or object from being cut multiple times with the beam when a beam is shot by remembering objects or triangles already cut with the beam. As a result, not so many beam-object or beam-triangle intersection calculations need to be carried out, which speeds up the calculation. The mailbox can be seen as a kind of intersection calculation cache, which, in contrast to a storage cache, does not prevent storage requests for storage, but intersection calculations. Standard caching methods such as 4-way caches can be used to implement the mailbox.

The units of the ray tracing architecture require a very high memory bandwidth, which means that a lot of data has to be transmitted per unit of time. Usually this can only be achieved by connecting a large number of memory chips in parallel. However, the required memory bandwidth can also be ensured by a suitable interconnection of several cache levels (n-level caches). What is essential here is a property of the ray tracing method, which is referred to as coherence. Coherence refers to the fact that rays that pass through similar areas of 3D space also access almost the same data in the acceleration structure and accordingly also the same objects. If this property is used, high cache hit rates can be achieved. This means that the required data will most likely be found in the cache and does not need to be loaded from the main memory in a time-consuming manner. As shown in FIG. 4, the caches themselves are arranged in a binary tree, for example, in order to supply several ray tracing units.

The device according to the invention can of course also be used in conjunction with a 3D display for the photorealistic three-dimensional real-time representation of complex moving scenes. Depending on the technology of the display used, three types of image output can be distinguished.

Firstly, an embodiment in which two images containing the stereo impression are alternately displayed horizontally offset on the display in time-division multiplexing. Secondly, an embodiment in which two horizontally offset images representing the stereo impression are shown on a display in alternating vertical stripes containing the image information of the two images.

Thirdly, an embodiment in which the two horizontally offset images are shown on two separate displays simultaneously or in time-division multiplexing.

The two horizontally offset images, each of which can be assigned to the right or left eye, can only be seen by one eye through appropriate spatial display arrangements or through the use of image separation devices (e.g. shutter glasses, strip-shaped Fresnel prisms / lenses, polarization filters). The 3D displays to be used and their requirements for video signal control correspond to the prior art and are not described in detail. Further explanations regarding the state of the art of 3D displays can be found in the following typefaces: Computer Graphics / Addison-Wesley ISBN 0201848406, DE 4331715, DE 4417664, DE 19753040, DE 19827590, DE 19737449

The use of computer-animated, photo-realistic, real-time display of three-dimensional moving scenes and images extends from the display of three-dimensional CAD data, the display of medical and technical-analytical data, film animation and use in flight and driving simulators to the so-called home applications in computer games with complex real-time graphics.

In addition, the same methods can also be used for non-photorealistic image generation (e.g. line drawings or comic style representations) without further changes to the functional design. Calculations can also be carried out without further technical changes, which are usually not directly associated with image calculation. For example, this includes collision detection between geometric objects (English collision detection) and the discrete solving of numerical problems. All of the applications described are not limited to the interactive area and can also be used in the offline area without changes to the method or device, for example in the calculation of cinema films or very complex physical simulations.

The functional implementation and the hardware implementation of the ray tracing processes takes place in complex and fast logic technologies, their implementation both as hardwired digital logic in the form of discrete digital logic consisting of individual standard ICs, or customer or application-specific ICs, for example ASIC, or of complex programmable logic modules / Logic circuits, for example CPLD or FPGA technologies with or without CPU core, or the combination of these technologies.

The exemplary embodiment of the invention described below describes the ray tracing unit of a computer graphics card, in which the hardware implementation of the ray tracing method can take place, for example, in a freely programmable logic module FPGA, in ASIC technology, or in a hard-wired special chip.

Insofar as methods and functional sequences are described, they can be implemented purely in terms of hardware. This means that appropriate logic units and hardware-implemented arithmetic units must be designed.

The standard function of the control electronics for controlling the data display (cathode ray tube, TFT, LCD or plasma monitor) and their timing correspond to the prior art, are assumed to be known and are not the subject of the description. An interface between the image memory of this standard function and the implementation of the ray tracing method according to the invention is described.

The description is divided into two parts. First, the ray casting pipeline is abbreviated to RCP. This is the core of the design, which traverses rays through the scene and returns the point of impact.

The second part describes an optimized ray tracing architecture for the ray casting pipeline, in which several of these ray casting pipelines work together.

FIG. 2 shows the ray casting pipeline unit (RCP), which consists of several subunits. The traversing unit traverses the beam through a suitable acceleration structure, preferably a k-D tree. The beam is traversed until it reaches an area of the scene in which there are possible hit objects. The objects in this area are stored in a list that is processed by the list unit. The list unit also includes a mailbox that prevents a triangle or object from being cut several times with the beam when a beam is shot, by remembering objects or triangles already cut with the beam. As a result, not so many beam-object or beam-triangle intersection calculations need to be carried out, which speeds up the calculation.

These objects must now be examined for a possible intersection. If there is no valid intersection, the traversing must be continued. The List unit sends the possible hit objects, which have not already been processed, one by one to the matrix loading unit, which loads the affine transformation associated with the object. This affine transformation can be represented by a 4x3 matrix. These can be object space transformation matrices or matrices that transform into the normal space of a primitive object. After the matrix loading unit has stored the matrix in the transformation unit, the beams are sent from the beam unit through the transformation unit.

After the transformation, two scenarios are now possible. On the one hand, it can be an object that contains other objects. If this is the case, the rays migrate back into the traversing unit and the transformed ray is traversed further in the object. However, if it is a primitive object, the beam goes directly to the intersection calculation unit, which cuts the steel with the standard object. The intersection calculation unit can support several standard objects (triangles, spheres, etc.) as described above.

The calculated intersection data is collected in the intersection unit. The intersection unit supplies a return channel to the traversing unit, so that it can recognize whether there are already valid intersection data.

The traversing unit, list unit and matrix loading unit are the only units of the ray casting pipeline that access external memory. The traversing unit accesses the acceleration structure, the list unit on lists of object addresses and the matrix loading unit on affine transformations in the form of matrices. All three units are connected to the main memory via their own cache in order to ensure the necessary memory bandwidth.

A simplified version of the ray casting pipeline is shown in FIG. 12, only the most important units being shown: the traversing unit, which traverses the beams through the acceleration structure, the list unit, which processes the lists, the transformation unit which applies the loaded transformation to the rays and the intersection calculation unit which intersects the transformed ray with the norm object.

As shown in FIG. 3, the ray casting pipeline is embedded in a suitable ray tracing architecture. The figure shows 4 ray casting pipeline units, each with 3 caches. There are 2 units each connected to a shading unit. These shading units use the ray casting pipeline units to calculate the colors of the pixels of the image. The shading unit shoots for this Primary rays, processes the impact information that the ray casting pipeline returns, and shoots secondary rays, for example, into light sources. The shading units can be hard-wired hardware or the programmable ray tracing processors described later.

The shading units have a channel to the transformation unit of the ray casting pipeline. This is used to load camera matrices and matrices for secondary beams, thereby minimizing the computing effort for the shading unit. It is advantageous if the shading units each have a separate texture cache and shading cache. The shading cache contains shading information about the geometry of the scene, such as colors and material data. The texture cache is connected to the texture memory and gives the shading units access to textures. It is advantageous if each shading unit has its own local frame buffer, on which it stores the colors and brightness values of the pixels just processed / calculated. Furthermore, it is advantageous if a Z-buffer is available, since it is required for the connection to the standard rasterization hardware described below.

Once the color of a pixel has been fully calculated by the shading unit, this color can be written to the global frame buffer using the optional tone mapping unit. The tone mapping unit applies a simple function to the color to map it in the 24 bit RGB space. The geometric depth values (Z values), which are stored in the optional local Z buffer, can now also be transferred to the optional global Z buffer.

However, the color or the new Z value are only written to the frame buffer or Z buffer if the Z value previously present in the Z buffer is larger. This ensures that pixels are only written if they are geometrically in front of values already calculated by the rasterization hardware or other ray tracing passes. It is thus possible to combine the ray tracing hardware with a standard rasterization hardware that works on the same frame buffer or Z-buffer, which also represents the interface to this standard rasterization hardware.

To further increase the system performance, additional shading units can optionally be connected in parallel with the associated ray casting pipelines and caches. The performance-enhancing effect lies in the broadening of the data and processing structure. It is not to be expected that the entire scene will fit into the local main memory of a ray tracing chip. To solve this problem, local memory can be used as a large cache that caches larger blocks of the scene. The actual scene is located elsewhere and is reloaded via DMA if necessary. This procedure for virtual memory treatment makes it possible to visualize very large scenes that do not fit into the main memory of the ray tracing chip.

Photon mapping is a standard process in which virtual photons are shot into the scene from the light sources and accumulated on the surfaces of the scene. The light distribution of the scene can thus be simulated. This applies especially to caustics. If photons are shot, an acceleration structure is built up over the photons, which can be a k-d tree, for example. This calculated photon light distribution can now be reproduced by visualizing the scene using standard ray tracing methods and by including the light intensity arriving at each point of impact in the color calculation in such a way that the energy of all the photons hitting in the vicinity of this point is added up. For this purpose, all neighboring photons must be searched for in the acceleration structure of the photons. This task can be supported by the traversing unit by traversing a volume rather than along a beam. In this way, all neighboring photons can be processed, for example by summing up their energy.

As a rule, ray tracing methods work with infinitely narrow rays, which leads to sampling artifacts (aliasing). A much better approximation results from using beam cones instead of beams. The light that illuminates a pixel of the camera comes not only from a discrete direction, but from a kind of pyramid that can be approximated very well by a cone. Traversing this room volume from front to back, described by a beam cone or a beam pyramid, can also be carried out by the traversing unit as a special function. Figure 6 shows a two-dimensional image of a beam cone.

The signal processing for controlling the display and the timing generation for the display or the monitor can be carried out in a known manner by means of the optional rasterization hardware or can be implemented by suitable functional units if advantageous for the desired application. So the basic functions of the standard rasterization hardware with the hardware-implemented ray tracing Procedures and functions can be combined to a very powerful real-time hardware architecture.

A second, further exemplary embodiment of the invention is based on the design and application of freely programmable ray tracing CPUs or ray tracing processors, which program-controlled execute the special ray tracing functions and methods described according to the invention. Due to the corresponding logic and function parallelism, only a few, preferably one or two CPU clock cycles are required to process the individual function.

Insofar as internal algorithms, procedures and functional sequences of the ray tracing processors are described, they must be created using a hardware description language, for example HDL, VHDL or JHDL and transferred to the corresponding hardware. The hardware implementation of the ray tracing processors with the implemented methods and functions can take place, for example, in a freely programmable logic module FPGA, in ASIC technology, in the combination of digital signal processors with FPGA / ASIC or in a hard-wired special chip.

This means that appropriate logic units and hardware-implemented arithmetic units are to be designed, the individual functions of which can be called up under program control. Depending on the complexity of the hardware technology used, one or more ray-tracing processors can be implemented per chip, with or without additional logic functions.

Ray tracing processors are fully programmable computing units that are designed to execute vector arithmetic instructions and special ray tracing instructions such as "traversing" and "creating acceleration structures". This can be done with the addition of functional units or with existing functional units with the addition of possibly a few logic modules. For example, traversing can be carried out using special functional units or by expanding the existing arithmetic functional units by a few logical functional units.

As can be seen in FIG. 11, several of the ray tracing processors shown in FIG. 5 are connected in parallel. The memory interface is formed by a cache hierarchy, which provides the necessary memory bandwidth. This approach is efficient because of the strong coherence of neighboring rays. With the ray tracing processor concept, each ray tracing processor processes either exactly one pixel of the Image or a package of several pixels. The calculation of each individual pixel corresponds to a calculation thread. Several of these threads are bundled into a package and processed as a whole, synchronized. Threads processed synchronously are characterized by the fact that they always execute the same instruction together. This enables the creation of efficient hardware and the execution of memory requests per whole package and not individually for each thread. Above all, this reduction in memory requests is a major advantage of processing packets of threads.

The thread generator of FIG. 11 creates threads that are executed on the ray tracing processors. The thread generator can also be programmed by the ray tracing processors. Special functions for scanning the pixels of the image in a cache-coherent manner (for example Hubert curve) allow the ray tracing processors to be optimally supplied with coherent threads. This reduces the memory bandwidth required. The thread generator also has access to the main memory via a DMA interface. The start values for the individual threads can also be created in a previous processing step and written to the memory in order to later generate new threads from this data.

A pixel is processed by a software program that runs on the ray tracing processor. This software program describes a recursive procedure for calculating the color value of a pixel, even if the hardware works on packets of rays. Package management is therefore transparent to the programming model.

FIG. 5 shows an exemplary structure of a ray tracing processor. You can see a standard processor core (RISC core) with two special coprocessors connected in parallel. The coprocessors each have their own registers. However, it is also possible to transfer these register contents from one coprocessor to another using special instructions. The traversing core is a special coprocessor that can efficiently traverse beams through an acceleration structure. For this he needs a special memory interface to the nodes of the acceleration structure (node cache). The vector arithmetic core is a special coprocessor that can efficiently perform operations in 3D space. The vector additions, scalar multiplications, cross products and vector products required by each ray tracing software can be calculated quickly using this unit. The vector arithmetic unit needs access to a special cache which enables whole vectors to be loaded in one cycle. The semantics of the traversal command could look like this. The ray tracing CPU describes special registers with the origin and the direction of the beam and the address of the acceleration structure that is to be traversed. Now a special command is called that starts a special traversing unit. This unit traverses the acceleration structure and sends all nodes that contain objects to a list unit, which can contain a mailboxing mechanism, which was also described in the first exemplary embodiment of the invention. A small program is now executed on the CPU for each object, which cuts the beam with this object. If a valid point of intersection of the beam with an object is found, the program returns to the program that shot the beam.

The individual instructions or logic functions implemented in hardware in the ray tracing processor contain the same algorithms that have already been described in the hard-wired first embodiment of the invention. In addition to this, this command set can be supplemented by further hard-wired commands and functions, which in turn are to be activated under program control. Special traversing operations, vector arithmetic, and the parallel processing of multiple threads on a ray tracing processor provide the computing power required for real-time use and at the same time the effects of memory latencies (waiting times for memory requests) on system speed are minimized or even irrelevant.

The fact that the - RTPU can be used in a program-controlled manner to recursively shoot the secondary beam through the scene, considerably simplifies the programming model compared to the hard-wired hardware.

Once the color of the pixel has been fully calculated, it can be written into the frame buffer. If necessary, the distance can also be written to the Z buffer. This enables a connection to rasterization hardware and the corresponding representation on a display.

Claim 10 describes the use of an additional space-dividing data structure that does not store or reference geometric objects, but rather contains spatial influences or material-changing parameters. Such room influences can be, for example, fog, haze, dust particles or hot smoke, whereby hot smoke, for example, can also result in a change in the scene visible through the volume of the smoke. Other room influences are light sources or material-changing auras as they can be used in the representation of fantasy creatures, for example. Here, the use of the space-dividing data structure allows the computational effort to be reduced considerably, since only influences that are spatially located in such a way that they can have an influence need to be taken into account.

Claim 11 describes an extension that makes it possible to process three-dimensional scenes that are not or not exclusively made up of triangles and that, if necessary, convert other geometric primitives into triangles or an easy-to-process intermediate representation. The additional functionality is achieved in that the hardware is either expanded by functional units or existing units implement the functionality.

Claim 12 describes an extension that makes it possible to calculate not only the closest, but also additional object-beam intersection points per beam as a result of beam processing. It is advantageous here if the results are sorted according to the distance from the beam origin. The maximum number of intersections per result can be set as a constant or per beam depending on parameters of the cut objects. This technique can be used to calculate rays through transparent objects more efficiently.

Claim 13 describes an extension that can count with additional and / or the existing functional units, how often a certain element was used in the calculations of an image. The elements to be counted can be very different and the elements could be classified using the address in the memory or the ID of an element. These elements include:

dynamic and geometric objects, parts or complete material descriptions, elements or subgroups of a room description data structure, programs or program functions, individual memory cells and entire memory pages or areas.

Claim 14 describes an extension of the functionality for calculating the spatial description data structures to parts or complete three-dimensional scenes, additional parameters per dynamic object or dynamic partial object or geometric object having an influence on the calculation method of the spatial description data structure. Such influences can be, for example, that an object specifies that it does not have any geometrical dimensions in certain spatial volumes Owns objects. This allows the space description data structure to be calculated more efficiently and the calculation effort per image to be further reduced.

Description of the drawings

Figure 1 shows how a tree can be created from several levels of objects. First, a leaf is modeled as object 101 of stage 1. This leaf 101 is now instantiated several times and placed on a branch 102, which creates a further object, but now an object of level 2. These small branches 102 can now be instantiated several times again to a tree 103 as an object of level 3.

FIG. 2 shows the ray casting pipeline 200 with the most important data paths and the interface to the caches. Arrow 201 goes to the shading unit, arrow 202 comes from the shading unit. Furthermore mean:

203 traversing unit 204 list unit which contains a mailbox 205 matrix loading unit 206 beam unit 207 transformation unit 208 intersection calculation unit 209 intersection unit 210 node cache 211 list cache 212 matrix cache

Figure 3 shows the top level diagram of an exemplary implementation of the invention. The 4 ray casting pipelines (RCP) are connected via a cache hierarchy to the separate memories for nodes, lists and matrices. In this example design, two RCPs are connected to a shading unit, which has access to a local frame buffer and Z buffer. These are used to write color values to a global frame buffer using a tone mapping unit and to send depth values directly to a global Z buffer. State-of-the-art rasterization hardware (rasterization pipeline) can be connected to this Z-buffer and frame buffer.

Mean:

301 rasterization hardware

302 frame buffer

303 Video Out

304 tone mapping unit

305 Z buffer

306 frame buffer 1

307 Z-buffer 1 308 Texture cache 1 309 Shading cache 1 310 Shading unit 1 311 node cache 1 312 list cache 1 313 matrix cache 1 314 RCP 1 315 node cache 2 316 list cache 2

317 Matrix Cache 2

318 RCP 2

319 frame buffer 2

320 Z buffer 2

321 texture cache 2

322 Shading Cache 2

323 shading unit 2

324 node cache 3

325 list cache 3

326 Matrix Cache 3

327 RCP 3

328 node cache 4

329 Listen Cache 4

330 matrix cache 4

331 RCP 4

Figure 4 shows the cache infrastructure, which provides the necessary internal memory bandwidth in the chip. The figure shows a binary n-level cache, but other hierarchical structures are also conceivable.

Mean:

401 node cache 1

402 nodes cache 2

403 node cache 3

404 node cache 4

405 node cache 5

406 node cache 6

407 node cache 7

408 nodes of storage

409 listen cache 1

410 listen cache 2

411 list cache 3

412 list cache 4

413 list cache 5

414 list cache 6

415 list cache 7

416 lists of memory

417 Matrix cache 1

418 matrix cache 2

419 matrix cache 3

420 matrix cache 4

421 Matrix Cache 5

422 Matrix cache 6 423 Matrix Cache 7

424 matrix memory

425 Texture Cache 1

426 Texture Cache 2

427 texture memory

428 Shading Cache 1

429 Shading Cache 2

430 shading cache 3

431 shading memory

FIG. 5 shows the exemplary embodiment of a ray tracing CPU. The blocks have the following meaning:

501 load instruction 502 instruction memory 503 RISC core 504 cache 505 traversal core 506 node cache 507 vector arithmetic core 508 vector cache

Figure 6 shows an example of the simplified geometry in the octre nodes. Reference number 601 denotes the beam cone and reference number 602 a "simplified geometry".

Figure 7 shows an example of the regular grid acceleration structure on a simple scene. For the sake of simplicity, the room is only drawn in 2D. The beam is designated by the reference number 701.

Figure 8 shows an example of the k-D tree acceleration structure on a simple scene. For the sake of simplicity, the room is only drawn in 2D. The beam is designated by reference number 801.

Figure 9 shows an example of the octree acceleration structure on a simple scene. For the sake of simplicity, the room is only drawn in 2D. The beam is designated by the reference number 901.

FIG. 10 shows the triangular transformation from the global space into the standard triangular space. A ray 1001 can be seen and a triangle 1002. An affine transformation, which is represented by the arrow 1003, results in a transformed ray 1004 and a standard triangle 1005. The global space is represented by the left coordinate system and the standard triangle space by the right coordinate system. FIG. 11 shows an exemplary embodiment of the invention based on special ray tracing processors. The blocks have the following meaning:

1101: Rasterization hardware

1102: Thread generator

1103: Z buffer / frame buffer

1104: Video Out

1105: Ray tracing processor 1

1106: Ray tracing processor 2

1107: Ray tracing processor 3

1108: Ray tracing processor 4

1109: cache 1

1110: cache 2

1111: cache 3

1112: cache 4

1113: cache 5

1114: cache 6

1115: cache 7

1116: main memory

FIG. 12 shows a simplified version of the ray casting pipeline shown in FIG. 2. The blocks have the following meaning:

1201 traversal unit 1202 node cache 1203 list unit 1204 list cache 1205 transformation unit 1206 matrix cache 1207 intersection calculation unit

Arrow 1208 comes from the shading unit

FIG. 13 shows a simple case in which, by evaluating the corner points of a triangle and the corner points of a box, it can be decided whether the triangle overlaps the box or not. If the X coordinates of the triangle are smaller than the smallest X coordinate of the box, the triangle is outside the box. The triangle is identified by reference number 1301, the box by reference number 1302.

Claims

claims

1. Device for the photorealistic representation of dynamic complex three-dimensional scenes by means of the ray tracing method, characterized in that it has at least one programmable ray tracing processor in which are implemented: • special traversing commands and / or • vector arithmetic commands and / or • commands to create ray tracing acceleration structures and / or • at least one decision unit (mailbox) with which it is suppressed that when the ray tracing method is executed when shooting a beam, objects or triangles already cut with the beam are cut several times with the beam , and that the device is organized so that multiple threads can be processed in parallel and multiple threads can be processed automatically synchronously and that the device has an n-level cache hierarchy and / or a virtual memory management and / or direct connection to the main memory.

2. Device for the photorealistic representation of dynamic complex three-dimensional scenes by means of the ray tracing method, characterized in that it has at least one special traversing unit and at least one list unit and at least one decision unit (mailbox) with which it is suppressed that when executing the ray tracing method when shooting a beam, objects or triangles already cut with the beam are cut several times with the beam, and at least one intersection calculation unit and at least one unit for creating acceleration structures and at least one transformation unit and / or at least one unit for solving linear systems of equations and that several beams or threads can be processed in parallel and several beams or threads can be processed automatically synchronously and any number of levels of dynamic objects in dynamic objects n can be realized and that the device has an n-level cache hierarchy and / or a virtual memory management and / or a direct connection to the main memory.

3. Device according to claim 1, characterized in that it has at least one special traversing unit and at least one list unit and at least one decision unit (mailbox) with which it is suppressed that when executing the ray tracing method when shooting a beam, objects or triangles already cut with the beam are cut several times with the beam, and at least one intersection calculation unit and at least one unit for creating acceleration structures and at least one ray tracing processor.

4. Device according to claims 1, 2 or 3, characterized in that the at least one unit for creating acceleration structures is implemented by special hardware or by programmable units or ray-tracing processors and carries out functional methods for building the data structure of the acceleration structure and the decision, whether a triangle or a box overlaps another box by the at least one unit making a decision based on vertex comparisons of the corners of the triangle or box and the corners of the second box and - if no decision is possible - making a conservative decision or in In this case, a program is started on the programmable ray tracing processor that makes the exact decision, or another special hardware unit makes the exact decision, or the entire calculation takes place on the ray tracing processor.

5. Device according to claims 3 or 4, characterized in that the at least one transformation unit and / or the at least one logic unit for solving linear systems of equations functionally for primary beam generation and / or object space transformation and / or triangle standard space transformation and / or reflection beam calculation and / or transparency beam calculation and / or shadow beam calculation and / or normal transformation is used.

6. Device according to one of claims 1 to 5, characterized in that the at least one traversing unit or the traversing command can not only traverse along a beam, but also a volume, so that all objects can be processed in this volume.

7. Device according to one of claims 1 to 6, characterized in that the at least one traversing unit or the traversing command can traverse not only along a beam, but also along a beam cone or a beam pyramid, so that all objects that are in the beam cone or the beam pyramid, can be worked from front to back.

8. Device according to one of claims 1 to 7, characterized in that the function of the at least one traversing unit and the hardware Implementation of the traversal commands is based on the fact that a beam is traversed by an acceleration structure which is based on the kD tree method or the octree method or the regular grid method or the bounding volume hierarchy method, with simplification in each acceleration structure node Geometry data are saved, which are used as soon as the beam cone under consideration passes through a large part of the volume belonging to this node.

9. Device according to one of claims 1 to 6, characterized in that several ray tracing units work in parallel on several chips and / or several boards.

10. The device according to one of claims 1 to 9, characterized in that the described ray tracing hardware additionally uses a space-dividing data structure in which room influences and / or material-changing parameters are stored, which are evaluated with the available and / or additional functional units.

11. The device according to any one of claims 1 to 10, characterized in that the ray tracing hardware processes three-dimensional scenes that are not made up exclusively of triangles but also contain other geometric objects that are converted into other geometric objects and / or directly if necessary can be processed with additional and / or existing functional units and / or the programmable ray tracing processor.

12. The device according to one of claims 1 to 11, characterized in that the described ray tracing hardware processes three-dimensional scenes and for a beam no, one or more object-beam intersection points are sorted or unsorted according to the distance, the number of calculated object-beam intersection points can be defined as a constant and / or can be described by additional parameters of the objects.

13. Device according to one of claims 1 to 12, characterized in that the described ray tracing hardware with additional and / or the existing functional units can count how often a dynamic and / or geometric object and / or a material description and / or a Element and / or a subgroup of the space description data structure and / or a program and / or a memory cell and / or a memory page were used in the calculations of an image.

14. Device according to one of claims 1 to 13, characterized in that the described ray tracing hardware with additional and / or the existing functional units can compute spatial description data structures to parts or complete three-dimensional scenes, with additional parameters per dynamic object and / or dynamic sub-object and / or geometric object have an influence on the calculation method of the room description data structure.

15. Device according to one of claims 1 to 14, characterized in that the described ray tracing hardware is connected via a common Z buffer and frame buffer to a rasterization hardware which is on the same or a separate chip as the ray Tracing hardware is located.

16. The device according to one of claims 1 to 15, characterized in that a plurality of ray tracing units work in parallel and the data required for the calculation are distributed to the memories of these ray tracing units, and if necessary each of the unit which the required data stores is reloaded.