WO2024002130A1

WO2024002130A1 - Image rendering method and related device thereof

Info

Publication number: WO2024002130A1
Application number: PCT/CN2023/103064
Authority: WO
Inventors: 周鹏; 龚哲; 徐维超; 林澈
Original assignee: 华为技术有限公司
Priority date: 2022-06-29
Filing date: 2023-06-28
Publication date: 2024-01-04
Also published as: CN117351134A

Abstract

Disclosed in the present application are an image rendering method and a related device thereof, capable of achieving a high rendering effect on a target object regardless of whether the phenomenon generated by forming light rays emitted onto the target object is diffuse reflection, mirror reflection, or highlight reflection. The method of the present application comprises: on the basis of an original light ray path between a target object and a light source, resampling a light ray path between the target object and the light source to obtain a new light ray path, wherein the original light ray path is obtained by means of light ray tracing, the original light ray path uses the target object as a start point and the light source as an end point, and the new light ray path uses the light source as a start point and the target object as an end point; obtaining a sample pool on the basis of the new light ray path, wherein the sample pool is used for indicating light rays emitted to the target object, and the new light ray path is formed by these light rays; and rendering the target object on the basis of the sample pool to obtain an image of the target object.

Description

An image rendering method and related equipment

This application claims priority to the Chinese patent application filed with the China Patent Office on June 29, 2022, with the application number 202210752953.8 and the invention title "An image rendering method and related equipment", the entire content of which is incorporated herein by reference. Applying.

Technical field

The embodiments of the present application relate to the technical field of computer graphics, and in particular, to an image rendering method and related equipment.

Background technique

With the rapid development of the computer industry, users' requirements for images are increasing day by day. Currently, electronic devices usually use ray tracing technology to render three-dimensional scenes to obtain more realistic images of three-dimensional scenes for users to view and use, thereby improving user experience.

Currently, related technologies propose a ray tracing algorithm based on path resampling. In this algorithm, for an object to be rendered in a three-dimensional scene, after determining the original light path formed by the light that intersects the object for the first time, the light path can be resampled based on the original light path, thereby collecting A new ray path formed by a new ray that intersects the object for the first time. Then, a corresponding sample pool can be obtained based on the new ray path, and the sample pool is used to indicate the new rays forming the new ray path. An image of the object can then be rendered based on this pool of samples.

In this algorithm, limited by the material of the object, if the effect of light on an object is diffuse reflection, the light re-emitted for the object will often have a large reflection range after the light hits the object. Even if the light continues to hit other objects, it has a higher probability of reaching the light source, so it can be easily resampled to a valid new light path. Then, the rendering effect displayed on the object will be better. If the effect of light on the object is specular reflection or specular reflection, for the light re-emitted by the object, the reflection range after the light hits the object is often small. After the reflected light continues to hit other objects, it will only There is a small chance of reaching the light source, so it is difficult to resample an effective new light path. As a result, a high-quality sample pool cannot be obtained, so the rendering effect displayed on the object is often poor.

Contents of the invention

Embodiments of the present application provide an image rendering method and related equipment, which can render the image of the target object regardless of whether the phenomenon caused by the light hitting the target object is diffuse reflection, specular reflection or specular reflection. The rendering effect is high enough.

A first aspect of the embodiments of the present application provides an image rendering method, which method includes:

When you need to render a target object in a three-dimensional scene, you can first obtain the original light path between the target object and the light source. Since the original light path between the target object and the light source has the target object as the starting point and the light source as the end point, the two endpoints of the target object and the light source can be determined, and the properties of the target object and the light source can be exchanged to match the target object and the light source. The light path between the light sources is reconstructed, that is, the light path between the two is resampled to obtain a new light path between the target object and the light source. The new light path starts with the light source and ends with the target object.

Obtaining the new light path between the target object and the light source is equivalent to obtaining the light rays that form the new light path. Information related to these rays can then be obtained and used to construct a sample pool. For example, the intensity value of these rays before hitting the target object can be obtained, and the intensity value of these rays before hitting the target object can be stored in the sample pool. In this way, the sample cell can be used to indicate these rays hitting the target object.

After obtaining the sample pool, the sample pool can be used to render the target object, thereby obtaining an image of the target object. Then, the image of the target object is the displayed image, which can be displayed on the screen for the user to view and use.

It can be seen from the above method: after obtaining the original light path between the target object and the light source, the light path between the target object and the light source can be resampled based on the original light path to obtain a new light path. The original light path passes through the light path. It is traced that the original light path starts from the target object and ends with the light source, and the new light path starts from the light source and ends with the target object. A pool of samples can then be obtained based on the new ray path, indicating the rays hitting the target object from which the new ray path is formed. Finally, the target object can be rendered based on the sample pool to obtain the image of the target object. Based on the foregoing process, it can be seen that the embodiment of the present application provides a new light path resampling method, which is contrary to the direction of the effective light path in ray tracing (with the light source as the starting point and the target object as the end point), and is targeted Ground resampling of light paths can improve the success rate of sampling to effective new light paths, that is, it can easily resample to effective new light paths between the target object and the light source. In this way, no matter the light that forms the new light path Regardless of whether the phenomenon produced by the target object is direct illumination, diffuse reflection, specular reflection or specular reflection, a high-quality and sufficient sample pool can be successfully obtained to target the target object. The rendering effect of the target object is high enough.

In a possible implementation, the new light path includes a first light path and a second light path. The first light path starts with the light source, ends with the target object, and passes through other objects. The second light path starts with the light source. The starting point is the target object as the end point, and does not pass through other objects; the sample pool includes the first sample pool and the second sample pool. The first sample pool is used to indicate the first light ray directed to the target object. The first light path is A first ray is formed, a second sample cell is used to direct a second ray directed to the target object, and a second ray path is formed by the second ray. In the aforementioned implementation method, after obtaining the original light path between the target object and the light source, based on the original light paths of the two, the light path between the target object and the light source can be reconstructed in some way, that is, the light path between the two can be reconstructed. Perform the first resampling of the light path between the target object and the light source to obtain the first light path between the target object and the light source (which can also be called the indirect light path). The first light path starts from the light source and ends with the target object, and in the middle Passing through other objects (that is, the first light path has the light source and the target object as two endpoints, and passes through at least one intermediate node in the middle. These intermediate nodes are the remaining objects in the space where the target object is located, except the target object), The light path between the two can also be reconstructed in another way based on the original light path of the two, that is, the light path between the two can be first resampled to obtain the relationship between the target object and the light source. The second light path (can also be called the direct light path), the second light path starts with the light source and ends with the target object, and does not pass through other objects in the middle (that is, the second light path starts with the light source and the target object as two endpoints, and does not pass through any intermediate nodes, that is, it does not pass through other objects in the space where the target object is located, except the target object). Obtaining the first ray path and the second ray path between the target object and the light source is equivalent to obtaining the first ray between the target object and the light source (which can also be called indirect rays. These rays start from the light source and hit After other objects, it shoots indirectly to the target object) and the second ray between the target object and the light source (also called direct rays, these rays start from the light source and shoot directly to the target object). Then, information related to the first ray can be obtained, and the information can be used to construct the first sample pool. For example, the light intensity value of the first ray after hitting other objects and before hitting the target object can be obtained, and the light intensity value of the first ray after hitting other objects and before hitting the target object can be stored in the first in this pool. In this way, the first sample cell can be used to indicate the first light ray directed to the target object. Information related to the second ray can also be obtained and used to construct a second sample pool. For example, the light intensity value of the second light ray before hitting the target object may be obtained, and the light intensity value of the second light ray before hitting the target object may be stored in the second sample pool. In this way, the second sample cell can be used to indicate the second light ray directed to the target object. Based on this, the target object can be rendered based on the first sample pool and the second sample pool together, thereby obtaining an image of the target object. It can be seen that the foregoing implementation provides a ray tracing technology based on multiple sample pools, where the multiple sample pools refer to at least one first sample pool and at least one second sample pool. In a common configuration, two first sample pools and one second sample pool can be constructed. The rendering effects corresponding to the two first sample pools are specular reflection and specular reflection respectively. The rendering effects corresponding to the second sample pool The effect is direct lighting. Then, even if the target object is an object of special material, the image of the target object rendered based on multiple sample pools can present not only direct lighting effects, but also indirect lighting effects such as specular reflection effects and highlight reflection effects. , the resulting image is more realistic and beautiful, which is conducive to improving user experience.

In a possible implementation, the phenomenon caused by the first light ray hitting the target object includes at least one of the following: diffuse reflection, specular reflection or specular reflection. In the aforementioned implementation, there may be two situations: (1) If only a single round of the first resampling is performed, a set of first light paths between the target object and the light source can be obtained, and thus the set of A group of first rays in the first ray path, so the phenomenon produced by the group of first rays after hitting the target object can be controlled to be one of the following: diffuse reflection, specular reflection or specular reflection, correspondingly, used for indication The first sample pool of the group of first rays corresponds to a certain rendering effect, and the rendering effect can be one of the following: diffuse reflection, specular reflection, or specular reflection. For example, if the phenomenon produced by the first set of rays after hitting the target object is diffuse reflection, the rendering effect corresponding to the first sample pool will appear as diffuse reflection, and so on. (2) If only multiple rounds of first resampling are performed, multiple sets of first light paths between the target object and the light source can be obtained, and multiple sets of first light paths forming these multiple sets of first light paths can be obtained. . For any group of first rays among multiple groups of first rays, the phenomenon produced by the group of first rays after hitting the target object can be controlled to be one of the following: diffuse reflection, specular reflection, or specular reflection. It should be noted that the phenomena produced by the multiple sets of first light rays after hitting the target object may be the same or different, and are not limited here. Correspondingly, for any first sample pool among the plurality of first sample pools used to indicate the plurality of sets of first rays, the first sample pool corresponds to a certain rendering effect, and the rendering effect can present Is one of the following: diffuse, specular, or specular. It should be noted that the rendering effects corresponding to the multiple first sample pools may be the same or different. For example, a conventional sample pool configuration can be: construct two first sample pools, the rendering effect corresponding to the first first sample pool is presented as specular reflection, and the rendering effect corresponding to the second first sample pool is For high light reflection. It can be seen that the multi-sample pool technology provided by the aforementioned implementation can not only make different first sample pools correspond to different rendering effects, but also make different first sample pools correspond to the same rendering effect. Depending on the material of the target object, Making choices increases the flexibility and comprehensiveness of the plan.

In a possible implementation, the number of first rays indicated by the first sample pool is less than the number of pixels displayed on the screen, and the second The number of second rays indicated by the sample pool is equal to the number of pixels displayed on the screen used to display the image of the target object. In the foregoing implementation, for any first sample pool, the first sample pool may contain a set of light intensity values of the first light after hitting other objects and before hitting the target object. Then, the first sample pool The size of this cell is the number of these light intensity values, that is, the number of first rays in the group (that is, the size of the first sample cell is the number of first rays indicated by the first sample cell ). It is worth noting that the size of the first sample pool is smaller than the resolution of the screen (that is, the number of pixels displayed on the screen), and the screen is used to display the final image of the target object. For any second sample pool, the second sample pool may contain a set of light intensity values before the second light hits the target object. Then, the size of the second sample pool is the number of these light intensity values, also It is the number of the second rays of the group (that is, the size of the second sample cell is the number of the second rays indicated by the second sample cell). It is worth noting that the size of the second sample pool is equal to the resolution of the screen (ie, the number of pixels displayed on the screen). It can be seen that the foregoing implementation method can make the size of the first sample pool smaller than the resolution of the screen, which can improve the efficiency of acquiring the first sample pool.

In a possible implementation, rendering the target object based on the first sample pool and the second sample pool to obtain an image of the target object includes: rendering the target object based on the first sample pool to obtain the target object The first illumination image of the object; rendering the target object based on the second sample pool to obtain the second illumination image of the target object; fusing the first illumination image and the second illumination image to obtain the image of the target object. In the aforementioned implementation manner, since at least one first sample pool can be obtained, the at least one first sample pool can be colored respectively to obtain at least one indirect illumination image of the target object (ie, the aforementioned first illumination image). Since at least one second sample pool can be obtained, the at least one second sample pool can be colored respectively to obtain at least one direct illumination image of the target object (ie, the aforementioned second illumination image). After obtaining at least one indirect illumination image of the target object and at least one direct illumination image of the target object, these multiple images can be regarded as multiple layers. Therefore, at least one indirect illumination image of the target object and at least one direct illumination image of the target object can be obtained. The direct illumination images are superimposed to obtain the final image of the target object that can be displayed.

In a possible implementation, before fusing the first illumination image and the second illumination image to obtain the image of the target object, the method further includes: obtaining a noise-free component of the material information of the target object; based on the noise-free component Perform first processing on the first illumination image to obtain the processed first illumination image. The first processing includes demodulation processing, noise reduction processing and modulation processing; perform second processing on the second illumination image based on the noise-free component to obtain Obtaining the processed second illumination image, the second processing includes demodulation processing, noise reduction processing and modulation processing; fusing the first illumination image and the second illumination image to obtain the image of the target object includes: The first illumination image and the processed second illumination image are fused to obtain an image of the target object. In the aforementioned implementation manner, after obtaining the noise-free component of the material information of the target object, at least one indirect illumination image of the target object can be divided by the noise-free component to obtain at least one demodulated indirect illumination image of the target object. After obtaining at least one demodulated indirect illumination image of the target object, the at least one demodulated indirect illumination image of the target object can be denoised to obtain at least one denoised indirect illumination image of the target object. After obtaining at least one denoised indirect illumination image of the target object, the at least one denoised indirect illumination image of the target object can be multiplied by the noise-free component to obtain at least one modulated indirect illumination image of the target object. As at least one processed indirect illumination image of the target object (the processed indirect illumination image is the aforementioned processed first illumination image). At the same time, after obtaining the noise-free component of the material information of the target object, at least one direct illumination image of the target object can be divided by the noise-free component to obtain at least one demodulated direct illumination image of the target object. After obtaining at least one demodulated direct illumination image of the target object, the at least one demodulated direct illumination image of the target object can be denoised to obtain at least one denoised direct illumination image of the target object. After obtaining at least one direct illumination image of the target object after denoising, the at least one direct illumination image after denoising of the target object can be multiplied by the noise-free component to obtain at least one modulated direct illumination image of the target object, that is, At least one processed direct illumination image of the target object (the processed direct illumination image is the aforementioned processed second illumination image). After obtaining at least one processed indirect illumination image of the target object and at least one processed direct illumination image of the target object, the at least one processed indirect illumination image of the target object and at least one processed direct illumination image of the target object can be obtained The images are superimposed to obtain the final image of the target object that can be displayed. It can be seen that the aforementioned implementation method provides a new image noise reduction method, which can realize demodulation, noise reduction and modulation of each layer based on the noise-free classification of the material information of the target object, so that each layer can be kept as smooth as possible. The details of the target object make the image of the target object obtained based on the superposition of each layer more realistic and clearly visible, which is conducive to improving the user's viewing experience and further improving the user experience.

In a possible implementation, the first processing also includes super-resolution processing, and the super-resolution processing is used to make the resolution of the processed first illumination image equal to the resolution of the screen. In the aforementioned implementation manner, since the size of the at least one first sample pool is smaller than the resolution of the screen, the resolution of the modulated at least one indirect illumination image of the target object is also smaller than the resolution of the screen. In order to achieve better integration of each layer, after obtaining at least one modulated indirect illumination image of the target object, you can also perform super-resolution on at least one modulated indirect illumination image of the target object. Processing, using at least one super-resolved indirect lighting image of the target object as at least one processed indirect lighting image of the target object, at this time, the resolution of the at least one processed indirect lighting image of the target object is equal to the resolution of the screen . It can be seen that the aforementioned implementation method can be used with super-resolution processing in the process of rendering target objects based on multi-sample pool technology, which can reduce the amount of calculation required in the process to a certain extent and is conducive to accelerating ray tracing based on multi-sample pools. The implementation of technology further improves user experience.

A second aspect of the embodiment of the present application provides an image rendering device. The device includes: a resampling module for resampling the light path between the target object and the light source based on the original light path between the target object and the light source. Sampling is performed to obtain a new light path. The original light path is obtained through ray tracing. The original light path starts with the target object and ends with the light source. The new light path starts with the light source and ends with the target object. The first acquisition module uses Based on the new light path, the sample pool is obtained. The sample pool is used to indicate the light rays directed to the target object. The new light path is formed by the light; the rendering module is used to render the target object based on the sample pool to obtain an image of the target object. .

It can be seen from the above device that after obtaining the original light path between the target object and the light source, the light path between the target object and the light source can be resampled based on the original light path to obtain a new light path. The original light path passes through the light path. It is traced that the original light path starts from the target object and ends with the light source, and the new light path starts from the light source and ends with the target object. A pool of samples can then be obtained based on the new ray path, indicating the rays hitting the target object from which the new ray path is formed. Finally, the target object can be rendered based on the sample pool to obtain the image of the target object. Based on the foregoing process, it can be seen that the embodiment of the present application provides a new light path resampling method, which is contrary to the direction of the effective light path in ray tracing (with the light source as the starting point and the target object as the end point), and is targeted Ground resampling of light paths can improve the success rate of sampling to effective new light paths, that is, it can easily resample to effective new light paths between the target object and the light source. In this way, no matter the light that forms the new light path Regardless of whether the phenomenon produced by the target object is direct illumination, diffuse reflection, specular reflection, or specular reflection, a high-quality and sufficient sample pool can be successfully obtained, so that the rendering effect of the target object is high enough.

In a possible implementation, the new light path includes a first light path and a second light path. The first light path starts with the light source, ends with the target object, and passes through other objects. The second light path starts with the light source. The starting point is the target object as the end point, and does not pass through other objects; the sample pool includes the first sample pool and the second sample pool. The first sample pool is used to indicate the first light ray directed to the target object. The first light path is A first ray is formed, a second sample cell is used to direct a second ray directed to the target object, and a second ray path is formed by the second ray.

In a possible implementation, the phenomenon caused by the first light ray hitting the target object is specular reflection or specular reflection.

In a possible implementation, the number of first rays indicated by the first sample pool is less than the number of pixels displayed on the screen, and the number of second rays indicated by the second sample pool is equal to the number of pixels displayed on the screen. Quantity, the screen is used to display the image of the target object.

In a possible implementation, the rendering module is configured to: render the target object based on the first sample pool to obtain the first illumination image of the target object; render the target object based on the second sample pool to obtain the The second illumination image of the target object; fuse the first illumination image and the second illumination image to obtain an image of the target object.

In a possible implementation, the rendering module is also used to: obtain the noise-free component of the material information of the target object; and perform the first processing on the first illumination image based on the noise-free component to obtain the processed first illumination image. , the first processing includes demodulation processing, noise reduction processing and modulation processing; the second processing is performed on the second illumination image based on the noise-free component to obtain the processed second illumination image, and the second processing includes demodulation processing, noise reduction Processing and modulation processing; a rendering module, used to fuse the processed first lighting image and the processed second lighting image to obtain an image of the target object.

In a possible implementation, the first processing also includes super-resolution processing, and the super-resolution processing is used to make the resolution of the processed first illumination image equal to the resolution of the screen.

A third aspect of the embodiment of the present application provides an electronic device. The electronic device includes a memory and a processor; the memory stores code, and the processor is configured to execute the code. When the code is executed, the electronic device executes the first step. The method described in any possible implementation manner of the aspect or the first aspect.

A fourth aspect of the embodiments of the present application provides a circuit system. The circuit system includes a processing circuit configured to perform the method described in the first aspect and any possible implementation manner of the first aspect.

A fifth aspect of the embodiments of the present application provides a chip system. The chip system includes a processor for calling a computer program or computer instructions stored in a memory, so that the processor executes the steps described in the first aspect and the first aspect. any possible implementation method.

In one possible implementation, the processor is coupled to the memory through an interface.

In a possible implementation, the chip system further includes a memory, and computer programs or computer instructions are stored in the memory.

A sixth aspect of the embodiments of the present application provides a computer storage medium. The computer storage medium stores one or more instructions. When executed by one or more computers, the instructions cause one or more computers to implement the first aspect or the third aspect. On the one hand, any possible implementation method is described.

The seventh aspect of the embodiments of the present application provides a computer program product. The computer program product stores instructions. When the instructions are executed by a computer, the computer implements the implementation as described in the first aspect or any possible implementation manner of the first aspect. Methods.

In the embodiment of the present application, after obtaining the original light path between the target object and the light source, the light path between the target object and the light source can be resampled based on the original light path to obtain a new light path. The original light path is traced through ray tracing. It is obtained that the original light path starts from the target object and ends with the light source, and the new light path starts from the light source and ends with the target object. A pool of samples can then be obtained based on the new ray path, indicating the rays hitting the target object from which the new ray path is formed. Finally, the target object can be rendered based on the sample pool to obtain the image of the target object. Based on the foregoing process, it can be seen that the embodiment of the present application provides a new light path resampling method, which is contrary to the direction of the effective light path in ray tracing (with the light source as the starting point and the target object as the end point), and is targeted Ground resampling of light paths can improve the success rate of sampling to effective new light paths, that is, it can easily resample to effective new light paths between the target object and the light source. In this way, no matter the light that forms the new light path Regardless of whether the phenomenon produced by the target object is direct illumination, diffuse reflection, specular reflection, or specular reflection, a high-quality and sufficient sample pool can be successfully obtained, so that the rendering effect of the target object is high enough.

Description of drawings

Figure 1 is a schematic diagram of the principle of ray tracing technology;

Figure 2 is a schematic diagram of the principle of rasterization technology;

Figure 3 is a schematic structural diagram of an electronic device provided by an embodiment of the present application;

Figure 4 is a schematic flow chart of an image rendering method provided by an embodiment of the present application;

Figure 5 is a schematic diagram of the BVH tree provided by the embodiment of the present application;

Figure 6 is a schematic diagram of light path resampling provided by an embodiment of the present application;

Figure 7 is a schematic diagram of an application example of the image rendering method provided by the embodiment of the present application;

Figure 8 is a schematic diagram of the noise reduction process provided by the embodiment of the present application;

Figure 9 is a schematic diagram of the comparison results provided by the embodiment of the present application;

Figure 10 is another schematic diagram of the comparison results provided by the embodiment of the present application;

Figure 11 is another schematic diagram of the comparison results provided by the embodiment of the present application;

Figure 12 is another schematic diagram of the comparison results provided by the embodiment of the present application;

Figure 13 is another schematic diagram of the comparison results provided by the embodiment of the present application;

Figure 14 is another schematic diagram of the comparison results provided by the embodiment of the present application;

Figure 15 is a schematic structural diagram of an image rendering device provided by an embodiment of the present application.

Detailed ways

The terms "first", "second", etc. in the description and claims of this application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the terms so used are interchangeable under appropriate circumstances, and are merely a way of distinguishing objects with the same attributes in describing the embodiments of the present application. Furthermore, the terms "include" and "having" and any variations thereof, are intended to cover non-exclusive inclusions, such that a process, method, system, product or apparatus comprising a series of elements need not be limited to those elements, but may include not explicitly other elements specifically listed or inherent to such processes, methods, products or equipment.

With the development of computer technology, more and more applications, such as game applications or video applications, require the display of fine-quality images on electronic devices. These images are usually rendered by electronic devices based on models in a three-dimensional (3D) scene.

In traditional image processing methods, rasterization processing is usually used to render 3D scenes to obtain images that can display 3D scenes. However, the quality of images obtained by rendering using rasterization technology is average, and it is often difficult to present realistic images. For example, it is often difficult to truly reproduce the effects of light reflection, refraction, and shadows in the scene in the rendered image. In view of this, a new rendering technology - ray tracing technology came into being. Ray tracing technology and rasterization technology are both methods used to achieve image rendering. Their main purpose is to project objects in 3D space into a two-dimensional screen space for display by calculating shading.

Figure 1 is a schematic diagram of the principle of ray tracing technology. As shown in Figure 1, the principle of ray tracing is: from the position of the camera, through the pixel position on the image plane, emit a beam of ray into the three-dimensional scene, find the nearest intersection point between the ray and the geometric figure, and then find the coloring of the intersection point . If the material of the intersection is reflective, you can continue tracing in the reflection direction of the intersection and continue to obtain the coloring of the reflected intersection. In other words, the ray tracing method calculates projection and global illumination by tracing the propagation process of light in a three-dimensional scene, thereby rendering a two-dimensional image.

Figure 2 is a schematic diagram of the principle of rasterization technology. As shown in Figure 2, the principle of rasterization is: segment the objects in the three-dimensional scene using triangles, transform the three-dimensional coordinates of the triangle vertices into two-dimensional coordinates on the image through coordinate change calculation, and finally within the triangle on the image Fill texture to achieve image rendering.

Since rasterization technology directly projects the content visible on the screen space onto the screen space to obtain the corresponding image, its processing difficulty is relatively low and the light and shadow effects provided are also poor. Ray tracing technology achieves real-life effects such as reflection, refraction, shadows, and ambient light occlusion by tracking each ray of light emitted from the camera. Therefore, the ray tracing method can provide realistic light and shadow effects. Therefore, in order to render more realistic images, current electronic devices usually give priority to ray tracing technology to render three-dimensional scenes to improve the user's viewing experience.

Currently, related technologies propose a ray tracing algorithm based on path resampling. In this algorithm, for an object to be rendered in a three-dimensional scene, after determining the original light path formed by the light that intersects the object for the first time, the light path can be resampled based on the original light path, thereby collecting A new ray path formed by a new ray that intersects the object for the first time. Then, a corresponding sample pool can be obtained based on the new light path, and the sample pool is used to indicate the new rays forming the new light path (that is, the sample pool can store light samples). An image of the object can then be rendered based on this pool of samples.

In this algorithm, limited by the material of the object, if the effect of light on an object is diffuse reflection, the light re-emitted for the object will often have a large reflection range after the light hits the object. Even if the light continues to hit other objects, it has a higher probability of reaching the light source, so it can be easily resampled to a valid new light path. Then, the rendering effect displayed on the object is better (that is, the rendered image of the object is often better quality). If the effect of light on the object is specular reflection or specular reflection, for the light re-emitted by the object, the reflection range after the light hits the object is often small. After the reflected light continues to hit other objects, it will only There is a small chance of reaching the light source, so it is difficult to resample an effective new light path, resulting in the inability to obtain a high-quality and sufficient sample pool. Therefore, the rendering effect displayed on the object is often poor (that is, the rendered image of the object often of poorer quality).

Furthermore, not only is the efficiency of obtaining the sample pool in this algorithm not high, but the amount of data in the sample pool is often very large, which makes the process of rendering objects based on the sample pool require a large amount of calculation, resulting in the entire rendering process It takes a lot of time and costs, resulting in poor user experience.

Furthermore, in the process of rendering objects, certain image noise reduction processing is often used to filter the noise in the image. However, the noise reduction processing of related technologies often filters the entire image, causing the final image of the object to become blurred. Such an image will lead to a poor user experience and reduce the user experience.

In order to solve the above problem, embodiments of the present application provide an image rendering method, which can be executed by an electronic device. The electronic device includes a CPU and a GPU, which can render images. Illustratively, the electronic device may be a mobile phone (mobile phone), a tablet computer, a notebook computer, a PC, a mobile internet device (mobile internet device, MID), a wearable device, a virtual reality (virtual reality, VR) device, an enhanced Augmented reality (AR) equipment, wireless electronic equipment in industrial control (industrial control), wireless electronic equipment in self-driving (self driving), wireless electronic equipment in remote medical surgery, smart grid ( Wireless electronic devices in smart grid, wireless electronic devices in transportation safety, wireless electronic devices in smart city, wireless electronic devices in smart home, etc. The electronic device can be a device running Android system, IOS system, Windows system and other systems. Applications that need to render 3D scenes to obtain two-dimensional images can be run in the electronic device, such as game applications, lock screen applications, map applications, or monitoring applications.

In order to facilitate understanding, the specific structure of the electronic device will be introduced in detail below with reference to Figure 3 . Please refer to FIG. 3 , which is a schematic structural diagram of an electronic device provided by an embodiment of the present application. As shown in Figure 3, the electronic device 3000 may include: a central processing unit 3001; Graphics processor 3002, display device 3003 and memory 3004. Optionally, the electronic device 3000 may also include at least one communication bus (not shown in FIG. 3) for realizing connection communication between various components.

It should be understood that various components in the electronic device 3000 can also be coupled through other connectors, and other connectors can include various types of interfaces, transmission lines or buses, etc. Each component in the electronic device 3000 may also be radially connected with the central processor 3001 as the center. In various embodiments of the present application, coupling refers to electrical connection or communication with each other, including direct connection or indirect connection through other devices.

There are many ways to connect the central processing unit 3001 and the graphics processor 3002, which are not limited to the way shown in Figure 2. The central processor 3001 and the graphics processor 3002 in the electronic device 3000 may be located on the same chip, or they may be separate chips.

The following briefly introduces the functions of the central processor 3001, the graphics processor 3002, the display device 3003, and the memory 3004.

Central processing unit 3001: used to run operating system 3005 and application programs 3006. The application 3006 may be a graphics application, such as a game, a video player, etc. The operating system 3005 provides a system graphics library interface, and the application program 3006 uses the system graphics library interface and the driver provided by the operating system 3005, such as the graphics library user mode driver and/or the graphics library kernel mode driver to generate graphics or graphics for rendering. A flow of instructions for image frames, along with the required associated rendering data. Among them, system graphics libraries include but are not limited to: open graphics library for embedded system (OpenGL ES), the khronos platform graphics interface (the khronos platform graphics interface) or Vulkan (a cross-platform graphics application Program interface) and other system graphics libraries. The instruction stream contains a series of instructions, which are usually instructions for calling the system graphics library interface.

Optionally, the central processor 3001 may include at least one of the following types of processors: an application processor, one or more microprocessors, a digital signal processor (DSP), a microcontroller unit, MCU) or artificial intelligence processor, etc.

The central processing unit 3001 may further include necessary hardware accelerators, such as application specific integrated circuits (ASICs), field programmable gate arrays (field programmable gate arrays, FPGAs), or integrated circuits used to implement logic operations. Processor 3001 may be coupled to one or more data buses for transmitting data and instructions between various components of electronic device 3000 .

Graphics processor 3002: used to receive the graphics instruction stream sent by the processor 3001, generate a rendering target through a rendering pipeline (pipeline), and display the rendering target to the display device 3003 through the layer synthesis display module of the operating system. The rendering pipeline may also be called a rendering pipeline, a pixel pipeline or a pixel pipeline, and is a parallel processing unit inside the graphics processor 3002 for processing graphics signals. The graphics processor 3002 may include multiple rendering pipelines, and the multiple rendering pipelines may process graphics signals in parallel independently of each other. For example, the rendering pipeline can perform a series of operations in the process of rendering graphics or image frames. Typical operations can include: vertex processing (vertex processing), primitive processing (primitive processing), rasterization (rasterization), fragment processing (fragment) processing) and so on.

Optionally, the graphics processor 3002 may include a general-purpose graphics processor that executes software, such as a GPU or other types of dedicated graphics processing units.

Display device 3003: used to display various images generated by the electronic device 3000, which can be a graphical user interface (GUI) of the operating system or image data (including still images and videos) processed by the graphics processor 3002 data).

Optionally, display device 3003 may include any suitable type of display screen. For example, liquid crystal display (LCD) or plasma display or organic light-emitting diode (OLED) display.

The memory 3004 is a transmission channel between the central processor 3001 and the graphics processor 3002, and can be double data rate synchronous dynamic random access memory (DDR SDRAM) or other types of cache.

The specific structure of the electronic device used in the image rendering method provided by the embodiment of the present application has been introduced above. The process of the image rendering method provided by the embodiment of the present application will be introduced in detail below. Figure 4 is a schematic flow chart of an image rendering method provided by an embodiment of the present application. As shown in Figure 4, the method includes:

401. Obtain the original light path between the target object and the light source. The original light path is obtained through ray tracing. The original light path starts from the target object and ends with the light source.

In this embodiment, after obtaining the model file of a certain three-dimensional scene, the rendering information of each object in the three-dimensional scene, the spatial information of the camera, the spatial information of the light source, etc. can be parsed from it. These information are briefly introduced below. :

(1) The rendering information of each object includes the spatial information of the object and the material information of the object. The spatial information of the object includes the vertex coordinates of the object, the vertex normal of the object, and the triangle index. The material information of the object includes The color of the object, the gold of the object Attributes and roughness of objects and other information.

It should be noted that hierarchical bounding box (bounding volume hierarchies, BVH) trees can be constructed based on the spatial information of each object, and the BVH tree can be used to implement subsequent ray tracing operations. Specifically, a BVH tree can be constructed based on the object's vertex coordinates, vertex normals, and triangle indexes. It can be understood that the BVH tree contains spatial information of multiple bounding boxes, and the spatial information of each bounding box contains the bounding box (cuboid). The coordinates of the 8 vertices and the vertical height of the 8 vertices, each bounding box is used to surround at least one object. In order to further understand the aforementioned BVH tree, further introduction will be made below in conjunction with Figure 5. Figure 5 is a schematic diagram of the BVH tree provided by the embodiment of the present application. As shown in Figure 5, assuming that there are 6 objects in the three-dimensional scene, the bounding box A (surrounding these 6 objects) can be determined based on the spatial information of these 6 objects. ), the spatial information of bounding box B (surrounding these 4 objects) can be determined based on the spatial information of 4 of the objects, and the spatial information of bounding box C (surrounding these 2 objects) can be determined based on the spatial information of the remaining 2 objects. The spatial information of bounding box A, the spatial information of bounding box B, and the spatial information of bounding box C are managed in a binary tree structure to obtain a BVH tree. It can be seen that the BVH tree can centrally manage the spatial information of bounding box A, bounding box B and bounding box C, which is equivalent to centralized management of bounding box A, bounding box B and bounding box C. Among them, bounding box A is the largest bounding box among all bounding boxes.

When implementing ray tracing based on BVH trees, it is necessary to calculate whether the light intersects with objects in the three-dimensional scene (i.e., intersection calculation). Due to the existence of the BVH tree, a bounding box surrounding the object can be determined based on the BVH tree, and then it is determined whether the light intersects the bounding box. If the light does not touch the bounding box, it means that the light will definitely not interact with the object in the bounding box. Intersect; if the light hits the bounding box, then calculate whether the light intersects with the object in the bounding box. For example, when it is detected that the light does not intersect with the B bounding box in the binary tree, it means that the light will definitely not intersect with the four objects in the B bounding box. Therefore, it is unnecessary to detect whether the light ray intersects with the four objects in the B bounding box. The step of intersecting objects, thereby only detecting whether the light intersects two objects in the C bounding box.

In addition, calculations can also be performed based on the material information of each object to obtain the noise-free component of the material information of each object, and the noise-free component of the material information of each object is processed into a table that can be queried to achieve subsequent targeting. The demodulation processing, noise reduction processing and modulation processing of the image of the object will not be discussed here.

(2) The spatial information of the camera may include the vertical height of the camera, the coordinates of the camera, etc. The camera is used to capture images that simulate a three-dimensional scene. It should be noted that when implementing ray tracing, the spatial position of the camera in the three-dimensional scene can be determined based on the spatial information of the camera. Then, the emission point of the light can be determined based on the spatial position of the camera, that is, the camera is used to emit rays to each object in the three-dimensional scene to implement route calculation and intersection calculation to determine the light path formed by these light rays in the three-dimensional scene. , thereby realizing subsequent rendering operations for each object in the three-dimensional scene.

(3) The spatial information of the light source is usually used to determine the direct light source and indirect light source in the three-dimensional scene. The direct light source refers to the light source that emits its own light in the three-dimensional scene. The indirect light source refers to the light source that does not emit its own light in the three-dimensional scene and relies on other light sources to achieve emitting light. , for example, each of the aforementioned objects can be used as an indirect light source. It should be noted that when implementing ray tracing, if the light emitted by the camera intersects with at least one object in the three-dimensional scene, is reflected on the surface of the object, and is finally received by the direct light source, such light will form The ray path is a valid ray path. Then, for some light rays emitted by the camera, assuming that these light rays first intersect with an object in the three-dimensional scene and are finally received by the direct light source (regardless of whether they intersect with other objects in the middle), then these rays can be placed on the object The effective light path formed between the object and the direct light source is used as the original light path between the object and the direct light source (that is, the light path between the object and the direct light source is first sampled). Similar operations can be performed for other objects in the three-dimensional scene, so the original light path between each object in the three-dimensional scene and the direct light source can be obtained. These original light paths can be used to resample the light paths of each object and the direct light source in the subsequent three-dimensional scene, and to render the image of each object.

It can be seen that after obtaining the rendering information of each object in the three-dimensional scene, the spatial information of the camera, and the spatial information of the light source, the original light path between each object and the direct light source (hereinafter referred to as the light source) can be obtained based on this information.

Since the operations performed on various objects in this embodiment are similar, for convenience of explanation, one of the objects is used as a schematic introduction below, and this object is called the target object. Then, according to the aforementioned introduction, based on the rendering information of each object in the three-dimensional scene, the spatial information of the camera, and the spatial information of the light source, the initial sampling of the light path between the target object and the direct light source can be completed, thereby obtaining the target object and the light source. The original light path between the target object and the light source starts with the target object and ends with the light source. It may or may not pass through other objects in the middle.

It is worth noting that the original light path between the target object and the light source involved in this embodiment can also be understood as a set of original light paths between the target object and the light source, because the light emitted by the camera first intersects with the target object. There are usually multiple rays, which can be called multiple original rays, that is, a group of original rays. Then, the set of original rays formed between the target object and the light source The light path is a set of original light paths between the target object and the light source.

402. Based on the original light path between the target object and the light source, resample the light path between the target object and the light source to obtain a new light path. The new light path starts with the light source and ends with the target object.

After obtaining the original light path between the target object and the light source, since the original light path between the target object and the light source starts with the target object and ends with the light source, the two endpoints of the target object and the light source can be determined, and the two endpoints can be determined. The properties of the two are exchanged to reconstruct the light path between the two in some way, that is, the light path between the two is first resampled to obtain the first light between the target object and the light source. Path (can also be called indirect light path), the first light path starts from the light source, ends at the target object, and passes through other objects in the middle.

Specifically, since the first resampling can be performed only once or repeatedly for multiple rounds, the first light path between the light path between the target object and the light source can be obtained in a variety of ways:

(1) Perform the first resampling based on a set of original light paths between the target object and the light source, and accordingly obtain a set of first light paths between the target object and the light source. The set of original light paths includes multiple original light paths. The light path corresponds one-to-one to a plurality of first light paths included in the group of first light paths. Among multiple original light paths, for any original light path, assume that the original light path starts from a certain surface point of the target object, passes through a certain surface point of other objects in the middle, and ends with a certain surface of the light source. point is the end point. For the convenience of explanation, below, a certain surface point of the target object passed by the original light path is called a first surface point, and a certain surface point of the other objects passed by the original light path is called a second surface point. A certain surface point of the light source that the original light path passes through is called a third surface point. Then, the original light path can be divided into two paths. The first path is the light path between the first surface point and the second surface point, and the second path is the light path between the second surface point and the third surface point. Light path. When performing the first resampling for the first path, a surface area can be drawn centered on the second surface point of the remaining object, and selected in the surface area (for example, it can be selected randomly or according to a certain preset Select according to the rules, no restrictions are made here) A surface point is called the fourth surface point (the fourth surface point can be the same surface point as the second surface point, or it can be a certain point around the second surface point. surface point (not limited here), and connect the fourth surface point with the first surface point to form a first new path from the fourth surface point to the first surface point. Similarly, when performing the first resampling for the second path, a surface area can be drawn centered on the third surface point of the light source, and a surface point is selected in the surface area, called the fifth surface point (the fifth surface point). The fifth surface point can be the same surface point as the third surface point, or it can be a surface point around the third surface point, there is no restriction here), and connect the fifth surface point with the fourth surface point, A second new path is formed from the fifth surface point to the fourth surface point. In this way, the first new path and the second new path constitute the first light path corresponding to the original light path. The first light path starts from the fifth surface point and passes through the fourth surface point in the middle. Surface points, with the first surface point as the end point. For the remaining original light paths except this original light path among the multiple original light paths, the same operations as the operations performed on the original light path can also be performed on the remaining original light paths, so a one-to-one correspondence with the multiple original light paths can be obtained. A plurality of first light paths, that is, a set of first light paths between the target object and the light source.

It should be understood that the foregoing example only uses one other object passing through the middle of the original light path for schematic introduction, and does not limit the number of other objects passing through the original light path. If the original light path passes through multiple other objects, the original light path will not be affected. The first resampling operation performed on the path is also similar and will not be described again here. Correspondingly, based on the first light path obtained from the original light paths passing through a plurality of other objects in the middle, a plurality of other objects can also be passed in the middle. In order to further understand the foregoing first resampling process, further explanation will be given below with reference to the example shown in FIG. 6 . Figure 6 is a schematic diagram of light path resampling provided by an embodiment of the present application. As shown in Figure 6, when performing ray tracing in a certain room, assume that the camera 100 emits a set of original light rays to the table 200, and select one of the original light rays. Light is introduced schematically. Assume that after starting from the camera 100, the original light first hits the surface point A of the table 200, then hits the surface point B of the wall 300 after reflection, hits the surface point C of the lampshade 400 after reflection again, and hits the light source 500 after reflection again. Surface point D, so the original light path of the original light between the table 200 and the light source 500 includes three paths. The first path 601 points from surface point A to surface point B, and the second path 602 points from surface point B. Surface point C, the third path 603 is directed from surface point C to surface point D. Then, you can take the surface point E around the surface point B on the wall 300, and construct the first new path 604 from the surface point E to the surface point A, and then keep the surface point C on the lampshade 400 unchanged, and construct A second new path 605 from surface point C to surface point E, then take the surface point F around surface point D on the light source 500, and construct a third new path 606 from surface point F to surface point C. . In this way, these three new paths constitute a first ray path. It can be understood that similar operations can also be performed on the original light paths formed by the remaining original light rays emitted by the camera 100 towards the table 200, so that multiple first light paths between the table 200 and the light source 500 can be obtained, that is, the table A first set of light paths between 200 and light source 500 .

(2) Repeat (1) for multiple rounds to obtain multiple sets of first light paths between the target object and the light source, that is, perform the first resampling based on a set of original light paths between the target object and the light source, After correspondingly obtaining a set of first light paths between the target object and the light source, the first resampling can be performed again based on a set of original light paths between the target object and the light source, and another corresponding set of light paths between the target object and the light source can be obtained. A set of first light paths, and so on, after performing multiple rounds of first resampling, multiple sets of first light paths between the target object and the light source can be obtained.

In addition, after obtaining the original light path between the target object and the light source, the light path between the two is reconstructed in another way based on the original light path between the two, that is, the light path between the two is reconstructed. Perform the first resampling to obtain the second light path between the target object and the light source (which can also be called the direct light path). The second light path starts from the light source and ends with the target object, and does not pass through other objects in the middle. .

Specifically, since the second resampling can be performed only once or repeatedly for multiple rounds, the second light path between the light path between the target object and the light source can be obtained in a variety of ways:

(1) The second resampling is performed based on a set of original light paths between the target object and the light source, and a set of second light paths between the target object and the light source can be obtained accordingly. The set of original light paths contains multiple original light paths. The light path has a one-to-one correspondence with the plurality of second light paths included in the set of second light paths. Among the multiple original light paths, for any original light path, let the original light path take a certain surface point of the target object as the starting point and a certain surface point of the light source as the end point. For convenience of explanation, below, the certain surface point of the target object that the original light path passes through is called a first surface point, and the certain surface point of the light source that the original light path passes through is called a second surface point. Then, when performing the second resampling for the original light path, a surface area can be drawn centered on the second surface point of the light source, and selected in the surface area (for example, it can be randomly selected, or it can be selected according to some predetermined The selection is based on the placement rules, and there are no restrictions here.) A surface point is called the third surface point (the third surface point can be the same surface point as the second surface point, or it can be a certain point around the second surface point. surface points (not limited here), and connect the third surface point with the first surface point to form a second light path from the third surface point to the first surface point. The second light path starts from the third surface point, does not pass through the surface points of other objects in the middle, and ends at the first surface point. For the remaining original light paths except this original light path among the multiple original light paths, the same operations as the operations performed on the original light path can also be performed on the remaining original light paths, so a one-to-one correspondence with the multiple original light paths can be obtained. A plurality of second light paths, that is, a set of second light paths between the target object and the light source.

(2) Repeat (1) for multiple rounds to obtain multiple sets of second light paths between the target object and the light source, that is, perform a second resampling based on a set of original light paths between the target object and the light source, After a set of second light paths between the target object and the light source is obtained, a second resampling can be performed based on a set of original light paths between the target object and the light source, and another set of light paths between the target object and the light source can be obtained accordingly. A set of second light paths, and by analogy, after performing multiple rounds of second resampling, multiple sets of second light paths between the target object and the light source can be obtained.

403. Based on the new light path between the target object and the light source, obtain a sample pool. The sample pool is used to indicate the light rays directed to the target object, and the new light path is formed by these light rays.

Obtaining the first light path between the target object and the light source is equivalent to obtaining the first light (also called indirect light) between the target object and the light source. The first light is the light that forms the first light path. It can be regarded as light that is emitted from the light source, hits other objects in the middle, hits the target object after reflection, and is received by the camera after reflection.

Then, information related to the first ray can be obtained, and the information can be used to construct the first sample pool. For example, the light intensity value (such as the brightness and color of the first light, etc.) of the first light after hitting other objects and before hitting the target object can be obtained, and the first light after hitting other objects and before hitting the target object can be obtained. The light intensity value before hitting the target object is stored in the first sample pool. In this way, the first sample cell can be used to indicate the first light ray directed to the target object.

Specifically, the first sample pool can be obtained in various ways:

(1) If only a set of first light paths between the target object and the light source are obtained, since one first light path is formed by one first light ray, the set of first light paths is formed by a set of first light rays. Since the group of first ray shapes includes a plurality of first rays, information related to the plurality of first rays can be used to construct a first sample pool (for example, the plurality of first rays are used to hit other first rays). The light intensity value after the object and before hitting the target object is stored in a first sample pool), so the first sample pool can be used to indicate the group of first rays that hit the target object. Still as in the above example, after obtaining a set of first light paths between the target object and the light source, the first new path 604, the second new path 605, and the third new path 606 constitute the first new path. For the light path, the light intensity value of the first light that forms the first light path after hitting the lampshade and the wall and before hitting the table can be obtained and stored in the first sample pool. Likewise, the group between the table and the light source In the first light path, in addition to the first light path, the light intensity values of the other first light rays that form the other first light paths after hitting other objects and before hitting the table can be obtained and stored in the first light path. in this pool. In this way, a complete first sample cell is obtained, which is used to indicate a set of first rays hitting the table.

Further, the phenomenon produced by the group of first rays after hitting the target object can be controlled to be one of the following: diffuse reflection, specular reflection or specular reflection. Correspondingly, the first sample pool corresponds to a certain kind of rendering. Effect, the rendering effect can appear as one of the following: diffuse reflection, specular reflection, or specular reflection. For example, if the phenomenon produced by the first set of rays after hitting the target object is diffuse reflection, the rendering effect corresponding to the first sample pool will appear as diffuse reflection, and so on.

(2) If multiple sets of first light paths between the target object and the light source are obtained, these multiple sets of first light paths are formed by multiple sets of first light rays (one-to-one correspondence between the two). For any group of first rays among multiple groups of first rays, since the group of first ray shapes includes multiple first rays, information related to the multiple first rays can be used to construct a first ray shape. Sample pool (for example, the light intensity values of the plurality of first rays after hitting other objects and before hitting the target object are stored in a first sample pool). Similarly, for the remaining groups of first rays, the same operations as for the group of first rays can also be performed, so that multiple first sample pools can finally be obtained, and one first sample pool is used to indicate the target object. of a set of first rays.

Further, for any group of first rays among the plurality of groups of first rays, the phenomenon produced by the group of first rays after hitting the target object can be controlled to be one of the following: diffuse reflection, specular reflection, or specular reflection. It should be noted that the phenomena produced by the multiple sets of first light rays after hitting the target object may be the same or different, and are not limited here. Correspondingly, for any first sample pool among the plurality of first sample pools, the first sample pool corresponds to a certain rendering effect, and the rendering effect can be presented as one of the following: diffuse reflection, specular reflection Reflection or specular reflection. It should be noted that the rendering effects corresponding to the multiple first sample pools may be the same or different, and are not limited here. For example, a conventional sample pool configuration can be: construct two first sample pools, the rendering effect corresponding to the first first sample pool is presented as specular reflection, and the rendering effect corresponding to the second first sample pool is For high light reflection.

Furthermore, whether in case (1) or case (2), for any first sample pool (which can also be understood as a first sample image), the first sample pool may include a The light intensity values of the group of first rays after hitting other objects and before hitting the target object, then the size of the first sample pool (the resolution of the first sample image) is the number of these light intensity values, That is, the number of the first rays of the group (that is, the size of the first sample pool (the resolution of the first sample image) is the number of the first rays indicated by the first sample pool). It is worth noting that the size of the first sample pool is smaller than the resolution of the screen (that is, the number of pixels displayed on the screen), and the screen is used to display the final image of the target object.

In addition, obtaining the second light path between the target object and the light source is equivalent to obtaining the second light ray (also called direct light) between the target object and the light source. The second light ray forms the second light ray path. Light can be regarded as light emitted from the light source, without hitting other objects in the middle, but directly hitting the target object, and then reflected by the light received by the camera.

Then, information related to the second ray can be obtained and used to construct the second sample pool. For example, the light intensity value of the second light ray before it hits the target object (such as the brightness and color of the second light ray, etc.) can be obtained, and the light intensity value of the second light ray before it hits the target object can be stored in the second in the sample pool. In this way, the second sample cell can be used to indicate the second light ray directed to the target object.

Specifically, the second sample pool can be obtained in various ways:

(1) If only a set of second ray paths between the target object and the light source are obtained, since one second ray path is formed by one second ray, the set of second ray paths is formed by a set of second rays. Since the set of second ray shapes includes a plurality of second rays, information related to the plurality of second rays can be used to construct a second sample pool (for example, the plurality of second rays are used to hit the target object The previous light intensity value is stored in a second sample cell), so the second sample cell can be used to indicate the set of second light rays emitted to the target object.

(2) If multiple sets of second light paths between the target object and the light source are obtained, these multiple sets of second light paths are formed by multiple sets of second light rays (one-to-one correspondence between the two). For any group of second rays among the plurality of groups of second rays, since the group of second ray shapes includes a plurality of second rays, information related to the plurality of second rays can be used to construct a second ray shape. Sample pool (for example, store the light intensity values of the plurality of second rays before hitting the target object in a second sample pool). Similarly, for the remaining sets of second rays, the same operations as for the set of second rays can also be performed, so that multiple second sample pools can finally be obtained, and one second sample pool is used to indicate a beam emitted to the target object. Set the second ray.

Further, whether in case (1) or case (2), for any second sample pool (which can also be understood as a second sample image), the second sample pool may contain a set of second rays The light intensity value before hitting the target object, then the size of the second sample pool (the resolution of the second sample image) is the number of these light intensity values, that is, the number of the second ray group (that is, Say, second The size of the sample pool (the resolution of the second sample image) is the number of second rays indicated by the second sample pool). It is worth noting that the size of the second sample pool is equal to the resolution of the screen (that is, the number of pixels displayed on the screen), and the screen is used to display the final image of the target object.

404. Based on the sample pool, render the target object to obtain an image of the target object.

After obtaining the first sample pool, the first sample pool can be used to render the target object, thereby obtaining an image of the target object. Then, the image of the target object is the displayed image, which can be displayed on the screen for the user to view and use.

Specifically, the image of the target object can be obtained in the following ways:

(1) Render the target object based on the first sample pool to obtain the indirect lighting image of the target object. It should be noted that since at least one first sample pool can be obtained (for example, in the aforementioned case (1), only one first sample pool is obtained, and for example, in the aforementioned case (2), multiple first sample pools are obtained ), the at least one first sample pool can be colored (for example, lighting operation, etc.) respectively, and at least one indirect lighting image of the target object can be obtained (for example, corresponding to the aforementioned situation (1), an indirect lighting image of the target object can be obtained Illumination image, for another example, corresponding to the aforementioned situation (2), multiple indirect illumination images of the target object can be obtained).

(2) Render the target object based on the second sample pool to obtain the direct illumination image of the target object. It should be noted that since at least one second sample pool can be obtained (for example, in the aforementioned case (1), only one second sample pool is obtained, and for example, in the aforementioned case (2), multiple second sample pools are obtained), it can be Perform coloring (for example, lighting operation, etc.) on at least one second sample pool respectively to obtain at least one direct lighting image of the target object (for example, corresponding to the aforementioned situation (1), a direct lighting image of the target object can be obtained, and For example, corresponding to the aforementioned situation (2), multiple direct illumination images of the target object can be obtained).

(3) Fusion of the indirect illumination image and the direct illumination image to obtain the image of the target object. It should be noted that after obtaining at least one indirect illumination image of the target object and at least one direct illumination image of the target object, these multiple images can be regarded as multiple layers, so at least one indirect illumination image of the target object and the At least one direct illumination image of the target object is superimposed to obtain the final image of the target object that can be displayed.

Furthermore, in order to obtain a higher-quality image of the target object, certain processing can be performed on each layer used to synthesize the image of the target object in advance:

(1) Obtain the noise-free component of the material information of the target object. It should be noted that before processing each layer, the noise-free component of the material information of the target object can be obtained from the aforementioned table.

(2) Perform first processing on the indirect illumination image of the target object based on the noise-free component of the material information of the target object to obtain a processed indirect illumination image. The first processing includes demodulation processing, noise reduction processing, and modulation processing. It should be noted that after obtaining the noise-free component of the material information of the target object, at least one indirect illumination image of the target object can be divided by the noise-free component to obtain at least one demodulated indirect illumination image of the target object. After obtaining at least one demodulated indirect illumination image of the target object, the at least one demodulated indirect illumination image of the target object can be denoised to obtain at least one denoised indirect illumination image of the target object. After obtaining at least one denoised indirect illumination image of the target object, the at least one denoised indirect illumination image of the target object can be multiplied by the noise-free component to obtain at least one modulated indirect illumination image of the target object.

Since the size of the at least one first sample pool is smaller than the resolution of the screen, the resolution of the modulated at least one indirect illumination image of the target object is also smaller than the resolution of the screen. In order to achieve better integration of each layer, after obtaining at least one modulated indirect illumination image of the target object, super-resolution processing can also be performed on at least one modulated indirect illumination image of the target object, thereby obtaining the processed image of the target object. At least one indirect illumination image, at this time, the resolution of the at least one indirect illumination image after processing of the target object is equal to the resolution of the screen.

(3) Perform second processing on the direct illumination image of the target object based on the noise-free component of the material information of the target object to obtain a processed direct illumination image. The second processing includes demodulation processing, noise reduction processing, and modulation processing. It should be noted that after obtaining the noise-free component of the material information of the target object, at least one direct illumination image of the target object can be divided by the noise-free component to obtain at least one demodulated direct illumination image of the target object. After obtaining at least one demodulated direct illumination image of the target object, the at least one demodulated direct illumination image of the target object can be denoised to obtain at least one denoised direct illumination image of the target object. After obtaining at least one direct illumination image of the target object after denoising, the at least one direct illumination image after denoising of the target object can be multiplied by the noise-free component to obtain at least one modulated direct illumination image of the target object, that is, At least one processed direct illumination image of the target object.

(4) Fusion of the processed indirect illumination image and the processed direct illumination image to obtain the image of the target object. It should be noted that, after obtaining at least one processed indirect illumination image of the target object and at least one processed direct illumination image of the target object, the at least one processed indirect illumination image of the target object and the processed at least one direct illumination image of the target object can be obtained. of at least one directly illuminated image Overlay to obtain the final image of the target object that can be displayed.

It should be noted that in this embodiment, the indirect illumination image is the aforementioned first illumination image, the direct illumination image is the aforementioned second illumination image, and the processed indirect illumination image is the aforementioned processed first illumination image. Image, the processed direct illumination image is the aforementioned processed second illumination image.

In order to further understand the image rendering method provided by the embodiment of the present application, a specific application example is provided below to further introduce the method. Figure 7 is a schematic diagram of an application example of the image rendering method provided by the embodiment of the present application. As shown in Figure 7, this application example is mainly used to display a virtual auto show for users. The application example includes:

(1) Resample the light path between the objects in the auto show (for example, vehicles, floors, etc.) and the light source to obtain indirect light paths and direct light paths.

(2) Construct the first type of sample pool based on the direct light path. The first type of sample pool contains 1 sample pool. The corresponding rendering effect of this sample pool is direct lighting, and the size of the sample pool is consistent with the size (resolution) of the screen. For example, if the resolution of the screen is 1920*1080, the size of the sample pool is 1920*1080, and the screen is used to display images of the virtual auto show. The second type of sample pool is constructed based on the indirect light path. The second type of sample pool contains n sample pools. The rendering effect corresponding to the first sample pool is specular reflection, the rendering effect corresponding to the second sample pool is specular reflection, and the rendering effect corresponding to the third sample pool is specular reflection. The rendering effect corresponding to the sample pools is diffuse reflection,..., and the rendering effect corresponding to the nth sample pool is specular reflection. Among these n sample pools, the size of each sample pool is 1/4 of the screen size. That is, the size of each sample pool is 480*270.

(3) Render based on the first type of sample pool to obtain a direct illumination image, which is used to present objects directly visible to the camera.

(4) Render based on the second type of sample to obtain an indirect lighting image. This image is used to further present effects such as specular reflection and highlight reflection on the basis of visible objects.

(5) The image obtained in step (3) is noisy and requires noise reduction processing. The noise reduction process mainly includes three steps, as shown in Figure 8 (Figure 8 is a schematic diagram of the noise reduction process provided by the embodiment of the present application): a. Calculate the noise-free component of the material information of the objects in the auto show; b. Divide the image obtained in step (3) by the noise-free component; c. Then perform noise reduction on the divided result, and then multiply the noise-reduced result by the noise-free component to obtain the direct illumination after removing the noise. image.

(6) The image obtained in step (4) is also noisy and can be processed according to the noise reduction process in step (5) to obtain the indirect illumination image after removing the noise.

(7) The resolution of the direct illumination image after removing noise obtained in step (5) is consistent with the resolution of the screen. The resolution of the indirect illumination image after removing noise obtained in step (6) is the resolution of the screen. 1/4, so the indirect illumination image after noise removal can be super-resolved. Finally, the super-resolution indirect illumination image and the direct illumination image after noise removal can be superimposed to obtain the image of the virtual car show and displayed on the screen. .

In addition, the image rendering method provided by the embodiment of the present application can also be compared with the image rendering method provided by the related technology. The comparison results are shown in Figures 9 to 14 (Figure 9 is a schematic diagram of the comparison results provided by the embodiment of the present application. , Figure 10 is another schematic diagram of the comparison results provided by the embodiment of the present application, Figure 11 is another schematic diagram of the comparison results provided by the embodiment of the present application, Figure 12 is another schematic diagram of the comparison results provided by the embodiment of the present application, Figure 13 is another schematic diagram of the comparison results provided by the embodiment of the present application, and Figure 14 is another schematic diagram of the comparison results provided by the embodiment of the present application). The test scene includes a vehicle, a ground and light sources in the environment.

Based on Figure 9, it can be seen that in the image rendered by the related technology, there is no reflection of the ground on the car body. Based on Figure 10, it can be seen that in the image rendered based on the multi-pool technology according to the embodiment of the present application, the car body shows a clear reflection on the ground.

Based on Figure 11, it can be seen that after the relevant technology rendering denoises the image, the details of the car seat are lost. Based on Figure 12, it can be seen that after the embodiment of the present application denoises the image, the details of the car seat can be better preserved.

Based on Figure 13, it can be seen that after the related technology performs super-resolution processing on the image, the resulting image has obvious aliasing. Based on Figure 14, it can be seen that the embodiment of the present application performs differentiated speculation on the effects of different lighting (direct lighting and indirect lighting). Pink, which can significantly reduce aliasing and the resulting image is smooth and clear.

In the embodiment of the present application, after obtaining the original light path between the target object and the light source, the light path between the target object and the light source can be resampled based on the original light path to obtain a new light path. The original light path is traced through ray tracing. It is obtained that the original light path starts from the target object and ends with the light source, and the new light path starts from the light source and ends with the target object. A pool of samples can then be obtained based on the new ray path, indicating the rays hitting the target object from which the new ray path is formed. Finally, the target object can be rendered based on the sample pool to obtain the image of the target object. Based on the foregoing process, it can be seen that the embodiment of the present application provides a new light path resampling method, which is contrary to the direction of the effective light path in ray tracing (with the light source as the starting point and the target object as the end point), and is targeted Consistently resampling the light path can improve the success rate of sampling an effective new light path, that is, it can easily resample an effective new light path between the target object and the light source. In this way, no matter what forms the new light path, Regardless of whether the phenomenon caused by light hitting the target object is direct illumination, diffuse reflection, specular reflection, or specular reflection, a high-quality and sufficient sample pool can be successfully obtained, so that the rendering effect of the target object is high enough. (That is, the rendered image of the target object is of sufficiently high quality).

Furthermore, embodiments of the present application provide a ray tracing technology based on multiple sample pools. The multiple sample pools refer to at least one first sample pool and at least one second sample pool. In a common configuration, two first sample pools and one second sample pool can be constructed. The rendering effects corresponding to the two first sample pools are specular reflection and specular reflection respectively. The rendering effects corresponding to the second sample pool The effect is direct lighting. Then, even if the target object is an object of special material, the image of the target object rendered based on multiple sample pools can present not only direct lighting effects, but also indirect lighting effects such as specular reflection effects and highlight reflection effects. , the resulting image is more realistic and beautiful, which is conducive to improving user experience.

Furthermore, in the embodiment of the present application, the size of the first sample pool is smaller than the resolution of the screen, so the efficiency of obtaining the first sample pool can be improved, and in the process of rendering the target object based on the multi-sample pool technology , can be used with super-resolution processing to reduce the amount of calculation required in the process to a certain extent, which is conducive to accelerating the implementation of ray tracing technology based on multi-sample pools and further improving user experience.

Furthermore, the embodiments of this application provide a new image noise reduction method, which can realize demodulation, noise reduction and modulation of each layer based on the noise-free classification of the material information of the target object, so that each layer can be The layers try to maintain the details of the target object, so that the image of the target object obtained based on the superposition of each layer is more realistic and clearly visible, which is conducive to improving the user's viewing experience and further improving the user experience.

The above is a detailed description of the image rendering method provided by the embodiment of the present application. The image rendering device provided by the embodiment of the present application will be introduced below. Figure 15 is a schematic structural diagram of an image rendering device provided by an embodiment of the present application. As shown in Figure 15, the device includes:

The resampling module 1501 is used to resample the light path between the target object and the light source based on the original light path between the target object and the light source to obtain a new light path. The original light path is obtained through ray tracing. The original light path Taking the target object as the starting point and the light source as the end point, the new light path takes the light source as the starting point and the target object as the end point;

The first acquisition module 1502 is used to acquire a sample pool based on the new light path. The sample pool is used to indicate the light rays directed to the target object. The new light path is formed by the light rays;

The rendering module 1503 is used to render the target object based on the sample pool to obtain an image of the target object.

In a possible implementation, the rendering module 1503 is configured to: render the target object based on the first sample pool to obtain the first lighting image of the target object; render the target object based on the second sample pool to obtain the first illumination image of the target object; Obtain a second illumination image of the target object; fuse the first illumination image and the second illumination image to obtain an image of the target object.

In a possible implementation, the rendering module 1503 is also used to: obtain the noise-free component of the material information of the target object; and perform the first processing on the first illumination image based on the noise-free component to obtain the processed first illumination. Image, the first processing includes demodulation processing, noise reduction processing processing and modulation processing; perform a second processing on the second illumination image based on the noise-free component to obtain a processed second illumination image. The second processing includes demodulation processing, noise reduction processing and modulation processing; rendering module 1503, for The processed first illumination image and the processed second illumination image are fused to obtain an image of the target object.

It should be noted that the information interaction, execution process, etc. between the modules/units of the above-mentioned device are based on the same concept as the method embodiments of the present application, and the technical effects they bring are the same as those of the method embodiments of the present application. The specific content can be Refer to the description in the method embodiments shown above in the embodiments of the present application, which will not be described again here.

Embodiments of the present application also relate to a circuit system, which circuit system includes a processing circuit configured to perform various steps in the embodiment shown in FIG. 4 .

Embodiments of the present application also relate to a chip system, which includes a processor for calling a computer program or computer instructions stored in a memory, so that the processor executes various steps in the embodiment shown in FIG. 4 .

Embodiments of the present application also relate to a computer storage medium. The computer-readable storage medium stores a program for performing signal processing. When the program is run on a computer, it causes the computer to execute each of the steps in the embodiment shown in Figure 4. step.

Embodiments of the present application also relate to a computer program product that stores instructions that, when executed by a computer, cause the computer to perform various steps in the embodiment shown in FIG. 4 .

Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, devices and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be described again here.

In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components may be combined or can be integrated into another system, or some features can be ignored, or not implemented. On the other hand, the coupling or direct coupling or communication connection between each other shown or discussed may be through some interfaces, and the indirect coupling or communication connection of the devices or units may be in electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or they may be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present application can be integrated into one processing unit, each unit can exist physically alone, or two or more units can be integrated into one unit. The above integrated units can be implemented in the form of hardware or software functional units.

If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it may be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or contributes to the existing technology, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium , including several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of this application. The aforementioned storage media include: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk and other media that can store program code. .

Claims

An image rendering method, characterized in that the method includes:

Based on the original light path between the target object and the light source, the light path between the target object and the light source is resampled to obtain a new light path. The original light path is obtained through ray tracing. The original light path The path takes the target object as the starting point and the light source as the end point, and the new light path takes the light source as the starting point and the target object as the end point;

Based on the new light path, obtain a sample pool, the sample pool is used to indicate the light rays directed to the target object, and the new light path is formed by the light rays;

Based on the sample pool, the target object is rendered to obtain an image of the target object.
The method according to claim 1, characterized in that the new light path includes a first light path and a second light path, the first light path starts from the light source and ends with the target object, And passing through other objects, the second light path takes the light source as the starting point and the target object as the ending point, and does not pass through other objects;

The sample pool includes a first sample pool and a second sample pool. The first sample pool is used to indicate the first light ray directed to the target object. The first light ray path is formed by the first light ray. The two sample cells are used to indicate the second light rays emitted to the target object, and the second light ray path is formed by the second light rays.
The method of claim 2, wherein the phenomenon caused by the first light ray hitting the target object includes at least one of the following: specular reflection or specular reflection.
The method according to claim 2 or 3, characterized in that the number of first rays indicated by the first sample pool is less than the number of pixels displayed on the screen, and the number of second rays indicated by the second sample pool is smaller than the number of pixels displayed on the screen. The number of rays is equal to the number of pixels displayed on the screen, which is used to display the image of the target object.
The method according to any one of claims 2 to 4, wherein rendering the target object based on the sample pool to obtain an image of the target object includes:

Render the target object based on the first sample pool to obtain a first illumination image of the target object;

Render the target object based on the second sample pool to obtain a second illumination image of the target object;

The first illumination image and the second illumination image are fused to obtain an image of the target object.
The method according to claim 5, characterized in that before fusing the first illumination image and the second illumination image to obtain the image of the target object, the method further includes:

Obtain the noise-free component of the material information of the target object;

Perform first processing on the first illumination image based on the noise-free component to obtain a processed first illumination image, where the first processing includes demodulation processing, noise reduction processing and modulation processing;

Perform the second processing on the second illumination image based on the noise-free component to obtain a processed second illumination image, where the second processing includes demodulation processing, noise reduction processing and modulation processing;

The fusion of the first illumination image and the second illumination image to obtain the image of the target object includes:

The processed first illumination image and the processed second illumination image are fused to obtain an image of the target object.
The method according to claim 6, wherein the first processing further includes super-resolution processing, and the super-resolution processing is used to make the resolution of the processed first illumination image equal to the resolution of the screen. .
An image rendering device, characterized in that the device includes:

A resampling module, configured to resample the light path between the target object and the light source based on the original light path between the target object and the light source to obtain a new light path. The original light path is traced through ray tracing. It is obtained that the original light path takes the target object as the starting point and the light source as the end point, and the new light path takes the light source as the starting point and the target object as the end point;

A first acquisition module, configured to acquire a sample pool based on the new light path, where the sample pool is used to indicate the light rays directed to the target object, and the new light path is formed by the light rays;

A rendering module, configured to render the target object based on the sample pool to obtain an image of the target object.
The device according to claim 8, wherein the new light path includes a first light path and a second light path, the first light path starting from the light source and ending with the target object, And passing through other objects, the second light path takes the light source as the starting point and the target object as the ending point, and does not pass through other objects;

The sample pool includes a first sample pool and a second sample pool. The first sample pool is used to indicate the first light ray directed to the target object. A first light path is formed by the first light ray, a second sample cell is used to indicate a second light ray directed to the target object, and the second light ray path is formed by the second light ray.
The device according to claim 9, characterized in that the phenomenon produced by the first light ray striking the target object is specular reflection or specular reflection.
The device according to claim 9 or 10, characterized in that the number of first rays indicated by the first sample pool is less than the number of pixels displayed on the screen, and the number of second rays indicated by the second sample pool is smaller than the number of pixels displayed on the screen. The number of rays is equal to the number of pixels displayed on the screen, which is used to display the image of the target object.
The device according to any one of claims 9 to 11, characterized in that the rendering module is used for:

Render the target object based on the first sample pool to obtain a first illumination image of the target object;

Render the target object based on the second sample pool to obtain a second illumination image of the target object;

The first illumination image and the second illumination image are fused to obtain an image of the target object.
The device according to claim 12, characterized in that the rendering module is also used to:

Obtain the noise-free component of the material information of the target object;

Perform first processing on the first illumination image based on the noise-free component to obtain a processed first illumination image, where the first processing includes demodulation processing, noise reduction processing and modulation processing;

Perform the second processing on the second illumination image based on the noise-free component to obtain a processed second illumination image, where the second processing includes demodulation processing, noise reduction processing and modulation processing;

The rendering module is used to fuse the processed first illumination image and the processed second illumination image to obtain an image of the target object.
The device according to claim 13, wherein the first processing further includes super-resolution processing, and the super-resolution processing is used to make the resolution of the processed first illumination image equal to the resolution of the screen.
An electronic device, characterized in that the device includes a memory and a processor; the memory stores code, the processor is configured to execute the code, and when the code is executed, the electronic device executes The method according to any one of claims 1 to 7.
A circuit system, characterized in that the circuit system includes a processing circuit configured to perform the method according to any one of claims 1 to 7.
A chip system, characterized in that the chip system includes a processor for calling a computer program or computer instructions stored in a memory, so that the processor executes the method according to any one of claims 1 to 7.
A computer storage medium, characterized in that the computer storage medium stores one or more instructions, which when executed by one or more computers cause the one or more computers to implement any of claims 1 to 7. The method described in 1.
A computer program product, characterized in that the computer program product stores instructions, which when executed by a computer, cause the computer to implement the method described in any one of claims 1 to 7.