CN116958362A - Image rendering method, device, equipment and storage medium - Google Patents
Image rendering method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN116958362A CN116958362A CN202310208155.3A CN202310208155A CN116958362A CN 116958362 A CN116958362 A CN 116958362A CN 202310208155 A CN202310208155 A CN 202310208155A CN 116958362 A CN116958362 A CN 116958362A
- Authority
- CN
- China
- Prior art keywords
- target object
- surface point
- spatial sampling
- light field
- color
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000009877 rendering Methods 0.000 title claims abstract description 168
- 238000000034 method Methods 0.000 title claims abstract description 130
- 238000005070 sampling Methods 0.000 claims abstract description 151
- 239000000463 material Substances 0.000 claims abstract description 103
- 210000005036 nerve Anatomy 0.000 claims abstract description 23
- 230000001537 neural effect Effects 0.000 claims description 99
- 238000012549 training Methods 0.000 claims description 40
- 239000003086 colorant Substances 0.000 claims description 34
- 230000006870 function Effects 0.000 claims description 23
- 230000015654 memory Effects 0.000 claims description 22
- 230000005855 radiation Effects 0.000 claims description 20
- 238000013528 artificial neural network Methods 0.000 claims description 16
- 230000003287 optical effect Effects 0.000 claims description 8
- 238000002360 preparation method Methods 0.000 claims description 7
- 238000004590 computer program Methods 0.000 claims description 6
- 238000005286 illumination Methods 0.000 abstract description 33
- 230000003993 interaction Effects 0.000 abstract description 25
- 230000015572 biosynthetic process Effects 0.000 abstract description 14
- 238000003786 synthesis reaction Methods 0.000 abstract description 14
- 230000001795 light effect Effects 0.000 abstract 1
- 230000000875 corresponding effect Effects 0.000 description 30
- 238000010586 diagram Methods 0.000 description 27
- 230000008859 change Effects 0.000 description 11
- 238000013473 artificial intelligence Methods 0.000 description 9
- 230000008569 process Effects 0.000 description 8
- 238000012545 processing Methods 0.000 description 7
- 230000003068 static effect Effects 0.000 description 7
- 230000000694 effects Effects 0.000 description 6
- 238000004891 communication Methods 0.000 description 5
- 238000005516 engineering process Methods 0.000 description 5
- 238000013507 mapping Methods 0.000 description 5
- 230000001360 synchronised effect Effects 0.000 description 5
- 238000005315 distribution function Methods 0.000 description 4
- 230000009471 action Effects 0.000 description 3
- 230000000903 blocking effect Effects 0.000 description 3
- 238000012512 characterization method Methods 0.000 description 3
- 238000009826 distribution Methods 0.000 description 3
- 230000010354 integration Effects 0.000 description 3
- 238000013459 approach Methods 0.000 description 2
- 230000003190 augmentative effect Effects 0.000 description 2
- 230000002457 bidirectional effect Effects 0.000 description 2
- 238000004364 calculation method Methods 0.000 description 2
- 238000013527 convolutional neural network Methods 0.000 description 2
- 238000012937 correction Methods 0.000 description 2
- 230000002596 correlated effect Effects 0.000 description 2
- 238000000605 extraction Methods 0.000 description 2
- 238000003780 insertion Methods 0.000 description 2
- 230000037431 insertion Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000001575 pathological effect Effects 0.000 description 2
- 230000000191 radiation effect Effects 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- 230000009466 transformation Effects 0.000 description 2
- 240000004282 Grewia occidentalis Species 0.000 description 1
- 230000003044 adaptive effect Effects 0.000 description 1
- 229940060587 alpha e Drugs 0.000 description 1
- 230000006399 behavior Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000002146 bilateral effect Effects 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 239000004568 cement Substances 0.000 description 1
- 239000002131 composite material Substances 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 230000003247 decreasing effect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000007613 environmental effect Effects 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 230000004927 fusion Effects 0.000 description 1
- 238000009499 grossing Methods 0.000 description 1
- 238000002620 method output Methods 0.000 description 1
- 238000010606 normalization Methods 0.000 description 1
- 230000002441 reversible effect Effects 0.000 description 1
- 230000006403 short-term memory Effects 0.000 description 1
- 238000004088 simulation Methods 0.000 description 1
- 238000000844 transformation Methods 0.000 description 1
- 230000001131 transforming effect Effects 0.000 description 1
- 238000013519 translation Methods 0.000 description 1
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T15/00—3D [Three Dimensional] image rendering
- G06T15/005—General purpose rendering architectures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Computer Graphics (AREA)
- Image Generation (AREA)
Abstract
Embodiments of the present disclosure provide an image rendering method, apparatus, device, and computer-readable storage medium. According to the method, the rendering color of the object surface point is decomposed into diffuse reflection color and specular reflection color generated by interaction of the material of the object and ambient light, the diffuse reflection color is determined by decoupling the material attribute of the object, and the light effect result of each space sampling point around the object in the nerve environment light field on the object surface point is generated by utilizing the decoupled material attribute, so that the rendering color of the object surface point in the nerve environment light field is determined. According to the method, the interaction and shielding relation between the light rays and the material of the object in the real scene can be expressed by considering the positions of a plurality of spatial sampling points around the object in the nerve environment light field and the viewpoint positions and directions of observers, and in addition, the independent control of the target object or the nerve environment light field can be realized for various task scenes such as new view synthesis or re-illumination.
Description
Technical Field
The present disclosure relates to the field of computer vision, and more particularly, to an image rendering method, apparatus, device, and storage medium.
Background
The expression mode of the ambient light field is an important ring for improving the sense of reality of the rendered image, and a series of works currently exist for providing natural sense of reality for new view synthesis and re-illumination algorithms and the like. For example, for view synthesis based on images, several images of known shooting viewpoints are required to be used as input, and three-dimensional objects or scenes shot by the images are expressed in terms of geometric, appearance, illumination and other properties, so that images of other non-shooting viewpoints can be synthesized, and finally, a drawing result with high reality is obtained, wherein how to express an environmental light field of the three-dimensional scene by using information of the input images is a key problem of new view synthesis.
However, in the real world, the observed object surface color is actually the result of the interaction between its surrounding ambient illumination and the material of the object itself, and it is obviously a difficult task to accurately express the ambient light field of the scene in which the object is located to reconstruct the ambient illumination of the real world, since the ambient illumination around the object varies with the position and direction of the viewpoint.
Therefore, there is a need for an efficient method that allows reconstructing the real world illumination, thereby obtaining image rendering results with high realism.
Disclosure of Invention
In order to solve the above-mentioned problem, the present disclosure obtains detailed real ambient illumination by modeling the environment around the object in five dimensions (5D), regarding each spatial sampling point in the environment around the object as a light source emitter, while considering the position of the light source and the viewpoint position and direction of the observer.
Embodiments of the present disclosure provide an image rendering method, apparatus, device, and computer-readable storage medium.
An embodiment of the present disclosure provides an image rendering method, including: acquiring a target object to be rendered and determining the material property of each surface point of the target object, wherein the material property comprises diffuse reflection color; acquiring a nerve environment light field for rendering the target object and the position of a viewpoint, and placing the target object in the nerve environment light field; in the neural environment light field, for each surface point of the target object, determining specular reflection colors generated at the surface point by a plurality of spatial sampling points in the neural environment light field, respectively, based on material properties of the surface point, and determining specular reflection colors of the surface point based on the specular reflection colors generated at the surface point by the plurality of spatial sampling points, respectively, wherein the plurality of spatial sampling points are associated with reflection directions corresponding to observation directions from the viewpoint to the surface point; and determining a rendering color of each surface point of the target object in the neural ambient light field based on the diffuse and specular reflection colors of each surface point of the target object.
Embodiments of the present disclosure provide an image rendering apparatus including: a material property acquisition module configured to acquire a target object to be rendered and determine a material property of each surface point of the target object, the material property including a diffuse reflection color; an ambient light field preparation module configured to acquire a neural ambient light field for rendering the target object and a position of a viewpoint, and place the target object in the neural ambient light field; a reflection color determination module configured to determine, for each surface point of the target object, a specular reflection color generated at the surface point by a plurality of spatial sampling points in the neuro-environmental light field, respectively, based on a material property of the surface point, and determine a specular reflection color of the surface point based on the specular reflection color generated at the surface point by the plurality of spatial sampling points, respectively, wherein the plurality of spatial sampling points are associated with a reflection direction corresponding to an observation direction from the viewpoint to the surface point; and a rendering color determination module configured to determine a rendering color of each surface point of the target object in the neural ambient light field based on the diffuse reflection color and the specular reflection color of each surface point of the target object.
Embodiments of the present disclosure provide an image rendering apparatus including: one or more processors; and one or more memories, wherein the one or more memories have stored therein a computer executable program which, when executed by the processor, performs the image rendering method as described above.
Embodiments of the present disclosure provide a computer readable storage medium having stored thereon computer executable instructions which, when executed by a processor, are for implementing an image rendering method as described above.
Embodiments of the present disclosure provide a computer program product or computer program comprising computer instructions stored in a computer readable storage medium. A processor of a computer device reads the computer instructions from a computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs an image rendering method according to an embodiment of the present disclosure.
Compared with the traditional image rendering method, the method provided by the embodiment of the disclosure can consider that the ambient light around the object changes along with the position and the direction of the viewpoint, and each space sampling point around the object is regarded as an independent light source emitter so as to model the blocking relation between the directional light source and the static three-dimensional environment, thereby realizing better reconstruction of the ambient light field.
According to the method provided by the embodiment of the disclosure, the rendering color of the object surface point is decomposed into the diffuse reflection color and the specular reflection color generated by interaction between the material of the object and the ambient light, the diffuse reflection color is respectively determined by decoupling the material attribute of the object, and the light action result of each spatial sampling point around the object in the neural ambient light field on the object surface point is generated, so that the rendering color of the object surface point in the neural ambient light field is determined. The method of the embodiment of the disclosure can simultaneously consider the positions of a plurality of spatial sampling points around the object in the nerve environment light field and the viewpoint position and direction of the observer to represent the interaction and shielding relation between the light rays and the material of the object in the real scene. Furthermore, in embodiments of the present disclosure, by decoupling the material properties of the target object from the neuro-ambient light field, separate control of the target object or neuro-ambient light field may be achieved for achieving true natural image rendering in a variety of task scenarios, such as new view synthesis or re-illumination.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings that are used in the description of the embodiments will be briefly described below. It should be apparent that the drawings in the following description are only some exemplary embodiments of the present disclosure, and that other drawings may be obtained from these drawings by those of ordinary skill in the art without undue effort.
FIG. 1A is a schematic view of a scene showing obtaining a rendered image based on an object to be rendered according to an embodiment of the present disclosure;
FIG. 1B is a network diagram illustrating image rendering based on a local ambient light map according to an embodiment of the present disclosure;
FIG. 2A is a flowchart illustrating an image rendering method according to an embodiment of the present disclosure;
FIG. 2B is a schematic flow diagram illustrating an image rendering method according to an embodiment of the present disclosure;
FIG. 3 is a schematic diagram illustrating a texture estimation network according to an embodiment of the present disclosure;
FIG. 4 is a schematic diagram illustrating interactions of spatially sampled points with object surface points within a neural ambient light field, according to an embodiment of the present disclosure;
FIG. 5 is a schematic flow diagram illustrating generation of specular reflection colors at object surface points based on at least one spatial sampling point in a neural ambient light field, according to an embodiment of the present disclosure;
FIG. 6A is a schematic flow diagram illustrating training of an image rendering network according to an embodiment of the present disclosure;
FIG. 6B is a schematic diagram illustrating training of an image rendering network according to an embodiment of the present disclosure;
fig. 7 is a schematic diagram illustrating an image rendering apparatus according to an embodiment of the present disclosure;
fig. 8 shows a schematic diagram of an image rendering device according to an embodiment of the present disclosure;
FIG. 9 illustrates a schematic diagram of an architecture of an exemplary computing device, according to an embodiment of the present disclosure; and
fig. 10 shows a schematic diagram of a storage medium according to an embodiment of the present disclosure.
Detailed Description
In order to make the objects, technical solutions and advantages of the present disclosure more apparent, exemplary embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. It should be apparent that the described embodiments are only some of the embodiments of the present disclosure and not all of the embodiments of the present disclosure, and that the present disclosure is not limited by the example embodiments described herein.
In the present specification and drawings, steps and elements having substantially the same or similar are denoted by the same or similar reference numerals, and repeated descriptions of the steps and elements will be omitted. Meanwhile, in the description of the present disclosure, the terms "first," "second," and the like are used merely to distinguish the descriptions, and are not to be construed as indicating or implying relative importance or order.
Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs. The terminology used herein is for the purpose of describing embodiments of the invention only and is not intended to be limiting of the invention.
For purposes of describing the present disclosure, the following presents concepts related to the present disclosure.
The image rendering method of the present disclosure may be artificial intelligence (Artificial intelligence, AI) based. Artificial intelligence is the theory, method, technique and application system that uses a digital computer or a digital computer-controlled machine to simulate, extend and expand human intelligence, sense the environment, acquire knowledge and use the knowledge to obtain optimal results. In other words, artificial intelligence is an integrated technology of computer science that attempts to understand the essence of intelligence and to produce a new intelligent machine that can react in a similar way to human intelligence. For example, for an artificial intelligence based image rendering method, it is possible to generate various views of a target object in a neural environment light field (Neural Ambient Illumination, neAI) in a manner similar to how a human being would obtain an image representation of the target object in the environment under various perspectives by the naked eye. Artificial intelligence by researching the design principles and implementation methods of various intelligent machines, the image rendering method disclosed by the invention has the functions of accurately extracting a nerve environment light field from a multi-viewpoint image in real time and applying the nerve environment light field to any target object represented by a grid so as to generate a real and natural new viewpoint image or render a re-illumination image for the target object.
The image rendering method of the present disclosure is also based on Computer Vision (CV) technology. The computer vision is a science for researching how to make a machine "see", and more specifically, a camera and a computer are used to replace human eyes to identify and measure targets, and the like, and further, graphic processing is performed, so that the computer is processed into images which are more suitable for human eyes to observe or transmit to an instrument to detect. As a scientific discipline, computer vision research-related theory and technology has attempted to build artificial intelligence systems that can acquire information from images or multidimensional data. Computer vision techniques typically include image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content/behavior recognition, three-dimensional object reconstruction, 3D techniques, virtual reality, augmented reality, synchronous positioning, and map construction, among others, as well as common biometric recognition techniques such as face recognition, fingerprint recognition, and others. In the image rendering method of the present disclosure, a neural ambient light field therein may be reconstructed based on a multi-viewpoint image, and then applied to a current image to obtain a new viewpoint image thereof or applied to other images to obtain re-illuminated images of the other images.
The image rendering method of the present disclosure may be based on an ambient light field. The expression mode of the environment light field is an important ring for improving the sense of reality of the rendered image. Recently, along with the proposal of the expression mode of the nerve radiation field, a great amount of research work based on the expression method further optimizes and expands the image rendering method, and good results are obtained in the aspects of accuracy, high efficiency and the like. Thus, in particular, the image rendering method of the present disclosure may be based on a neural radiation field (Neural Radiance Field, neRF). NeRF has attracted considerable attention in the field of computer vision as a novel field of view synthesis technology with implicit scene representation. In its basic form, the NeRF model represents a three-dimensional scene as a radiation field approximated by a neural network. The radiation field describes the color and volume density of each point in the scene and each viewing direction. The input to NeRF is a five-dimensional vector comprising the position of the (object) spatial sampling point and the (camera) viewing direction (viewpoint direction), and its output comprises the volume density (which represents the probability that a ray of light will terminate after reaching the spatial sampling point, which can be understood as transparency) σ and the spatial sampling point color c= (r, g, b) of the object based on the viewing angle. The color and bulk density of the path spatial sampling point are calculated for each beam of light emitted from the camera, and then voxel Rendering (Volume Rendering) is performed based on the obtained color and bulk density of the spatial sampling point to obtain a prediction of the pixel value of the spatial sampling point. The NeRF-implemented function is actually to implement a mapping from "spatial sample point locations+observation angles" to "spatial sample point colors+bulk densities". As a novel view synthesis and three-dimensional reconstruction method, neRF realizes view synthesis of high degree of realism of complex scenes, and has wide application in the fields of robots, urban maps, autonomous navigation, virtual reality/augmented reality and the like.
In view of the foregoing, embodiments of the present disclosure relate to techniques of artificial intelligence, computer vision, neural radiation fields, etc., and are further described below with reference to the accompanying drawings.
Fig. 1A is a schematic view illustrating obtaining a rendered image based on an object to be rendered according to an embodiment of the present disclosure.
As shown in fig. 1A, a user may send an object to be rendered (e.g., geometric data of a target object of the present disclosure, or an image including the target object to be rendered, etc.) to a server through a user terminal for the server to perform an image rendering process based on the object to be rendered. The server may then return the generated rendered image to the user terminal over the network for display to the user. Alternatively, the user terminal may specifically include a smart phone, a tablet computer, a laptop portable computer, a desktop computer, a vehicle-mounted terminal, a wearable device, etc., but is not limited thereto, and for example, the user terminal may also be an image capturing device, etc., so as to directly send the captured image to the processor end for image processing. The user terminal may also be a client that installs a browser or various applications, including system applications and third party applications. Alternatively, the network may be an internet of things (Internet of Things) based on the internet and/or a telecommunication network, which may be a wired network or a wireless network, for example, it may be a Local Area Network (LAN), a Metropolitan Area Network (MAN), a Wide Area Network (WAN), a cellular data communication network, or an electronic network capable of implementing an information exchange function, where the user terminal and the server may be directly or indirectly connected through wired or wireless communication, and the present application is not limited herein. The server may be an independent physical server, a server cluster or a distributed system formed by a plurality of physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content Delivery Networks (CDNs), basic cloud computing services such as big data and artificial intelligence platforms, and the like.
In the image rendering process of an object to be rendered by a server, in order to generate a natural and lifelike image, an ambient light field needs to be accurately expressed so as to improve the realism of the rendered image. Considering that the observed object surface color is actually the result of interaction between ambient light and object materials, the problem of inverse rendering of ambient light field and object materials estimated directly from images in the past is considered to be pathological, whereas existing image rendering techniques generally use ambient light maps to represent ambient light field, so as to solve the pathological problem of inverse rendering. However, these image rendering techniques tend to assume that the light sources in the environment are at infinity, when the incident light on the object surface is only related to the viewpoint direction. In the real world, the ambient light around the object will vary with the position and direction of the viewpoint, which is obviously not taken into account by such simplification of the ambient light map. In addition, there are methods for image rendering using global illumination (global illumination) technology, which simulate the interaction of rays and problem materials in a real scene by means of ray tracing and path tracing. However, this approach typically requires modeling of the material and geometry of all objects surrounding the target object, and also requires accurate estimation of the active light source, which can introduce significant computational complexity.
There is also an image rendering method that defines a local ambient light map for each vertex of the target object surface, so as to consider occlusion and interaction of illumination. Fig. 1B is a network diagram illustrating image rendering based on a local ambient light map according to an embodiment of the present disclosure. As shown in fig. 1B, for any vertex x, L of an object, which is the reflection direction at that vertex when viewed from the observation direction d, the formula in fig. 1B represents solving the color L of the outgoing light of the spatial coordinate x when viewed from the direction d o (d, x) which is equal to all incident light L on a hemisphere Ω defined by the normal to the spatial coordinate x i Integral value co-acting with the material parameter f (.). In practice, for each surface point of the object, the BRDF (Bidirectional Reflectance DistributionFunction, bi-directional reflectance distribution function) coefficients { b, r, m } are estimated by feeding into the texture network (e.g., via a multi-layer perceptron (MLP) neural network). And obtaining positional correspondence of object surface points through a neural incident light field (NeILF) networkIs a graph of the ambient light. Finally, obtaining the final emergent light color L through a rendering formula of a classical BRDF model o (d, x). In the training process of the model, the network parameters are iteratively optimized by carrying out gradient feedback on the error between the estimated emergent light color and the true value.
However, in the image rendering method, three-dimensional modeling of the ambient light field around the object is not considered, and the light field information observed at different positions cannot be fused. In addition, the method outputs an environment light map for each vertex of the object, the degree of freedom of the environment light map is too high, the environment light map is easy to converge to a local optimal value, decoupling ambiguity of the material of the object and the light field is caused, and the estimation of the light field by the method tends to be fuzzy due to the fact that the angle input observed by each vertex is little and sufficient information supervision is lacked.
The present disclosure is based on the present disclosure, and provides an image rendering method, which obtains real-detail ambient illumination by modeling an ambient environment of an object in five dimensions, regarding each spatial sampling point in the ambient environment around the object as a light source emitter, and considering a position of a light source and a viewpoint position and direction of an observer.
Compared with the traditional image rendering method, the method provided by the embodiment of the disclosure can consider that the ambient light around the object changes along with the position and the direction of the viewpoint, and each space sampling point around the object is regarded as an independent light source emitter so as to model the blocking relation between the directional light source and the static three-dimensional environment, thereby realizing better reconstruction of the ambient light field.
According to the method provided by the embodiment of the disclosure, the rendering color of the object surface point is decomposed into the diffuse reflection color and the specular reflection color generated by interaction between the material of the object and the ambient light, the diffuse reflection color is respectively determined by decoupling the material attribute of the object, and the light action result of each spatial sampling point around the object in the neural ambient light field on the object surface point is generated, so that the rendering color of the object surface point in the neural ambient light field is determined. The method of the embodiment of the disclosure can simultaneously consider the positions of a plurality of spatial sampling points around the object in the nerve environment light field and the viewpoint position and direction of the observer to represent the interaction and shielding relation between the light rays and the material of the object in the real scene. Furthermore, in embodiments of the present disclosure, by decoupling the material properties of the target object from the neuro-ambient light field, separate control of the target object or neuro-ambient light field may be achieved for achieving true natural image rendering in a variety of task scenarios, such as new view synthesis or re-illumination.
Fig. 2A is a flowchart illustrating an image rendering method 200 according to an embodiment of the present disclosure. Fig. 2B is a schematic flow diagram illustrating an image rendering method according to an embodiment of the present disclosure.
As shown in fig. 2B, the image rendering method of the present disclosure may take a target object to be rendered as an input and take a rendered image for illumination interaction using a neural environment light field as an output, where specifically, the target object to be rendered may be geometric information of the target object estimated in advance using other methods, which may include surface points of the target object, the surface points may be represented in a grid form, and the rendered image output for the target object may be a new viewpoint image generated according to an environment light field where the target object is originally located, or may be an image obtained by re-illuminating the target object using other environment light fields, which may depend on the neural environment light field adopted in the rendering process of the target object, and the present disclosure is not limited in particular.
According to an embodiment of the present disclosure, the image rendering method may include: determining the material property of each surface point of the target object through a material estimation network; generating the neural environment light field through a light field reconstruction network; and determining, by an image rendering network, a rendering color of each surface point of the target object in the neural ambient light field. As shown in fig. 2B, for an input target object, a final rendered image may be obtained through a texture estimation network, a light field reconstruction network, and an image rendering network, respectively, where texture properties of the target object may be estimated in the texture estimation network, then a neural environment light field for rendering the target object may be obtained in the light field reconstruction network, and finally the target object located in the neural environment light field at a specific viewpoint may be rendered through the image rendering network, and specific operations in these networks will be described in detail below with reference to steps 201-204 and fig. 2B-5.
In step 201, a target object to be rendered may be acquired and a texture attribute for each surface point of the target object may be determined, which may include a diffuse reflection color.
As described above, geometric information of a target object to be rendered, which may be represented in a grid form, may be first acquired, wherein a plurality of surface points of the target object may be included, and a specific rendering color thereof may be determined for each of these surface points in embodiments of the present disclosure, and then the target object may be rendered based on the determined rendering color of each surface point, thereby generating a final rendering image.
According to an embodiment of the present disclosure, acquiring a target object to be rendered may include: geometric information of a target object to be rendered is acquired, wherein the geometric information of the target object can comprise a plurality of surface points of the target object. In particular, the acquisition of the geometric information of the target object to be rendered may be estimated in advance by various geometric extraction methods, and the present disclosure is not limited to a specific method for acquiring the geometric information of the target object, and any method that can achieve the above object may be applied to the geometric information acquisition operation of the present disclosure.
In embodiments of the present disclosure, for each surface point of the target object (or, for any pixel of the target object surface), its rendering color may be considered as a result of the interaction of the diffuse reflection color of the object surface independent of the direction of observation and the specular reflection color produced by the interaction of the object surface material and ambient light at that surface point.
According to an embodiment of the present disclosure, determining the material properties of each surface point of the target object may include: a material property of each of the plurality of surface points of the target object is determined by a trained multi-layer perceptron neural network.
Thus, alternatively, embodiments of the present disclosure may determine the material properties of the target object based on diffuse and specular reflections produced in the object and ambient light. In particular, considering that specular reflection and diffuse reflection of the surface of an object both follow the law of reflection, there is a certain distinction between the two: specular reflection occurs only in the case of light striking objects with very smooth surfaces (such as water surface mirrors, etc.), and such reflection can only be seen at certain angles, but cannot be seen when the viewing angle changes, while diffuse reflection occurs in the case of light striking objects with rough surfaces, such as book pages, cement floors, wooden seating, etc., and such reflection can be seen at various angles, and embodiments of the present disclosure can divide the material properties of the target object based on diffuse reflection and specular reflection at the object surface. For example, diffuse reflectance color may be considered alone as the object's own properties, independent of interactions between the object and ambient light, whereas for factors affecting the specular reflectance effect of ambient light on the object surface, it may be included in the material properties to take into account the specular reflectance color of a particular ray in ambient light on the object surface when rendering the object in a subsequent neural ambient light field. Alternatively, the roughness of the target object surface may be considered as a material property of the target object, which may affect the degree of blurring of the reflection, e.g., a smoother object whose roughness is lower, the ambient light reflection is clearer, and a rougher object whose roughness is higher, the ambient light reflection is more blurred, that is, for each surface point of the target object, the roughness property of the target object will affect the amount of light in the neural ambient light field that will produce specular reflection for that surface point, e.g., for each surface point on the object whose roughness is higher, it may receive specular reflection thereon from more light in the light field environment (i.e., may correspond to a larger specular reflection lobe).
According to an embodiment of the present disclosure, the material properties may include at least one of diffuse reflectance color, tonal modulation, and roughness. As an example, in embodiments of the present disclosure, diffuse reflection color may be considered as the only factor affecting diffuse reflection of the object surface and as the own material properties of the target object, while for specular reflection of the target object surface, material properties including hue modulation and roughness may be considered. Of course, it should be understood that the above-described material properties affecting diffuse and specular reflection of an object surface are given in embodiments of the present disclosure by way of example only and are described below as examples, and that other material properties that may affect diffuse and specular reflection of an object surface may likewise be included in the image rendering methods of the present disclosure, which are not limiting to the present disclosure.
Fig. 3 is a schematic diagram illustrating a material estimation network according to an embodiment of the present disclosure. As shown in fig. 3, for each surface point x of the target object s The corresponding Diffuse reflection Color (Diffuse Color) C can be determined through the texture estimation network d Tone modulation (Tint) α and Roughness (Roughness) ρ.
Alternatively, the texture properties corresponding to their pairs may be determined from object surface points by a trained network structure such as a multi-layer perceptron neural network. It should be understood that the embodiments of the present disclosure are not limited to the specific structure of the texture estimation network, and other network structures that can achieve the same texture estimation purpose may be equally applicable to the image rendering method of the present disclosure.
As described above, the material properties of the target object may include the diffuse reflection color C d The modulation α of the hue and the roughness ρ. Wherein, optionally, the color C is diffusely reflected d Can be used to effect a change in the color of the object, the change in the value of the attribute of which does not affect the specular color. In addition, the tone modulation α may be used to enhance or attenuate the intensity of the specular reflection or to change the tone of the specular reflection, while the roughness ρ may affect the degree of blurring of the specular reflection, e.g., smoother objects having lower roughness and therefore sharper specular reflection, while rougher objects having higher roughness and thus more blurring of the mirrorSurface reflection. Hereinafter, the interaction between these material properties and the neural ambient light field defined in the present disclosure will be described in detail based on these material properties.
In step 202, a neuro-ambient light field for rendering the target object and a location of a viewpoint may be acquired and the target object placed in the neuro-ambient light field.
As described above, in embodiments of the present disclosure, the neural ambient light field used to render the target object may include the original neural ambient light field in which the target object is located, as well as other neural ambient light fields, thereby generating a new viewpoint image or re-illumination image of the target object. Thus, the acquisition of the neural ambient light field for rendering the target object in the present disclosure may include acquiring an original neural ambient light field in which the target object is located, or may include acquiring other neural ambient light fields for rendering the target object, which is not limited by the present disclosure, since the acquisition operation of the neural ambient light field is similar for different target objects (or, including multi-viewpoint images of different target objects), which will be described in detail below with reference to fig. 6A and 6B for ease of understanding.
According to embodiments of the present disclosure, the neural ambient light field may be comprised of spatial sampling points, where each spatial sampling point may have known optical radiation properties, which may include bulk density and color. Alternatively, the neuro-ambient light field of the present disclosure may be comprised of a plurality of voxels in the light field environment, wherein each voxel may have known optical radiation properties and may produce optical radiation effects on a surface point of the target object. Alternatively, the spatial sampling points in the present disclosure may be the same as the voxels here, or may be the result of sampling all voxels in the light field environment, which may be determined according to actual computational demand, which is not limiting of the present disclosure.
Fig. 4 is a schematic diagram illustrating interactions of spatially sampled points with object surface points within a neural ambient light field, according to an embodiment of the present disclosure. As shown in fig. 4, based on the determined neuro-environmental light field and the viewpoint position, an observation direction from the viewpoint to each surface point of the target object and a reflection direction corresponding thereto may be determined, wherein the reflection direction may be determined based on the observation direction and a normal direction of the surface point. Optionally, a series of spatial sampling points associated with the reflection direction will produce specular reflection at the surface point and generate a corresponding specular reflection color. Note that the "series of spatial sampling points associated with the reflection direction" described herein is different from the series of spatial sampling points in the reflection direction, because the roughness of the object surface will affect the reflected lobe size at the object surface point, so that the spatial sampling points around the spatial sampling points in the reflection direction may also produce specular reflection at the surface point.
In step 203, in the neural environment light field, for each surface point of the target object, a specular reflection color generated at the surface point by a plurality of spatial sampling points in the neural environment light field, respectively, may be determined based on a material property of the surface point, and the specular reflection color of the surface point may be determined based on the specular reflection color generated at the surface point by the plurality of spatial sampling points, respectively, wherein the plurality of spatial sampling points are associated with a reflection direction corresponding to an observation direction from the viewpoint to the surface point.
As described above, in embodiments of the present disclosure, the rendered color of the object surface point may be represented as diffuse reflection color C d And specular reflection color C at the surface point generated by interaction of the object surface material and ambient light s Is a combination of (a) and (b). Wherein specular reflection colors generated by interaction of object surface materials and ambient light at the surface point can be regarded as superposition of specular reflection colors generated at the surface point by spatial sampling points associated with reflection directions corresponding to observation directions from the viewpoint to the surface point, respectively, in the neural ambient light field.
According to an embodiment of the present disclosure, determining specular reflection colors generated at the surface points by a plurality of spatial sampling points in the neural ambient light field, respectively, based on material properties of the surface points may include: and determining a specular reflection color generated by each spatial sampling point at the surface point based on the reflection direction and the position of each spatial sampling point by using each spatial sampling point of the plurality of spatial sampling points as an independent light source emitter.
Note that unlike conventional neural radiation fields (NeRF) where a light source is considered to be at infinity and the rendered color of an object surface point is represented as a superposition of the rendered colors of multiple sampling points on a ray incident from the light source on the object surface point, the neural ambient light field of the present disclosure treats each voxel (or spatial sampling point) in the environment surrounding the target object as a separate light source emitter, each of which can independently (i.e., without affecting each other) illuminate the target object, thereby affecting the specular reflection color of the target object. By designing the expression mode of the nerve environment light field, the overall environment illumination can be expressed three-dimensionally, and the shielding relation between the directional light source and the static three-dimensional environment is modeled, so that the rendering result of the target object is more attached to the environment to which the target object belongs.
Optionally, in an embodiment of the present disclosure, for a surface point x of the target object s The rendering color at this surface point observed from the observation direction d can be expressed as follows:
C(x s ，d)＝C d (x s )+∫ Ω f(l，-d，x s )L in (x s ，l)(l·n)dl (1)
wherein n and l respectively represent a surface point x s Normal and incident light L at in F represents a bi-directional reflection distribution function (bidirectional reflectance distribution function, BRDF), equation (1) is integrated for all directions of incidence within a hemisphere Ω, where hemisphere Ω satisfies l·n > 0.
Optionally, the diffuse reflection color C d (x s ) Can be determined from the own properties of the target object (i.e. the diffuse reflection color included in the material properties of the target object), and can be determined from the above description and equation (1) by considering all incident light in all incident directions within the hemisphere ΩObject surface point x s Illumination at to determine specular reflection color C s (x s D) (i.e. C s (x s ，d)＝∫ Ω f(l，-d，x s )L in (x s L) (l·n) dl), it should be appreciated that, although in the above description it is believed that light sources in the neuro-ambient light field may contribute to illumination of the target object, based on the material properties of the target object, typically only a fraction of these light sources associated with a reflection direction corresponding to the direction of observation from the viewpoint to the object surface point will have an impact on the specular reflection color at that surface point, i.e. the specular reflection color generated by other light sources at that surface point is almost negligible. For example, in embodiments of the present disclosure, specular reflection color C s (x s D) can be determined in the neuro-ambient light field from the roughness and tone modulation properties included in the material properties of the target object.
As an example, in an embodiment of the present disclosure, the BRDF function f is based on the reflection direction i r Assuming rotational symmetry, spatially varying object surface texture properties can be defined as roughness ρ ε [0,1 ]]Diffuse reflection color C d ∈[0，1] 3 Tone modulation alpha e 0,1]Thereby generating a specular lobe as shown in fig. 4 (e.g., shown in fig. 4 as a largest cone represented by a dashed line). Thus, in embodiments of the present disclosure, the BRDF function f described above may be approximated with von Mises-Fisher, vMF distributions, i.e., with a spherical harmonic gaussian function representing one normalized unit lobe, as follows:
f(l，-d，x s )≈G(l，l r ，x s )＝αexp(ρ(x s )(l·l r -1)) (2)
wherein l r And l represents the unit vector in the reflection direction corresponding to the observation direction and the unit vector in all reflection directions within the mirror lobe, respectively, the function values f (l, -d, x s ) And l.l r The results of (2) are positively correlated. Note that l r Represents the central axis of the specular lobe, while the roughness ρ (x s ) The angular width of the specular lobe can be controlled and the tonal modulation a can be considered as the amplitude of the specular lobe. According to the above equation, a larger ρ (x s ) May correspond to a rougher surface, i.e. to a wider vMF distribution.
Thus, according to an embodiment of the present disclosure, the plurality of spatial sampling points may be all spatial sampling points in a reflection lobe having the reflection direction as a central axis, which is determined based on a material property of the surface point, wherein a width of the reflection lobe is related to a roughness of the surface point. That is, the specular color at an object surface point for all spatial sampling points in the neural ambient light field can be considered as a superposition of the specular color produced at that surface point for all spatial sampling points in the reflected lobe.
Based on the above description, for each surface point x of the target object s The specular reflection color at this surface point for all spatial sampling points in the neural ambient light field of the present disclosure can be expressed as surrounding reflected rays r (t) =x in the neural ambient light field s The superposition of the specular reflection colors produced at this surface point by all reflected light rays within one reflection lobe of +tl, wherein the unit vector l of the reflection direction can be determined on the basis of the observation direction, for example by 2 (-d·n) n+d.
According to an embodiment of the present disclosure, using each spatial sampling point of the plurality of spatial sampling points as an independent light source emitter, determining a specular reflection color generated by the spatial sampling point at the surface point based on the reflection direction and the position of each spatial sampling point may include: the positions of the spatial sampling points are position-coded, and specular reflection colors and volume densities generated by the spatial sampling points at the surface points are determined through a multi-layer perceptron network.
Alternatively, the image rendering method of the present disclosure may express the ambient light field around the target object directly as one continuous voxel radiation field, wherein each of the plurality of spatial sampling points in the voxel radiation field may be position-encoded to utilize a neural network structure such as MLP to encodeThe spatially encoded spatial sampling point x and the reflection direction l are used as inputs to output the specular reflection color c and the volume density σ corresponding to the spatial sampling point x at the object surface point, which can be expressed as:(x, l) → (c, σ). Of course, it should be understood that the neural network structure used in this disclosure is not particularly limited to the model type and topology of the deep neural network, and may be replaced with other various effective novel model structures, such as a Convolutional Neural Network (CNN) in combination with other network structures, or other network structures such as a long short term memory network (LSTM), etc., and the neural network structures such as MLP employed in this disclosure are used herein by way of example only and not limitation.
As an example, in an embodiment of the present disclosure, in order to enable a neural ambient light field to learn the change of high frequency detail according to the roughness of a target object, a new characterization approach is proposed, i.e. an integral lobe encoding (Integrated Lobe Encoding, ILE) method for the above-mentioned position encoding, to efficiently and simply construct the position encoding of all voxel coordinates within the above-mentioned specular lobe. Specifically, the specular lobe may be discretized into a series of circular tables, each of which may be approximated by a multi-element gaussian function, so that the encoded values of all voxel coordinates within a circular table may be integrated using the multi-element gaussian function, thereby obtaining a corresponding high-dimensional feature representation of any spatial sampling point.
For example, as shown in FIG. 4, for the reflection direction l r The specular lobe in the reflection direction may be discretized based on at least one spatial sampling point thereon (e.g., shown in five four-pointed star patterns in fig. 4) to generate a corresponding at least one circular truncated cone. Thus, by integrating the at least one circular table, a high-dimensional representation of the feature with high frequency detail of all voxel coordinates within the specular lobe can be determined.
Fig. 5 is a schematic flow diagram illustrating generation of specular reflection colors at object surface points based on at least one spatial sampling point in a neural ambient light field, according to an embodiment of the present disclosure.
According to an embodiment of the present disclosure, using each spatial sampling point of the plurality of spatial sampling points as an independent light source emitter, determining a specular reflection color generated by the spatial sampling point at the surface point based on the reflection direction and the position of each spatial sampling point may include: dividing the reflection light valve based on at least one spatial sampling point positioned in the reflection direction in the plurality of spatial sampling points to generate at least one round table, wherein a one-to-one correspondence exists between each spatial sampling point in the at least one spatial sampling point and each round table; for each spatial sampling point in the at least one spatial sampling point, performing position coding on all coordinates in a circular table corresponding to the spatial sampling point to generate high-frequency position characteristics of the spatial sampling point relative to the surface point; and determining specular reflection colors and bulk densities produced by the spatially sampled points at the surface points through a multi-layer perceptron network based on the high-frequency location features.
As shown in fig. 5, for each surface point x of the target object s The reflection direction l (here the reflection direction l is taken as the surface point x s Center axis of specular lobe at), can be calculated for reflected ray r (t) =x s +tl to determine the at least one spatial sampling point along the reflection direction l (i.e., K spatial sampling points x in fig. 5) 0 (t)、x k (t)、……、x K (t)), where t represents the step size of the sample, e.g., the distance between two adjacent four-corner star patterns in FIG. 4.
Thus, the integration lobe coding can be performed separately for each circular truncated cone to generate a high frequency location characterization of the spatial sampling points corresponding to each circular truncated cone with respect to the object surface points. Specifically, according to an embodiment of the present disclosure, position encoding all coordinates within a circular table corresponding to the spatial sampling point to generate a high frequency position feature of the spatial sampling point with respect to the surface point may include: based on the roughness of the surface points, approximating the round table by utilizing a multi-element Gaussian function, and carrying out position coding on all coordinates in the round table; and integrating the results of the position coding of all coordinates in the circular table to generate high-frequency position features of the spatial sampling points relative to the surface points. Alternatively, the position coordinates may be encoded using a fourier series, wherein the high frequency position features of the surface points may be obtained by transforming the position information of the surface points from the spatial domain to the frequency domain through a fourier transform. Thus, by mapping successive input position coordinates to the frequency domain, a frequency characteristic representation with richer position coordinates, i.e. a high frequency position characteristic, can be obtained.
For example, alternatively, for each circular table, its high frequency position feature may be defined by a multiple gaussian function (μ, Σ (ρ)), where μ represents the mean position coordinates of all spatial sampling points in each circular table, and the variance Σ along the circular table radius direction may be determined by the material roughness ρ at the object surface point. Thus, for each of the at least one circular truncated cone, its high frequency location characteristics may be represented as follows:
wherein L represents the corresponding order of the high-frequency position characteristic of the current round table.
Thus, based on the integrated lobe-encoded high frequency location features ILE (p) of each truncated cone, the specular reflection color (c) produced at the object surface point at all spatial sampling points in its corresponding lobe portion can be determined by a neural network structure such as MLP i ，σ i ) Which includes color c i Sum volume density sigma i Wherein i represents the ith round table.
The final specular reflection color at the surface point of the target object may then be determined based on the specular reflection colors produced by the spatially sampled points, respectively, at that surface point. According to an embodiment of the present disclosure, determining the specular reflection color of the surface point based on the specular reflection color generated at the surface point by the plurality of spatial sampling points, respectively, may include: the specular reflection color of the surface point is determined by integrating the specular reflection color produced at the surface point by all spatially sampled points in the reflection lobe, respectively.
Optionally, for the surface point x of the target object s And either reflection direction L, in order to obtain the incident light L in equation (1) in In embodiments of the present disclosure, the object surface point x may be rendered by voxels s The values of all spatial sampling points starting from and along the reflection direction l are integrated as follows:
where T represents the integrated opacity, which can be expressed as follows:
thus, the above equation represents that all spatial sampling points are considered as one independent light emitter in the embodiments of the present disclosure, so that the image rendering method of the present disclosure can model occlusion relations in directional light sources and static three-dimensional environments, while also enabling scenes compatible with other NeRF representations.
By using the ray sampling shown in equation (4), the specular reflection color portion in equation (1) can be expressed as:
as indicated above, equation (6) above may represent determining that all spatial sampling points within the specular lobe are at object surface point x s An integral of the optical radiation value at, wherein the specular lobe may be defined by a vector of reflections, an amplitude, etc.
For a discrete sampling scenario as shown in fig. 5, determining a specular reflection color of the surface point based on specular reflection colors generated by the plurality of spatial sampling points at the surface point, respectively, may include: the specular reflection color of the surface point is determined by voxel rendering based on the specular reflection color generated at the surface point by each of the at least one spatial sampling point, respectively.
Alternatively, as shown in the above equation (6), the integral value thereof is performed under vMF distribution, that is, it will follow l.l r The values decay. The ILE-encoded location features described above may be used as inputs to an MLP network to output the volume density and color values of points at the object surface.
The above-described integral lobe coding method allows a neural network such as an MLP to parameterize all incident light within a specular lobe as a function of roughness, thereby efficiently mapping successive input voxel coordinates to high frequency location features.
Thus, by doing so, the specular color of the object surface point in the neural ambient light field can be determined. The final rendering color of each object surface point can then be determined based on its specular and diffuse reflection colors to achieve image rendering at any point of view. In step 204, a rendered color of each surface point of the target object in the neural ambient light field may be determined based on the diffuse and specular reflection colors of each surface point of the target object.
According to an embodiment of the present disclosure, determining a rendering color of each surface point of the target object in the neural ambient light field based on the diffuse and specular reflection colors of each surface point of the target object may include: the diffuse reflection color and the specular reflection color of each surface point of the target object are superimposed to generate a rendered color of each surface point of the target object in the neural ambient light field.
Optionally, for each surface point x of the target object s Its final rendering color value C can be represented as diffuse reflection color C d And to the sameVoxel integral value C in mirror lobe s Is a combination of (a) and (b). For example, the final rendering color value C may be expressed as follows:
C(x s ，d)＝γ(C d (x s )+C s (x s ，d)) (7)
where γ may represent a learnable High Dynamic Range (HDR) to Low Dynamic Range (LDR) mapping, which in embodiments of the present disclosure may be approximated as gamma (gamma) γ correction, which is a learnable transformation (e.g., exposure and white balance correction of an image).
Therefore, based on the image rendering method described above with reference to steps 201-204, the present disclosure proposes a completely new representation of an ambient light field, namely a neural ambient light field, which can perform 5D neural ambient light field modeling on the surrounding of the target object, wherein each spatial sampling point in the surrounding of the target object is regarded as an independent light source emitter by using a neural radiation field technique, and simultaneously, the position of the light source and the viewpoint direction of the observer are considered, and by such neural ambient light field modeling, the real natural ambient illumination considering the viewpoint position and direction can be obtained. Meanwhile, the image rendering method also provides an integral lobe coding method, so that incident light in a mirror lobe adaptive to the roughness of a target object can be obtained through single calculation, and the image rendering speed is greatly improved.
Next, a training process of a material estimation network, a light field reconstruction network, and an image rendering network employed by the image rendering method of the present disclosure may be described with reference to fig. 6A and 6B.
According to embodiments of the present disclosure, the texture estimation network, the light field reconstruction network, and the image rendering network may be jointly trained based on multi-view training images. Optionally, the material estimation network, the light field reconstruction network and the image rendering network may be trained by taking a training target object as input and taking multi-viewpoint training images as supervision information, where the multi-viewpoint training images may be real images obtained for the training target object based on a plurality of different viewpoint observations, so that the training images may be used as supervision information to perform supervision training on light field reconstruction and image rendering in the light field reconstruction network and the image rendering network.
Fig. 6A is a schematic flow diagram illustrating training of an image rendering network according to an embodiment of the present disclosure. As shown in fig. 6A, the joint training of the texture estimation network, the light field reconstruction network, and the image rendering network may take training target objects as inputs, and multi-viewpoint training images as supervisory information. Wherein each network may have a corresponding task, for example, for a material estimation network, it is considered that it may comprise decoupling of material properties of an object, for a light field reconstruction network, it may comprise reconstruction and expression of a neural environment light field in a multi-view training image, and for an image rendering network, it may comprise generating a plug-and-play rendering pipeline, wherein plug-and-play herein may be understood as insertion of a target object or insertion of a neural environment light field.
Alternatively, the training target object may be the same or different than the target object above, and the neuro-ambient light field in which the training target object is located in the multi-view training image may also be the same or different than the neuro-ambient light field in which the target object above is located, which may depend on the specific task scenario (e.g., new viewpoint image generation or re-illumination image generation task, etc.) as described above. For example, for a new view image generation task, the training target object may be the same as the target object above, and the neuro-environmental light field in which the training target object is located in the multi-view training image may be the same as the neuro-environmental light field in which the target object is located above, the neuro-environmental light field may be generated by the above-described joint training, and then the new view image of the target object may be generated by placing the target object (i.e., training target object) in the neuro-environmental light field and setting a desired observation direction.
According to embodiments of the present disclosure, the joint training may utilize background information obtained by performing multi-scale pre-convolution on the training image as supervision information.
Alternatively, in view of the complex high-dimensional illumination information, diffuse reflection colors may be easily fused into the ambient illumination when the material properties of the surface points of the training target object and the ambient illumination are jointly optimized according to the above equations (6) and (7). Although in theory, the above-mentioned image rendering method of the present disclosure may still generate a new viewpoint composite image with high fidelity, since the material information of the training target object is not accurately distinguished from the background information in the environment, the above-mentioned method may be erroneous in the case of a re-illumination task. Thus, the above-described joint training can be supervised using the background information of the training target object.
As described above, in equation (6), the integral of all incident light rays within one specular lobe can be regarded as a convolution process, and the width of the specular lobe is correlated with the roughness of the target object, so for larger roughness it can provide a wider integral of reflected light rays within the specular lobe at a surface point such that the reflection at the surface point becomes more blurred. Thus, optionally, the image rendering method of the present disclosure may also utilize a multi-scale pre-convolution technique, using a set of multi-scale gaussian convolution kernels to blur the background information of the training target object, which directs the roughness decoupled from the ambient light field. It should be appreciated that the above method is similar to the pre-filtered environment map in CG rendering, but the problem of view directionality is ignored in CG rendering, whereas in the image rendering method of the present disclosure, the view direction (observation direction) is an important input in BRDF. Thus, in embodiments of the present disclosure, the multi-scale pre-convolution may take into account the problem of viewpoint direction, so that high quality specular reflection results may be obtained. In particular, in practice, the radius of the blurred pixel may be set to 3σr 0 Where σ represents the variance of the gaussian convolution kernel and r 0 Representing the radius of the unprocessed pixel.
Fig. 6B is a schematic diagram illustrating training of an image rendering network according to an embodiment of the present disclosure. As shown in fig. 6B, for the surface point x of the target object s The decoupled texture properties, including diffuse reflectance color C, can be obtained from the neural network structure in FIG. 6B d Roughness ρ, and tone modulation α. The roughness obtained may be obtained by normalization processing ρ/r such as in a graph 0 But is applied to the light field of the nerve environment.
Alternatively, in embodiments of the present disclosure, the above-described joint training may use the same loss function/as NeRF recon Calculating a rendered color predicted by an image rendering network, e.g., using an L-2 normAnd the error between the true color C, and jointly optimizing a material estimation network and a light field reconstruction network through gradient return: />
Wherein S is I Representing a set of all pixels used for the loss function calculation. In addition, in order to further reduce ambiguity between the material of the target object and the environment, the object surface may be dotted with x s Roughness ρ (x) s ) And specular tone modulation alpha (x) s ) The constraint is relatively smooth. Specifically, according to the image gradient of the pixel point p, a bilateral smoothing regularization term may be defined as follows:
Equation (9) above may constrain the gradient requirements of a material to be related to the gradient of the pixel point it projects. As an example, in practice, image gradientsMay be pre-calculated and the gradient of the material may be derived by the chain law.
Furthermore, as shown in fig. 6B, embodiments of the present disclosure may also employ hierarchical sampling techniques in NeRF, including both coarse and fine sampling steps. In particular, during the joint training process, it is possible toThe micro-renderable manner shown in fig. 6B is adopted for all the surface points of the training target object, wherein the material estimation and the light field reconstruction of the training target object are jointly optimized. In addition, for the pixel points in the background, the diffuse reflection color C can be ignored d And specular tonal modulation alpha, and directly imparts a gaussian convolution kernel radius r p ＝3σr 0 。
Therefore, by the above-described image rendering method of the present disclosure, first, high-quality new viewpoint image synthesis and re-illumination image synthesis can be achieved. In addition, since the above method decouples the material of the target object from the neural environment light field, the image rendering method of the present disclosure can edit any part thereof (e.g., the material of the target object or the neural environment light field) individually.
Optionally, for editing the material of the object, the material attribute parameters output by the material estimation network may be modified. For example, by reflecting color C for diffuse reflection d By modifying, a change in the colour of the object can be achieved, in which case the specular colour C s No change occurs. In addition, editing the specular tonal modulation α can increase or decrease the intensity of the reflection, or change the tonal of the reflection, and increasing or decreasing the roughness ρ can modify the degree of blurring of the reflection, thereby enabling the simulation of objects of different roughness, i.e., smoother objects with less roughness, sharper ambient light reflections, and rougher objects with greater roughness, more blurring of ambient light reflections. In addition, these edits may also be applied to certain portions of the target object through the setting of specific constraints.
Optionally, for editing of the neural ambient light field, coordinate transformations (such as rotations and offsets) may be performed on spatially sampled points in the neural ambient light field to effect translation and rotation of the neural ambient light field. In addition, under the new neural environment light field condition, the image rendering method disclosed by the invention can recover reasonable object shielding relation or details such as high light, strong directivity light sources and the like. Thus, with this feature, the image rendering method of the present disclosure may enable replacement of a neural ambient light field, i.e., replacing a current neural ambient light field network with, for example, a pre-trained neural radiation field. By properly adjusting the position and the material of the target object, a natural fusion result can be obtained.
Thus, by way of example and not limitation, the image rendering method of the present disclosure may have the following beneficial technical effects: (1) By providing the expression mode of the nerve environment light field, the three-dimensional information of the object ignored by the expression mode of the traditional environment light map can be solved, and the problem of a directional light source which is difficult to process can be represented; (2) By proposing a new rendering framework (comprising a material estimation network, a light field reconstruction network and an image rendering network), any object represented by a grid can be placed in a three-dimensional scene expressed by a nerve radiation field and a true natural reflection effect is generated, furthermore, by proposing integral lobe coding (ILE), the integral of all rays in a mirror lobe can be obtained efficiently by one voxel rendering, and the size of the mirror lobe can be adaptively adjusted by considering the roughness of the material of a target object; (3) By using the pre-convolution background to guide the decoupling of the material and the decoupling of the light field, the background information ignored in the previous method is effectively utilized, so that the high-precision reflection information can be reconstructed in the new viewpoint image synthesis task, and the evaluation index and the look and feel are greatly improved.
Fig. 7 is a schematic diagram illustrating an image rendering apparatus 700 according to an embodiment of the present disclosure.
According to an embodiment of the present disclosure, the image rendering apparatus 700 may include a material property acquisition module 701, an ambient light field preparation module 702, a reflected color determination module 703, and a rendered color determination module 704.
The texture property acquisition module 701 may be configured to acquire a target object to be rendered and determine a texture property of each surface point of the target object, the texture property including a diffuse reflection color. Alternatively, the texture property acquisition module 701 may perform the operations described above with reference to step 201.
Alternatively, geometric information of the target object to be rendered may be first acquired, which may be represented in a grid form, wherein a plurality of surface points of the target object may be included, each of which may be determined a specific rendering color in embodiments of the present disclosure, and then the target object may be rendered based on the determined rendering color of each surface point, thereby generating a final rendered image. The method for obtaining the geometric information of the target object to be rendered may be estimated in advance by various geometric extraction methods, and the present disclosure is not limited to a specific method for obtaining the geometric information of the target object, and any method that can achieve the above object may be applied to the geometric information obtaining operation of the present disclosure.
For example, for each surface point of the target object, its rendering color may be considered as a result of the interaction of the diffuse reflection color of the object surface independent of the direction of observation and the specular reflection color produced by the interaction of the object surface material and ambient light at that surface point. Thus, alternatively, embodiments of the present disclosure may determine the material properties of the target object based on diffuse and specular reflections produced in the object and ambient light. As an example, in embodiments of the present disclosure, diffuse reflection color may be considered as the only factor affecting diffuse reflection of the object surface and as the own material properties of the target object, while for specular reflection of the target object surface, material properties including hue modulation and roughness may be considered. In particular, diffuse reflectance colors may be used to effect a change in the color of an object whose property value change does not affect the specular reflectance color, tonal modulation may be used to enhance or attenuate the intensity of the specular reflectance or to change the tonal color of the specular reflectance, and roughness may affect the degree of blurring of the specular reflectance.
Alternatively, the texture properties corresponding to their pairs may be determined from object surface points by a trained network structure such as a multi-layer perceptron neural network. It should be understood that the embodiments of the present disclosure are not limited to the specific structure of the texture estimation network, and other network structures that can achieve the same texture estimation purpose may be equally applicable to the image rendering method of the present disclosure.
The ambient light field preparation module 702 may be configured to obtain a neural ambient light field for rendering the target object and a location of a viewpoint, and place the target object in the neural ambient light field. Alternatively, ambient light field preparation module 702 may perform the operations described above with reference to step 202.
Optionally, in embodiments of the present disclosure, the neural ambient light field used to render the target object may include the original neural ambient light field in which the target object is located, as well as other neural ambient light fields, thereby generating a new viewpoint image or re-illumination image of the target object. Thus, the operation of the ambient light field preparation module 702 to obtain the neural ambient light field for rendering the target object may include obtaining the original neural ambient light field in which the target object is located, or may include obtaining other neural ambient light fields for rendering the target object, which may be selected according to actual task needs.
As an example, the neural ambient light field may be comprised of spatially sampled points, where each spatially sampled point may have known optical radiation properties, which may include bulk density and color. Alternatively, the neuro-ambient light field of the present disclosure may be comprised of a plurality of voxels in the light field environment, wherein each voxel may have known optical radiation properties and may produce optical radiation effects on a surface point of the target object. Alternatively, the spatial sampling points in the present disclosure may be the same as the voxels here, or may be the result of sampling all voxels in the light field environment, which may be determined according to actual computational demand, which is not limiting of the present disclosure.
Thus, based on the neural ambient light field and the viewpoint location determined by the ambient light field preparation module 702, an observation direction from the viewpoint to each surface point of the target object and its corresponding reflection direction may be determined, where the reflection direction may be determined based on the observation direction and a normal direction to the surface point. Optionally, a series of spatial sampling points associated with the reflection direction will produce specular reflection at the surface point and generate a corresponding specular reflection color.
The reflection color determination module 703 may be configured to determine, in the neural environment light field, for each surface point of the target object, a specular reflection color generated at the surface point by a plurality of spatial sampling points in the neural environment light field, respectively, based on a material property of the surface point, and determine a specular reflection color of the surface point based on the specular reflection color generated at the surface point by the plurality of spatial sampling points, respectively, wherein the plurality of spatial sampling points are associated with a reflection direction corresponding to an observation direction from the viewpoint to the surface point. Alternatively, the reflected color determination module 703 may perform the operations described above with reference to step 203.
As an example, each spatial sampling point of the plurality of spatial sampling points may be considered as an independent light source emitter, and specular reflection color generated by the spatial sampling point at the surface point may be determined based on the reflection direction and the position of each spatial sampling point. In particular, the neural ambient light field of the present disclosure may treat each voxel (or spatially sampled point) in the environment surrounding the target object as an independent light source that may both contribute to the illumination of the target object, thereby affecting the specular reflectance color of the target object. By designing the expression mode of the nerve environment light field, the overall environment illumination can be expressed three-dimensionally, and the shielding relation between the directional light source and the static three-dimensional environment is modeled, so that the rendering result of the target object is more attached to the environment to which the target object belongs.
In embodiments of the present disclosure, the rendered color of an object surface point at which the specular reflection color produced by the object surface material and ambient light interaction may be regarded as a superposition of specular reflection colors produced at the surface point by spatially sampled points in the neural ambient light field associated with reflection directions corresponding to the observation direction from the viewpoint to the surface point, respectively, may be expressed as a combination of the diffuse reflection color and the specular reflection color produced by the object surface material and ambient light interaction at the surface point. Wherein, optionally, the diffuse reflection color may be determined according to the own properties of the target object (i.e. the diffuse reflection color included in the material properties of the target object), and the specular reflection color may be determined in the neural ambient light field according to the roughness and tone modulation properties included in the material properties of the target object.
For example, the plurality of spatial sampling points may be all spatial sampling points in a reflection lobe having the reflection direction as a central axis, which is determined based on a material property of the surface point, wherein a width of the reflection lobe is related to roughness of the surface point. That is, the specular color at an object surface point for all spatial sampling points in the neural ambient light field can be considered as a superposition of the specular color produced at that surface point for all spatial sampling points in the reflected lobe. Alternatively, the ambient light field around the target object may be directly expressed as one continuous voxel radiation field, wherein each of the plurality of spatial sampling points in the voxel radiation field may be position-encoded to take as input the position-encoded spatial sampling points and the reflection direction using a neural network structure such as MLP, thereby outputting the specular reflection color and the volume density corresponding to the spatial sampling points at the object surface point. Therein, as an example, the position encoding of all voxel coordinates within a specular lobe can be efficiently and simply constructed using an integral lobe encoding method.
Specifically, the specular lobe may be discretized into a series of circular truncated cones, each of which may be approximated by a multi-element gaussian function, so that the encoded values of all voxel coordinates within a circular truncated cone may be integrated using the multi-element gaussian function, thereby obtaining a corresponding high-dimensional feature representation of any spatial sampling point. For example, for a reflection direction corresponding to an observation direction, a specular lobe in that reflection direction may be discretized based on at least one spatial sampling point thereon to generate a corresponding at least one circular truncated cone. Thus, by integrating the at least one circular table, a high-dimensional representation of the feature with high frequency detail of all voxel coordinates within the specular lobe can be determined.
For example, the integration lobe encoding may be performed separately for each circular truncated cone to generate a high frequency location characterization of the spatial sampling points corresponding to each circular truncated cone with respect to the object surface points. Specifically, the circular truncated cone may be approximated by a multiple gaussian function based on the roughness of the surface point, and all coordinates within the circular truncated cone may be position-coded, and the result of the position-coding of all coordinates within the circular truncated cone may be integrated to generate a high-frequency position characteristic of the spatial sampling point with respect to the surface point.
Thus, based on the high frequency position characteristics of the integrated lobe codes of each circular truncated cone, the specular reflection color produced at the object surface point by all the spatial sampling points in its corresponding lobe portion can be determined by a neural network structure such as MLP. The above-described integral lobe coding method allows a neural network such as an MLP to parameterize all incident light within a specular lobe as a function of roughness, thereby efficiently mapping successive input voxel coordinates to high frequency location features.
The final specular reflection color at the surface point of the target object may then be determined based on the specular reflection colors produced by the spatially sampled points, respectively, at that surface point. For example, the specular reflection color of the surface point may be determined by integrating the specular reflection color produced at the surface point by all spatially sampled points in the reflection lobe, respectively. Alternatively, for a surface point of the target object and any reflection direction, the values of all spatial sampling points starting from the object surface point and along the reflection direction may be integrated by voxel rendering.
Thus, the specular reflection color of the object surface point in the neural ambient light field can be determined by the reflection color determination module 703 described above. Next, in a rendering color determination module 704, its final rendering color may be determined based on the specular and diffuse reflection colors of the respective object surface points. The rendering color determination module 704 may be configured to determine a rendering color of each surface point of the target object in the neural ambient light field based on the diffuse and specular reflection colors of each surface point of the target object. Alternatively, rendering color determination module 704 may perform the operations described above with reference to step 204. For example, the diffuse and specular reflection colors of each surface point of the target object may be superimposed to generate a rendered color of each surface point of the target object in the neural ambient light field. Wherein the final rendered color value of the surface point may be represented as a combination of diffuse reflection color and voxel integration values within the specular lobe as described above.
The image rendering method and the image rendering apparatus of the present disclosure can be applied to any target object whose geometry is represented in a mesh, and can render an image truly natural thereto by placing it in existing various types of nerve radiation fields, unlike rendering engines such as Blender, UE, etc., the image rendering method and the image rendering apparatus of the present disclosure have a faster rendering speed. In addition, the image rendering method and the image rendering device can decouple the existing illumination scene, and can assist the present disclosure to reconstruct the nerve environment light field better by means of the background information of multi-scale pre-convolution through decoupling the diffuse reflection color, roughness and tone modulation attribute of the target object.
By decoupling the material of the target object and the nerve environment light field, the image rendering method and the image rendering device can realize the image synthesis of the new viewpoint of the current scene, and under the condition of editing the current target object such as rendering, moving and the like, the reflection of the current target object can accurately and naturally change along with the current target object. Meanwhile, the reconstructed nerve environment light field can be applied to other target objects, natural and real images are synthesized through the image rendering pipeline, or high-definition environment maps are derived to be accessed into an existing rendering engine.
According to still another aspect of the present disclosure, there is also provided an image rendering apparatus. Fig. 8 shows a schematic diagram of an image rendering device 2000 according to an embodiment of the present disclosure.
As shown in fig. 8, the image rendering device 2000 may include one or more processors 2010, and one or more memories 2020. Wherein said memory 2020 has stored therein computer readable code which, when executed by said one or more processors 2010, can perform an image rendering method as described above.
The processor in embodiments of the present disclosure may be an integrated circuit chip having signal processing capabilities. The processor may be a general purpose processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), an off-the-shelf programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components. The disclosed methods, steps, and logic blocks in the embodiments of the present application may be implemented or performed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like, and may be of the X86 architecture or ARM architecture.
In general, the various example embodiments of the disclosure may be implemented in hardware or special purpose circuits, software, firmware, logic, or any combination thereof. Some aspects may be implemented in hardware, while other aspects may be implemented in firmware or software which may be executed by a controller, microprocessor or other computing device. While aspects of the embodiments of the present disclosure are illustrated or described as block diagrams, flow charts, or using some other pictorial representation, it is well understood that the blocks, apparatus, systems, techniques or methods described herein may be implemented in, as non-limiting examples, hardware, software, firmware, special purpose circuits or logic, general purpose hardware or controller or other computing devices, or some combination thereof.
For example, a method or apparatus according to embodiments of the present disclosure may also be implemented by means of the architecture of computing device 3000 shown in fig. 9. As shown in fig. 9, computing device 3000 may include a bus 3010, one or more CPUs 3020, a Read Only Memory (ROM) 3030, a Random Access Memory (RAM) 3040, a communication port 3050 connected to a network, an input/output component 3060, a hard disk 3070, and the like. A storage device in the computing device 3000, such as a ROM 3030 or a hard disk 3070, may store various data or files used for processing and/or communication of the image rendering method provided by the present disclosure and program instructions executed by the CPU. The computing device 3000 may also include a user interface 3080. Of course, the architecture shown in FIG. 9 is merely exemplary, and one or more components of the computing device shown in FIG. 9 may be omitted as may be practical in implementing different devices.
According to yet another aspect of the present disclosure, a computer-readable storage medium is also provided. Fig. 10 shows a schematic diagram 4000 of a storage medium according to the present disclosure.
As shown in fig. 10, the computer storage medium 4020 has stored thereon computer readable instructions 4010. The image rendering method according to the embodiments of the present disclosure described with reference to the above drawings may be performed when the computer readable instructions 4010 are executed by a processor. The computer readable storage medium in embodiments of the present disclosure may be volatile memory or nonvolatile memory, or may include both volatile and nonvolatile memory. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be Random Access Memory (RAM), which acts as external cache memory. By way of example, and not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), dynamic Random Access Memory (DRAM), synchronous Dynamic Random Access Memory (SDRAM), double data rate synchronous dynamic random access memory (ddr SDRAM), enhanced Synchronous Dynamic Random Access Memory (ESDRAM), synchronous Link Dynamic Random Access Memory (SLDRAM), and direct memory bus random access memory (DRRAM). It should be noted that the memory of the methods described herein is intended to comprise, without being limited to, these and any other suitable types of memory. It should be noted that the memory of the methods described herein is intended to comprise, without being limited to, these and any other suitable types of memory.
Embodiments of the present disclosure also provide a computer program product or computer program comprising computer instructions stored in a computer readable storage medium. A processor of a computer device reads the computer instructions from a computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs an image rendering method according to an embodiment of the present disclosure.
Embodiments of the present disclosure provide an image rendering method, apparatus, device, and computer-readable storage medium.
Compared with the traditional image rendering method, the method provided by the embodiment of the disclosure can consider that the ambient light around the object changes along with the position and the direction of the viewpoint, and each space sampling point around the object is regarded as an independent light source emitter so as to model the blocking relation between the directional light source and the static three-dimensional environment, thereby realizing better reconstruction of the ambient light field.
According to the method provided by the embodiment of the disclosure, the rendering color of the object surface point is decomposed into the diffuse reflection color and the specular reflection color generated by interaction between the material of the object and the ambient light, the diffuse reflection color is respectively determined by decoupling the material attribute of the object, and the light action result of each spatial sampling point around the object in the neural ambient light field on the object surface point is generated, so that the rendering color of the object surface point in the neural ambient light field is determined. The method of the embodiment of the disclosure can simultaneously consider the positions of a plurality of spatial sampling points around the object in the nerve environment light field and the viewpoint position and direction of the observer to represent the interaction and shielding relation between the light rays and the material of the object in the real scene. Furthermore, in embodiments of the present disclosure, by decoupling the material properties of the target object from the neuro-ambient light field, separate control of the target object or neuro-ambient light field may be achieved for achieving true natural image rendering in a variety of task scenarios, such as new view synthesis or re-illumination.
It is noted that the flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises at least one executable instruction for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
In general, the various example embodiments of the disclosure may be implemented in hardware or special purpose circuits, software, firmware, logic, or any combination thereof. Some aspects may be implemented in hardware, while other aspects may be implemented in firmware or software which may be executed by a controller, microprocessor or other computing device. While aspects of the embodiments of the present disclosure are illustrated or described as block diagrams, flow charts, or using some other pictorial representation, it is well understood that the blocks, apparatus, systems, techniques or methods described herein may be implemented in, as non-limiting examples, hardware, software, firmware, special purpose circuits or logic, general purpose hardware or controller or other computing devices, or some combination thereof.
The exemplary embodiments of the present disclosure described in detail above are illustrative only and are not limiting. Those skilled in the art will understand that various modifications and combinations of these embodiments or features thereof may be made without departing from the principles and spirit of the disclosure, and such modifications should fall within the scope of the disclosure.
Claims (16)
1. An image rendering method, comprising:
acquiring a target object to be rendered and determining the material property of each surface point of the target object, wherein the material property comprises diffuse reflection color;
acquiring a nerve environment light field for rendering the target object and the position of a viewpoint, and placing the target object in the nerve environment light field;
in the neural environment light field, for each surface point of the target object, determining specular reflection colors generated at the surface point by a plurality of spatial sampling points in the neural environment light field, respectively, based on material properties of the surface point, and determining specular reflection colors of the surface point based on the specular reflection colors generated at the surface point by the plurality of spatial sampling points, respectively, wherein the plurality of spatial sampling points are associated with reflection directions corresponding to observation directions from the viewpoint to the surface point; and
A rendering color of each surface point of the target object in the neural ambient light field is determined based on the diffuse and specular reflection colors of each surface point of the target object.
2. The method of claim 1, wherein the neuro-ambient light field is comprised of spatially sampled points, wherein each spatially sampled point has known optical radiation properties including bulk density and color;
based on the material properties of the surface points, determining specular reflection colors produced at the surface points by a plurality of spatially sampled points in the neural ambient light field, respectively, comprises:
and determining a specular reflection color generated by each spatial sampling point at the surface point based on the reflection direction and the position of each spatial sampling point by using each spatial sampling point of the plurality of spatial sampling points as an independent light source emitter.
3. The method of claim 2, wherein the material properties further comprise roughness;
the plurality of spatial sampling points are all spatial sampling points in a reflection light lobe which takes the reflection direction as a central axis and is determined based on the material property of the surface point, wherein the width of the reflection light lobe is related to the roughness of the surface point.
4. The method of claim 3, wherein determining the specular reflection color produced by the spatial sampling points at the surface point based on the reflection direction and the position of each spatial sampling point using each spatial sampling point of the plurality of spatial sampling points as an independent light source emitter comprises:
the positions of the spatial sampling points are position-coded, and specular reflection colors and volume densities generated by the spatial sampling points at the surface points are determined through a multi-layer perceptron network.
5. The method of claim 4, wherein determining the specular reflection color of the surface point based on the specular reflection color produced by the plurality of spatially sampled points, respectively, at the surface point comprises:
the specular reflection color of the surface point is determined by integrating the specular reflection color produced at the surface point by all spatially sampled points in the reflection lobe, respectively.
6. The method of claim 3, wherein determining the specular reflection color produced by the spatial sampling points at the surface point based on the reflection direction and the position of each spatial sampling point using each spatial sampling point of the plurality of spatial sampling points as an independent light source emitter comprises:
Dividing the reflection light valve based on at least one spatial sampling point positioned in the reflection direction in the plurality of spatial sampling points to generate at least one round table, wherein a one-to-one correspondence exists between each spatial sampling point in the at least one spatial sampling point and each round table;
for each spatial sampling point in the at least one spatial sampling point, performing position coding on all coordinates in a circular table corresponding to the spatial sampling point to generate high-frequency position characteristics of the spatial sampling point relative to the surface point; and
the specular reflection color and bulk density generated by the spatially sampled points at the surface points are determined by a multi-layer perceptron network based on the high frequency positional characteristic.
7. The method of claim 6, wherein position encoding all coordinates within the circular table corresponding to the spatial sampling points to generate high frequency positional features of the spatial sampling points with respect to the surface points comprises:
based on the roughness of the surface points, approximating the round table by utilizing a multi-element Gaussian function, and carrying out position coding on all coordinates in the round table; and
And integrating the position coding results of all coordinates in the circular table to generate high-frequency position characteristics of the space sampling points relative to the surface points.
8. The method of claim 7, wherein determining the specular reflection color of the surface point based on the specular reflection color produced by the plurality of spatially sampled points, respectively, at the surface point comprises:
the specular reflection color of the surface point is determined by voxel rendering based on the specular reflection color generated at the surface point by each of the at least one spatial sampling point, respectively.
9. The method of claim 1, wherein determining a rendered color of each surface point of the target object in the neural ambient light field based on the diffuse and specular reflection colors of each surface point of the target object comprises:
the diffuse reflection color and the specular reflection color of each surface point of the target object are superimposed to generate a rendered color of each surface point of the target object in the neural ambient light field.
10. The method of claim 1, wherein acquiring the target object to be rendered comprises:
obtaining geometric information of a target object to be rendered, wherein the geometric information of the target object comprises a plurality of surface points of the target object; and
Determining the material properties of each surface point of the target object includes:
a texture property of each of the plurality of surface points of the target object is determined by a trained multi-layer perceptron neural network, the texture property comprising at least one of diffuse reflectance color, tonal modulation, and roughness.
11. The method of claim 1, wherein the image rendering method further comprises:
determining the material property of each surface point of the target object through a material estimation network;
generating the neural environment light field through a light field reconstruction network; and
determining a rendering color of each surface point of the target object in the neural environment light field through an image rendering network;
wherein the material estimation network, the light field reconstruction network, and the image rendering network are based on multi-view training image joint training.
12. The method of claim 11, wherein the joint training utilizes background information obtained by multi-scale pre-convolution of the training image as supervision information.
13. An image rendering apparatus comprising:
a material property acquisition module configured to acquire a target object to be rendered and determine a material property of each surface point of the target object, the material property including a diffuse reflection color;
An ambient light field preparation module configured to acquire a neural ambient light field for rendering the target object and a position of a viewpoint, and place the target object in the neural ambient light field;
a reflection color determination module configured to determine, for each surface point of the target object, a specular reflection color generated at the surface point by a plurality of spatial sampling points in the neuro-environmental light field, respectively, based on a material property of the surface point, and determine a specular reflection color of the surface point based on the specular reflection color generated at the surface point by the plurality of spatial sampling points, respectively, wherein the plurality of spatial sampling points are associated with a reflection direction corresponding to an observation direction from the viewpoint to the surface point; and
a rendering color determination module configured to determine a rendering color of each surface point of the target object in the neural ambient light field based on the diffuse and specular reflection colors of each surface point of the target object.
14. An image rendering apparatus comprising:
one or more processors; and
one or more memories in which a computer executable program is stored which, when executed by the processor, performs the method of any of claims 1-12.
15. A computer program product stored on a computer readable storage medium and comprising computer instructions which, when executed by a processor, cause a computer device to perform the method of any of claims 1-12.
16. A computer readable storage medium having stored thereon computer executable instructions which when executed by a processor are for implementing the method of any of claims 1-12.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310208155.3A CN116958362A (en) | 2023-02-27 | 2023-02-27 | Image rendering method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310208155.3A CN116958362A (en) | 2023-02-27 | 2023-02-27 | Image rendering method, device, equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116958362A true CN116958362A (en) | 2023-10-27 |
Family
ID=88448057
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310208155.3A Pending CN116958362A (en) | 2023-02-27 | 2023-02-27 | Image rendering method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116958362A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117333637A (en) * | 2023-12-01 | 2024-01-02 | 北京渲光科技有限公司 | Modeling and rendering method, device and equipment for three-dimensional scene |
| CN117422815A (en) * | 2023-12-19 | 2024-01-19 | 北京渲光科技有限公司 | Reverse rendering method and system based on nerve radiation field |
-
2023
- 2023-02-27 CN CN202310208155.3A patent/CN116958362A/en active Pending
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117333637A (en) * | 2023-12-01 | 2024-01-02 | 北京渲光科技有限公司 | Modeling and rendering method, device and equipment for three-dimensional scene |
| CN117333637B (en) * | 2023-12-01 | 2024-03-08 | 北京渲光科技有限公司 | Modeling and rendering method, device and equipment for three-dimensional scene |
| CN117422815A (en) * | 2023-12-19 | 2024-01-19 | 北京渲光科技有限公司 | Reverse rendering method and system based on nerve radiation field |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| AU2020200811B2 (en) | Direct meshing from multiview input using deep learning | |
| Li et al. | Monocular real-time volumetric performance capture | |
| Khakhulin et al. | Realistic one-shot mesh-based head avatars | |
| CN115100339B (en) | Image generation method, device, electronic equipment and storage medium | |
| WO2022001236A1 (en) | Three-dimensional model generation method and apparatus, and computer device and storage medium | |
| Rosu et al. | Permutosdf: Fast multi-view reconstruction with implicit surfaces using permutohedral lattices | |
| CN116958362A (en) | Image rendering method, device, equipment and storage medium | |
| JP2022032937A (en) | Computer vision method and system | |
| Weng et al. | Vid2actor: Free-viewpoint animatable person synthesis from video in the wild | |
| WO2021164759A1 (en) | Three-dimensional facial reconstruction | |
| US11669986B2 (en) | Generating enhanced three-dimensional object reconstruction models from sparse set of object images | |
| CN115239861A (en) | Face data enhancement method and device, computer equipment and storage medium | |
| DE102022113244A1 (en) | Joint shape and appearance optimization through topology scanning | |
| Yuan et al. | Neural radiance fields from sparse RGB-D images for high-quality view synthesis | |
| Ren et al. | Facial geometric detail recovery via implicit representation | |
| Duan et al. | Bakedavatar: Baking neural fields for real-time head avatar synthesis | |
| Mittal | Neural Radiance Fields: Past, Present, and Future | |
| US20230298243A1 (en) | 3d digital avatar generation from a single or few portrait images | |
| US20230154101A1 (en) | Techniques for multi-view neural object modeling | |
| CN116664422A (en) | Image highlight processing method and device, electronic equipment and readable storage medium | |
| WO2022251619A1 (en) | Hybrid differentiable rendering for light transport simulation systems and applications | |
| RU2729166C1 (en) | Neural dot graphic | |
| Jin | Variational methods for shape reconstruction in computer vision | |
| Miao et al. | Research on 3d reconstruction of furniture based on differentiable renderer | |
| Suppan et al. | Neural Screen Space Rendering of Direct Illumination. |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| REG | Reference to a national code |
Ref country code: HK Ref legal event code: DE Ref document number: 40099461 Country of ref document: HK |