CN113096228B - Real-time illumination estimation and rendering method and system based on neural network - Google Patents
Real-time illumination estimation and rendering method and system based on neural network Download PDFInfo
- Publication number
- CN113096228B CN113096228B CN202110639919.5A CN202110639919A CN113096228B CN 113096228 B CN113096228 B CN 113096228B CN 202110639919 A CN202110639919 A CN 202110639919A CN 113096228 B CN113096228 B CN 113096228B
- Authority
- CN
- China
- Prior art keywords
- real
- rendering
- virtual object
- neural network
- time
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T15/00—3D [Three Dimensional] image rendering
- G06T15/10—Geometric effects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- General Health & Medical Sciences (AREA)
- General Engineering & Computer Science (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Biomedical Technology (AREA)
- Life Sciences & Earth Sciences (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Geometry (AREA)
- Computer Graphics (AREA)
- Image Generation (AREA)
- Processing Or Creating Images (AREA)
Abstract
The invention provides a real-time illumination estimation and rendering system based on a neural network, which comprises a camera reading module: real-time camera video stream acquisition for end-side devices; the neural network reasoning module: using a lightweight network for generating a result of illumination estimation, and using spherical harmonic wave coefficients to render a virtual object; a virtual object loading module: the system comprises a three-dimensional virtual object loading unit, a three-dimensional virtual object loading unit and a three-dimensional virtual object loading unit, wherein the three-dimensional virtual object is used as an object rendered by an illumination result; the spherical harmonic wave real-time rendering and fusion module comprises: and rendering the virtual object by utilizing the spherical harmonic wave coefficient for realizing cross-platform rendering and fusion. The light-weight neural network is adopted, and the spherical harmonic wave coefficient is calculated according to the video sequence frame, so that real-time and efficient illumination estimation is achieved; meanwhile, the virtual-real fusion rendering module is adopted to render the three-dimensional virtual object in real time and display the three-dimensional virtual object in front of the camera video stream, so that real-time fusion of the virtual object and the real background is realized.
Description
Technical Field
The invention relates to the technical field of illumination, in particular to a real-time illumination estimation and rendering method and system based on a neural network.
Background
In the existing illumination estimation technology on the market, one type needs extra hardware to collect illumination information, and the other type needs a neural network with huge parameter quantity to calculate a high dynamic panoramic illumination map, and both the two types are difficult to achieve real-time illumination estimation. In addition, after the illumination information is obtained, the virtual object needs to be rendered, which needs to be parallel to the camera reading module and the neural network reasoning module, however, the computing power of the mobile terminal computing platform is often limited. Therefore, the prior art is difficult to solve the real-time illumination estimation and rendering of the mobile terminal.
Through retrieval, patent document CN103440684A discloses a method for applying spherical harmonic illumination technology to surface rendering, which includes, firstly, performing discrete sampling on each patch in a surface model to convert the patch into a point model; then obtaining a spherical harmonic coefficient group of each point by a spherical harmonic illumination method; generating a group of spherical harmonic coefficient textures for each surface by a texture backfill method according to the point dispersed from each surface and the corresponding spherical harmonic coefficient group; and finally, drawing the surface model by using a surface drawing method, and programming through a GPU rendering pipeline to complete the spherical harmonic illumination simulation. The method has the defects that the spherical harmonic coefficient is obtained by calculating after the surface model is converted into the point model, and then the spherical harmonic coefficient is drawn into the surface model, so that the calculated amount is large, the conversion is needed to obtain related data, the process is complicated, and the efficiency is low.
Therefore, it is necessary to develop a system that involves a small amount of parameters and is capable of performing illumination estimation and rendering in real time.
Disclosure of Invention
Aiming at the defects in the prior art, the invention aims to provide a real-time illumination estimation and rendering method and system based on a neural network, which solve the problems that the real-time illumination estimation and rendering of a mobile terminal are not required to adopt extra hardware to collect illumination information, and the neural network with huge parameter quantity is not required to calculate a high-dynamic panoramic illumination map.
The invention provides a real-time illumination estimation and rendering system based on a neural network, which comprises the following modules:
a camera reading module: real-time camera video stream acquisition for end-side devices;
the neural network reasoning module: using a lightweight network for generating a result of illumination estimation, and using spherical harmonic wave coefficients to render a virtual object;
a virtual object loading module: the system comprises a three-dimensional virtual object loading unit, a three-dimensional virtual object loading unit and a three-dimensional virtual object loading unit, wherein the three-dimensional virtual object is used as an object rendered by an illumination result;
the spherical harmonic wave real-time rendering and fusion module comprises: and rendering the virtual object by utilizing the spherical harmonic wave coefficient for realizing cross-platform rendering and fusion. Preferably, the camera reading module continuously acquires a real-time video stream of the camera, and performs a preprocessing operation on the video frame by frame to generate video sequence frames.
Preferably, the video sequence frames after pre-processing are used as input for the illumination estimation.
Preferably, the neural network inference module adopts a lightweight convolutional neural network as a backbone network.
Preferably, the neural network inference module improves learning ability using an optimized inverse residual bottleneck structure.
Preferably, for the video sequence frame of the input network, a potential space vector of 1280 dimensions is generated through a neural network inference module, and then an illumination result, namely a spherical harmonic wave coefficient, is obtained from the potential space vector through a full connection layer.
Preferably, when the virtual object loading module loads a three-dimensional virtual object, the entire model can be packaged into a scene object, and the scene object includes a series of scene nodes for storing vertex data and surface indexes.
Preferably, the spherical harmonic wave real-time rendering and fusion module renders the loaded three-dimensional virtual object in real time according to the spherical harmonic wave coefficient.
According to the real-time illumination estimation and rendering method based on the neural network, the real-time illumination estimation and rendering system based on the neural network is adopted to carry out real-time illumination estimation and rendering.
Preferably, the method comprises the following steps:
a camera reading step: acquiring a real-time camera video stream of a terminal side device;
neural network reasoning step: generating an illumination estimation result by utilizing a lightweight network;
loading a virtual object: loading a three-dimensional virtual object as an object rendered by an illumination result;
real-time rendering and fusing spherical harmonic waves: and rendering the virtual object by utilizing the spherical harmonic wave coefficient to realize cross-platform rendering and fusion. Compared with the prior art, the invention has the following beneficial effects:
1. the invention solves the problems of real-time illumination estimation and rendering of the mobile terminal, does not need extra hardware to collect illumination information, and does not need a neural network with huge parameter quantity to calculate a high-dynamic panoramic illumination map.
2. The light-weight neural network is adopted, and the spherical harmonic wave coefficient is calculated according to the video sequence frame, so that real-time and efficient illumination estimation is achieved.
3. The invention adopts the rendering module of virtual-real fusion to render the three-dimensional virtual object in real time and display the three-dimensional virtual object in front of the camera video stream, thereby realizing the real-time fusion of the virtual object and the real background.
Drawings
Other features, objects and advantages of the invention will become more apparent upon reading of the detailed description of non-limiting embodiments with reference to the following drawings:
FIG. 1 is a general flowchart of a real-time illumination estimation and rendering method based on neural network according to the present invention;
FIG. 2 is a flowchart of the operation of the rendering and fusion module in the real-time illumination estimation and rendering system based on neural network according to the present invention;
FIG. 3 is a flowchart illustrating the operation of the neural network module in the real-time illumination estimation and rendering system based on the neural network according to the present invention.
Detailed Description
The present invention will be described in detail with reference to specific examples. The following examples will assist those skilled in the art in further understanding the invention, but are not intended to limit the invention in any way. It should be noted that it would be obvious to those skilled in the art that various changes and modifications can be made without departing from the spirit of the invention. All falling within the scope of the present invention.
As shown in fig. 1, the present invention provides a real-time illumination estimation and rendering system based on a neural network, which includes any one or more of the following modules:
a camera reading module: the camera reading module is responsible for acquiring the real-time camera video stream of the end-side device, continuously acquiring the real-time video stream of the camera, and performing preprocessing operations such as interception, scaling, filtering and the like on the video frame by frame. The preprocessed video sequence frames will be used as input for the illumination estimation. The present invention is capable of calculating illumination in real time at a rate of 60 frames per second.
A neural network module: and adopting a lightweight convolutional neural network as a backbone network for generating a result of the illumination estimation. Network 95% of the convolutional layers are convolved with 1x1, and the depth separable convolutions reduce the amount of computation. Therefore, the method can be ensured to run at the mobile terminal efficiently and in real time. Meanwhile, the learning capability of the network is improved by adopting an optimized inverse residual bottleneck structure. For a video sequence frame input into the network, a 1280-dimensional potential space vector is generated through the backbone network, and then an illumination result, namely a spherical harmonic wave coefficient, is obtained from the potential space vector through the full connection layer. As shown in fig. 3, specifically, a video sequence frame input to the network is subjected to convolution and deconvolution operations by a lightweight convolutional neural network, so that a 1280-dimensional potential space vector can be obtained; wherein, the potential space vector of 1280 dimension can output the spherical harmonic wave coefficient of 27 dimensions after the convolution calculation of the full connection layer. The parameters of the neural network are obtained through training of the paired input images and the corresponding actual spherical harmonic wave coefficients.
A virtual object loading module: the method is used for loading the three-dimensional virtual object as an object rendered by the illumination result. When a three-dimensional virtual object is loaded, the module encapsulates the whole model into a scene object, the scene object comprises a series of scene nodes for storing vertex data and face indexes, and the scene object also stores material information. The virtual object loading module adopts a recursive analysis method to recursively acquire the grid object and process the grid object to obtain vertex data, surface indexes, textures, material data and the like required by rendering.
The spherical harmonic wave real-time rendering and fusion module comprises: the method is used for realizing cross-platform rendering and fusion and rendering the loaded three-dimensional virtual object in real time according to the spherical harmonic wave coefficient. The method comprises the steps of firstly binding Vertex data, surface indexes, textures and material data to a Vertex Array Object (VAO) in combination with a virtual Object loading module, then obtaining information such as a camera video stream and a spherical harmonic wave coefficient, finally updating a shader, and realizing the effect of virtual-real fusion before displaying a rendered three-dimensional virtual Object on the camera video stream.
As shown in fig. 2, the present invention further provides a real-time illumination estimation and rendering method based on the neural network, which adopts the real-time illumination estimation and rendering system based on the neural network to perform real-time illumination estimation and rendering; comprising any one or more of the following steps:
a camera reading step: acquiring a real-time camera video stream of a terminal side device;
neural network reasoning step: generating a result of the illumination estimation;
loading a virtual object: loading a three-dimensional virtual object as an object rendered by an illumination result;
real-time rendering and fusing spherical harmonic waves: and realizing cross-platform rendering and fusion.
Specifically, a binding grid model is loaded firstly, then a neural network is loaded, and then a camera is initialized; the method comprises the steps of obtaining camera video stream after initializing a camera, ending if an ending command is received, entering neural network reasoning if the ending command is not received, generating a result of illumination estimation, continuously updating illumination, motion and visual angle, finally obtaining the camera video stream after rendering and displaying, and ending if the ending command is received.
Those skilled in the art will appreciate that, in addition to implementing the system and its various devices, modules, units provided by the present invention as pure computer readable program code, the system and its various devices, modules, units provided by the present invention can be fully implemented by logically programming method steps in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers, embedded microcontrollers and the like. Therefore, the system and various devices, modules and units thereof provided by the invention can be regarded as a hardware component, and the devices, modules and units included in the system for realizing various functions can also be regarded as structures in the hardware component; means, modules, units for performing the various functions may also be regarded as structures within both software modules and hardware components for performing the method.
The foregoing description of specific embodiments of the present invention has been presented. It is to be understood that the present invention is not limited to the specific embodiments described above, and that various changes or modifications may be made by one skilled in the art within the scope of the appended claims without departing from the spirit of the invention. The embodiments and features of the embodiments of the present application may be combined with each other arbitrarily without conflict.
Claims (4)
1. A real-time illumination estimation and rendering system based on a neural network is characterized by comprising the following modules:
a camera reading module: the method comprises the steps that a real-time camera video stream used for end-side equipment is obtained, and video sequence frames are generated by performing preprocessing operation on the video frame by frame;
the neural network reasoning module: adopting a lightweight convolutional neural network as a backbone network to generate an illumination estimation result, adopting an optimized inverse residual bottleneck structure to improve the learning capacity of the network, inputting video sequence frames into the lightweight convolutional neural network, generating 1280-dimensional potential space vectors through the backbone network, and then obtaining an illumination result, namely a spherical harmonic wave coefficient, from the potential space vectors through a full connection layer;
a virtual object loading module: the system is used for loading a three-dimensional virtual object as an object rendered by an illumination result, and when a three-dimensional virtual object model is loaded, the whole model can be packaged into a scene object, the scene object comprises a series of scene nodes for storing vertex data and surface indexes, and the vertex data, the surface indexes, the textures and the material data required by rendering are obtained by recursively obtaining a grid object and processing the grid object;
the spherical harmonic wave real-time rendering and fusion module comprises: the cross-platform rendering and fusion are realized by rendering the virtual object by utilizing the spherical harmonic wave coefficient, the vertex data, the surface index, the texture and the material data are bound to the vertex group object by combining with the virtual object loading module, then the camera video stream and the spherical harmonic wave coefficient are obtained, finally the shader is updated, and the rendered three-dimensional virtual object is displayed in front of the camera video stream.
2. The neural network-based real-time illumination estimation and rendering system of claim 1, wherein the preprocessed video sequence frames are used as input to the illumination estimation.
3. A real-time illumination estimation and rendering method based on a neural network, characterized in that the real-time illumination estimation and rendering is performed by using the real-time illumination estimation and rendering system based on the neural network of any one of claims 1-2.
4. The real-time illumination estimation and rendering method based on neural network as claimed in claim 3, comprising the steps of:
a camera reading step: acquiring a real-time camera video stream of a terminal side device;
neural network reasoning step: generating a result of the illumination estimation by using the lightweight network;
loading a virtual object: loading a three-dimensional virtual object as an object rendered by an illumination result;
real-time rendering and fusing spherical harmonic waves: and rendering the virtual object by utilizing the spherical harmonic wave coefficient to realize cross-platform rendering and fusion.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110639919.5A CN113096228B (en) | 2021-06-09 | 2021-06-09 | Real-time illumination estimation and rendering method and system based on neural network |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110639919.5A CN113096228B (en) | 2021-06-09 | 2021-06-09 | Real-time illumination estimation and rendering method and system based on neural network |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113096228A CN113096228A (en) | 2021-07-09 |
| CN113096228B true CN113096228B (en) | 2021-08-31 |
Family
ID=76664482
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110639919.5A Active CN113096228B (en) | 2021-06-09 | 2021-06-09 | Real-time illumination estimation and rendering method and system based on neural network |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113096228B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115619989B (en) * | 2022-10-28 | 2024-04-26 | 如你所视(北京)科技有限公司 | Fusion effect diagram generation method and device, electronic equipment and storage medium |
| CN116152419B (en) * | 2023-04-14 | 2023-07-11 | 腾讯科技（深圳）有限公司 | Data processing method, device, equipment and storage medium |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108509887A (en) * | 2018-03-26 | 2018-09-07 | 深圳超多维科技有限公司 | A kind of acquisition ambient lighting information approach, device and electronic equipment |
| CN109410310A (en) * | 2018-10-30 | 2019-03-01 | 安徽虚空位面信息科技有限公司 | A kind of real-time lighting Rendering algorithms based on deep learning network |
| CN109523617A (en) * | 2018-10-15 | 2019-03-26 | 中山大学 | A kind of illumination estimation method based on monocular-camera |
| CN110211061A (en) * | 2019-05-20 | 2019-09-06 | 清华大学 | List depth camera depth map real time enhancing method and device neural network based |
| CN110310224A (en) * | 2019-07-04 | 2019-10-08 | 北京字节跳动网络技术有限公司 | Light efficiency rendering method and device |
| CN110458964A (en) * | 2019-08-21 | 2019-11-15 | 四川大学 | A kind of real-time computing technique of actual environment dynamic illumination |
| CN111698497A (en) * | 2020-06-15 | 2020-09-22 | 中航华东光电有限公司 | Real-time transmission and monitoring method of panoramic display system on AR glasses |
| CN112509109A (en) * | 2020-12-10 | 2021-03-16 | 上海影创信息科技有限公司 | Single-view illumination estimation method based on neural network model |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20070024835A1 (en) * | 2005-08-01 | 2007-02-01 | Kuo-Chun Huang | Method for improving illumination uniformity in exposure process, and exposure apparatus |
| US20090102843A1 (en) * | 2007-10-17 | 2009-04-23 | Microsoft Corporation | Image-based proxy accumulation for realtime soft global illumination |
| CN109448084A (en) * | 2017-08-23 | 2019-03-08 | 当家移动绿色互联网技术集团有限公司 | It is a kind of to carry out the algorithm that light textures are baked and banked up with earth based on voxelization global illumination algorithm |
-
2021
- 2021-06-09 CN CN202110639919.5A patent/CN113096228B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108509887A (en) * | 2018-03-26 | 2018-09-07 | 深圳超多维科技有限公司 | A kind of acquisition ambient lighting information approach, device and electronic equipment |
| CN109523617A (en) * | 2018-10-15 | 2019-03-26 | 中山大学 | A kind of illumination estimation method based on monocular-camera |
| CN109410310A (en) * | 2018-10-30 | 2019-03-01 | 安徽虚空位面信息科技有限公司 | A kind of real-time lighting Rendering algorithms based on deep learning network |
| CN110211061A (en) * | 2019-05-20 | 2019-09-06 | 清华大学 | List depth camera depth map real time enhancing method and device neural network based |
| CN110310224A (en) * | 2019-07-04 | 2019-10-08 | 北京字节跳动网络技术有限公司 | Light efficiency rendering method and device |
| CN110458964A (en) * | 2019-08-21 | 2019-11-15 | 四川大学 | A kind of real-time computing technique of actual environment dynamic illumination |
| CN111698497A (en) * | 2020-06-15 | 2020-09-22 | 中航华东光电有限公司 | Real-time transmission and monitoring method of panoramic display system on AR glasses |
| CN112509109A (en) * | 2020-12-10 | 2021-03-16 | 上海影创信息科技有限公司 | Single-view illumination estimation method based on neural network model |
Non-Patent Citations (4)
| Title |
|---|
| Reconstructing Reflection Maps using a Stacked-CNN for Mixed Reality Rendering;Andrew Chalmers等;《IEEE Transactions on Visualization and Computer Graphics 》;20200612;1-12 * |
| 基于卷积神经网络的人体姿态估计方法研究;张锋;《中国博士学位论文全文数据库.信息科技辑》;20210115(第1期);I138-214 * |
| 基于深度学习的轻量级实时多目标检测系统的研究与实现;张逸凡;《中国优秀硕士学位论文全文数据库.信息科技辑》;20181015(第10期);I138-571 * |
| 基于轻量级网络的实时人脸识别算法研究;张典等;《计算机科学与探索》;20200229;第14卷(第2期);317-324 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113096228A (en) | 2021-07-09 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Liu et al. | Infinite nature: Perpetual view generation of natural scenes from a single image | |
| CN110910486B (en) | Indoor scene illumination estimation model, method and device, storage medium and rendering method | |
| CN113096228B (en) | Real-time illumination estimation and rendering method and system based on neural network | |
| CN115100339B (en) | Image generation method, device, electronic equipment and storage medium | |
| CN114820906B (en) | Image rendering method and device, electronic equipment and storage medium | |
| CN114863038B (en) | Real-time dynamic free visual angle synthesis method and device based on explicit geometric deformation | |
| CN112950471A (en) | Video super-resolution processing method and device, super-resolution reconstruction model and medium | |
| CN110942512B (en) | Indoor scene reconstruction method based on meta-learning | |
| CN116977522A (en) | Rendering method and device of three-dimensional model, computer equipment and storage medium | |
| CN111028279A (en) | Point cloud data processing method and device, electronic equipment and storage medium | |
| Liu et al. | Real-time neural rasterization for large scenes | |
| CN111768467B (en) | Image filling method, device, equipment and storage medium | |
| CN116385667B (en) | Reconstruction method of three-dimensional model, training method and device of texture reconstruction model | |
| CN113066018A (en) | Image enhancement method and related device | |
| CN103116897B (en) | A kind of Three-Dimensional Dynamic data compression based on image space and smoothing method | |
| CN115346000A (en) | Three-dimensional human body reconstruction method and device, computer readable medium and electronic equipment | |
| CN116342782A (en) | Method and apparatus for generating avatar rendering model | |
| CN115797561A (en) | Three-dimensional reconstruction method, device and readable storage medium | |
| CN117576312A (en) | Hand model construction method and device and computer equipment | |
| CN117456128A (en) | Three-dimensional reconstruction method, device, equipment and storage medium | |
| CN113902631A (en) | Image processing method, electronic device, and storage medium | |
| CN115713581A (en) | Dynamic model generation method, device and equipment | |
| CN115578561A (en) | Real-time semantic segmentation method and device based on multi-scale context aggregation network | |
| CN115393498A (en) | Drawing method and device based on implicit light transfer function combination | |
| Zhang et al. | Research on image super-resolution reconstruction based on deep learning |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| PP01 | Preservation of patent right |
Effective date of registration: 20221226 Granted publication date: 20210831 |
|
| PP01 | Preservation of patent right |