CN115457188A

CN115457188A - 3D rendering display method and system based on fixation point

Info

Publication number: CN115457188A
Application number: CN202211139620.4A
Authority: CN
Inventors: 黄来响; 李宁; 马玉广; 苟振兴
Original assignee: Yaozai Shandong Digital Technology Co ltd
Current assignee: Yaozai Shandong Digital Technology Co ltd
Priority date: 2022-09-19
Filing date: 2022-09-19
Publication date: 2022-12-09

Abstract

The invention relates to the technical field of 3D rendering, and particularly discloses a 3D rendering display method and a system based on a fixation point, wherein the method comprises the steps of establishing a training set according to a preset sampling rule, and training a nerve radiation field model based on the training set; extracting a Mesh model in the nerve radiation field model, and rendering and outputting a 3D scene based on the Mesh model; sending the 3D scene to VR head display equipment; acquiring a gaze point region of a user in the VR head display device, and rendering an image of the gaze point region at the 3D scene based on the nerve radiation field model. According to the technical scheme, the Mesh model scene is generated by using the nerve radiation field, the function of predicting the image watched by the user is realized, and the problems of low speed and low definition of the nerve radiation field in reasoning and 3D scene image rendering are solved.

Description

3D rendering display method and system based on fixation point

Technical Field

The invention relates to the technical field of 3D rendering, in particular to a 3D rendering display method and system based on a fixation point.

Background

Neural radiation Field (NeRF) is a depth rendering method, and the main characteristics of the method are scene implicit expression and volume rendering of images. Different from the traditional three-dimensional reconstruction method, the scene is represented as point cloud, grid, voxel and other explicit expressions, the NeRF models the scene into a continuous 5D radiation field and implicitly stores the radiation field in a neural network, a sparse multi-angle image with a position is input to train to obtain a neural radiation field model, and a clear picture under any visual angle can be rendered according to the model. Generally speaking, an implicit rendering process is constructed, the input of which is the position o, the direction d and the corresponding coordinates (x, y, z) of light rays emitted under a certain visual angle, the input is sent into a nerve radiation field to obtain the volume density and the color, and finally, the final image is obtained through volume rendering.

The traditional three-dimensional reconstruction roughly comprises the following processes: sparse point cloud reconstruction, dense point cloud reconstruction, grid reconstruction, texture mapping and material mapping. Experience shows that the modeling tool based on the Photogrammetry strongly depends on the shooting environment, has poor surface reduction on weak textures and smooth areas, and generally depends on manual repair of model grids and textures and endows materials. The traditional three-dimensional modeling project is measured by photography, a Mesh grid model and a mapping method are adopted, the modeling project amount is large, the visual effect of a modeled scene is poor, and the real-time rendering effect depends on hardware.

The main reason why NeRF works well is to use implicit representation of 3D scenes. Implicit representation (implicit scene representation) generally describes scene geometry by using a function, and it can be understood that complex three-dimensional scene representation information is stored in parameters of the function. Because a description function of a 3D scene is often learned, the amount of parameters is small relative to the "display representation" (explicit scene representation) when a large-resolution scene is expressed, and the implicit representation function is a continuous representation, which is more detailed for the scene representation. NeRF achieves the photo-level perspective synthesis effect by using 'implicit representation', selects Volume as the intermediate 3D scene representation, and then achieves the specific perspective photo synthesis effect by Volume rendering. The NeRF realizes learning an implicit Volume expression from a discrete photo set, and then obtains a photo at a certain view angle by using the implicit Volume expression and Volume rendering.

At present, eye tracking technology for gaze point rendering is built in VR head display equipment in the market, and the gaze point rendering technology can improve the display effect by more than half through hardware equipment such as a computer vision algorithm, an optical sensor and the like.

Disclosure of Invention

The present invention provides a 3D rendering display method and system based on a gaze point, so as to solve the problems mentioned in the background art.

In order to achieve the purpose, the invention provides the following technical scheme:

a 3D rendering display method based on a point of regard, the method comprising:

establishing a training set according to a preset sampling rule, and training a nerve radiation field model based on the training set;

extracting a Mesh model in the nerve radiation field model, and rendering and outputting a 3D scene based on the Mesh model;

sending the 3D scene to VR head display equipment;

acquiring a gaze point region of a user in the VR head display device, and rendering an image of the gaze point region at a 3D scene based on the nerve radiation field model.

As a further scheme of the invention: the method comprises the following steps of establishing a training set according to a preset sampling rule, and training a nerve radiation field model based on the training set, wherein the steps comprise:

shooting image information containing shooting parameters at a preset position point to obtain a training set; the shooting parameters comprise camera pose, camera internal parameters and scene range information;

sequentially reading images in the training set, and inputting the images into a preset multilayer perceptron to obtain the volume density and the RGB color values;

projecting the volume density and the RGB color value into an image according to a volume rendering technology to obtain a complete image;

a neural radiation field model is trained based on the complete image.

As a further scheme of the invention: the step of extracting a Mesh model in the nerve radiation field model and rendering and outputting a 3D scene based on the Mesh model comprises the following steps of:

extracting Mesh grids according to the volume density in the nerve radiation field model; the Mesh model supports rasterization rendering;

and importing the generated Mesh grid into a 3D engine Unity project, receiving a chartlet and an illumination parameter input by a user, and rendering and outputting a 3D scene.

As a further scheme of the invention: the step of acquiring a gaze point region of a user in the VR head display device, rendering an image of the gaze point region at a 3D scene based on the nerve radiation field model, comprising:

acquiring the fixation point information of a user in VR head display equipment based on an eye movement tracking technology, and determining a fixation point area;

acquiring coordinate information of a fixation point area, and inputting the coordinate information into a multilayer perception layer of a nerve radiation field neural network model;

predicting an image of the gaze point region based on a nerve radiation field model;

rendering the predicted image in the VR head display device into a gaze point region within the user's field of view.

The technical scheme of the invention also provides a 3D rendering display system based on the fixation point, which comprises:

the model training module is used for establishing a training set according to a preset sampling rule and training the nerve radiation field model based on the training set;

the scene rendering module is used for extracting a Mesh model in the nerve radiation field model and rendering and outputting a 3D scene based on the Mesh model;

a scene sending module, configured to send the 3D scene to a VR head-mounted display device;

an image rendering module, configured to acquire a gaze point region of a user in the VR head display device, and render an image of the gaze point region at the 3D scene based on the nerve radiation field model.

As a further scheme of the invention: the model training module comprises:

the training set generating unit is used for shooting image information containing shooting parameters at a preset position point to obtain a training set; the shooting parameters comprise camera pose, camera internal parameters and scene range information;

the parameter determining unit is used for sequentially reading the images in the training set and inputting the images into a preset multilayer perceptron to obtain the volume density and the RGB color values;

the projection unit is used for projecting the volume density and the RGB color value into an image according to a volume rendering technology to obtain a complete image;

and the first execution unit is used for training the nerve radiation field model based on the complete image.

As a further scheme of the invention: the scene rendering module includes:

the grid extraction unit is used for extracting Mesh grids according to the volume density in the nerve radiation field model; the Mesh model supports rasterization rendering;

and the output unit is used for importing the generated Mesh grid into the 3D engine Unity project, receiving the chartlet and the illumination parameter input by the user, and rendering and outputting the 3D scene.

As a further scheme of the invention: the image rendering module includes:

the area positioning unit is used for acquiring the fixation point information of the user in the VR head display equipment based on an eye tracking technology and determining a fixation point area;

the system comprises a coordinate acquisition unit, a neural network model and a data processing unit, wherein the coordinate acquisition unit is used for acquiring coordinate information of a fixation point area and inputting the coordinate information into a multilayer perception layer of the neural network model of the neural radiation field;

an image prediction unit for predicting an image of the gaze point region based on the nerve radiation field model;

a second execution unit to render the predicted image into a gaze point region within a user's field of view in the VR head display device.

Compared with the prior art, the invention has the beneficial effects that: according to the technical scheme, the Mesh model scene is generated by using the nerve radiation field, the function of predicting the image watched by the user is realized, and the problems of low speed and low definition of the nerve radiation field in reasoning and 3D scene image rendering are solved. The key point is that a Mesh model and a watching prediction image are combined, a 3D scene image is generated in a rendering mode in VR equipment, and a user can view a high-definition real-time rendered 3D image in VR head display equipment.

Drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention.

Fig. 1 is a flowchart of a 3D rendering display method based on a gaze point.

Fig. 2 is a first sub-flowchart block diagram of a 3D rendering display method based on a point of regard.

Fig. 3 is a second sub-flow block diagram of a 3D rendering display method based on a gaze point.

Fig. 4 is a third sub-flow block diagram of a 3D rendering display method based on a gaze point.

Fig. 5 is a block diagram illustrating a composition structure of a 3D rendering display system based on a gaze point.

Detailed Description

In order to make the technical problems, technical solutions and advantageous effects to be solved by the present invention more clearly apparent, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

Example 1

Fig. 1 is a flow chart of a 3D rendering display method based on a gaze point, and in an embodiment of the present invention, the 3D rendering display method based on the gaze point includes:

step S100: establishing a training set according to a preset sampling rule, and training a nerve radiation field model based on the training set;

step S200: extracting a Mesh model in the nerve radiation field model, and rendering and outputting a 3D scene based on the Mesh model;

step S300: sending the 3D scene to VR head display equipment;

step S400: acquiring a gaze point region of a user in the VR head display device, and rendering an image of the gaze point region at a 3D scene based on the nerve radiation field model.

In an example of the technical scheme, the Mesh model scene is generated by using the nervus radiation field NeRF, the 3D modeling and rendering technology based on the neural rendering is improved by combining the method of the gaze point image prediction, and the rendering speed of the NeRF image and the definition of the image are innovatively improved.

Aiming at the problems of low NeRF rendering speed and overlarge model, the method is based on the trained NeRF model, extracts the density sigma of the NeRF model and extracts the Mesh model in advance. After that, the Mesh grid model is led into the Unity project to be matched with the chartlet and the illumination to generate a complete 3D scene, and a user can view the real-time rendered 3D scene in VR head display equipment. According to the invention, the Mesh model with a smaller volume is generated through the NeRF model, so that the rendering speed of the 3D scene image is accelerated.

Aiming at the problem of insufficient definition of a NeRF rendering picture, the invention acquires the fixation point information of a user based on a VR eye movement tracking fixation point rendering technology, only inputs the coordinates of a fixation point area into a NeRF neural network model, and then predicts the fixation point image through the NeRF model. According to the invention, the image generated by NeRF is rendered to the gaze point area of the user in the VR head display device, so that the definition of the rendered image in the visual field of the user is ensured.

Further, from a three-dimensional reconstruction perspective, neRF has several significant disadvantages:

1. the training and reasoning speed is slow, images with 1920-1080 resolution are output, the NeRF reasoning speed is 50 s/frame, the practical requirement is that more than 30 frames/s is required, the difference is 1500 times, and more than 90 frames/s are required in VR head display equipment. The 1 1080P graph reasoning time exceeds 50s, and the modeling time of an object generally needs more than 2 days.

2. The rendered picture is not clear enough and the detail texture cannot be recovered.

3. Implicit expression (explicit scene representation) cannot directly lead to a graphics tool, and does not support explicit use, such as CAD scene collocation design. And NeRF can only restore the illumination of a shooting scene, and cannot support the scene application of environment illumination change.

The reason that the NeRF speed is low is in several aspects, firstly, the number of effective pixels is small, the effective pixels of the generated 2D image is less than 1/3, the effective pixels can be obtained quickly, and the reasoning speed can be improved. Second, 192 points are sampled with few valid voxels, and only the density σ of points near the surface is large, and no inference is necessary for other points. Thirdly, the network reasoning speed is low, the color and density of 1 individual pixel can be obtained only by reasoning in a 12-layer fully-connected network, and the reasoning speed can be greatly accelerated by optimizing the performance. Optimization can be performed by a space time changing method, including methods such as FastNeRF and plenOctree, but the storage space is too large to reach the size of 200M to 1G only by simply remembering the output result of the NeRF, and the large model is not practical.

The NeRF has no more successful commercial application at present, and has the problems of low training speed, low rendering speed, only being used for static scenes, poor generalization performance, large number of visual angles, difficulty in fusion with the traditional rendering pipeline and the like.

Fig. 2 is a first sub-flow block diagram of a 3D rendering display method based on a fixation point, where the training set is established according to a preset sampling rule, and the step of training a neural radiation field model based on the training set includes:

step S101: shooting image information containing shooting parameters at a preset position point to obtain a training set; the shooting parameters comprise camera pose, camera internal parameters and range information of a scene;

the method takes pictures of the same scene from different positions, and simultaneously records the camera pose, camera internal parameters and scene range information of the pictures, and uses the pictures as an input data set for training the nerve radiation field.

Step S102: sequentially reading images in the training set, and inputting the images into a preset multilayer perceptron to obtain the volume density and the RGB color values;

the present invention inputs the image data set into a Multi-Layer perceptron MLP (Multi-Layer Perception), and outputs a Volume intensity and RGB color values.

Step S103: projecting the volume density and the RGB color value into an image according to a volume rendering technology to obtain a complete image;

the invention takes different positions and uses volume rendering technology (volume rendering) to project the output color and volume density on the image, thereby synthesizing the values into a complete image.

Step S104: training a nerve radiation field model based on the complete image;

the present invention trains an optimized neural radiation field scene representation by minimizing the difference between rendered synthetic, real images. Specifically, the invention optimizes a deep fully-connected neural network, which is also called a multilayer perceptron (MLP), and the invention expresses such a function through the MLP: and outputting the RGB color related to the volume density and the visual angle according to 5D coordinate regression.

After the neural radiation field training is completed, a model represented by the weight of a plurality of layers of perception layers is obtained. The model contains only the information of the scene and does not have the capability of generating a picture of another scene. The nerve radiation field represents the scene as a volume density σ and a color value c for any point in space. After a representation of a scene in NeRF form is available, the scene may be rendered to generate a simulated picture of the new perspective. The present invention uses the classical volume rendering principle to solve the color of any light passing through the scene, rendering a synthetic new image.

Although the nerve radiation field implicitly represents the 3D model, some method can be adopted to extract the model of the nerve radiation field explicitly and perform visual analysis.

Fig. 3 is a second sub-flow block diagram of a 3D rendering display method based on a gaze point, where the step of extracting a Mesh model in a nerve radiation field model and rendering and outputting a 3D scene based on the Mesh model includes:

step S201: extracting Mesh grids according to the volume density in the nerve radiation field model; the Mesh model supports rasterization rendering;

the Mesh model is extracted in advance from the volume density sigma of the nerve radiation field model generated in the figure 1. The Mesh model supports rasterization rendering, can quickly obtain 2D effective pixels, and can greatly improve rendering speed only aiming at effective pixel rendering.

Step S202: importing the generated Mesh grid into a 3D engine Unity project, receiving a chartlet and an illumination parameter input by a user, and rendering and outputting a 3D scene;

the generated Mesh grid is led into a 3D engine Unity project, and a 3D scene with a definite shape is formed through rendering output in cooperation with charting and illumination.

In an example of the technical scheme of the invention, all 3D scenes are rendered in real time in VR head display equipment, and the rendering speed of the 3D scene images is greatly accelerated because the 3D scenes are generated by a Mesh grid model with a small volume.

Fig. 4 is a third sub-flow block diagram of a 3D rendering display method based on a gaze point, where the step of acquiring a gaze point region of a user in the VR head display device and rendering an image of the gaze point region at a 3D scene based on the nerve radiation field model includes:

step S401: acquiring the fixation point information of a user in VR head display equipment based on an eye movement tracking technology, and determining a fixation point area;

according to the invention, the gaze point information of the user is obtained in the VR head display device through an eye tracking technology.

Step S402: acquiring coordinate information of a fixation point area, and inputting the coordinate information into a multilayer perception layer of a neural radiation field neural network model;

the invention inputs the coordinate information of the fixation point area into a multilayer perception layer of a nerve radiation field neural network model.

Step S403: predicting an image of the gaze point region based on a nerve radiation field model;

the invention predicts the image of the fixation point area through the nerve radiation field model.

Step S404: rendering the predicted image in a gaze point region within a user's field of view in a VR head display device;

the invention renders the predicted image to a gazing point area in the visual field of a user in a VR head display device.

In one example of the technical solution of the present invention, a 3D scene with a definite shape is rendered and output by using a Mesh model with a relatively small volume. This step is rendered very fast in the VR head-mounted display device to ensure that the user can view the 3D image in real time. Meanwhile, the invention renders the image generated by the neural field to the user so as to ensure that the user can watch the 3D image.

As a preferred embodiment of the technical scheme of the invention, the sampling efficiency of the nerve radiation field is improved by a scheme of sampling effective voxels. A large number of points in space are not near the surface of the object and are randomly picked

The efficiency of the sample is very low, and the Mesh model can be used for quickly obtaining the surface information of the object, so that the sampling is only carried out near the surface, and the method has the advantages that

If, the efficiency of sampling can promote greatly, can not need thick sampling, direct accurate sampling, and accurate sampling also only needs 32 points can reach same effect.

As a preferred embodiment of the technical scheme of the invention, a network optimization scheme is adopted to optimize the training process of the nerve radiation field. Model table of nerve radiation field through 12-layer neural network

It shows that a large number of redundant nodes are contained, and the network can be accelerated by network pruning or model distillation. Method for pruning using network

Is superior to model distillation in the aspect of the optimization of the neural radiation field network. 80% of nodes can be optimized through the L1 regularization, the network scale is reduced to 1/5 of the original scale, and the effect is kept unchanged.

To facilitate a better understanding of the above solutions for those skilled in the art, the technical terms used in the present invention are described as follows:

the NeRF: neural Radiance Field Neural radiation Field. The method is one of the current research fields of the most fire and heat, and the problem to be solved is how to generate the images under new view angles given some shot images, which is different from the traditional three-dimensional reconstruction method that the scene is expressed as point clouds, grids, voxels and other explicit expressions, and the NeRF scene is modeled into a continuous 5D radiation field and is stored in a neural network implicitly. In colloquial, the nerve radiation field is an implicit rendering process, and the input is the position of light emitted under a certain visual angle. Direction d and corresponding coordinates (x, y, z). Obtaining the volume density and the color through a nerve radiation field F theta, and finally obtaining a final image through rendering.

Mesh model: the grid three-dimensional model is mainly used for three-dimensional reconstruction. Mesh is a component in Unity, called a grid component. Generally speaking, mesh refers to a Mesh of a model, a 3D model is formed by splicing polygons, and a complex polygon is actually formed by splicing a plurality of triangular faces. The surface of a 3D model is made up of a plurality of triangular faces connected to each other. In the three-dimensional space, the set of points constituting the triangular faces and the sides of the triangle is the Mesh.

Implicit scene representation (implicitscreen representation): the precursor to deep learning based rendering is the implicit representation of three-dimensional scenes using neural networks. Many 3D-aware image generation methods use voxels, meshes, point clouds, etc. to represent three-dimensional scenes, typically based on a convolutional architecture. On the other hand, on CVPR 2019, work begins to occur to represent a three-dimensional scene using a neural network to fit a scalar function.

MLP: a Multi-Layer persistence Multi-Layer perceptron. Is a feedforward artificial neural network model that maps multiple data sets of an input onto a single data set of an output. MLP is the most classical model of neural networks, and variants of neural networks are probabilistic neural networks, convolutional neural networks, temporal recurrent neural networks, and the like.

VR: virtual Reality. The virtual reality technology is a brand-new practical technology developed in the 20 th century, and the virtual reality technology is mainly a computer technology, and utilizes various new technologies such as a three-dimensional graphic technology, a sensing technology, a simulation technology and a display technology including artificial intelligence and the like to generate a vivid virtual world with multi-sensory experience by means of VR equipment.

3D Engine Unity: a real-time 3D interactive content authoring and operation platform. Creatives including game development, art, architecture, automobile design, and movie are realized by Unity. The Unity platform provides a complete set of complete software solutions that can be used to author, operate and render any real-time interactive 2D and 3D content, and the support platforms include cell phones, tablets, PCs, game consoles, augmented reality and virtual reality devices.

Example 2

Fig. 5 is a block diagram of a composition structure of a 3D rendering display system based on a gaze point, in an embodiment of the present invention, a 3D rendering display system based on a gaze point includes:

the model training module 11 is used for establishing a training set according to a preset sampling rule and training a nerve radiation field model based on the training set;

the scene rendering module 12 is configured to extract a Mesh model in the nerve radiation field model, and render and output a 3D scene based on the Mesh model;

a scene sending module 13, configured to send the 3D scene to a VR head display device;

an image rendering module 14, configured to obtain a gaze point region of a user in the VR head display device, and render an image of the gaze point region at the 3D scene based on the nerve radiation field model.

The model training module comprises:

the training set generating unit is used for shooting image information containing shooting parameters at a preset position point to obtain a training set; the shooting parameters comprise camera pose, camera internal parameters and range information of a scene;

The scene rendering module includes:

the Mesh extraction unit is used for extracting Mesh meshes according to the volume density in the nerve radiation field model; the Mesh model supports rasterization rendering;

8. The point-of-gaze based 3D rendering display system of claim 5, wherein the image rendering module comprises:

the coordinate acquisition unit is used for acquiring coordinate information of the fixation point area and inputting the coordinate information into a multilayer perception layer of the neural network model of the nerve radiation field;

an image prediction unit for predicting an image of the gaze point region based on the neural radiation field model;

a second execution unit to render the predicted image into a gaze point region within a user field of view in the VR head display device.

The functions that can be implemented by the gaze point based 3D rendering display method are all performed by a computer device comprising one or more processors and one or more memories, wherein at least one program code is stored in the one or more memories, and loaded and executed by the one or more processors to implement the functions of the gaze point based 3D rendering display method.

The processor fetches instructions and analyzes the instructions one by one from the memory, then completes corresponding operations according to the instruction requirements, generates a series of control commands, enables all parts of the computer to automatically, continuously and coordinately act to form an organic whole, realizes the input of programs, the input of data, the operation and the output of results, and the arithmetic operation or the logic operation generated in the process is completed by the arithmetic unit; the Memory comprises a Read-Only Memory (ROM) for storing a computer program, and a protection device is arranged outside the Memory.

Illustratively, the computer program may be partitioned into one or more modules, stored in memory and executed by a processor, to implement the invention. One or more of the modules may be a series of computer program instruction segments capable of performing certain functions, which are used to describe the execution of the computer program in the terminal device.

Those skilled in the art will appreciate that the above description of the service device is merely exemplary and not limiting of the terminal device, and may include more or less components than those described, or combine certain components, or different components, such as may include input output devices, network access devices, buses, etc.

The Processor may be a Central Processing Unit (CPU), other general purpose Processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), an off-the-shelf Programmable Gate Array (FPGA) or other Programmable logic device, discrete Gate or transistor logic, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may be any conventional processor or the like, which is the control center of the terminal equipment and connects the various parts of the entire user terminal using various interfaces and lines.

The memory may be used to store computer programs and/or modules, and the processor may implement various functions of the terminal device by operating or executing the computer programs and/or modules stored in the memory and calling data stored in the memory. The memory mainly comprises a storage program area and a storage data area, wherein the storage program area can store an operating system, application programs (such as an information acquisition template display function, a product information publishing function and the like) required by at least one function and the like; the storage data area may store data created according to the use of the berth-state display system (e.g., product information acquisition templates corresponding to different product types, product information that needs to be issued by different product providers, etc.), and the like. In addition, the memory may include high speed random access memory, and may also include non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), at least one magnetic disk storage device, a Flash memory device, or other volatile solid state storage device.

The terminal device integrated modules/units, if implemented in the form of software functional units and sold or used as separate products, may be stored in a computer readable storage medium. Based on such understanding, all or part of the modules/units in the system according to the above embodiment may be implemented by a computer program, which may be stored in a computer-readable storage medium and used by a processor to implement the functions of the embodiments of the system. Wherein the computer program comprises computer program code, which may be in the form of source code, object code, an executable file or some intermediate form, etc. The computer readable medium may include: any entity or device capable of carrying computer program code, recording medium, U.S. disk, removable hard disk, magnetic diskette, optical disk, computer Memory, read-Only Memory (ROM), random Access Memory (RAM), electrical carrier wave signal, telecommunications signal, software distribution medium, etc.

It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrases "comprising a component of' 8230; \8230;" does not exclude the presence of another like element in a process, method, article, or apparatus that comprises the element.

The above description is only a preferred embodiment of the present invention, and not intended to limit the scope of the present invention, and all modifications of equivalent structures and equivalent processes, which are made by using the contents of the present specification and the accompanying drawings, or directly or indirectly applied to other related technical fields, are included in the scope of the present invention.

Claims

1. A3D rendering display method based on a fixation point is characterized by comprising the following steps:

sending the 3D scene to VR head display equipment;

2. The method for 3D rendering and displaying based on a point of regard according to claim 1, wherein the training set is established according to a preset sampling rule, and the step of training the neural radiation field model based on the training set comprises:

a neural radiation field model is trained based on the complete image.

3. The gaze point-based 3D rendering and displaying method according to claim 1, wherein the step of extracting a Mesh model in the neural radiation field model and rendering and outputting a 3D scene based on the Mesh model comprises:

4. The method for gaze point based 3D rendering display of claim 1, wherein the step of obtaining a user's gaze point area in the VR head display device, rendering an image of the gaze point area at the 3D scene based on the neuroradiation field model comprises:

the predicted image is rendered in the VR head display device into a gaze point area within the user's field of view.

5. A point-of-gaze based 3D rendering display system, the system comprising:

an image rendering module, configured to acquire a gaze point region of a user in the VR head display device, and render an image of the gaze point region at a 3D scene based on the neural radiation field model.

6. The point-of-gaze based 3D rendering display system of claim 5, wherein the model training module comprises:

7. The point-of-gaze based 3D rendering display system of claim 5, wherein the scene rendering module comprises: