CN117319631A - Light field display system and image rendering method based on human eye tracking - Google Patents
Light field display system and image rendering method based on human eye tracking Download PDFInfo
- Publication number
- CN117319631A CN117319631A CN202311235605.4A CN202311235605A CN117319631A CN 117319631 A CN117319631 A CN 117319631A CN 202311235605 A CN202311235605 A CN 202311235605A CN 117319631 A CN117319631 A CN 117319631A
- Authority
- CN
- China
- Prior art keywords
- light field
- rendering
- display
- unit
- human eye
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000009877 rendering Methods 0.000 title claims abstract description 113
- 238000000034 method Methods 0.000 title claims abstract description 40
- 210000001508 eye Anatomy 0.000 claims abstract description 86
- 230000005855 radiation Effects 0.000 claims abstract description 24
- 239000004973 liquid crystal related substance Substances 0.000 claims abstract description 18
- 238000013461 design Methods 0.000 claims abstract description 12
- 210000005036 nerve Anatomy 0.000 claims abstract description 10
- 230000000007 visual effect Effects 0.000 claims description 25
- 238000012549 training Methods 0.000 claims description 14
- 230000001537 neural effect Effects 0.000 claims description 11
- 238000005070 sampling Methods 0.000 claims description 10
- 230000003993 interaction Effects 0.000 claims description 9
- 210000005252 bulbus oculi Anatomy 0.000 claims description 8
- 230000008569 process Effects 0.000 claims description 8
- 230000006870 function Effects 0.000 claims description 7
- 230000001360 synchronised effect Effects 0.000 claims description 5
- 238000004891 communication Methods 0.000 claims description 4
- 230000006854 communication Effects 0.000 claims description 4
- 238000005192 partition Methods 0.000 claims description 3
- 238000012545 processing Methods 0.000 claims description 3
- 238000013135 deep learning Methods 0.000 claims description 2
- 238000007781 pre-processing Methods 0.000 claims description 2
- 238000005516 engineering process Methods 0.000 abstract description 29
- 230000000694 effects Effects 0.000 abstract description 13
- 238000007654 immersion Methods 0.000 abstract description 9
- 239000011521 glass Substances 0.000 description 10
- 230000004888 barrier function Effects 0.000 description 8
- 230000003287 optical effect Effects 0.000 description 8
- 238000010586 diagram Methods 0.000 description 6
- 238000003384 imaging method Methods 0.000 description 6
- 208000003464 asthenopia Diseases 0.000 description 3
- 230000008901 benefit Effects 0.000 description 3
- 230000001133 acceleration Effects 0.000 description 2
- 238000013528 artificial neural network Methods 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 2
- 210000004556 brain Anatomy 0.000 description 2
- 230000006872 improvement Effects 0.000 description 2
- 230000011218 segmentation Effects 0.000 description 2
- 208000012886 Vertigo Diseases 0.000 description 1
- 230000003044 adaptive effect Effects 0.000 description 1
- 230000004075 alteration Effects 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 239000002131 composite material Substances 0.000 description 1
- 230000006835 compression Effects 0.000 description 1
- 238000007906 compression Methods 0.000 description 1
- 230000001186 cumulative effect Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 208000002173 dizziness Diseases 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 230000004438 eyesight Effects 0.000 description 1
- 230000004907 flux Effects 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012634 optical imaging Methods 0.000 description 1
- 239000002245 particle Substances 0.000 description 1
- 230000005043 peripheral vision Effects 0.000 description 1
- 238000004886 process control Methods 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000002834 transmittance Methods 0.000 description 1
- 231100000889 vertigo Toxicity 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/30—Image reproducers
- H04N13/398—Synchronisation thereof; Control thereof
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
- H04N13/106—Processing image signals
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/30—Image reproducers
- H04N13/366—Image reproducers using viewer tracking
- H04N13/368—Image reproducers using viewer tracking for two or more viewers
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/30—Image reproducers
- H04N13/366—Image reproducers using viewer tracking
- H04N13/383—Image reproducers using viewer tracking for tracking with gaze detection, i.e. detecting the lines of sight of the viewer's eyes
Abstract
The application discloses a light field display system and an image rendering method based on human eye tracking, wherein the positions of eyes of a viewer are positioned and tracked, and basic information such as real-time viewing angles, directions and the like required by light field reconstruction is calculated, so that the real-time rendering of light field images is completed through a nerve radiation field body rendering algorithm. By combining the human eye tracking technology and the nerve radiation field rendering technology, the spatial bandwidth product of the light field display is greatly improved, and simultaneous watching of multiple persons is supported, so that the design dependence of the light field display on the high pixel density of the liquid crystal panel is reduced, the stereoscopic display effect of the light field display is improved, and a viewer can obtain better stereoscopic immersion feeling at different watching positions. In addition, the proposed nerve radiation field is adopted to reconstruct the light field image, so that the problem of depth inversion is effectively avoided, and meanwhile, the data volume and the computational complexity of a light field image rendering algorithm can be reduced.
Description
Technical Field
The application relates to the fields of image processing, stereoscopic display and artificial intelligence, in particular to a light field display system based on human eye tracking and an image rendering method.
Background
Current stereoscopic display technology generally adopts a Parallax Barrier PB (Parallax Barrier) technology to generate Parallax so as to allow a viewer to obtain stereoscopic immersion. The stereoscopic display technology can be roughly divided into two main categories, namely auxiliary and non-auxiliary, and the stereoscopic display with auxiliary can be further divided into switch glasses, polarized glasses, VR helmets, chromatic aberration glasses and the like. Unassisted and can be divided into parallax barrier type and true three-dimensional displays. Parallax barrier type mainly comprises a lenticular lens grating, a slit grating, a directional backlight (Directional Backlight) technology and the like, and true three-dimensional display is further divided into integrated imaging, holographic display, light field display, volume display and other technologies. Among them, parallax barrier type stereoscopic display technology is the most mature display technology, but once the process control is poor, the stereoscopic display technology can cause larger stereoscopic crosstalk. The optical field display system is a geometrical optical system, which skillfully uses crosstalk, and the initial optical field image acquisition path is obtained by directly imaging through a lens array, but the direct display has the problem of depth inversion.
In the existing auxiliary stereoscopic display mode, the driver of the stereoscopic display will alternately render the left and right eye images, if the first frame is the left eye image, then the latter frame corresponds to the right eye image. The viewer wears liquid crystal shutter glasses, the shutter glasses are synchronized with the display card and the display in a wired or wireless manner, when the left eye image is displayed on the display, the glasses open the shutter of the left lens and simultaneously close the shutter of the right lens, and when the right eye image is displayed on the display, the glasses open the shutter of the right lens and simultaneously close the shutter of the left lens. Since the refresh rate is fast, the view of a certain eye which is not seen at the moment is synthesized by the brain according to the vision temporary storage effect, so that the stereoscopic image is watched from the display only by wearing the shutter type stereoscopic glasses interface in the coverage range of the synchronous signal.
The auxiliary stereoscopic display requires that the left and right lenses of the stereoscopic shutter glasses must be synchronized with the corresponding projection image signals, and a higher stereoscopic display refresh rate is required to obtain a good stereoscopic silence effect. The stereoscopic display technology has the advantages of low cost and good stereoscopic immersion, can realize seamless switching of the plane image and the stereoscopic image, and has the disadvantages of low stereoscopic brightness and easy visual fatigue.
The unassisted autostereoscopic display technology is generally called as "naked eye 3D display" technology, that is, two images with parallax can be seen by left and right eyes from a display screen without any tool, and reflected to the brain, so that a viewer can generate stereoscopic immersion. The unassisted stereoscopic display also utilizes the parallax principle of human eyes, and different display pictures are respectively provided for the left eye and the right eye of a viewer, so that the stereoscopic visual viewing effect is achieved. Currently, the unassisted stereoscopic display technology is mainly a multi-view stereoscopic display technology, and is commonly used in public places for displaying and propaganda because an observer can wear no glasses. However, the multi-viewpoint autostereoscopic display technique has disadvantages of reduced resolution, limited viewing angle, low brightness, large crosstalk, stereoscopic vertigo, and the like, and has relatively poor display effects.
The directional backlight technology is a practical stereo display technology in the existing unassisted stereo display technology, the directional backlight stereo display technology is matched with two groups of LEDs, and is matched with a fast-response LCD panel and a driving method, so that 3D content enters left and right eye interchange images of a viewer in a Sequential mode to generate parallax, the eyes of the viewer feel 3D stereo effect, and communication and interaction based on mobile equipment are greatly enhanced. The display device has the advantages that resolution and light transmittance are guaranteed, an existing display design architecture cannot be affected, and the 3D display effect is excellent. The defect is that the technology is still under development and the product is not mature.
Compared with the traditional parallax barrier type stereoscopic display technology, the light field display technology has the advantages of good stereoscopic immersion sense, multiple display information dimensions, no dizziness sense, high resolution, quick refreshing and the like, is considered as one of the most promising next generation reserve display technologies, and is a key technology for realizing large-scale stereoscopic display.
In summary, the prior art has the following disadvantages:
1. parallax barrier type auto-stereoscopic displays are limited by the resolution of the liquid crystal display panel, and the spatial bandwidth product is difficult to increase, and generally only about 40% is achieved.
2. The parallax barrier type autostereoscopic display generates stereoscopic crosstalk in the stereoscopic display process due to the optical structural characteristics thereof, and the parallax barrier type autostereoscopic display needs to modulate light to the left and right eye positions of a user due to the optical structural characteristics thereof, thereby causing lower brightness of the autostereoscopic display and aggravating the asthenopia phenomenon.
3. The dense viewpoint stereoscopic display technology has higher requirement on the display density of the liquid crystal panel, difficult increase of the spatial bandwidth product and large data volume of image rendering, so that the rendering speed of the stereoscopic image is slow.
4. The light field display adopts the traditional light ray chasing pipeline rendering scheme, so that the algorithm rendering process is complex, the speed is low, the accuracy is low, the depth inversion phenomenon can be generated, and the process of sampling and reconstructing the target model to be displayed is complex, thereby the display efficiency of the display is lower and the three-dimensional performance of the three-dimensional display is deteriorated.
Disclosure of Invention
In order to solve the problems, the application provides a light field display system based on human eye tracking and an image rendering method.
According to a first aspect of the present application, there is provided a light field display system based on human eye tracking, comprising:
the display control unit is used for providing three-dimensional display data and man-machine interaction related interaction information for the light field display;
the light field rendering unit is connected with the display control unit and is used for performing light field image real-time rendering according to the light field model and human eye position information output by the video input interface in the display control unit;
the driving units are connected with the light field rendering units, the driving units are in a modularized design, and the display resolution of each driving module is 2K multiplied by 1K or 4K multiplied by 2K and is used for carrying out partition refreshing and synchronous control on the high-density liquid crystal display panel;
the display unit is used for providing a display carrier for the presentation of the light field information and constructing a high-density display physical model with the resolution of 16K multiplied by 8K;
the high-speed eyeball three-dimensional tracking unit is connected with the display unit and is used for positioning and tracking the human eye position information of a viewer in real time and outputting the human eye position information from the serial bus according to a preset communication protocol;
the visual driving unit is connected with the light field rendering unit and the high-speed eyeball three-dimensional tracking unit and is used for bridging the visual angle information of the viewer with the light field rendering model so as to output expected visual angle rendering parameters for the light field rendering unit according to the three-dimensional position information of the human eye of the viewer.
The light field display system based on eye tracking is characterized in that the display control unit is a 2D interaction unit, a user can interact with an operating system by being provided with a 2D display screen on one hand, and can also switch between 2D and 3D switching control by a mouse or gestures by setting the working mode of the light field display to be a 2D working mode.
The light field display system based on human eye tracking is characterized in that the light field rendering unit is a parallel rendering core with high stability and high instantaneity, and adopts a nerve radiation field deep learning algorithm to conduct rendering.
The light field display system based on human eye tracking is characterized in that the driving unit is a Tcon driving module with independent driving and array synchronization functions, and the resolution of the driving board is 1920×1080 or 3840×2160.
The light field display system based on human eye tracking is characterized in that the visual driving unit provides visual angle information of image rendering for the light field rendering unit, wherein the image rendering adopts an inverse sequence ray integral rendering technology.
According to a second aspect of the present application, the present application provides a light field image rendering method based on eye tracking, which is characterized by comprising two stages of model training and model reasoning, and comprises the following steps:
a step of acquiring a target scene: the target scene to be displayed is used for providing an original training data set for the light field image rendering unit;
and (3) self-adaptive scene acquisition: sampling the target scene by adopting a multi-scale self-adaptive method according to the target scene, and generating a digital image signal to be output;
and a camera pose estimation step: estimating the pose of a shooting camera according to the sampled scene information;
training a nerve radiation field model: preprocessing the digital waveform signals output in the data acquisition step and outputting processing results;
obtaining an implicit expression step: according to a neural radiation field algorithm, training a model by adopting a fully connected network to obtain node information and weight parameters of the model;
a light field reproduction step: and converting the human eye position signal in the neural radiation field rendering step into visual angle information by taking parameters in the model implicit expression as network weights, so as to complete model reasoning. And rasterizing the data output according to the human eye position rendering step and converting the data into a display screen coordinate system so as to achieve the aim of light field display.
The light field image rendering method based on eye tracking is characterized in that in the target scene acquisition step, the data can be a real scene to be displayed or a virtual scene in a game or a three-dimensional model in the animation field.
The light field image rendering method based on human eye tracking is characterized in that the data acquisition step can sample a scene in real time through an aircraft, sample data of a light field camera or virtually sample a three-dimensional model by adopting a virtual camera.
The light field image rendering method based on human eye tracking is characterized in that in the step of obtaining model parameters, a reverse rendering algorithm is adopted in the rendering process, so that a front-looking observed image is obtained.
The light field image rendering method based on human eye tracking is characterized in that the light field rendering algorithm is a real-time rendering algorithm
The beneficial effects of this application are:
according to the light field display system and the image rendering method based on human eye tracking, the positions of eyes of a viewer are tracked, and corresponding image rendering viewpoint information is calculated, so that light field images are rendered through a neural radiation field reconstruction algorithm. The spatial bandwidth product of the light field stereoscopic display is greatly improved by combining with human eyes, and simultaneous watching of multiple persons is supported, so that the design requirement of the light field display on the high density of pixels of the liquid crystal panel is reduced, the display effect of the light field display is improved, a viewer can obtain better stereoscopic immersion feeling at different watching positions, and meanwhile, the data volume and the computational complexity of a light field image rendering algorithm can be reduced.
Drawings
FIG. 1 is a schematic diagram of a light field display system based on human eye tracking in accordance with the present application;
FIG. 2 is a schematic diagram of a light field display system based on eye tracking;
FIG. 3 is a schematic diagram of a driving circuit of a light field display system based on human eye tracking;
FIG. 4 is a flow chart of a light field image rendering method based on human eye tracking;
FIG. 5 is a schematic diagram of a light field acquisition and reproduction method based on human eye tracking;
FIG. 6 is a schematic diagram of a light field image rendering method based on human eye tracking;
Detailed Description
The present application is described in further detail below with reference to the accompanying drawings by way of specific embodiments.
Referring to fig. 1, the present application discloses a light field display system (hereinafter referred to as light field display) based on human eye tracking, which includes a display control unit 10, a light field rendering unit 11, a driving unit 12, a display unit 13, a high-speed eyeball three-dimensional tracking unit 14 and a visual driving unit 15, which are described below.
The display control unit 10 is arranged to provide stereoscopic display data and related interaction information for a light field display. In one embodiment, the display control unit 10 is a computer with high stability and high computing power. For example, the display control unit 10 may be a server, and of course, in other embodiments, the display control unit 10 may be a conventional computer or a computer with an HDMI video interface.
The light field rendering unit 11 is connected to the display control unit 10, and is configured to perform light field image rendering according to the light field model and the human eye position information output by the video input interface in the display control unit.
The driving unit 12 is connected with the light field rendering unit 11, and adopts a modularized design, and each module is driven to 1920×1080 in size and is used for carrying out partition refreshing and control on the high-density liquid crystal display panel.
The resolution of the driving unit 12 is 1920×1080, and the driving unit is used for displaying, distributing and refreshing the light field digital signal signals output by the light field rendering unit. In a particular embodiment, the driver may support HDMI, DP, or Type-C inputs.
The display unit 13 is used for providing display information with a physical resolution of 16K x 8K for a presentation carrier of the light field display information; in one embodiment, the display unit 13 is a composite display carrier composed of a liquid crystal display panel with high resolution and an optical microstructure. For example, the display unit 13 may be a display device composed of a backlight structure, a liquid crystal display panel, and a grating, and of course, in other embodiments, the display unit 13 may be a non-liquid crystal display panel. In another embodiment, as shown in fig. 2, the backlight module 24 provides a light source for the lcd panel, the eye tracking module 26 can track the viewing position of the user in real time, the input end of the high-density lcd module 27 is connected to the display driving module 23, and the front panel of the high-density lcd module is attached to the aspheric microlens array module 28.
And the high-speed eyeball three-dimensional tracking unit 14 is connected with the display unit 13 and is used for positioning and tracking the human eye position information of the viewer in real time and outputting the human eye position information from a serial port according to a preset communication protocol. In one embodiment, the high-speed three-dimensional eye tracking unit 14 is a visual tracking device having three-dimensional position detection and tracking functions for the human eye.
The visual driving unit 15 is connected with the light field rendering unit 11 and the high-speed eyeball three-dimensional tracking unit 14, and is used for bridging the visual angle information of the viewer and the light field rendering model so as to output expected visual angle rendering parameters for the light field rendering unit according to the three-dimensional information of the human eye of the viewer.
Accordingly, the light field display image rendering method based on human eye tracking is characterized by comprising two stages of model training and model reasoning, as shown in fig. 4, and comprises the following steps:
s41, a target scene acquisition step: the target scene to be displayed may provide the original training data set for the light field image rendering unit, as shown in fig. 4. In a specific embodiment, the target scene may be a virtual three-dimensional scene in computer software or may be a real object directly acquired in a real environment;
s42, self-adaptive scene acquisition: and according to the target scene, performing scene sampling on the target scene by adopting a multi-scale self-adaptive method to generate a digital image signal to be output. In a specific embodiment, sampling with different scales is performed according to the complexity of the model so as to reduce information redundancy and image acquisition of key areas;
s43, estimating the pose of the camera: and estimating the pose of the photographed camera according to the sampled scene information. In a specific embodiment, estimating the pose of the camera by carrying out feature extraction and feature matching on the acquired image;
s44, training a nerve radiation field model: according to the position and direction of the visual angle, taking the color as a supervision item, and performing model training by adopting a fully-connected network;
s45, obtaining an implicit expression step: according to a neural radiation field algorithm, in a specific embodiment, node information and weight parameters of a model are obtained from the model which is trained by the fully-connected network;
s46, light field reproduction: and converting the human eye position signal in the neural radiation field rendering step into visual angle information by taking parameters in the model implicit expression as network weights, so as to complete model reasoning. In a specific embodiment, the data output according to the human eye position rendering step is rasterized and then converted into a display screen coordinate system, so as to achieve the purpose of light field display.
According to the light field display system and the image rendering method based on human eye tracking, due to the fact that the positions of two eyes of a viewer are tracked, corresponding image rendering viewpoint information is calculated, and therefore light field image rendering is conducted through a nerve radiation field reconstruction algorithm. The spatial bandwidth product of the light field stereoscopic display is greatly improved by combining with human eyes, and simultaneous watching of multiple persons is supported, so that the design requirement of the light field display on the high density of pixels of the liquid crystal panel is reduced, the display effect of the light field display is improved, a viewer can obtain better stereoscopic immersion feeling at different watching positions, the data volume and the computational complexity of a light field image rendering algorithm can be reduced, and the method is specifically analyzed.
As shown in fig. 1, in the technical solution for displaying a light field provided in the present application, a display control unit is used to control displaying contents to be displayed, where a light field model output by the display control unit may output a standard three-dimensional point cloud model (such as a ply, pcl, txt, etc.), and may also be an implicit expression of the three-dimensional model, including a weight file of the model and a connection relationship between all fully connected layers. The data such as the nonstandard three-dimensional model is encoded and then transmitted through the standard HDMI output interface, so that the method has good compatibility, can be used for compression transmission, encryption transmission and the like of the optical field data, has high timeliness and safety, and the output resolution of the optical field data is 1920 multiplied by 1080.
As shown in fig. 1, for a large light field model, the rendering unit 11 needs to perform data segmentation, so as to distribute the segmented data to the arrayed parallel acceleration unit for accelerating the rendering algorithm, and the segmented data blocks respectively complete the rendering of the light field image in the arrayed rendering unit. In fig. 1, the data segmentation module, the light field image rendering module and the video interface module output are collectively called a light field rendering unit, and the light field data rendered by the light field rendering unit needs to be output to the driving unit through the video output interface module.
As shown in fig. 1, the driving unit 12 further drives the high-density liquid crystal display unit 13 to display after receiving the light field data of the light field rendering unit 11. In order to increase the spatial bandwidth product of the light field display, on one hand, the information amount of the light field display needs to be increased, and on the other hand, a high-density liquid crystal panel is used as a physical support, but as the display information amount increases, huge computing power expenditure is undoubtedly caused for the light field image rendering unit 11, so that the display system cannot realize smooth light field image playing and interaction.
Inspired by a peripheral vision mechanism, the application adopts the eye tracking technology to realize acceleration of a rendering algorithm and improvement of a display bandwidth product, and as shown in fig. 1, the application determines the watching position of a viewer through the high-speed eyeball three-dimensional tracking unit 14, so that display information is accurately generated by a user, the regions which are not watched by the viewer are not rendered with light field images, the data volume of rendering the light field images is reduced, a large number of effective pixels are saved, and more liquid crystal display resources are contributed to the effective viewer regions. Therefore, the spatial bandwidth product of the light field display is increased by the high-speed three-dimensional eye tracking unit 14, and the spatial bandwidth product (Space Bandwidth Product, SBP) is generally calculated by sbp=horizontal resolution×vertical resolution×field frequency×1.344. In order to achieve good stereoscopic viewing effect, it is generally required that the spatial bandwidth product SBP of the light field display is not less than 10 8 The visual driving unit 15 is therefore of central value in establishing a physical logical link between the light field rendering unit and the high-speed eye three-dimensional tracking unit.
As shown in fig. 2, which is a system structure diagram of a light field display system based on eye tracking in the present application, the system includes a video interface module 21, a unit module 22, a driving module 23, a backlight module 24, a light field rendering module 25, an eye tracking module 26, a high-density liquid crystal display module 27, and an aspheric lens array module 28, it can be seen that in the present application, the eye tracking module cooperates with the light field image rendering module and the like.
As shown in fig. 3, the video interface unit is composed of a unit module 31, a PC interface module 32, an ESD protection module 33, a signal conversion module 34, a display panel 35, an audio decoding module 36, a video synchronization interface module 37, and a core processor module 38, and adopts a modularized design concept, so that the design difficulty of the arrayed display screen drive is reduced.
Referring to fig. 4, an algorithm of a light field display image rendering method based on human eye tracking includes a target scene acquisition step S41, an adaptive scene acquisition step S42, a camera pose estimation step S43, a neural radiation field model training step S44, an implicit expression acquisition step S45, and a light field reproduction step S46. The target scene can obtain multi-view information of the scene after multi-scale self-adaptive sampling, pose estimation is carried out according to the sampled scene information, network input of a nerve radiation field can be obtained according to pose parameters and multi-view images after pose estimation is completed, a NeRF function is a function which is used for representing a continuous scene as a 5D vector input, the function comprises a 3D coordinate position x= (x, y, z) and a direction (theta, phi) of a space point, and the color c= (r, g, b) of the 3D point related to the view angle and the volume density sigma of the position are output. The bulk density σ (x) is understood to mean that at the spatial x-position an infinitely small derivative of the particle (derivative of the opacity at the position) is independent of the direction of radiation. Camera imaging radiation r (t) =o+td; the camera position o and the imaging point d constitute a camera imaging ray, which accompanies a range t of near and far points, the color C (r) of which is expressed in an integrated manner as:
wherein t is f And t n Representing the far and near points of the ray, respectively, σ representing the bulk density of the current point, c representing the RGB value of the current point, T (T) representing the ray from T n Cumulative transparency to this segment of t:
wherein t is n Representing the near point of the ray, t represents any point on the ray,σ represents the bulk density of the current point.
In the continuous radiation field, rendering is performed for any view angle, and the ray passing through each pixel of the target virtual camera needs to be integrated with the color, so that the color of each pixel is obtained, and an imaging picture under the view angle is rendered. And obtaining a difference value between the color and the actual color through rendering of the L2 loss calculation body as a loss function to supervise the learning of the neural network.
And (3) training the neural network is completed, and the scene is converted into implicit expression. If new visual angle information is needed to be obtained, the visual angle information of the current viewing position can be rendered by only inputting the visual angle information of a viewer, and the visual angle image can be matched with a physical coordinate system of the light field display after rasterization, so that a three-dimensional display effect is achieved.
As shown in fig. 5, in the present application, a three-dimensional scene 51 is captured by a capture lens array 52 through a capture plane 53, and this process is a capture process of light field display, and because each lens of the capture lens array 52 can capture specific information of a corresponding scene from different angles, each capture lens can capture specific unit images at specific positions of the capture plane 53, and all the unit images form a sub-image array 54. The sub-image array 54 is loaded onto the display plane 55, and a reconstruction lens array 56 is arranged behind the display plane, the reconstruction lens array 56 corresponds to the sub-image array 54 precisely, and a reconstructed three-dimensional scene 57 can be obtained according to the principle of reversibility of the light path, which is a reproduction process of the light field display.
As shown in fig. 6, in order to illustrate a light field image rendering method based on eye tracking in the present application, the aforementioned display plane 61 can reconstruct the light field information 63 of the three-dimensional scene by loading the sub-image array and the reconstruction lens array 62 above the display plane, but the light field information of the whole scene is not required to be actually viewed, and only the image required for human eye viewing needs to be generated. The light that the observer can see at the current position is called effective light, and the light that the observer cannot see at the current position is called ineffective light 66, so the eyes 64 and 65 of the observer are tracked by the eye tracking module, the eye positions are sent to the light field rendering unit, only an effective sub-image array for the observer is needed to be rendered by the nerve radiation field, and the data volume and the computational complexity of the light field image rendering algorithm can be effectively reduced.
In summary, since the positions of the eyes of the viewer are tracked and the corresponding image rendering viewpoint information is calculated, the rendering of the light field image is performed through the neural radiation field reconstruction algorithm. The spatial bandwidth product of the light field stereoscopic display is greatly improved by combining with human eyes, and simultaneous watching of multiple persons is supported, so that the design requirement of the light field display on the high density of pixels of the liquid crystal panel is reduced, the display effect of the light field display is improved, a viewer can obtain better stereoscopic immersion feeling at different watching positions, and meanwhile, the data volume and the computational complexity of a light field image rendering algorithm can be reduced. The method and the device combine the eye tracking technology to reduce the data volume of data rendering, thereby being capable of reducing the requirement of light field image rendering on high performance of a processor, reducing the design difficulty of a light field display, enabling the real-time rendering of the light field image and the design of a high-density liquid crystal display panel not to be the bottleneck for improving the space bandwidth product of the light field display any more, and enabling the output signal bandwidth of the light field display to break through the space bandwidth product (10 8 ) Limiting, the spatial bandwidth product refers to the information flux of optical imaging of the light field display screen, the spatial bandwidth product is abbreviated as SPB, spb=horizontal resolution×vertical resolution×field frequency×1.344, and the spatial bandwidth product SPB of the light field display screen needs to be greater than 10 8 And a better viewing effect can be obtained at any viewing angle by adopting the neural radiation field rendering technology.
In addition, the visual fatigue phenomenon caused by crosstalk and the like of the traditional parallax type auto-stereoscopic display, such as convergence point and focusing not in a unified plane and the like, is relieved due to the increase of the number of rays and the improvement of the spatial bandwidth product in light field display, so that the stereoscopic immersion and stereoscopic viewing comfort of the light field display are improved.
The foregoing is a further detailed description of the present application in connection with the specific embodiments, and it is not intended that the practice of the present application be limited to such descriptions. It will be apparent to those skilled in the art from this disclosure that several simple deductions or substitutions can be made without departing from the inventive concepts of the present application.
Claims (10)
1. A light field display system based on eye tracking, comprising:
the display control unit is used for providing three-dimensional display data and man-machine interaction related interaction information for the light field display;
the light field rendering unit is connected with the display control unit and is used for performing light field image real-time rendering according to the light field model and human eye position information output by the video input interface in the display control unit;
the driving units are connected with the light field rendering units, the driving units are in a modularized design, and the display resolution of each driving module is 2K multiplied by 1K or 4K multiplied by 2K and is used for carrying out partition refreshing and synchronous control on the high-density liquid crystal display panel;
the display unit is used for providing a display carrier for the presentation of the light field information and constructing a high-density display physical model with the resolution of 16K multiplied by 8K;
the high-speed eyeball three-dimensional tracking unit is connected with the display unit and is used for positioning and tracking the human eye position information of a viewer in real time and outputting the human eye position information from the serial bus according to a preset communication protocol;
the visual driving unit is connected with the light field rendering unit and the high-speed eyeball three-dimensional tracking unit and is used for bridging the visual angle information of the viewer with the light field rendering model so as to output expected visual angle rendering parameters for the light field rendering unit according to the three-dimensional position information of the human eye of the viewer.
2. The light field display system based on eye tracking as claimed in claim 1, wherein the display control unit is a 2D interaction unit, a user can interact with the operating system by providing a 2D display screen on one hand, and can also set the operation mode of the light field display to be a 2D operation mode, and the 2D and 3D switching control can be switched by a mouse or a gesture.
3. The eye-tracking based light field display system of claim 1, wherein the light field rendering unit is a parallel rendering core with high stability and high real-time, and performs rendering using a neural radiation field deep learning algorithm.
4. The eye-tracking based light field display system of claim 1, wherein the driving unit is a Tcon driving module having independent driving and array synchronization functions, and the resolution of the driving board is 1920 x 1080 or 3840 x 2160.
5. The eye-tracking based light field display system of any one of claims 1-4, wherein the visual drive unit provides visual angle information for image rendering to the light field rendering unit, wherein the image rendering employs an inverse ray-integral rendering technique.
6. The light field image rendering method based on human eye tracking is characterized by comprising two stages of model training and model reasoning, and comprises the following steps:
a step of acquiring a target scene: the target scene to be displayed is used for providing an original training data set for the light field image rendering unit;
and (3) self-adaptive scene acquisition: sampling the target scene by adopting a multi-scale self-adaptive method according to the target scene, and generating a digital image signal to be output;
and a camera pose estimation step: estimating the pose of a shooting camera according to the sampled scene information;
training a nerve radiation field model: preprocessing the digital waveform signals output in the data acquisition step and outputting processing results;
obtaining an implicit expression step: according to a neural radiation field algorithm, training a model by adopting a fully connected network to obtain node information and weight parameters of the model;
a light field reproduction step: and converting the human eye position signal in the neural radiation field rendering step into visual angle information by taking parameters in the model implicit expression as network weights, so as to complete model reasoning. And rasterizing the data output according to the human eye position rendering step and converting the data into a display screen coordinate system so as to achieve the aim of light field display.
7. The method for rendering a light field image based on eye tracking as claimed in claim 6, wherein in said target scene obtaining step, the data may be a real scene to be displayed, or may be a virtual scene in a game or a three-dimensional model in an animation field.
8. The method of claim 6, wherein the data sampling step is performed by sampling the scene in real time by an aircraft, sampling data by a light field camera, or virtually sampling a three-dimensional model by a virtual camera.
9. The method of light field image rendering based on eye tracking as claimed in claim 6, wherein in the step of obtaining model parameters, a reverse rendering algorithm is adopted in the rendering process, so as to obtain a front-looking observation image.
10. A light field image rendering method based on human eye tracking as claimed in any one of claims 6 to 9, wherein the light field rendering algorithm is a real-time rendering algorithm.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311235605.4A CN117319631A (en) | 2023-09-22 | 2023-09-22 | Light field display system and image rendering method based on human eye tracking |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311235605.4A CN117319631A (en) | 2023-09-22 | 2023-09-22 | Light field display system and image rendering method based on human eye tracking |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117319631A true CN117319631A (en) | 2023-12-29 |
Family
ID=89261455
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311235605.4A Pending CN117319631A (en) | 2023-09-22 | 2023-09-22 | Light field display system and image rendering method based on human eye tracking |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117319631A (en) |
-
2023
- 2023-09-22 CN CN202311235605.4A patent/CN117319631A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10939085B2 (en) | Three dimensional glasses free light field display using eye location | |
| EP3110518B1 (en) | Image encoding and display | |
| EP3111640B1 (en) | Image encoding and display | |
| CN101909219B (en) | Stereoscopic display method, tracking type stereoscopic display | |
| US10554960B2 (en) | Unassisted stereoscopic display device using directional backlight structure | |
| CN103096109B (en) | Multiple view automatic stereoscopic displayer and display method | |
| JP2008522270A (en) | System and method for composite view display of single 3D rendering | |
| CN112866676B (en) | Naked eye three-dimensional display algorithm based on single-pixel multi-view reconstruction | |
| CN114339191B (en) | Naked eye three-dimensional display method based on multi-viewpoint reconstruction | |
| CN117319631A (en) | Light field display system and image rendering method based on human eye tracking | |
| Surman et al. | Towards the reality of 3D imaging and display | |
| CN203377985U (en) | Multidimensional led display screen | |
| Wetzstein | Computational Near-Eye Displays: Engineering the Interface to the Digital World | |
| Kejian et al. | The development of stereoscopic display technology | |
| Lu | Research on optical display technology of virtual reality technology based on optical image | |
| Yan et al. | Research summary on light field display technology based on projection | |
| Wetzstein et al. | Factored displays: improving resolution, dynamic range, color reproduction, and light field characteristics with advanced signal processing | |
| US20230239456A1 (en) | Display system with machine learning (ml) based stereoscopic view synthesis over a wide field of view | |
| Zhang et al. | 3D communication system integrating 3D reconstruction and rendering display | |
| CN117176936A (en) | Freely-expandable stereoscopic digital sand table system and light field rendering method | |
| US20220232201A1 (en) | Image generation system and method | |
| Megyesi et al. | 3D video visualization on the holovizio system | |
| Wang et al. | An Introduction to Immersive Display Technologies | |
| Surman et al. | Super multiview 3D display systems | |
| EP3598271A1 (en) | Method and device for disconnecting user's attention |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |