CN116597288A - Gaze point rendering method, gaze point rendering system, computer and readable storage medium - Google Patents
Gaze point rendering method, gaze point rendering system, computer and readable storage medium Download PDFInfo
- Publication number
- CN116597288A CN116597288A CN202310879693.5A CN202310879693A CN116597288A CN 116597288 A CN116597288 A CN 116597288A CN 202310879693 A CN202310879693 A CN 202310879693A CN 116597288 A CN116597288 A CN 116597288A
- Authority
- CN
- China
- Prior art keywords
- gaze point
- rendering
- data
- gaze
- training
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000009877 rendering Methods 0.000 title claims abstract description 129
- 238000000034 method Methods 0.000 title claims abstract description 53
- 238000012549 training Methods 0.000 claims abstract description 73
- 238000013528 artificial neural network Methods 0.000 claims abstract description 46
- 210000005252 bulbus oculi Anatomy 0.000 claims abstract description 19
- 210000001508 eye Anatomy 0.000 claims abstract description 19
- 238000007499 fusion processing Methods 0.000 claims abstract description 7
- 238000012545 processing Methods 0.000 claims description 64
- 238000012360 testing method Methods 0.000 claims description 48
- 238000005457 optimization Methods 0.000 claims description 33
- 238000012795 verification Methods 0.000 claims description 18
- 238000004590 computer program Methods 0.000 claims description 9
- 238000013527 convolutional neural network Methods 0.000 claims description 8
- 238000010586 diagram Methods 0.000 claims description 7
- 230000004424 eye movement Effects 0.000 claims description 6
- 238000010200 validation analysis Methods 0.000 claims description 2
- 230000000694 effects Effects 0.000 abstract description 11
- 238000005516 engineering process Methods 0.000 description 13
- 230000008569 process Effects 0.000 description 9
- 230000008859 change Effects 0.000 description 4
- 238000003491 array Methods 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 239000000463 material Substances 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
- 238000004088 simulation Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
- G06F3/013—Eye tracking input arrangements
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/0464—Convolutional networks [CNN, ConvNet]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/80—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
- G06V10/806—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level of extracted features
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Evolutionary Computation (AREA)
- Health & Medical Sciences (AREA)
- Artificial Intelligence (AREA)
- Computing Systems (AREA)
- General Health & Medical Sciences (AREA)
- General Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Databases & Information Systems (AREA)
- Multimedia (AREA)
- Life Sciences & Earth Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Molecular Biology (AREA)
- Medical Informatics (AREA)
- Mathematical Physics (AREA)
- Human Computer Interaction (AREA)
- Image Analysis (AREA)
Abstract
The application provides a gaze point rendering method, a gaze point rendering system, a computer and a readable storage medium, wherein the method comprises the following steps: calibrating an eye tracking device by a user, and collecting gaze point data generated by eyeballs of the user when observing a scene image in real time by the calibrated eye tracking device; correlating the gaze point data with the scene image to generate a corresponding target data set, and training a corresponding gaze point prediction model according to the target data set and a preset neural network; and outputting a gaze point distribution map corresponding to the gaze point data through the gaze point prediction model, and carrying out fusion processing on the gaze point distribution map and the scene image so as to complete image rendering of each gaze point. The method and the device can greatly improve the rendering fidelity of each fixation point, simultaneously improve the rendering effect of the scene image and correspondingly improve the use experience of the user.
Description
Technical Field
The present application relates to the field of virtual reality technologies, and in particular, to a gaze point rendering method, a gaze point rendering system, a gaze point rendering computer, and a gaze point rendering program.
Background
With the progress of technology and the rapid development of productivity, the virtual reality technology has also been rapidly developed, and the basic implementation manner of the existing virtual reality technology is mainly based on computer technology, and utilizes and integrates the latest development achievements of various high technologies such as three-dimensional graphics technology, multimedia technology, simulation technology, display technology, servo technology and the like, so that people can generate an immersive feeling.
The gaze point rendering is an important technology in virtual reality, and is used for simulating visual effects generated when eyes gaze a scene in real time, and correspondingly improving the effect of picture rendering.
Most of the prior art finishes the rendering effect of the fixation point by uniformly sampling all pixels in the scene, however, the rendering mode cannot accurately simulate the fixation point change of human eyes in the process of observing the scene, so that the rendering fidelity is lower, and the use experience of a user is correspondingly reduced.
Disclosure of Invention
Based on this, the present application aims to provide a gaze point rendering method, a system, a computer and a readable storage medium, so as to solve the problem that the rendering fidelity is lower because the gaze point change of human eyes in the process of observing a picture cannot be accurately simulated in the rendering mode in the prior art.
An embodiment of the present application provides a gaze point rendering method, where the method includes:
calibrating an eye tracking device by a user, and collecting gaze point data generated by eyeballs of the user when observing a scene image in real time by the calibrated eye tracking device;
correlating the gaze point data with the scene image to generate a corresponding target data set, and training a corresponding gaze point prediction model according to the target data set and a preset neural network;
and outputting a gaze point distribution map corresponding to the gaze point data through the gaze point prediction model, and carrying out fusion processing on the gaze point distribution map and the scene image so as to complete image rendering of each gaze point.
The beneficial effects of the application are as follows: the method comprises the steps of collecting the gaze point data generated by eyeballs of a user when the scene image is observed in real time, correspondingly associating the gaze point data with the current scene image to generate a corresponding target data set, training a corresponding gaze point prediction model through the target data set based on the gaze point data set, and finally predicting the positions of all gaze points in the scene image through the gaze point prediction model, so that the rendering of images around all gaze points can be finally completed, the positions of the gaze points generated by the eyes of the user in the process of observing the picture can be accurately found, the rendering of all gaze points can be further carried out, the rendering fidelity of all gaze points can be greatly improved, the rendering effect of the scene image is improved, and the use experience of the user is correspondingly improved.
Preferably, the step of associating the gaze point data with the scene image comprises:
when the fixation point data and the scene images are acquired, the fixation point coordinates marked in each scene image are acquired in real time based on the fixation point data, and a plurality of fixation points generated in real time by eyeballs of the user in the scene images are found out according to the fixation point coordinates;
acquiring first time stamps corresponding to each scene image respectively, and acquiring second time stamps corresponding to each gaze point respectively;
comparing the first timestamp with the second timestamp one by one, and associating the same gaze point with scene images, each of which contains one or more of the gaze points.
Preferably, the training the corresponding gaze point prediction model according to the target data set and the preset neural network includes:
performing interpolation processing on the target data set to generate corresponding continuous and consistent time series data, and performing image enhancement processing on the time series data;
and dividing a corresponding training set, a verification set and a test set according to the time sequence data after the image enhancement processing, and training an R-CNN neural network through the training set, the verification set and the test set to generate the gaze point prediction model.
Preferably, the step of training the R-CNN neural network by the training set, the verification set and the test set includes:
inputting the training set into a data processing layer of the R-CNN neural network to conduct predictive training on the data processing layer and generate a first predictive result;
inputting the verification set into the data processing layer to output a corresponding second prediction result, and simultaneously inputting the first prediction result and the second prediction result into a data optimization layer of the R-CNN neural network to enable the data optimization layer to output a corresponding optimization scheme;
inputting the optimization scheme into the data processing layer to perform optimization processing on initial parameters in the data processing layer, and inputting the test set into the optimized data processing layer to output a corresponding test result;
judging whether the training of the R-CNN neural network is completed or not according to the test result.
Preferably, the step of determining whether training of the R-CNN neural network is completed according to the test result includes:
calculating the prediction accuracy corresponding to the test result, and judging whether the prediction accuracy is higher than a preset threshold in real time;
and if the prediction accuracy rate is judged to be higher than the preset threshold value in real time, judging that the test result is qualified, and finishing training of the R-CNN neural network.
Preferably, the step of fusing the gaze point distribution map with the scene image to complete image rendering of each gaze point includes:
when the gaze point distribution map is acquired, finding out target positions corresponding to the gaze points in the scene image according to the gaze point distribution map, and detecting all pixel points contained in the scene image;
and adding corresponding rendering weights to pixel points around each gaze point through a gaze weight algorithm based on a preset rule, and completing image rendering of each gaze point based on the rendering weights.
Preferably, the preset rule is:
the size of the rendering weight is inversely proportional to the size of the distance between the pixel point and the nearest gaze point.
A second aspect of an embodiment of the present application proposes a gaze point rendering system, wherein the system includes:
the acquisition module is used for calibrating the eye movement tracking equipment through a user and acquiring fixation point data generated when the eyeball of the user observes a scene image in real time through the calibrated eye movement tracking equipment;
the training module is used for associating the gaze point data with the scene image to generate a corresponding target data set, and training a corresponding gaze point prediction model according to the target data set and a preset neural network;
and the rendering module is used for outputting a gaze point distribution diagram corresponding to the gaze point data through the gaze point prediction model, and carrying out fusion processing on the gaze point distribution diagram and the scene image so as to complete image rendering of each gaze point.
In the gaze point rendering system, the training module is specifically configured to:
when the fixation point data and the scene images are acquired, the fixation point coordinates marked in each scene image are acquired in real time based on the fixation point data, and a plurality of fixation points generated in real time by eyeballs of the user in the scene images are found out according to the fixation point coordinates;
acquiring first time stamps corresponding to each scene image respectively, and acquiring second time stamps corresponding to each gaze point respectively;
comparing the first timestamp with the second timestamp one by one, and associating the same gaze point with scene images, each of which contains one or more of the gaze points.
In the gaze point rendering system, the training module is further specifically configured to:
performing interpolation processing on the target data set to generate corresponding continuous and consistent time series data, and performing image enhancement processing on the time series data;
and dividing a corresponding training set, a verification set and a test set according to the time sequence data after the image enhancement processing, and training an R-CNN neural network through the training set, the verification set and the test set to generate the gaze point prediction model.
In the gaze point rendering system, the training module is further specifically configured to:
inputting the training set into a data processing layer of the R-CNN neural network to conduct predictive training on the data processing layer and generate a first predictive result;
inputting the verification set into the data processing layer to output a corresponding second prediction result, and simultaneously inputting the first prediction result and the second prediction result into a data optimization layer of the R-CNN neural network to enable the data optimization layer to output a corresponding optimization scheme;
inputting the optimization scheme into the data processing layer to perform optimization processing on initial parameters in the data processing layer, and inputting the test set into the optimized data processing layer to output a corresponding test result;
judging whether the training of the R-CNN neural network is completed or not according to the test result.
In the gaze point rendering system, the training module is further specifically configured to:
calculating the prediction accuracy corresponding to the test result, and judging whether the prediction accuracy is higher than a preset threshold in real time;
and if the prediction accuracy rate is judged to be higher than the preset threshold value in real time, judging that the test result is qualified, and finishing training of the R-CNN neural network.
In the gaze point rendering system, the rendering module is specifically configured to:
when the gaze point distribution map is acquired, finding out target positions corresponding to the gaze points in the scene image according to the gaze point distribution map, and detecting all pixel points contained in the scene image;
and adding corresponding rendering weights to pixel points around each gaze point through a gaze weight algorithm based on a preset rule, and completing image rendering of each gaze point based on the rendering weights.
In the gaze point rendering system, the preset rule is as follows:
the size of the rendering weight is inversely proportional to the size of the distance between the pixel point and the nearest gaze point.
A third aspect of an embodiment of the present application proposes a computer comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the gaze point rendering method as described above when executing the computer program.
A fourth aspect of the embodiments of the present application proposes a readable storage medium having stored thereon a computer program, wherein the program, when executed by a processor, implements a gaze point rendering method as described above.
Additional aspects and advantages of the application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the application.
Drawings
Fig. 1 is a flowchart of a gaze point rendering method according to a first embodiment of the present application;
fig. 2 is a block diagram of a gaze point rendering system according to a third embodiment of the present application.
The application will be further described in the following detailed description in conjunction with the above-described figures.
Detailed Description
In order that the application may be readily understood, a more complete description of the application will be rendered by reference to the appended drawings. Several embodiments of the application are presented in the figures. This application may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete.
It will be understood that when an element is referred to as being "mounted" on another element, it can be directly on the other element or intervening elements may also be present. When an element is referred to as being "connected" to another element, it can be directly connected to the other element or intervening elements may also be present. The terms "vertical," "horizontal," "left," "right," and the like are used herein for illustrative purposes only.
Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein in the description of the application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The term "and/or" as used herein includes any and all combinations of one or more of the associated listed items.
Referring to fig. 1, a gaze point rendering method according to a first embodiment of the present application is shown, where the gaze point rendering method according to the present embodiment can greatly improve the rendering fidelity of each gaze point, and simultaneously improve the rendering effect of a scene image, and correspondingly improve the use experience of a user.
Specifically, the gaze point rendering method provided by the embodiment specifically includes the following steps:
step S10, calibrating an eye tracking device by a user, and acquiring gaze point data generated by eyeballs of the user when observing a scene image in real time by the calibrated eye tracking device;
specifically, in this embodiment, it should be first described that the gaze point rendering method provided in this embodiment is specifically applied to the field of virtual reality technology, and is used to track each gaze point generated when an eyeball of a user observes a scene image in real time, and perform image rendering based on each gaze point, so as to correspondingly improve a display effect of the scene image, and correspondingly improve a use experience of the user.
It should be noted that, the scene image provided in this embodiment refers to an image or video that is watched by the user, so when the scene image is a picture, the current scene image is static, and correspondingly, when the scene image is a video or an application program, the current scene image is dynamic.
Based on this, in this step, it should be noted that, in this step, the gaze point rendering method is first started by the existing eye tracking device, and specifically, the eye tracking device can detect the dynamic change process of the eyeball of the user in real time, that is, can track the eyeball of the user in real time. Furthermore, before the formal rendering starts, the current eye tracking device needs to be calibrated first, specifically, the current eye tracking device needs to be calibrated by the current user, that is, when each new user uses the current eye tracking device, the corresponding calibration needs to be performed. Furthermore, the eye tracking device after calibration can acquire the gaze point data generated by the eyeballs of the user when observing the scene image in real time.
Step S20, associating the gaze point data with the scene image to generate a corresponding target data set, and training a corresponding gaze point prediction model according to the target data set and a preset neural network;
further, in this step, it should be noted that, because the eyeballs of the user dynamically change when observing the scene image, a plurality of gaze points are generated in the process of observation, that is, the gaze point data is generated, and based on this, the current gaze point data and the scene image that is currently played in real time need to be associated together in real time in this step, so as to correspondingly generate a target data set that can facilitate subsequent processing.
On the basis, a required gaze point prediction model can be trained further according to the current target data set and a preset neural network, and particularly, the gaze point prediction model can predict the distribution position of each gaze point in the current scene image so as to facilitate subsequent processing.
And step S30, outputting a gaze point distribution diagram corresponding to the gaze point data through the gaze point prediction model, and carrying out fusion processing on the gaze point distribution diagram and the scene image so as to complete image rendering of each gaze point.
Further, in this step, after the gaze point prediction model is constructed in real time, the present step further outputs a gaze point distribution map corresponding to the current gaze point data through the current gaze point prediction model, and specifically, the distribution position of each gaze point can be obtained through the distribution map.
Based on the method, the current gaze point distribution map and the current scene image are fused, so that each gaze point can be correspondingly displayed in the current scene image, and on the basis, corresponding image rendering is carried out in the current scene image by taking each gaze point as the center, so that the area of image rendering can be correspondingly reduced, the image rendering efficiency is correspondingly improved, and meanwhile, the use experience of a user is improved.
When the method is used, the gaze point data generated by eyeballs of a user when the scene image is observed are collected in real time, the gaze point data are correspondingly associated with the current scene image to generate a corresponding target data set, based on the gaze point data, a corresponding gaze point prediction model is trained through the target data set, and finally, the positions of all gaze points in the scene image are predicted through the gaze point prediction model, so that the rendering of images around all gaze points can be finally completed, the positions of gaze points generated by eyes of the user in the process of observing the picture can be accurately found, and the gaze points can be further rendered, so that the rendering fidelity of all gaze points can be greatly improved, the rendering effect of the scene image is improved, and the use experience of the user is correspondingly improved.
It should be noted that the foregoing implementation procedure is only for illustrating the feasibility of the present application, but this does not represent that the gaze point rendering method of the present application is only one implementation procedure, and may be incorporated into the feasible implementation of the gaze point rendering method of the present application, as long as it can be implemented.
In summary, the gaze point rendering method provided by the embodiment of the application can greatly improve the rendering fidelity of each gaze point, and simultaneously improve the rendering effect of the scene image, and correspondingly improve the use experience of the user.
The second embodiment of the present application also provides a gaze point rendering method, which is different from the gaze point rendering method provided in the first embodiment in that:
specifically, in this embodiment, it should be noted that the step of associating the gaze point data with the scene image includes:
when the fixation point data and the scene images are acquired, the fixation point coordinates marked in each scene image are acquired in real time based on the fixation point data, and a plurality of fixation points generated in real time by eyeballs of the user in the scene images are found out according to the fixation point coordinates;
acquiring first time stamps corresponding to each scene image respectively, and acquiring second time stamps corresponding to each gaze point respectively;
comparing the first timestamp with the second timestamp one by one, and associating the same gaze point with scene images, each of which contains one or more of the gaze points.
Specifically, in this embodiment, it should be noted that, the gaze point data provided in this embodiment includes a plurality of gaze points acquired in real time, where it should be noted that, in this embodiment, when each gaze point is acquired, a corresponding timestamp is added to each gaze point, where the timestamp corresponds to the acquisition time. Similarly, when each scene image is played, the embodiment also adds a corresponding timestamp to each scene image, and specifically, the timestamp corresponds to the playing time of each scene image. Based on this, after a plurality of gaze points are acquired through the gaze point data, a first timestamp corresponding to the current scene image and a second timestamp corresponding to the current gaze point are acquired simultaneously.
Based on the above, each first timestamp and each second timestamp are compared one by one, and in the process, if the same point of regard and scene images are detected, the same point of regard and scene images are associated together in real time. In addition, it should be noted that one or more gaze points may be included in each scene image, requiring a differential association.
Specifically, in this embodiment, it should be further noted that the step of training the corresponding gaze point prediction model according to the target data set and the preset neural network includes:
performing interpolation processing on the target data set to generate corresponding continuous and consistent time series data, and performing image enhancement processing on the time series data;
and dividing a corresponding training set, a verification set and a test set according to the time sequence data after the image enhancement processing, and training an R-CNN neural network through the training set, the verification set and the test set to generate the gaze point prediction model.
In particular, in this embodiment, it should be further noted that, in order to effectively train a desired gaze point prediction model, this embodiment needs to perform secondary processing on the target data set, and in particular, because in the process of collecting gaze point data, there may occur a situation that data is lost or recording intervals are inconsistent, so that the obtained gaze point data is incomplete, and therefore, in this embodiment, it is also necessary to perform interpolation processing on the current target data set first, so as to generate corresponding continuous time series data with consistent data formats, and similarly, the time series data includes the gaze point data and the scene image. Based on this, the present embodiment further performs image enhancement processing such as brightness adjustment, contrast adjustment, image inversion, rotation, and clipping on the current time-series data, so as to correspondingly improve the robustness of the constructed gaze point prediction model.
After the image enhancement processing of the time-series data is completed, a required training set, a validation set and a test set can be partitioned based on the current time-series data at this time, so as to further perform corresponding training on the R-CNN (Region-Convolutional Neural Networks, deep learning target detection algorithm) provided in the present embodiment, so as to generate the gaze point prediction model.
In addition, in this embodiment, it should be noted that the step of training the R-CNN neural network through the training set, the verification set and the test set includes:
inputting the training set into a data processing layer of the R-CNN neural network to conduct predictive training on the data processing layer and generate a first predictive result;
inputting the verification set into the data processing layer to output a corresponding second prediction result, and simultaneously inputting the first prediction result and the second prediction result into a data optimization layer of the R-CNN neural network to enable the data optimization layer to output a corresponding optimization scheme;
inputting the optimization scheme into the data processing layer to perform optimization processing on initial parameters in the data processing layer, and inputting the test set into the optimized data processing layer to output a corresponding test result;
judging whether the training of the R-CNN neural network is completed or not according to the test result.
In addition, in this embodiment, it should be noted that, in particular, the R-CNN neural network provided in this embodiment includes a data input layer, a data processing layer, a data optimization layer, and a data output layer, and in particular, this embodiment inputs the training set described above to the data processing layer through the data input layer, where an initial parameter is set in the data processing layer that just starts training, and based on this, the current initial parameter is trained through the training set described above, so that the data output layer outputs a corresponding first prediction result. Further, the verification set is input into the current data processing layer through the data input layer, so that a second prediction result can be correspondingly output, and on the basis, the current first prediction result and the second prediction result are simultaneously input into the data optimization layer, so that the current data optimization layer can output a corresponding optimization scheme according to a deviation value between the current first prediction result and the second prediction result. The optimization scheme is used for optimizing the initial parameters.
Further, the current optimization scheme is input into the data processing layer, so that the optimization processing can be performed on the initial parameters in the current data processing layer to complete the optimization of the initial parameters. Based on the above, the test set is input into the optimized data processing layer, so that a corresponding test result can be output, and whether the training of the current R-CNN neural network is completed can be judged in real time according to the test result.
In addition, in this embodiment, it should be further noted that the step of determining whether to complete training of the R-CNN neural network according to the test result includes:
calculating the prediction accuracy corresponding to the test result, and judging whether the prediction accuracy is higher than a preset threshold in real time;
and if the prediction accuracy rate is judged to be higher than the preset threshold value in real time, judging that the test result is qualified, and finishing training of the R-CNN neural network.
In addition, in this embodiment, it should also be noted that, immediately after the required prediction accuracy is obtained through the above steps, the present embodiment compares the current prediction accuracy with the preset threshold, and determines in real time whether the current prediction accuracy is higher than the current preset threshold, and preferably, the prediction accuracy provided in this embodiment is set to 95%.
Further, if the current prediction accuracy is judged to be higher than the current preset threshold in real time, the test result is judged to be qualified to finish the training of the R-CNN neural network, and correspondingly, if the current prediction accuracy is judged to be lower than the current preset threshold in real time, the test result is judged to be unqualified, and the test process is required to be repeated until the output prediction accuracy is higher than the current preset threshold to finish the training of the R-CNN neural network.
In this embodiment, it should be noted that, the step of fusing the gaze point distribution map and the scene image to complete image rendering of each gaze point includes:
when the gaze point distribution map is acquired, finding out target positions corresponding to the gaze points in the scene image according to the gaze point distribution map, and detecting all pixel points contained in the scene image;
and adding corresponding rendering weights to pixel points around each gaze point through a gaze weight algorithm based on a preset rule, and completing image rendering of each gaze point based on the rendering weights.
In this embodiment, it should be noted that, after the required gaze point distribution map is obtained through the above steps, the target positions corresponding to the current gaze points in the corresponding scene images can be quickly found according to the current gaze point distribution map, and at the same time, all the pixel points included in the current scene images are synchronously detected.
Furthermore, according to preset rendering rules, corresponding rendering weights are added to pixel points around each current gaze point through a preset gaze weight algorithm, so that image rendering of each gaze point can be completed only based on the rendering weights of the pixel points.
In this embodiment, it should be noted that, the preset rule is:
the size of the rendering weight is inversely proportional to the size of the distance between the pixel point and the nearest gaze point.
In this embodiment, it should be noted that, in the preset rule provided in this embodiment, the magnitude of the rendering weight is inversely proportional to the magnitude of the distance between each pixel point and the gaze point closest thereto, that is, the greater the distance between the pixel point and the gaze point closest thereto, the smaller the rendering weight of the current pixel point, and conversely, the smaller the distance between the pixel point and the gaze point closest thereto, the greater the rendering weight of the current pixel point, so that the rendering resources can be effectively saved, and the rendering efficiency is correspondingly improved.
It should be noted that, for the sake of brevity, the method according to the second embodiment of the present application, which implements the same principle and some of the technical effects as the first embodiment, is not mentioned here, and reference is made to the corresponding content provided by the first embodiment.
In summary, the gaze point rendering method provided by the embodiment of the application can greatly improve the rendering fidelity of each gaze point, and simultaneously improve the rendering effect of the scene image, and correspondingly improve the use experience of the user.
Referring to fig. 2, a gaze point rendering system according to a third embodiment of the present application is shown, wherein the system includes:
the acquisition module 12 is used for calibrating the eye movement tracking device by a user and acquiring fixation point data generated when the eyeball of the user observes a scene image in real time by the calibrated eye movement tracking device;
the training module 22 is configured to correlate the gaze point data with the scene image to generate a corresponding target data set, and train a corresponding gaze point prediction model according to the target data set and a preset neural network;
and the rendering module 32 is configured to output a gaze point distribution map corresponding to the gaze point data through the gaze point prediction model, and perform fusion processing on the gaze point distribution map and the scene image, so as to complete image rendering of each gaze point.
The training module 22 in the gaze point rendering system is specifically configured to:
when the fixation point data and the scene images are acquired, the fixation point coordinates marked in each scene image are acquired in real time based on the fixation point data, and a plurality of fixation points generated in real time by eyeballs of the user in the scene images are found out according to the fixation point coordinates;
acquiring first time stamps corresponding to each scene image respectively, and acquiring second time stamps corresponding to each gaze point respectively;
comparing the first timestamp with the second timestamp one by one, and associating the same gaze point with scene images, each of which contains one or more of the gaze points.
The training module 22 in the gaze point rendering system is specifically further configured to:
performing interpolation processing on the target data set to generate corresponding continuous and consistent time series data, and performing image enhancement processing on the time series data;
and dividing a corresponding training set, a verification set and a test set according to the time sequence data after the image enhancement processing, and training an R-CNN neural network through the training set, the verification set and the test set to generate the gaze point prediction model.
The training module 22 in the gaze point rendering system is specifically further configured to:
inputting the training set into a data processing layer of the R-CNN neural network to conduct predictive training on the data processing layer and generate a first predictive result;
inputting the verification set into the data processing layer to output a corresponding second prediction result, and simultaneously inputting the first prediction result and the second prediction result into a data optimization layer of the R-CNN neural network to enable the data optimization layer to output a corresponding optimization scheme;
inputting the optimization scheme into the data processing layer to perform optimization processing on initial parameters in the data processing layer, and inputting the test set into the optimized data processing layer to output a corresponding test result;
judging whether the training of the R-CNN neural network is completed or not according to the test result.
The training module 22 in the gaze point rendering system is specifically further configured to:
calculating the prediction accuracy corresponding to the test result, and judging whether the prediction accuracy is higher than a preset threshold in real time;
and if the prediction accuracy rate is judged to be higher than the preset threshold value in real time, judging that the test result is qualified, and finishing training of the R-CNN neural network.
The rendering module 32 in the gaze point rendering system is specifically configured to:
when the gaze point distribution map is acquired, finding out target positions corresponding to the gaze points in the scene image according to the gaze point distribution map, and detecting all pixel points contained in the scene image;
and adding corresponding rendering weights to pixel points around each gaze point through a gaze weight algorithm based on a preset rule, and completing image rendering of each gaze point based on the rendering weights.
The preset rule in the gaze point rendering system is as follows:
the size of the rendering weight is inversely proportional to the size of the distance between the pixel point and the nearest gaze point.
A fourth embodiment of the present application provides a computer, including a memory, a processor, and a computer program stored on the memory and executable on the processor, where the processor implements the gaze point rendering method provided in the above embodiments when executing the computer program.
A fifth embodiment of the present application provides a readable storage medium having stored thereon a computer program, wherein the program, when executed by a processor, implements the gaze point rendering method as provided by the above embodiments.
In summary, the gaze point rendering method, the gaze point rendering system, the gaze point rendering computer and the gaze point rendering medium provided by the embodiments of the present application can greatly improve the rendering fidelity of each gaze point, and simultaneously improve the rendering effect of the scene image, and correspondingly improve the use experience of the user.
The above-described respective modules may be functional modules or program modules, and may be implemented by software or hardware. For modules implemented in hardware, the various modules described above may be located in the same processor; or the above modules may be located in different processors in any combination.
Logic and/or steps represented in the flowcharts or otherwise described herein, e.g., a ordered listing of executable instructions for implementing logical functions, can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor-containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. For the purposes of this description, a "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device.
More specific examples (a non-exhaustive list) of the computer-readable medium would include the following: an electrical connection (electronic device) having one or more wires, a portable computer diskette (magnetic device), a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). In addition, the computer readable medium may even be paper or other suitable medium on which the program is printed, as the program may be electronically captured, via, for instance, optical scanning of the paper or other medium, then compiled, interpreted or otherwise processed in a suitable manner, if necessary, and then stored in a computer memory.
It is to be understood that portions of the present application may be implemented in hardware, software, firmware, or a combination thereof. In the above-described embodiments, the various steps or methods may be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, may be implemented using any one or combination of the following techniques, as is well known in the art: discrete logic circuits having logic gates for implementing logic functions on data signals, application specific integrated circuits having suitable combinational logic gates, programmable Gate Arrays (PGAs), field Programmable Gate Arrays (FPGAs), and the like.
In the description of the present specification, a description referring to terms "one embodiment," "some embodiments," "examples," "specific examples," or "some examples," etc., means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the present application. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiments or examples. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.
The foregoing examples illustrate only a few embodiments of the application and are described in detail herein without thereby limiting the scope of the application. It should be noted that it will be apparent to those skilled in the art that several variations and modifications can be made without departing from the spirit of the application, which are all within the scope of the application. Accordingly, the scope of protection of the present application is to be determined by the appended claims.
Claims (10)
1. A gaze point rendering method, the method comprising:
calibrating an eye tracking device by a user, and collecting gaze point data generated by eyeballs of the user when observing a scene image in real time by the calibrated eye tracking device;
correlating the gaze point data with the scene image to generate a corresponding target data set, and training a corresponding gaze point prediction model according to the target data set and a preset neural network;
and outputting a gaze point distribution map corresponding to the gaze point data through the gaze point prediction model, and carrying out fusion processing on the gaze point distribution map and the scene image so as to complete image rendering of each gaze point.
2. A gaze point rendering method according to claim 1, characterized by: the step of associating the gaze point data with the scene image comprises:
when the fixation point data and the scene images are acquired, the fixation point coordinates marked in each scene image are acquired in real time based on the fixation point data, and a plurality of fixation points generated in real time by eyeballs of the user in the scene images are found out according to the fixation point coordinates;
acquiring first time stamps corresponding to each scene image respectively, and acquiring second time stamps corresponding to each gaze point respectively;
comparing the first timestamp with the second timestamp one by one, and associating the same gaze point with scene images, each of which contains one or more of the gaze points.
3. A gaze point rendering method according to claim 1, characterized by: the step of training a corresponding gaze point prediction model according to the target data set and a preset neural network comprises the following steps:
performing interpolation processing on the target data set to generate corresponding continuous and consistent time series data, and performing image enhancement processing on the time series data;
and dividing a corresponding training set, a verification set and a test set according to the time sequence data after the image enhancement processing, and training an R-CNN neural network through the training set, the verification set and the test set to generate the gaze point prediction model.
4. A gaze point rendering method according to claim 3, characterized in that: the step of training the R-CNN neural network by the training set, the validation set, and the test set includes:
inputting the training set into a data processing layer of the R-CNN neural network to conduct predictive training on the data processing layer and generate a first predictive result;
inputting the verification set into the data processing layer to output a corresponding second prediction result, and simultaneously inputting the first prediction result and the second prediction result into a data optimization layer of the R-CNN neural network to enable the data optimization layer to output a corresponding optimization scheme;
inputting the optimization scheme into the data processing layer to perform optimization processing on initial parameters in the data processing layer, and inputting the test set into the optimized data processing layer to output a corresponding test result;
judging whether the training of the R-CNN neural network is completed or not according to the test result.
5. The gaze point rendering method of claim 4, wherein: the step of judging whether the training of the R-CNN neural network is completed according to the test result comprises the following steps:
calculating the prediction accuracy corresponding to the test result, and judging whether the prediction accuracy is higher than a preset threshold in real time;
and if the prediction accuracy rate is judged to be higher than the preset threshold value in real time, judging that the test result is qualified, and finishing training of the R-CNN neural network.
6. A gaze point rendering method according to claim 1, characterized by: the step of fusing the gaze point distribution map and the scene image to complete image rendering of each gaze point comprises the following steps:
when the gaze point distribution map is acquired, finding out target positions corresponding to the gaze points in the scene image according to the gaze point distribution map, and detecting all pixel points contained in the scene image;
and adding corresponding rendering weights to pixel points around each gaze point through a gaze weight algorithm based on a preset rule, and completing image rendering of each gaze point based on the rendering weights.
7. The gaze point rendering method of claim 6, wherein: the preset rule is as follows:
the size of the rendering weight is inversely proportional to the size of the distance between the pixel point and the nearest gaze point.
8. A gaze point rendering system, the system comprising:
the acquisition module is used for calibrating the eye movement tracking equipment through a user and acquiring fixation point data generated when the eyeball of the user observes a scene image in real time through the calibrated eye movement tracking equipment;
the training module is used for associating the gaze point data with the scene image to generate a corresponding target data set, and training a corresponding gaze point prediction model according to the target data set and a preset neural network;
and the rendering module is used for outputting a gaze point distribution diagram corresponding to the gaze point data through the gaze point prediction model, and carrying out fusion processing on the gaze point distribution diagram and the scene image so as to complete image rendering of each gaze point.
9. A computer comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the gaze point rendering method of any of claims 1 to 7 when executing the computer program.
10. A readable storage medium having stored thereon a computer program, which when executed by a processor implements a gaze point rendering method as claimed in any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310879693.5A CN116597288B (en) | 2023-07-18 | 2023-07-18 | Gaze point rendering method, gaze point rendering system, computer and readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310879693.5A CN116597288B (en) | 2023-07-18 | 2023-07-18 | Gaze point rendering method, gaze point rendering system, computer and readable storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN116597288A true CN116597288A (en) | 2023-08-15 |
| CN116597288B CN116597288B (en) | 2023-09-12 |
Family
ID=87599566
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310879693.5A Active CN116597288B (en) | 2023-07-18 | 2023-07-18 | Gaze point rendering method, gaze point rendering system, computer and readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116597288B (en) |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110378914A (en) * | 2019-07-22 | 2019-10-25 | 北京七鑫易维信息技术有限公司 | Rendering method and device, system, display equipment based on blinkpunkt information |
| CN112164016A (en) * | 2020-09-23 | 2021-01-01 | 京东方科技集团股份有限公司 | Image rendering method and system, VR (virtual reality) equipment, device and readable storage medium |
| WO2021135607A1 (en) * | 2019-12-31 | 2021-07-08 | 歌尔股份有限公司 | Method and device for optimizing neural network-based target classification model |
| CN113992907A (en) * | 2021-10-29 | 2022-01-28 | 南昌虚拟现实研究院股份有限公司 | Eyeball parameter checking method, system, computer and readable storage medium |
| CN114816060A (en) * | 2022-04-23 | 2022-07-29 | 中国人民解放军军事科学院国防科技创新研究院 | User fixation point estimation and precision evaluation method based on visual tracking |
| CN115147819A (en) * | 2022-07-07 | 2022-10-04 | 西安电子科技大学 | Driver fixation point prediction method based on fixation point prediction model |
| CN115457188A (en) * | 2022-09-19 | 2022-12-09 | 遥在(山东)数字科技有限公司 | 3D rendering display method and system based on fixation point |
-
2023
- 2023-07-18 CN CN202310879693.5A patent/CN116597288B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110378914A (en) * | 2019-07-22 | 2019-10-25 | 北京七鑫易维信息技术有限公司 | Rendering method and device, system, display equipment based on blinkpunkt information |
| WO2021135607A1 (en) * | 2019-12-31 | 2021-07-08 | 歌尔股份有限公司 | Method and device for optimizing neural network-based target classification model |
| CN112164016A (en) * | 2020-09-23 | 2021-01-01 | 京东方科技集团股份有限公司 | Image rendering method and system, VR (virtual reality) equipment, device and readable storage medium |
| CN113992907A (en) * | 2021-10-29 | 2022-01-28 | 南昌虚拟现实研究院股份有限公司 | Eyeball parameter checking method, system, computer and readable storage medium |
| CN114816060A (en) * | 2022-04-23 | 2022-07-29 | 中国人民解放军军事科学院国防科技创新研究院 | User fixation point estimation and precision evaluation method based on visual tracking |
| CN115147819A (en) * | 2022-07-07 | 2022-10-04 | 西安电子科技大学 | Driver fixation point prediction method based on fixation point prediction model |
| CN115457188A (en) * | 2022-09-19 | 2022-12-09 | 遥在(山东)数字科技有限公司 | 3D rendering display method and system based on fixation point |
Non-Patent Citations (1)
| Title |
|---|
| 戴帅凡;田丰;: "虚拟现实影视渲染优化技术现状概述", 现代电影技术, no. 04, pages 28 - 31 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN116597288B (en) | 2023-09-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108197618B (en) | Method and device for generating human face detection model | |
| CN108122234A (en) | Convolutional neural networks training and method for processing video frequency, device and electronic equipment | |
| CN106228293A (en) | teaching evaluation method and system | |
| US20190171280A1 (en) | Apparatus and method of generating machine learning-based cyber sickness prediction model for virtual reality content | |
| Conati et al. | Further Results on Predicting Cognitive Abilities for Adaptive Visualizations. | |
| KR20170104846A (en) | Method and apparatus for analyzing virtual reality content | |
| CN116843196A (en) | Intelligent training method and system applied to military training | |
| CN112101320A (en) | Model training method, image generation method, device, equipment and storage medium | |
| CN116597288B (en) | Gaze point rendering method, gaze point rendering system, computer and readable storage medium | |
| KR20170105905A (en) | Method and apparatus for analyzing virtual reality content | |
| CN109102486A (en) | Detection method of surface flaw and device based on machine learning | |
| CN116994084A (en) | Regional intrusion detection model training method and regional intrusion detection method | |
| CN116977209A (en) | Image noise reduction method, electronic device and storage medium | |
| Gomes et al. | A QoE evaluation of an immersive virtual reality autonomous driving experience | |
| KR101717399B1 (en) | The automatic system and method for measuring subjective quality score, recording medium thereof | |
| Zováthi et al. | ST-DepthNet: A spatio-temporal deep network for depth completion using a single non-repetitive circular scanning Lidar | |
| KR102256004B1 (en) | Prediction apparatus of protein solvation free energy using deep learning and method thereof | |
| CN109166182B (en) | AR simulation processing method and device, electronic equipment and readable storage medium | |
| CN112633065A (en) | Face detection method, system, storage medium and terminal based on data enhancement | |
| CN111275642A (en) | Low-illumination image enhancement method based on significant foreground content | |
| CN117237358B (en) | Stereoscopic image quality evaluation method based on metric learning | |
| KR102669002B1 (en) | Apparatus and Method for Predicting Motion Sickness and Dizziness in Virtual Reality Content | |
| CN115761871B (en) | Detection image generation method, device, equipment and medium based on eye movement detection | |
| CN113283402B (en) | Differential two-dimensional fixation point detection method and device | |
| CN114748872B (en) | Game rendering updating method based on information fusion |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |