WO2024245263A1 - 一种视线预测模型的构建方法、装置、设备及存储介质 - Google Patents

一种视线预测模型的构建方法、装置、设备及存储介质 Download PDF

Info

Publication number
WO2024245263A1
WO2024245263A1 PCT/CN2024/095931 CN2024095931W WO2024245263A1 WO 2024245263 A1 WO2024245263 A1 WO 2024245263A1 CN 2024095931 W CN2024095931 W CN 2024095931W WO 2024245263 A1 WO2024245263 A1 WO 2024245263A1
Authority
WO
WIPO (PCT)
Prior art keywords
sight
line
label
model
prediction
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2024/095931
Other languages
English (en)
French (fr)
Inventor
王亮亮
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Zitiao Network Technology Co Ltd
Original Assignee
Beijing Zitiao Network Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Zitiao Network Technology Co Ltd filed Critical Beijing Zitiao Network Technology Co Ltd
Publication of WO2024245263A1 publication Critical patent/WO2024245263A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/18—Eye characteristics, e.g. of the iris
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0464—Convolutional networks [CNN, ConvNet]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/047—Probabilistic or stochastic networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/082—Learning methods modifying the architecture, e.g. adding, deleting or silencing nodes or connections
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/084—Backpropagation, e.g. using gradient descent
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/09—Supervised learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/096—Transfer learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/72—Data preparation, e.g. statistical preprocessing of image or video features
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/774—Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
    • Y02T10/00—Road transport of goods or passengers
    • Y02T10/10—Internal combustion engine [ICE] based vehicles
    • Y02T10/40—Engine management systems

Definitions

  • the present disclosure relates to the field of computer vision technology, and in particular to a method, device, equipment and storage medium for constructing a sight line prediction model.
  • Gaze prediction technology is a technology that outputs gaze direction through a network structure given an eye image. It is widely used in human-computer interaction, virtual reality, augmented reality, medical analysis and other fields.
  • a method for constructing a sight line prediction model comprising:
  • Input a plurality of the eye sample images into the student model to be constructed to obtain a plurality of sight line prediction information, wherein each of the eye sample images and each of the sight line reference information corresponds to each of the sight line prediction information.
  • the model parameters of the student model to be constructed are updated based on the multiple line of sight prediction information and the corresponding target line of sight labels; otherwise, the student model to be constructed is determined to be the line of sight prediction model.
  • a sight line prediction method comprising:
  • An eye image is input into a sight line prediction model to obtain sight line information, wherein the sight line prediction model is constructed by the sight line prediction model construction method described in the exemplary embodiment of the present disclosure.
  • a construction device comprising:
  • the prediction module is used to input multiple eye sample images into the constructed teacher model to obtain multiple sight reference information.
  • a construction module is used to input a plurality of the eye sample images into the student model to be constructed, obtain a plurality of sight line prediction information, each of the eye sample images, each of the sight line reference information corresponds to each of the sight line prediction information, and if the loss of the student model to be constructed is determined based on the plurality of sight line prediction information and the corresponding target sight line labels, the loss does not meet the collection requirement.
  • Convergence condition based on the multiple line of sight prediction information and the corresponding target line of sight labels, update the model parameters of the student model to be constructed, if it is determined based on the multiple line of sight prediction information and the corresponding target line of sight labels that the loss of the student model to be constructed meets the convergence condition, determine that the student model to be constructed is the line of sight prediction model.
  • the target gaze label of the eye sample image is the corresponding gaze reference information.
  • a sight line prediction device comprising:
  • the acquisition module is used to acquire eye images.
  • a prediction module is used to input an eye image into a sight line prediction model to obtain sight line information.
  • the sight line prediction model is constructed by the sight line prediction model construction method described in the exemplary embodiment of the present disclosure.
  • an electronic device comprising:
  • Memory for storing programs.
  • the program includes instructions, which, when executed by the processor, cause the processor to perform the method according to the exemplary embodiment of the present disclosure.
  • a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to execute the method according to the exemplary embodiments of the present disclosure.
  • FIG1 shows a flow chart of a method for constructing a sight line prediction model according to an exemplary embodiment of the present disclosure
  • FIG2 shows a flow chart of a sight line prediction method according to an exemplary embodiment of the present disclosure
  • FIG3 shows a schematic block diagram of functional modules of a construction device according to an exemplary embodiment of the present disclosure
  • FIG4 shows a schematic block diagram of functional modules of a sight line prediction device according to an exemplary embodiment of the present disclosure
  • FIG5 shows a schematic block diagram of a chip according to an exemplary embodiment of the present disclosure
  • FIG. 6 shows a block diagram of an exemplary electronic device that can be used to implement an embodiment of the present disclosure.
  • Virtual reality also known as virtual reality or spiritual realm technology
  • Virtual reality technology encompasses computer, electronic information, and simulation technology. Its basic implementation method is based on computer technology, using and integrating the latest developments of various high technologies such as 3D graphics technology, multimedia technology, simulation technology, display technology, and servo technology. With the help of computers and other equipment, a realistic 3D visual, tactile, olfactory and other sensory experiences are generated in the virtual world, so that people in the virtual world have an immersive feeling.
  • Knowledge distillation is a construction method based on the "teacher-student network concept". As the name suggests, it is to distill the knowledge contained in the already constructed teacher model into the student model. By introducing soft targets related to the teacher model as part of the total loss, it can induce the construction of the student model and realize knowledge transfer.
  • the loss function also known as the objective function, is an important equation used to measure the difference between the predicted value and the target value.
  • Model parameters refer to the optimal parameters obtained by constructing samples during model construction, which can be understood as the "internal parameters" of the model.
  • the VR device can collect the user's eye image through the eye tracking system, and predict the user's eye line based on the user's eye image, and then use the predicted user's eye line to control the VR device to perform eye tracking to improve the user's comfort.
  • Gaze prediction technology is a technology that outputs gaze direction through a network structure given an eye image. It is widely used in human-computer interaction, virtual reality, augmented reality, medical analysis and other fields.
  • the subject will be guided to look at different points, and a data can be obtained by collecting the eye image at this time, and each point corresponds to a known gaze direction, which can be used as a label for this data.
  • a known gaze direction which can be used as a label for this data.
  • the user's eye image can be input into the line of sight prediction model to obtain the user's line of sight.
  • the line of sight prediction model a large number of eye sample images with line of sight labels can be used as a data set to construct the line of sight prediction model.
  • the subject can be guided to look at the preset gaze point, but the subject may not follow the guidance to look at the preset gaze point for some reason, resulting in a mismatch between the collected eye sample image and the corresponding line of sight label.
  • there is a certain amount of dirty data in the data set used are used. These dirty data can be manifested as a mismatch between the eye sample image and the corresponding real line of sight label, resulting in the low prediction accuracy of the constructed line of sight prediction model.
  • the exemplary embodiments of the present disclosure provide a method for constructing a line of sight prediction model and a line of sight prediction method, which can be based on the knowledge distillation theory and use the line of sight reference information output by the teacher model to clean the line of sight labels corresponding to the eye sample images used by the student model, thereby reducing the impact of dirty data on the construction of the line of sight prediction model and improving the prediction accuracy of the line of sight prediction model.
  • the relevant eye images involved in the method for constructing a line of sight prediction model and the line of sight prediction method provided by the exemplary embodiments of the present disclosure can be images containing human eyes or images containing animal eyes, which are not limited here.
  • the model parameters of the student model to be constructed can be updated based on the multiple line of sight prediction information and the corresponding target line of sight labels to ensure that the model parameters of the student model to be constructed are accurate.
  • the exemplary embodiment of the present disclosure determines the loss of the student model to be constructed
  • the loss determined by the line of sight prediction information and the corresponding real line of sight label meets the label screening condition
  • the corresponding line of sight reference information output by the constructed teacher model based on the eye sample image can be determined as the target line of sight label of the eye sample image to reduce the interference caused by the poor matching between the real line of sight label and the corresponding eye sample image, thereby improving the accuracy of the loss of the determined student model to be constructed, and ensuring that the line of sight prediction model finally obtained has a high accuracy.
  • the method of the exemplary embodiment of the present disclosure may be performed by an electronic device or a chip applied to the electronic device.
  • the electronic device may be an electronic device with a display function, such as a mobile phone, a tablet computer, a wearable device, a vehicle-mounted device, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), and a wearable device based on augmented reality (AR) and/or virtual reality (VR) technology.
  • a display function such as a mobile phone, a tablet computer, a wearable device, a vehicle-mounted device, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), and a wearable device based on augmented reality (AR) and/or virtual reality (VR) technology.
  • AR augmented reality
  • VR virtual reality
  • the wearable device can also be a general term for wearable devices that are intelligently designed and developed using wearable technology for daily wear, such as glasses, gloves, watches, clothing and shoes.
  • a wearable device is a portable device that is worn directly on the body or integrated into the user's clothes or accessories.
  • Wearable devices are not just hardware devices, but also achieve powerful functions through software support, data interaction, and cloud interaction.
  • wearable smart devices include those that are fully functional, large in size, and can achieve complete or partial functions without relying on smartphones, such as smart watches or smart glasses, as well as those that only focus on a certain type of application function and need to be used in conjunction with other devices such as smartphones, such as various smart bracelets and smart jewelry for vital sign monitoring.
  • FIG1 illustrates a flow chart of a method for constructing a sight line prediction model provided by an exemplary embodiment of the present disclosure.
  • the method for constructing a sight line prediction model of an exemplary embodiment of the present disclosure includes:
  • Step 101 Input multiple eye sample images to the constructed teacher model to obtain multiple sight line reference information.
  • the eye sample images may be eye sample images collected by an image acquisition device, and the eye sample images may be eye pictures or eye videos.
  • eye sample images can be collected by the camera in the VR device, and then the collected multiple eye sample images can be input into the constructed teacher model for training.
  • the teacher model can be any existing open source neural network model for line of sight prediction, which is not limited here.
  • VCG Visual Geometry Group
  • ResNet Deep Residual Network
  • CNN Convolutional Neural Networks
  • the construction process of the teacher model can be: by inputting multiple eye sample images with sight line labels into the teacher model to obtain the corresponding multiple sight line reference information, solving the loss based on the sight line reference information corresponding to the multiple eye sample images and the real sight line labels, if the loss meets the iteration condition, it means that the sight line reference information of the eye sample images of the teacher model is inaccurate, and the model parameters of the teacher model can be updated based on the sight line reference information corresponding to the multiple eye sample images and the real sight line labels (the model parameters may include weights and offset values). If the loss meets the iteration termination condition, it means that the sight line reference information of the eye sample images predicted by the teacher model is relatively accurate, and the constructed teacher model can be obtained.
  • the real sight line labels carried by the above eye sample images should be as accurate as possible to improve the accuracy of the teacher model.
  • the eye sample images used in the process of building the teacher model can be the same as or different from the eye sample images input into the teacher model when building the student model, which is not limited here.
  • Step 102 Input multiple eye sample images to the student model to be constructed to obtain multiple sight line prediction information.
  • the sample image, the sight reference information and the sight prediction information correspond to each other.
  • the description of inputting multiple eye sample images to be constructed into the student model can refer to the related description above, which will not be repeated here.
  • the multiple eye sample images input to the student model to be constructed and the multiple eye sample images input to the completed teacher model of the exemplary embodiment of the present disclosure may be the same.
  • the same eye sample image is input to the completed teacher model and the student model to be constructed, and the same eye sample image may correspond to the sight line reference information output by the completed teacher model and the sight line prediction information output by the student model to be constructed, respectively, and there is also a corresponding relationship between the sight line reference information and the sight line prediction information.
  • the exemplary embodiment of the present disclosure may also clean the real sight labels so that the real sight labels corresponding to the eye sample images used in the loss calculation have a high degree of matching with the eye sample images, thereby improving the accuracy of the student model to be constructed.
  • the method of the exemplary embodiment of the present disclosure may also include:
  • Step 103 Determine whether the loss determined by the sight prediction information and the corresponding real sight label meets the label screening condition.
  • the real sight label carried by the eye sample image corresponding to the sight prediction information may be the real sight label corresponding to the sight prediction information.
  • the loss can be determined by the sight prediction information and the corresponding real sight label.
  • step 104 can be executed.
  • step 105 can be executed.
  • Step 104 Determine that the target sight line label of the eye sample image includes the corresponding sight line reference information.
  • the sight line reference information corresponding to the eye sample image can be used to replace the true sight line label carried by the eye sample image, and the sight line reference information corresponding to the eye sample image can be used as the target sight line label to determine the convergence of the student model to be constructed, thereby improving the convergence accuracy of the student model to be constructed.
  • the exemplary embodiment of the present disclosure can be used to determine the loss of the line of sight prediction information and the corresponding real line of sight label.
  • the label screening conditions are met, the construction interference caused by the poor matching between the real sight line label and the corresponding eye sample image is reduced.
  • Step 105 Determine that the target sight line label of the eye sample image includes the corresponding real sight line label.
  • the loss determined by the sight line prediction information and the corresponding real sight line label does not meet the label screening condition, so there is no need to clean the real sight line label, and therefore, the target sight line label of the eye sample image can be determined to be the corresponding real sight line label.
  • Step 106 Determine whether the loss of the student model to be constructed based on the multiple sight line prediction information and the corresponding target sight line label meets the convergence condition.
  • the architecture of the student model can select a lightweight model architecture, such as SqueezeNet, MobileNet or ShuffleNet.
  • the multiple sight line prediction information and the corresponding target sight line labels can be substituted into the loss function to determine the loss of the student model to be constructed.
  • the loss function can be a softmax loss function or other possible loss functions, such as a cross entropy loss.
  • the convergence condition here can be that the loss of the student model to be constructed is less than or equal to the preset loss, or that the loss of the student model to be constructed is stable, that is, the loss of the student model to be constructed remains constant or fluctuates within a certain range during different iterations.
  • step 107 can be executed.
  • step 108 can be executed.
  • Step 107 Update the model parameters of the student model to be constructed based on the multiple sight line prediction information and the corresponding target sight line labels.
  • the model parameters of the student model to be constructed can be updated by a back propagation algorithm, and then the student model to be constructed after the updated model parameters is reconstructed based on the next batch of eye sample images. It should be understood that the next batch of eye sample images and the current batch of eye sample images can be at least partially the same, or completely different.
  • Step 108 Determine that the student model to be constructed is a sight line prediction model. For example, if the loss of the student model to be constructed is determined to meet the convergence condition based on multiple sight line prediction information and corresponding target sight line labels, the construction of the student model to be constructed can be directly terminated, and the current student model to be constructed can be used as the sight line prediction model.
  • the label screening condition of the exemplary embodiment of the present disclosure may be limited by threshold comparison or by loss sorting.
  • the label screening condition includes: the loss determined based on the line of sight prediction information and the corresponding real line of sight label is greater than the loss threshold. It should be understood that here The loss threshold can be determined empirically.
  • the eye sample images whose loss determined by the gaze prediction information and the corresponding true gaze label is greater than the loss threshold are determined as dirty data, and the gaze reference information output by the constructed teacher model with higher gaze prediction accuracy is used as the target gaze label, and the loss is re-determined with the corresponding gaze prediction information, thereby cleaning the dirty data in the eye sample images and avoiding the influence of dirty data on the accuracy of constructing the student model to be constructed.
  • the method provided by the exemplary embodiment of the present disclosure may further include: sorting in descending order according to the losses determined based on the line of sight prediction information and the corresponding real line of sight labels, and the label filtering condition includes: the order of the losses determined based on the line of sight prediction information and the corresponding real line of sight labels in the sorting is less than or equal to M, where M represents an integer less than the total number of line of sight prediction information.
  • the loss corresponding to each eye sample image can be obtained by bringing the gaze prediction information and the corresponding true gaze label into the loss function for solution, and the losses of different eye sample images are sorted in descending order. If the order of the loss determined by the gaze prediction information and the corresponding true gaze label in the sorting is less than or equal to M, it means that the true gaze label may not match the corresponding eye sample image. Therefore, it can be considered that the loss determined by the gaze prediction information and the corresponding true gaze label meets the label filtering condition, and the target gaze label of the eye sample image corresponding to the gaze prediction information is set as the corresponding gaze reference information, thereby avoiding the influence of dirty data on the accuracy of constructing the student model to be constructed.
  • the above M can be considered as the number of sample errors in the eye sample image construction set, which can be determined by the cleaning rate of the eye sample images and the total number of eye sample images.
  • the cleaning rate of the eye sample images can reflect the error rate of the eye sample images, which can be determined by statistics or customized according to actual conditions.
  • the total number of eye sample images can be the number of eye sample images required to update the model parameters of the student model to be constructed once, or the number of eye sample images in a batch of the student model to be constructed.
  • the above M is positively correlated with the cleaning rate of the eye sample images, and M is positively correlated with the total number of the eye sample images.
  • the sight prediction information corresponding to the eye sample image and its corresponding true sight label can be brought into the loss function for solution to obtain the loss corresponding to each eye sample image, and the sight prediction information corresponding to each loss is sorted based on the order of the losses of different eye sample images from large to small.
  • the target sight labels corresponding to the first 10 sight prediction information are updated to their corresponding sight reference information, and the target sight labels corresponding to the remaining 990 sight prediction information are updated to their corresponding true sight labels, so as to re-determine the convergence of the student model to be constructed and reduce the influence of dirty data on the accuracy of the student model to be constructed.
  • all target sight lines corresponding to the multiple eye sample images may include real sight lines labels corresponding to different eye sample images and sight line reference information corresponding to different eye sample images.
  • the sight line reference information can be used to clean dirty data in all real sight lines labels corresponding to the multiple eye sample images, thereby reducing the impact of dirty data on the accuracy of constructing the student model to be constructed.
  • the loss determined by the line of sight prediction information and the corresponding real line of sight label does not meet the label filtering condition, it is determined that the target line of sight label corresponding to the line of sight prediction information also includes corresponding line of sight reference information, and the loss determined by the line of sight prediction information and the corresponding target line of sight label includes a first loss and a second loss. Based on the first loss and the second loss, the loss of the corresponding eye sample image can be determined.
  • the loss determined by the sight prediction information and the corresponding real sight label does not meet the label screening condition, it means that there is no dirty data in the current eye sample image, and the real sight label corresponding to the eye sample image can be directly used to calculate the loss.
  • the sight reference information and the real sight label output by the teacher model with higher accuracy can be used to supervise the student model to be constructed at the same time.
  • the target sight label of the sight prediction information corresponding to the same eye sample image can be selected from the corresponding sight reference information and the corresponding real sight label, and the first loss is determined by the sight prediction information and the corresponding real sight label, and the second loss is determined by the sight prediction information and the corresponding sight reference information, and then the loss of the corresponding eye sample image is determined based on the first loss and the second loss.
  • the loss of the corresponding eye sample image determined based on the first loss and the second loss here can be obtained by weighted summing the first loss and the second loss, or by using other methods, which are not limited here. In some examples, the weight of the first loss is greater than the weight of the second loss.
  • the loss function involved in the sight line prediction model of the exemplary embodiments of the present disclosure can also be expressed as:
  • the exemplary embodiment of the present disclosure also provides a sight line prediction method for predicting the sight line direction in an eye image.
  • FIG2 illustrates a flow chart of the sight line prediction method provided by the exemplary embodiment of the present disclosure. As shown in FIG2 , the sight line prediction method includes:
  • Step 201 Obtain an eye image.
  • the eye sample image is an eye sample image containing a sight line direction. It should be understood that the eye sample image here can be a picture or a video image.
  • the eye image can be acquired by an image acquisition device, which can be a camera or a video camera. It can be a separate device, or a camera of a computing device with a display screen, or a camera installed on a bicycle, motorcycle, car or other vehicle.
  • the camera can be a monocular camera or a binocular camera, etc.
  • Step 202 Input the eye image into the sight line prediction model to obtain sight line information.
  • the sight line prediction model can be constructed by a method for constructing a sight line prediction model.
  • the sight line information can be the "gaze direction" of the user collected by the above-mentioned image acquisition device.
  • the sight line information is the "gaze direction” of the user
  • the sight line prediction method is used in the offline retail field, the user's interest in the products on the shelf can be obtained by obtaining the user's current "gaze direction", thereby realizing product selection and timely adjustment of the placement.
  • the device of the present disclosure includes hardware structures and/or software modules corresponding to the execution of each function. It should be easily appreciated by those skilled in the art that, in combination with the units and algorithm steps of each example described in the embodiment disclosed herein, the present disclosure can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present disclosure.
  • the embodiments of the present disclosure may divide the functional units of the device of the present disclosure according to the above method examples.
  • each functional module may be divided corresponding to each function, or two or more functions may be integrated into one processing module.
  • the above integrated modules may be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiments of the present disclosure is schematic and is only a logical functional division. There may be other division methods in actual implementation.
  • the exemplary embodiment of the present disclosure provides a construction device, which can be an electronic device or a chip applied to an electronic device.
  • FIG3 shows a schematic block diagram of the functional modules of the construction device according to the exemplary embodiment of the present disclosure.
  • the construction device 300 includes:
  • the prediction module 301 is used to input multiple eye sample images into the constructed teacher model to obtain multiple sight line reference information. Input multiple eye sample images into the student model to be constructed to obtain multiple sight line prediction information. Each eye sample image and each sight line reference information corresponds to each sight line prediction information.
  • the determination module 302 is used to determine the target sight line label of the eye sample image as the corresponding sight line reference information when the loss determined by the sight line prediction information and the corresponding real sight line label meets the label screening condition, otherwise, determine the target sight line label of the eye sample image as the corresponding real sight line label. If the loss of the student model to be constructed based on multiple sight line prediction information and the corresponding target sight line labels meets the convergence condition, the student model to be constructed is determined to be a sight line prediction model.
  • the updating module 303 is used to update the model parameters of the student model to be constructed based on the multiple sight line prediction information and the corresponding target sight line labels if it is determined based on the multiple sight line prediction information and the corresponding target sight line labels that the loss of the student model to be constructed does not meet the convergence condition.
  • the label screening condition includes: a loss determined by the line of sight prediction information and the corresponding real line of sight label is greater than a loss threshold.
  • the label screening condition includes: sorting multiple line of sight prediction information in descending order according to the loss determined based on the line of sight prediction information and the corresponding target line of sight label, the line of sight prediction information is the kth line of sight prediction information, k is greater than 0 and less than or equal to M, and M represents an integer less than the total number of line of sight prediction information.
  • the above M is positively correlated with the cleaning rate of the eye sample images, and M is positively correlated with the total number of the eye sample images.
  • the loss determined by the line of sight prediction information and the corresponding real line of sight label does not meet the label filtering condition, it is determined that the target line of sight label corresponding to the line of sight prediction information also includes corresponding line of sight reference information, and the loss determined by the line of sight prediction information and the corresponding target line of sight label includes a first loss and a second loss.
  • the first loss is determined by the sight line prediction information and the corresponding true sight line label
  • the second loss is determined by the sight line prediction information and the corresponding sight line reference information.
  • the exemplary embodiment of the present disclosure further provides a sight line prediction device, which can be an electronic device or a chip applied to an electronic device.
  • FIG4 shows a schematic block diagram of the functional modules of the sight line prediction device according to the exemplary embodiment of the present disclosure. As shown in FIG4, the sight line prediction device 400 includes:
  • the acquisition module 401 is used to acquire an eye image.
  • the prediction module 402 is used to input the eye image into the sight line prediction model to obtain the sight line information.
  • the sight line prediction model is constructed by the above-mentioned sight line prediction model construction method.
  • FIG5 shows a schematic block diagram of a chip according to an exemplary embodiment of the present disclosure.
  • the chip 500 includes one or more (including two) processors 501 and a communication interface 502.
  • the communication interface 502 can support the electronic device to perform the data transceiving steps in the above method, and the processor 501 can support the electronic device to perform the data processing steps in the above method.
  • the chip 500 further includes a memory 503, which may include a read-only memory and a random access memory, and provides operation instructions and data to the processor.
  • a portion of the memory may also include a non-volatile random access memory (NVRAM).
  • NVRAM non-volatile random access memory
  • the processor 501 performs corresponding operations by calling operation instructions stored in the memory (the operation instructions may be stored in the operating system).
  • the processor 501 controls the processing operations of any one of the terminal devices, and the processor may also be referred to as a central processing unit (CPU).
  • the memory 503 may include a read-only memory and a random access memory, and provides instructions and data to the processor 501.
  • a portion of the memory 503 may also include NVRAM.
  • the memory, the communication interface, and the memory are coupled together through a bus system, wherein the bus system may include a power bus, a control bus, and a status signal bus in addition to a data bus.
  • various buses are labeled as bus system 504 in FIG. 5 .
  • the method disclosed in the above-mentioned embodiment of the present disclosure can be applied to a processor or implemented by a processor.
  • the processor may be an integrated circuit chip with signal processing capabilities.
  • each step of the above-mentioned method can be completed by an integrated logic circuit of hardware in the processor or an instruction in the form of software.
  • the above-mentioned processor can be a general-purpose processor, a digital signal processor (digital signal processing, DSP), an application-specific integrated circuit (application specific integrated circuit, ASIC), a field-programmable gate array (field-programmable gate array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.
  • DSP digital signal processing
  • ASIC application specific integrated circuit
  • FPGA field-programmable gate array
  • the disclosed methods, steps and logic block diagrams in the embodiments of the present disclosure can be implemented or executed.
  • the general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc.
  • the steps of the method disclosed in conjunction with the embodiments of the present disclosure can be directly embodied as a hardware decoding processor to be executed, or a combination of hardware and software modules in the decoding processor can be executed.
  • the software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc.
  • the storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.
  • the exemplary embodiment of the present disclosure also provides an electronic device, including: a processor; and a memory communicating with the processor and storing a program.
  • the program includes instructions, and when the instructions are executed by the processor, the processor executes the method according to the exemplary embodiment of the present disclosure.
  • the exemplary embodiments of the present disclosure also provide a non-transitory computer-readable storage medium storing a computer program, wherein the computer instructions are used to cause the computer to execute the method according to the embodiments of the present disclosure.
  • Electronic devices 600 that can be used as a server or a client of an exemplary embodiment of the present disclosure will now be described.
  • Electronic devices are intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers.
  • Electronic devices can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices.
  • the components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and/or required herein.
  • the electronic device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 606 to a random access memory (RAM) 603.
  • ROM read-only memory
  • RAM random access memory
  • various programs and data required for the operation of the device 600 can also be stored.
  • the computing unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604.
  • An input/output (I/O) interface 605 is also connected to the bus 604.
  • a plurality of components in the electronic device 600 are connected to the I/O interface 605, including: an input unit 606, an output unit 607, a storage unit 608, and a communication unit 609.
  • the input unit 606 may be any type of device capable of inputting information to the electronic device 600, and the input unit 606 may receive input digital or character information, and generate key signal inputs related to user settings and/or function control of the electronic device.
  • the output unit 607 may be any type of device capable of presenting information, and may include, but is not limited to, a display, a speaker, a video/audio output terminal, a vibrator, and/or a printer.
  • the storage unit 604 may include, but is not limited to, a disk, an optical disk.
  • the communication unit 609 allows the electronic device 600 to exchange information/data with other devices through a computer network such as the Internet and/or various telecommunication networks, and may include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver, and/or a chipset, such as a Bluetooth TM device, a WiFi device, a WiMax device, a cellular communication device, and/or the like.
  • the computing unit 601 may be a variety of general and/or special processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc.
  • the computing unit 601 performs the various methods and processes described above. For example, in some embodiments, the aforementioned method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 608.
  • part or all of the computer program may be loaded and/or installed on the electronic device 600 via the ROM 602 and/or the communication unit 609.
  • the computing unit 601 may be configured to perform the aforementioned method in any other appropriate manner (e.g., by means of firmware).
  • the program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that when the program code is executed by the processor or controller, the functions/operations specified in the flow chart and/or block diagram are executed.
  • the program code may execute entirely on the machine, partly on the machine, as a stand-alone software package, partly on the machine and partly on a remote machine or entirely on the remote machine or server.
  • a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment.
  • a machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium.
  • a machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing.
  • a more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
  • RAM random access memory
  • ROM read-only memory
  • EPROM or flash memory erasable programmable read-only memory
  • CD-ROM portable compact disk read-only memory
  • CD-ROM compact disk read-only memory
  • magnetic storage device or any suitable combination of the foregoing.
  • machine-readable medium and “computer-readable medium” refer to any computer program product, apparatus, and/or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and/or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal.
  • machine-readable signal refers to any signal for providing machine instructions and/or data to a programmable processor.
  • the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a cathode ray tube (CRT) or a liquid crystal display (LCD) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer.
  • a display device e.g., a cathode ray tube (CRT) or a liquid crystal display (LCD) monitor
  • a keyboard and pointing device e.g., a mouse or trackball
  • Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).
  • the systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components.
  • the components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.
  • a computer system may include clients and servers.
  • Clients and servers are generally remote from each other and usually interact through a communication network.
  • the relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.
  • the computer program product includes one or more computer programs or instructions.
  • the computer may be a general-purpose computer, a special-purpose computer, a computer network, a terminal, a user device or other programmable device.
  • the computer program or instruction may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer program or instruction may be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired or wireless means.
  • the computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server, data center, etc. that integrates one or more available media.
  • the available medium may be a magnetic medium, for example, a floppy disk, a hard disk, a tape; it may also be an optical medium, for example, a digital video disc (DVD); it may also be a semiconductor medium, for example, a solid state drive (SSD).

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Health & Medical Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • General Physics & Mathematics (AREA)
  • Evolutionary Computation (AREA)
  • Artificial Intelligence (AREA)
  • Software Systems (AREA)
  • Computing Systems (AREA)
  • Biophysics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Data Mining & Analysis (AREA)
  • General Engineering & Computer Science (AREA)
  • Computational Linguistics (AREA)
  • Mathematical Physics (AREA)
  • Biomedical Technology (AREA)
  • Molecular Biology (AREA)
  • Multimedia (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Databases & Information Systems (AREA)
  • Medical Informatics (AREA)
  • Probability & Statistics with Applications (AREA)
  • Ophthalmology & Optometry (AREA)
  • Human Computer Interaction (AREA)
  • Image Analysis (AREA)

Abstract

一种视线预测模型的构建方法、装置、设备及存储介质，方法包括：向构建完成的老师模型和待构建学生模型输入多个眼部样本图像，分别获得多个视线参考信息和多个视线预测信息；若基于多个视线预测信息和对应的目标视线标签确定待构建学生模型的损失不满足收敛条件，更新待构建学生模型的模型参数，否则确定待构建学生模型为视线预测模型。当视线预测信息和对应的真实视线标签确定的损失满足标签筛选条件时，眼部样本图像的目标视线标签包括对应的视线参考信息，否则眼部样本图像的目标视线标签包括对应的真实视线标签。

Description

一种视线预测模型的构建方法、装置、设备及存储介质
相关申请的交叉引用
本申请以申请号为202310620447.8，申请日为2023年5月29日，题为“一种视线预测模型的构建方法、装置、设备及存储介质”的中国申请为基础，并主张其优先权，该中国申请的公开内容在此作为整体引入本申请中。
技术领域
本公开涉及计算机视觉技术领域，尤其涉及一种视线预测模型的构建方法、装置、设备及存储介质。
背景技术
视线预测技术就是给定眼睛图像，经过网络结构输出视线方向的技术。广泛应用于人机交互、虚拟现实、增强现实、医学分析等领域。
发明内容
根据本公开的一方面，提供了一种视线预测模型的构建方法，包括：
向构建完成的老师模型输入多个眼部样本图像，获得多个视线参考信息。
向待构建学生模型输入多个所述眼部样本图像，获得多个视线预测信息，每个所述眼部样本图像、每个所述视线参考信息与每个所述视线预测信息对应。
当所述视线预测信息和对应的真实视线标签确定的损失满足标签筛选条件时，确定所述眼部样本图像的目标视线标签包括对应的所述视线参考信息，否则，确定所述眼部样本图像的目标视线标签包括对应的所述真实视线标签；
若基于多个所述视线预测信息和对应的目标视线标签确定所述待构建学生模型的损失不满足收敛条件，基于多个所述视线预测信息和对应的目标视线标签更新所述待构建学生模型的模型参数，否则，确定所述待构建学生模型为所述视线预测模型。
根据本公开的另一方面，提供了一种视线预测方法，包括：
获取眼部图像。
向视线预测模型中输入眼部图像，获得视线信息，所述视线预测模型由本公开示例性实施例所述的视线预测模型的构建方法构建。
根据本公开的另一方面，提供了一种构建装置，包括：
预测模块，用于向构建完成的老师模型输入多个眼部样本图像，获得多个视线参考信息。
构建模块，用于向待构建学生模型输入多个所述眼部样本图像，获得多个视线预测信息，每个所述眼部样本图像、每个所述视线参考信息与每个所述视线预测信息对应，若基于多个所述视线预测信息和对应的目标视线标签确定所述待构建学生模型的损失不满足收 敛条件，基于多个所述视线预测信息和对应的目标视线标签更新所述待构建学生模型的模型参数，若基于多个所述视线预测信息和对应的目标视线标签确定所述待构建学生模型的损失满足收敛条件，确定所述待构建学生模型为所述视线预测模型。
当所述视线预测信息和对应的真实视线标签确定的损失满足标签筛选条件时，所述眼部样本图像的目标视线标签为对应的所述视线参考信息。
根据本公开的另一方面，提供了一种视线预测装置，包括：
获取模块，用于获取眼部图像。
预测模块，用于向视线预测模型中输入眼部图像，获得视线信息，所述视线预测模型由本公开示例性实施例所述的视线预测模型的构建方法构建。
根据本公开的另一方面，提供了一种电子设备，包括：
处理器；以及
存储程序的存储器。
其中，所述程序包括指令，所述指令在由所述处理器执行时使所述处理器执行根据本公开示例性实施例所述的方法。
根据本公开的另一方面，提供了一种存储有计算机指令的非瞬时计算机可读存储介质，其中，所述计算机指令用于使所述计算机执行根据本公开示例性实施例所述的方法。
附图说明
在下面结合附图对于示例性实施例的描述中，本公开的更多细节、特征和优点被公开，在附图中：
图1示出了根据本公开示例性实施例的视线预测模型的构建方法的流程图；
图2示出了根据本公开示例性实施例的视线预测方法的流程图；
图3示出了根据本公开示例性实施例的构建装置的功能模块示意性框图；
图4示出了根据本公开示例性实施例的视线预测装置的功能模块示意性框图；
图5示出了根据本公开示例性实施例的芯片的示意性框图；
图6示出了能够用于实现本公开的实施例的示例性电子设备的结构框图。
具体实施方式
下面将参照附图更详细地描述本公开的实施例。虽然附图中显示了本公开的某些实施例，然而应当理解的是，本公开可以通过各种形式来实现，而且不应该被解释为限于这里阐述的实施例，相反提供这些实施例是为了更加透彻和完整地理解本公开。应当理解的是，本公开的附图及实施例仅用于示例性作用，并非用于限制本公开的保护范围。
应当理解，本公开的方法实施方式中记载的各个步骤可以按照不同的顺序执行，和/或并行执行。此外，方法实施方式可以包括附加的步骤和/或省略执行示出的步骤。本公开的 范围在此方面不受限制。
本文使用的术语“包括”及其变形是开放性包括，即“包括但不限于”。术语“基于”是“至少部分地基于”。术语“一个实施例”表示“至少一个实施例”；术语“另一实施例”表示“至少一个另外的实施例”；术语“一些实施例”表示“至少一些实施例”。其他术语的相关定义将在下文描述中给出。需要注意，本公开中提及的“第一”、“第二”等概念仅用于对不同的装置、模块或单元进行区分，并非用于限定这些装置、模块或单元所执行的功能的顺序或者相互依存关系。
需要注意，本公开中提及的“一个”、“多个”的修饰是示意性而非限制性的，本领域技术人员应当理解，除非在上下文另有明确指出，否则应该理解为“一个或多个”。
在介绍本公开实施例之前首先对本公开实施例中涉及到的相关名词作如下释义：
虚拟现实技术(Virtual Reality，VR)，又称虚拟实境或灵境技术，是20世纪发展起来的一项全新的实用技术。虚拟现实技术囊括计算机、电子信息、仿真技术，其基本实现方式是以计算机技术为主，利用并综合三维图形技术、多媒体技术、仿真技术、显示技术、伺服技术等多种高科技的最新发展成果，借助计算机等设备产生一个逼真的三维视觉、触觉、嗅觉等多种感官体验的虚拟世界，从而使处于虚拟世界中的人产生一种身临其境的感觉。
知识蒸馏(Knowledge Distillation，KD)，是一种基于“老师-学生网络思想”的构建方式，顾名思义，就是将已经构建好的老师模型包含的知识(Knowledge)，蒸馏(Distill)提取到学生模型里面。通过引入与老师模型相关的软目标(Soft-target)作为总损失的一部分，以诱导学生模型的构建，实现知识迁移。
损失函数(loss function)又称目标函数(objective function)，用于衡量预测值和目标值的差异的重要方程，损失函数的输出值(loss)越高表示差异越大，那么深度神经网络的构建就变成了尽可能缩小这个loss的过程。
模型参数，是指在模型构建中通过构建样本获得的最优参数，可理解为模型的“内部参数”。
在用户使用VR设备时，VR设备可以通过视线跟踪系统采集用户眼部图像，并基于用户眼部图像预测用户视线，然后利用所预测的用户视线控制VR设备进行视线跟踪，以提高用户的使用舒适性。
视线预测技术就是给定眼睛图像，经过网络结构输出视线方向的技术。广泛应用于人机交互、虚拟现实、增强现实、医学分析等领域。在应用的过程中，被试对象会被引导注视不同的点位，通过采集此时的眼部图像即可得到一个数据，并且每个点位对应一个已知的视线方向，该视线方向即可作为此数据的标签。但是在实际操作过程中，往往存在这样 的情况，例如，被试对象被引导注视点位x，而实际并没有准确的注视该点位，导致视线估计的数据中存在大量脏数据，对模型精度的影响比较大。
具体而言，在相关技术中，可以将用户眼部图像输入视线预测模型，获得用户视线，在构建视线预测模型时，可以使用大量带视线标签的眼部样本图像作为数据集，构建视线预测模型。在采集带视线标签的眼部样本图像时，可以通过引导被试对象注视预设注视点，但是被试对象有可能因为一些原因并未按照引导去注视预设注视点，导致所采集的眼部样本图像与对应的视线标签存在不匹配的问题。基于此，在构建视线预测模型时，所使用的数据集中存在一定数量的脏数据，这些脏数据可以表现为眼部样本图像和对应的真实视线标签不匹配，导致构建完成的视线预测模型的预测精度不高。
针对上述问题，本公开示例性实施例提供了一种视线预测模型的构建方法及视线预测方法，可以基于知识蒸馏理论，利用老师模型输出的视线参考信息清洗学生模型所使用的眼部样本图像对应的视线标签，从而减少脏数据对视线预测模型构建所产生的影响，提高视线预测模型的预测精度。应理解，本公开示例性实施例提供的视线预测模型的构建方法和视线预测方法所涉及的相关眼部图像，其可以是包含人类眼部的图像，也可以是包含动物眼部的图像，此处不作限制。
本公开示例性实施例中提供的一个或多个技术方案中，通过在构建视线预测模型的过程中，向待构建学生模型输入多个眼部样本图像，获得多个视线预测信息，然后基于多个视线预测信息和对应的目标视线标签确定待构建学生模型的损失，若满足收敛条件，则说明待构建学生模型的模型参数准确，待构建学生模型可以比较准确的预测眼部样本图像的视线预测信息。若不满足收敛条件，则说明待构建学生模型的模型参数的准确性不够，待构建学生模型无法准确预测眼部样本图像的视线预测信息，因此，可以基于多个视线预测信息和对应的目标视线标签更新待构建学生模型的模型参数，保证待构建学生模型的模型参数准确。
而且，本公开示例性实施例在确定待构建学生模型的损失时，在视线预测信息和对应的真实视线标签确定的损失满足标签筛选条件时，说明视线预测信息和对应的真实视线标签的差异较大，而造成这种差异的原因可能包括两方面，一方面可能是因为待构建学生模型的模型参数存在问题，另一方面也有可能是因为真实视线标签与对应的眼部样本图像之间的匹配性较差，也就是说该真实视线标签以及对应的眼部样本图像实质为构建样本中的脏数据。基于此，可以将构建完成的老师模型基于该眼部样本图像所输出的对应的视线参考信息确定为眼部样本图像的目标视线标签，以减少因为真实视线标签与对应的眼部样本图像之间匹配性差带来的干扰，从而提高确定的待构建学生模型的损失的准确性，保证最终所获得的视线预测模型具有较高的精度。
本公开示例性实施例的方法可以由电子设备或应用于电子设备的芯片执行。该电子设备可以是具有显示功能的电子设备，例如：手机、平板电脑、可穿戴设备、车载设备、笔记本电脑、超级移动个人计算机(ultra-mobile personal computer，UMPC)、上网本、掌上电脑(Personal Digital Assistant，PDA)以及基于增强现实(augmented reality，AR)和/或虚拟现实(virtual reality，VR)技术的可穿戴设备等电子设备。
示例性的，当电子设备为可穿戴设备时，该可穿戴设备还可以是应用穿戴技术对日常穿戴进行智能化设计、开发出可以穿戴的设备的总称，如眼镜、手套、手表、服饰及鞋等。可穿戴设备即直接穿在身上，或是整合到用户的衣服或配件的一种便携式设备。可穿戴设备不仅仅是一种硬件设备，更是通过软件支持以及数据交互、云端交互来实现强大的功能。广义穿戴式智能设备包括功能全、尺寸大、可不依赖智能手机实现完整或者部分的功能，如智能手表或智能眼镜等，以及只专注于某一类应用功能，需要和其它设备如智能手机配合使用，如各类进行体征监测的智能手环、智能首饰等。
图1示例出了本公开示例性实施例提供的视线预测模型的构建方法的流程图。如图1所示，本公开示例性实施例的视线预测模型的构建方法包括：
步骤101：向构建完成的老师模型输入多个眼部样本图像，获得多个视线参考信息。眼部样本图像可以是通过图像采集装置采集的眼部样本图像，该眼部样本图像可以为眼部图片或眼部视频。
在实际应用中，可以通过VR设备内的摄像头采集的眼部样本图像，然后将采集到的多个眼部样本图像输入至构建完成的老师模型中进行训练。该老师模型可以是现有任意开源的用于视线预测的神经网络模型，此处不做限定。例如：可以是视觉几何图像组(Visual Geometry Group，VGG)网络、深度残差神经网络(Deep residual network，ResNet)或者卷积神经(Convolutional Neural Networks，CNN)网络等。
老师模型的构建过程可以为：通过向老师模型中输入多个带视线标签的眼部样本图像获得对应的多个视线参考信息，基于多个眼部样本图像对应的视线参考信息和真实视线标签求解损失，若损失满足迭代条件，说明老师模型的眼部样本图像的视线参考信息不准确，可以基于多个眼部样本图像对应的视线参考信息和真实视线标签更新老师模型的模型参数(模型参数可以包括权重以及偏移值)。若损失满足迭代终止条件，说明老师模型预测的眼部样本图像的视线参考信息比较准确，可以获得构建完成的老师模型。
上述眼部样本图像所携带的真实视线标签应该尽可能的准确，以提高老师模型的精度。构建老师模型的过程中所使用的眼部样本图像，与构建学生模型时向老师模型中输入的眼部样本图像可以相同，也可以不同，此处不作限定。
步骤102：向待构建学生模型输入多个眼部样本图像，获得多个视线预测信息，眼部 样本图像、视线参考信息与视线预测信息对应。输入待构建学生模型的多个眼部样本图像的描述可以参考前文相关描述，此处不作赘述。
本公开示例性实施例的输入待构建学生模型的多个眼部样本图像和输入构建完成的老师模型的多个眼部样本图像可以相同。例如，当眼部样本图像、视线参考信息与视线预测信息对应时，向构建完成的老师模型和待构建学生模型输入同一眼部样本图像，该同一眼部样本图像可以分别与构建完成的老师模型输出的视线参考信息和待构建学生模型输出的视线预测信息对应，且视线参考信息和视线预测信息之间也具有对应关系。
考虑到所使用的眼部样本图像携带真实视线标签时，可能会存在一定数量的脏数据，本公开示例性实施例还可以对真实视线标签进行清洗，以使得在损失计算时所使用的眼部样本图像对应的真实视线标签与眼部样本图像具有较高的匹配性，从而提高待构建学生模型的精度。基于此，本公开示例性实施例的方法还可以包括：
步骤103：判断视线预测信息和对应的真实视线标签确定的损失是否满足标签筛选条件。此处视线预测信息对应的眼部样本图像所携带的真实视线标签，可以是该视线预测信息对应的真实视线标签。对于任意一个眼部样本图像中，可以通过视线预测信息和对应的真实视线标签确定损失。
针对任意一个眼部样本图像，其损失可以被认为是单样本损失，如果该单样本损失满足标签筛选条件时，说明在损失计算时所使用的眼部样本图像对应的真实视线标签与眼部样本图像的差异较大，而造成这种差异的原因可能包括两方面，一方面可能是因为待构建学生模型的模型参数存在问题，另一方面也有可能是因为真实视线标签与对应的眼部样本图像之间的匹配性较差，也就是说该真实视线标签以及对应的眼部样本图像实质为构建样本中的脏数据，无法直接使用视线预测信息和对应真实视线标签进行待构建学生模型的收敛性确认，此时，可以执行步骤104。当视线预测信息和对应的真实视线标签确定的损失不满足标签筛选条件时，说明在损失计算时所使用的眼部样本图像对应的真实视线标签与眼部样本图像的匹配性较好，可以直接使用视线预测信息和对应真实视线标签进行待构建学生模型的收敛性确认，此时，可以执行步骤105。
步骤104：确定眼部样本图像的目标视线标签包括对应的视线参考信息。此处由于将同一个眼部样本图像分别输入待构建学生模型和构建完成的老师模型中，而构建完成的老师模型具有较高的视线预测精度，因此，在视线预测信息和对应的真实视线标签确定的损失满足标签筛选条件时，可以使用该眼部样本图像对应的视线参考信息替换眼部样本图像携带的真实视线标签，并将该眼部样本图像对应的视线参考信息作为目标视线标签，用以确定待构建学生模型的收敛性，从而提高待构建学生模型的收敛准确性。
可见，本公开示例性实施例可以在视线预测信息和对应的真实视线标签确定的损失满 足标签筛选条件时，减少因为真实视线标签与对应的眼部样本图像之间匹配性差带来的构建干扰。
步骤105：确定眼部样本图像的目标视线标签包括对应的真实视线标签。这种情况下，视线预测信息和对应的真实视线标签确定的损失不满足标签筛选条件，因此，不需要对真实视线标签进行清洗，因此，可以确定眼部样本图像的目标视线标签为对应的真实视线标签。
步骤106：判断基于多个视线预测信息和对应的目标视线标签确定待构建学生模型的损失是否满足收敛条件。此处学生模型的架构可以选择轻量型的模型架构，例如SqueezeNet，MobileNet或ShuffleNet。
在实际应用中，在基于多个视线预测信息和对应的目标视线标签确定待构建学生模型的损失时，可以将多个视线预测信息和对应的目标视线标签代入损失函数中，从而确定待构建学生模型的损失。该损失函数可以为softmax损失函数，还可以是其它可能的损失函数，如交叉熵损失等。
此处的收敛条件可以是待构建学生模型的损失小于或等于预设损失，还可以是待构建学生模型的损失稳定，也就是在不同迭代过程中待构建学生模型的损失保持恒定或者在某个范围内波动。
当待构建学生模型的损失不满足收敛条件时，说明学生模型中输出的视线预测信息与目标视线标签的差异较大，无法满足推理阶段的视线预测要求(例如视线预测精度未达到要求)，因此，可以执行步骤107。当待构建学生模型的损失满足收敛条件时，说明待构建学生模型中输出的视线预测信息与目标视线标签的匹配性较好，且满足推理阶段的视线预测要求(例如视线预测精度达到要求)，因此，可以执行步骤108。
步骤107：基于多个视线预测信息和对应的目标视线标签更新待构建学生模型的模型参数。此处，可以通过反向传播算法更新待构建学生模型的模型参数，然后重新基于下一批次的眼部样本图像构建更新模型参数后的待构建学生模型。应理解，下一批次的眼部样本图像和当前批次的眼部样本图像可以至少部分相同，也可以完全不同。
步骤108：确定待构建学生模型为视线预测模型。例如，若基于多个视线预测信息和对应的目标视线标签确定待构建学生模型的损失满足收敛条件，可以直接结束待构建学生模型的构建，将当前待构建学生模型作为视线预测模型。
作为一种可能的实现方式，本公开示例性实施例的标签筛选条件可以通过阈值比较方式限制，也可以通过损失排序的方式限制。
当本公开示例性实施例的标签筛选条件通过阈值比较方式限制时，该标签筛选条件包括：基于视线预测信息和对应的真实视线标签确定的损失大于损失阈值。应理解，这里的 损失阈值可以根据经验确定。
在实际应用中，通过判断视线预测信息和对应的真实视线标签确定的损失与损失阈值的关系，将视线预测信息和对应的真实视线标签确定的损失大于损失阈值的眼部样本图像确定为脏数据，使用视线预测精度较高的构建完成的老师模型输出的视线参考信息作为目标视线标签，与对应的视线预测信息重新确定损失，从而将眼部样本图像中的脏数据清洗掉，避免了脏数据对构建待构建学生模型的精度产生影响。
当本公开示例性实施例的标签筛选条件通过损失排序的方式限制时，本公开示例性实施例提供的方法还可以包括：按照基于视线预测信息和对应的真实视线标签确定的损失从大到小的顺序排序，标签筛选条件包括：基于视线预测信息和对应的真实视线标签确定的损失在排序中的次序小于或等于M，M表示小于视线预测信息总数的整数。
在实际应用中，可以通过将视线预测信息和对应的真实视线标签带入损失函数中进行求解，得到每个眼部样本图像对应的损失，并基于不同眼部样本图像的损失由大到小的顺序进行排序，如果该视视线预测信息和对应的真实视线标签确定的损失在排序中的次序小于或等于M，则说明该真实视线标签有可能与其对应的眼部样本图像不匹配，因此，可以认为该视线预测信息和对应的真实视线标签确定的损失满足标签筛选条件，将该视线预测信息对应的眼部样本图像的目标视线标签设定为对应的视线参考信息，从而避免了脏数据对构建待构建学生模型的准确性产生影响。
上述M可以被认为是眼部样本图像构建集的样本错误数量，其可以由眼部样本图像的清洗率和眼部样本图像的总数量确定。此处眼部样本图像的清洗率可以反映眼部样本图像的错误率，其可以通过统计方式确定，也可以根据实际情况自定义。眼部样本图像的总数量，可以是该更新一次待构建学生模型的模型参数所需要的眼部样本图像数量，或者说待构建学生模型的一批次的眼部样本图像数量。
示例性的，上述M与所述眼部样本图像的清洗率正相关，M与所述眼部样本图像的总数量正相关。
以：M＝rate*batch_size为例，当眼部样本图像的清洗率rate＝0.01，眼部样本图像的总数量batch_size＝1000时，M＝10。说明在这一批眼部样本图像中需要清洗的脏数据的数量为10，此时，可以将该眼部样本图像对应的视线预测信息和其对应的真实视线标签带入损失函数中进行求解，得到每个眼部样本图像对应的损失，并基于不同眼部样本图像的损失由大到小的顺序，对每个损失对应的视线预测信息进行排序，基于排序完成的视线预测信息，将前10个视线预测信息对应的目标视线标签更新为其对应的视线参考信息，剩余的990个视线预测信息对应的目标视线标签为其对应的真实视线标签，以重新确定待构建学生模型的收敛性，减少脏数据对待构建学生模型的精度的影响。
由上可见，本公开示例性实施例在构建待构建学生模型时，不管采用何种标签筛选条件，其多个眼部样本图像所对应的所有目标视线标签可以包括对应不同眼部样本图像的真实视线标签和对应不同眼部样本图像的视线参考信息，该视线参考信息可以用于清洗多个眼部样本图像所对应的所有真实视线标签中的脏数据，减少脏数据对构建待构建学生模型的精度产生的影响。
作为一种可能的实现方式，当视线预测信息和对应的真实视线标签确定的损失不满足标签筛选条件时，确定视线预测信息对应的目标视线标签还包括对应的视线参考信息，视线预测信息和对应目标视线标签确定的损失包括第一损失和第二损失，基于第一损失和第二损失可以确定对应的眼部样本图像的损失。
在实际应用中，当视线预测信息和对应的真实视线标签确定的损失不满足标签筛选条件时，说明当前眼部样本图像中不存在脏数据，可以直接使用该眼部样本图像对应的真实视线标签求损失。但为了提高待构建学生模型的预测精准度，可以使用精准度较高的老师模型输出的视线参考信息和真实视线标签同时对待构建学生模型进行监督。此时，可以将同一眼部样本图像对应的视线预测信息的目标视线标签分别选用对应的视线参考信息和对应的真实视线标签，由视线预测信息和对应的真实视线标签确定第一损失，由视线预测信息和对应的所述视线参考信息确定第二损失，然后再基于第一损失和第二损失确定对应的眼部样本图像的损失。应理解，此处基于第一损失和第二损失确定对应的眼部样本图像的损失可以是将第一损失和第二损失进行加权求和得到的，也可以是使用其他方法，在此不做限定，在一些示例中，第一损失的权重大于第二损失的权重。
在另一些示例中，本公开示例性实施例的视线预测模型所涉及的损失函数还可以表示为：
式中，Lsoft是由在温度＝T下的构建完成的老师模型输出的视线参考信息和温度＝T下的待构建学生模型输出的视线预测信息求得的损失。包括眼部样本图像与真实视线标签匹配的情况下，通过视线预测信息和真实视线标签确定的损失。包括眼部样本图像与真实视线标签不匹配的情况下，通过视线预测信息和视线参考信息确定的损失。
其中， 表示构建完成的老师模型在温度＝T下的softmax输出在第r类上的视线参考信息；表示待构建学生模型在温度＝T下的softmax输出在第r类上的视线预测信息，N表示输入待构建的学生模型的眼部样本图像的总数量，r表示大于或等于1且小于或等于N的整数。
cj表示不满足标签筛选条件的第j类上的真实视线标签； 表示待构建学生模型在温度＝1下的softmax输出在第j类上的视线预测信息。
表示构建完成的老师模型在温度＝1下的softmax输出在第k类上的视线参考信息；表示待构建学生模型在温度＝1下的softmax输出在第k类上的视线预测信息。
本公开示例性实施例还提供了一种视线预测方法，用于预测眼部图像中的视线方向。图2示例出了本公开示例性实施例提供的视线预测方法的流程图。如图2所示，该视线预测方法包括：
步骤201：获取眼部图像。该眼部样本图像为含有视线方向的眼部样本图像。应理解，这里的眼部样本图像可以为图片，也可以为视频图像。
在实际应用中，该眼部图像可以由图像采集装置采集，图像采集装置可以为相机或者摄像机，其可以为单独设备存在，也可以为具有显示屏的计算设备所具有的摄像头，还可以为安装在自行车、摩托书、汽车等车辆上的相机，该相机可以为单目相机或双目相机等。
步骤202：向视线预测模型中输入眼部图像，获得视线信息。其中，视线预测模型可以由视线预测模型的构建方法构建。这里需要注意的是，视线信息可以为上述图像采集装置采集到的用户的“注视方向”。例如，当该视线信息为用户的“注视方向”，视线预测方法用于线下零售领域时，可以通过获取用户当前的“注视方向”获知用户对货架上产品的兴趣，从而实现产品的选择以及摆放位置的及时调整。
上述主要对本公开实施例提供的方案进行了介绍。可以理解的是，为了实现上述功能，本公开的装置包含了执行各个功能相应的硬件结构和/或软件模块。本领域技术人员应该很容易意识到，结合本文中所公开的实施例描述的各示例的单元及算法步骤，本公开能够以硬件或硬件和计算机软件的结合形式来实现。某个功能究竟以硬件还是计算机软件驱动硬件的方式来执行，取决于技术方案的特定应用和设计约束条件。专业技术人员可以对每个特定的应用来使用不同方法来实现所描述的功能，但是这种实现不应认为超出本公开的范围。
本公开实施例可以根据上述方法示例对本公开的装置进行功能单元的划分，例如，可以对应各个功能划分各个功能模块，也可以将两个或两个以上的功能集成在一个处理模块中。上述集成的模块既可以采用硬件的形式实现，也可以采用软件功能模块的形式实现。需要说明的是，本公开实施例中对模块的划分是示意性的，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式。
在采用对应各个功能划分各个功能模块的情况下，本公开示例性实施例提供一种构建装置，该构建装置可以为电子设备或应用于电子设备的芯片。图3示出了根据本公开示例性实施例的构建装置的功能模块示意性框图。如图3所示，该构建装置300包括：
预测模块301，用于向构建完成的老师模型输入多个眼部样本图像，获得多个视线参考信息。向待构建学生模型输入多个眼部样本图像，获得多个视线预测信息。每个眼部样本图像、每个视线参考信息与每个视线预测信息对应。
确定模块302，用于当视线预测信息和对应的真实视线标签确定的损失满足标签筛选条件时，确定眼部样本图像的目标视线标签为对应的视线参考信息，否则，确定眼部样本图像的目标视线标签为对应的真实视线标签。若基于多个视线预测信息和对应的目标视线标签确定待构建学生模型的损失满足收敛条件，确定待构建学生模型为视线预测模型。
更新模块303，用于若基于多个视线预测信息和对应的目标视线标签确定待构建学生模型的损失不满足收敛条件，基于多个视线预测信息和对应的目标视线标签更新待构建学生模型的模型参数。
在一种可能的实现方式中，上述标签筛选条件包括：视线预测信息和对应的真实视线标签确定的损失大于损失阈值。
在一种可能的实现方式中，上述标签筛选条件包括：按照基于视线预测信息和对应的目标视线标签确定的损失从大到小的顺序对多个视线预测信息排序，视线预测信息为第k个视线预测信息，k大于0且小于或等于M，M表示小于视线预测信息总数的整数。
在一种可能的实现方式中，上述M与眼部样本图像的清洗率正相关，M与眼部样本图像的总数量正相关。
在一种可能的实现方式中，当视线预测信息和对应的真实视线标签确定的损失不满足标签筛选条件时，确定视线预测信息对应的目标视线标签还包括对应的视线参考信息，视线预测信息和对应目标视线标签确定的损失包括第一损失和第二损失。
在一种可能的实现方式中，第一损失由视线预测信息和对应的真实视线标签确定，第二损失由视线预测信息和对应的视线参考信息确定。
在采用对应各个功能划分各个功能模块的情况下，本公开示例性实施例还提供一种视线预测装置，该视线预测装置可以为电子设备或应用于电子设备的芯片。图4示出了根据本公开示例性实施例的视线预测装置的功能模块示意性框图。如图4所示，该视线预测装置400包括：
获取模块401，用于获取眼部图像。
预测模块402，用于向视线预测模型中输入眼部图像，获得视线信息，视线预测模型由上述视线预测模型的构建方法构建。
图5示出了根据本公开示例性实施例的芯片的示意性框图。如图5所示，该芯片500包括一个或两个以上(包括两个)处理器501和通信接口502。通信接口502可以支持电子设备执行上述方法中的数据收发步骤，处理器501可以支持电子设备执行上述方法中的数据处理步骤。
可选的，如图5所示，该芯片500还包括存储器503，存储器503可以包括只读存储器和随机存取存储器，并向处理器提供操作指令和数据。存储器的一部分还可以包括非易失性随机存取存储器(non-volatile random access memory，NVRAM)。
在一些实施方式中，如图5所示，处理器501通过调用存储器存储的操作指令(该操作指令可存储在操作系统中)，执行相应的操作。处理器501控制终端设备中任一个的处理操作，处理器还可以称为中央处理单元(central processing unit，CPU)。存储器503可以包括只读存储器和随机存取存储器，并向处理器501提供指令和数据。存储器503的一部分还可以包括NVRAM。例如应用中存储器、通信接口以及存储器通过总线系统耦合在一起，其中总线系统除包括数据总线之外，还可以包括电源总线、控制总线和状态信号总线等。但是为了清楚说明起见，在图5中将各种总线都标为总线系统504。
上述本公开实施例揭示的方法可以应用于处理器中，或者由处理器实现。处理器可能是一种集成电路芯片，具有信号的处理能力。在实现过程中，上述方法的各步骤可以通过处理器中的硬件的集成逻辑电路或者软件形式的指令完成。上述的处理器可以是通用处理器、数字信号处理器(digital signal processing，DSP)、专用集成电路(application specific integrated circuit，ASIC)、现成可编程门阵列(field-programmable gate array，FPGA)或者其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件。可以实现或者执行本公开实施例中的公开的各方法、步骤及逻辑框图。通用处理器可以是微处理器或者该处理器也可以是任何常规的处理器等。结合本公开实施例所公开的方法的步骤可以直接体现为硬件译码处理器执行完成，或者用译码处理器中的硬件及软件模块组合执行完成。软件模块可以位于随机存储器，闪存、只读存储器，可编程只读存储器或者电可擦写可编程存储器、寄存器等本领域成熟的存储介质中。该存储介质位于存储器，处理器读取存储器中的信息，结合其硬件完成上述方法的步骤。
本公开示例性实施例还提供一种电子设备，包括：处理器；以及与处理器通信连接的存储程序的存储器。程序包括指令，指令在由处理器执行时使处理器执行根据本公开示例性实施例的方法。
本公开示例性实施例还提供一种存储有计算机程序的非瞬时计算机可读存储介质，其中，计算机指令用于使所述计算机执行根据本公开实施例的方法。
参考图6，现将描述可以作为本公开示例性实施例的服务器或客户端的电子设备600 的结构框图，其是可以应用于本公开的各方面的硬件设备的示例。电子设备旨在表示各种形式的数字电子的计算机设备，诸如，膝上型计算机、台式计算机、工作台、个人数字助理、服务器、刀片式服务器、大型计算机、和其它适合的计算机。电子设备还可以表示各种形式的移动装置，诸如，个人数字处理、蜂窝电话、智能电话、可穿戴设备和其它类似的计算装置。本文所示的部件、它们的连接和关系、以及它们的功能仅仅作为示例，并且不意在限制本文中描述的和/或者要求的本公开的实现。
如图6所示，电子设备600包括计算单元601，其可以根据存储在只读存储器(read only memory，ROM)602中的计算机程序或者从存储单元606加载到随机访问存储器(random access memory，RAM)603中的计算机程序，来执行各种适当的动作和处理。在RAM 603中，还可存储设备600操作所需的各种程序和数据。计算单元601、ROM 602以及RAM 603通过总线604彼此相连。输入/输出(I/O)接口605也连接至总线604。
电子设备600中的多个部件连接至I/O接口605，包括：输入单元606、输出单元607、存储单元608以及通信单元609。输入单元606可以是能向电子设备600输入信息的任何类型的设备，输入单元606可以接收输入的数字或字符信息，以及产生与电子设备的用户设置和/或功能控制有关的键信号输入。输出单元607可以是能呈现信息的任何类型的设备，并且可以包括但不限于显示器、扬声器、视频/音频输出终端、振动器和/或打印机。存储单元604可以包括但不限于磁盘、光盘。通信单元609允许电子设备600通过诸如因特网的计算机网络和/或各种电信网络与其他设备交换信息/数据，并且可以包括但不限于调制解调器、网卡、红外通信设备、无线通信收发机和/或芯片组，例如蓝牙TM设备、WiFi设备、WiMax设备、蜂窝通信设备和/或类似物。
计算单元601可以是各种具有处理和计算能力的通用和/或专用处理组件。计算单元601的一些示例包括但不限于中央处理单元(CPU)、图形处理单元(GPU)、各种专用的人工智能(AI)计算芯片、各种运行机器学习模型算法的计算单元、数字信号处理器(DSP)、以及任何适当的处理器、控制器、微控制器等。计算单元601执行上文所描述的各个方法和处理。例如，在一些实施例中，前述方法可被实现为计算机软件程序，其被有形地包含于机器可读介质，例如存储单元608。在一些实施例中，计算机程序的部分或者全部可以经由ROM 602和/或通信单元609而被载入和/或安装到电子设备600上。在一些实施例中，计算单元601可以通过其他任何适当的方式(例如，借助于固件)而被配置为执行前述方法。
用于实施本公开的方法的程序代码可以采用一个或多个编程语言的任何组合来编写。这些程序代码可以提供给通用计算机、专用计算机或其他可编程数据处理装置的处理器或控制器，使得程序代码当由处理器或控制器执行时使流程图和/或框图中所规定的功能/操 作被实施。程序代码可以完全在机器上执行、部分地在机器上执行，作为独立软件包部分地在机器上执行且部分地在远程机器上执行或完全在远程机器或服务器上执行。
在本公开的上下文中，机器可读介质可以是有形的介质，其可以包含或存储以供指令执行系统、装置或设备使用或与指令执行系统、装置或设备结合地使用的程序。机器可读介质可以是机器可读信号介质或机器可读储存介质。机器可读介质可以包括但不限于电子的、磁性的、光学的、电磁的、红外的、或半导体系统、装置或设备，或者上述内容的任何合适组合。机器可读存储介质的更具体示例会包括基于一个或多个线的电气连接、便携式计算机盘、硬盘、随机存取存储器(RAM)、只读存储器(ROM)、可擦除可编程只读存储器(EPROM或快闪存储器)、光纤、便捷式紧凑盘只读存储器(CD-ROM)、光学储存设备、磁储存设备、或上述内容的任何合适组合。
如本公开使用的，术语“机器可读介质”和“计算机可读介质”指的是用于将机器指令和/或数据提供给可编程处理器的任何计算机程序产品、设备、和/或装置(例如，磁盘、光盘、存储器、可编程逻辑装置(PLD))，包括，接收作为机器可读信号的机器指令的机器可读介质。术语“机器可读信号”指的是用于将机器指令和/或数据提供给可编程处理器的任何信号。
为了提供与用户的交互，可以在计算机上实施此处描述的系统和技术，该计算机具有：用于向用户显示信息的显示装置(例如，阴极射线管(cathode ray tube，CRT)或者液晶显示器(liquid crystal display，LCD)监视器)；以及键盘和指向装置(例如，鼠标或者轨迹球)，用户可以通过该键盘和该指向装置来将输入提供给计算机。其它种类的装置还可以用于提供与用户的交互；例如，提供给用户的反馈可以是任何形式的传感反馈(例如，视觉反馈、听觉反馈、或者触觉反馈)；并且可以用任何形式(包括声输入、语音输入或者、触觉输入)来接收来自用户的输入。
可以将此处描述的系统和技术实施在包括后台部件的计算系统(例如，作为数据服务器)、或者包括中间件部件的计算系统(例如，应用服务器)、或者包括前端部件的计算系统(例如，具有图形用户界面或者网络浏览器的用户计算机，用户可以通过该图形用户界面或者该网络浏览器来与此处描述的系统和技术的实施方式交互)、或者包括这种后台部件、中间件部件、或者前端部件的任何组合的计算系统中。可以通过任何形式或者介质的数字数据通信(例如，通信网络)来将系统的部件相互连接。通信网络的示例包括：局域网(LAN)、广域网(WAN)和互联网。
计算机系统可以包括客户端和服务器。客户端和服务器一般远离彼此并且通常通过通信网络进行交互。通过在相应的计算机上运行并且彼此具有客户端-服务器关系的计算机程序来产生客户端和服务器的关系。
在上述实施例中，可以全部或部分地通过软件、硬件、固件或者其任意组合来实现。当使用软件实现时，可以全部或部分地以计算机程序产品的形式实现。所述计算机程序产品包括一个或多个计算机程序或指令。在计算机上加载和执行所述计算机程序或指令时，全部或部分地执行本公开实施例所述的流程或功能。所述计算机可以是通用计算机、专用计算机、计算机网络、终端、用户设备或者其它可编程装置。所述计算机程序或指令可以存储在计算机可读存储介质中，或者从一个计算机可读存储介质向另一个计算机可读存储介质传输，例如，所述计算机程序或指令可以从一个网站站点、计算机、服务器或数据中心通过有线或无线方式向另一个网站站点、计算机、服务器或数据中心进行传输。所述计算机可读存储介质可以是计算机能够存取的任何可用介质或者是集成一个或多个可用介质的服务器、数据中心等数据存储设备。所述可用介质可以是磁性介质，例如，软盘、硬盘、磁带；也可以是光介质，例如，数字视频光盘(digital video disc，DVD)；还可以是半导体介质，例如，固态硬盘(solid state drive，SSD)。
尽管结合具体特征及其实施例对本公开进行了描述，显而易见的，在不脱离本公开的精神和范围的情况下，可对其进行各种修改和组合。相应地，本说明书和附图仅仅是所附权利要求所界定的本公开的示例性说明，且视为已覆盖本公开范围内的任意和所有修改、变化、组合或等同物。显然，本领域的技术人员可以对本公开进行各种改动和变型而不脱离本公开的精神和范围。这样，倘若本公开的这些修改和变型属于本公开权利要求及其等同技术的范围之内，则本公开也意图包括这些改动和变型在内。

Claims (13)

  1. 一种视线预测模型的构建方法，所述构建方法包括：
    向构建完成的老师模型输入多个眼部样本图像，获得多个视线参考信息；
    向待构建学生模型输入多个所述眼部样本图像，获得多个视线预测信息，所述眼部样本图像、所述视线参考信息与所述视线预测信息对应；
    当所述视线预测信息和对应的真实视线标签确定的损失满足标签筛选条件时，确定所述眼部样本图像的目标视线标签包括对应的所述视线参考信息，否则，确定所述眼部样本图像的目标视线标签包括对应的所述真实视线标签；
    若基于多个所述视线预测信息和对应的目标视线标签确定所述待构建学生模型的损失不满足收敛条件，基于多个所述视线预测信息和对应的目标视线标签更新所述待构建学生模型的模型参数，否则，确定所述待构建学生模型为所述视线预测模型。
  2. 根据权利要求1所述的视线预测模型的构建方法，其中，所述标签筛选条件包括：基于所述视线预测信息和对应的真实视线标签确定的损失大于损失阈值。
  3. 根据权利要求1所述的视线预测模型的构建方法，其中，所述构建方法还包括：按照基于所述视线预测信息和对应的真实视线标签确定的损失从大到小的顺序排序，所述标签筛选条件包括：基于所述视线预测信息和对应的真实视线标签确定的损失在所述排序中的次序小于或等于M，M表示小于视线预测信息总数的整数。
  4. 根据权利要求3所述的视线预测模型的构建方法，其中，所述M与所述眼部样本图像的清洗率正相关，所述M与所述眼部样本图像的总数量正相关。
  5. 根据权利要求1～4任一项所述的视线预测模型的构建方法，其中，所述构建方法还包括：
    当所述视线预测信息和对应的真实视线标签确定的损失不满足标签筛选条件时，确定所述视线预测信息对应的目标视线标签还包括对应的所述视线参考信息，所述视线预测信息和对应所述目标视线标签确定的损失包括第一损失和第二损失，所述第一损失由所述视线预测信息和对应的所述真实视线标签确定，所述第二损失由所述视线预测信息和对应的所述视线参考信息确定。
  6. 根据权利要求5所述的视线预测模型的构建方法，其中，所述第一损失的权重大于所述第二损失的权重。
  7. 一种视线预测方法，包括：
    获取眼部图像；
    向视线预测模型中输入眼部图像，获得视线信息，所述视线预测模型由权利要求1～6任一项所述的视线预测模型的构建方法构建。
  8. 一种构建装置，所述构建装置包括：
    预测模块，被配置为向构建完成的老师模型输入多个眼部样本图像，获得多个视线参考信息，向待构建学生模型输入多个所述眼部样本图像，获得多个视线预测信息，每个所述眼部样本图像、每个所述视线参考信息与每个所述视线预测信息对应，
    确定模块，被配置为当所述视线预测信息和对应的真实视线标签确定的损失满足标签筛选条件时，确定所述眼部样本图像的目标视线标签为对应的所述视线参考信息，否则，确定所述眼部样本图像的目标视线标签为对应的所述真实视线标签，若基于多个所述视线预测信息和对应的目标视线标签确定所述待构建学生模型的损失满足收敛条件，确定所述待构建学生模型为视线预测模型；
    更新模块，被配置为若基于多个所述视线预测信息和对应的目标视线标签确定所述待构建学生模型的损失不满足收敛条件，基于多个所述视线预测信息和对应的目标视线标签更新所述待构建学生模型的模型参数。
  9. 一种视线预测装置，所述视线预测装置包括：
    获取模块，被配置为获取眼部图像；
    预测模块，被配置为向视线预测模型中输入眼部图像，获得视线信息，所述视线预测模型由权利要求1～6任一项所述方法构建。
  10. 一种电子设备，包括：
    处理器；以及
    与所述处理器通信连接的存储程序的存储器，
    其中，所述程序包括指令，所述指令在由所述处理器执行时使所述处理器执行根据权利要求1～7中任一项所述的方法。
  11. 一种存储有计算机指令的非瞬时计算机可读存储介质，其中，所述计算机指令用于使所述计算机执行根据权利要求1～7中任一项所述的方法。
  12. 一种计算机程序产品，包括一个或多个计算机程序，所述计算机程序在由计算机执行时使所述计算机执行根据权利要求1～7中任一项所述的方法。
  13. 一种计算机程序，所述计算机程序在由计算机执行时使所述计算机执行根据权利要求1～7中任一项所述的方法。
PCT/CN2024/095931 2023-05-29 2024-05-29 一种视线预测模型的构建方法、装置、设备及存储介质 Ceased WO2024245263A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202310620447.8A CN119049114A (zh) 2023-05-29 2023-05-29 一种视线预测模型的构建方法、装置、设备及存储介质
CN202310620447.8 2023-05-29

Publications (1)

Publication Number Publication Date
WO2024245263A1 true WO2024245263A1 (zh) 2024-12-05

Family

ID=93569638

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2024/095931 Ceased WO2024245263A1 (zh) 2023-05-29 2024-05-29 一种视线预测模型的构建方法、装置、设备及存储介质

Country Status (2)

Country Link
CN (1) CN119049114A (zh)
WO (1) WO2024245263A1 (zh)

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111192262A (zh) * 2020-01-03 2020-05-22 腾讯云计算（北京）有限责任公司 基于人工智能的产品缺陷分类方法、装置、设备及介质
CN112307815A (zh) * 2019-07-26 2021-02-02 北京三星通信技术研究有限公司 图像处理方法、装置、电子设备及可读存储介质
US20220344049A1 (en) * 2019-09-23 2022-10-27 Presagen Pty Ltd Decentralized artificial intelligence (ai)/machine learning training system
CN115797373A (zh) * 2023-01-09 2023-03-14 苏州浪潮智能科技有限公司 一种图像分割方法、装置、电子设备及介质
CN116091541A (zh) * 2022-12-21 2023-05-09 哲库科技(上海)有限公司 眼动追踪方法、装置、电子设备、存储介质和程序产品

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112307815A (zh) * 2019-07-26 2021-02-02 北京三星通信技术研究有限公司 图像处理方法、装置、电子设备及可读存储介质
US20220344049A1 (en) * 2019-09-23 2022-10-27 Presagen Pty Ltd Decentralized artificial intelligence (ai)/machine learning training system
CN111192262A (zh) * 2020-01-03 2020-05-22 腾讯云计算（北京）有限责任公司 基于人工智能的产品缺陷分类方法、装置、设备及介质
CN116091541A (zh) * 2022-12-21 2023-05-09 哲库科技(上海)有限公司 眼动追踪方法、装置、电子设备、存储介质和程序产品
CN115797373A (zh) * 2023-01-09 2023-03-14 苏州浪潮智能科技有限公司 一种图像分割方法、装置、电子设备及介质

Also Published As

Publication number Publication date
CN119049114A (zh) 2024-11-29

Similar Documents

Publication Publication Date Title
JP2022058915A (ja) 画像認識モデルをトレーニングするための方法および装置、画像を認識するための方法および装置、電子機器、記憶媒体、並びにコンピュータプログラム
EP4369159A1 (en) Deep machine learning to perform touch motion prediction
CN111681726A (zh) 电子病历数据的处理方法、装置、设备和介质
WO2020125445A1 (zh) 分类模型训练方法、分类方法、设备及介质
US20180122517A1 (en) Methods and apparatus related to electronic display of a human avatar with display properties particularized to health risks of a patient
CN111814056A (zh) 基于信息处理的供应商推荐方法及相关设备
EP4538894A1 (en) Operation prediction method and related apparatus
EP3152640A1 (en) Systems and methods for serving product recommendations
WO2020007177A1 (zh) 计算机执行的报价方法、报价装置、电子设备及存储介质
CN107368820A (zh) 一种精细化手势识别方法、装置及设备
CN116304314A (zh) 资源召回方法、装置、电子设备及存储介质
WO2025122922A1 (en) Synthetic training data for generative models
CN114791982A (zh) 对象推荐方法和装置
EP4700608A1 (en) Data processing method and related apparatus
CN114550236A (zh) 图像识别及其模型的训练方法、装置、设备和存储介质
WO2024245263A1 (zh) 一种视线预测模型的构建方法、装置、设备及存储介质
WO2024251044A1 (zh) 视线预测模型的构建方法、视线预测方法、装置及电子设备和存储介质
CN114693977A (zh) 图像处理方法、模型训练方法、装置、设备及介质
US20240346081A1 (en) Systems and methods for query engine analysis
CN118485819A (zh) 一种关键点检测方法、装置、设备及存储介质
CN113920998A (zh) 语音识别方法、装置、电子设备以及存储介质
US12443983B2 (en) Systems and methods for recommendation system analysis
Cai One framework to rule them all: Unifying rl-based and rl-free methods in rlhf
CN115331811A (zh) 一种疾病类型的预测方法、装置、电子设备及存储介质
CN110706772A (zh) 一种排序方法及装置、电子设备、存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 24814460

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE