US20230210364A1 - Eye tracking system, eye tracking method, and eye tracking program - Google Patents

Eye tracking system, eye tracking method, and eye tracking program Download PDF

Info

Publication number
US20230210364A1
US20230210364A1 US17/997,976 US202117997976A US2023210364A1 US 20230210364 A1 US20230210364 A1 US 20230210364A1 US 202117997976 A US202117997976 A US 202117997976A US 2023210364 A1 US2023210364 A1 US 2023210364A1
Authority
US
United States
Prior art keywords
area
user
eye tracking
content
guidance
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Abandoned
Application number
US17/997,976
Other languages
English (en)
Inventor
Nobuo Kawakami
Yuri ODAGIRI
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Dwango Co Ltd
Original Assignee
Dwango Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Dwango Co Ltd filed Critical Dwango Co Ltd
Assigned to DWANGO CO., LTD. reassignment DWANGO CO., LTD. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: KAWAKAMI, NOBUO, ODAGIRI, Yuri
Publication of US20230210364A1 publication Critical patent/US20230210364A1/en
Abandoned legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G06F3/013—Eye tracking input arrangements
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B3/00—Apparatus for testing the eyes; Instruments for examining the eyes
    • A61B3/10—Objective types, i.e. instruments for examining the eyes independent of the patients' perceptions or reactions
    • A61B3/113—Objective types, i.e. instruments for examining the eyes independent of the patients' perceptions or reactions for determining or recording eye movement
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer

Definitions

  • An aspect of the present disclosure relates to an eye tracking system, an eye tracking method, and an eye tracking program.
  • Patent Document 1 describes a calibration method for a head-mounted eye tracking device.
  • eye data related to the position of the wearer's eye is acquired by the eye tracking device while the wearer of the eye tracking device is looking in a reference direction, and the eye data is associated with a viewing direction that corresponds to the reference direction.
  • the eye tracking device includes a spectacle frame with an ophthalmic lens, and a viewing direction that corresponds to a reference direction is determined taking into account an optical refractive function of the ophthalmic lens.
  • Other examples of calibration methods are described in Patent Document 2 and Patent Document 3.
  • DOCUMENT PATENT 1 Japanese Patent Publication No. 6656156
  • PATENT DOCUMENT 2 Japanese Unexamined Patent Publication No. 2010-259605
  • PATENT DOCUMENT 3 Japanese Unexamined Patent Publication No. 2001-204692
  • An eye tracking system includes at least one processor.
  • the at least one processor is configured to: dynamically set a partial area of first content displayed on a screen as a guidance area to have a user gaze at the partial area; identify first viewpoint coordinates on the screen based on an eye movement of the user gazing at the guidance area; calculate a difference between the first viewpoint coordinates identified and area coordinates of the guidance area on the screen; and calibrate second viewpoint coordinates of the user viewing second content displayed on the screen by using the difference.
  • a guidance area for the user to gaze at is dynamically set for any content (first content), and the guidance area is used to calculate the difference to be used in calibration. Accordingly, there is no need to prepare content for calibration in advance. This makes calibration for eye tracking easier to perform.
  • eye tracking calibration can be easily executed.
  • FIG. 1 is a diagram illustrating an exemplary application of an assist system in accordance with an embodiment.
  • FIG. 2 is a diagram illustrating an exemplary hardware configuration related to the assist system according to the embodiment.
  • FIG. 3 is a diagram illustrating an exemplary function configuration related to the assist system according to the embodiment.
  • FIG. 4 is a flowchart illustrating an exemplary operation of the assist system in accordance with the embodiment.
  • FIG. 5 is a flowchart illustrating an exemplary operation of an eye tracking system according to an embodiment.
  • FIG. 6 is a diagram illustrating an exemplary guidance area set in a first content.
  • FIG. 7 is a diagram illustrating another exemplary guidance area set in a first content.
  • FIG. 8 is a flowchart illustrating an exemplary operation of the assist system in accordance with the embodiment.
  • FIG. 9 is a flowchart illustrating an exemplary operation of the assist system in accordance with the embodiment.
  • FIG. 10 is a diagram illustrating exemplary assist information.
  • the assist system related to an embodiment is a computer system that assists a user who visually recognizes content.
  • the content herein refers to information in a human-recognizable form, which is provided by a computer or computer system.
  • Electronic data representing the content is referred to as content data.
  • No particular limitation is imposed on the form of expressing the content and the content may be expressed, for example, in the form of documents, images (e.g., photographs and videos), or a combination thereof.
  • No particular limitation is imposed on the purpose and the usage scenes for the content, and the content may be utilized for a variety of purposes such as, for example, education, news, lecture, commercial transaction, entertainment, medical treatment, game, and chat.
  • the assist system provides the content to a user by transmitting the content data to a user terminal.
  • the user is a person who seeks to obtain information from the assist system, that is, a viewer of the content.
  • the user terminal may also be referred to as a “viewer terminal”.
  • the assist system may provide the content data to the user terminal in response to a request from the user, or may provide the content data to the user terminal based on an instruction from a distributor apart from the user.
  • the distributor is a person who intends to convey information to a user (viewer), that is, a sender of content.
  • the assist system provides the user not only with the content but also with assist information corresponding to a user understanding level, as needed.
  • the user understanding level is an indicator of the user understanding level for the content.
  • the user understanding level may be an indicator of how much the user understands the sentence (e.g., whether or not the user understands the meaning of the words in the sentence or whether or not the user understands the grammar of the sentence).
  • the assist information is information for promoting the user understanding for the content.
  • the assist information may be information indicating a meaning of each word in the sentence, a grammar of the sentence, or the like.
  • the assist system estimates the target user understanding level based on the target user viewpoint movement on the screen displaying the target content.
  • the assist system refers to correlation data indicating a correlation between the user viewpoint movement and the user understanding level.
  • the correlation data is electronic data generated by performing statistical processing on sample data acquired in advance.
  • Sample data is electronic data indicating a pair of the movement of the user viewpoint while he or she visually recognizes the content and the user understanding level for the content.
  • a user who provides sample data for generating correlation data is referred to as a sample user, and content visually recognized by the sample user is referred to as sample content.
  • the assist system acquires data indicating the target user viewpoint movement from the user terminal of the target user.
  • the data indicating the viewpoint movement is data indicating how the user viewpoint has moved on the screen of the user terminal, and is also referred to as viewpoint data in the present disclosure.
  • data indicating the target user viewpoint movement (that is, viewpoint data of the target user) is referred to as target data.
  • the assist system estimates the target user understanding level by using the correlation data and the target data.
  • the assist system then outputs assist information corresponding to the target user understanding level to the user terminal of the target user as needed.
  • the present disclosure may collectively refer the sample user and the target user as the user, where the sample user and the target user do not need to be distinguished from each other.
  • the viewpoint data is acquired by an eye tracking system.
  • the eye tracking system identifies user viewpoint coordinates at each given time interval based on the movements of the user's eyes, and obtains viewpoint data indicating coordinates of a plurality of viewpoints in a time sequence.
  • the viewpoint coordinates are coordinates indicating the position of the viewpoint on the user terminal screen.
  • the viewpoint coordinates may be represented using a two-dimensional coordinate system.
  • the eye tracking system may be mounted on the user terminal or may be mounted on another computer separate from the user terminal. Alternatively, the tracking system may be implemented by the user terminal in cooperation with another computer.
  • the eye tracking system performs calibration that is a process for more accurately identifying the viewpoint coordinates of the user.
  • the eye tracking system first sets a partial area of the content displayed on the screen of the user terminal as a guidance area for the user to gaze at.
  • the content in which the guidance area is set is referred to as first content.
  • the eye tracking system identifies the coordinates of the viewpoint of the user gazing at the guidance area as the first viewpoint coordinates based on the user's eye movements, and calculates the difference between the first viewpoint coordinates and the area coordinates of the guidance area.
  • the area coordinates of the guidance area are coordinates indicating the position of the guidance area on the screen of the user terminal.
  • the eye tracking system then identifies the coordinates of the viewpoint of the user looking at the second content as the second viewpoint coordinates based on the user's eye movements while the user visually recognizes the content displayed on the screen of the user terminal (second content).
  • the eye tracking system then calibrates the second viewpoint coordinates identified by using the pre-calculated difference.
  • the second content is content that the user is looking at while the second viewpoint coordinates are calibrated.
  • educational content is shown as an exemplary content, and the assist system assists a student who visually recognizes the educational content. Therefore, the target content is “target content for education”, and the sample content is “sample content for education”.
  • the educational content is content used to educate students, and may be, for example, tests such as exercise questions and examination questions, or textbooks.
  • the educational content may include a sentence, a mathematical expression, a graph, a figure, or the like.
  • the term “student” refers to a person who receives teaching such as academic work and handicraft. The student is an example of a user (viewer). As described above, the distribution of the content to the viewer may be performed based on an instruction of the distributor.
  • the distributor may be a teacher.
  • the teacher refers to a person who teaches schoolwork, techniques, and the like to students.
  • the teacher may be a person with a teacher's license or a person without a teacher's license.
  • the age and affiliation are not limited for each of the teacher and student. Therefore, the purpose and the usage scenes of the educational content are not limited.
  • the educational content may be used in various schools such as nursery schools, kindergarten schools, elementary schools, junior high schools, high schools, universities, graduates, specialty schools, preparatory schools, and online schools, or may be used in places or scenes other than schools.
  • educational content may be used for a variety of purposes, such as infant education, compulsory education, higher education, lifelong learning, and the like.
  • the educational content includes not only content used in school education but also content used in a seminar or a training scene of a company or the like.
  • FIG. 1 is a diagram illustrating an exemplary application of an assist system 1 in accordance with an embodiment.
  • the assist system 1 includes a server 10 .
  • the server 10 is connected to and in communication with a user terminal 20 and a database 30 via a communication network N.
  • the configuration of the communication network N is not limited.
  • the communication network N may include the Internet or an intranet.
  • the server 10 is a computer that distributes content to the user terminal 20 and provides assist information to the user terminal 20 as needed.
  • the server 10 may be configured by one or more computers.
  • the user terminal 20 is a computer used by a user.
  • the user is a student who views educational content.
  • the user terminal 20 has a function of accessing the assist system 1 to receive and display content data and assist information, and a function of transmitting viewpoint data to the assist system 1 .
  • the type of the user terminal 20 is not limited, and may be, for example, a mobile terminal such as a high-function mobile phone (smartphone), a tablet terminal, a wearable terminal (e.g., a head-mounted display (HMD), smart glasses, or the like), a laptop personal computer, or a mobile phone.
  • the user terminal 20 may be a stationary terminal such as a desktop personal computer. Although three user terminals 20 are shown in FIG.
  • the number of user terminals 20 is not limited.
  • the terminal of the sample user and the terminal of the target user are distinguished from each other, the terminal of the sample user is referred to as a “user terminal 20 A”, and the terminal of the target user is referred to as a “user terminal 20 B”.
  • the user can operate the user terminal 20 to log in to the assist system 1 and view content. In the present embodiment, it is assumed that the user of the assist system 1 has already logged in.
  • the database 30 is a non-transitory storage device that stores data used by the assist system 1 .
  • the database 30 stores content data, sample data, correlation data, and assist information.
  • the database 30 may be a single database or a collection of multiple databases.
  • FIG. 2 is a diagram illustrating an exemplary hardware configuration related to the assist system 1 .
  • FIG. 2 shows a server computer 100 serving as the server 10 , and a terminal computer 200 serving as the user terminal 20 .
  • the server computer 100 includes a processor 101 , a main storage 102 , an auxiliary storage 103 , and a communication unit 104 as hardware components.
  • the processor 101 is a computing device that executes an operating system and application programs. Examples of the processor include a central processing unit (CPU) and a graphics processing unit (GPU). However, the type of the processor 101 is not limited to these.
  • the main storage 102 is a device that stores a program for achieving the server 10 , computation results output from the processor 101 , and the like.
  • the main storage 102 is configured by, for example, at least one of a read-only memory (ROM) or random access memory (RAM).
  • the auxiliary storage 103 is generally a device capable of storing a larger amount of data than the main storage 102 .
  • the auxiliary storage 103 is configured by a non-volatile storage medium such as a hard disk or a flash memory.
  • the auxiliary storage 103 stores a server program P 1 that causes the server computer 100 to function as the server 10 and stores various types of data.
  • the assist program is implemented as a server program P 1 .
  • the communication unit 104 is a device that executes data communication with another computer via the communication network N.
  • the communication unit 104 is configured by, for example, a network card or a wireless communication module.
  • Each functional element of the server 10 is achieved by causing the processor 101 or the main storage 102 to read the server program P 1 and causing the processor 101 to execute the program.
  • the server program P 1 includes codes that achieve the functional elements of the server 10 .
  • the processor 101 operates the communication unit 104 according to the server program P 1 , and executes reading and writing of data from and to the main storage 102 or the auxiliary storage 103 . Through such processing, each functional element of the server 10 is achieved.
  • the server 10 may be configured by one or more computers. In a case of using a plurality of computers, the computers are connected to each other via the communication network N, thereby logically configuring single server 10 .
  • the terminal computer 200 includes, as hardware components, a processor 201 , a main storage 202 , an auxiliary storage 203 , a communication unit 204 , an input interface 205 , an output interface 206 , and an imaging unit 207 .
  • the processor 201 is a computing device that executes an operating system and application programs.
  • the processor 201 may be, for example, a CPU or a GPU, but the type of the processor 201 is not limited to these.
  • the main storage 202 is a device that stores a program for achieving the user terminal 20 , computation results output from the processor 201 , and the like.
  • the main storage 202 is configured by, for example, at least one of ROM or RAM.
  • the auxiliary storage 203 is generally a device capable of storing a larger amount of data than the main storage 202 .
  • the auxiliary storage 203 is configured by a non-volatile storage medium such as a hard disk or a flash memory.
  • the auxiliary storage 203 stores a client program P 2 for causing the terminal computer 200 to function as the user terminal 20 , and various data.
  • the communication unit 204 is a device that executes data communication with another computer via the communication network N.
  • the communication unit 204 is configured by, for example, a network card or a wireless communication module.
  • the input interface 205 is a device that receives data based on a user's operation or action.
  • the input interface 205 is configured by at least one of a keyboard, an operation button, a pointing device, a touch panel, a microphone, a sensor, or a camera.
  • the output interface 206 is a device that outputs data processed by the terminal computer 200 .
  • the output interface 206 is configured by at least one of a monitor, a touch panel, an HMD, or a speaker.
  • the imaging unit 207 is a device that captures an image of the real world, and is a camera, specifically.
  • the imaging unit 207 may capture a moving image (video) or a still image (photograph).
  • the imaging unit 207 can also function as the input interface 205 .
  • Each functional element of the user terminals 20 is achieved by causing the processor 201 or the main storage 202 to read the client program P 2 and causing the processor 201 to execute the program.
  • the client program P 2 includes code for achieving each functional element of the user terminal 20 .
  • the processor 201 operates the communication unit 204 , the input interface 205 , the output interface 206 , or the imaging unit 207 in accordance with the client program P 2 to read and write data from and to the main storage 202 or the auxiliary storage 203 . Through this processing, each functional element of the user terminal 20 is achieved.
  • At least one of the server program P 1 or the client program P 2 may be provided after being non-temporarily recorded on a tangible recording medium such as a CD-ROM, a DVD-ROM, or a semiconductor memory.
  • a tangible recording medium such as a CD-ROM, a DVD-ROM, or a semiconductor memory.
  • at least one of these programs may be provided via a communication network N as a data signal superimposed on a carrier wave. These programs may be separately provided or may be provided together.
  • FIG. 3 is a diagram illustrating an exemplary function configuration related to the assist system 1 .
  • the server 10 includes a content distributor 11 , a statistical processor 12 , an estimation unit 13 , and an assist unit 14 as functional elements.
  • the statistical processor 12 is a functional element that generates correlation data.
  • the statistical processor 12 generates correlation data by performing statistical processing on the sample data stored in the database 30 , and stores the correlation data in the database 30 .
  • the estimation unit 13 is a functional element that estimates the target user understanding level for the target content.
  • the estimation unit 13 acquires target data indicating the target user viewpoint movement from the user terminal 20 B, and estimates the target user understanding level based on the target data and the correlation data.
  • the assist unit 14 is a functional element that transmits assist information corresponding to the target user understanding level to the user terminal 20 B.
  • the user terminal 20 includes a setting unit 21 , an identification unit 22 , a calculation unit 23 , a tracking unit 24 , and a display controller 25 as functional elements.
  • the setting unit 21 is a functional element that sets a partial area of the first content displayed on the screen of the user terminal 20 as a guidance area.
  • the identification unit 22 is a functional element that identifies the first viewpoint coordinates of the user based on the eye movement of the user gazing at the guidance area.
  • the calculation unit 23 is a functional element that calculates a difference between the area coordinates of the guidance area set by the setting unit 21 and the first viewpoint coordinates identified by the identification unit 22 .
  • the tracking unit 24 is a functional element that generates viewpoint data by observing the eye movements of the user viewing the content displayed on the screen of the user terminal 20 .
  • the tracking unit 24 calibrates the second viewpoint coordinates of the user viewing the second content by using the calculated difference, and generates viewpoint data indicating the calibrated second viewpoint coordinates.
  • the display controller 25 is a functional element that controls display of a screen on the user terminal 20 .
  • the eye tracking system includes a setting unit 21 , an identification unit 22 , a calculation unit 23 , a tracking unit 24 , and a display controller 25 .
  • FIG. 4 is a flowchart illustrating, as a process flow 51 , the operation of the assist system 1 . The overview of the process by the assist system 1 will be described with reference to FIG. 4 .
  • step S 11 a statistical processor 12 of a server 10 performs statistical processing on a plurality of sets of sample data to generate correlation data.
  • a content distributor 11 distributes sample content to each of a plurality of user terminals 20 A.
  • the timing of distributing the sample content to each of the user terminals 20 A is not limited.
  • the content distributor 11 may distribute the sample content to each of the user terminals 20 A in response to a request from the user terminals 20 A, or may simultaneously distribute the sample content to two or more user terminals 20 A.
  • the display controller 25 receives and displays the sample content.
  • the tracking unit 24 of each user terminal 20 A then generates viewpoint data indicating the viewpoint movement of the sample user having visually recognized that sample content.
  • the sample user inputs his/her understanding level for the sample content to the user terminal 20 A in the form of answering a questionnaire, and the user terminal 20 A receives the input.
  • the user terminal 20 A or the server 10 may estimate the understanding level of the sample user based on the answer of the user to the sample content (e.g., answer to question).
  • the understanding level to be input or estimated indicates, for example, whether or not the meaning of a word included in a sentence has been understood, whether or not the grammar of the sentence has been understood, or the like.
  • the user terminal 20 A generates sample data representing a pair of viewpoint data generated and the understanding level input or estimated, and transmits the sample data to the server 10 .
  • the user terminal 20 A may transmit the viewpoint data to the server 10 , and the server 10 may generate sample data indicating a pair of the viewpoint data and the understanding level estimated.
  • the server 10 stores the sample data in the database 30 .
  • the server 10 stores, for a specific set of sample content, a plurality of sets of sample data obtained from a plurality of user terminals 20 A in a database 30 .
  • the server 10 may store a plurality of sets of sample data for each of the plurality of sets of sample contents.
  • the assist system 1 collects sample data through this series of processing.
  • the statistical processor 12 reads a plurality of sets of sample data from the database 30 and performs statistical processing on the plurality of sets of sample data to generate correlation data.
  • the method of the statistical processing by the statistical processor 12 and how the correlation data generated is expressed is not limited.
  • the statistical processor 12 generates correlation data by clustering a plurality of sets of sample data based on the viewpoint movement of the sample user and the user understanding level for the sample content.
  • the statistical processor 12 may determine the similarity in the viewpoint movements based on at least one of the following: the viewpoint movement speed, the number of reversals of the viewpoint (the number of changes in the direction of viewpoint movement), and the area of a region where the viewpoint moved.
  • the statistical processor 12 may determine the similarity of understanding level of content based on at least one of the understanding level of the meaning of each word or the understanding level of the grammar of each sentence.
  • the statistical processor 12 may vectorize features related to the viewpoint movement and features related to understanding level as a feature vector, and make the sample data with common or similar feature vectors belong to the same cluster.
  • the statistical processor 12 derives the correlation between the user viewpoint movement and the user understanding level from the clustering results. More specifically, this correlation indicates a pair of a user viewpoint movement tendency and the understanding level corresponding thereto.
  • the statistical processor 12 generates correlation data indicating the correlation and stores the correlation data in the database 30 .
  • the statistical processor 12 may generate correlation data by performing regression analysis. Specifically, the statistical processor 12 quantifies the viewpoint movement and the understanding level of each sample user based on predetermined rules. The statistical processor 12 then performs regression analysis on the quantified data, and generates a regression equation with the understanding level of the sample user as an objective variable and the viewpoint movement of the sample user as an explanatory variable. In this case, the statistical processor 12 may break down the viewpoint movement of the sample user into a plurality of elements, such as the viewpoint movement speed and the number of viewpoint reversals, and set a plurality of explanatory variables for these elements.
  • the statistical processor 12 may quantify the viewpoint movement speed and the number of viewpoint reversals as independent explanatory variables and perform a multi-regression analysis using the plurality of explanatory variables.
  • the statistical processor 12 stores the regression equation generated through the regression analysis in the database 30 as correlation data.
  • the method of regression analysis performed by the statistical processor 12 may be partial least squares regression (PLS) or support vector regression (SVR).
  • PLS partial least squares regression
  • SVR support vector regression
  • the correlation data also shows pairs of the user viewpoint movement tendency and the understanding level corresponding thereto.
  • the statistical processor 12 may generate correlation data by analyzing the correlation between the movement of the sample user viewpoint and the sample user understanding level through machine learning.
  • the machine learning may be deep learning using a neural network.
  • the statistical processor 12 performs supervised learning using sample data as learning data, with a machine learning model configured to output data indicating the user understanding level when data indicating the user viewpoint movement is input to the input layer, and adjusts the weighting parameters within that learning model.
  • the statistical processor 12 stores, as correlation data, the model (learned model) whose weighting parameters have been adjusted in the database 30 .
  • the statistical processor 12 may pre-process the sample data stored in the database 30 and convert it into data in a format suitable for machine learning.
  • the statistical processor 12 may generate various types of correlation data by selecting sample data for statistical processing as appropriate. For example, for each of a plurality of sets of sample content, the statistical processor 12 may generate correlation data using sample data obtained from a plurality of sample users viewing the sets of sample content. In this case, correlation data is generated for each set of content. This correlation data is hereinafter referred to as “content-specific correlation data”. Alternatively, the statistical processor 12 may generate correlation data using sample data from a plurality of sets of sample content (e.g., a plurality of sets of sample content under the same category). In this case, correlation data common to a plurality of sets of content (e.g., a plurality of sets of content under the same category) is generated. This correlation data will be hereinafter referred to as “generalized correlation data”.
  • step S 12 the assist unit 14 provides assist information to the target user visually recognizing the target content as needed.
  • the assist unit 14 estimates the target user understanding level for the target content, and provides assist information corresponding to the understanding level as needed.
  • the process for outputting assist information is detailed later.
  • the correlation between the user understanding level and the assist information is determined in advance, and the assist information is stored in the database 30 in advance in such a way that the correlation can be specified.
  • the user understanding level may be associated with the assist information so that the assist information can compensate for a part of the target content lacking the target user understanding. For example, for a user understanding level indicating that he/she does not understand the meaning of a word in a sentence in the content, the meaning of the word may be associated as the assist information.
  • FIG. 5 is a flowchart illustrating, as a process flow S 2 , the operation of the eye tracking system.
  • the processing by the eye tracking system is roughly divided into a process of calculating a difference used for calibrating the viewpoint coordinates (from step S 21 to step S 23 ) and a process of calibrating the user viewpoint coordinates by using the calculated difference (from step S 24 to step S 25 ).
  • the setting unit 21 dynamically sets, as a guidance area, a partial area of the first content displayed on the screen of the user terminal 20 .
  • the first content is any content distributed by the content distributor 11 and displayed by the display controller 25 .
  • the first content may be educational content or may be content that is not for educational purposes.
  • the guidance area is an area for causing the user to gaze, and is configured by a plurality of pixels that are continuously arranged. Dynamically setting the guidance area refers to setting a guidance area in the first content in response to display of the first content in which an area for causing the user to gaze is not set in advance, on the screen. In one example, the guidance area is set only while the first content is displayed on the screen.
  • the position of the guidance area in the first content displayed on the screen is not limited.
  • the setting unit 21 may set the guidance area at any given position such as a central part, an upper part, a lower part, or a corner part of the first content.
  • the display controller 25 displays the guidance area in the first content based on that setting.
  • the shape and area (number of pixels) of the guidance area are not limited either. Since the guidance area is the area that the user is prompted to gaze so as to calibrate the viewpoint coordinates, the setting unit 21 typically sets the area of the guidance area to be much smaller than the area of the first content displayed on the screen (i.e., the area of the display device).
  • the method of dynamically setting the guidance area is not limited.
  • the setting unit 21 may visually distinguish the guidance area from areas other than the guidance area (hereinafter referred to as the non-guidance areas) by making the display mode of the guidance area different from the non-guidance areas.
  • the method of setting the display mode is not limited.
  • the setting unit 21 may distinguish the guidance areas from the non-guidance areas by decreasing the resolution of non-guidance areas so that the resolution of the guidance area relatively increases without a change in the resolution of the guidance areas.
  • the setting unit 21 may distinguish the guidance area from the non-guidance area by performing a blurring the non-guidance area without changing the display mode of the guidance area.
  • the setting unit 21 may perform the blurring process by setting the color of a target pixel in a non-guidance area to the average color of the plurality of pixels adjacent to that target pixel.
  • the setting unit 21 may perform the blurring process while maintaining the resolution of the non-guidance area, or may perform the blurring process after reducing the resolution.
  • the setting unit 21 may distinguish the guidance area from the non-guidance area by surrounding the outer edge of the guidance area with a specific color or a specific type of frame line.
  • the setting unit 21 may distinguish the guidance area from the other areas by combining any two or more out of resolution adjustment, blurring process, and border drawing.
  • the setting unit 21 may set the area having the selectable object as the guidance area.
  • the setting unit 21 may identify that selectable object as a partial area and set that selectable object as the guidance area.
  • the selectable object may be a selection button or link displayed on a tutorial screen of an application program.
  • the selectable object may be a button for selecting a question or a button for starting the exercise or the test.
  • the setting unit 21 may decrease the resolution of the non-guidance area while maintaining the resolution of the selectable object set as the guidance area.
  • the setting unit 21 may perform a blurring process on the non-guidance area, or may surround the outer edge of the selectable object set as the guidance area with a specific color or a specific type of frame line.
  • the setting unit 21 sets the area coordinates of the guidance area using any given method.
  • the setting unit 21 may set the coordinates of the center of the guidance area or the center of gravity of the guidance area as the area coordinates.
  • the setting unit 21 may set the position of any one of the pixels in the guidance area as the area coordinates.
  • the identification unit 22 identifies the viewpoint coordinates of the user gazing at the guidance area as first viewpoint coordinates.
  • the identification unit 22 identifies the viewpoint coordinates based on the user's eye movement.
  • the method of identifying the viewpoint coordinates is not limited.
  • the identification unit 22 may take a peripheral image of the user's eye by the imaging unit 207 of the user terminal 20 and identifies the viewpoint coordinates based on the position of the iris with the user's inner canthus as the reference point.
  • the identification unit 22 may identify the viewpoint coordinates of the user by using a pupil center corneal reflection method (PCCR).
  • PCCR pupil center corneal reflection method
  • the user terminal 20 may include an infrared emitting device and an infrared camera as a hardware configuration.
  • step S 23 the calculation unit 23 calculates a difference between the first viewpoint coordinates identified by the identification unit 22 and the area coordinates of the guidance area set by the setting unit 21 .
  • the calculation unit 23 stores the calculated difference in any storage device, such as the main storage 202 , or the auxiliary storage 203 .
  • the user terminal 20 may repeat the process from step S 21 to step S 23 multiple times while changing the position of the guidance area.
  • the calculation unit 23 may set a statistical value (e.g., a mean value) of the plurality of differences calculated, as the difference to be used in the subsequent calibration process (step S 25 ).
  • the tracking unit 24 identifies the viewpoint coordinates of the user who looks at the second content as second viewpoint coordinates.
  • the second content is any content distributed by the content distributor 11 and displayed by the display controller 25 .
  • the second content may be sample content or target content.
  • the tracking unit 24 may identify the second viewpoint coordinates in a manner similar to identifying the first viewpoint coordinates by the identification unit 22 (i.e., through a method similar to step S 22 ).
  • the second content may be different from or the same as the first content.
  • the tracking unit 24 may repeat the processing of steps S 24 and S 25 , acquire calibrated coordinates of a plurality of second viewpoints arranged in a time sequence, and generate viewpoint data indicating the movement of the user viewpoint.
  • the tracking unit 24 may acquire calibrated coordinates of a plurality of second viewpoints, and the server 10 may generate viewpoint data based on the coordinates of the plurality of second viewpoints.
  • FIG. 6 and FIG. 7 is a diagram illustrating an exemplary guidance area set in the first content by the setting unit 21 .
  • the setting unit 21 sets the guidance area by reducing the resolution of the non-guidance area.
  • the user terminal 20 displays first content C 11 including a child, lawn, and a ball, and calculates a difference while changing the position of the guidance area on the first content C 11 .
  • the display changes in an order of screens D 11 , D 12 , and D 13 .
  • the non-guidance area is represented by a dashed line.
  • the setting unit 21 sets a part of the child's face as the guidance area A 11 .
  • the screen D 11 corresponds to this setting.
  • the setting unit 21 lowers the resolutions of the areas (non-guidance areas) other than the guidance area A 11 without changing the resolution of the guidance area A 11 .
  • the setting unit 21 may reduce the resolution of the non-guidance area so that the resolution of the guidance area A 11 is more than a double or a quadruple of the resolution of the non-guidance area. For example, where the resolution of the guidance area A 11 is 300 ppi, the resolution of the non-guidance area may be 150 ppi or less or 75 ppi or less.
  • the non-guidance area appears blur as compared to the guidance area A 11 , so that the user's line of sight is usually directed to the clearly displayed guidance area A 11 . Therefore, it is possible to identify the viewpoint coordinates (first viewpoint coordinates) of the user gazing at the guidance area A 11 . While the screen D 11 is displayed, the identification unit 22 acquires the first viewpoint coordinates of the user. The calculation unit 23 then calculates the difference between that first viewpoint coordinates and the area coordinates of the guidance area A 11 .
  • the setting unit 21 sets a part of the ball as the guidance area A 12 .
  • the screen D 12 corresponds to this setting.
  • the setting unit 21 restores the resolution of the guidance area A 12 to the original value, and lowers the resolutions of the areas (non-guidance areas) other than the guidance area A 12 .
  • the line of sight of the user is usually directed to the guidance area A 12 .
  • the identification unit 22 acquires the first viewpoint coordinates of the user.
  • the calculation unit 23 then calculates the difference between first viewpoint coordinates and the area coordinates of the guidance area A 12 .
  • the setting unit 21 sets the lower right part of the first content C 11 (the lawn area) as the guidance area A 13 .
  • the screen D 13 corresponds to this setting.
  • the setting unit 21 restores the resolution of the guidance area A 13 to the original value, and lowers the resolutions of the areas (non-guidance areas) other than the guidance area A 13 .
  • the line of sight of the user is usually directed to the guidance area A 13 .
  • the identification unit 22 acquires the first viewpoint coordinates of the user.
  • the calculation unit 23 then calculates the difference between that first viewpoint coordinates and the area coordinates of the guidance area A 13 .
  • the calculation unit 23 obtains a statistical value of a plurality of differences calculated. This statistical value is used for calibration (step S 25 ) of the second viewpoint coordinates by the tracking unit 24 .
  • the setting unit 21 sets the selectable object in the first content C 21 as the guidance area.
  • the first content C 21 is a tutorial for an online academic test. With a progress in the tutorial, the display changes in an order of the screens D 11 , D 12 , and D 13 .
  • Screen D 21 includes a text string “Questions for Japanese language.” and an OK button.
  • the OK button is a selectable object.
  • the setting unit 21 sets the area with the OK button as a guidance area A 21 . Normally, the user gazes at the selectable object while operating the selectable object. Therefore, the viewpoint coordinates (first viewpoint coordinates) of the user gazing at the guidance area A 21 can be identified.
  • the identification unit 22 acquires the first viewpoint coordinates of the user.
  • the calculation unit 23 then calculates the difference between that first viewpoint coordinates and the area coordinates of the guidance area A 21 .
  • the display controller 25 switches the screen D 21 to the screen D 22 .
  • Screen D 22 includes the text string “Please select the number of questions.” and three selection buttons: “5 questions”, “10 questions” and “15 questions”. These selection buttons are selectable objects.
  • the setting unit 21 sets the areas with the three selection buttons as a guidance area A 22 , a guidance area A 23 , and a guidance area A 24 , respectively.
  • the identification unit 22 identifies the viewpoint coordinates (first viewpoint coordinates) of the user.
  • the calculation unit 23 calculates the difference between the first viewpoint coordinates and the area coordinates of the guidance area corresponding to the selectable object selected by the user (any one of the guidance areas A 22 to A 24 ).
  • the display controller 25 switches the screen D 22 to the screen D 23 .
  • Screen D 23 includes the text string “Start test?” and a start button.
  • the start button is a selectable object.
  • the setting unit 21 sets the area with the start button as a guidance area A 25 .
  • the identification unit 22 acquires the first viewpoint coordinates of the user.
  • the calculation unit 23 calculates the difference between the first viewpoint coordinates and the area coordinates of the guidance area A 25 .
  • the calculation unit 23 obtains a statistical value of a plurality of differences calculated. This statistical value is used for calibration (step S 25 ) of the second viewpoint coordinates by the tracking unit 24 .
  • FIG. 8 is a flowchart illustrating, as a process flow S 3 , an example of the operation of the assist system 1 .
  • the process flow S 3 indicates a processing procedure for providing assist information to the target user who views the target content.
  • the process flow S 3 is based on the premise that the target user has logged into the assist system 1 . It is also assumed that the eye tracking system has already calculated the differences that are used for calibrating the viewpoint coordinates.
  • step S 31 the display controller 25 of the user terminal 20 B displays the target content on the screen of the user terminal 20 B.
  • the display controller 25 receives, from the server 10 , content data distributed from the content distributor 11 , and displays the target content based on the content data.
  • step S 32 the tracking unit 24 of the user terminal 20 B acquires the viewpoint coordinates (second viewpoint coordinates) of the target user who visually recognizes the target content. Specifically, the tracking unit 24 identifies the viewpoint coordinates (viewpoint coordinates before calibration) based on the eye movements of the target user looking at the target content, and calibrates the viewpoint coordinates identified, by using the difference calculated in advance. The tracking unit 24 may acquire the calibrated viewpoint coordinates at each given time interval and generate viewpoint data (i.e., target data indicating the movement of the target user viewpoint) in which coordinates of a plurality of viewpoints are arranged in a time sequence.
  • viewpoint data i.e., target data indicating the movement of the target user viewpoint
  • the estimation unit 13 acquires the target data.
  • the estimation unit 13 may receive the target data from the tracking unit 24 of the user terminal 20 B.
  • the tracking unit 24 may sequentially transmit calibrated coordinates of a plurality of viewpoints to the server 10 , and the estimation unit 13 may generate viewpoint data (target data) in which coordinates of a plurality of viewpoints are arranged in a time sequence.
  • the estimation unit 13 refers to the database 30 to obtain correlation data, and estimates the target user understanding level for the target content based on the target data and correlation data.
  • the estimation unit 13 estimates the understanding level indicated by the cluster to which the target data belongs as the target user understanding level.
  • the estimation unit 13 applies the target data to the regression equation to estimate the target user understanding level.
  • the estimation unit 13 estimates the target user understanding level by inputting the target data into that learned model.
  • step S 35 the assist unit 14 acquires assist information corresponding to the target user understanding level from the database 30 , and transmits the assist information to the user terminals 20 B.
  • the display controller 25 of the user terminal 20 B displays the assist information on the screen of the user terminal 20 B.
  • the output timing of the assist information is not limited.
  • the display controller 25 may output the assist information after a predetermined time (e.g., 15 seconds) has elapsed from the point of displaying the target content on the screen of the user terminal 20 .
  • the display controller 25 may output the assist information in response to a request from the user.
  • the display controller 25 may adjust the display time of the assist information according to the user understanding level.
  • the display controller 25 may display the assist information only during a display time set in advance by the user or others.
  • the assist unit 14 may display the assist information until the display of the target content is switched, or until user input is made for the target content (e.g., answers to questions). If the estimated understanding level indicates that the target user understanding level for the target content is sufficient, the assist unit 14 may terminate the process without outputting any assist information.
  • An output mode of the assist information is not limited.
  • the assist information includes voice data
  • the user terminal 20 may output the voice data from a speaker.
  • step S 36 the assist system 1 repeats the process from step S 32 to step S 35 while the user terminal 20 B is displaying the target content.
  • the assist system 1 repeats the series of processes while the target content is displayed.
  • FIG. 9 is a flowchart illustrating as a process flow S 4 an exemplary operation of the assist system 1 .
  • the process flow S 4 also relates to a process of providing the assist information to the target user who views the target content, but different from that of the process flow S 3 in the specific steps thereof.
  • the process flow S 4 also assumes that the target user is logged into the assist system 1 and that the eye tracking system has already calculated the difference.
  • step S 41 the display controller 25 of the user terminal 20 B displays the target content on the screen of the user terminal 20 B.
  • step S 42 the tracking unit 24 of the user terminal 20 B acquires the viewpoint coordinates (second viewpoint coordinates) of the target user who visually recognizes the target content.
  • step S 43 the estimation unit 13 acquires target data indicating the target user viewpoint movement. This series of processes is similar to steps S 31 to S 33 .
  • step S 44 the estimation unit 13 refers to the database 30 to obtain generalized correlation data, and estimates the target user understanding level (first understanding level) for the target content based on the target data and generalized correlation data.
  • a specific estimation method is similar to the method in step S 34 .
  • step S 45 the assist unit 14 acquires assist information corresponding to the target user's first understanding level from the database 30 , and transmits the assist information to the user terminal 20 B.
  • the display controller 25 of the user terminal 20 B outputs assist information to the screen of the user terminal 20 B.
  • step S 46 the assist unit 14 determines whether to provide additional assistance to the target user, that is, whether to provide additional assist information to the target user.
  • the process proceeds to step S 49 .
  • the assist unit 14 determines to perform additional assistance, the process proceeds to step S 47 .
  • the assist unit 14 may determine that no additional assistance is provided if there is a user input to the target content (e.g., an answer to a question) within a predetermined time period, and if there is no user input within the predetermined time period, the assist unit 14 may determine that additional assistance is provided.
  • step S 47 the estimation unit 13 refers to the database 30 to obtain correlation data specific to the target content, and estimates the target user understanding level for the target content (second understanding level) based on the target data and content-specific correlation data. This process assumes that the same content is used as sample content and target content.
  • a specific estimation method is similar to the method in step S 34 .
  • step S 48 the assist unit 14 acquires additional assist information corresponding to the target user's second understanding level from the database 30 , and transmits the assist information to the user terminal 20 B.
  • the display controller 25 of the user terminal 20 B outputs additional assist information to the screen of the user terminal 20 B.
  • step S 49 the assist system 1 repeats the process from step S 42 to step S 48 while the user terminal 20 B is displaying the target content.
  • the assist system 1 repeats the series of processes while the target content is displayed.
  • FIG. 10 is a diagram illustrating exemplary assist information.
  • the assist system 1 refers to correlation data that includes information about an understanding level Ra indicating “lack of vocabulary”, an understanding level Rb indicating “lack of grammatical competence” and an understanding level Rc indicating “lack of understanding about the background of the sentence”.
  • the assist unit 14 outputs assist information B 11 corresponding to the user's understanding level.
  • the assist unit 14 If the estimation unit 13 estimates that the target user's grammatical ability is insufficient, the assist unit 14 outputs assist information B 12 corresponding to the user's understanding level. If the estimation unit 13 estimates that the target user does not understand the background of the sentence, the assist unit 14 outputs assist information B 13 corresponding to the user's understanding level.
  • the display controller 25 of the user terminal 20 B displays the assist information output. The target user can refer to the assist information to solve the problem.
  • an eye tracking system includes at least one processor.
  • the at least one processor is configured to: dynamically set a partial area of first content displayed on a screen as a guidance area to have a user gaze at the partial area; identify first viewpoint coordinates on the screen based on an eye movement of the user gazing at the guidance area; calculate a difference between the first viewpoint coordinates identified and area coordinates of the guidance area on the screen; and calibrate second viewpoint coordinates of the user viewing second content displayed on the screen by using the difference.
  • An eye tracking method is executed by an eye tracking system including at least one processor.
  • the eye tracking method includes: dynamically setting a partial area of first content displayed on a screen as a guidance area to have a user gaze at the partial area; identifying first viewpoint coordinates on the screen based on an eye movement of the user gazing at the guidance area; calculating a difference between the first viewpoint coordinates identified and area coordinates of the guidance area on the screen; and calibrating second viewpoint coordinates of the user viewing second content displayed on the screen by using the difference.
  • An eye tracking program related to an aspect of the present disclosure causes a computer to execute: dynamically setting a partial area of first content displayed on a screen as a guidance area to have a user gaze at the partial area; identifying first viewpoint coordinates on the screen based on an eye movement of the user gazing at the guidance area; calculating a difference between the first viewpoint coordinates identified and area coordinates of the guidance area on the screen; and calibrating second viewpoint coordinates of the user viewing second content displayed on the screen by using the difference.
  • a guidance area for the user to gaze at is dynamically set for any content (first content), and the guidance area is used to calculate the difference to be used in calibration. Accordingly, there is no need to prepare content for calibration in advance. This makes calibration for eye tracking easier to perform.
  • the at least one processor may set the partial area as the guidance area by adjusting a resolution of the screen such that a resolution of the partial area is higher than a resolution of an area other than the partial area.
  • the at least one processor sets the partial area as the guidance area by performing blurring processing on an area other than the partial area.
  • the guidance area is displayed more clearly than the non-guidance area, so that the user's line of sight can be naturally brought to the guidance area while he or she views the first content.
  • the at least one processor may set the partial area as the guidance area by surrounding an outer edge of the partial area with a frame line. In this case, since the guidance area and the other area are distinguished from each other by the frame line, the user can easily recognize the position of the guidance area.
  • the at least one processor may set an area with a selectable object to be selected by the user as the guidance area.
  • the user usually gazes at the selectable object at the time of selecting the selectable object. Therefore, by setting the display area of the selectable object as the guidance area, the user's line of sight can be naturally brought to the guidance area.
  • the at least one processor may calculate the difference multiple times while changing the position of the guidance area in the first content. In this case, multiple differences calculated can be used to calibrate the second viewpoint coordinates, which improves the calibration accuracy.
  • the assist system 1 is configured by using the server 10 ; however, the assist system 1 may be configured without the server 10 .
  • each functional element of the server 10 may be implemented in any one of the user terminals 20 , and for example, may be implemented in any one of a terminal used by a distributor of content and a terminal used by a viewer of content.
  • each one of the functional elements of the server 10 may be implemented separately in a plurality of user terminals 20 , e.g., separate terminals of the distributor and of the viewer.
  • the assist program may be implemented as a client program. Since the user terminal 20 has the function of the server 10 , it is possible to reduce the load on the server 10 .
  • information about the viewer of the content such as students (e.g., data indicating viewpoint movement), is not transmitted outside of the user terminal 20 , making it possible to more reliably protect the viewer's confidentiality.
  • the eye tracking system is configured only with the user terminal 20 ; however, the system may be configured by using the server 10 .
  • some functional elements of the user terminal 20 may be implemented in the server 10 .
  • a functional element corresponding to the calculation unit 23 may be implemented in the server 10 .
  • the assist information is displayed separately from the target content; however, the assist information may be displayed in such a manner as to constitute a part of the target content.
  • the assist unit 14 may highlight parts of the sentence (e.g., parts that are important for understanding the text) as assist information.
  • the assist information may be a visual effect added to the target content.
  • the assist unit 14 may perform the highlighting by making the color or font of a part of the sentence subject to the assist information different from other parts.
  • the assist system 1 outputs the assist information corresponding to the target user understanding level.
  • the assist system 1 may output the assist information without using the understanding level. This variation is described below.
  • the server 10 obtains viewpoint data indicating the movement of the viewpoint of the sample user who visually recognized the sample contents and sample data indicating assist information presented to the sample user from respective user terminals 20 A, and stores the sample data in a database 30 .
  • the assist information presented to the sample user i.e., the assist information corresponding to the sample user
  • the statistical processor 12 performs statistical processing on the sample data in the database 30 , generates correlation data indicating the correlation between the user viewpoint movement and the content assist information, and stores the correlation data in the database 30 .
  • the statistical processing methods and the form of expression of the generated correlation data are not limited. Therefore, statistical processor 12 may generate correlation data through various methods such as clustering, regression analysis, machine learning, etc.
  • Server 10 outputs the assist information corresponding to the target data received from user terminal 20 B based on the target data and its correlation data.
  • the estimation unit 13 refers to the database 30 to obtain the correlation data and identify the assist information corresponding to the target data.
  • the estimation unit 13 identifies the assist information indicated by the cluster to which the target data belongs.
  • the estimation unit 13 applies the target data to the regression equation to identify the assist information.
  • the estimation unit 13 identifies the assist information by inputting the target data into that learned model.
  • the assist unit 14 obtains the assist information identified from database 30 and transmits the assist information to user terminal 20 B.
  • the expression “at least one processor executes a first process, a second process, and . . . executes an n-th process.” or the expression corresponding thereto is a concept including the case where the execution bodies (i.e., processors) of the n processes from the first process to the n-th process change in the middle.
  • this expression is a concept including both a case where all of the n processes are executed by the same processor and a case where the processor changes during the n processes, according to any given policy.
  • the processing procedure of the method executed by the at least one processor is not limited to the example of the above embodiments. For example, a part of the above-described steps (processing) may be omitted, or each step may be executed in another order. Any two or more of the above-described steps may be combined, or some of the steps may be modified or deleted. Alternatively, other steps may be executed in addition to the steps described above.

Landscapes

  • Engineering & Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Human Computer Interaction (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • User Interface Of Digital Computer (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Position Input By Displaying (AREA)
  • Health & Medical Sciences (AREA)
  • Heart & Thoracic Surgery (AREA)
  • Biophysics (AREA)
  • Ophthalmology & Optometry (AREA)
  • Biomedical Technology (AREA)
  • Eye Examination Apparatus (AREA)
  • Medical Informatics (AREA)
  • Molecular Biology (AREA)
  • Surgery (AREA)
  • Animal Behavior & Ethology (AREA)
  • General Health & Medical Sciences (AREA)
  • Public Health (AREA)
  • Veterinary Medicine (AREA)
US17/997,976 2020-09-30 2021-09-01 Eye tracking system, eye tracking method, and eye tracking program Abandoned US20230210364A1 (en)

Applications Claiming Priority (3)

Application Number Priority Date Filing Date Title
JP2020166486A JP7034228B1 (ja) 2020-09-30 2020-09-30 アイトラッキングシステム、アイトラッキング方法、及びアイトラッキングプログラム
JP2020-166486 2020-09-30
PCT/JP2021/032163 WO2022070746A1 (ja) 2020-09-30 2021-09-01 アイトラッキングシステム、アイトラッキング方法、およびアイトラッキングプログラム

Related Parent Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2021/032163 A-371-Of-International WO2022070746A1 (ja) 2020-09-30 2021-09-01 アイトラッキングシステム、アイトラッキング方法、およびアイトラッキングプログラム

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US19/001,237 Continuation US12554324B2 (en) 2020-09-30 2024-12-24 Eye tracking system, eye tracking method, and eye tracking program

Publications (1)

Publication Number Publication Date
US20230210364A1 true US20230210364A1 (en) 2023-07-06

Family

ID=80950024

Family Applications (2)

Application Number Title Priority Date Filing Date
US17/997,976 Abandoned US20230210364A1 (en) 2020-09-30 2021-09-01 Eye tracking system, eye tracking method, and eye tracking program
US19/001,237 Active US12554324B2 (en) 2020-09-30 2024-12-24 Eye tracking system, eye tracking method, and eye tracking program

Family Applications After (1)

Application Number Title Priority Date Filing Date
US19/001,237 Active US12554324B2 (en) 2020-09-30 2024-12-24 Eye tracking system, eye tracking method, and eye tracking program

Country Status (4)

Country Link
US (2) US20230210364A1 (https=)
JP (2) JP7034228B1 (https=)
CN (1) CN115916032A (https=)
WO (1) WO2022070746A1 (https=)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20250110550A1 (en) * 2023-09-29 2025-04-03 Apple Inc. Adaptive blurring of virtual content
US12554324B2 (en) 2020-09-30 2026-02-17 Dwango Co., Ltd. Eye tracking system, eye tracking method, and eye tracking program

Citations (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20140226131A1 (en) * 2013-02-14 2014-08-14 The Eye Tribe Aps Systems and methods of eye tracking calibration
US20140375541A1 (en) * 2013-06-25 2014-12-25 David Nister Eye tracking via depth camera
US20160116980A1 (en) * 2013-03-01 2016-04-28 Tobii Ab Two step gaze interaction
US20160139665A1 (en) * 2014-11-14 2016-05-19 The Eye Tribe Aps Dynamic eye tracking calibration
US20170076503A1 (en) * 2015-09-16 2017-03-16 Bandai Namco Entertainment Inc. Method for generating image to be displayed on head tracking type virtual reality head mounted display and image generation device
US20170344112A1 (en) * 2016-05-31 2017-11-30 Fove, Inc. Gaze detection device
US20180286105A1 (en) * 2017-04-01 2018-10-04 Intel Corporation Motion biased foveated renderer
US20180308252A1 (en) * 2017-04-19 2018-10-25 The Nielsen Company (Us), Llc Methods and systems to increase accuracy of eye tracking
US10268268B1 (en) * 2016-09-02 2019-04-23 Facebook Technologies, Llc Waveguide integrated eye tracking
US20210093192A1 (en) * 2019-09-27 2021-04-01 Alcon Inc. Instant eye gaze calibration systems and methods
US20220050522A1 (en) * 2020-08-11 2022-02-17 Inseye LLC Eye tracking system for use in head-mounted display units

Family Cites Families (29)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP4710508B2 (ja) * 2005-09-21 2011-06-29 富士ゼロックス株式会社 画像処理プログラム、画像処理装置
US8154583B2 (en) * 2007-05-31 2012-04-10 Eastman Kodak Company Eye gazing imaging for video communications
JP5664064B2 (ja) * 2010-09-22 2015-02-04 富士通株式会社 視線検出装置および補正係数算出プログラム
US20150103096A1 (en) * 2012-05-30 2015-04-16 Pioneer Corporation Display device, head mount display, calibration method, calibration program and recording medium
JP2015152939A (ja) * 2014-02-10 2015-08-24 ソニー株式会社 情報処理装置、情報処理方法、及びプログラム
WO2015125243A1 (ja) * 2014-02-19 2015-08-27 三菱電機株式会社 表示制御装置、表示制御装置の表示制御方法、視線方向検出システムおよび視線方向検出システムのキャリブレーション制御方法
GB2528446B (en) * 2014-07-21 2021-08-04 Tobii Tech Ab Method and apparatus for detecting and following an eye and/or the gaze direction thereof
JP6319349B2 (ja) * 2015-04-03 2018-05-09 株式会社デンソー 情報提示装置
JP2017176414A (ja) * 2016-03-30 2017-10-05 ソニー株式会社 制御装置および方法並びに手術システム
JP2017176411A (ja) 2016-03-30 2017-10-05 株式会社平和 遊技機
JP2017215928A (ja) * 2016-05-31 2017-12-07 フォーブ インコーポレーテッド 視線検出システム
JP2018004950A (ja) * 2016-07-01 2018-01-11 フォーブ インコーポレーテッド 映像表示システム、映像表示方法、映像表示プログラム
US20180059773A1 (en) * 2016-08-29 2018-03-01 Korea Automotive Technology Institute System and method for providing head-up display information according to driver and driving condition
CN111107779B (zh) * 2017-06-23 2023-06-30 自适应传感技术公司 用于测试和分析视敏度及其变化的系统和方法
JP6856200B2 (ja) * 2017-07-26 2021-04-07 国立研究開発法人産業技術総合研究所 視線検出校正方法、システム、及びコンピュータプログラム
JP6279136B1 (ja) * 2017-08-15 2018-02-14 株式会社テクテック 位置ゲームにおけるオブジェクト制御システム、プログラム及び方法
JP2019091253A (ja) * 2017-11-15 2019-06-13 株式会社ソニー・インタラクティブエンタテインメント 視線検出装置
JP2019101137A (ja) * 2017-11-30 2019-06-24 国立大学法人東京工業大学 視線誘導システム
JP7064345B2 (ja) * 2018-02-09 2022-05-10 株式会社デンソーテン 表示処理装置および表示処理方法
TWI732342B (zh) * 2018-11-25 2021-07-01 宏達國際電子股份有限公司 傳輸眼球追蹤資訊的方法、頭戴式顯示器及電腦裝置
JP2020111202A (ja) * 2019-01-11 2020-07-27 株式会社リコー 表示制御装置、表示装置、表示システム、移動体、プログラム、画像生成方法
US10810966B1 (en) * 2019-01-15 2020-10-20 Ambarella International Lp Fusion of electronic mirror systems and driver monitoring for increased convenience and added safety
JP6559375B1 (ja) * 2019-03-28 2019-08-14 株式会社ドワンゴ コンテンツ配信システム、コンテンツ配信方法、およびコンテンツ配信プログラム
CN113109941B (zh) * 2020-01-10 2023-02-10 未来（北京）黑科技有限公司 一种分层成像的抬头显示系统
CN115668101A (zh) * 2020-05-21 2023-01-31 奇跃公司 使用眼睛跟踪对空间光调制显示器进行扭曲
JP7034228B1 (ja) 2020-09-30 2022-03-11 株式会社ドワンゴ アイトラッキングシステム、アイトラッキング方法、及びアイトラッキングプログラム
EP4144560B1 (en) * 2021-09-03 2025-03-05 Volkswagen Ag Method, computer program and apparatus for controlling operation of two or more configurable systems of a motor vehicle
US11682357B1 (en) * 2022-02-18 2023-06-20 Harman International Industries, Incorporated Heads up display
US20230359419A1 (en) * 2022-05-09 2023-11-09 Toyota Motor Engineering & Manufacturing North America, Inc. Intelligent heads up display

Patent Citations (17)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20140226131A1 (en) * 2013-02-14 2014-08-14 The Eye Tribe Aps Systems and methods of eye tracking calibration
US20160116980A1 (en) * 2013-03-01 2016-04-28 Tobii Ab Two step gaze interaction
US20140375541A1 (en) * 2013-06-25 2014-12-25 David Nister Eye tracking via depth camera
US9851791B2 (en) * 2014-11-14 2017-12-26 Facebook, Inc. Dynamic eye tracking calibration
US20160139665A1 (en) * 2014-11-14 2016-05-19 The Eye Tribe Aps Dynamic eye tracking calibration
US10636212B2 (en) * 2015-09-16 2020-04-28 Bandai Namco Entertainment Inc. Method for generating image to be displayed on head tracking type virtual reality head mounted display and image generation device
US20170076503A1 (en) * 2015-09-16 2017-03-16 Bandai Namco Entertainment Inc. Method for generating image to be displayed on head tracking type virtual reality head mounted display and image generation device
US20170344112A1 (en) * 2016-05-31 2017-11-30 Fove, Inc. Gaze detection device
US10268268B1 (en) * 2016-09-02 2019-04-23 Facebook Technologies, Llc Waveguide integrated eye tracking
US20180286105A1 (en) * 2017-04-01 2018-10-04 Intel Corporation Motion biased foveated renderer
US10152822B2 (en) * 2017-04-01 2018-12-11 Intel Corporation Motion biased foveated renderer
US20180308252A1 (en) * 2017-04-19 2018-10-25 The Nielsen Company (Us), Llc Methods and systems to increase accuracy of eye tracking
US10304209B2 (en) * 2017-04-19 2019-05-28 The Nielsen Company (Us), Llc Methods and systems to increase accuracy of eye tracking
US20210093192A1 (en) * 2019-09-27 2021-04-01 Alcon Inc. Instant eye gaze calibration systems and methods
US11786119B2 (en) * 2019-09-27 2023-10-17 Alcon Inc. Instant eye gaze calibration systems and methods
US20220050522A1 (en) * 2020-08-11 2022-02-17 Inseye LLC Eye tracking system for use in head-mounted display units
US11740695B2 (en) * 2020-08-11 2023-08-29 Inseye Inc. Eye tracking system for use in head-mounted display units

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US12554324B2 (en) 2020-09-30 2026-02-17 Dwango Co., Ltd. Eye tracking system, eye tracking method, and eye tracking program
US20250110550A1 (en) * 2023-09-29 2025-04-03 Apple Inc. Adaptive blurring of virtual content
US12346496B2 (en) * 2023-09-29 2025-07-01 Apple Inc. Adaptive blurring of virtual content

Also Published As

Publication number Publication date
US12554324B2 (en) 2026-02-17
CN115916032A (zh) 2023-04-04
US20250138631A1 (en) 2025-05-01
JP2022058150A (ja) 2022-04-11
JP2022057959A (ja) 2022-04-11
WO2022070746A1 (ja) 2022-04-07
JP7034228B1 (ja) 2022-03-11

Similar Documents

Publication Publication Date Title
US12554324B2 (en) Eye tracking system, eye tracking method, and eye tracking program
Keighrey et al. A physiology-based QoE comparison of interactive augmented reality, virtual reality and tablet-based applications
US20220198952A1 (en) Assessment and training system
US20230360548A1 (en) Assist system, assist method, and assist program
US20230237922A1 (en) Artificial intelligence-driven avatar-based personalized learning techniques
Pise et al. Estimation of learning affects experienced by learners: an approach using relational reasoning and adaptive mapping
JP7099377B2 (ja) 情報処理装置及び情報処理方法
WO2023041940A1 (en) Gaze-based behavioural monitoring system
KR102245319B1 (ko) 원격지 학습자의 집중도를 분석하는 시스템
Shekar et al. VREye: Exploring human visual acuity test using virtual reality
KR20190048144A (ko) 발표 및 면접 훈련을 위한 증강현실 시스템
Adiani et al. Evaluation of webcam-based eye tracking for a job interview training platform: Preliminary results
JP7607272B2 (ja) 学習支援装置および学習支援システム
KR20230153552A (ko) Adhd 아동 청소년의 주의산만 및 충동성 개선을 위한 vr 기반 훈련 시스템 및 방법
WO2022158160A1 (ja) 支援システム、支援方法、及びプログラム
Sadeghzadeh et al. Assistive Visual Tool: Enhancing Safe Navigation with Video Remapping in AR Headsets
Bennett et al. Looking at faces: autonomous perspective invariant facial gaze analysis
Evju Predicting student performance in game-based learning with Artificial Intelligence ensemble pipelines
Gådin Factors for good text legibility: eye-tracking in virtual reality
Artiran Analysis and Practice of Nonverbal Communication and Attention in Autism with Virtual Reality Job Interviews Using Machine Learning
Du et al. Talk to Me, Not the Slides: A Real-Time Wearable Assistant for Improving Eye Contact in Presentations
Yakoubi et al. Towards an Assistive Eye-Controlled Interface
Gorach Designing and Evaluating Extended Reality Solutions for Sign Reading Assistance for People with Low Vision
Abinaya et al. A Survey of AI-Enhanced Augmented Reality in Phobia Treatment: Innovations, Challenges, and Societal Impacts
Ishimaru Meta-Augmented Human: From Physical to Cognitive Towards Affective State Recognition

Legal Events

Date Code Title Description
AS Assignment

Owner name: DWANGO CO., LTD., JAPAN

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:KAWAKAMI, NOBUO;ODAGIRI, YURI;REEL/FRAME:061662/0251

Effective date: 20221018

STPP Information on status: patent application and granting procedure in general

Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION

STPP Information on status: patent application and granting procedure in general

Free format text: NON FINAL ACTION MAILED

STPP Information on status: patent application and granting procedure in general

Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER

STPP Information on status: patent application and granting procedure in general

Free format text: FINAL REJECTION MAILED

STPP Information on status: patent application and granting procedure in general

Free format text: RESPONSE AFTER FINAL ACTION FORWARDED TO EXAMINER

STPP Information on status: patent application and granting procedure in general

Free format text: ADVISORY ACTION MAILED

STPP Information on status: patent application and granting procedure in general

Free format text: RESPONSE AFTER FINAL ACTION FORWARDED TO EXAMINER

STPP Information on status: patent application and granting procedure in general

Free format text: ADVISORY ACTION MAILED

STPP Information on status: patent application and granting procedure in general

Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION

STPP Information on status: patent application and granting procedure in general

Free format text: NON FINAL ACTION MAILED

STPP Information on status: patent application and granting procedure in general

Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER

STPP Information on status: patent application and granting procedure in general

Free format text: FINAL REJECTION MAILED

STCB Information on status: application discontinuation

Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION