US20230393653A1 - Calibration for gaze detection - Google Patents
Calibration for gaze detection Download PDFInfo
- Publication number
- US20230393653A1 US20230393653A1 US18/248,832 US202118248832A US2023393653A1 US 20230393653 A1 US20230393653 A1 US 20230393653A1 US 202118248832 A US202118248832 A US 202118248832A US 2023393653 A1 US2023393653 A1 US 2023393653A1
- Authority
- US
- United States
- Prior art keywords
- image
- eye
- user
- gaze
- camera
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000001514 detection method Methods 0.000 title claims abstract description 48
- 238000000034 method Methods 0.000 claims abstract description 55
- 230000004418 eye rotation Effects 0.000 claims abstract description 17
- 210000001508 eye Anatomy 0.000 claims description 56
- 210000001747 pupil Anatomy 0.000 claims description 45
- 210000003128 head Anatomy 0.000 claims description 26
- 210000005252 bulbus oculi Anatomy 0.000 claims description 19
- 238000012545 processing Methods 0.000 claims description 19
- 230000000007 visual effect Effects 0.000 claims description 18
- 230000006835 compression Effects 0.000 claims description 10
- 238000007906 compression Methods 0.000 claims description 10
- 238000009826 distribution Methods 0.000 claims description 10
- 238000003384 imaging method Methods 0.000 claims description 7
- 210000002159 anterior chamber Anatomy 0.000 claims description 5
- 238000004891 communication Methods 0.000 description 21
- 210000004087 cornea Anatomy 0.000 description 10
- 230000008569 process Effects 0.000 description 10
- 238000012546 transfer Methods 0.000 description 10
- 238000010586 diagram Methods 0.000 description 7
- 238000005457 optimization Methods 0.000 description 5
- 210000000554 iris Anatomy 0.000 description 4
- 230000004044 response Effects 0.000 description 4
- 238000004220 aggregation Methods 0.000 description 3
- 230000002776 aggregation Effects 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 241000254158 Lampyridae Species 0.000 description 2
- 230000001815 facial effect Effects 0.000 description 2
- 239000003550 marker Substances 0.000 description 2
- 238000003860 storage Methods 0.000 description 2
- 241000282412 Homo Species 0.000 description 1
- 241001465754 Metazoa Species 0.000 description 1
- 238000009825 accumulation Methods 0.000 description 1
- 230000003190 augmentative effect Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 230000000295 complement effect Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005401 electroluminescence Methods 0.000 description 1
- 230000004424 eye movement Effects 0.000 description 1
- 210000000887 face Anatomy 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 239000011159 matrix material Substances 0.000 description 1
- 229910044991 metal oxide Inorganic materials 0.000 description 1
- 150000004706 metal oxides Chemical class 0.000 description 1
- 238000002360 preparation method Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 210000001525 retina Anatomy 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
- G06F3/013—Eye tracking input arrangements
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
- G06F3/012—Head tracking input arrangements
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/13—Edge detection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/50—Depth or shape recovery
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/60—Analysis of geometric attributes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T9/00—Image coding
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/10—Image acquisition
- G06V10/12—Details of acquisition arrangements; Constructional details thereof
- G06V10/14—Optical characteristics of the device performing the acquisition or on the illumination arrangements
- G06V10/141—Control of illumination
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/25—Determination of region of interest [ROI] or a volume of interest [VOI]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/18—Eye characteristics, e.g. of the iris
- G06V40/193—Preprocessing; Feature extraction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20212—Image combination
- G06T2207/20221—Image fusion; Image merging
Definitions
- the present invention relates to a video system, a video generating method, a video distribution method, a video generating program, and a video distribution program. particularly in the context of a video system comprising a head-mounted display and a gaze detection device.
- the calibration refers to causing a user to gaze at a specific indicator and specifying a position relationship between a position at which the specific indicator is displayed and a corneal center of the user gazing at the specific indicator.
- a gaze detection system that performs calibration to perform gaze detection can specify a point at which a user is looking.
- the present invention has been made in consideration of the above problems, and an object thereof is to provide a technology capable of accurately executing calibration for realizing gaze detection of a user wearing a head mounted display.
- an aspect of the present invention is a method comprising: measuring head rotation speed in a direction; measuring eye rotation speed in the direction; and performing calibration of a gaze detection unit when the head rotation speed and eye rotation speed is lower than a threshold.
- the present invention it is possible to provide a technology for detecting a gaze direction of a user wearing a head mounted display.
- FIG. 1 shows a schematic overview of the video system 1 according to the first embodiment.
- FIG. 2 is a block diagram illustrating the configuration of the video system 1 according to the embodiments.
- FIG. 3 shows a diagram showing the location of each part.
- FIG. 4 shows a flowchart of an eye-tracking method.
- FIG. 5 shows a physical location of a virtual camera and a lens.
- FIG. 6 shows camera images for lens shape.
- FIG. 7 shows a flow chart of the process of 3D model based pupil prediction.
- FIG. 8 shows an example of a scene image for calibration.
- FIG. 9 shows a flowchart of the process of hidden calibration.
- FIG. 10 shows a schematic overview of the video system.
- FIG. 11 shows a flowchart of the process related to communication between the head-mounted display and the cloud server.
- FIG. 12 shows a functional configuration diagram of the video system.
- FIG. 13 shows another example of a functional configuration diagram of the video system.
- FIG. 14 shows a graph showing the rotation speeds of head and eye.
- FIG. 15 shows a physical structure of the eyeball.
- FIG. 16 shows an example of a method of calibration of the ACD.
- FIG. 17 shows the refraction model of the single point calibration.
- FIG. 18 shows branches of the implicit calibration.
- FIG. 19 shows the overview of the implicit calibration.
- FIG. 20 shows a flow chart of the implicit calibration.
- FIG. 1 shows a schematic overview of the video system 1 according to the first embodiment.
- video system 1 comprises a head-mounted display 100 and a gaze detection device 200 .
- the head-mounted display 100 is used while secured to the head of the user 300 .
- a gaze detection device 200 detects a gaze direction of at least one of a right eye and a left eye of the user wearing the head mounted display 100 and specifies the user's focal point, that is, a point gazed by the user in a three-dimensional image displayed on the head mounted display.
- the gaze detection device 200 also functions as a video generation device that generates a video to be displayed by the head mounted display 100 .
- the gaze detection device 200 is a device capable of reproducing videos of stationary game machines, portable game machines, PCs, tablets, smartphones, phablets, video players, TVs, or the like, but the present invention is not limited thereto.
- the gaze detection device 200 is wirelessly or wiredly connected to the head mounted display 100 . In the example illustrated in FIG.
- the gaze detection device 200 is wirelessly connected to the head mounted display 100 .
- the wireless connection between the gaze detection device 200 and the head mounted display 100 can be realized using a known wireless communication technique such as Wi-Fi (registered trademark) or Bluetooth (registered trademark).
- Wi-Fi registered trademark
- Bluetooth registered trademark
- transfer of videos between the head mounted display 100 and the gaze detection device 200 is executed according to a standard such as Miracast (registered trademark), WiGig (registered trademark), or WHDI (registered trademark).
- Other communication techniques may be used and, for example, acoustic communication techniques or optical transmission techniques may be used.
- the head-mounted display 100 comprises a housing 150 , a fitting harness 160 , and headphones 170 .
- the housing 150 encloses an image display system, such as an image display element for presenting video images to the user 300 , and, not shown in the figure, a Wi-Fi (registered trademark) module, a Bluetooth (registered trademark) module, or other type wireless communication module.
- the head-mounted display 100 is secured to the head of the user 300 with a fitting harness 160 .
- the fitting harness 160 may be implemented with the help of, for example, belts or elastic bands.
- the headphones 170 output the audio of the video reproduced by the video generating device 200 .
- the headphones 170 do not need to be fixed to the head-mounted display 100 . Even when the head-mounted display 100 is secured with the fitting harness 160 , the user 300 may freely put on or remove the headphones 170 .
- FIG. 2 is a block diagram illustrating the configuration of the video system 1 according to the embodiments.
- the head-mounted display 100 comprises a video presentation unit 110 , an imaging unit 120 , and a communication unit 130 .
- the video presentation unit 110 presents a video to the user 300 .
- the video presentation unit 110 may, for example, be implemented as a liquid crystal monitor or an organic EL (electroluminescence) display.
- the imaging unit 120 captures images of the user's eye.
- the imaging unit 120 may, for example, be implemented as a CCD (charge-coupled device), CMOS (complementary metal oxide semiconductor) or other image sensor disposed in the housing 150 .
- CCD charge-coupled device
- CMOS complementary metal oxide semiconductor
- the communication unit 130 provides a wireless or wired connection to the video generating device 200 for information transfer between the head-mounted display 100 and the video generating device 200 . Specifically, the communication unit 130 transfers images captured by the imaging unit 120 to the video generating device 200 , and receives video from the video generating device 200 for presentation by the video presentation unit 110 .
- the communication unit 130 may be implemented as, for example, a Wi-Fi module, a Bluetooth (registered trademark) module or another wireless communication module.
- the gaze detection device 200 shown FIG. 2 will be introduced.
- the gaze detection device 200 comprises a communication unit 210 , a gaze detection unit 220 , a calibration unit 230 , and a storage unit 240 .
- the communication unit 210 provides a wireless or wired connection to the head-mounted display 100 .
- the communication unit 210 receives from the head-mounted display 100 images captured by the imaging unit 120 , and transmits video to the head-mounted display 100 .
- the gaze detection unit 220 detects a gaze of the user viewing an image displayed on the display 100 , and generates gaze data.
- the calibration unit 230 performs the calibration of the gaze detection.
- the storage unit 240 stores data for gaze detection and calibration.
- the eye-tracking with lens compensation may be a method comprising:
- the method may further comprises:
- FIG. 3 shows a schematic view of the eye-tracking with lens compensation.
- FIG. 3 shows human eye, lens, virtual camera and the head mounted display screen.
- the ray from the camera go through the standard or fresnel lens and reach the human eye.
- the gaze detection unit 220 uses the ray in order to compute eye-tracking.
- a standard or fresnel lens is provided between the camera and the human eye.
- the gaze detection unit 220 detects glints and the pupil on the image of the human eye using rays from the camera to each of the glints and the pupil. In the eye-tracking with lens compensation, the ray transfers through the lens. Therefore, the gaze detection unit 220 must compute such transfer.
- the gaze detection unit 220 may compute a ray (ray before lens) from the camera to a glint position detected from the image using intrinsic and extrinsic matrices to give the 3d ray for any 2d point (glint) on the camera image.
- the gaze detection unit 220 may apply snell's law ray tracing or use a precalculated transfer matrix in order to calculate the ray after lens.
- the gaze detection unit 220 uses this ray after lens in order to compute eye-tracking (gaze direction).
- Lens compensation may be done with polynomial fitting. Assuming (x, y) represents a pixel on the camera image, (xp, yp) represents the x-y position on the lens, and (xd, yd, zd) represents the x-y-z direction of the ray from the lens. Then for any pixel on the camera image the gaze detection unit 220 can find a ray after in pass the lens:
- (x, y) can be anything that can be directly derived from pixel coordinate, such as angle in spherical coordinates etc. Further, (xd, yd, zd) can also have alternative representation (e.g. spherical coordinates).
- FIG. 4 shows a flowchart of an eye-tracking method.
- the left shows a conventional flow, and the right shows the eye-tracking with lens compensation according to the present embodiment.
- the gaze detection unit 220 obtains eye images from a camera. Then, the gaze detection unit 220 finds glints and a pupil by conducting image processing. The gaze detection unit 220 uses intrinsic and extrinsic matrices to get rays from the camera to every glint.
- the gaze detection unit 220 transfers the rays through the lens.
- the transfer is computed matrices or polynomial fitting described above.
- the gaze detection unit 220 solves the inverse problem to find cornea center/radius.
- the gaze detection unit 220 uses intrinsic and extrinsic matrices to get a ray from camera to the pupil.
- the gaze detection unit 220 transfers this ray through the lens.
- the gaze detection unit 220 intersects this ray with cornea sphere.
- the resulting optical axis is the vector from cornea center to 3D pupil position.
- the camera optimization through lens fitting may be a method comprising:
- FIG. 5 shows a physical location of a virtual camera and a lens.
- the expected position and orientation of the camera means a lot to computing the gaze direction since the ray from the camera to the eye of the user transfers through the lens.
- FIG. 6 shows camera images for lens shape.
- the left side picture shows an expected camera image when the orientation of the camera is correct.
- the right side picture shows the camera image when the orientation of the camera is wrong.
- the lens shape (white circle) is not located in the center of the image.
- the calibration unit 230 runs numerical optimization to correct camera position and orientation. As an optimization cost function the calibration unit 230 tries to fit the observed lens to the expected lens shape.
- 3D model based prediction may be a method comprising:
- FIG. 7 shows a flow chart of the process of 3D model based pupil prediction.
- the eye-tracking system obtains an eye image by the camera. Then, based on the pupil and iris eccentricity, and glint position, and eye image from the camera, the system conducts image processing. Then, it estimates eyeball model parameters, such as position and orientation radiuses of eyeball, pupil and iris. it outputs 3D gaze point estimation. Then, it creates a 3D eyeball model from previous image frames, and it estimates pupil and iris eccentricity, and glints position from the 3D model. Then the pupil and iris eccentricity, and glint position are used for the next cycle of the image processing.
- Calibration process burdens the user with additional effort.
- the calibration is performed during the user viewing contents.
- the hidden calibration may be a method comprising:
- the calibration may be performed every time a scene changes.
- FIG. 8 shows an example of a scene image for calibration.
- the left side figure of FIG. 8 shows a scream image of conventional calibration.
- the conventional calibration before the content starts, the moving dot is displayed on the screen and the user watches the dot. Or in case recalibration is conducted, it is required to stop the content to display the moving dot again. But stopping content for calibration stresses users. To address this problem, it is better to conduct calibration without stopping content.
- the video content has a scene during a specific period of time which shows only moving object on the screen such as a logo, a firefly and a bright object.
- the user watches the moving object and the calibration unit can conduct the calibration process.
- the right side figure of FIG. 8 shows an example of a scene displayed with a firefly.
- FIG. 9 shows a flowchart of the process of hidden calibration.
- An application such as Video player
- the application sends 3D location information (3D coordinates) of the object to the eye tracking unit.
- the eye tracking unit uses the location information to calibrate in real time. If the eye tracking unit conducts the calibration, the application sends further timestamp information with the 3D location information.
- the foveated camera streaming may be a method comprising:
- a resolution of the region of interest is higher than a resolution of the outer region.
- the image may be a video, in the step of compressing the region of interest, encoding the region of interest in a first video, in the step of compressing the outer region, encoding the outer region into a second video, wherein a frame rate of the first video is higher than a frame rate of the second video.
- the foveated camera streaming also may be a method comprising:
- FIG. 10 shows a schematic overview of the video system.
- the video system comprises the head-mounted display 100 , a gaze detection device 200 and a cloud server.
- the head-mounted display 100 further comprises an external camera.
- the external camera is fixed with the housing 150 and arranged to record video images of the front direction of the user's head facing.
- the external camera records video images with full resolution for the whole world where the external camera can record.
- the video system has two image streams including high resolution images for the gazing area of the user and low resolution images for the other area.
- the images including the high resolution images and the low resolution images are sent to the cloud server by a public communication network directly from the head-mounted display 100 or via the gaze detection device 200 .
- the video system 1 can reduce the bandwidth for sending images because it sends full resolution images for only limited area (a gazing area) where the user looking at, and low resolution images for the other area, instead of sending full resolution images for the whole world where the external camera can record.
- the cloud server Based on the received two types of image information, the cloud server creates contextual information which is used for AR (Augmented Reality) or MR (Mixed Reality) display.
- the cloud server aggregates information (e.g. object identification, facial recognition, video image, etc.) to create contextual information and sends the contextual information to the head-mounted display 100 .
- FIG. 11 shows a flowchart of the process related to communication between the head-mounted display and the cloud server.
- the external camera facing outward on the head mount display takes images of the world (S 1101 ).
- the control unit splits the video images into two streams based on the eye tracking coordinates (S 1102 ).
- the control unit detects gazing point coordinates of the user based on the eye tracking coordinates, and splits the video images into a region of interest area and the other area.
- the region of interest can be obtained from the video images by splitting a certain sized area including the gazing point.
- the two video image streams are sent to the cloud server by a communication network (e.g. 5G network) (S 1103 ).
- a communication network e.g. 5G network
- images of the region of interest area are sent to the server as high resolution images
- images of the other area are sent to the server as low resolution images.
- the cloud server processes the images and adds contextual information (S 1104 ).
- the images and contextual information are sent back to the head mount display to display AR or MR images to the user (S 1105 ).
- FIG. 12 shows a functional configuration diagram of the video system.
- the head mount display and gaze detection device comprise the external camera, the control unit, the eye tracking unit, a sensing unit, communication unit and Display unit.
- the cloud server comprises a general awareness processing unit and a high-detail processing unit, and an information aggregation unit.
- the external camera obtains video images and the obtained raw video image with high resolution is inputted to the control unit.
- the eye tracking unit detects looking at point (gaze coordinates) based on eye tracking and input the gaze coordinates information to the control unit.
- the control unit determines the region of interest in each image based on the gaze coordinates. For example, the region of interest can be obtained from the video images by splitting a certain sized area including the gazing point.
- the image data of the region of interest is compressed at lower compression ratio and inputted to the communication unit.
- the communication unit also receives sensing data such as headset angle and other metadata which is obtained by the sensing unit.
- the sensing unit can be configured by GPS or geomagnetic sensor.
- the image data of the region of interest is sent to the cloud server with a higher resolution image.
- the image data outside of the region of interest is compressed at a higher compression ratio and inputted to the communication unit.
- the image data outside of the region of interest is sent to the cloud server with a lower resolution image.
- the general awareness processing unit in the cloud server receives the image data outside of “the region of interest” which is lower resolution (and the headset angle and the metadata), and performs image processing in order to identify objects in the image, such as object type and number.
- the High-detail processing unit in the cloud server receives the image data of the region of interest which is higher resolution (and the headset angle and the metadata), and performs image processing in order to identify detail things, such as facial recognition, text recognition.
- the information aggregation unit receives the identification result of the general awareness processing unit and the recognition result of the High-detail processing unit.
- the information aggregation unit aggregates received results in order to create a display image, and send the display image to the head mount display via the communication network.
- FIG. 13 shows another example of a functional configuration diagram of the video system.
- the image data of the region of interest (higher resolution) and the image data outside of the region of interest (lower resolution) are sent to the cloud server separately. But these image data can be sent with one video stream as shown in FIG. 7 - 4 .
- the control unit After obtaining the region of interest, the control unit stretches the image in order to reduce data outside of the region of interest. Then, it sends the stretched image and sensing data from the sensing unit to an unstretched unit in the cloud server. The unstretched unit un-stretches received image data and sends the un-stretched image data to the general awareness processing unit and the high-detail processing unit.
- the eye tracking calibration can be made using optokinetic response, i.e. the calibration method may comprise:
- Optokinetic response is an eye movement that occurs in response to the image movement on the retina.
- the sum of the head rotation speed and the eye rotation speed is zero (0) during the head rotations.
- the calibration unit 230 can perform the calibration of the gaze detection device 200 when the user gazes at a stable point which can be detected by detecting that the sum of the head rotation speed and the eye rotation speed is zero. That is, when the user rotates his/her head to the right, the user should rotate his/her eyes to the left to gaze at a point.
- FIG. 14 shows a graph showing the rotation speeds of head and eye.
- the dotted line shows the eye rotation speed in a direction.
- the solid line shows the inverted head rotation speed (the head rotation speed multiplied by ⁇ 1) in the direction. As shown FIG. 14 , the inverted head rotation speed almost aligns the eye rotation speed.
- the head-mounted display 100 comprises an IMU.
- the IMU can measure the head rotation speed of the user 300 .
- the gaze detection unit can measure the eye rotation speed of the user.
- the eye rotation speed can be represented by the speed of the movement of the gaze point.
- the calibration unit 230 may calculate the head rotation speeds in up-down direction and left-right direction from the measured value by the IMU.
- the calibration unit 230 may also calculate the eye rotation speed in up-down direction and left-right direction from the history of the gaze point.
- the calibration unit 230 displays a marker in a virtual space rendered in the display. The marker can be moved and be stable.
- the calibration unit 230 calculates the head rotation speeds in left-right direction and up-down direction and the eye rotation speeds in left-right direction and up-down direction.
- the calibration unit 230 may perform calibration when the sum of the head rotation speeds and the eye rotation speeds is lower than a pre-determined threshold.
- the single point calibration may be a method comprising:
- the calibration method determine a direction from a cornea center to the position of the pupil as the gaze direction.
- the calibration method it may be possible to determine a direction from a eyeball center to the position of the pupil as the gaze direction.
- the calibration method may further comprise: compensating a position of the pupil to angle against a direction from the camera to the pupil image.
- FIG. 15 shows a physical structure of the eyeball.
- the eyeball is configured with some parts including a pupil, a cornea and an anterior chamber.
- the position of the pupil may be recognized by camera image.
- anterior chamber depth ACD
- it is required to compensate the position of the pupil taking the ACD into consideration in order to improve accuracy of gaze estimation.
- the compensated pupil position is used to estimate the gaze direction.
- FIG. 16 shows an example of a method of calibration.
- the eye is looking at the calibration point which is known by the system, and the pupil is observed by a camera.
- P0 shows an intersection of a ray (observed pupil by a camera) with cornea sphere.
- P0 is the observed pupil on the camera.
- P0 is used in general gaze estimation.
- the pupil is positioned at P1 inside the cornea sphere by ACD.
- the direction from the center of the eyeball (or the center of the cornel sphere) to the center of the pupil is considered to be the gaze direction of the eye.
- the calibration may be made using the gaze direction and the known calibration point.
- the calibration unit 230 may calibrate the parameter of the gaze estimation.
- the calibration unit 230 may calibrate the ACD.
- the calibration unit 230 may calibrate the center of the cornea sphere.
- the calibration unit 230 may calibrate the angle of the camera.
- the compensation will be adjusted assuming the cornea refracts the ray. That is the pupil position is compensated taking the anterior chamber depth (ACD) and the horizontal optical to visual axis into consideration.
- ACD anterior chamber depth
- FIG. 17 shows the refraction model of the single point calibration.
- the intersection of ray (observed pupil by camera) with the cornea sphere gives P0.
- ACD a certain distance between pupil P1 and cornea sphere surface (P0).
- Direction from cornea center (or eyeball center) to P1 should direct towards known calibration point. If not, the calibration unit 230 optimizes ACD. Thus the calibration unit 230 calibrates the ACD.
- the implicit calibration may be a method for calibrating gaze detection comprising:
- the point to which the gaze point is adjusted may be the point with the highest probability of an edge occurring.
- the implicit calibration may further comprises:
- the screen image is referred to as a visual field.
- the visual field may cover both screen image in VR and outside camera image in AR. It is also important to note that we do not require the user to look at any specific targets.
- the scene content can be arbitrary. Since we have two types of images: eye images and screen images, it would be better to specify explicitly which image is referred to.
- the correlation of gaze points and images of visual field can be defined as an assumption about human behavior: In circumstances A people are likely to look at B, where A and B are something that we can automatically extract from the visual field image. Currently we use an assumption In all circumstances people are likely to look at edges.
- FIG. 18 shows branches of the implicit calibration.
- the “bias” means that the difference between estimated gaze point and actual gaze point is constant. People tend to look at points with high contrast, i.e. object edges. Therefore, in the implicit calibration using the edge accumulation includes:
- FIG. 19 shows the overview of the implicit calibration.
- the circles are the gaze points from the eye tracker (gaze detection unit 220 ).
- the stars are the actual gaze points.
- the rectangles are the accumulated regions of the visual field. We use the point with the highest amount of edges as a ground truth for calibration. Therefore, there is no need to provide any calibration point in the display field.
- FIG. 20 shows a flow chart of the implicit calibration.
- the exe tracker gaze detection unit 220
- the head-mounted display provides the image of the full visual field that the user sees.
- the calibration unit 230 uses the approximate gaze direction and the image of the full visual field, the calibration unit 230 computes statistics over time, estimates the eye-tracking parameters, and feeds back the parameters to the exe tracker. Thus, gradually the gaze direction is calibrated.
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Human Computer Interaction (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Multimedia (AREA)
- Health & Medical Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Ophthalmology & Optometry (AREA)
- Geometry (AREA)
- Eye Examination Apparatus (AREA)
- Position Input By Displaying (AREA)
- User Interface Of Digital Computer (AREA)
- Image Analysis (AREA)
Applications Claiming Priority (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2020172238 | 2020-10-12 | ||
JP2020-172238 | 2020-10-12 | ||
PCT/IB2021/059329 WO2022079585A1 (en) | 2020-10-12 | 2021-10-12 | Calibration for gaze detection |
Publications (1)
Publication Number | Publication Date |
---|---|
US20230393653A1 true US20230393653A1 (en) | 2023-12-07 |
Family
ID=81207753
Family Applications (3)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US18/248,832 Pending US20230393653A1 (en) | 2020-10-12 | 2021-10-12 | Calibration for gaze detection |
US18/248,847 Pending US20240192771A1 (en) | 2020-10-12 | 2021-10-12 | Visual line detection device and visual line detection program |
US18/248,838 Pending US20240134448A1 (en) | 2020-10-12 | 2021-10-12 | Viewpoint detection device, calibration method, and program |
Family Applications After (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US18/248,847 Pending US20240192771A1 (en) | 2020-10-12 | 2021-10-12 | Visual line detection device and visual line detection program |
US18/248,838 Pending US20240134448A1 (en) | 2020-10-12 | 2021-10-12 | Viewpoint detection device, calibration method, and program |
Country Status (3)
Country | Link |
---|---|
US (3) | US20230393653A1 (ja) |
JP (3) | JPWO2022079587A1 (ja) |
WO (3) | WO2022079584A1 (ja) |
Families Citing this family (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
KR20240020960A (ko) * | 2022-08-09 | 2024-02-16 | 삼성전자주식회사 | 시선 방향을 식별하는 전자 장치 및 그 작동 방법 |
Family Cites Families (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JPH0764709A (ja) * | 1993-08-26 | 1995-03-10 | Olympus Optical Co Ltd | 指示処理装置 |
JP2013252301A (ja) * | 2012-06-07 | 2013-12-19 | Toyota Central R&D Labs Inc | 眼球中心推定装置及びプログラム |
EP3329316B1 (en) * | 2016-03-11 | 2023-09-20 | Facebook Technologies, LLC | Corneal sphere tracking for generating an eye model |
US10976813B2 (en) * | 2016-06-13 | 2021-04-13 | Apple Inc. | Interactive motion-based eye tracking calibration |
CN109690553A (zh) * | 2016-06-29 | 2019-04-26 | 醒眸行有限公司 | 执行眼睛注视跟踪的系统和方法 |
US10820796B2 (en) * | 2017-09-08 | 2020-11-03 | Tobii Ab | Pupil radius compensation |
CN108038884B (zh) * | 2017-11-01 | 2020-12-11 | 北京七鑫易维信息技术有限公司 | 校准方法、装置、存储介质和处理器 |
-
2021
- 2021-10-12 US US18/248,832 patent/US20230393653A1/en active Pending
- 2021-10-12 US US18/248,847 patent/US20240192771A1/en active Pending
- 2021-10-12 WO PCT/IB2021/059328 patent/WO2022079584A1/ja active Application Filing
- 2021-10-12 WO PCT/IB2021/059329 patent/WO2022079585A1/en active Application Filing
- 2021-10-12 JP JP2022557220A patent/JPWO2022079587A1/ja active Pending
- 2021-10-12 JP JP2023546576A patent/JP2024514380A/ja active Pending
- 2021-10-12 US US18/248,838 patent/US20240134448A1/en active Pending
- 2021-10-12 WO PCT/IB2021/059332 patent/WO2022079587A1/ja active Application Filing
- 2021-10-12 JP JP2022557219A patent/JPWO2022079584A1/ja active Pending
Also Published As
Publication number | Publication date |
---|---|
WO2022079585A1 (en) | 2022-04-21 |
US20240134448A1 (en) | 2024-04-25 |
WO2022079587A1 (ja) | 2022-04-21 |
JPWO2022079584A1 (ja) | 2022-04-21 |
WO2022079584A1 (ja) | 2022-04-21 |
US20240192771A1 (en) | 2024-06-13 |
JPWO2022079587A1 (ja) | 2022-04-21 |
JP2024514380A (ja) | 2024-04-02 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US11703947B2 (en) | Apparatus and method for dynamic graphics rendering based on saccade detection | |
US20190235624A1 (en) | Systems and methods for predictive visual rendering | |
US10182720B2 (en) | System and method for interacting with and analyzing media on a display using eye gaze tracking | |
US20150097772A1 (en) | Gaze Signal Based on Physical Characteristics of the Eye | |
CN109791605A (zh) | 基于眼睛跟踪信息的图像区域中的自适应参数 | |
US20120200667A1 (en) | Systems and methods to facilitate interactions with virtual content | |
US20200120322A1 (en) | Image generating device, image display system, and image generating method | |
US11507184B2 (en) | Gaze tracking apparatus and systems | |
JP2018196730A (ja) | 眼の位置を監視するための方法およびシステム | |
US11983310B2 (en) | Gaze tracking apparatus and systems | |
US20210382316A1 (en) | Gaze tracking apparatus and systems | |
US20230393653A1 (en) | Calibration for gaze detection | |
CN112926523B (zh) | 基于虚拟现实的眼球追踪方法、系统 | |
US20220035449A1 (en) | Gaze tracking system and method | |
US20240031551A1 (en) | Image capturing apparatus for capturing a plurality of eyeball images, image capturing method for image capturing apparatus, and storage medium | |
US20210271075A1 (en) | Information processing apparatus, information processing method, and program | |
US20220014729A1 (en) | Information processing apparatus and image display method | |
US10915169B2 (en) | Correcting method and device for eye-tracking | |
US10962779B2 (en) | Display control device, method for controlling display control device, and storage medium | |
US11579690B2 (en) | Gaze tracking apparatus and systems | |
US11222394B2 (en) | Devices and headsets | |
US11023041B1 (en) | System and method for producing images based on gaze direction and field of view | |
US20230308753A1 (en) | Camera system for focusing on and tracking objects | |
Blignaut | The effect of real-time headbox adjustments on data quality |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
STPP | Information on status: patent application and granting procedure in general |
Free format text: APPLICATION UNDERGOING PREEXAM PROCESSING |
|
AS | Assignment |
Owner name: FOVE, INC., CALIFORNIA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:CHERNYAK, IAKOV;CHERNYAK, GRIGORY;REEL/FRAME:064596/0517 Effective date: 20230606 |