US20170140224A1 - Gaze detection device - Google Patents
Gaze detection device Download PDFInfo
- Publication number
- US20170140224A1 US20170140224A1 US15/322,413 US201415322413A US2017140224A1 US 20170140224 A1 US20170140224 A1 US 20170140224A1 US 201415322413 A US201415322413 A US 201415322413A US 2017140224 A1 US2017140224 A1 US 2017140224A1
- Authority
- US
- United States
- Prior art keywords
- user
- display
- invisible light
- eye
- head
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Abandoned
Links
Images
Classifications
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
- G02B27/02—Viewing or reading apparatus
-
- G06K9/00604—
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
- G02B27/0093—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
- G02B27/01—Head-up displays
- G02B27/017—Head mounted
- G02B27/0172—Head mounted characterised by optical features
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
- G02B27/10—Beam splitting or combining systems
- G02B27/14—Beam splitting or combining systems operating by reflection only
- G02B27/149—Beam splitting or combining systems operating by reflection only using crossed beamsplitting surfaces, e.g. cross-dichroic cubes or X-cubes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
- G06F3/013—Eye tracking input arrangements
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/18—Eye characteristics, e.g. of the iris
- G06V40/19—Sensors therefor
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/57—Mechanical or electrical details of cameras or camera modules specially adapted for being embedded in other devices
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/90—Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
-
- H04N5/247—
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/30—Transforming light or analogous information into electric information
- H04N5/33—Transforming infrared radiation
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
- G02B27/01—Head-up displays
- G02B27/0101—Head-up displays characterised by optical features
- G02B2027/0138—Head-up displays characterised by optical features comprising image capture systems, e.g. camera
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
- G02B27/01—Head-up displays
- G02B27/0179—Display position adjusting means not related to the information to be displayed
- G02B2027/0187—Display position adjusting means not related to the information to be displayed slaved to motion of at least a part of the body of the user, e.g. head, eye
Definitions
- the present invention relates to a gaze detection device.
- the invention relates to a gaze detection device used in conjunction with a head-mounted display (HMD).
- HMD head-mounted display
- Conventional gaze detection devices that estimate a user's gaze direction from images of the user's eye comprise, as illustrated in FIG. 8 , a lens 61 , an infrared emitter device 62 , a hot mirror 63 that reflects infrared light and transmits visible light, a camera 64 , and a display 65 .
- the hot mirror 63 is positioned at an angle of about 45 degrees relative to the optical axis 66 that is perpendicular to the surface of the display 65 , with the angle ⁇ between the surface of the display and the hot mirror being 45 degrees.
- the camera 64 is positioned below the optical axis 66 and is oriented perpendicular to and pointing towards the optical axis 66 .
- the infrared light emitted by the infrared emitter device 62 reflects from the eye 60 of the user, then reflects from the hot mirror 63 , and reaches the camera 64 .
- the infrared light optical path connects the camera 64 and the user's eye 60 through a right angle reflection in the hot mirror 63 .
- FIG. 8 of patent document 1 discloses a medical eye camera comprising a hot mirror that is tilted at an angle of 45 degrees. In contrast, as illustrated in FIG.
- patent document 1 discloses a configuration wherein the space wasted by the hot mirror is eliminated by making the angle y between the optical axis 66 and the dichroic mirror (hot mirror) 63 larger than 45 degrees and the angle ⁇ between the display and the optical axis smaller than 45 degrees.
- FIG. 10 Technology for another known method for gaze detection is shown in FIG. 10 , wherein a camera 64 is pointed towards an eye 60 of the user and an image of the user's eye 60 can be captured directly where the captured image is used for gaze detection.
- Patent document 3 discloses a gaze detection device that uses a half mirror.
- the half mirror in patent document 3 changes the optical path by reflecting light incident from a specific direction and transmitting light incident from directions other than the specific direction (Patent Document 3 [0015]).
- a lens is positioned in front of the user's eye
- a liquid crystal display (LCD) is positioned orthogonally to the user's gaze direction
- a half mirror is oriented at a 45 degree angle with respect to the user's gaze direction.
- Light emitted from the LCD is reflected by the half mirror, reflected again by the lens, and reaches the eye of the user after being transmitted through the half mirror.
- Light from the LCD emitted towards the gaze detection device that faces the LCD is blocked by the half mirror and therefore cannot enter the gaze detection device.
- Patent document 1 JP Application No. PH08-9205 (U.S. Pat. No. 6,018,630)
- Patent document 2 JP Application No. PH05-076497
- Patent document 3 JP Application No. 2001-108933
- Patent document 4 JP Application No. PH08-179239
- Patent document 5 JP Application No. 2014-21272
- Patent document 6 JP Application No. 2006-163383 (USP 2006/0103591)
- Patent document 7 JP Application No. P11-155152 (U.S. Pat. No. 6,611,283)
- a hot mirror is tilted at an angle of 45 degrees relative to the optical axis 66 that is perpendicular to the display 65 surface. Therefore the large space required by the hot mirror is the main reason why the size of a head-mounted display cannot be
- An existing technology to solve this problem is to positon the hot mirror at a larger angle than 45 degrees with respect to the optical axis 66 .
- the optical axis of the camera 64 that captures images of the user's eye 60 is substantially horizontal with respect to the surface of the display.
- Light emitted from the display 65 adds a noise component to the image of the user's eye used for gaze detection, and it is thus necessary to orient the camera 64 in a direction where light emitted by the display 65 does not enter the camera. For this reason, even if the angle ⁇ between the display 65 and the hot mirror 63 is made smaller than 45 degrees, the angle ⁇ still has to be considerably large. Thus, the distance between the lens and the display still cannot be sufficiently reduced and miniaturization is impossible.
- edges of the hot mirror between the user's eye and the display would create a visual disruption due to unintended reflection or refraction of light, disturbing the video presentation.
- the camera 64 can directly capture images of the user's eye. This means that the camera 64 is in the user's field of view, disrupting the user's immersion in the virtual realty world created by the head-mounted display.
- the half mirror has to be position at an angle of 45 degrees to the gaze direction of the user, wasting space and making miniaturization of a head-mounted display impossible.
- the purpose of the present invention is to provide a gaze detection device that allows a head-mounted display (HMD) to be miniaturized, or a HMD or another system employing the same.
- HMD head-mounted display
- the gaze detection device of the present invention is used in conjunction with a head-mounted display that is used while secured to the head of a user, wherein the head-mounted display comprises at least one display, at least one invisible light emitter illuminating an eye of the user with invisible light, at least one invisible light mirror that reflects the invisible light and transmits visible light, at least one filter that transmits the invisible light and does not transmit visible light, and at least one imaging device that can capture images using the invisible light, wherein at least one of the invisible light mirrors is positioned between at least one of the displays and an eye of the user and between at least one of the imaging devices and at least one of the displays, wherein at least one of the invisible light mirrors is positioned at an angle relative to the surface of at least one of the displays, wherein of the two perpendicular axes defined in the surface plane of at least one of the displays, the first rotation angle of at least one of the invisible light mirrors around one of the two axes
- one or more of the invisible light mirrors may be implemented as a combination of two invisible light mirrors, wherein one of the two invisible light mirrors is positioned between at least one of the displays and an eye of the user, and each of the first rotation angle and the second rotation angle of the invisible light mirror are set to reflection angles that direct invisible light reflected from an eye of the user towards one of the one or more imaging devices, while the other of the two invisible light mirrors is positioned between at least one display and another eye of the user, and each of the first rotation angle and the second rotation angle of the invisible light mirror are set to reflection angles that direct invisible light reflected from the user's other eye towards the one imaging device.
- the gaze analysis device may be connected to at least one imaging device, wherein one or more imaging devices transfer the information related to the captured images of an eye of the user to the gaze analysis device, and the gaze analysis device may detect the user's gaze based on the information related to the captured images of the user's eyes.
- a hot mirror may be used as the invisible light mirror.
- On or more lenses may be disposed between an eye of the user and one or more displays.
- the distances between at least one lens, at least one display, and the user's eye may be determined based on at least the user's visual acuity and the focal length of the user's eyes.
- the lens may be detachable from the device.
- the gaze detection device of the present invention may be detachable from the head-mounted display.
- At least one imaging device may be disposed at a position allowing imaging of the user's eye from a lower direction.
- the head-mounted display of the present invention comprises the aforementioned gaze detection device.
- the system of the present invention includes a head-mounted display described above, the head-mounted display comprises at least one display, the video control system comprises a gaze analysis device and a video control device connected to one or more displays, wherein the gaze analysis device transmits the user's gaze detection information to the video control device, and based on the user's gaze detection information received from the gaze analysis device, the video control device generates a video signal that is sent to one or more displays, and one or more displays display a video based on the video signal.
- the image control device may be connected to a user input device, wherein the user input device transmits user inputs to the video control device and, based on the user input, the video control device may generate appropriate image information to be transmitted to the display.
- the imaging device is positioned and oriented so as to allow the field of view of the imaging device to include at least a part of the display surface.
- the degree of freedom in selecting the angle between the invisible light mirror and the display is increased, making it possible to further miniaturize the head-mounted display. Further, since there is no need to use a smaller-sized invisible light mirror to miniaturize the device, there is no effect on the image presentation due to the edges of the invisible light mirror entering the field of view. Therefore, it is possible to miniaturize the device and maintain high video presentation quality.
- the infrared light reflects from the eye of the user, is then further reflected by the invisible light mirror, and can be delivered to an imaging device that is positioned outside of the user's field of view. That is, by adopting such a configuration, since it is possible to position the imaging device outside of the user's field of view, the imaging device does not impede the visibility of the video image for the user and the immersion of the user in the virtual reality world created by the head-mounted display can be improved.
- an imaging device installed above the eye acquires images of a user's eye from the upper direction an eyelid or eyelashes may obstruct imaging by a camera.
- an eyelid or eyelashes may obstruct imaging by a camera.
- an eyelid or eyelashes have no effect and images of an eye of the user can be reliably captured.
- the gaze detection function can be added to a head-mounted display that does not have a built-in gaze detection device. Further, by enclosing in a casing a gaze detection device with an optical configuration tailored to a specific user, there is no need to purchase whole head-mounted display units for each user. Further, by setting the distances between at least one lens, at least one display and an eye of the user on the basis of the visual acuity and focal distance of the user's eye, head-mounted displays can be tailored to accommodate each user.
- FIG. 1 is a schematic side view of the HMD according to the first embodiment of the present invention.
- FIG. 2 is an explanatory diagram of the X-axis and Y-axis of the HMD.
- FIG. 3 is a schematic plan view of a HMD according to the first embodiment of the present invention.
- FIG. 4 is a schematic plan view of a HMD according to the second embodiment of the present invention.
- FIG. 5 is a schematic plan view of a HMD according to the third embodiment of the present invention.
- FIG. 6 is a schematic side view of the HMD according to the fourth embodiment of the present invention.
- FIG. 7 is an example of a display screen according to the fourth embodiment of the present invention.
- FIG. 8 is a schematic side view of a conventional gaze detection device.
- FIG. 9 is a schematic side view of a conventional gaze detection device.
- FIG. 10 is a schematic side view of a conventional gaze detection device.
- FIG. 1 A schematic side view of the head-mounted display according to the first embodiment is shown in FIG. 1 .
- the head-mounted display 1 that can be secured to the head of a user contains a display unit 14 and the display unit 14 has an image display surface 16 .
- the gaze detection device installed in the head-mounted display comprises invisible light emitter devices 20 that illuminate the eye 10 of the user with invisible light, an invisible light mirror 13 that reflects the invisible light and transmits visible light, an imaging device 12 that can capture images with the invisible light, and a filter 15 that transmits the invisible light and does not transmit visible light.
- the gaze detection device further comprises a lens 11 that is positioned between the display unit 14 and the eye 10 of the user.
- the invisible light emitter device 20 is implemented as four infrared emitter devices 20 - 1 ⁇ 20 - 4 that are positioned at the top, bottom, left, and right edges of a lens.
- the invisible light emitter device 20 is implemented as four infrared emitter devices 20 - 1 ⁇ 20 - 4 that are positioned at the top, bottom, left, and right edges of a lens.
- the emitted infrared light can reach the imaging device 12 after being reflected from the eye 10 of the user, other arrangements of the light emitter positions are also possible, while a smaller number of light emitters, for example one, may be used.
- the imaging device 12 in FIG. 1 is positioned in the lower part of the head-mounted display 1 , but may be positioned in the upper part, sides, or elsewhere.
- the invisible light mirror 13 is a hot mirror that reflects infrared light and transmits visible light.
- the invisible light emitter device 20 is a light emitter with a wavelength outside of the infrared range
- the invisible light mirror should reflects light at the operating wavelength of the invisible light emitter device.
- a hot mirror is disposed at a position where it is between an eye of the user and a display, and between the display and the imaging device.
- a hot mirror 13 is positioned between the display 14 and the lens 11 . As shown in FIG.
- the angle ⁇ between the hot mirror 13 and the image display surface 16 of the display unit 14 is larger than 0 degrees and less than 45 degrees, preferably in the range from 0 degrees to less than 20 degrees. Smaller angle ⁇ leads to greater reduction of the head-mounted display thickness and weight. For example, the edges of the hot mirror 13 and the display 14 may be in contact.
- the angle between the hot mirror surface and the image display surface 16 of the display 14 can be defined by the rotation angle of the hot mirror 13 around the X-axis and the rotation angle of the hot mirror 13 around the Y-axis.
- the axes are defined in the plane of the image display surface 16 of the display 14 such that when the head-mounted display is mounted on the head, the X-axis points along the left-right direction and the Y-axis points along the up-down direction.
- the Z-axis points in the direction of the user.
- the angle ⁇ in FIG. 1 is the rotation angle of the hot mirror 13 around the X-axis. In the present embodiment, as shown in FIG.
- the hot mirror 13 rotation angle ⁇ around the Y-axis is 0 degrees.
- the hot mirror 13 thus reflects the infrared light emitted by the infrared emitter device 20 and reflected from the eye 10 of the user towards the imaging device 12 .
- both angles ⁇ and ⁇ are small, the wasted space can be reduced and the head-mounted display can be made smaller.
- Filter 15 is positioned between the display 14 and the imaging device 12 .
- the filter 15 is a visible light cut-off filter that transmits infrared light and blocks visible light.
- the filter 15 may be any filter that can block visible light and transmit the invisible light emitted by the invisible light emitter device 20 and reflected from the eye of the user.
- the imaging device 12 is oriented in a direction that places at least a part of the image display surface 16 in the field of view of the imaging device and the imaging device captures images of the user's eye using infrared light reflected by the hot mirror 13 . That is, the imaging device 12 is positioned in a direction that allows light emitted from the image display surface 16 of the display 14 to reach the imaging device directly, while infrared light reaches the imaging device after being reflected in the hot mirror 13 .
- the image display surface 16 of the display 14 is directed towards the eyes of the user and displays a video.
- the video may represent a computer-generated animation, a video game, a movie, etc.
- the image display surface 16 of the display 14 displays a video by emitting visible light. This emitted visible light is transmitted through the hot mirror 13 , and reaches the eye 10 of the user through the lens 11 .
- the lens 11 magnifies the image on the display 14 for viewing by the user.
- the lens can also contribute to vision correction for the user.
- the lens 11 is not essential for the operation of the gaze detection device.
- the infrared emitter devices 20 - 1 ⁇ 20 - 4 illuminate the eye 10 of the user with infrared light at a fixed illumination angle.
- the illuminating infrared light is reflected from the eye 10 of the user, passes through the lens 11 , and reaches the hot mirror 13 .
- the hot mirror 13 reflects the infrared light towards the imaging device 12 .
- the hot mirror 13 is in the field of view 21 of the user and preferably covers the entire surface of the display 14 . It is thereby possible to view the display without the edges of the hot mirror creating boundaries within the field of view 21 . However, the hot mirror 13 does not necessarily have to cover the entire surface of the display 14 .
- the imaging device 12 captures an image of the user's eye, wherein the image is formed by the infrared light reflected from the eye 10 of the user. It is possible to determine the user's gaze direction based on the captured image of the eye of the user. By reflecting the infrared light, the hot mirror 13 makes it possible for an imaging device located outside of the user's field of view 21 to acquire images of the user's eye. The user can view the video shown on the display without being obstructed by the imaging device.
- using a plurality of infrared emitter devices 20 - 1 ⁇ 20 - 4 allows the intensity of the infrared light reflected from the eye 10 of the user and reaching the imaging device 12 to be increased. It is possible to obtain sufficient received infrared light intensity with a single infrared emitter device as well. Increasing the infrared light emission intensity of the single infrared emitter device may be used to increase the intensity of the reflected infrared light reaching the imaging device 12 . However, in that case strong infrared light irradiates a particular part of the user's eye, increasing the risk of injury to the eye of the user. By using a plurality of infrared emitter devices, the relatively weak infrared irradiation does not damage the eyes of the user, while sufficient infrared light intensity reaches the imaging device 12 for gaze detection.
- the imaging device 12 Since the imaging device 12 is positioned such that at least a part of the surface of the display 14 is included in the field of view of the imaging device, visible light emitted from the image display surface 16 of the display 14 passes through the hot mirror 13 and illuminates the imaging device 12 . However, the light emitted from the display 14 is visible light, which is blocked by the visible light cut-off filter 15 , positioned between the display 14 and the imaging device 12 , preventing the emitted light from reaching the imaging device 12 . In contrast, the infrared light reflected by the hot mirror 13 passes through the visible light cut-off filter 15 and reaches the imaging device.
- the imaging device 12 may be oriented so that at least a part of the image display surface of the display 14 is included in the field of view of the imaging device, the visible light emitted by the display 14 that reaches the imaging device 12 does not give rise to noise and the received infrared light can be used for capturing images of the user's eye. Therefore, there is greater freedom in the positioning direction of the imaging device, and also greater freedom in setting the angle ⁇ between the hot mirror 13 and the display 14 . For example, the angle ⁇ may be set to 0 degrees. Accordingly, the head-mounted display can be made thinner and smaller.
- the imaging device 12 is preferably arranged at a position that allows capturing images of an eye of the user from below.
- the imaging device 12 is preferably arranged at a position that allows capturing images of an eye of the user from below.
- FIG. 4 shows a schematic top-view diagram of the configuration of the head-mounted display according to the second embodiment.
- explanations are omitted for components that are identical to those described in the first embodiment.
- the head-mounted display 1 comprises a display 14 a, a hot mirror 13 a that is positioned between the display 14 a and the right eye 10 a, and between the display 14 a and the imaging device 12 a, a lens 11 a that is positioned between the display 14 a and the user's right eye 10 a, infrared emitter devices 20 a - 1 ⁇ 20 a - 4 that illuminate the right eye 10 a of the user with infrared light, an imaging device 12 a that is oriented so that at least a part of the image display surface of the display 14 a is included in the field of view of the imaging device, and a visible light cut-off filter 15 a that is positioned between the imaging device 12 a and the display 14 a.
- the hot mirror 13 a is rotated around an X-axis defined within the image display surface 16 a plane of the display 14 a, wherein the rotation angle ⁇ (referred to as ⁇ 1 in this embodiment) between the surface of the hot mirror 13 a and the display 14 a is greater than or equal to 0 degrees and less than 45 degrees, but preferably greater than or equal to 0 degrees and less than 20 degrees.
- the angle ⁇ between the image display surface 16 a of the display 14 a and the surface of the hot mirror 13 a is 0 degrees.
- the gaze detection device further comprises a display 14 b, a hot mirror 13 b, a lens 11 b, infrared emitter devices 20 b - 1 20 b - 4 , a visible light cut-off filter 15 b, and an imaging device 12 b.
- the hot mirror 13 b is rotated around the X-axis defined within the image display surface 16 b plane of the display 14 b, wherein the rotation angle ⁇ (referred to as ⁇ 2 in this embodiment) between the hot mirror 13 b and the display 14 b is greater than or equal to 0 degrees and less than 45 degrees, but preferably greater than or equal to 0 degrees and less than 20 degrees, while the rotation angle f between the surface of the hot mirror 13 b and the image display surface 16 b of the display 14 b for a rotation of the hot mirror 13 b around the Y-axis is 0 degrees.
- the angles ⁇ 1 and ⁇ 2 do not need to be equal to each other.
- images formed by infrared light of the user's right eye 10 a and the left eye 10 b are captured by the infrared imaging devices 12 a and 12 b, respectively.
- a display 14 c and a display 14 d are located on the sides of the front part of the head-mounted display. Since the user can see the video images on the displays 14 c and 14 d in addition to the displays 14 a and 14 b that are directly facing the user, the user may gain close to a real-world experience.
- the visible light cut-off filters 15 a and 15 b are disposed between the side-mounted displays 14 c, 14 d and the imaging devices 12 a, 12 b, blocking the visible light emitted from these displays, making it possible to properly image the motion of the user's eye.
- a visible light cut-off filter that covers the lens of the imaging device may be used. It is also possible to use a plurality of visible light cut-off filters to block light from a plurality of displays.
- displays may be located along the top surface or the bottom surface of the front part of the head-mounted display 1 .
- FIG. 5 shows a schematic top view of the configuration of the head-mounted display according to the present embodiment.
- the explanation will focus on differences from the second embodiment. For the sake of simplifying the explanation of the display, only a configuration with two front displays 14 a and 14 b will be considered. Further displays may be added on the sides and at the bottom.
- the displays 14 a and 14 b may be implemented as a single display unit.
- a single common imaging device 12 is used for the left eye and the right eye.
- This single imaging device is positioned in a direction that allows at least a part of the image display surface of at least one of the displays 14 a and 14 b to be included in the field of view of the imaging device.
- two hot mirrors 13 a and 13 b are used, each of which is positioned between one of the eyes of the user and one of the two displays 14 a and 14 b.
- the angle ⁇ between the image display surface 16 a of the display 14 a and the surface of the hot mirror 13 a is the rotation angle of the hot mirror 13 a around the Y-axis of the display 14 a.
- a schematic side view of this embodiment has the same configuration as that shown in FIG. 1 .
- the rotation angle between the image display surface 16 a of the display 14 a and the surface of the hot mirror 13 a is ⁇ (referred to as ⁇ 3).
- the angles ⁇ and ⁇ 3 are set so as to reflect the infrared light reflected from the right eye of the user towards the common imaging device 12 .
- the other hot mirror 13 b is positioned between the display 14 b and the user's left eye 10 b.
- the angles ⁇ and ⁇ of this embodiment are set to predetermined values that are greater than 0.
- the angles ⁇ , ⁇ , ⁇ 3, and ⁇ 4 are not necessarily equal to each other.
- the imaging device 12 receives infrared light reflected from each of the hot mirrors 13 a and 13 b, and captures images of both the right and the left eyes 10 a and 10 b of the user.
- the imaging device is directed in a direction that allows at least a portion of the image display surface of a display to be included in the field of view of the imaging device, and the hot mirror is rotated not only around the horizontal axis of the head-mounted display, but also around the vertical axis to incline the hot mirror relative to the display, a single imaging device can be used to capture images of the gaze of both eyes. Therefore, the head-mounted display can be made more compact and manufactured at a lower cost.
- FIG. 6 shows a structure diagram of the video control system of the present embodiment.
- the figure shows a schematic side view of the head-mounted display 1 a.
- the head mounted-display 1 a further comprises a video control device 24 .
- the head-mounted display 1 a is a variation of the head-mounted display 1 described in the first embodiment.
- the gaze detection device of the head-mounted display 1 a further comprises a gaze analysis device 22 that is connected to the imaging device 12 .
- the gaze analysis device 22 is connected to a video control device 24
- the video control device 24 is connected to the display 14 .
- a user input device 25 for receiving input from the user is connected to the video control device 24 .
- These connections may be wired or wireless.
- the video control device 24 of the present embodiment is a video game machine for controlling a video game
- the user input device is a controller 25 connected to the video game machine.
- the gaze analysis device of the present embodiment is placed in a casing 23 that is detachable from the head-mounted display 1 a . Since the gaze detection device is detachable from the head-mounted display, it is possible to add the gaze detection function to an existing head-mounted display that does not include a gaze detection device.
- the selection of the lens 11 and the distances between the lens 11 , the display 14 , and the user's eye 10 are preferably determined based on the user's visual acuity and the user's focal length. Since the optical system of the gaze detection device is designed based on the user's eye and the position of the lens, an entire head-mounted display unit would need to be purchased for each user to provide a customized configuration suitable for the user, placing a large burden on the user.
- the lens 11 , and the distances between the lens 11 , the display 14 , and the user's eye 10 can be selected based on the user's eyesight and the user's focal length, which means that the user would only need to purchase the gaze detection device and mount it on a head-mounted display.
- the adjustment can be done, for example, by fixing the distance between the lens and the display, and changing the distance between the lens and the user's eye. Further, if the distance between the lens and the user's eye is fixed, the adjustment can be done by moving the display. It is also possible to fix the distance between the display and the user's eye, and perform the adjustment by moving the lens in between. Specifically, as shown in FIG.
- the lens 11 may be attached to a rail 26 that is disposed between the eye 10 of the user and the display 14 , wherein the lens can slide along the rail 26 , allowing the lens 11 position to be adjusted to suit the user.
- the same configuration can be used for the display 14 .
- the lens and the display may be configured to slide simultaneously.
- the distance between the lens 11 and the eye of the user may be significantly different for each user.
- the video control device 24 is described as being a device that is external to the head-mounted display, it is also possible to provide a device that is integrated in a head-mounted display.
- the gaze analysis device 22 has is described as being a part of the gaze detection device, it is also possible to implement it as an external device for a head-mounted display or a gaze detection device.
- the imaging device 12 transmits the image data of the captured images of the user's eye to the gaze analysis device 22 , and based on such image information, the gaze analysis device 22 analyzes the user's gaze, eye movement frequency, nystagmus, etc. In the present embodiment, the gaze analysis device 22 determines the gaze direction and speed, and transmits the gaze information to the video control device 24 . Based on the user's gaze information received from the gaze analysis device 22 , the video control device 24 generates the video information to be displayed on the display 14 . In the present embodiment, the user plays a shooting game using the head-mounted display 1 a . A video game machine 24 , functioning as a video control device, displays the video of the shooting game on the image display surface 16 of the display 14 .
- a cursor is displayed on the image display surface of the display 14 , marking the firing point in the vicinity of the user's gaze location, while the cursor moves in accordance with the gaze movement. For example, as shown in FIG. 7 , a circular cursor 32 is shown on the display in the video image 30 around the position 31 of the user's gaze. When the user's gaze location moves to the right, the cursor 32 is also moved to the right. The position and the moving speed of the cursor are calculated on the basis of the position and movement speed of the gaze location.
- the firing command is input from the controller 25 , and according to this command, the video game machine 24 generate a video that is shown on the display 14 .
- the game machine 24 may decide that a user input action has occurred.
- the gaze location can be used to control the mouse pointer position.
- the user input device 25 is a mouse.
- the image control device 24 used for displaying a movie etc., in accordance with the movement of the gaze position, may performs a control function such as scrolling the image.
- a user input device 25 is not necessarily required.
Abstract
Problem: The purpose is to reduce the size of a gaze detection device.
Solution: A gaze detection device that is used with a head-mounted display and comprises an invisible light emitter device for illuminating the user's eye with invisible light, an invisible light mirror that reflects the invisible light and transmits visible light, a filter that transmits the invisible light but blocks visible light, and an imaging device that can capture images with the invisible light, wherein the invisible light mirror is positioned between a display and the user's eye, and between the display and the imaging device at an angle between the surface of the invisible light mirror and the image display surface of the display that is greater than or equal to 0 degrees and less than 45 degrees, reflecting the invisible light emitted by the invisible light emitter device and reflected from the user's eye, the filter is positioned between the display and the imaging device, the imaging device is positioned so that at least a part of the image display surface of the display is included in the field of view of the imaging device, allowing images of the user's eye to be captured using the invisible light reflected by the invisible light mirror.
Description
- The present invention relates to a gaze detection device. In particular, the invention relates to a gaze detection device used in conjunction with a head-mounted display (HMD).
- Conventional gaze detection devices that estimate a user's gaze direction from images of the user's eye comprise, as illustrated in
FIG. 8 , alens 61, aninfrared emitter device 62, ahot mirror 63 that reflects infrared light and transmits visible light, acamera 64, and adisplay 65. Thehot mirror 63 is positioned at an angle of about 45 degrees relative to theoptical axis 66 that is perpendicular to the surface of thedisplay 65, with the angle θ between the surface of the display and the hot mirror being 45 degrees. Thecamera 64 is positioned below theoptical axis 66 and is oriented perpendicular to and pointing towards theoptical axis 66. The infrared light emitted by theinfrared emitter device 62 reflects from theeye 60 of the user, then reflects from thehot mirror 63, and reaches thecamera 64. The infrared light optical path connects thecamera 64 and the user'seye 60 through a right angle reflection in thehot mirror 63. An example of the aforementioned structure is disclosed inFIG. 8 ofpatent document 1. Further,patent document 2 discloses a medical eye camera comprising a hot mirror that is tilted at an angle of 45 degrees. In contrast, as illustrated inFIG. 9 herein,patent document 1 discloses a configuration wherein the space wasted by the hot mirror is eliminated by making the angle y between theoptical axis 66 and the dichroic mirror (hot mirror) 63 larger than 45 degrees and the angle θ between the display and the optical axis smaller than 45 degrees. - Technology for another known method for gaze detection is shown in
FIG. 10 , wherein acamera 64 is pointed towards aneye 60 of the user and an image of the user'seye 60 can be captured directly where the captured image is used for gaze detection. -
Patent document 3 discloses a gaze detection device that uses a half mirror. The half mirror inpatent document 3 changes the optical path by reflecting light incident from a specific direction and transmitting light incident from directions other than the specific direction (Patent Document 3 [0015]). In this gaze detection device, a lens is positioned in front of the user's eye, a liquid crystal display (LCD) is positioned orthogonally to the user's gaze direction, and a half mirror is oriented at a 45 degree angle with respect to the user's gaze direction. Light emitted from the LCD is reflected by the half mirror, reflected again by the lens, and reaches the eye of the user after being transmitted through the half mirror. Light from the LCD emitted towards the gaze detection device that faces the LCD is blocked by the half mirror and therefore cannot enter the gaze detection device. - Techniques for changing the video shown by the video display in a HMD in response to user's head movement are disclosed in
patent document 4. Other technologies related to HMDs or gaze detection are disclosed in patent documents 5 to 7. - Patent document 1: JP Application No. PH08-9205 (U.S. Pat. No. 6,018,630)
- Patent document 2: JP Application No. PH05-076497
- Patent document 3: JP Application No. 2001-108933
- Patent document 4: JP Application No. PH08-179239
- Patent document 5: JP Application No. 2014-21272
- Patent document 6: JP Application No. 2006-163383 (USP 2006/0103591)
- Patent document 7: JP Application No. P11-155152 (U.S. Pat. No. 6,611,283)
- In existing technology described in
FIG. 8 , a hot mirror is tilted at an angle of 45 degrees relative to theoptical axis 66 that is perpendicular to thedisplay 65 surface. Therefore the large space required by the hot mirror is the main reason why the size of a head-mounted display cannot be - An existing technology to solve this problem, as illustrated in
FIG. 9 , is to positon the hot mirror at a larger angle than 45 degrees with respect to theoptical axis 66. However, because the angle is restricted, it is impossible to achieve the desired reduction in size. In this structure, the optical axis of thecamera 64 that captures images of the user'seye 60 is substantially horizontal with respect to the surface of the display. Light emitted from thedisplay 65 adds a noise component to the image of the user's eye used for gaze detection, and it is thus necessary to orient thecamera 64 in a direction where light emitted by thedisplay 65 does not enter the camera. For this reason, even if the angle θ between thedisplay 65 and thehot mirror 63 is made smaller than 45 degrees, the angle θ still has to be considerably large. Thus, the distance between the lens and the display still cannot be sufficiently reduced and miniaturization is impossible. - If a smaller hot mirror is used to obtain a more compact device, edges of the hot mirror between the user's eye and the display would create a visual disruption due to unintended reflection or refraction of light, disturbing the video presentation.
- In a conventional configuration described in
FIG. 10 , thecamera 64 can directly capture images of the user's eye. This means that thecamera 64 is in the user's field of view, disrupting the user's immersion in the virtual realty world created by the head-mounted display. - In the configuration using a half mirror as described in
patent document 3, the half mirror has to be position at an angle of 45 degrees to the gaze direction of the user, wasting space and making miniaturization of a head-mounted display impossible. - Therefore, the purpose of the present invention is to provide a gaze detection device that allows a head-mounted display (HMD) to be miniaturized, or a HMD or another system employing the same.
- The present invention has been made in view of the above considerations, and has the following features. The gaze detection device of the present invention is used in conjunction with a head-mounted display that is used while secured to the head of a user, wherein the head-mounted display comprises at least one display, at least one invisible light emitter illuminating an eye of the user with invisible light, at least one invisible light mirror that reflects the invisible light and transmits visible light, at least one filter that transmits the invisible light and does not transmit visible light, and at least one imaging device that can capture images using the invisible light, wherein at least one of the invisible light mirrors is positioned between at least one of the displays and an eye of the user and between at least one of the imaging devices and at least one of the displays, wherein at least one of the invisible light mirrors is positioned at an angle relative to the surface of at least one of the displays, wherein of the two perpendicular axes defined in the surface plane of at least one of the displays, the first rotation angle of at least one of the invisible light mirrors around one of the two axes, and the second rotation angle of the invisible light mirror around the other of the two axes, are both larger than or equal to 0 and less than 45 degrees, such that when invisible light emitted by at least one of the invisible light emitter devices illuminating an eye of the user is reflected from an eye of the user, at least one of the filters is positioned between at least one of the displays and at least one of the imaging devices, and one or more imaging devices are oriented so that at least a part of the image display surface of at least one of the displays is in the imaging range of the imaging device, at least one of the imaging devices can capture an image of an eye of the user by using invisible light reflected in at least one of the invisible light mirrors.
- In one aspect of the present invention, one or more of the invisible light mirrors may be implemented as a combination of two invisible light mirrors, wherein one of the two invisible light mirrors is positioned between at least one of the displays and an eye of the user, and each of the first rotation angle and the second rotation angle of the invisible light mirror are set to reflection angles that direct invisible light reflected from an eye of the user towards one of the one or more imaging devices, while the other of the two invisible light mirrors is positioned between at least one display and another eye of the user, and each of the first rotation angle and the second rotation angle of the invisible light mirror are set to reflection angles that direct invisible light reflected from the user's other eye towards the one imaging device.
- Also, the gaze analysis device may be connected to at least one imaging device, wherein one or more imaging devices transfer the information related to the captured images of an eye of the user to the gaze analysis device, and the gaze analysis device may detect the user's gaze based on the information related to the captured images of the user's eyes.
- Furthermore, if the non-visible light is infrared light, a hot mirror may be used as the invisible light mirror.
- On or more lenses may be disposed between an eye of the user and one or more displays.
- The distances between at least one lens, at least one display, and the user's eye may be determined based on at least the user's visual acuity and the focal length of the user's eyes.
- The lens may be detachable from the device.
- The gaze detection device of the present invention may be detachable from the head-mounted display.
- At least one imaging device may be disposed at a position allowing imaging of the user's eye from a lower direction.
- The head-mounted display of the present invention comprises the aforementioned gaze detection device.
- Furthermore, the system of the present invention includes a head-mounted display described above, the head-mounted display comprises at least one display, the video control system comprises a gaze analysis device and a video control device connected to one or more displays, wherein the gaze analysis device transmits the user's gaze detection information to the video control device, and based on the user's gaze detection information received from the gaze analysis device, the video control device generates a video signal that is sent to one or more displays, and one or more displays display a video based on the video signal.
- Furthermore, the image control device may be connected to a user input device, wherein the user input device transmits user inputs to the video control device and, based on the user input, the video control device may generate appropriate image information to be transmitted to the display.
- In the aforementioned configuration of the present invention, the imaging device is positioned and oriented so as to allow the field of view of the imaging device to include at least a part of the display surface. The degree of freedom in selecting the angle between the invisible light mirror and the display is increased, making it possible to further miniaturize the head-mounted display. Further, since there is no need to use a smaller-sized invisible light mirror to miniaturize the device, there is no effect on the image presentation due to the edges of the invisible light mirror entering the field of view. Therefore, it is possible to miniaturize the device and maintain high video presentation quality.
- In the present invention, when the user's eye is illuminated with infrared light from an infrared emitter device, the infrared light reflects from the eye of the user, is then further reflected by the invisible light mirror, and can be delivered to an imaging device that is positioned outside of the user's field of view. That is, by adopting such a configuration, since it is possible to position the imaging device outside of the user's field of view, the imaging device does not impede the visibility of the video image for the user and the immersion of the user in the virtual reality world created by the head-mounted display can be improved.
- When the user slightly lowers an eyelid, the eyelid conceals the upper half of the eyeball. Since an imaging device installed above the eye acquires images of a user's eye from the upper direction an eyelid or eyelashes may obstruct imaging by a camera. In case the imaging device for capturing images of an eye of the user is positioned so as to capture images of an eye of the user from a lower direction, an eyelid or eyelashes have no effect and images of an eye of the user can be reliably captured.
- By enclosing the gaze detection device of the present invention in a casing that is detachable from a head-mounted display, the gaze detection function can be added to a head-mounted display that does not have a built-in gaze detection device. Further, by enclosing in a casing a gaze detection device with an optical configuration tailored to a specific user, there is no need to purchase whole head-mounted display units for each user. Further, by setting the distances between at least one lens, at least one display and an eye of the user on the basis of the visual acuity and focal distance of the user's eye, head-mounted displays can be tailored to accommodate each user.
-
FIG. 1 is a schematic side view of the HMD according to the first embodiment of the present invention. -
FIG. 2 is an explanatory diagram of the X-axis and Y-axis of the HMD. -
FIG. 3 is a schematic plan view of a HMD according to the first embodiment of the present invention. -
FIG. 4 is a schematic plan view of a HMD according to the second embodiment of the present invention. -
FIG. 5 is a schematic plan view of a HMD according to the third embodiment of the present invention. -
FIG. 6 is a schematic side view of the HMD according to the fourth embodiment of the present invention. -
FIG. 7 is an example of a display screen according to the fourth embodiment of the present invention. -
FIG. 8 is a schematic side view of a conventional gaze detection device. -
FIG. 9 is a schematic side view of a conventional gaze detection device. -
FIG. 10 is a schematic side view of a conventional gaze detection device. - A schematic side view of the head-mounted display according to the first embodiment is shown in
FIG. 1 . The head-mounteddisplay 1 that can be secured to the head of a user contains adisplay unit 14 and thedisplay unit 14 has animage display surface 16. Further, the gaze detection device installed in the head-mounted display comprises invisible light emitter devices 20 that illuminate theeye 10 of the user with invisible light, aninvisible light mirror 13 that reflects the invisible light and transmits visible light, animaging device 12 that can capture images with the invisible light, and afilter 15 that transmits the invisible light and does not transmit visible light. The gaze detection device further comprises alens 11 that is positioned between thedisplay unit 14 and theeye 10 of the user. - Although in the present embodiment infrared emitter devices 20-1˜20-4 are used as the invisible light emitter device 20, as long as the emitted light is invisible, the use of a light source with a different wavelength is possible as well. Further, as described in
FIG. 1 , in the present embodiment, the invisible light emitter device 20 is implemented as four infrared emitter devices 20-1˜20-4 that are positioned at the top, bottom, left, and right edges of a lens. However, as long as the emitted infrared light can reach theimaging device 12 after being reflected from theeye 10 of the user, other arrangements of the light emitter positions are also possible, while a smaller number of light emitters, for example one, may be used. The number of installed infrared light sources may be larger as well. Theimaging device 12 inFIG. 1 is positioned in the lower part of the head-mounteddisplay 1, but may be positioned in the upper part, sides, or elsewhere. In the present embodiment, theinvisible light mirror 13 is a hot mirror that reflects infrared light and transmits visible light. In case the invisible light emitter device 20 is a light emitter with a wavelength outside of the infrared range, the invisible light mirror should reflects light at the operating wavelength of the invisible light emitter device. A hot mirror is disposed at a position where it is between an eye of the user and a display, and between the display and the imaging device. In this embodiment, ahot mirror 13 is positioned between thedisplay 14 and thelens 11. As shown inFIG. 1 , the angle θ between thehot mirror 13 and theimage display surface 16 of the display unit 14 (the hot mirror side of the display 14) is larger than 0 degrees and less than 45 degrees, preferably in the range from 0 degrees to less than 20 degrees. Smaller angle θ leads to greater reduction of the head-mounted display thickness and weight. For example, the edges of thehot mirror 13 and thedisplay 14 may be in contact. - As shown in
FIG. 2 , the angle between the hot mirror surface and theimage display surface 16 of thedisplay 14 can be defined by the rotation angle of thehot mirror 13 around the X-axis and the rotation angle of thehot mirror 13 around the Y-axis. Here, the axes are defined in the plane of theimage display surface 16 of thedisplay 14 such that when the head-mounted display is mounted on the head, the X-axis points along the left-right direction and the Y-axis points along the up-down direction. In the present embodiment, the Z-axis points in the direction of the user. The angle θ inFIG. 1 is the rotation angle of thehot mirror 13 around the X-axis. In the present embodiment, as shown inFIG. 3 , thehot mirror 13 rotation angle α around the Y-axis is 0 degrees. Thehot mirror 13 thus reflects the infrared light emitted by the infrared emitter device 20 and reflected from theeye 10 of the user towards theimaging device 12. When both angles α and θ are small, the wasted space can be reduced and the head-mounted display can be made smaller. -
Filter 15 is positioned between thedisplay 14 and theimaging device 12. Thefilter 15 is a visible light cut-off filter that transmits infrared light and blocks visible light. Thefilter 15 may be any filter that can block visible light and transmit the invisible light emitted by the invisible light emitter device 20 and reflected from the eye of the user. Theimaging device 12 is oriented in a direction that places at least a part of theimage display surface 16 in the field of view of the imaging device and the imaging device captures images of the user's eye using infrared light reflected by thehot mirror 13. That is, theimaging device 12 is positioned in a direction that allows light emitted from theimage display surface 16 of thedisplay 14 to reach the imaging device directly, while infrared light reaches the imaging device after being reflected in thehot mirror 13. - Next, the operation according to the present embodiment will be explained. First, the user secures the head-mounted
display 1 to the head. Theimage display surface 16 of thedisplay 14 is directed towards the eyes of the user and displays a video. The video may represent a computer-generated animation, a video game, a movie, etc. Theimage display surface 16 of thedisplay 14 displays a video by emitting visible light. This emitted visible light is transmitted through thehot mirror 13, and reaches theeye 10 of the user through thelens 11. Thelens 11 magnifies the image on thedisplay 14 for viewing by the user. The lens can also contribute to vision correction for the user. Thelens 11 is not essential for the operation of the gaze detection device. - The infrared emitter devices 20-1˜20-4 illuminate the
eye 10 of the user with infrared light at a fixed illumination angle. The illuminating infrared light is reflected from theeye 10 of the user, passes through thelens 11, and reaches thehot mirror 13. Thehot mirror 13 reflects the infrared light towards theimaging device 12. In the present embodiment, thehot mirror 13 is in the field ofview 21 of the user and preferably covers the entire surface of thedisplay 14. It is thereby possible to view the display without the edges of the hot mirror creating boundaries within the field ofview 21. However, thehot mirror 13 does not necessarily have to cover the entire surface of thedisplay 14. Theimaging device 12 captures an image of the user's eye, wherein the image is formed by the infrared light reflected from theeye 10 of the user. It is possible to determine the user's gaze direction based on the captured image of the eye of the user. By reflecting the infrared light, thehot mirror 13 makes it possible for an imaging device located outside of the user's field ofview 21 to acquire images of the user's eye. The user can view the video shown on the display without being obstructed by the imaging device. - In the present embodiment, using a plurality of infrared emitter devices 20-1˜20-4 allows the intensity of the infrared light reflected from the
eye 10 of the user and reaching theimaging device 12 to be increased. It is possible to obtain sufficient received infrared light intensity with a single infrared emitter device as well. Increasing the infrared light emission intensity of the single infrared emitter device may be used to increase the intensity of the reflected infrared light reaching theimaging device 12. However, in that case strong infrared light irradiates a particular part of the user's eye, increasing the risk of injury to the eye of the user. By using a plurality of infrared emitter devices, the relatively weak infrared irradiation does not damage the eyes of the user, while sufficient infrared light intensity reaches theimaging device 12 for gaze detection. - Since the
imaging device 12 is positioned such that at least a part of the surface of thedisplay 14 is included in the field of view of the imaging device, visible light emitted from theimage display surface 16 of thedisplay 14 passes through thehot mirror 13 and illuminates theimaging device 12. However, the light emitted from thedisplay 14 is visible light, which is blocked by the visible light cut-off filter 15, positioned between thedisplay 14 and theimaging device 12, preventing the emitted light from reaching theimaging device 12. In contrast, the infrared light reflected by thehot mirror 13 passes through the visible light cut-off filter 15 and reaches the imaging device. - In existing technology, because visible light emitted from a display can enter the imaging device, noise is added to the user's gaze detection and it is therefore necessary to direct the imaging device in a direction that prevents visible light emitted by the display from entering the imaging device, preferably in a vertical direction. For this reason, the tilt angle of the hot mirror with respect to the display has to be large.
- In contrast, by employing the configuration of the present invention, even though the
imaging device 12 may be oriented so that at least a part of the image display surface of thedisplay 14 is included in the field of view of the imaging device, the visible light emitted by thedisplay 14 that reaches theimaging device 12 does not give rise to noise and the received infrared light can be used for capturing images of the user's eye. Therefore, there is greater freedom in the positioning direction of the imaging device, and also greater freedom in setting the angle θ between thehot mirror 13 and thedisplay 14. For example, the angle θ may be set to 0 degrees. Accordingly, the head-mounted display can be made thinner and smaller. - Further, as shown in
FIG. 1 , theimaging device 12 is preferably arranged at a position that allows capturing images of an eye of the user from below. When imaging the eye of the user from below, without being obstructed by the eyelid or the eyelashes, reliable capturing of images of the user's eye is possible. -
FIG. 4 shows a schematic top-view diagram of the configuration of the head-mounted display according to the second embodiment. In the following description, explanations are omitted for components that are identical to those described in the first embodiment. - The configuration of the present embodiment is similar to the configuration of the first embodiment but applied for each eye of the user. In other words, for the
right eye 10 a of the user, the head-mounteddisplay 1 comprises adisplay 14 a, ahot mirror 13 a that is positioned between thedisplay 14 a and theright eye 10 a, and between thedisplay 14 a and theimaging device 12 a, alens 11 a that is positioned between thedisplay 14 a and the user'sright eye 10 a, infrared emitter devices 20 a-1˜20 a-4 that illuminate theright eye 10 a of the user with infrared light, animaging device 12 a that is oriented so that at least a part of the image display surface of thedisplay 14 a is included in the field of view of the imaging device, and a visible light cut-off filter 15 a that is positioned between theimaging device 12 a and thedisplay 14 a. In this embodiment, as in the first embodiment, illustrated inFIG. 1 , thehot mirror 13 a is rotated around an X-axis defined within theimage display surface 16 a plane of thedisplay 14 a, wherein the rotation angle θ (referred to as θ1 in this embodiment) between the surface of thehot mirror 13 a and thedisplay 14 a is greater than or equal to 0 degrees and less than 45 degrees, but preferably greater than or equal to 0 degrees and less than 20 degrees. On the other hand, as shown inFIG. 4 , for a rotation of thehot mirror 13 a around the Y-axis, the angle α between theimage display surface 16 a of thedisplay 14 a and the surface of thehot mirror 13 a is 0 degrees. - Similar to the above, for the user's
left eye 10 b, the gaze detection device further comprises adisplay 14 b, ahot mirror 13 b, alens 11 b,infrared emitter devices 20 b-1 20 b-4, a visible light cut-off filter 15 b, and animaging device 12 b. Thehot mirror 13 b is rotated around the X-axis defined within theimage display surface 16 b plane of thedisplay 14 b, wherein the rotation angle θ (referred to as θ2 in this embodiment) between thehot mirror 13 b and thedisplay 14 b is greater than or equal to 0 degrees and less than 45 degrees, but preferably greater than or equal to 0 degrees and less than 20 degrees, while the rotation angle f between the surface of thehot mirror 13 b and theimage display surface 16 b of thedisplay 14 b for a rotation of thehot mirror 13 b around the Y-axis is 0 degrees. The angles θ1 and θ2 do not need to be equal to each other. - To apply the method described in the first embodiment to both, images formed by infrared light of the user's
right eye 10 a and theleft eye 10 b are captured by theinfrared imaging devices - In the present embodiment, a
display 14 c and adisplay 14 d are located on the sides of the front part of the head-mounted display. Since the user can see the video images on thedisplays displays filters displays imaging devices - Similarly, displays may be located along the top surface or the bottom surface of the front part of the head-mounted
display 1. -
FIG. 5 shows a schematic top view of the configuration of the head-mounted display according to the present embodiment. The explanation will focus on differences from the second embodiment. For the sake of simplifying the explanation of the display, only a configuration with twofront displays displays - In the present embodiment, a single
common imaging device 12 is used for the left eye and the right eye. This single imaging device is positioned in a direction that allows at least a part of the image display surface of at least one of thedisplays hot mirrors displays FIG. 5 , the angle α between theimage display surface 16 a of thedisplay 14 a and the surface of thehot mirror 13 a is the rotation angle of thehot mirror 13 a around the Y-axis of thedisplay 14 a. - A schematic side view of this embodiment has the same configuration as that shown in
FIG. 1 . For rotations of thehot mirror 13 a around the X-axis defined in the plane of the image display surface of thedisplay 14 a, the rotation angle between theimage display surface 16 a of thedisplay 14 a and the surface of thehot mirror 13 a is θ (referred to as θ3). The angles α and θ3 are set so as to reflect the infrared light reflected from the right eye of the user towards thecommon imaging device 12. - Similarly, the other
hot mirror 13 b is positioned between thedisplay 14 b and the user'sleft eye 10 b. The angle between the hot mirror surface and theimage display surface 16 b of thedisplay 14 b, wherein the rotation angle of thehot mirror 13 b around the Y-axis defined in the plane of the image display surface of thedisplay 14 b is β and the rotation angle of thehot mirror 13 b around the X-axis is θ(referred to as θ4), is set so as to reflect infrared light reflected from the left eye of the user towards theimaging device 12. The angles α and β of this embodiment are set to predetermined values that are greater than 0. The angles α, β, θ3, and θ4 are not necessarily equal to each other. - The
imaging device 12 receives infrared light reflected from each of thehot mirrors left eyes - Since the imaging device is directed in a direction that allows at least a portion of the image display surface of a display to be included in the field of view of the imaging device, and the hot mirror is rotated not only around the horizontal axis of the head-mounted display, but also around the vertical axis to incline the hot mirror relative to the display, a single imaging device can be used to capture images of the gaze of both eyes. Therefore, the head-mounted display can be made more compact and manufactured at a lower cost.
-
FIG. 6 shows a structure diagram of the video control system of the present embodiment. The figure shows a schematic side view of the head-mounteddisplay 1 a. - In this embodiment, the head mounted-
display 1 a further comprises avideo control device 24. The head-mounteddisplay 1 a is a variation of the head-mounteddisplay 1 described in the first embodiment. In addition to the configuration of the gaze detection device of the first embodiment, the gaze detection device of the head-mounteddisplay 1 a further comprises agaze analysis device 22 that is connected to theimaging device 12. Thegaze analysis device 22 is connected to avideo control device 24, and thevideo control device 24 is connected to thedisplay 14. Further, auser input device 25 for receiving input from the user is connected to thevideo control device 24. These connections may be wired or wireless. Thevideo control device 24 of the present embodiment is a video game machine for controlling a video game, the user input device is acontroller 25 connected to the video game machine. - The gaze analysis device of the present embodiment is placed in a
casing 23 that is detachable from the head-mounteddisplay 1 a. Since the gaze detection device is detachable from the head-mounted display, it is possible to add the gaze detection function to an existing head-mounted display that does not include a gaze detection device. - The selection of the
lens 11 and the distances between thelens 11, thedisplay 14, and the user'seye 10 are preferably determined based on the user's visual acuity and the user's focal length. Since the optical system of the gaze detection device is designed based on the user's eye and the position of the lens, an entire head-mounted display unit would need to be purchased for each user to provide a customized configuration suitable for the user, placing a large burden on the user. - With the detachable gaze detection device of the present embodiment, the
lens 11, and the distances between thelens 11, thedisplay 14, and the user'seye 10 can be selected based on the user's eyesight and the user's focal length, which means that the user would only need to purchase the gaze detection device and mount it on a head-mounted display. Thus, it is possible to provide head-mounted displays with gaze detection devices suitable for each user at a lower cost. - Further, by adjusting the respective distances between the
lens 11, thedisplay 14, and user'seye 10 appropriately, it is possible to set an optimal optical configuration for each user in the gaze detection device. The adjustment can be done, for example, by fixing the distance between the lens and the display, and changing the distance between the lens and the user's eye. Further, if the distance between the lens and the user's eye is fixed, the adjustment can be done by moving the display. It is also possible to fix the distance between the display and the user's eye, and perform the adjustment by moving the lens in between. Specifically, as shown inFIG. 6 , for example, thelens 11 may be attached to arail 26 that is disposed between theeye 10 of the user and thedisplay 14, wherein the lens can slide along therail 26, allowing thelens 11 position to be adjusted to suit the user. The same configuration can be used for thedisplay 14. Alternatively, the lens and the display may be configured to slide simultaneously. - For a head-mounted display, depending on the build of the cheekbones of the user and the shape of the user's nose, the distance between the
lens 11 and the eye of the user may be significantly different for each user. By adopting the configuration described in this embodiment, it is possible to build a head-mounted display that fits every user. Further, it is possible to use a detachable lens to configure the gaze detection device for each user by replacing the lens. - Although in the present embodiment, the
video control device 24 is described as being a device that is external to the head-mounted display, it is also possible to provide a device that is integrated in a head-mounted display. Further, although thegaze analysis device 22 has is described as being a part of the gaze detection device, it is also possible to implement it as an external device for a head-mounted display or a gaze detection device. - The
imaging device 12 transmits the image data of the captured images of the user's eye to thegaze analysis device 22, and based on such image information, thegaze analysis device 22 analyzes the user's gaze, eye movement frequency, nystagmus, etc. In the present embodiment, thegaze analysis device 22 determines the gaze direction and speed, and transmits the gaze information to thevideo control device 24. Based on the user's gaze information received from thegaze analysis device 22, thevideo control device 24 generates the video information to be displayed on thedisplay 14. In the present embodiment, the user plays a shooting game using the head-mounteddisplay 1 a. Avideo game machine 24, functioning as a video control device, displays the video of the shooting game on theimage display surface 16 of thedisplay 14. Based on the user's gaze information from the analysis by thegaze analysis device 21, a cursor is displayed on the image display surface of thedisplay 14, marking the firing point in the vicinity of the user's gaze location, while the cursor moves in accordance with the gaze movement. For example, as shown inFIG. 7 , acircular cursor 32 is shown on the display in thevideo image 30 around the position 31 of the user's gaze. When the user's gaze location moves to the right, thecursor 32 is also moved to the right. The position and the moving speed of the cursor are calculated on the basis of the position and movement speed of the gaze location. After the user moves the cursor to the desired position, the firing command is input from thecontroller 25, and according to this command, thevideo game machine 24 generate a video that is shown on thedisplay 14. Instead of the user input received through thecontroller 25, if the user keeps the gaze fixed at one location for a predetermined period of time, thegame machine 24 may decide that a user input action has occurred. - If instead of a video game machine the user is using a personal computer, the gaze location can be used to control the mouse pointer position. In this case the
user input device 25 is a mouse. Also, theimage control device 24 used for displaying a movie etc., in accordance with the movement of the gaze position, may performs a control function such as scrolling the image. In this case auser input device 25 is not necessarily required. - The above embodiments have been presented as examples for the purpose of explaining the present invention. The preset invention is not limited to these embodiments. The present invention, without departing from the gist thereof can be implemented in various forms.
-
- 1 HMD
- 10 User's eye
- 11 Lens
- 12 Imaging device
- 13 Invisible light mirror
- 14 Display
- 15 Filter
- 16 Image display surface
- 20 Invisible light emitter device
- 21 Field of view
- 22 Gaze analysis unit
- 23 Casing
- 24 Video control device
- 25 User input unit
- 26 Rail
- 30 Display surface plane
- 31 Gaze location
- 32 Cursor
- 60 User's eye
- 61 Lens
- 62 Infrared emitter device
- 63 Hot mirror
- 64 Camera
- 65 Display
- 66 Optical axis
Claims (14)
1. A gaze detection device for use with a head-mounted display that comprises at least one display and is used while secured to the head of the user, comprising:
at least one invisible light emitter device for illuminating an eye of the user with invisible light;
at least one invisible light mirrors that reflects the invisible light and transmits visible light;
at least one filter that transmits the invisible light and does not transmit visible light; and
at least one imaging device that can capture images formed by the invisible light, wherein
at least one of the invisible light mirrors is disposed both between at least one of the displays and an eye of the user, and between the display and at least one of the imaging devices,
the angle between the image display surface of the display and the surface of the invisible light mirror is defined in reference to two orthogonal axes within the plane that includes the image display surface of the display, wherein the first rotation angle of the invisible light mirror around one of the two axes and the second rotation angle of the invisible light mirror around the other axis are both set to angles greater than or equal to 0 degrees and less than 45 degrees;
the invisible light mirror reflects invisible light emitted by at least one of the invisible light emitter devices and reflected from the user's eye;
at least one of the filters is disposed between at least one of the displays and at least one of the imaging devices;
at least one of the imaging devices is oriented so as to include at least a part of the image display surface of at least one of the displays in the field of view of the imaging device, allowing the imaging device to capture an image of the user's eye using the invisible light reflected by the invisible light mirror; and
the imaging device is positioned so as to make the crossing angle between the optical axis of the imaging device reflected from the invisible light mirror, and the ocular axis that is orthogonal to at least one of the one or more displays and passes through the corneal apex, greater than zero when the user has mounted the head-mounted display.
2. The device of claim 1 , wherein at least one of the invisible light mirrors comprises two invisible light mirrors, one of which is disposed between at least one of the displays and one of the eyes of the user and oriented by setting each of the first rotation angle and the second rotation angle so as to reflect the invisible light reflected from the eye of the user towards one imaging device of the one or more imaging devices; and the other of the two invisible mirrors is disposed between at least one of the displays and the other eye of the user and oriented by setting each of the first rotation angle and the second rotation angle so as to reflect the invisible light reflected from the other eye of the user towards the same imaging device.
3. The device of claim 1 that further comprises a gaze analysis device connected to at least one imaging device, wherein one or more imaging devices capture images of the user's eye and send the image data to the gaze analysis device that detects the gaze of the user on the basis of the captured image data of the user's eyes.
4. The device of claim 1 , wherein the invisible light is infrared light and the invisible light mirror is a hot mirror.
5. The device of claim 1 that further comprises at least one lens disposed between at least one of the displays and an eye of the user.
6. The device of claim 5 , wherein the distances between at least one of the lenses, at least one of the displays and an eye of the user are set on the basis of at least the user's visual acuity and the user's focal length.
7. The device of claim 5 , wherein the lens is detachable from the gaze detection device.
8. The device of claim 1 , wherein the device is detachable from the head-mounted display.
9. The device of claim 1 , wherein at least one of the imaging devices is disposed at a location that allows the user's eye to be imaged from a lower direction.
10. A head-mounted display comprising the device of claim 1 .
11. A system that comprises
a head-mounted display of claim 10 , wherein the head-mounted display comprises at least one display; and
a video control system comprising a video control device that is connected to the gaze analysis device and at least one display, wherein
the gaze analysis device sends the user's gaze detection information to the video control device;
the video control device generates video data for at least one display based on the users gaze information received from the gaze analysis device; and
at least one display shows a video based on the video data.
12. The system of claim 11 that further comprises a user input device connected to the video control device, wherein the user input device send the user inputs to the video control device and the video control device sends to the display video data based on the user input.
13. The device of claim 5 that further comprises an adjustment unit for adjusting the distance between the lens and the eye of the user.
14. The device of claim 1 , wherein the device comprises a plurality of invisible light emitter devices.
Applications Claiming Priority (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
PCT/JP2014/067197 WO2015198477A1 (en) | 2014-06-27 | 2014-06-27 | Sight line detection device |
JPPCT/JP2014/067197 | 2014-06-27 | ||
PCT/JP2014/080254 WO2015198502A1 (en) | 2014-06-27 | 2014-11-14 | Device for detecting line of sight |
Publications (1)
Publication Number | Publication Date |
---|---|
US20170140224A1 true US20170140224A1 (en) | 2017-05-18 |
Family
ID=54937600
Family Applications (3)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US15/322,401 Active US10445573B2 (en) | 2014-06-27 | 2014-06-27 | Gaze detection device |
US15/322,413 Abandoned US20170140224A1 (en) | 2014-06-27 | 2014-11-14 | Gaze detection device |
US16/563,407 Active US10817721B2 (en) | 2014-06-27 | 2019-09-06 | Gaze detection device |
Family Applications Before (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US15/322,401 Active US10445573B2 (en) | 2014-06-27 | 2014-06-27 | Gaze detection device |
Family Applications After (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US16/563,407 Active US10817721B2 (en) | 2014-06-27 | 2019-09-06 | Gaze detection device |
Country Status (5)
Country | Link |
---|---|
US (3) | US10445573B2 (en) |
JP (2) | JP6539654B2 (en) |
KR (1) | KR20170039622A (en) |
CN (2) | CN106797423B (en) |
WO (2) | WO2015198477A1 (en) |
Cited By (25)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20180157320A1 (en) * | 2016-12-01 | 2018-06-07 | Oculus Vr, Llc | Air spaced optical assembly with integrated eye tracking |
US20180338089A1 (en) * | 2015-11-27 | 2018-11-22 | Lg Innotek Co., Ltd. | Camera Module for Both Normal Photography and Infrared Photography |
CN108919960A (en) * | 2018-07-27 | 2018-11-30 | 上海理工大学 | A kind of system showing missing image |
CN108965644A (en) * | 2018-07-27 | 2018-12-07 | 上海理工大学 | A kind of system and method for arranging showing missing image |
CN109109744A (en) * | 2018-07-27 | 2019-01-01 | 上海理工大学 | A kind of system and method for arranging showing missing image |
CN109109745A (en) * | 2018-07-27 | 2019-01-01 | 上海理工大学 | A kind of system showing missing image |
US10209773B2 (en) | 2016-04-08 | 2019-02-19 | Vizzario, Inc. | Methods and systems for obtaining, aggregating, and analyzing vision data to assess a person's vision performance |
US20190073025A1 (en) * | 2016-03-07 | 2019-03-07 | Sensomotoric Instruments Gesellschaft Fur Innovative Sensorik Mbh | Method and device for carrying out eye gaze mapping |
US10299673B2 (en) | 2008-01-14 | 2019-05-28 | Vizzario, Inc. | Method and system of enhancing ganglion cell function to improve physical performance |
US10310598B2 (en) * | 2017-01-17 | 2019-06-04 | Facebook Technologies, Llc | Varifocal head-mounted display including modular air spaced optical assembly |
US20190353909A1 (en) * | 2017-02-27 | 2019-11-21 | Alibaba Group Holding Limited | Virtual reality head-mounted apparatus |
CN110850594A (en) * | 2018-08-20 | 2020-02-28 | 余姚舜宇智能光学技术有限公司 | Head-mounted visual equipment and eyeball tracking system for same |
US10599215B2 (en) * | 2016-04-26 | 2020-03-24 | Facebook Technologies, Llc | Off-axis eye tracker |
US10867174B2 (en) | 2018-02-05 | 2020-12-15 | Samsung Electronics Co., Ltd. | System and method for tracking a focal point for a head mounted device |
US10880542B1 (en) * | 2018-12-19 | 2020-12-29 | Facebook Technologies, Llc | Near-eye optical element with embedded hot mirror |
US10983592B2 (en) * | 2018-03-13 | 2021-04-20 | Ricoh Company, Ltd. | Eyeball-tilt position device and apparatus including same |
US20210311549A1 (en) * | 2020-04-07 | 2021-10-07 | Eye Tech Digital Systems, Inc. | Compact Eye-Tracking Camera Systems and Methods |
US11150674B2 (en) * | 2018-02-13 | 2021-10-19 | Wistron Corp. | Head-mounted display device and adjusting method thereof |
US20220004001A1 (en) * | 2018-11-08 | 2022-01-06 | Lumus Ltd. | Light-Guide Display with Reflector |
US11256667B2 (en) | 2017-10-26 | 2022-02-22 | Druva Inc. | Deduplicated merged indexed object storage file system |
US11300794B2 (en) * | 2018-02-28 | 2022-04-12 | Konami Digital Entertainment Co., Ltd. | Recording medium, information processing system, and information processing method |
US11442270B2 (en) * | 2017-02-27 | 2022-09-13 | Advanced New Technologies Co., Ltd. | Virtual reality head-mounted apparatus with a partial-reflection partial-transmission wedge |
US11523039B2 (en) | 2017-02-27 | 2022-12-06 | Advanced New Technologies Co., Ltd. | Virtual reality head-mounted apparatus |
US11822081B2 (en) * | 2019-08-29 | 2023-11-21 | Apple Inc. | Optical module for head-mounted device |
US11885965B1 (en) | 2019-09-23 | 2024-01-30 | Apple Inc. | Head-mounted display and display modules thereof |
Families Citing this family (31)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US9625989B2 (en) * | 2015-04-01 | 2017-04-18 | Fove, Inc. | Head mounted display |
JP6474900B2 (en) * | 2015-08-11 | 2019-02-27 | 株式会社ソニー・インタラクティブエンタテインメント | Head mounted display |
CN205594581U (en) * | 2016-04-06 | 2016-09-21 | 北京七鑫易维信息技术有限公司 | Module is tracked to eyeball of video glasses |
WO2017186320A1 (en) * | 2016-04-29 | 2017-11-02 | Tobii Ab | Eye-tracking enabled wearable devices |
US10095307B2 (en) * | 2016-05-13 | 2018-10-09 | Google Llc | Eye tracking systems and methods for virtual reality environments |
US10877556B2 (en) * | 2016-10-21 | 2020-12-29 | Apple Inc. | Eye tracking system |
WO2018105223A1 (en) * | 2016-12-08 | 2018-06-14 | ソニー株式会社 | Imaging device |
US10545347B2 (en) * | 2017-02-23 | 2020-01-28 | Google Llc | Compact eye tracking using folded display optics |
CN106932905A (en) | 2017-02-27 | 2017-07-07 | 阿里巴巴集团控股有限公司 | Virtual reality helmet |
CN106873158A (en) * | 2017-02-27 | 2017-06-20 | 阿里巴巴集团控股有限公司 | Virtual reality helmet |
JP2018169428A (en) * | 2017-03-29 | 2018-11-01 | セイコーエプソン株式会社 | Image display device |
JP6952123B2 (en) * | 2017-05-26 | 2021-10-20 | グーグル エルエルシーGoogle LLC | Near-eye display with extended adjustment range adjustment |
US20190049722A1 (en) * | 2017-08-10 | 2019-02-14 | Htc Corporation | Head-mounted display apparatus |
US10976811B2 (en) * | 2017-08-11 | 2021-04-13 | Microsoft Technology Licensing, Llc | Eye-tracking with MEMS scanning and reflected light |
US10788677B2 (en) * | 2017-10-03 | 2020-09-29 | Facebook Technologies, Llc | Fresnel assembly for light redirection in eye tracking systems |
EP3698359A1 (en) * | 2017-10-18 | 2020-08-26 | Soapbox Labs Ltd. | Methods and systems for speech detection |
WO2019089811A1 (en) | 2017-11-01 | 2019-05-09 | Vrgineers, Inc. | Interactive augmented or virtual reality devices |
US10845594B1 (en) | 2017-12-21 | 2020-11-24 | Facebook Technologies, Llc | Prism based light redirection system for eye tracking systems |
CN109068050A (en) * | 2018-01-11 | 2018-12-21 | 王富云 | Free scope |
US10488917B2 (en) * | 2018-02-17 | 2019-11-26 | Varjo Technologies Oy | Gaze-tracking system and method of tracking user's gaze using reflective element |
CN111665622B (en) * | 2019-03-06 | 2022-07-08 | 株式会社理光 | Optical device, retina projection display device, and head-mounted display device |
CN111754728A (en) * | 2019-03-29 | 2020-10-09 | 京东方科技集团股份有限公司 | Detection assembly, detection device and detection method for fatigue driving |
US11300784B2 (en) * | 2020-02-21 | 2022-04-12 | Fotonation Limited | Multi-perspective eye acquisition |
RU2738070C1 (en) * | 2020-05-13 | 2020-12-07 | Общество С Ограниченной Ответственностью «Тотал Вижен» (Ооо "Тотал Вижен") | Device for determining coordinates of observer line in real time |
KR20210156613A (en) | 2020-06-18 | 2021-12-27 | 삼성전자주식회사 | Augmented reality glass and operating method thereof |
KR102378673B1 (en) | 2020-07-31 | 2022-03-25 | 주식회사 리버스톤미디어 | Eye contact display apparatus with stretch display panel as light source |
KR102378047B1 (en) | 2020-07-31 | 2022-03-24 | 주식회사 리버스톤미디어 | Image reflector apparatus for face and pupil capture |
KR20220111144A (en) * | 2021-02-01 | 2022-08-09 | 삼성전자주식회사 | Image processing method of head mounted display and head mounted display implementing the same |
US11914764B2 (en) * | 2021-03-26 | 2024-02-27 | Htc Corporation | Head mounted display device |
KR20220136561A (en) | 2021-03-31 | 2022-10-11 | 삼성디스플레이 주식회사 | Display device, and control method of display device |
US11630510B2 (en) | 2021-06-23 | 2023-04-18 | Huawei Technologies Co., Ltd. | System, method and storage medium for 2D on-screen user gaze estimation |
Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6091378A (en) * | 1998-06-17 | 2000-07-18 | Eye Control Technologies, Inc. | Video processing methods and apparatus for gaze point tracking |
US20030234823A1 (en) * | 2002-05-17 | 2003-12-25 | Shigemi Sato | Image processing apparatus and image processing method, and image processing program and recording medium of the same |
Family Cites Families (30)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP3149462B2 (en) | 1991-06-25 | 2001-03-26 | 株式会社島津製作所 | Eye camera |
US5515266A (en) | 1992-01-12 | 1996-05-07 | Meyer; Urs | Textile spinning machine management system |
JPH0777665A (en) | 1993-03-29 | 1995-03-20 | Canon Inc | Image display device and image photographing device for the same |
US6018630A (en) | 1994-05-17 | 2000-01-25 | Canon Kabushiki Kaisha | Camera viewfinder having a viewpoint detecting apparatus |
JPH089205A (en) * | 1994-06-17 | 1996-01-12 | Canon Inc | Picture monitor device |
JPH08179239A (en) | 1994-12-21 | 1996-07-12 | Canon Inc | Head mounted type display device and driving control method therefor |
JPH11155152A (en) | 1997-11-21 | 1999-06-08 | Canon Inc | Method and system for three-dimensional shape information input, and image input device thereof |
JP2001108933A (en) | 1999-08-03 | 2001-04-20 | Sony Corp | Head mount display device |
JP2003225207A (en) | 2002-02-04 | 2003-08-12 | Minolta Co Ltd | Visual axis detector |
JP3925222B2 (en) * | 2002-02-07 | 2007-06-06 | コニカミノルタホールディングス株式会社 | Gaze detection device |
JP4148700B2 (en) * | 2002-05-30 | 2008-09-10 | 松下電器産業株式会社 | Eye imaging device |
JP2005195821A (en) * | 2004-01-06 | 2005-07-21 | Scalar Corp | Image display apparatus |
JP2006053321A (en) * | 2004-08-11 | 2006-02-23 | Olympus Corp | Projection observation device |
JP2006163383A (en) | 2004-11-15 | 2006-06-22 | Canon Inc | Information processing apparatus and information processing method |
JP2008241822A (en) * | 2007-03-26 | 2008-10-09 | Mitsubishi Electric Corp | Image display device |
JP5216761B2 (en) * | 2007-09-26 | 2013-06-19 | パナソニック株式会社 | Beam scanning display device |
JP5237268B2 (en) * | 2007-11-21 | 2013-07-17 | パナソニック株式会社 | Display device |
CN101677762B (en) * | 2008-02-28 | 2012-08-22 | 松下电器产业株式会社 | Sight line detector and method for detecting sight line |
JP5829850B2 (en) * | 2011-03-22 | 2015-12-09 | パナソニック株式会社 | Pupil measurement device |
US9025252B2 (en) * | 2011-08-30 | 2015-05-05 | Microsoft Technology Licensing, Llc | Adjustment of a mixed reality display for inter-pupillary distance alignment |
JP2014021272A (en) * | 2012-07-18 | 2014-02-03 | Nikon Corp | Information input/output device and information input/output method |
JP2014022942A (en) * | 2012-07-18 | 2014-02-03 | Nikon Corp | Head-mounted device |
US20140055322A1 (en) * | 2012-08-21 | 2014-02-27 | Hon Hai Precision Industry Co., Ltd. | Display system and head-mounted display apparatus |
US20140118243A1 (en) * | 2012-10-25 | 2014-05-01 | University Of Seoul Industry Cooperation Foundation | Display section determination |
US20140160157A1 (en) * | 2012-12-11 | 2014-06-12 | Adam G. Poulos | People-triggered holographic reminders |
CN203608262U (en) * | 2013-09-10 | 2014-05-21 | 许玉珍 | Low-illumination-level infrared ray color photographic device |
KR20150034558A (en) * | 2013-09-26 | 2015-04-03 | 엘지전자 주식회사 | A head mounted display and the method of controlling the same |
CN106258010B (en) * | 2014-02-04 | 2019-11-22 | 弗劳恩霍夫应用研究促进协会 | 2D image dissector |
US9791924B2 (en) * | 2014-12-23 | 2017-10-17 | Mediatek Inc. | Eye tracking with mobile device in a head-mounted display |
CA2995746C (en) * | 2015-11-25 | 2020-06-30 | Google Llc | Prism-based eye tracking |
-
2014
- 2014-06-27 WO PCT/JP2014/067197 patent/WO2015198477A1/en active Application Filing
- 2014-06-27 US US15/322,401 patent/US10445573B2/en active Active
- 2014-06-27 JP JP2016528961A patent/JP6539654B2/en active Active
- 2014-06-27 CN CN201480080185.XA patent/CN106797423B/en active Active
- 2014-11-14 US US15/322,413 patent/US20170140224A1/en not_active Abandoned
- 2014-11-14 WO PCT/JP2014/080254 patent/WO2015198502A1/en active Application Filing
- 2014-11-14 JP JP2015530207A patent/JP6656921B2/en active Active
- 2014-11-14 KR KR1020167036205A patent/KR20170039622A/en active IP Right Grant
- 2014-11-14 CN CN201480080184.5A patent/CN106797422B/en active Active
-
2019
- 2019-09-06 US US16/563,407 patent/US10817721B2/en active Active
Patent Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6091378A (en) * | 1998-06-17 | 2000-07-18 | Eye Control Technologies, Inc. | Video processing methods and apparatus for gaze point tracking |
US20030234823A1 (en) * | 2002-05-17 | 2003-12-25 | Shigemi Sato | Image processing apparatus and image processing method, and image processing program and recording medium of the same |
Cited By (36)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11096570B2 (en) | 2008-01-14 | 2021-08-24 | Vizzario, Inc. | Method and system of enhancing ganglion cell function to improve physical performance |
US10299673B2 (en) | 2008-01-14 | 2019-05-28 | Vizzario, Inc. | Method and system of enhancing ganglion cell function to improve physical performance |
US20180338089A1 (en) * | 2015-11-27 | 2018-11-22 | Lg Innotek Co., Ltd. | Camera Module for Both Normal Photography and Infrared Photography |
US11212450B2 (en) * | 2015-11-27 | 2021-12-28 | Lg Innotek Co., Ltd. | Camera module for both normal photography and infrared photography |
US11579686B2 (en) * | 2016-03-07 | 2023-02-14 | Apple Inc. | Method and device for carrying out eye gaze mapping |
US20190073025A1 (en) * | 2016-03-07 | 2019-03-07 | Sensomotoric Instruments Gesellschaft Fur Innovative Sensorik Mbh | Method and device for carrying out eye gaze mapping |
US11561614B2 (en) | 2016-04-08 | 2023-01-24 | Sphairos, Inc. | Methods and systems for obtaining, aggregating, and analyzing vision data to assess a person's vision performance |
US10209773B2 (en) | 2016-04-08 | 2019-02-19 | Vizzario, Inc. | Methods and systems for obtaining, aggregating, and analyzing vision data to assess a person's vision performance |
US10599215B2 (en) * | 2016-04-26 | 2020-03-24 | Facebook Technologies, Llc | Off-axis eye tracker |
US20180157320A1 (en) * | 2016-12-01 | 2018-06-07 | Oculus Vr, Llc | Air spaced optical assembly with integrated eye tracking |
US10310598B2 (en) * | 2017-01-17 | 2019-06-04 | Facebook Technologies, Llc | Varifocal head-mounted display including modular air spaced optical assembly |
US10416766B1 (en) * | 2017-01-17 | 2019-09-17 | Facebook Technologies, Llc | Varifocal head-mounted display including modular air spaced optical assembly |
US20190353909A1 (en) * | 2017-02-27 | 2019-11-21 | Alibaba Group Holding Limited | Virtual reality head-mounted apparatus |
US11523039B2 (en) | 2017-02-27 | 2022-12-06 | Advanced New Technologies Co., Ltd. | Virtual reality head-mounted apparatus |
US10996477B2 (en) * | 2017-02-27 | 2021-05-04 | Advanced New Technologies Co., Ltd. | Virtual reality head-mounted apparatus |
US11442270B2 (en) * | 2017-02-27 | 2022-09-13 | Advanced New Technologies Co., Ltd. | Virtual reality head-mounted apparatus with a partial-reflection partial-transmission wedge |
US11599507B2 (en) | 2017-10-26 | 2023-03-07 | Druva Inc. | Deduplicated merged indexed object storage file system |
US11256667B2 (en) | 2017-10-26 | 2022-02-22 | Druva Inc. | Deduplicated merged indexed object storage file system |
US10867174B2 (en) | 2018-02-05 | 2020-12-15 | Samsung Electronics Co., Ltd. | System and method for tracking a focal point for a head mounted device |
US11150674B2 (en) * | 2018-02-13 | 2021-10-19 | Wistron Corp. | Head-mounted display device and adjusting method thereof |
TWI749172B (en) * | 2018-02-13 | 2021-12-11 | 緯創資通股份有限公司 | Head-mounted display device and adjusting method thereof |
US11300794B2 (en) * | 2018-02-28 | 2022-04-12 | Konami Digital Entertainment Co., Ltd. | Recording medium, information processing system, and information processing method |
US10983592B2 (en) * | 2018-03-13 | 2021-04-20 | Ricoh Company, Ltd. | Eyeball-tilt position device and apparatus including same |
CN109109744A (en) * | 2018-07-27 | 2019-01-01 | 上海理工大学 | A kind of system and method for arranging showing missing image |
CN109109745A (en) * | 2018-07-27 | 2019-01-01 | 上海理工大学 | A kind of system showing missing image |
CN108965644A (en) * | 2018-07-27 | 2018-12-07 | 上海理工大学 | A kind of system and method for arranging showing missing image |
CN108919960A (en) * | 2018-07-27 | 2018-11-30 | 上海理工大学 | A kind of system showing missing image |
CN110850594A (en) * | 2018-08-20 | 2020-02-28 | 余姚舜宇智能光学技术有限公司 | Head-mounted visual equipment and eyeball tracking system for same |
US20220004001A1 (en) * | 2018-11-08 | 2022-01-06 | Lumus Ltd. | Light-Guide Display with Reflector |
US11940625B2 (en) * | 2018-11-08 | 2024-03-26 | Lumus Ltd. | Light-guide display with reflector |
US10880542B1 (en) * | 2018-12-19 | 2020-12-29 | Facebook Technologies, Llc | Near-eye optical element with embedded hot mirror |
US11822081B2 (en) * | 2019-08-29 | 2023-11-21 | Apple Inc. | Optical module for head-mounted device |
US11885965B1 (en) | 2019-09-23 | 2024-01-30 | Apple Inc. | Head-mounted display and display modules thereof |
WO2021207227A1 (en) * | 2020-04-07 | 2021-10-14 | Eyetech Digital Systems, Inc. | Compact eye-tracking camera systems and methods |
US20210311549A1 (en) * | 2020-04-07 | 2021-10-07 | Eye Tech Digital Systems, Inc. | Compact Eye-Tracking Camera Systems and Methods |
US11921917B2 (en) * | 2020-04-07 | 2024-03-05 | Eyetech Digital Systems, Inc. | Compact eye-tracking camera systems and methods |
Also Published As
Publication number | Publication date |
---|---|
US10817721B2 (en) | 2020-10-27 |
CN106797423B (en) | 2019-12-06 |
CN106797422A (en) | 2017-05-31 |
US20170140223A1 (en) | 2017-05-18 |
WO2015198502A1 (en) | 2015-12-30 |
WO2015198477A1 (en) | 2015-12-30 |
US10445573B2 (en) | 2019-10-15 |
JPWO2015198477A1 (en) | 2017-04-20 |
JP6539654B2 (en) | 2019-07-03 |
US20190392210A1 (en) | 2019-12-26 |
JP6656921B2 (en) | 2020-03-04 |
KR20170039622A (en) | 2017-04-11 |
CN106797423A (en) | 2017-05-31 |
CN106797422B (en) | 2020-03-27 |
JPWO2015198502A1 (en) | 2017-04-20 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US20170140224A1 (en) | Gaze detection device | |
US10382748B2 (en) | Combining video-based and optic-based augmented reality in a near eye display | |
US10725535B2 (en) | Head-worn image display apparatus for stereoscopic microsurgery | |
JP2020515895A (en) | Operable fovea display | |
US10488917B2 (en) | Gaze-tracking system and method of tracking user's gaze using reflective element | |
WO2018100241A1 (en) | Gaze-tracking system and method of tracking user's gaze | |
KR102570188B1 (en) | vision aid device | |
JP6780502B2 (en) | Image display device and display device | |
CN114503011A (en) | Compact retinal scanning device that tracks the movement of the pupil of the eye and uses thereof | |
WO2019235059A1 (en) | Video projection system, video projection device, optical element for diffracting video display light, tool, and method for projecting video | |
JP2017092628A (en) | Display device and display device control method | |
JP6379639B2 (en) | Glasses wearing parameter measurement imaging device, glasses wearing parameter measuring imaging program | |
JP3709921B2 (en) | Image providing method and image providing apparatus for patients with eye diseases | |
JP2016116066A (en) | Display device and control method of display device | |
JP2013228573A (en) | Electronic device with rsd function, control method for electronic device with rsd function, and computer program | |
CN115857164A (en) | Near-to-eye display device and near-to-eye display device visual angle adjusting method | |
CN116437180A (en) | Display apparatus, viewfinder apparatus, and image pickup apparatus | |
TWM575869U (en) | Retinal scan imaging device |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: FOVE, INC., CALIFORNIA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:WILSON, LOCHLAINN;CHOU, BAKUI;SIGNING DATES FROM 20170110 TO 20170126;REEL/FRAME:041481/0384 |
|
STCB | Information on status: application discontinuation |
Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION |