WO2017062268A1 - Eye gaze responsive virtual reality headset - Google Patents
Eye gaze responsive virtual reality headset Download PDFInfo
- Publication number
- WO2017062268A1 WO2017062268A1 PCT/US2016/054648 US2016054648W WO2017062268A1 WO 2017062268 A1 WO2017062268 A1 WO 2017062268A1 US 2016054648 W US2016054648 W US 2016054648W WO 2017062268 A1 WO2017062268 A1 WO 2017062268A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- user
- eyes
- displays
- headset
- support arm
- Prior art date
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/30—Image reproducers
- H04N13/332—Displays for viewing with the aid of special glasses or head-mounted displays [HMD]
- H04N13/344—Displays for viewing with the aid of special glasses or head-mounted displays [HMD] with head-mounted left-right displays
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
- G02B27/0093—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
- G02B27/01—Head-up displays
- G02B27/017—Head mounted
- G02B27/0172—Head mounted characterised by optical features
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
- G02B27/01—Head-up displays
- G02B27/017—Head mounted
- G02B27/0176—Head mounted characterised by mechanical features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/002—Specific input/output arrangements not covered by G06F3/01 - G06F3/16
- G06F3/005—Input arrangements through a video camera
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
- G06F3/012—Head tracking input arrangements
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
- G06F3/013—Eye tracking input arrangements
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/30—Image reproducers
- H04N13/363—Image reproducers using image projection screens
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/30—Image reproducers
- H04N13/366—Image reproducers using viewer tracking
- H04N13/383—Image reproducers using viewer tracking for tracking with gaze detection, i.e. detecting the lines of sight of the viewer's eyes
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/30—Image reproducers
- H04N13/398—Synchronisation thereof; Control thereof
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
- G02B27/01—Head-up displays
- G02B27/0101—Head-up displays characterised by optical features
- G02B2027/0132—Head-up displays characterised by optical features comprising binocular systems
- G02B2027/0134—Head-up displays characterised by optical features comprising binocular systems of stereoscopic type
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
- G02B27/01—Head-up displays
- G02B27/0101—Head-up displays characterised by optical features
- G02B2027/0138—Head-up displays characterised by optical features comprising image capture systems, e.g. camera
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
- G02B27/01—Head-up displays
- G02B27/0101—Head-up displays characterised by optical features
- G02B2027/014—Head-up displays characterised by optical features comprising information/image processing systems
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
- G02B27/01—Head-up displays
- G02B27/0149—Head-up displays characterised by mechanical features
- G02B2027/0154—Head-up displays characterised by mechanical features with movable elements
- G02B2027/0159—Head-up displays characterised by mechanical features with movable elements with mechanical means other than scaning means for positioning the whole image
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
- G02B27/01—Head-up displays
- G02B27/0179—Display position adjusting means not related to the information to be displayed
- G02B2027/0187—Display position adjusting means not related to the information to be displayed slaved to motion of at least a part of the body of the user, e.g. head, eye
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N2213/00—Details of stereoscopic systems
- H04N2213/001—Constructional or mechanical details
Definitions
- the present invention relates to a head-mounted display device and a control method for the head-mounted display device.
- the present state of the art includes a head-mounted display (“HMD") device that is mounted on the user's head and includes one or more display devices for presenting stereoscopic images to a user's left eye and a user's right eye so that the user perceives images in three dimensions.
- the images in the HMD can be related to a virtual or real world scene and the images are related to the user's head's orientation so that the user feels immersed in a scene presented by the display.
- the display devices in the HMD device may be liquid crystal displays and a light source that guides the generated image light to the eyes through a projection optical system and a light guide plate for proper vision at small distance.
- the resolution of the images presented to the eye is limited by the capacity of the displays.
- the higher resolution of a display presents an improved image in terms of clarity and definition.
- the angle of view of systems of the present state of the art is determined by the size of the displays in front of the eyes. If the display in front of the eyes is increased, then the angle of view is also increased.
- Systems of the present state of the art do not allow the user to move their eyes left and right outside the viewing cone, which is determined by the angle of view of the displays. Instead, the user changes his or her perspective in the virtual or real world by rotating his head while keeping his eyes generally straight ahead focused on the displays.
- the present invention provides, generally, a wearable device for viewing images, having displays that move in response to the movement of a user's (wearer's) eye movement.
- the user's eye movement is detected, sensed, measured, or otherwise captured, and processed.
- the results of such processing are used to move the displays to correspond to the eye movement.
- a wearable device for a user's head presents images to a user's left eye and a user's right eye.
- the images presented to the user are related to a virtual world, an animation, digital images, or to recorded images of a real- world or other recording, and the images are stereoscopic so that world is perceived by the user as being three-dimensional.
- the images are presented to the left eye and the right eye through a left display and a right display, with each display being adjustable and movable, either together or independently, and the displays are oriented in the direction of the left eye and the right eye as the eyes move.
- the displays move and orient in response to eye movement.
- the system records or otherwise detects the movement the user's eyes though one or more cameras or other sensors focused on the eyes, and uses the imagery captured by the camera to calculate data related to the relative position of the user's eyes.
- the displays are attached to a support that is moved via motors that are controlled by a controller.
- the data related to movement of the user's eyes is used to position the displays.
- the data related to movement of the user's eyes is also used to control the image displayed on the displays. Therefore, the effective field of view of the headset is increased without deterioration of resolution of the image presented to the user. This allows the user to more naturally and comfortably view the virtual world.
- a user can gaze around a virtual world using eye movement, as in the real world.
- a user's gaze is detected through one or more cameras, or other sensors or detectors, viewing the user's eyes and through software that detects the movement of parts of the eyes, e.g. the pupils of the eyes.
- the system of the present invention includes displays that are in the field of view of the user's eyes. The displays are moved and repositioned as the user's eyes move so that the displays are constantly positioned in front of the user's eyes.
- the system of the present invention also causes the image displayed to the user to change depending on the movement of the user's eyes so that the displayed image is related to the field of view commensurate with the state and positioning of the user's eyes.
- Figures 1A and IB illustrate a first embodiment of a headset according in an upward and downward position.
- Figure 2 illustrates the first embodiment of the headset.
- Figure 3 is a flow diagram illustrating a process for positioning displays based on tracking of the user's eyes.
- Figure 4 illustrates an alternate embodiment of the headset including lenses.
- Figures 5A and 5B illustrate displays showing an image in two different orientations based on a user's eyes.
- Figure 6 is a schematic drawing of the headset of Figures 1A, IB, and
- Figure 7 is a schematic drawing of a headset according to another embodiment.
- a system 100 for displaying images to a user.
- the system 100 includes a headset 11 that can be fastened to a user's head or otherwise attached to a user.
- a strap, multiple straps, or a helmet arrangement can be used to attach the headset 11 to the user.
- the system 100 includes two displays 1, 2 that are positioned in front of a user's eyes 9, 10.
- the images provided by the displays 1, 2 can include any type of image, such as a video game, recorded images, live images, film, three-dimensional virtual world, or other type of image.
- the displays 1, 2 are attached to a display support arm 3.
- the display support arm 3 is illustrated as forked-type bar in the drawings. As shown in Figures 1A, and IB, a crossbar 3a is provided between separate prongs of the display support bar 3. In Figure 2, the display support arm 3 lacks any such crossbar 3a.
- the display support arm 3 could include multiple articulated joints and a plurality of arms.
- the displays 1, 2 are each attached to a separately articulated display support arm, and the displays 1, 2, are independently positioned from each other.
- the display support arm 3 is attached to a pitch rotating motor 4, and the display support arm 3 is configured to be manipulated by the pitch rotating motor 4.
- the pitch rotating motor 4 moves the display support arm 3 in three dimensions to move the displays 1, 2 with respect to the user's eyes 9, 10.
- the pitch rotating motor 4 is also attached to a yaw rotating motor 5 so that the pitch motor 4 and the yaw rotating motor 5 control the position of the displays 1, 2 in front of a user's eyes 9, 10.
- Other types of motors can be used, as long as the motors have sufficient a sufficient degree of motion to move the display support arm 3.
- Actuation of the pitch rotating motor 4 and the yaw rotating motor 5 is controlled by a processor 7 within the headset 11.
- the processor 7 sends signals to the motors 4, 5 to drive the motors 4, 5 and move the display support arm 3 in multiple directions.
- the system 100 also includes a camera 6 for tracking the user's eyes 9,
- the camera 6 is fixed within the headset 11 relatively close to the user's eyes 9, 10 such that the camera 6 has an unobstructed line of sight to the user's eyes 9, 10. Data from the camera 6 is transmitted to the processor 7.
- the camera 6 preferably includes a pupil tracking lens that specifically locates and maps movement of the eye.
- the camera 6 can track rotation of the eyes, position of the pupil, or other characteristics of the eye.
- the camera 6 has a very high sampling rate, preferably of at least 30 Hz, more preferably between 50-350 Hz, and most preferably of at least 1000 Hz. Although only one camera 6 is shown in the figures, one of ordinary skill in the art would recognize that multiple cameras 6 can be used.
- the camera 6 is an active light projecting infrared camera system focused on the user's eyes 9, 10.
- the camera 6 acquires data comprising an image sequence depicting the pupil of the eye in high contract.
- Sequential images are analyzed through software in the processor 7 to determine a movement vector that maps a deviation of a center of the pupil from a reference point.
- This reference point may be the center of the image or it may be a point detected through a calibration procedure related to an orthogonal, straight forward, gaze of the user with a reference point in the image.
- the user may be prompted to look straight ahead and press a button, i.e. a key on a keyboard, to relate in software running in processor 7 the current image acquired with the orthogonal gaze.
- the movement vector then is used to provide motion input commands to the motors 4, 5, and/or to adjust the images provided to the displays 1,2.
- Data from the camera 6 is processed by the processor 7 and is used to drive the motors 4, 5 so that the displays 1, 2 remain in front of the user's eyes 9, 10 as the user moves their eyes 9, 10.
- the processor 7 sends data related to the eye movement to the processor 7, which then quickly processes this data to provide instructions to the motors 4, 5 to move the display arms 3.
- sensors or detectors may be used to measure, track or otherwise determine the position of a user's eyes, such as IR (infra-red) sensors, optical sensors, beam sensors, or laser sensors.
- the headset 11 also includes an accelerometer 12 or other sensor that detects and generates data related the orientation of the headset 11.
- the accelerometer 12 can detect motion, orientation, and/or positioning of the headset 11.
- Data from the accelerometer 12 is transmitted to the processor 7, and the processor 7 transmits both (1) the data related to the movement of the user's eyes 9, 10, and (2) the data related to the orientation of the headset 11 to a central processor 14 through transmitter/receiver units 8, 13.
- the processor 7 also receives image data related to a virtual or real world from the central processor 14.
- the transmitter/receiver unit 13 and the central processor 14 can be in the form of a central processing unit (CPU).
- Figures 1A and IB show the displays 1, 2 are moved from (1) a first position in Figure 1A when the user's eyes 9, 10 are moved upward to (2) a second position shown in Figure IB when the user's eyes 9, 10 are moved downward.
- the camera 6 sends input related to data gathered during movement of the user's eyes 9, 10 to the processor 7, and the processor 7 then uses this data to provide input to the motors 4, 5.
- the motors 4, 5 then drive the display support arm 3 to different positions based on the data generated by the camera 6.
- the processor 7 also provides images to the displays 1, 2 based on the position of the user's eyes 9, 10. For example, if the user is looking downward, the processor 7 sends images to the displays 1, 2 for a downward view.
- Figures la and lb show a user moving their eyes up and down
- the system 100 including a full range of motion, to provide responsive and appropriate images to the displays 1, 2.
- Figure 5A illustrates the displays 1, 2 showing an image in a first orientation
- Figure 5B illustrates the displays 1, 2 showing the image in a second orientation
- the displays 1, 2 show an image of two characters in a first orientation in which the entire figures are visible.
- Figure 5A corresponds to a first condition in which a user is looking straight forward.
- Figure 5B corresponds to a second condition in which a user moves their eyes downward compared to the position depicted in Figure 5A.
- Figure 5B only the lower halves of the figures are visible.
- Figures 5A and 5B illustrate the concept that as the user moves their eyes, the image is adjusted to correspond to the position of the user's eyes.
- a flow diagram of a method 300 for controlling the system 100 is provided in Figure 3.
- input from the accelerometer 310 and input from the camera 320 are used in data processing 330 to control the display support arms 340.
- the display support arm moves the displays in front of a user's eyes based on movement and orientation of the user's eyes.
- Input from the accelerometer 310 and input from the camera 320 are also used by the controller in data processing 330 to generate image data 350 for the displays related to the orientation of headset and related to the orientation of the user's eyes, and the displays show images 360 based on the data.
- the flow diagram of the method 300 generally discloses two input steps, both the accelerometer input 310 and pupil/eye tracking step 320.
- the data from both of these steps 310, 320 are processed in data processing step 330.
- the display arms and displays are moved according to the data input in step 340.
- the image generation step 350 determines the appropriate type of view that will be provided to the displays.
- the displays are instructed to provide a particular display in step 360 based on the previous steps 310, 320, and 330.
- Figure 4 shows an alternative embodiment of the system 200 including lenses 15, 16 positioned between the user's eyes 9, 10 and the displays 1, 2. All of the components with the same reference numerals in Figure 4 are the same as previously discussed with respect to Figures 1A, IB, and 2.
- the lenses 15, 16 are preferably convex and are preferably arranged on a secondary support 23 connected to the display support arm 3.
- the secondary support 23 is arranged between the user's eyes 9, 10 and the displays 1, 2.
- the convex lenses 15, 16 focus light from the displays 1, 2 to make viewing the displays 1, 2 at smaller distances possible.
- the head mounted display device of the system 200 places the displays 1, 2 at a distance from the user's eyes 9, 10 that is less than a minimum that a human eye can focus.
- the system 200 essentially provides very strong reading/magnifying glasses in the form of the convex lenses 15, 16 with, for example, refractive power of 20 diopters or more.
- the convex lenses 15, 16 adjust the images provided by the displays 1, 2.
- the convex lenses 15, 16 are physically connected to displays 1, 2 so that they move with displays, 1, 2 such that the images of the displays 1, 2 can be orientated and positioned for the user's eyes 9, 10.
- the displays 1, 2 effectively provide an increased field of view while the resolution of the image displayed to the user is not reduced.
- the system 100 also allows a more natural and comfortable viewing of the virtual world because the user uses natural movement of the user's eyes as the user's eyes are used to manipulate the virtual world view instead of requiring the user to move their entire head and neck to change the view of the virtual world.
- Other embodiments can include multiples cameras to track a user's eyes, and may include a myoelectric detector 17, such as shown in Figure 2.
- the displays 1, 2 can be stationary, and light refracting lenses are moved, instead of the displays 1, 2, as the user moves their eyes 9, 10.
- the two displays 1, 2 are combined in one display that is moved according to the movements of the user's eyes 9, 10.
- Figure 6 illustrates a schematic view of the system 100 of Figures 1A
- Figure 6 provides a simplified representation of the displays 1, 2, the display support arm 3, the motors 4 and 5 arranged in a single housing, the processor 7, and the camera 6 facing towards the user's eyes 9, 10.
- one or more projectors 20 are provided that transmit images to a display surface, generically identified as element 1'.
- the display surface 1' can include a surface on an interior of a helmetlike headset 11, screen, glass, or other surface.
- the projector 20 transmits one or more beams 20' of images to the display surface 1'
- the beam 20' can include a set of images, i.e. one image for a left eye and one image for a right eye, that together provide a stereoscopic image for the user's eyes 9, 10.
- the projector 20 is motorized and can change a position of the beam 20' based on movement of the user's head and/or position of the user's eyes 9, 10. Tracking eye movement, processing the images, and positioning the images displays based on a user's eye positioning can be accomplished in a manner similar to what has been described.
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Human Computer Interaction (AREA)
- Signal Processing (AREA)
- Optics & Photonics (AREA)
Abstract
A headset for providing images to a user is provided. The headset is configured to attach to a user's head and includes displays positioned adjacent to a user's eyes. A display support arm is connected to the displays. A motor is connected to the display support arm and drives the display support arm between multiple positions. A camera is positioned adjacent to the user's eyes and records the user's eyes. A processor receives data from the camera regarding a position and movement of the user's eyes. The processor uses the data from the camera to drive the motor and to position the display support arm with respect to the user's eyes such that the displays maintain a position adjacent to the user's eyes.
Description
EYE GAZE RESPONSIVE VIRTUAL REALITY HEADSET
CROSS-REFERENCE TO RELATED APPLICATIONS
[0001] This application claims the benefit of U.S. provisional patent application 62/236,964, filed on October 4, 2015, the entire contents of which is incorporated by reference as if fully set forth herein.
FIELD OF THE INVENTION
[0002] The present invention relates to a head-mounted display device and a control method for the head-mounted display device.
BACKGROUND
[0003] The present state of the art includes a head-mounted display ("HMD") device that is mounted on the user's head and includes one or more display devices for presenting stereoscopic images to a user's left eye and a user's right eye so that the user perceives images in three dimensions. The images in the HMD can be related to a virtual or real world scene and the images are related to the user's head's orientation so that the user feels immersed in a scene presented by the display. The display devices in the HMD device may be liquid crystal displays and a light source that guides the generated image light to the eyes through a projection optical system and a light guide plate for proper vision at small distance.
[0004] In the HMD of the present state of the art, the resolution of the images presented to the eye is limited by the capacity of the displays. The higher resolution of a display presents an improved image in terms of clarity and definition. The angle of view of systems of the present state of the art is determined by the size of the displays in front of the eyes. If the display in front of the eyes is increased, then the angle of view is also increased. Systems of the present state of the art do not allow the user to move their eyes left and right outside the viewing cone, which is determined by the angle of view of the displays. Instead, the user changes his or her
perspective in the virtual or real world by rotating his head while keeping his eyes generally straight ahead focused on the displays. The need to move one's head for moderate changes in perspective may place an unnecessary burden on the user's neck and shoulder muscles. In addition to this ergonomic problem, a user's natural tendency is to change their view by slightly adjusting the direction of his eyes. Therefore, the method of changing the direction of view by changing the angle of the entire head as is required with current headsets is not natural and can be strenuous on the user.
[0005] It would be desirable to provide a virtual reality headset that allows a user to explore a corresponding virtual world by allowing minor eye movement by the user to adjust the corresponding view in the virtual world.
SUMMARY OF THE INVENTION
[0006] The present invention provides, generally, a wearable device for viewing images, having displays that move in response to the movement of a user's (wearer's) eye movement. The user's eye movement is detected, sensed, measured, or otherwise captured, and processed. The results of such processing are used to move the displays to correspond to the eye movement.
[0007] A wearable device for a user's head is disclosed that presents images to a user's left eye and a user's right eye. The images presented to the user are related to a virtual world, an animation, digital images, or to recorded images of a real- world or other recording, and the images are stereoscopic so that world is perceived by the user as being three-dimensional. In one embodiment, the images are presented to the left eye and the right eye through a left display and a right display, with each display being adjustable and movable, either together or independently, and the displays are oriented in the direction of the left eye and the right eye as the eyes move.
[0008] Generally, the displays move and orient in response to eye movement.
The system records or otherwise detects the movement the user's eyes though one
or more cameras or other sensors focused on the eyes, and uses the imagery captured by the camera to calculate data related to the relative position of the user's eyes. The displays are attached to a support that is moved via motors that are controlled by a controller. The data related to movement of the user's eyes is used to position the displays. The data related to movement of the user's eyes is also used to control the image displayed on the displays. Therefore, the effective field of view of the headset is increased without deterioration of resolution of the image presented to the user. This allows the user to more naturally and comfortably view the virtual world. A user can gaze around a virtual world using eye movement, as in the real world.
[0009] In one embodiment, a user's gaze is detected through one or more cameras, or other sensors or detectors, viewing the user's eyes and through software that detects the movement of parts of the eyes, e.g. the pupils of the eyes. The system of the present invention includes displays that are in the field of view of the user's eyes. The displays are moved and repositioned as the user's eyes move so that the displays are constantly positioned in front of the user's eyes. The system of the present invention also causes the image displayed to the user to change depending on the movement of the user's eyes so that the displayed image is related to the field of view commensurate with the state and positioning of the user's eyes.
DESCRIPTION OF THE DRAWINGS
[0010] The foregoing Summary and the following detailed description will be better understood when read in conjunction with the appended drawings, which illustrate a preferred embodiment of the invention. In the drawings:
[0011] Figures 1A and IB illustrate a first embodiment of a headset according in an upward and downward position.
[0012] Figure 2 illustrates the first embodiment of the headset.
[0013] Figure 3 is a flow diagram illustrating a process for positioning displays based on tracking of the user's eyes.
[0014] Figure 4 illustrates an alternate embodiment of the headset including lenses.
[0015] Figures 5A and 5B illustrate displays showing an image in two different orientations based on a user's eyes.
[0016] Figure 6 is a schematic drawing of the headset of Figures 1A, IB, and
2.
[0017] Figure 7 is a schematic drawing of a headset according to another embodiment.
DETAILED DESCRIPTION
[0018] Certain terminology is used in the following description for convenience only and is not limiting. The words "front," "rear," "upper" and "lower" designate directions in the drawings to which reference is made. The words "inwardly" and "outwardly" refer to directions toward and away from the parts referenced in the drawings. "Axially" refers to a direction along the axis of a shaft. A reference to a list of items that are cited as "at least one of a, b, or c" (where a, b, and c represent the items being listed) means any single one of the items a, b, or c, or combinations thereof. The terminology includes the words specifically noted above, derivatives thereof and words of similar import.
[0019] As shown in Figures 1A, IB, and 2, a system 100 is provided for displaying images to a user. The system 100 includes a headset 11 that can be fastened to a user's head or otherwise attached to a user. For example, a strap, multiple straps, or a helmet arrangement can be used to attach the headset 11 to the user. The system 100 includes two displays 1, 2 that are positioned in front of a user's eyes 9, 10. The images provided by the displays 1, 2 can include any type of image, such as a video game, recorded images, live images, film, three-dimensional virtual world, or other type of image.
[0020] The displays 1, 2 are attached to a display support arm 3. The display support arm 3 is illustrated as forked-type bar in the drawings. As shown in Figures 1A, and IB, a crossbar 3a is provided between separate prongs of the
display support bar 3. In Figure 2, the display support arm 3 lacks any such crossbar 3a. One of ordinary skill in the art will recognize from the present disclosure that alternative configurations of the display support arm 3 can be used. For example, the display support arm 3 could include multiple articulated joints and a plurality of arms. In one embodiment, the displays 1, 2 are each attached to a separately articulated display support arm, and the displays 1, 2, are independently positioned from each other.
[0021] The display support arm 3 is attached to a pitch rotating motor 4, and the display support arm 3 is configured to be manipulated by the pitch rotating motor 4. The pitch rotating motor 4 moves the display support arm 3 in three dimensions to move the displays 1, 2 with respect to the user's eyes 9, 10. The pitch rotating motor 4 is also attached to a yaw rotating motor 5 so that the pitch motor 4 and the yaw rotating motor 5 control the position of the displays 1, 2 in front of a user's eyes 9, 10. Other types of motors can be used, as long as the motors have sufficient a sufficient degree of motion to move the display support arm 3. Actuation of the pitch rotating motor 4 and the yaw rotating motor 5 is controlled by a processor 7 within the headset 11. The processor 7 sends signals to the motors 4, 5 to drive the motors 4, 5 and move the display support arm 3 in multiple directions.
[0022] The system 100 also includes a camera 6 for tracking the user's eyes 9,
10. The camera 6 is fixed within the headset 11 relatively close to the user's eyes 9, 10 such that the camera 6 has an unobstructed line of sight to the user's eyes 9, 10. Data from the camera 6 is transmitted to the processor 7. The camera 6 preferably includes a pupil tracking lens that specifically locates and maps movement of the eye. The camera 6 can track rotation of the eyes, position of the pupil, or other characteristics of the eye. The camera 6 has a very high sampling rate, preferably of at least 30 Hz, more preferably between 50-350 Hz, and most preferably of at least 1000 Hz. Although only one camera 6 is shown in the figures, one of ordinary skill in the art would recognize that multiple cameras 6 can be used.
[0023] In one embodiment, the camera 6 is an active light projecting infrared camera system focused on the user's eyes 9, 10. The camera 6 acquires data comprising an image sequence depicting the pupil of the eye in high contract. Sequential images are analyzed through software in the processor 7 to determine a movement vector that maps a deviation of a center of the pupil from a reference point. This reference point may be the center of the image or it may be a point detected through a calibration procedure related to an orthogonal, straight forward, gaze of the user with a reference point in the image. In one embodiment, the user may be prompted to look straight ahead and press a button, i.e. a key on a keyboard, to relate in software running in processor 7 the current image acquired with the orthogonal gaze. The movement vector then is used to provide motion input commands to the motors 4, 5, and/or to adjust the images provided to the displays 1,2.
[0024] Data from the camera 6 is processed by the processor 7 and is used to drive the motors 4, 5 so that the displays 1, 2 remain in front of the user's eyes 9, 10 as the user moves their eyes 9, 10. For example, if a user moves their eyes 9, 10 quickly to the left, right, up, or down, then the camera 6 sends data related to the eye movement to the processor 7, which then quickly processes this data to provide instructions to the motors 4, 5 to move the display arms 3. It is appreciated that other types of sensors or detectors may be used to measure, track or otherwise determine the position of a user's eyes, such as IR (infra-red) sensors, optical sensors, beam sensors, or laser sensors.
[0025] In one embodiment, the headset 11 also includes an accelerometer 12 or other sensor that detects and generates data related the orientation of the headset 11. The accelerometer 12 can detect motion, orientation, and/or positioning of the headset 11. Data from the accelerometer 12 is transmitted to the processor 7, and the processor 7 transmits both (1) the data related to the movement of the user's eyes 9, 10, and (2) the data related to the orientation of the headset 11 to a central processor 14 through transmitter/receiver units 8, 13. The processor 7 also
receives image data related to a virtual or real world from the central processor 14. The transmitter/receiver unit 13 and the central processor 14 can be in the form of a central processing unit (CPU).
[0026] Figures 1A and IB show the displays 1, 2 are moved from (1) a first position in Figure 1A when the user's eyes 9, 10 are moved upward to (2) a second position shown in Figure IB when the user's eyes 9, 10 are moved downward. The camera 6 sends input related to data gathered during movement of the user's eyes 9, 10 to the processor 7, and the processor 7 then uses this data to provide input to the motors 4, 5. The motors 4, 5 then drive the display support arm 3 to different positions based on the data generated by the camera 6. The processor 7 also provides images to the displays 1, 2 based on the position of the user's eyes 9, 10. For example, if the user is looking downward, the processor 7 sends images to the displays 1, 2 for a downward view. While Figures la and lb show a user moving their eyes up and down, one of ordinary skill in the art will recognize from the present disclosure that other movement of the eyes are captured by the system 100, including a full range of motion, to provide responsive and appropriate images to the displays 1, 2.
[0027] Figure 5A illustrates the displays 1, 2 showing an image in a first orientation, and Figure 5B illustrates the displays 1, 2 showing the image in a second orientation. As shown in Figure 5A, the displays 1, 2 show an image of two characters in a first orientation in which the entire figures are visible. Figure 5A corresponds to a first condition in which a user is looking straight forward. Figure 5B corresponds to a second condition in which a user moves their eyes downward compared to the position depicted in Figure 5A. In Figure 5B, only the lower halves of the figures are visible. Figures 5A and 5B illustrate the concept that as the user moves their eyes, the image is adjusted to correspond to the position of the user's eyes.
[0028] A flow diagram of a method 300 for controlling the system 100 is provided in Figure 3. As shown in Figure 3, input from the accelerometer 310 and
input from the camera 320 are used in data processing 330 to control the display support arms 340. The display support arm moves the displays in front of a user's eyes based on movement and orientation of the user's eyes. Input from the accelerometer 310 and input from the camera 320 are also used by the controller in data processing 330 to generate image data 350 for the displays related to the orientation of headset and related to the orientation of the user's eyes, and the displays show images 360 based on the data. The flow diagram of the method 300 generally discloses two input steps, both the accelerometer input 310 and pupil/eye tracking step 320. Next, the data from both of these steps 310, 320 are processed in data processing step 330. Based on this processing step 330, then the display arms and displays are moved according to the data input in step 340. At the same time, the image generation step 350 determines the appropriate type of view that will be provided to the displays. Finally, the displays are instructed to provide a particular display in step 360 based on the previous steps 310, 320, and 330.
[0029] Figure 4 shows an alternative embodiment of the system 200 including lenses 15, 16 positioned between the user's eyes 9, 10 and the displays 1, 2. All of the components with the same reference numerals in Figure 4 are the same as previously discussed with respect to Figures 1A, IB, and 2. In Figure 4, the lenses 15, 16 are preferably convex and are preferably arranged on a secondary support 23 connected to the display support arm 3. The secondary support 23 is arranged between the user's eyes 9, 10 and the displays 1, 2. The convex lenses 15, 16 focus light from the displays 1, 2 to make viewing the displays 1, 2 at smaller distances possible. The head mounted display device of the system 200 places the displays 1, 2 at a distance from the user's eyes 9, 10 that is less than a minimum that a human eye can focus. Therefore, the system 200 essentially provides very strong reading/magnifying glasses in the form of the convex lenses 15, 16 with, for example, refractive power of 20 diopters or more. The convex lenses 15, 16 adjust the images provided by the displays 1, 2. The convex lenses 15, 16 are physically
connected to displays 1, 2 so that they move with displays, 1, 2 such that the images of the displays 1, 2 can be orientated and positioned for the user's eyes 9, 10.
[0030] Based on the configuration of the system 100, the displays 1, 2 effectively provide an increased field of view while the resolution of the image displayed to the user is not reduced. The system 100 also allows a more natural and comfortable viewing of the virtual world because the user uses natural movement of the user's eyes as the user's eyes are used to manipulate the virtual world view instead of requiring the user to move their entire head and neck to change the view of the virtual world.
[0031] Other embodiments can include multiples cameras to track a user's eyes, and may include a myoelectric detector 17, such as shown in Figure 2. In another embodiment, the displays 1, 2 can be stationary, and light refracting lenses are moved, instead of the displays 1, 2, as the user moves their eyes 9, 10. In another embodiment, the two displays 1, 2 are combined in one display that is moved according to the movements of the user's eyes 9, 10.
[0032] Figure 6 illustrates a schematic view of the system 100 of Figures 1A,
IB, and 2. Figure 6 provides a simplified representation of the displays 1, 2, the display support arm 3, the motors 4 and 5 arranged in a single housing, the processor 7, and the camera 6 facing towards the user's eyes 9, 10.
[0033] In another embodiment shown in Figure 7, one or more projectors 20 are provided that transmit images to a display surface, generically identified as element 1'. The display surface 1' can include a surface on an interior of a helmetlike headset 11, screen, glass, or other surface. The projector 20 transmits one or more beams 20' of images to the display surface 1' The beam 20' can include a set of images, i.e. one image for a left eye and one image for a right eye, that together provide a stereoscopic image for the user's eyes 9, 10. The projector 20 is motorized and can change a position of the beam 20' based on movement of the user's head and/or position of the user's eyes 9, 10. Tracking eye movement, processing the
images, and positioning the images displays based on a user's eye positioning can be accomplished in a manner similar to what has been described.
[0034] It will be appreciated that the foregoing is presented by way of illustration only and not by way of any limitation. It is contemplated that various alternatives and modifications may be made to the described embodiments without departing from the spirit and scope of the invention. Having thus described the present invention in detail, it is to be appreciated and will be apparent to those skilled in the art that many physical changes, only a few of which are exemplified in the detailed description of the invention, could be made without altering the inventive concepts and principles embodied therein. It is also to be appreciated that numerous embodiments incorporating only part of the preferred embodiment are possible which do not alter, with respect to those parts, the inventive concepts and principles embodied therein. The present embodiment and optional configurations are therefore to be considered in all respects as exemplary and/or illustrative and not restrictive, the scope of the invention being indicated by the appended claims rather than by the foregoing description, and all alternate embodiments and changes to this embodiment which come within the meaning and range of equivalency of said claims are therefore to be embraced therein.
• * *
Claims
1. A headset for providing images to a user comprising:
a headset configured to attach to a user's head;
displays positioned adjacent to a user's eyes;
a display support arm connected to the displays;
a motor connected to the display support arm that drives the display support arm between multiple positions;
a camera positioned adjacent to the user's eyes that records the user's eyes; and
a processor that receives data from the camera regarding a position and movement of the user's eyes, and the processor uses the data from the camera to drive the motor and to position the display support arm with respect to the user's eyes such that the displays maintain a position adjacent to the user's eyes.
2. The headset of claim 1, wherein the displays include two displays.
3. The headset of claim 1, wherein the motor includes a pitch motor and a yaw motor.
4. The headset of claim 1, wherein the camera tracks a user's eye movement.
5. The headset of claim 1, further comprising an accelerometer that measures movement of the user's head.
6. The headset of claim 1, further comprising convex lenses positioned between the displays and the user's eyes.
7. The headset of claim 1, further comprising a transmission/receiving unit that sends data from the processor to a central processor.
8. The headset of claim 1, wherein the displays project stereoscopic images to the user.
9. The headset of claim 1, wherein the processor generates images related to the position and the movement of the user's eyes.
10. A headset for providing images to a user comprising:
a headset configured to attach to a user's head;
two displays, each one of the two displays positioned in front a respective one of a user's eyes, such that the two displays provide stereoscopic images to the user's eyes;
a display support arm connected to the two displays;
a pitch motor and a yaw motor, at least one of the pitch motor or the yaw motor connected to the display support arm to drive the display support arm between multiple positions;
a camera positioned adjacent to the user's eyes that records the user's eyes; an accelerometer that measures movement of a user's head; and
a processor that receives data from the camera regarding a position and movement of the user's eyes and data from the accelerometer regarding movement of the user's head, and the processor uses the data from the camera and the accelerometer to drive the pitch motor and the yaw motor and to position the display support arm with respect to the user's eyes such that the two displays maintain a position adjacent to the user's eyes, and images provided to the displays are modified based on the data from the camera and the accelerometer.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
EP16854109.2A EP3356877A4 (en) | 2015-10-04 | 2016-09-30 | Eye gaze responsive virtual reality headset |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US201562236964P | 2015-10-04 | 2015-10-04 | |
US62/236,964 | 2015-10-04 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2017062268A1 true WO2017062268A1 (en) | 2017-04-13 |
Family
ID=58446944
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/US2016/054648 WO2017062268A1 (en) | 2015-10-04 | 2016-09-30 | Eye gaze responsive virtual reality headset |
Country Status (4)
Country | Link |
---|---|
US (2) | US10200680B2 (en) |
EP (1) | EP3356877A4 (en) |
TW (1) | TW201721228A (en) |
WO (1) | WO2017062268A1 (en) |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN107977076A (en) * | 2017-11-17 | 2018-05-01 | 国网山东省电力公司泰安供电公司 | A kind of wearable virtual reality device |
CN108008535A (en) * | 2017-11-17 | 2018-05-08 | 国网山东省电力公司 | A kind of augmented reality equipment |
Families Citing this family (12)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11366521B2 (en) | 2014-11-17 | 2022-06-21 | Thika Holdings Llc | Device for intuitive dexterous touch and feel interaction in virtual worlds |
US10169919B2 (en) * | 2016-09-09 | 2019-01-01 | Oath Inc. | Headset visual displacement for motion correction |
CN107315470B (en) * | 2017-05-25 | 2018-08-17 | 腾讯科技(深圳)有限公司 | Graphic processing method, processor and virtual reality system |
US10810773B2 (en) * | 2017-06-14 | 2020-10-20 | Dell Products, L.P. | Headset display control based upon a user's pupil state |
US11311188B2 (en) * | 2017-07-13 | 2022-04-26 | Micro Medical Devices, Inc. | Visual and mental testing using virtual reality hardware |
TWI646355B (en) * | 2017-12-26 | 2019-01-01 | 宏碁股份有限公司 | Head-mounted display and adjusting method of the same |
US20200008666A1 (en) * | 2018-07-03 | 2020-01-09 | Tarseer, Inc. | Methods and systems for vision monitoring |
US11119321B2 (en) * | 2018-09-10 | 2021-09-14 | Apple Inc. | Electronic device with a display attached to a lens element |
CN109188699A (en) * | 2018-10-27 | 2019-01-11 | 歌尔科技有限公司 | Head-mounted display |
CN109856796A (en) * | 2018-11-20 | 2019-06-07 | 成都理想境界科技有限公司 | Image source mould group, waveguide, near-eye display system and its control method |
US20240013480A1 (en) * | 2020-08-28 | 2024-01-11 | Telefonaktiebolaget Lm Ericsson (Publ) | Projection of an image in an extended reality device |
US20220211266A1 (en) * | 2021-01-05 | 2022-07-07 | Corey Joseph Brewer | Police assistance device and methods of use |
Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20080002262A1 (en) * | 2006-06-29 | 2008-01-03 | Anthony Chirieleison | Eye tracking head mounted display |
US9094677B1 (en) * | 2013-07-25 | 2015-07-28 | Google Inc. | Head mounted display device with automated positioning |
Family Cites Families (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JPH10161058A (en) * | 1996-12-05 | 1998-06-19 | Sony Corp | Display device |
US6409599B1 (en) | 1999-07-19 | 2002-06-25 | Ham On Rye Technologies, Inc. | Interactive virtual reality performance theater entertainment system |
US8860634B2 (en) * | 2012-10-11 | 2014-10-14 | Sony Computer Entertainment Europe Limited | Head mountable display |
US9058053B2 (en) * | 2012-10-26 | 2015-06-16 | The Boeing Company | Virtual reality display system |
US9019174B2 (en) * | 2012-10-31 | 2015-04-28 | Microsoft Technology Licensing, Llc | Wearable emotion detection and feedback system |
US9788714B2 (en) * | 2014-07-08 | 2017-10-17 | Iarmourholdings, Inc. | Systems and methods using virtual reality or augmented reality environments for the measurement and/or improvement of human vestibulo-ocular performance |
-
2016
- 2016-09-30 EP EP16854109.2A patent/EP3356877A4/en not_active Withdrawn
- 2016-09-30 WO PCT/US2016/054648 patent/WO2017062268A1/en active Application Filing
- 2016-10-03 US US15/283,725 patent/US10200680B2/en active Active
- 2016-10-03 TW TW105131898A patent/TW201721228A/en unknown
-
2019
- 2019-02-01 US US16/265,352 patent/US20190238827A1/en not_active Abandoned
Patent Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20080002262A1 (en) * | 2006-06-29 | 2008-01-03 | Anthony Chirieleison | Eye tracking head mounted display |
US9094677B1 (en) * | 2013-07-25 | 2015-07-28 | Google Inc. | Head mounted display device with automated positioning |
Non-Patent Citations (1)
Title |
---|
See also references of EP3356877A4 * |
Cited By (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN107977076A (en) * | 2017-11-17 | 2018-05-01 | 国网山东省电力公司泰安供电公司 | A kind of wearable virtual reality device |
CN108008535A (en) * | 2017-11-17 | 2018-05-08 | 国网山东省电力公司 | A kind of augmented reality equipment |
CN107977076B (en) * | 2017-11-17 | 2018-11-27 | 国网山东省电力公司泰安供电公司 | A kind of wearable virtual reality device |
Also Published As
Publication number | Publication date |
---|---|
EP3356877A4 (en) | 2019-06-05 |
US20170099478A1 (en) | 2017-04-06 |
TW201721228A (en) | 2017-06-16 |
EP3356877A1 (en) | 2018-08-08 |
US20190238827A1 (en) | 2019-08-01 |
US10200680B2 (en) | 2019-02-05 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US10200680B2 (en) | Eye gaze responsive virtual reality headset | |
JP6339239B2 (en) | Head-mounted display device and video display system | |
JP2023504373A (en) | Predictive eye-tracking system and method for foveal rendering of electronic displays | |
JP6378781B2 (en) | Head-mounted display device and video display system | |
US9077973B2 (en) | Wide field-of-view stereo vision platform with dynamic control of immersive or heads-up display operation | |
CN104067160B (en) | Make picture material method placed in the middle in display screen using eyes tracking | |
US9213185B1 (en) | Display scaling based on movement of a head-mounted display | |
JP2020515895A (en) | Operable fovea display | |
US20060098087A1 (en) | Housing device for head-worn image recording and method for control of the housing device | |
EP3023828B1 (en) | Image display apparatus | |
EP2786196A1 (en) | Wide field-of-view 3d stereo vision platform with dynamic control of immersive or heads-up display operation | |
WO2012009115A1 (en) | Head mounted display having a panoramic field of view | |
JPH11202256A (en) | Head-mounting type image display device | |
US20180205932A1 (en) | Stereoscopic video see-through augmented reality device with vergence control and gaze stabilization, head-mounted display and method for near-field augmented reality application | |
JP2008256946A (en) | Sickness prevention device for image display device | |
JPWO2019187808A1 (en) | Information processing equipment, information processing methods, and programs | |
EP2859399A1 (en) | Apparatus and method for a bioptic real time video system | |
US20200285310A1 (en) | Information processing apparatus, information processing method, and program | |
US11743447B2 (en) | Gaze tracking apparatus and systems | |
EP4365663A2 (en) | Gaze tracking apparatus and systems | |
US20230106173A1 (en) | Head mounted display (hmd) apparatus, method, and system | |
US20230049899A1 (en) | System and method for enhancing visual acuity | |
WO2022254782A1 (en) | Line-of-sight detection device, display device, line-of-sight detection system, and line-of-sight detection method | |
KR102315674B1 (en) | Sight recovery apparatus through virtual reality-based eye movement induction | |
JP2022514217A (en) | Maneuverable positioning element |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 16854109 Country of ref document: EP Kind code of ref document: A1 |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
WWE | Wipo information: entry into national phase |
Ref document number: 2016854109 Country of ref document: EP |