WO2014199155A1 - Appareil et systèmes à réalité virtuelle - Google Patents

Appareil et systèmes à réalité virtuelle Download PDF

Info

Publication number
WO2014199155A1
WO2014199155A1 PCT/GB2014/051798 GB2014051798W WO2014199155A1 WO 2014199155 A1 WO2014199155 A1 WO 2014199155A1 GB 2014051798 W GB2014051798 W GB 2014051798W WO 2014199155 A1 WO2014199155 A1 WO 2014199155A1
Authority
WO
WIPO (PCT)
Prior art keywords
user
display
hmd
image
head
Prior art date
Application number
PCT/GB2014/051798
Other languages
English (en)
Inventor
Jeremy David ASHFORTH
Simon Mark BENSON
Ian Henry BICKERSTAFF
Sharwin Winesh RAGHOEBARDAYAL
Original Assignee
Sony Computer Entertainment Europe Limited
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Sony Computer Entertainment Europe Limited filed Critical Sony Computer Entertainment Europe Limited
Publication of WO2014199155A1 publication Critical patent/WO2014199155A1/fr

Links

Classifications

    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/0093—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
    • A—HUMAN NECESSITIES
    • A63—SPORTS; GAMES; AMUSEMENTS
    • A63F—CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
    • A63F13/00—Video games, i.e. games using an electronically generated display having two or more dimensions
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/017—Head mounted
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G06F3/012—Head tracking input arrangements
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G06F3/013—Eye tracking input arrangements
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0481—Interaction techniques based on graphical user interfaces [GUI] based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
    • G06F3/0482—Interaction with lists of selectable items, e.g. menus
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/18—Eye characteristics, e.g. of the iris
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/0179—Display position adjusting means not related to the information to be displayed
    • G02B2027/0187—Display position adjusting means not related to the information to be displayed slaved to motion of at least a part of the body of the user, e.g. head, eye

Definitions

  • This invention relates to head-mountable apparatus and systems.
  • a head-mountable display is one example of a head-mountable apparatus.
  • Audio headphones comprising a frame supporting one or more audio transducers are another example of a head-mountable apparatus.
  • a head-mounted torch or light is a further example of a head-mountable apparatus. The following background discussions will relate mainly to HMDs, but the principles are also applicable to other types of head-mountable apparatus.
  • an image or video display device is provided which may be worn on the head or as part of a helmet. Either one eye or both eyes are provided with small electronic display devices.
  • Some HMDs allow a displayed image to be superimposed on a real-world view.
  • This type of HMD can be referred to as an optical see-through HMD and generally requires the display devices to be positioned somewhere other than directly in front of the user's eyes. Some way of deflecting the displayed image so that the user may see it is then required. This might be through the use of a partially reflective mirror placed in front of the user's eyes so as to allow the user to see through the mirror but also to see a reflection of the output of the display devices.
  • a waveguide arrangement employing total internal reflection is used to convey a displayed image from a display device disposed to the side of the user's head so that the user may see the displayed image but still see a view of the real world through the waveguide.
  • a virtual image of the display is created (using known techniques) so that the user sees the virtual image at an appropriate size and distance to allow relaxed viewing.
  • the virtual image may be arranged so as to be perceived by the user at a distance of (for example) 20 m from the user, having a perceived size of 5 m x 5m.
  • HMDs allow the user only to see the displayed images, which is to say that they obscure the real world environment surrounding the user.
  • This type of HMD can position the actual display devices in front of the user's eyes, in association with appropriate lenses or other optical components which place a virtual displayed image at a suitable distance for the user to focus in a relaxed manner - for example, at a similar virtual distance and perceived size as the optical see-through HMD described above.
  • This type of device might be used for viewing movies or similar recorded content, or for viewing so-called virtual reality content representing a virtual space surrounding the user. It is of course however possible to display a real-world view on this type of HMD, for example by using a forward-facing camera to generate images for display on the display devices.
  • HMDs Although the original development of HMDs was perhaps driven by the military and professional applications of these devices, HMDs are becoming more popular for use by casual users in, for example, computer game or domestic computing applications.
  • Figure 1 schematically illustrates an HMD worn by a user
  • Figure 2 is a schematic plan view of an HMD
  • Figure 3 schematically illustrates the formation of a virtual image by an HMD
  • Figure 4 schematically illustrates another type of display for use in an HMD
  • Figure 5 schematically illustrates a pair of stereoscopic images
  • FIGS 6 and 7 schematically illustrate a user wearing an HMD connected to a Sony®
  • Figure 8 schematically illustrates a change of view of user of an HMD
  • Figures 9a and 9b schematically illustrate HM Ds with motion sensing
  • Figure 10 schematically illustrates a position sensor based on optical flow detection
  • Figure 1 1 schematically illustrates the generation of images in response to HMD position or motion detection
  • Figure 12 is a further schematic plan view of an HMD
  • Figure 13 schematically illustrates a pair of eye tracking images
  • Figure 14 schematically illustrates another technique for capturing eye tracking images
  • Figure 15 schematically illustrates a control technique according to eye orientation detection
  • Figure 16 schematically illustrates a control technique according to head orientation detection
  • Figure 17 schematically illustrates a flat menu
  • Figure 18 schematically illustrates a part-cylindrical menu
  • Figure 19 schematically illustrates a part-spherical menu
  • Figure 20 schematically illustrates a user menu
  • Figure 21 schematically illustrates another arrangement of a user menu
  • Figure 22 schematically illustrates a paged user menu
  • Figure 23 schematically illustrates a dial control
  • Figure 24 schematically illustrates a slider control
  • FIGS. 25 and 26 schematically illustrate game environment hotspots
  • Figures 27 and 28 schematically illustrate the highlighting of a menu item
  • Figure 29 schematically illustrates an audio processor
  • Figures 30 and 31 schematically illustrate examples of eye rotation
  • Figures 32 and 33 schematically illustrate a peripheral view item
  • Figures 34 and 35 are schematic flowcharts relating to the display of a peripheral view item.
  • a user 10 is wearing an HMD 20 (as an example of a generic head-mountable apparatus - other examples, where the technical context allows, including audio headphones or a head-mountable light source) on the user's head 30.
  • the HMD comprises a frame 40, in this example formed of a rear strap and a top strap, and a display portion 50.
  • HMD of Figure 1 may comprise further features, to be described below in connection with other drawings, but which are not shown in Figure 1 for clarity of this initial explanation.
  • the HMD of Figure 1 completely (or at least substantially completely) obscures the user's view of the surrounding environment. All that the user can see is the pair of images displayed within the HMD.
  • the HMD has associated headphone audio transducers or earpieces 60 which fit into the user's left and right ears 70.
  • the earpieces 60 replay an audio signal provided from an external source, which may be the same as the video signal source which provides the video signal for display to the user's eyes.
  • this HMD may be considered as a so-called "full immersion" HMD.
  • the HMD is not a full immersion HMD, and may provide at least some facility for the user to see and/or hear the user's surroundings.
  • a camera for example a camera mounted on the HMD
  • a front-facing camera 122 may capture images to the front of the HMD, in use.
  • a Bluetooth® antenna 124 may provide communication facilities or may simply be arranged as a directional antenna to allow a detection of the direction of a nearby Bluetooth transmitter.
  • a video signal is provided for display by the HMD.
  • This could be provided by an external video signal source 80 such as a video games machine or data processing apparatus (such as a personal computer), in which case the signals could be transmitted to the HMD by a wired or a wireless connection 82.
  • suitable wireless connections include Bluetooth® connections.
  • Audio signals for the earpieces 60 can be carried by the same connection.
  • any control signals passed from the HMD to the video (audio) signal source may be carried by the same connection.
  • a power supply 83 including one or more batteries and/or being connectable to a mains power outlet
  • the power supply 83 and the video signal source 80 may be separate units or may be embodied as the same physical unit. There may be separate cables for power and video (and indeed for audio) signal supply, or these may be combined for carriage on a single cable (for example, using separate conductors, as in a USB cable, or in a similar way to a "power over Ethernet" arrangement in which data is carried as a balanced signal and power as direct current, over the same collection of physical wires).
  • the video and/or audio signal may be carried by, for example, an optical fibre cable.
  • at least part of the functionality associated with generating image and/or audio signals for presentation to the user may be carried out by circuitry and/or processing forming part of the HMD itself.
  • a power supply may be provided as part of the HMD itself.
  • embodiments of the invention are applicable to an HMD having at least one electrical and/or optical cable linking the HMD to another device, such as a power supply and/or a video (and/or audio) signal source. So, embodiments of the invention can include, for example:
  • an HMD having a cabled connection to a power supply and to a video and/or audio signal source, embodied as a single physical cable or more than one physical cable;
  • an HMD having its own video and/or audio signal source (as part of the HMD arrangement) and a cabled connection to a power supply;
  • an HMD having a wireless connection to a video and/or audio signal source and a cabled connection to a power supply.
  • the physical position at which the cable 82 and/or 84 enters or joins the HMD is not particularly important from a technical point of view. Aesthetically, and to avoid the cable(s) brushing the user's face in operation, it would normally be the case that the cable(s) would enter or join the HMD at the side or back of the HMD (relative to the orientation of the user's head when worn in normal operation). Accordingly, the position of the cables 82, 84 relative to the HMD in Figure 1 should be treated merely as a schematic representation.
  • Figure 1 provides an example of a head-mountable display system comprising a frame to be mounted onto an observer's head, the frame defining one or two eye display positions which, in use, are positioned in front of a respective eye of the observer and a display element mounted with respect to each of the eye display positions, the display element providing a virtual image of a video display of a video signal from a video signal source to that eye of the observer.
  • Figure 1 shows just one example of an HMD.
  • an HMD could use a frame more similar to that associated with conventional eyeglasses, namely a substantially horizontal leg extending back from the display portion to the top rear of the user's ear, possibly curling down behind the ear.
  • the user's view of the external environment may not in fact be entirely obscured; the displayed images could be arranged so as to be superposed (from the user's point of view) over the external environment. An example of such an arrangement will be described below with reference to Figure 4.
  • FIG. 1 A schematic plan view of how this is achieved is provided as Figure 2, which illustrates the positions 100 of the user's eyes and the relative position 1 10 of the user's nose.
  • the display portion 50 in schematic form, comprises an exterior shield 120 to mask ambient light from the user's eyes and an internal shield 130 which prevents one eye from seeing the display intended for the other eye.
  • the combination of the user's face, the exterior shield 120 and the interior shield 130 form two compartments 140, one for each eye.
  • a display element 150 and one or more optical elements 160 In each of the compartments there is provided a display element 150 and one or more optical elements 160. The way in which the display element and the optical element(s) cooperate to provide a display to the user will be described with reference to Figure 3.
  • the display element 150 generates a displayed image which is (in this example) refracted by the optical elements 160 (shown schematically as a convex lens but which could include compound lenses or other elements) so as to generate a virtual image 170 which appears to the user to be larger than and significantly further away than the real image generated by the display element 150.
  • the virtual image may have an apparent image size (image diagonal) of more than 1 m and may be disposed at a distance of more than 1 m from the user's eye (or from the frame of the HMD). In general terms, depending on the purpose of the HMD, it is desirable to have the virtual image disposed a significant distance from the user.
  • FIG. 4 An alternative arrangement is shown in Figure 4. This arrangement may be used where it is desired that the user's view of the external environment is not entirely obscured. However, it is also applicable to HMDs in which the user's external view is wholly obscured.
  • the display element 150 and optical elements 200 cooperate to provide an image which is projected onto a mirror 210, which deflects the image towards the user's eye position 220. The user perceives a virtual image to be located at a position 230 which is in front of the user and at a suitable distance from the user.
  • the mirror 210 can be a substantially 100% reflective mirror.
  • the arrangement of Figure 4 then has the advantage that the display element and optical elements can be located closer to the centre of gravity of the user's head and to the side of the user's eyes, which can produce a less bulky HMD for the user to wear.
  • the mirror 210 can be made partially reflective so that the user sees the external environment, through the mirror 210, with the virtual image superposed over the real external environment.
  • the lateral displacements in Figure 5 could in fact be the other way round, which is to say that the left eye image as drawn could in fact be the right eye image, and the right eye image as drawn could in fact be the left eye image.
  • the left eye image as drawn could in fact be the right eye image
  • the right eye image as drawn could in fact be the left eye image.
  • some stereoscopic displays tend to shift objects to the right in the right eye image and to the left in the left eye image, so as to simulate the idea that the user is looking through a stereoscopic window onto the scene beyond.
  • some HMDs use the arrangement shown in Figure 5 because this gives the impression to the user that the user is viewing the scene through a pair of binoculars. The choice between these two arrangements is at the discretion of the system designer.
  • an HMD may be used simply to view movies and the like. In this case, there is no change required to the apparent viewpoint of the displayed images as the user turns the user's head, for example from side to side. In other uses, however, such as those associated with virtual reality (VR) or augmented reality (AR) systems, the user's viewpoint needs to track movements with respect to a real or virtual space in which the user is located.
  • VR virtual reality
  • AR augmented reality
  • Figure 6 schematically illustrates a user wearing an HMD connected to a Sony®
  • PlayStation 3® games console 300 as an example of a base device.
  • the games console 300 is connected to a mains power supply 310 and (optionally) to a main display screen (not shown).
  • a cable acting as the cables 82, 84 discussed above (and so acting as both power supply and signal cables), links the HMD 20 to the games console 300 and is, for example, plugged into a USB socket 320 on the console 300.
  • a single physical cable is provided which fulfils the functions of the cables 82, 84.
  • a hand-held controller 330 which may be, for example, a Sony® Move® controller which communicates wirelessly with the games console 300 to control (or to contribute to the control of) game operations relating to a currently executed game program.
  • the video displays in the HMD 20 are arranged to display images generated by the games console 300, and the earpieces 60 in the HMD 20 are arranged to reproduce audio signals generated by the games console 300. Note that if a USB type cable is used, these signals will be in digital form when they reach the HMD 20, such that the HMD 20 comprises a digital to analogue converter (DAC) to convert at least the audio signals back into an analogue form for reproduction.
  • DAC digital to analogue converter
  • Images from the camera 122 mounted on the HMD 20 are passed back to the games console 300 via the cable 82, 84.
  • signals from those sensors may be at least partially processed at the HMD 20 and/or may be at least partially processed at the games console 300. The use and processing of such signals will be described further below.
  • the USB connection from the games console 300 also provides power to the HMD 20, according to the USB standard.
  • FIG. 7 schematically illustrates a similar arrangement in which the games console is connected (by a wired or wireless link) to a so-called "break out box" acting as a base or intermediate device 350, to which the HMD 20 is connected by a cabled link 82,84.
  • the breakout box has various functions in this regard.
  • One function is to provide a location, near to the user, for some user controls relating to the operation of the HMD, such as (for example) one or more of a power control, a brightness control, an input source selector, a volume control and the like.
  • Another function is to provide a local power supply for the HMD (if one is needed according to the embodiment being discussed).
  • Another function is to provide a local cable anchoring point.
  • the break-out box 350 is fixed to the ground or to a piece of furniture, but rather than having a very long trailing cable from the games console 300, the break-out box provides a locally weighted point so that the cable 82, 84 linking the HMD 20 to the break-out box will tend to move around the position of the break-out box. This can improve user safety and comfort by avoiding the use of very long trailing cables.
  • an HMD may form part of a set or cohort of interconnected devices (that is to say, interconnected for the purposes of data or signal transfer, but not necessarily connected by a physical cable).
  • processing which is described as taking place “at” one device, such as at the HMD could be devolved to another device such as the games console (base device) or the break-out box.
  • Processing tasks can be shared amongst devices.
  • Source signals, on which the processing is to take place, could be distributed to another device, or the processing results from the processing of those source signals could be sent to another device, as required. So any references to processing taking place at a particular device should be understood in this context.
  • the user's viewpoint needs to track movements with respect to a real or virtual space in which the user is located.
  • FIG. 8 schematically illustrates the effect of a user head movement in a VR or AR system.
  • a virtual environment is represented by a (virtual) spherical shell 250 around a user. Because of the need to represent this arrangement on a two-dimensional paper drawing, the shell is represented by a part of a circle, at a distance from the user equivalent to the separation of the displayed virtual image from the user.
  • a user is initially at a first position 260 and is directed towards a portion 270 of the virtual environment. It is this portion 270 which is represented in the images displayed on the display elements 150 of the user's HMD.
  • the displayed portion of the virtual environment also moves so that, at the end of the movement, a new portion 290 is displayed by the HMD.
  • the apparent viewpoint within the virtual environment moves with the head movement. If the head rotates to the right side, for example, as shown in Figure 8, the apparent viewpoint also moves to the right from the user's point of view. If the situation is considered from the aspect of a displayed object, such as a displayed object 300, this will effectively move in the opposite direction to the head movement. So, if the head movement is to the right, the apparent viewpoint moves to the right but an object such as the displayed object 300 which is stationary in the virtual environment will move towards the left of the displayed image and eventually will disappear off the left-hand side of the displayed image, for the simple reason that the displayed portion of the virtual environment has moved to the right whereas the displayed object 300 has not moved in the virtual environment.
  • FIGS 9a and 9b schematically illustrated HMDs with motion sensing.
  • the two drawings are in a similar format to that shown in Figure 2. That is to say, the drawings are schematic plan views of an HMD, in which the display element 150 and optical elements 160 are represented by a simple box shape. Many features of Figure 2 are not shown, for clarity of the diagrams. Both drawings show examples of HMDs with a motion detector for detecting motion of the observer's head.
  • a forward-facing camera 322 is provided on the front of the HMD. This may be the same camera as the camera 122 discussed above, or may be an additional camera. This does not necessarily provide images for display to the user (although it could do so in an augmented reality arrangement). Instead, its primary purpose in the present embodiments is to allow motion sensing. A technique for using images captured by the camera 322 for motion sensing will be described below in connection with Figure 10.
  • the motion detector comprises a camera mounted so as to move with the frame; and an image comparator operable to compare successive images captured by the camera so as to detect inter-image motion.
  • Figure 9b makes use of a hardware motion detector 332.
  • This can be mounted anywhere within or on the HMD.
  • suitable hardware motion detectors are piezoelectric accelerometers or optical fibre gyroscopes. It will of course be appreciated that both hardware motion detection and camera-based motion detection can be used in the same device, in which case one sensing arrangement could be used as a backup when the other one is unavailable, or one sensing arrangement (such as the camera) could provide data for changing the apparent viewpoint of the displayed images, whereas the other (such as an accelerometer) could provide data for image stabilisation.
  • Figure 10 schematically illustrates one example of motion detection using the camera 322 of Figure 9a.
  • the camera 322 is a video camera, capturing images at an image capture rate of, for example, 25 images per second. As each image is captured, it is passed to an image store 400 for storage and is also compared, by an image comparator 410, with a preceding image retrieved from the image store. The comparison uses known block matching techniques (so- called "optical flow” detection) to establish whether substantially the whole image has moved since the time at which the preceding image was captured.
  • Localised motion might indicate moving objects within the field of view of the camera 322, but global motion of substantially the whole image would tend to indicate motion of the camera rather than of individual features in the captured scene, and in the present case because the camera is mounted on the HMD, motion of the camera corresponds to motion of the HMD and in turn to motion of the user's head.
  • the displacement between one image and the next, as detected by the image comparator 410, is converted to a signal indicative of motion by a motion detector 420. If required, the motion signal is converted by to a position signal by an integrator 430.
  • the HMD can detect head motion using a mechanical or solid state detector 332 such as an accelerometer.
  • a mechanical or solid state detector 332 such as an accelerometer.
  • the detector 332 can be better suited for use with higher frequency motion detection.
  • a high image rate camera such as a 200 Hz capture rate camera
  • a camera-based system may be more appropriate.
  • the detector 332 could take the place of the camera 322, the image store 400 and the comparator 410, so as to provide an input directly to the motion detector 420. Or the detector 332 could take the place of the motion detector 420 as well, directly providing an output signal indicative of physical motion.
  • position or motion detecting techniques are of course possible.
  • a mechanical arrangement by which the HMD is linked by a moveable pantograph arm to a fixed point may be used, with position and orientation sensors detecting changes in the deflection of the pantograph arm.
  • a system of one or more transmitters and receivers, mounted on the HMD and on a fixed point can be used to allow detection of the position and orientation of the HMD by triangulation techniques.
  • the HMD could carry one or more directional transmitters, and an array of receivers associated with known or fixed points could detect the relative signals from the one or more transmitters.
  • the transmitters could be fixed and the receivers could be on the HMD.
  • transmitters and receivers examples include infra-red transducers, ultrasonic transducers and radio frequency transducers.
  • the radio frequency transducers could have a dual purpose, in that they could also form part of a radio frequency data link to and/or from the HMD, such as a Bluetooth® link.
  • Figure 11 schematically illustrates image processing carried out in response to a detected position or change in position of the HMD.
  • the apparent viewpoint of the video being displayed to the user of the HMD is changed in response to a change in actual position or orientation of the user's head.
  • a motion sensor 450 (such as the arrangement of Figure 10 and/or the motion detector 332 of Figure 9b) supplying data indicative of motion and/or current position to a required image position detector 460, which translates the actual position of the HMD into data defining the required image for display.
  • An image generator 480 accesses image data stored in an image store 470 if required, and generates the required images from the appropriate viewpoint for display by the HMD.
  • the external video signal source can provide the functionality of the image generator 480 and act as a controller to compensate for the lower frequency component of motion of the observer's head by changing the viewpoint of the displayed image so as to move the displayed image in the opposite direction to that of the detected motion so as to change the apparent viewpoint of the observer in the direction of the detected motion.
  • FIG. 12 is a further schematic plan view of an HMD similar to the plan view of Figure 2 described above. Features already described in connection with that Figure will not be described again.
  • a pair of eye-tracking cameras 500, 510 are provided within the compartments corresponding to each eye tries to generate images of the wearer's eyes in use.
  • the eye-tracking cameras 500, 510 are directed in a backwards direction relative to the orientation of the user's head, so that they look back at the user's eyes.
  • the cameras 500, 510 in Figure 12 can be disposed anywhere with respect to the compartments corresponding to each eye, as long as they do not obscure the user's view of the displayed images; they are just shown in the outer corners by way of one schematic example.
  • the cameras 500, 510 can rely on illumination provided by the displayed images within the HMD or, if that is insufficient, on infrared or other illumination directed towards the user's eyes.
  • Figure 13 schematically illustrates a pair of eye tracking images as captured by the cameras 500, 510.
  • Figure 13 shows an image 520 of the user's left eye and an image 530 of the user's right eye. In general, these will be captured as separate images but are shown alongside one another in Figure 13 for the purposes of this explanation.
  • a significant feature to be derived from the captured images 520, 530 is the location of the pupils of the wearer's eyes.
  • the cameras 500, 510 are mounted in a fixed relationship relative to the frame of the HMD, and that, in use, the HMD adopts a fixed relationship to the user's head. Accordingly, from the image position of the pupils within the captured images 520, 530, the position of each pupil relative to the HMD, and therefore relative to the display elements of the HMD, can be directly established.
  • a first stage in detecting the pupil position is to detect the extent of the iris or coloured portion 540. Then, using known image processing techniques, a central region of the iris is scanned to establish the upper 542 and lower 544 bounds of the pupil (a dark area within the iris) to allow the vertical centre 546 of the pupil to be detected. As a crosscheck, the system establishes whether the vertical centre 546 is also the approximate vertical centre of the iris 540.
  • a horizontal scan of the captured image 520, 530 is carried out at the vertical position indicated by the vertical centre 546. This gives left 550 and right 552 boundaries of the pupil, from which a horizontal centre 554 of the pupil can be established.
  • each pupil provides a set of coordinates of the pupil position.
  • This detection can be carried out frequently.
  • the detection is carried out at least as frequently as the image display rate of the display elements of the HMD, so that a next image to be displayed by the HMD can be corrected using techniques to be described below according to the detected pupil positions found by a detection process carried out immediately before the display of that image.
  • the eye position may be detected less frequently than the image display frequency, but a most-recently detected eye position is used in the correction of each image for display.
  • a nominal or central eye position may be detected.
  • One way to detect this is for the HMD to display material (such as an instruction which the user has to read) at a central position in the HMD display, and for the eye position detection arrangement to detect the eye position at the time that the user is viewing that centrally displayed information.
  • calibration can be carried out to detect the user's eye position when the user is viewing information at various extreme positions (such as each corner) of the HMD display. This calibration can be carried out without the user necessarily knowing, simply by providing information for the user to read at different positions in the HMD display. From the calibration, a mapping between detected eye position and region of the HMD display which the user is viewing can be generated.
  • Further information can be derived from the images captured by the cameras 500, 510.
  • a detection of whether the eyes are shut can be obtained by the lack of a detection of an iris or pupil at or near to the expected position.
  • FIG. 14 A different arrangement is illustrated schematically in Figure 14.
  • a user's eye 560 views a display element 570 through an optical system shown schematically by a lens 580 but which (as discussed above) may include other optical elements.
  • An eye tracking camera 590 is disposed so as to capture images of the user's eye 560 via at least a part of the optical system 580. So, where the optical system 580 comprises one or more lenses, the camera 590 is disposed so as to capture images of the user's eye through one or more of those lenses.
  • the camera 590 may be disposed so as to capture images of the user's eye via one or more of the reflectors.
  • the camera 590 is disposed alongside the display element 570, but in other embodiments the camera 590 could be positioned within a compound optical system 580.
  • the position of only one eye and its associated pupil is tracked, so that only one of the eye tracking cameras is required. This is on the basis that, for most people, movements of one eye mapped directly to corresponding movements of the other eye.
  • a feature of using two cameras is that the results for the pair of eyes can be checked against one another, and that individual detections can be obtained for each of the user's eyes.
  • Figure 15 schematically illustrates a control technique according to eye orientation detection.
  • the current state of a game or other data processing operation is such that a menu of the type discussed above is to be displayed or that the game environment of a current game includes one or more hotspots of the type discussed above.
  • a detector 800 comprises one or two eye tracking cameras of the type described above, along with processing to carry out the steps described with reference to Figure 13.
  • the output of the detector 800 comprises a set of data defining either or both of: (a) the absolute coordinates or position of one or each pupil (relative to a frame of reference defined relative to the HMD), and (b) deviations of the pupil position(s) from a nominal or expected pupil position.
  • a controller 810 detects the image location, with respect to the images being displayed by the HMD, which the user is looking at.
  • the controller 810 can make use of calibration data obtained during a calibration stage as discussed above, in order to provide a mapping between pupil position and image position being looked at.
  • the controller can derive control data to be passed to a data processing or game engine to carry out a certain data processing function in response to the user selection.
  • An image renderer 820 renders a menu item, an enlarged hotspot or other display feature according to the detection, by the controller 810, of the image area of the displayed image at which the user is looking. So, for example, the image renderer might render an enlarged menu item or a brought forward menu item of the type discussed above, an enlarged hotspot or other feature if the user is looking at that image region.
  • Figure 16 illustrates a similar arrangement, but using head orientation rather than eye orientation as an input.
  • Head orientation (relative to a central position of the user's viewpoint, for example) is detected by a head orientation detector 830 using techniques described earlier.
  • a controller 810' operates in a similar manner to the controller 810, and an image renderer 820' operates in a similar manner to the image renderer 820.
  • references to the user "looking at" a screen item such as a menu item or a so-called hotspot (see below) may be references to the user directing the user's head at that item (so that a head orientation detection is relevant) and/or references to the user directing the user's eyes at that item (so that an eye orientation detection is relevant).
  • an item can be selected by the user looking at it, but this could lead to false selections as the user gazes around the displayed scene.
  • item could be provisionally selected by the user looking at it, with the selection being confirmed by either the operation of a manual control such as a control button, or by the user carrying out some other head or eye based operation such as looking at a "select" icon within a predetermined time of looking at the provisionally selected item.
  • Figures 17-19 provide schematic illustrations of different techniques for displaying items such as textual items to the user, particularly in the context of a stereoscopic (3-D) HMD display. The techniques may also be applicable to other 3-D displays.
  • Figure 17 schematically illustrates a flat textual menu box 1000 displayed within an overall display image 1010.
  • the term "flat” means that all parts of the menu 1000 are at the same apparent depth with respect to the user's viewpoint.
  • planar menu could be used but it could be tilted in 3-D display space so that its apparent depth varies linearly with one or both dimensions of the planar menu.
  • Figure 18 schematically illustrates a part-cylindrical menu 1020.
  • This has an apparent depth which is constant with relation to a vertical dimension of the menu but which varies along a horizontal dimension of the menu so as to give the impression that, in the horizontal direction, different portions of the menu are at different virtual distances from the viewer.
  • the depth may vary in the horizontal direction so that all parts of the menu appear to be on the surface of a part of a cylinder having its centre at the position in the virtual 3-D space of the viewer. But in other examples, they could just be a simple variations so that a lateral central portion of the menu is perceived to be further away in the virtual 3-D space than lateral edges of the menu.
  • a central portion could be displayed so as to be perceived to be closer to the user then the lateral edges.
  • Figure 19 schematically illustrates a part-spherical menu 1030.
  • the apparent depth in a 3-D display space varies across the menu item, but in this example the variation is with respect to horizontal and vertical directions.
  • the menu may be displayed so as to appear to be on the surface of a portion of a sphere centred at the viewer's virtual position.
  • any of the menus of Figures 17 to 19 could extend outside the central region of a user's view, and indeed could be normally displayed in a 3-D display space at or beyond a periphery of the user's normal view.
  • a menu could be displayed below a normal view, so that the user has to look down to see it.
  • an icon for an alert or the like could be placed in the user's peripheral vision, so that it becomes visible (it is enlarged, brought forward, moved more centrally, rendered with detail or the like) when the user looks at it. This can make use of the fact that some HMD displays have a lower display resolution and a more visible flicker at their periphery, so the periphery is a useful place to display such icons or alerts.
  • FIG. 20 schematically illustrates a user menu as displayed by the HMD. Multiple menu items are displayed as small boxes 600, 610, 620, 630. Examples of these selectable menu items could include commands such as "restart game”, “reset score”, “invite friend”, “upload score to social media” and the like. The user's pupil or head position is detected in order to detect which (if any) of the menu items the user is looking at.
  • menu item is displayed differently, as shown schematically as a larger box 640.
  • the menu item is simply enlarged to indicate that it is currently being looked at. Not only does this confirm the user's selection or provisional selection of the menu item, it also makes it easier for the user to read.
  • the menu item is moved towards the user in a 3-D display space.
  • the overall result might be that the size, in pixels on the HMD display, of the brought-forward menu item is the same as the size of the enlarged menu item discussed above, but the change in depth associated with bringing the menu item forward can provide a more natural indication to the user that the menu item is selected or provisionally selected.
  • the user can indicate a final selection of the many item by operating a manual user control such as a button or, in some embodiments, by using an eye control such as (a) looking directly from the provisionally selected menu item to a "select" menu item, or (b) closing one or both of the user's eyes for a least a predetermined period such as two seconds.
  • Figure 21 schematically illustrates another arrangement of a user menu.
  • a collection of base menu items 1100, 1110, 1120, 1130 are displayed.
  • Each one of the base menu items provides a top-level menu with two or more underlying sub-levels of information which are not normally displayed. If, however, the user looks at one of the base menu items (in the example of Figure 21 , the base menu item 1100) then the underlying sub-levels of information are displayed, in this example as further boxes 1101 , 1102 alongside the corresponding base menu item.
  • Figure 22 schematically illustrates a paged user menu comprising a front menu page 1200 which overlies, in 3-D space, a plurality of further menu pages 1210, 1220, 1230.
  • the front menu page 1200 may be displayed to the user says the most prominent by, for example, (a) setting its depth position to be closest to the user out of all of the menu pages, and/or (b) displaying it in a more prominent fashion than that applicable to the other menu pages - for example, by displaying the other menu pages in a semi-transparent format.
  • the user can switch between the setup menu pages, for example in a rotational (carousel) fashion by looking at different ones of the menu pages or by using manual controls.
  • Figure 23 schematically illustrates a dial control 1300 and Figure 24 schematically illustrates a slider control 1320. These can be used for the user to change certain settings associated with the current HMD activity, such as a volume control or a difficulty level.
  • a rotatable pointer 1310 indicates the current setting.
  • an extreme edge 1330 of a laterally movable bar 1340 indicates a current setting.
  • the user In order to operate the controls of Figures 23 and 24 in the context of an HMD using head and/or eye tracking, the user first looks at the control which has the effect of selecting it for adjustment. Then, the user can adjust the control by maintaining his gaze at the control but rotating his head to the left or right around a general access between the user's head and the control. When the desired setting is reached, the user looks away from the control.
  • Figures 25 and 26 schematically illustrates game environment "hotspots". These relate to the display, within a game environment, of user-selectable information points shown schematically as small circles 1650.
  • the rectangular area 1660 of Figure 26 schematically represents the extent of the displayed image as seen by the HMD wearer, and it is assumed that the displayed image represents a current game environment including (but purely by way of example) a landscape 1670, buildings 1680, game characters 1690 and the like.
  • the display of the game environment may change according to the current position of the user's head, as discussed earlier.
  • the game environment hotspots 1650 are stationary with respect to the environment, so they move with the environment according to changes in the position of the user's head.
  • the hotspots 1650 are designed so as to be small and relatively inconspicuous when not selected. This avoids the hotspots occupying too much of the user's view and distracting the user from the game environment. However, if the user looks directly at a hotspot, as detected by the pupil detection process discussed above, the hotspot is expanded from its initial displayed as a small circle into a temporarily displayed wider region 1700 ( Figure 26) containing information to the user. Because the hotspots are locked or located relative to the game environment, the information can be relevant to the particular region of the game environment at which the hotspot is located. Examples here might be user suggestions or clues as to how to deal with a particular game issue (such as dealing with an enemy or exiting from a difficult region).
  • the hotspot returns to his previous display size as a small circle.
  • the precise details of how the hotspots are displayed, either in their small or their enlarged state, are open to the skilled person to determine.
  • Figures 27 and 28 schematically illustrate the highlighting of a menu item.
  • a user-selectable menu item (shown here in schematic form as the word "menu") 1800 is displayed in a semi-transparent form until the user looks at it.
  • the user looks at it it is displayed at the same position but in a bolder form as in the item 1810 of Figure 28. If the user looks away, the item returns to its semi-transparent form.
  • the controller of Figure 15 or Figure 16 can act as a controller for generating one or more of the image features and for generating an indicator for display to indicate to the user to look at the image features.
  • the controller can also generate a pointer or similar indication that tells the user that if the user moves the user's head or gaze in a certain direction (as indicated by the indicator) then the user will see the image features in question.
  • the indicator can be, for example, an arrow or similar directional marker, or its direction can be indicated by its position within the field of view (so that, for example, an indicator to the right of the field of view indicates to the user to move the user's head or gaze to the right).
  • Indicators can be used for some or all image features, but in some example systems they are used only for messages of a certain importance level, such as system messages and warnings.
  • Messages and image features can be used, along with other game features, to calibrate the detection of head movement and/or gaze during early stages of the game or other interaction, so as to avoid the need for a separate calibration phase.
  • the user interface controls for the other user(s) can be greyed out - or displayed in a less prominent colour such as a monotone.
  • a set of user interface controls is implemented as a succession of display elements, with the user having to select "continue”, “next” or the like to move from one to another.
  • all (or at least more than one) of these elements can be displayed simultaneously alongside one another, so that the user can move from one to another by changing the position at which the user is looking.
  • the image renderer 820, 820' could revert all other image material to a 2D display format, or (if notionally remaining in a 3D mode) push all of the other material back to a distance from the viewer in the 3D domain of infinity (or at least a large virtual distance such as more than 10m).
  • the perceived depth of 3D material could be changed by the renderer 820, 820' to be behind the displayed GUI item.
  • Figure 29 schematically illustrates a corresponding audio process applicable in respect of audio signals associated with the video material for display. So, the arrangement of Figure 29 operates in association with the arrangements of Figures 15 and 16, for example.
  • Figure 29 provides an example of an audio processor configured to provide audio content in an audio presentation format dependent upon the current operations of the controllers 810, 810'.
  • a selector 1800 receives data form the controller 810, 810' of Figure 15 or 16 indicative of image rendering actions to be taken (such as drawing or expanding a menu) dependent upon the detected eye or head position.
  • the selector 1800 selects an audio format for presentation of audio material 1805 to the user.
  • An audio processor 1810 processes the audio material 1805 according to the selected audio format and supplies the processed audio data, via a headphone driver (such as an amplifier and impedance matcher) 1820 to earpieces to be worn by the HMD wearer.
  • a headphone driver such as an amplifier and impedance matcher
  • the audio material can be, for example, material provided for indicating an acknowledgement that the user has selected a menu item (such as a bell or beep sound).
  • the audio processor 1805 is operable with respect to a so-called 3-D audio system so that virtual sound sources can be simulated in any direction with respect to the earpiece wearer's head. Accordingly, the arrangement of Figure 29 operates so as to provide the processed audio data at an apparent spatial position, relative to the listener's head, which corresponds to (or substantially corresponds to, such as being in the same solid angle octant as or being within an angular deviation of, say, 10% of the three dimensional angular position of) the item activated, selected or otherwise indicated by the user's head or eye movement.
  • a user may open a menu or display or select a menu item using a head or eye movement.
  • a similarly located sound is provided to confirm that the user has done this.
  • Figures 30 and 31 schematically illustrate examples of eye rotation. These drawings illustrate a phenomenon which occurs when the user rotates his or her eyes to look at a peripheral object.
  • a user's eyeball 1850 is schematically illustrated.
  • the eye socket and other parts of the user's head are not shown, merely for clarity of the drawing.
  • the user is looking straight ahead, in a direction 1855.
  • the user is able to see peripheral objects despite the fact that the user is looking straight ahead.
  • a schematic barrier 1860 is a representation, for the purposes of this schematic discussion, of a peripheral part of the HMD display.
  • the user can see objects between the direction 1855 and a direction 1870, that is to say, within an angular range 1875. Any items which are further displaced from the straight ahead direction are blocked by the barrier 1860.
  • the new viewing direction 1880 is slightly more acute than the previous viewing direction 1870, which in turn means that items in the angular region 1890 were visible in the viewer's peripheral vision when the viewer was looking ahead in Figure 30 but are now invisible in Figure 31 when the viewer is actually trying to look in that direction.
  • This phenomenon can be used advantageously in the presentation of menu and display items, as discussed below.
  • optical properties of the compartments 140 applicable to each eye can provide the equivalent of the barrier 1860, or an additional physical barrier can be provided so as to achieve the effect that an extreme periphery of the display field of view is visible when the user is not looking at it but is masked when the user is looking in that direction.
  • Figures 32 and 33 schematically illustrate a peripheral view item.
  • a broken line rectangle 1900 provides a very schematic representation of the field of view available using the HMD displays.
  • a trigger icon or indicator 1910 is generated, for example by the controller 810 or the Tenderer 820, and displayed at an extreme periphery of the HMD display field of view. This could be at a left edge, a right edge, an upper edge or a lower edge. If the field of view is not rectangular then any peripheral position may be used.
  • the trigger icon 1910 When the user is looking straight ahead, the user can see the trigger icon 1910.
  • the user looks at the trigger icon 1910.
  • the user cannot actually see the trigger icon 1910 when the user looks in that direction.
  • the menu or other display item 1920 is displayed (for example, as discussed above, in a more prominent or brought-forward position in 3-D virtual space).
  • the item 1920 In order that the viewer can see the whole of the item 1920, the item 1920s displaced inwards (towards the centre of the field of view, or at least away from the periphery of the field of view) by a displacement 1930.
  • Figures 34 and 35 are schematic flowcharts relating to the display of a peripheral view item.
  • a trigger icon is displayed.
  • the detector 800, 830 detects the head and/or eye orientation of the user. If that orientation is directed towards the trigger icon, then at step 2023 trigger icon is expanded so as to display a fuller item such as a more detailed menu, a display item or the like. As discussed above, in examples, the fuller item can be displayed so as to be larger and/or so as to be brought forward in 3-D space.
  • trigger icon such as the icon 1910 is displayed at a peripheral position, which in some embodiments may be a position and visible when the user is looking straight ahead but not visible when the user is looking to the periphery of the display field of view.
  • the detector 800 detects the eye orientation and, at a step 2050 if the user is looking towards the trigger icon it is expanded to a fuller item (for example, in the manner discussed above) but which is displayed at a less peripheral position in the field of view, again as discussed above.
  • HMD head mountable display
  • the HMD system comprising: a detector (such as the detector 800 and/or 830) configured to detect the eye position and/or orientation and/or the head orientation of the HMD wearer; and a controller (such as the controller 810, 810', operating in cooperation with the renderer 820, 820') configured to control the generation of images for display, at least in part, according to the detection of the eye position and/or orientation and/or the head orientation of the HMD wearer; in which the controller is configured to change the display of one or more image features according to whether or not the user is currently looking at those image features, the image features are menu items or information items, by rendering an image feature so as to be more prominent on the display if the user is looking at it, such that the image feature is enlarged, moved from a peripheral display position, replaced by a larger image feature and/or brought forward in a 3-D display space if the user is looking at it, and/or
  • embodiments of the present invention can provide a head mountable display (HMD) system in which images are generated for display to the user, with the image generation being controlled, at least in part, according to the detection of the eye position and/or orientation and/or the head orientation of the HMD wearer.
  • HMD head mountable display
  • the HMD may comprise one or more cameras directed (in use) at the wearer's eyes, and a detection arrangement associated with the cameras to detect either or both of the eye position and/or orientation, and whether one or both eyes are shut.
  • the display of one or more image features can be changed according to whether or not the user is currently looking at those image features.
  • the image features may be menu items or information items.
  • Such an item can be made more prominent on the display if the user is looking at it.
  • such an item can be enlarged, moved from a peripheral display position and/or brought forward (in a 3-D display space) if the user is looking at it.
  • the enlarged and/or brought forward item can include more information than the item before being enlarged or brought forward.
  • Embodiments of the invention can provide a system in which menu items are displayed as flat, part-cylindrical, part spherical or other formats, for example with relation to a 3-D display space.
  • Embodiments of the invention can provide a system in which menu or other items are displayed more prominently if the user is detected to be looking at them.
  • Embodiments of the invention can provide a nested set of information for display such that a lower level of the nested information is displayed if the user looks at a displayed item representing a higher level of the nested set.
  • Embodiments of the invention also provide a system comprising an HMD and one or more of a base device such as a games console and an intermediate device such as a breakout box.
  • a base device such as a games console
  • an intermediate device such as a breakout box.
  • Embodiments of the present invention can provide a method of operation of a head mountable display (HMD) system in which images are generated for display to the user, comprising: controlling the generation of images, at least in part, according to the detection of the eye position and/or orientation and/or the head orientation of the HMD wearer.
  • Embodiments of the present invention can provide a method of operation of a head mountable display (HMD) comprising displaying menu items as flat, part-cylindrical, part spherical or other formats.
  • Embodiments of the present invention can provide a method of operation of a head mountable display (HMD) comprising displaying menu or other items more prominently if the user is detected to be looking at them.
  • HMD head mountable display
  • Embodiments of the present invention can provide a method of operation of a head mountable display (HMD) comprising displaying a lower level of a nested set of information if the user looks at a displayed item representing a higher level of the nested set.
  • HMD head mountable display

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Human Computer Interaction (AREA)
  • Multimedia (AREA)
  • Optics & Photonics (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Ophthalmology & Optometry (AREA)
  • User Interface Of Digital Computer (AREA)

Abstract

La présente invention concerne un système d'affichage à réalité virtuelle (HMD, Head mountable display) dans lequel des images sont générées afin d'être présentées à l'utilisateur, comprenant un détecteur configuré pour détecter la position et/ou l'orientation des yeux et/ou l'orientation de la tête du porteur du système HMD ; et une unité de commande configurée pour commander la production d'images à afficher conformément, au moins en partie, à la détection de la position et/ou de l'orientation des yeux et/ou de l'orientation de la tête du porteur du système HMD. L'unité de commande est configurée pour modifier l'affichage d'un ou plusieurs éléments caractéristiques de l'image en fonction du fait que l'utilisateur regarde ou non ces éléments caractéristiques de l'image, les éléments caractéristiques de l'image étant des éléments de menu ou des éléments d'informations, par rendu d'un élément caractéristique de l'image de manière à ce qu'il ressorte plus clairement sur l'affichage si l'utilisateur le regarde, de manière à ce que l'élément caractéristique de l'image soit agrandi, déplacé par rapport à une position d'affichage périphérique, remplacé par un élément caractéristique d'image plus grand et/ou mis en avant dans un espace d'affichage tridimensionnel si l'utilisateur le regarde.
PCT/GB2014/051798 2013-06-11 2014-06-11 Appareil et systèmes à réalité virtuelle WO2014199155A1 (fr)

Applications Claiming Priority (6)

Application Number Priority Date Filing Date Title
GB1310376.7 2013-06-11
GBGB1310376.7A GB201310376D0 (en) 2013-06-11 2013-06-11 Head-mountable apparatus and systems
GBGB1314982.8A GB201314982D0 (en) 2013-06-11 2013-08-21 Head-mountable apparatus and systems
GB1314982.8 2013-08-21
GB1404741.9 2014-03-17
GBGB1404741.9A GB201404741D0 (en) 2013-06-11 2014-03-17 Head-mountable apparatus and systems

Publications (1)

Publication Number Publication Date
WO2014199155A1 true WO2014199155A1 (fr) 2014-12-18

Family

ID=48876080

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/GB2014/051798 WO2014199155A1 (fr) 2013-06-11 2014-06-11 Appareil et systèmes à réalité virtuelle

Country Status (2)

Country Link
GB (4) GB201310376D0 (fr)
WO (1) WO2014199155A1 (fr)

Cited By (15)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2016107519A1 (fr) * 2014-12-31 2016-07-07 青岛歌尔声学科技有限公司 Procédé d'affichage d'images en 3d et dispositif de visiocasque
WO2017086708A1 (fr) * 2015-11-17 2017-05-26 Samsung Electronics Co., Ltd. Procédé et appareil d'affichage de contenu
RU2631151C2 (ru) * 2015-03-31 2017-09-19 Сяоми Инк. Способ и устройство для управления воспроизведением и электронное оборудование
WO2018002801A1 (fr) * 2016-06-30 2018-01-04 Nokia Technologies Oy Procédé et appareil pour fournir une indication visuelle d'un point d'intérêt à l'extérieur de la vue d'un utilisateur
CN107797280A (zh) * 2016-08-31 2018-03-13 乐金显示有限公司 个人沉浸式显示装置及其驱动方法
US10031718B2 (en) 2016-06-14 2018-07-24 Microsoft Technology Licensing, Llc Location based audio filtering
WO2018183027A1 (fr) * 2017-03-27 2018-10-04 Microsoft Technology Licensing, Llc Rendu sélectif d'affichages périphériques creux sur la base de mouvements d'utilisateur
US10216260B2 (en) 2017-03-27 2019-02-26 Microsoft Technology Licensing, Llc Selective rendering of sparse peripheral displays based on element saliency
EP3396511A4 (fr) * 2015-12-21 2019-07-24 Sony Interactive Entertainment Inc. Dispositif de traitement d'informations et procédé de réception d'opération
US10419655B2 (en) 2015-04-27 2019-09-17 Snap-Aid Patents Ltd. Estimating and using relative head pose and camera field-of-view
EP3683656A4 (fr) * 2017-10-20 2020-11-04 Huawei Technologies Co., Ltd. Procédé et appareil de génération d'interface de réalité virtuelle
CN113686350A (zh) * 2020-05-18 2021-11-23 华为技术有限公司 道路信息的显示方法、装置和智能可穿戴设备
US11255663B2 (en) 2016-03-04 2022-02-22 May Patents Ltd. Method and apparatus for cooperative usage of multiple distance meters
US11290708B2 (en) 2019-02-19 2022-03-29 Edgy Bees Ltd. Estimating real-time delay of a video data stream
WO2022256152A1 (fr) * 2021-06-02 2022-12-08 Dathomir Laboratories Llc Procédé et dispositif de navigation dans des fenêtres en 3d

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111934704B (zh) * 2015-12-24 2023-02-24 索尼互动娱乐股份有限公司 频带确定装置、头戴式显示器和频带确定方法

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20110043644A1 (en) * 2008-04-02 2011-02-24 Esight Corp. Apparatus and Method for a Dynamic "Region of Interest" in a Display System
US20120212414A1 (en) * 2010-02-28 2012-08-23 Osterhout Group, Inc. Ar glasses with event and sensor triggered control of ar eyepiece applications
US20120272179A1 (en) * 2011-04-21 2012-10-25 Sony Computer Entertainment Inc. Gaze-Assisted Computer Interface
US20130128364A1 (en) * 2011-11-22 2013-05-23 Google Inc. Method of Using Eye-Tracking to Center Image Content in a Display

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5689619A (en) * 1996-08-09 1997-11-18 The United States Of America As Represented By The Secretary Of The Army Eyetracker control of heads-up displays
US5982555A (en) * 1998-01-20 1999-11-09 University Of Washington Virtual retinal display with eye tracking
US6456262B1 (en) * 2000-05-09 2002-09-24 Intel Corporation Microdisplay with eye gaze detection
US8531394B2 (en) * 2010-07-23 2013-09-10 Gregory A. Maltz Unitized, vision-controlled, wireless eyeglasses transceiver

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20110043644A1 (en) * 2008-04-02 2011-02-24 Esight Corp. Apparatus and Method for a Dynamic "Region of Interest" in a Display System
US20120212414A1 (en) * 2010-02-28 2012-08-23 Osterhout Group, Inc. Ar glasses with event and sensor triggered control of ar eyepiece applications
US20120272179A1 (en) * 2011-04-21 2012-10-25 Sony Computer Entertainment Inc. Gaze-Assisted Computer Interface
US20130128364A1 (en) * 2011-11-22 2013-05-23 Google Inc. Method of Using Eye-Tracking to Center Image Content in a Display

Cited By (27)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2016107519A1 (fr) * 2014-12-31 2016-07-07 青岛歌尔声学科技有限公司 Procédé d'affichage d'images en 3d et dispositif de visiocasque
US10104358B2 (en) 2014-12-31 2018-10-16 Qingdao Goertek Technology Co., Ltd. 3D image display method and head-mounted device
RU2631151C2 (ru) * 2015-03-31 2017-09-19 Сяоми Инк. Способ и устройство для управления воспроизведением и электронное оборудование
US9997197B2 (en) 2015-03-31 2018-06-12 Xiaomi Inc. Method and device for controlling playback
US10419655B2 (en) 2015-04-27 2019-09-17 Snap-Aid Patents Ltd. Estimating and using relative head pose and camera field-of-view
US11019246B2 (en) 2015-04-27 2021-05-25 Snap-Aid Patents Ltd. Estimating and using relative head pose and camera field-of-view
US10594916B2 (en) 2015-04-27 2020-03-17 Snap-Aid Patents Ltd. Estimating and using relative head pose and camera field-of-view
WO2017086708A1 (fr) * 2015-11-17 2017-05-26 Samsung Electronics Co., Ltd. Procédé et appareil d'affichage de contenu
US10976808B2 (en) 2015-11-17 2021-04-13 Samsung Electronics Co., Ltd. Body position sensitive virtual reality
EP3396511A4 (fr) * 2015-12-21 2019-07-24 Sony Interactive Entertainment Inc. Dispositif de traitement d'informations et procédé de réception d'opération
US11906290B2 (en) 2016-03-04 2024-02-20 May Patents Ltd. Method and apparatus for cooperative usage of multiple distance meters
US11255663B2 (en) 2016-03-04 2022-02-22 May Patents Ltd. Method and apparatus for cooperative usage of multiple distance meters
US10031718B2 (en) 2016-06-14 2018-07-24 Microsoft Technology Licensing, Llc Location based audio filtering
US11032471B2 (en) 2016-06-30 2021-06-08 Nokia Technologies Oy Method and apparatus for providing a visual indication of a point of interest outside of a user's view
WO2018002801A1 (fr) * 2016-06-30 2018-01-04 Nokia Technologies Oy Procédé et appareil pour fournir une indication visuelle d'un point d'intérêt à l'extérieur de la vue d'un utilisateur
CN107797280A (zh) * 2016-08-31 2018-03-13 乐金显示有限公司 个人沉浸式显示装置及其驱动方法
CN110447001B (zh) * 2017-03-27 2021-11-23 微软技术许可有限责任公司 基于用户移动的稀疏外围显示器的选择性绘制
WO2018183027A1 (fr) * 2017-03-27 2018-10-04 Microsoft Technology Licensing, Llc Rendu sélectif d'affichages périphériques creux sur la base de mouvements d'utilisateur
US10216260B2 (en) 2017-03-27 2019-02-26 Microsoft Technology Licensing, Llc Selective rendering of sparse peripheral displays based on element saliency
CN110447001A (zh) * 2017-03-27 2019-11-12 微软技术许可有限责任公司 基于用户移动的稀疏外围显示器的选择性绘制
US10277943B2 (en) 2017-03-27 2019-04-30 Microsoft Technology Licensing, Llc Selective rendering of sparse peripheral displays based on user movements
EP3683656A4 (fr) * 2017-10-20 2020-11-04 Huawei Technologies Co., Ltd. Procédé et appareil de génération d'interface de réalité virtuelle
US11294535B2 (en) 2017-10-20 2022-04-05 Huawei Technologies Co., Ltd. Virtual reality VR interface generation method and apparatus
US11290708B2 (en) 2019-02-19 2022-03-29 Edgy Bees Ltd. Estimating real-time delay of a video data stream
US11563932B2 (en) 2019-02-19 2023-01-24 Edgy Bees Ltd. Estimating real-time delay of a video data stream
CN113686350A (zh) * 2020-05-18 2021-11-23 华为技术有限公司 道路信息的显示方法、装置和智能可穿戴设备
WO2022256152A1 (fr) * 2021-06-02 2022-12-08 Dathomir Laboratories Llc Procédé et dispositif de navigation dans des fenêtres en 3d

Also Published As

Publication number Publication date
GB201404741D0 (en) 2014-04-30
GB201310376D0 (en) 2013-07-24
GB201314982D0 (en) 2013-10-02
GB2517263A (en) 2015-02-18
GB201410405D0 (en) 2014-07-23

Similar Documents

Publication Publication Date Title
WO2014199155A1 (fr) Appareil et systèmes à réalité virtuelle
US9645398B2 (en) Electronic correction based on eye tracking
EP3008691B1 (fr) Appareil et systèmes tête haute
EP3008698B1 (fr) Systèmes et appareil pouvant être montés sur la tête
EP3008548B1 (fr) Systèmes et appareil de casque
EP3070513B1 (fr) Système d'affichage à assemblage avec tête
US11277603B2 (en) Head-mountable display system
US20140362113A1 (en) Change nature of display according to overall motion
US20140361987A1 (en) Eye controls
US20140362370A1 (en) Directional light beams for angle detection
US11045733B2 (en) Virtual reality
GB2517008A (en) Head-mountable apparatus and systems
GB2571286A (en) Virtual reality
WO2018115842A1 (fr) Affichage de réalité virtuelle par visiocasque
EP3559782B1 (fr) Traitement de données
WO2018115843A1 (fr) Visiocasque
WO2018096315A1 (fr) Réalité virtuelle
GB2515130A (en) Head-mountable apparatus and systems

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 14732310

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 14732310

Country of ref document: EP

Kind code of ref document: A1