US20170097753A1 - Systems, devices, and methods for interacting with content displayed on head-mounted displays - Google Patents

Systems, devices, and methods for interacting with content displayed on head-mounted displays Download PDF

Info

Publication number
US20170097753A1
US20170097753A1 US15/282,535 US201615282535A US2017097753A1 US 20170097753 A1 US20170097753 A1 US 20170097753A1 US 201615282535 A US201615282535 A US 201615282535A US 2017097753 A1 US2017097753 A1 US 2017097753A1
Authority
US
United States
Prior art keywords
interface device
user
portable interface
display
head
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Abandoned
Application number
US15/282,535
Other languages
English (en)
Inventor
Matthew Bailey
Stefan Alexander
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Google LLC
Original Assignee
Thalmic Labs Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Thalmic Labs Inc filed Critical Thalmic Labs Inc
Priority to US15/282,535 priority Critical patent/US20170097753A1/en
Publication of US20170097753A1 publication Critical patent/US20170097753A1/en
Priority to US15/799,642 priority patent/US10656822B2/en
Assigned to Thalmic Labs Inc. reassignment Thalmic Labs Inc. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: BAILEY, MATTHEW, ALEXANDER, STEFAN
Assigned to GOOGLE LLC reassignment GOOGLE LLC ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: NORTH INC.
Assigned to NORTH INC. reassignment NORTH INC. CHANGE OF NAME (SEE DOCUMENT FOR DETAILS). Assignors: Thalmic Labs Inc.
Abandoned legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0487—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/017—Head mounted
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/017—Head mounted
    • G02B27/0172—Head mounted characterised by optical features
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/017—Head mounted
    • G02B27/0176—Head mounted characterised by mechanical features
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G06F3/013—Eye tracking input arrangements
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G06F3/014—Hand-worn input/output arrangements, e.g. data gloves
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/02—Input arrangements using manually operated switches, e.g. using keyboards or dials
    • G06F3/0227—Cooperation and interconnection of the input arrangement with other functional units of a computer
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
    • G06F3/033—Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor
    • G06F3/038—Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
    • G06F3/0383—Signal control means within the pointing device
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0484—Interaction techniques based on graphical user interfaces [GUI] for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
    • G06F3/04842—Selection of displayed objects or displayed text elements
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/0101—Head-up displays characterised by optical features
    • G02B2027/014—Head-up displays characterised by optical features comprising information/image processing systems
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/017—Head mounted
    • G02B2027/0178—Eyeglass type
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/0179—Display position adjusting means not related to the information to be displayed
    • G02B2027/0187—Display position adjusting means not related to the information to be displayed slaved to motion of at least a part of the body of the user, e.g. head, eye
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2203/00—Indexing scheme relating to G06F3/00 - G06F3/048
    • G06F2203/033—Indexing scheme relating to G06F3/033
    • G06F2203/0331—Finger worn pointing device
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2203/00—Indexing scheme relating to G06F3/00 - G06F3/048
    • G06F2203/038—Indexing scheme relating to G06F3/038
    • G06F2203/0381—Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2203/00—Indexing scheme relating to G06F3/00 - G06F3/048
    • G06F2203/038—Indexing scheme relating to G06F3/038
    • G06F2203/0384—Wireless input, i.e. hardware and software details of wireless interface arrangements for pointing devices
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/02—Input arrangements using manually operated switches, e.g. using keyboards or dials

Definitions

  • the present systems, devices, and methods generally relate to interacting with content displayed on head-mounted displays and particularly relate to a multi-input interface that combines eye tracking with a wireless portable interface device.
  • Such “portable” electronic devices may include on-board power supplies (such as batteries or other power storage systems) and may be “wireless” (i.e., designed to operate without any wire-connections to other, non-portable electronic systems); however, a small and lightweight electronic device may still be considered portable even if it includes a wire-connection to a non-portable electronic system.
  • a microphone may be considered a portable electronic device whether it is operated wirelessly or through a wire-connection.
  • a wearable electronic device is any portable electronic device that a user can carry without physically grasping, clutching, or otherwise holding onto the device with their hands.
  • a wearable electronic device may be attached or coupled to the user by a strap or straps, a band or bands, a clip or clips, an adhesive, a pin and clasp, an article of clothing, tension or elastic support, an interference fit, an ergonomic form, etc.
  • Examples of wearable electronic devices include digital wristwatches, electronic armbands, electronic rings, electronic ankle-bracelets or “anklets,” head-mounted electronic display units, hearing aids, and so on.
  • form factor i.e., size, geometry, and appearance
  • a head-mounted display is a form of wearable electronic device that is worn on the user's head and, when so worn, positions a display in the user's field of view. This enables the user to see content displayed on the display at all times, without using their hands to hold the display and regardless of the direction in which the user's head is facing.
  • a wearable head-mounted display may completely occlude the external environment from the user's view, in which case the display is well-suited for virtual reality applications.
  • An example of a virtual reality head-mounted display is the Oculus Rift®.
  • a head-mounted display may be at least partially transparent and/or sized and positioned to only occupy a portion of the user's field of view.
  • a wearable heads-up display is a head-mounted display that enables the user to see displayed content but does not prevent the user from being able to see their external environment. Wearable heads-up displays are well-suited for augmented reality applications. Examples of wearable heads-up displays include: the Google Glass®, the Optinvent Ora®, the Epson Moverio®, the Microsoft HoloLens®, and the Sony Glasstron®, just to name a few.
  • a human-electronics interface mediates communication between a human and one or more electronic device(s).
  • a human-electronics interface is enabled by one or more electronic interface device(s) that: a) detect inputs effected by the human and convert those inputs into electric signals that can be processed or acted upon by the one or more electronic device(s), and/or b) provide sensory outputs (e.g., typically visual, auditory, and/or tactile) to the human from the one or more electronic device(s), where the user is able to sense the outputs and understand some information represented by the outputs.
  • a human-electronics interface may be one directional or bidirectional, and a complete interface may make use of multiple interface devices.
  • the computer mouse is a one-way interface device that detects inputs effected by a user of a computer and converts those inputs into electric signals that can be processed by the computer, while the computer's display or monitor is a one-way interface device that provides outputs to the user in a visual form through which the user can understand information.
  • the computer mouse and display complete a bidirectional human-computer interface (“HCI”).
  • HCI is an example of a human-electronics interface.
  • a wearable electronic device may function as an interface device if, for example, the wearable electronic device: a) includes sensors that detect inputs effected by a user, and b) transmits signals to another electronic device based on those inputs.
  • Sensor-types and input-types may each take on a variety of forms, including but not limited to: tactile sensors (e.g., buttons, switches, touchpads, or keys) providing manual control, acoustic sensors providing voice-control, electromyography sensors providing gestural control, and/or accelerometers providing gestural control.
  • Portable electronic devices that include display screens typically require the user to use their hand(s) to carry the device and/or to orient the device so that the user may see, access, receive feedback from, and/or generally interact with the device's display screen. Occupying the user's hand(s) is an inconvenience that can significantly hinder the user's ability to interact with the portable electronic device and/or to interact with other aspects of their environment while operating the portable electronic device. However, this hindrance is at least partially overcome by making the display screen of the portable electronic device wearable, as is the case with head-mounted displays. Making the display screen of the portable electronic device wearable enables the user to see, access, and/or receive feedback from the display screen without using their hand(s). In recent years, head-mounted displays have begun to gain wider acceptance, with a number of recently introduced head-mounted display devices having the potential for widespread adoption by consumers.
  • a system that enables interaction with content displayed on a head-mounted display may be summarized as including: a head-mounted display including: at least one display positioned in a field of view of at least one eye of a user when the head-mounted display is worn on a head of the user; a processor communicatively coupled to the at least one display; a non-transitory processor-readable storage medium communicatively coupled to the processor, wherein the non-transitory processor-readable storage medium stores processor-executable instructions and/or data that, when executed by the processor, cause the at least one display to display at least one object that is responsive to a selection operation performed by the user; and a wireless receiver to wirelessly receive signals, the wireless receiver communicatively coupled to the processor; an eye-tracker communicatively coupled to the processor, the eye-tracker to detect that the user is gazing at the at least one object displayed by the at least one display; and a wireless portable interface device with a form factor to be carried by or on the user, the portable interface device physically separate from the
  • the selection operation performed by the user may comprise a substantially concurrent combination of gazing at the least one object displayed by the at least one display and activating the at least actuator of the portable interface device.
  • a visual effect may be displayed on the at least one display of the head-mounted display.
  • the non-transitory processor-readable storage medium may further store processor-executable instructions and/or data that, when executed by the processor in response to the wireless receiver wirelessly receiving the signal from the portable interface device, cause the processor to: request current gaze direction data from the eye-tracker; identify a particular object at which the user is gazing based on the current gaze direction data received from the eye-tracker, the particular object identified among the at least one object displayed by the at least one display; and cause the at least one display to display the visual effect on the particular object.
  • the portable interface device may be batteryless and may include a piezoelectric element communicatively coupled to the actuator and an antenna communicatively coupled to the piezoelectric element.
  • the wireless receiver of the head-mounted display may include a radio frequency receiver.
  • the actuator When activated by the user, the actuator may mechanically actuate the piezoelectric element. In response to the mechanical actuation, the piezoelectric element may generate an electric signal. In response to the electric signal generated by the piezoelectric element, the antenna may transmit a radio frequency signal.
  • the portable interface device may be batteryless and may include a mechanical resonator physically coupled to the actuator.
  • the wireless receiver of the head-mounted display may include at least one of: a microphone and/or a piezoelectric element tuned to be responsive to a sonic signal.
  • the actuator When activated by the user, the actuator may mechanically actuate the mechanical resonator. In response to the mechanical actuation, the mechanical resonator may generate the sonic signal.
  • the sonic signal may include an ultrasonic signal.
  • the object may include at least one object selected from a group consisting of: a menu item, a graphical button, a keyboard key, a notification, one of multiple objects displayed by the at least one display of the head-mounted display, a file, a folder, and an alphanumeric character.
  • the portable interface device may include a wearable device selected from a group consisting of: a ring, a wristband, and an armband.
  • the actuator of the portable interface device may include a button.
  • the eye-tracker may be carried by and physically coupled to the head-mounted display.
  • a method of operating a system wherein the system comprises a head-mounted display, an eye-tracker, and a wireless portable interface device, the portable interface device physically separate from the head-mounted display, may be summarized as including: displaying an object within a field of view of at least one eye of a user by at least one display of the head-mounted display; receiving a selection operation from the user by the system, wherein receiving a selection operation from the user by the system includes: detecting, by the eye tracker, that the user is gazing at the object; while detecting, by the eye tracker, that the user is gazing at the object, receiving, by the portable interface device, an activation from the user of an actuator of the portable interface device; in response to receiving, by the portable interface device, the activation from the user of the actuator of the portable interface device, wirelessly transmitting a signal by the portable interface device; and wirelessly receiving the signal by a wireless receiver of the head-mounted display; and in response to receiving the selection operation from the user by the system, displaying a visual effect by the at
  • the head-mounted display may include a processor and a non-transitory processor-readable storage medium communicatively coupled to the processor.
  • the non-transitory processor-readable storage medium may store processor-executable instructions and/or data and the method may include executing, by the processor, the processor-executable instructions and/or data stored in the non-transitory processor-readable storage medium to cause: the at least one display of the head-mounted display to display the object within the field of view of at least one eye of the user; and the at least one display of the head-mounted display to display the visual effect in response to the system receiving the selection operation from the user.
  • the portable interface device may be batteryless and may include a piezoelectric element communicatively coupled to the actuator and an antenna communicatively coupled to the piezoelectric element, and the wireless receiver of the head-mounted display may include a radio frequency receiver.
  • receiving, by the portable interface device, an activation from the user of an actuator of the portable interface device may include receiving, by the portable interface device, a mechanical actuation of the piezoelectric element from the user.
  • the piezoelectric element may generate an electric signal.
  • Wirelessly transmitting a signal by the portable interface device may include wirelessly transmitting a radio frequency signal by the antenna of the portable interface device in response to the electric signal generated by the piezoelectric element.
  • Wirelessly receiving the signal by a wireless receiver of the head-mounted display may include wirelessly receiving the radio frequency signal by the radio frequency receiver of the head-mounted display.
  • the portable interface device may be batteryless and may include a mechanical resonator physically coupled to the actuator.
  • receiving, by the portable interface device, an activation from the user of an actuator of the portable interface device may include receiving, by the portable interface device, a mechanical actuation of the mechanical resonator from the user.
  • the mechanical resonator may generate a sonic signal.
  • Wirelessly transmitting a signal by the portable interface device may include wirelessly transmitting the sonic signal by mechanical resonator of the portable interface device.
  • Wirelessly receiving the signal by a wireless receiver of the head-mounted display may include wirelessly receiving the sonic signal by the wireless receiver of the head-mounted display, wherein the wireless receiver of the head-mounted display includes at least one of: a microphone and/or a piezoelectric element tuned to be responsive to the sonic signal.
  • the sonic signal generated by the mechanical resonator of the portable interface device may include an ultrasonic signal.
  • the portable interface device may include an on-board power source and a radio frequency transmitter.
  • Wirelessly transmitting a signal by the portable interface device may include wirelessly transmitting a radio signal by the radio frequency transmitter of the portable interface device, the radio signal having a frequency in a range of 10 MHz to 10 GHz.
  • Wirelessly receiving the signal by a wireless receiver of the head-mounted display may include wirelessly receiving the radio signal by the wireless receiver of the head-mounted display, and wherein the wireless receiver of the head-mounted display includes a radio frequency receiver.
  • a completely wearable human-electronics interface may be summarized as including: a wearable heads-up display including a wireless receiver; an eye-tracker carried by the wearable heads-up display; and a wearable actuator including a wireless transmitter to transmit wireless signals to the wireless receiver of the wearable heads-up display.
  • the wearable heads-up display may further include a processor that is communicatively coupled to both the eye-tracker and the wireless receiver, the processor to effect interactions with content displayed by the wearable heads-up display in response to concurrent inputs from both the eye-tracker and the wearable actuator.
  • FIG. 1 is an illustrative diagram showing an exemplary system that enables interaction with content displayed on a head-mounted display in accordance with the present systems, devices, and methods.
  • FIG. 2 is an illustrative diagram showing a human-electronics interface in which a user wears a system that enables the user to interact with displayed content in accordance with the present systems, devices, and methods.
  • FIG. 3 is a flow-diagram showing an exemplary method of operating a system in accordance with the present systems, devices, and methods.
  • the various embodiments described herein provide systems, devices, and methods for interacting with content displayed on head-mounted displays. Such includes an interface having a minimal form factor that enables sophisticated control interactions to be carried out in an inconspicuous, substantially hands-free manner. All of this is achieved with a multi-modal and fully-wearable interface that combines substantially concurrent inputs from both an eye tracker and a wireless portable interface device.
  • FIG. 1 is an illustrative diagram showing an exemplary system 100 that enables interaction with content displayed on a head-mounted display (“HMD”) 110 in accordance with the present systems, devices, and methods.
  • the HMD 110 of system 100 includes at least one display 111 (two such displays illustrated in FIG. 1 ) positioned in the field of view of at least one eye of a user when HMD 110 is worn on the user's head.
  • One or more display(s) 111 may employ one or more waveguide(s), one or more microdisplay(s), and/or any or all of the display technologies described in US Patent Application Publication 2015-0205134, U.S. Non-Provisional patent application Ser. No. 14/749,341 (now US Patent Application Publication 2015-0378164), U.S.
  • Non-Provisional patent application Ser. No. 14/749,351 now US Patent Application Publication 2015-0378161
  • U.S. Non-Provisional patent application Ser. No. 14/749,359 now US Patent Application Publication 2015-0378162
  • U.S. Provisional Patent Application Ser. No. 62/117,316 now US Patent Application Publication 2016-0274365
  • U.S. Provisional Patent Application Ser. No. 62/156,736 now U.S. Non-Provisional patent application Ser. Nos. 15/145,576, 15/145,609, and 15/145,583).
  • HMD 110 also includes a processor 112 communicatively coupled to the at least one display 111 and a non-transitory processor-readable storage medium or memory 112 communicatively coupled to processor 112 .
  • Memory 113 stores data and/or instructions 114 (i.e., processor-executable instructions) that, when executed by processor 112 of HMD 110 , cause the at least one display 111 to display at least one object 115 that is responsive to a selection operation performed by the user.
  • HMD 110 also includes a receiver 116 (e.g., a wireless receiver, or a wireless transceiver including a wireless receiver) operative to wirelessly receive signals.
  • Receiver 116 is communicatively coupled to processor 112 .
  • System 100 further includes an eye-tracker 117 that is operative to detect the eye position and/or gaze direction of at least one eye the user and communicatively coupled to processor 112 .
  • Eye-tracker 117 includes at least one camera or photodetector to measure light (e.g., visible light or infrared light) reflected from the at least one eye and processor 112 may determine the eye position or gaze direction of the at least one eye based on the measured reflections.
  • Eye-tracker 117 may implement the technology described in U.S. Provisional Patent Application Ser. No. 62/167,767 (now U.S. Non-Provisional patent application Ser. Nos. 15/167,458 and 15/167,472).
  • eye-tracker 117 is operative to detect that the user is gazing at (e.g., looking, staring or generally pointing his or her eye(s) in the direction of) the at least one object 115 displayed by the at least one display 111 .
  • eye-tracker 117 is carried by HMD 110 , though in alternative implementations eye-tracker 117 may be physically separate from HMD 110 .
  • System 100 provides a multi-modal interface for interacting with content displayed on a head-mounted display.
  • a first mode of interaction i.e., via eye position and/or gaze direction
  • system 100 further includes a wireless portable interface device 120 with a form factor to be carried by or on the user.
  • portable interface device 120 has the general size and geometry of a ring to be worn on a finger or thumb of the user.
  • portable interface device 120 may be a wristband or an armband, or may adopt a non-annular form factor that clips, sticks, or otherwise attaches to the user or the user's clothing (e.g., a pen with a clip).
  • Portable interface device 120 is physically separate from HMD 110 and includes at least one actuator 121 (e.g., a button, switch, toggle, lever, dial, or similar component) that, when activated by the user, causes portable interface device 120 to wirelessly transmit a signal from a wireless signal generator 122 .
  • actuator 121 e.g., a button, switch, toggle, lever, dial, or similar component
  • Portable interface device 120 may include a portable power source, such as a battery or a supercapacitor (i.e., capacitor with capacitance on the order of 0.01 F or greater).
  • portable interface device 120 may be “batteryless.”
  • the term “batteryless” literally means “without any battery or batteries” (or any other equivalent device providing a similar function, such as a supercapacitor) and is generally used to indicate that the corresponding device (e.g., portable interface device 120 ) has no on-board battery or other source of pre-stored (i.e., generated off of the device and stored therein) power.
  • Portable interface device 120 is generally described as a wireless device.
  • the term “wireless” literally means “without any external wire-connections to anything” and is generally used to indicate that the corresponding device (e.g., portable interface device 120 ) is untethered with no external wire-connection(s) (or optical fiber connections, or cable connections, etc.) to any other electronic device or to any source of electric power.
  • portable interface device 120 is both batteryless and wireless, then in the absence of any actuation (as described in more detail later on), portable interface device 120 is generally without any electric power.
  • Wearable electronic devices are typically larger and bulkier than other wearable accessories, such as traditional jewelry.
  • portable interface device 120 is wireless (and may, in some implementations, be batteryless) in order to remove the large and bulky electric components (e.g., a battery and/or a charging port, if batteryless) and provide a small and compact form factor not typically seen among wearable electronic devices.
  • portable interface device 120 may still operate electrically using electric signals generated upon mechanical actuation by, for example, one or more on-board piezoelectric component(s).
  • Portable interface device 120 only includes one actuator or “button” 121 .
  • Other implementations may include a second and even a third actuator, but in general portable interface device 120 includes very few actuators in order to minimize its form factor.
  • actuator 121 may provide a “select” function in combination with whatever the user is gazing at on at least one display 111 of HMD 110 as detected by eye-tracker 117 and determined by processor 112 .
  • memory 113 of HMD 110 stores processor-executable instructions and/or data 114 that, when executed by processor 112 of HMD 110 , cause the at least one display 111 to display at least one object 115 that is responsive to a selection operation performed by the user.
  • the selection operation performed by the user may comprise a substantially concurrent combination of gazing at the least one object 115 displayed by the at least one display 111 (as detected by eye-tracker 117 ) and activating the at least actuator 121 of the portable interface device 120 .
  • the selection operation may be effected by HMD 110 (e.g., by processor 112 of HMD 110 ) in response to receipt of a wireless “selection signal” 150 at receiver 116 transmitted from wireless signal generator 122 of portable interface device 120 , and the selection operation may include “selecting” whatever object 115 on display 111 that eye tracker 117 identifies the user is looking/gazing at when the wireless selection signal 150 is receiver at receiver 116 .
  • processor 112 executes processor-executable instructions and/or data 114 stored in memory 113 , which cause processor 112 to: i) request current gaze direction data from eye-tracker 117 ; ii) identify a particular object 115 at which the user is gazing based on the current gaze direction data received from eye-tracker 117 (e.g., the particular object identified among at least one object displayed by at least one display 111 ); and iii) cause at least one display 111 to display the visual effect on the particular object 115 .
  • a visual effect may be displayed or rendered on the at least one display 111 of HMD 110 .
  • the visual effect may include: highlighting the object 115 , visually changing or modifying the object 115 , displaying new or changed content elsewhere on the display 111 , or changing other aspects of the displayed content (including replacing the displayed content with new displayed content) based on the object 115 selected by the user.
  • the object 115 may be any displayed image depending on the specific application and/or user interface, including without limitation: a menu item, a graphical button, a keyboard key, a notification, one of multiple objects displayed by the at least one display 111 of HMD 110 , a file, a folder, and/or an alphanumeric character.
  • display 111 displays a representation of a virtual keyboard and the specific object 115 selected by the user corresponds to a specific key (i.e., letter) of the keyboard. Based on a typical qwerty keyboard configuration, the position of the selected key/letter corresponds to the letter “T.”
  • system 100 may be used to enable the user to type by: i) displaying a virtual keyboard on at least one display 111 and, over a number of instances: ii) detecting which letter the user is gazing at by eye-tracker 117 , and iii) selecting the letter that the user is gazing at when the user activates actuator 121 of portable interface device 120 .
  • the form factor of wearable electronic devices is a very important consideration in their design and can ultimately determine whether or not a wearable electronic device will be adopted by users.
  • the present systems, devices, and methods provide a portable interface device 120 for interacting with content displayed on a HMD 110 , where the portable interface device 120 has minimal bulk associated with its technological capabilities and may approximate the form factor of traditional jewelry or other accessories.
  • This aspect of the portable interface device 120 is enabled by making the portable interface device 120 wireless and, in some implementations, batteryless.
  • portable interface device 120 Even batteryless and wireless implementations of portable interface device 120 may still be used to wirelessly transmit signals to HMD 110 .
  • the present systems, devices, and methods provide two example configurations that enable portable interface device 120 to wirelessly transmit signals despite being a batteryless and wireless device.
  • portable interface device 120 may include a piezoelectric element communicatively coupled to actuator 121 and a radio frequency antenna 122 communicatively coupled to the piezoelectric element.
  • actuator 121 When activated (e.g., pressed, pushed, depressed, switched, or similar) by the user, actuator 121 mechanically actuates the piezoelectric element. In response to the mechanical actuation, the piezoelectric element generates an electric signal. This electric signal is communicatively coupled to radio frequency antenna 122 where, in response to the electric signal, antenna 122 wirelessly transmits a wireless (e.g., radio frequency) signal.
  • a wireless e.g., radio frequency
  • Antenna 122 may be designed to wirelessly transmit a radio frequency or microwave frequency signal 150 having a specific frequency or within a specific range of frequencies.
  • receiver 116 of HMD 110 includes a radio frequency or microwave receiver that is advantageously tuned to be responsive to radio or microwave signals in the range of signal 150 wirelessly transmitted by antenna 122 of portable interface device 120 .
  • portable interface device 120 may include a mechanical resonator 122 physically coupled to actuator 121 .
  • actuator 121 When activated (e.g., pressed, pushed, depressed, switched, or similar) by the user, actuator 121 mechanically actuates (e.g., strikes, impacts, oscillates, vibrates, or similar) mechanical resonator 122 .
  • mechanical resonator 122 In response to the mechanical actuation, mechanical resonator 122 generates a sonic, acoustic, or aural signal 150 of a specific frequency (or in a specific range of frequencies).
  • the sonic signal may be an ultrasonic signal.
  • receiver 116 of HMD 110 includes a microphone and/or a piezoelectric element that may be tuned to be responsive to sonic signals in the range of sonic signal 150 wirelessly transmitted by mechanical resonator 122 of portable interface device 120 .
  • a signal 150 that is “wirelessly transmitted” may exist in a variety of different forms, including without limitation: a radio frequency signal, a sonic signal (such as an ultrasonic signal), an optical signal (generated by a mechanoluminescent material, such as a piezoluminescent alkali halide or a triboluminescent mineral), a photonic signal, a thermal signal, and so on.
  • a signal that is “wirelessly transmitted” is any signal that is transmitted through any medium other than a conductive wire.
  • portable interface device 120 may be configured to transmit (and receiver 116 of HMD 110 may be configured to receive) more conventional wireless signals, such as short-wavelength radio wave signals in the range of 10 MHz-10 GHz.
  • wireless signal generator 122 may include a wireless transmitter (e.g., a wireless transceiver including a wireless transmitter) designed and operated to transmit (and receiver 116 may be designed and operated to receive) wireless signals using an established wireless communication protocol, including without limitation: Bluetooth®, Bluetooth® Low-Energy, Bluetooth Smart®, ZigBee®, WiFi®, Near-Field Communication (NFC), or the like.
  • a wireless transmitter e.g., a wireless transceiver including a wireless transmitter
  • receiver 116 may be designed and operated to receive wireless signals using an established wireless communication protocol, including without limitation: Bluetooth®, Bluetooth® Low-Energy, Bluetooth Smart®, ZigBee®, WiFi®, Near-Field Communication (NFC), or the like.
  • FIG. 2 is an illustrative diagram showing a human-electronics interface 200 in which a user 201 wears a system that enables the user to interact with displayed content in accordance with the present systems, devices, and methods.
  • the system comprises a HMD 210 and a portable interface device 220 .
  • HMD 210 is substantially similar to HMD 110 from FIG. 1 and portable interface device 220 is substantially similar to portable interface device 120 from FIG. 1 .
  • portable interface device 220 is substantially similar to portable interface device 120 from FIG. 1 .
  • portable interface device 220 is shown having the form factor of a ring and worn on a finger of user 201 ; however, in alternative implementations portable interface device 220 may adopt a different form factor and be worn elsewhere on/by user 201 , such as a wristband, an armband, or a device that clips, affixes, or otherwise couples to user 201 or to an article of clothing worn by user 201 . In general, it is advantageous for the actuator ( 121 in FIG. 1 , not visible in FIG. 2 ) of portable interface device 220 to be easily and inconspicuously accessible to user 201 .
  • an actuator on portable interface device 220 may be easily and inconspicuously activated by the adjacent thumb of user 201 .
  • activation of the actuator causes portable interface device 220 to wirelessly transmit a signal 250 (e.g., a radio frequency signal, a sonic signal such as an ultrasonic signal, an optical or photonic signal, or similar) and HMD 210 includes a receiver that wirelessly receives signal 250 .
  • a signal 250 e.g., a radio frequency signal, a sonic signal such as an ultrasonic signal, an optical or photonic signal, or similar
  • HMD 210 includes a receiver that wirelessly receives signal 250 .
  • HMD 210 If signal 250 is received by HMD 210 while an eye-tracker on HMD 210 detects that user 201 is gazing at an object displayed on HMD 210 that is responsive to a selection operation, then the combination of user 201 gazing at the object displayed by HMD 210 and substantially concurrently activating the actuator of portable interface device 220 effects the selection operation. In response to the selection operation, HMD 210 may display a visual effect to user 201 .
  • FIG. 3 is a flow-diagram showing an exemplary method 300 of operating a system in accordance with the present systems, devices, and methods.
  • the system e.g., substantially similar to system 100 from FIG. 1
  • the system comprises a HMD (e.g., 110 ), an eye-tracker (e.g., 117 ), and a portable interface device (e.g., 120 ), the portable interface device ( 120 ) physically separate from the HMD ( 110 ).
  • the portable interface device is a wireless device and, in some implementations, may be a batteryless device.
  • Method 300 includes three acts 310 , 320 , and 330 , though act 320 further includes four sub-acts 321 a , 321 b , 322 , and 323 .
  • act 320 further includes four sub-acts 321 a , 321 b , 322 , and 323 .
  • certain acts/sub-acts may be omitted and/or additional acts/sub-acts may be added.
  • the illustrated order of the acts/sub-acts is shown for exemplary purposes only and may change in alternative embodiments.
  • At 310 at least one display ( 111 ) of the HMD ( 110 ) displays an object ( 115 ) within the field of view of at least one eye of the user.
  • the object ( 115 ) is responsive to a selection operation performed by the user and may, as previously described, include without limitation: a menu item, a graphical button, a keyboard key, a notification, one of multiple objects displayed by the at least one display ( 111 ) of the HMD ( 110 ), a file, a folder, and/or an alphanumeric character.
  • the system ( 100 ) receives a selection operation from the user.
  • the selection operation performed by the user may comprise a combination of two substantially concurrent portions: i) the user gazing at the at least one object ( 115 ) displayed by the at least one display ( 111 ) per 310 and ii) the user activating at least one actuator of the portable interface device ( 120 ).
  • the selection operation involves multiple inputs of different modes and communication between different devices in system ( 100 ).
  • Act 320 comprises sub-acts 321 a , 321 b , 322 , and 323 that collectively define the receipt of the components of the selection operation by and between the components of the system ( 100 ).
  • the eye-tracker ( 117 ) of the system ( 100 ) detects that the user is gazing at the object ( 115 ) displayed at 310 .
  • Sub-act 321 a provides a first portion of act 320 . More specifically, at sub-act 321 a the system ( 100 ) receives a first portion of the selection operation that is received at 320 , that first portion corresponding to the “user gazing at the object ( 115 )” portion of the selection operation detected and/or determined by the eye tracker ( 117 ).
  • the portable interface device ( 120 ) of the system ( 100 ) receives a second portion of the selection operation an activation of at least one actuator ( 121 ) by the user).
  • the portable interface device ( 120 ) may receive an activation of at least one actuator ( 121 ) from the user by, for example, having at least one actuator ( 121 ) be activated by the user.
  • Sub-acts 321 a and 321 b are connected by a horizontal arrow in FIG. 3 to indicate that the two sub-acts are substantially concurrent (i.e., sub-acts 321 a and 321 b are performed substantially concurrently).
  • the user activates the actuator ( 121 ) of the portable interface device ( 120 ) per 321 b while the user is gazing at an object ( 115 ) that the user wishes to select, as detected by the eye-tracker ( 117 ) per 321 a .
  • This combination of actions performed by the user e.g. gazing while activating the actuator
  • a transmitter or signal generator ( 122 ) of the portable interface device ( 120 ) wirelessly transmits a signal ( 150 ) in response to the at least one actuator ( 121 ) being activated by the user at 321 b.
  • a receiver ( 116 ) of the HMD ( 110 ) wirelessly receives the signal ( 150 ) wirelessly transmitted by the portable interface device ( 120 ) at 322 .
  • the combination of sub-acts 321 b , 322 , and 323 provides a second portion of act 320 .
  • the system ( 100 ) receives, via the portable interface device ( 120 ), a second portion of the selection operation that is received at 320 , that second portion being the “user activating an actuator” portion of the selection operation provided by the portable interface device ( 120 ). With the selection operation thus received, method 300 proceeds to act 330 .
  • the at least one display ( 111 ) of the HMD ( 110 ) displays or renders a visual effect in response to receiving the selection operation from the user per 320 .
  • the visual effect may include, as non-limiting examples, any or all of: highlighting the object ( 115 ), visually changing or modifying the object ( 115 ), displaying new or changed content elsewhere on the display ( 111 ), or changing other aspects of the displayed content (including replacing the displayed content with new displayed content) based on the object ( 115 ) selected by the user.
  • the HMD ( 110 ) of the system ( 100 ) may include a processor ( 112 ) communicatively coupled to the at least one display ( 111 ) and a non-transitory processor-readable storage medium or memory ( 113 ) communicatively coupled to the processor ( 112 ).
  • the memory ( 113 ) may store data and/or instructions (i.e., processor-executable instructions 114 ) that, when executed by the processor ( 112 ), cause: i) the at least one display ( 111 ) of the HMD ( 110 ) to display the object ( 115 ) within the field of view of at least one eye of the user per act 310 , and ii) the at least one display ( 111 ) of the HMD ( 110 ) to display the visual effect to the user per act 330 in response to the system ( 100 ) receiving the selection operation from the user per act 320 .
  • data and/or instructions i.e., processor-executable instructions 114
  • the portable interface device ( 120 ) of the system ( 100 ) may be implemented in a variety of different ways. Further details of sub-acts 321 b , 322 , and 323 of method 300 may depend on the nature of the portable interface device ( 12 ) being implemented.
  • the portable interface device ( 120 ) may comprise a batteryless and wireless portable interface device that includes a piezoelectric element communicatively coupled to the actuator ( 121 ) and a radio frequency antenna ( 122 ) communicatively coupled to the piezoelectric element and tuned to wirelessly transmit radio frequency signals ( 150 ) of a specific frequency or in a specific range of frequencies.
  • the receiver ( 116 ) of the HMD ( 110 ) may comprise a radio frequency receiver ( 116 ) tuned to wirelessly receive radio frequency signals ( 150 ) of the specific frequency or in the specific range of frequencies.
  • the portable interface device ( 120 ) may receive a mechanical actuation of the piezoelectric element from the user at sub-act 321 b of method 300 .
  • the portable interface device ( 120 ) may receive or experience a mechanical actuation of the piezoelectric element from the user by, for example, having the piezoelectric element be mechanically actuated by the user.
  • the piezoelectric element may generate an electric signal that communicatively couples to the radio frequency antenna ( 122 ) of the portable interface device ( 120 ), and in response to this electric signal the radio frequency antenna may wirelessly transmit a radio frequency signal ( 150 ) at sub-act 322 of method 300 .
  • the radio frequency receiver ( 116 ) of the HMD ( 110 ) may wirelessly receive the radio frequency signal ( 150 ).
  • the portable interface device ( 120 ) may comprise a batteryless and wireless portable interface device that includes a mechanical resonator ( 122 ) physically coupled to the actuator ( 121 ) and tuned to wirelessly transmit sonic, acoustic, or aural signals ( 150 ), such as ultrasonic signals, of a specific frequency or in a specific range of frequencies.
  • the receiver ( 116 ) of the HMD ( 110 ) may comprise a microphone and/or a piezoelectric element tuned to be responsive to sonic signals ( 150 ) of the specific frequency or in the specific range of frequencies.
  • the portable interface device ( 120 ) may receive a mechanical actuation of the mechanical resonator ( 122 ) from the user at sub-act 321 b of method 300 .
  • the portable interface device ( 120 ) may receive or experience a mechanical actuation of the mechanical resonator ( 122 ) from the user by, for example, having the mechanical resonator ( 122 ) be mechanically actuated by the user.
  • the mechanical resonator ( 122 ) may generate a sonic signal ( 150 ), such as an ultrasonic signal, which is wirelessly transmitted at sub-act 322 of method 300 .
  • the microphone and/or tuned piezoelectric element ( 116 ) of the HMD ( 110 ) may wirelessly receive the sonic signal ( 150 ).
  • the portable interface device ( 120 ) may comprise a wireless portable interface device that includes an on-board power source, such as a battery or a supercapacitor (and either a wireless charging coil or a tethered connector port for charging said on-board power source).
  • an on-board power source such as a battery or a supercapacitor (and either a wireless charging coil or a tethered connector port for charging said on-board power source).
  • the portable interface device ( 120 ) may include a wireless transmitter ( 122 ; e.g., an antenna and/or a wireless transceiver that includes a wireless transmitter) electrically coupled to the actuator ( 121 ) and operative to wirelessly transmit radio frequency signals ( 150 ) that embody an established wireless communication protocol, such as without limitation: Bluetooth®, Bluetooth® Low-Energy, Bluetooth Smart®, ZigBee®, WiFi®, Near-Field Communication (NFC), or the like.
  • an established wireless communication protocol such as without limitation: Bluetooth®, Bluetooth® Low-Energy, Bluetooth Smart®, ZigBee®, WiFi®, Near-Field Communication (NFC), or the like.
  • Such protocols typically employ radio frequency signals in the range of 1 GHz to 10 GHz (with the exception of NFC, which operates in the 10 MHz-20 MHz range) and may include pairing or otherwise establishing a wireless communicative link between the portable interface device ( 120 ) and the HMD ( 110 ).
  • the receiver ( 116 ) of the HMD ( 110 ) may comprise a wireless receiver or antenna (e.g., a wireless transceiver that includes a wireless receiver or antenna) tuned to be responsive to radio frequency signals ( 150 ) of the specific frequency or in the specific range of frequencies transmitted by the wireless transmitter ( 122 ) of the portable interface device ( 120 ).
  • the portable interface device ( 120 ) may receive an actuation of the actuator ( 121 ) from the user at sub-act 321 b of method 300 .
  • the portable interface device ( 120 ) may receive or experience an actuation of the actuator ( 121 ) from the user by, for example, having the actuator ( 121 ) be physically actuated (e.g., depressed, switched, twisted, dialed, or similar, depending on the nature of the specific actuator) by the user.
  • the wireless transmitter ( 122 ) may generate a radio frequency signal ( 150 ) which is wirelessly transmitted at sub-act 322 of method 300 .
  • the wireless receiver ( 116 ) of the HMD ( 110 ) may wirelessly receive the radio frequency signal ( 150 ).
  • the multi-modal interface described herein (comprising an eye tracker and a wireless actuator or button, both of which communicate with a wearable heads-up display) enables sophisticated control of and/or interactions with content displayed by the wearable heads-up display in an inconspicuous, substantially hands-free manner and has a minimal form factor.
  • the sophisticated and inconspicuous aspects of the control interactions are achieved, at least in part, by using the eyes (based on data provided by the eye tracker) to carry out the pointing and “identifying one among many” tasks that can be difficult to perform using hand-controlled interfaces.
  • the eye (more particularly, the gaze thereof) is able to scan over and hone in on aspects of displayed content much more quickly and easily than a hand- or finger-controlled cursor; however, actually specifying a selection operation with the eye alone can be cumbersome because: a) the user is likely to inadvertently gaze at something he or she does not wish to actually select, and b) the mechanisms for doing so, such as a deliberate blink or an extended dwell time, are impractical and lead to an unpleasant user experience.
  • the present systems, devices, and methods take advantage of the versatility and scanning/honing capabilities of the eye/gaze but avoid the specification/selection issues by employing a secondary input mode, a simple wearable actuator such as ring-based button, to actuate the specification/selection function.
  • a simple wearable actuator such as ring-based button
  • This wearable actuator is too simplistic to enable sophisticated control interactions on its own, but when used in conjunction with an eye tracker the resulting interface enjoys the best features of both modes.
  • the wearable actuator is similarly inconspicuous and it is “substantially hands-free” because it does not need to be carried by or held in the user's hand.
  • the user may actuate the button simply using his or her thumb and this action may be performed while the user's hand or hands is/are also devoted to another task, such as carrying something(s).
  • the single-button actuator is also extremely low power (even batteryless in some implementations) and its simplicity enables very compact, minimal form factor designs.
  • the various embodiments described herein provide a multi-modal, portable, and completely wearable interface that enables a user to perform sophisticated interactions with content displayed on a wearable heads-up display in an inconspicuous, substantially hands-free manner.
  • the eye tracker component is integrated into the wearable heads-up display (even integrated with the projection elements of the wearable heads-up display, as described in U.S. Provisional Patent Application Ser. No. 62/167,767, now U.S. Non-Provisional patent application Ser. Nos. 15/167,458 and 15/167,472) and the wireless actuator component is a wearable element, such as a ring-based button, the entire system provides a minimal and inconspicuous form factor.
  • a user wearing the multi-modal interface described herein may be substantially indistinguishable from the same user wearing a conventional pair of eyeglasses and a conventional ring on his or her finger.
  • Such is achieved, at least in part, by: the compact form factor of the wearable heads-up display; the compact form factor of the wearable (e.g., ring-based) actuator; integration of the eye tracker with the wearable heads-up display; and/or the wireless communication between the wearable actuator and the wearable heads-up display.
  • infinitive verb forms are often used. Examples include, without limitation: “to detect,” “to provide,” “to transmit,” “to communicate,” “to process,” “to route,” and the like. Unless the specific context requires otherwise, such infinitive verb forms are used in an open, inclusive sense, that is as “to, at least, detect,” “to, at least, provide,” “to, at least, transmit,” and so on.
  • logic or information can be stored on any processor-readable medium for use by or in connection with any processor-related system or method.
  • a memory is a processor-readable medium that is an electronic, magnetic, optical, or other physical device or means that contains or stores a computer and/or processor program.
  • Logic and/or the information can be embodied in any processor-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor-containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions associated with logic and/or information.
  • a “non-transitory processor-readable medium” can be any element that can store the program associated with logic and/or information for use by or in connection with the instruction execution system, apparatus, and/or device.
  • the processor-readable medium can be, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device.
  • the computer readable medium would include the following: a portable computer diskette (magnetic, compact flash card, secure digital, or the like), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM, EEPROM, or Flash memory), a portable compact disc read-only memory (CDROM), digital tape, and other non-transitory media.
  • a portable computer diskette magnetic, compact flash card, secure digital, or the like
  • RAM random access memory
  • ROM read-only memory
  • EPROM erasable programmable read-only memory
  • CDROM compact disc read-only memory
  • digital tape digital tape

Landscapes

  • Engineering & Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Human Computer Interaction (AREA)
  • Optics & Photonics (AREA)
  • User Interface Of Digital Computer (AREA)
US15/282,535 2015-10-01 2016-09-30 Systems, devices, and methods for interacting with content displayed on head-mounted displays Abandoned US20170097753A1 (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
US15/282,535 US20170097753A1 (en) 2015-10-01 2016-09-30 Systems, devices, and methods for interacting with content displayed on head-mounted displays
US15/799,642 US10656822B2 (en) 2015-10-01 2017-10-31 Systems, devices, and methods for interacting with content displayed on head-mounted displays

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US201562236060P 2015-10-01 2015-10-01
US15/282,535 US20170097753A1 (en) 2015-10-01 2016-09-30 Systems, devices, and methods for interacting with content displayed on head-mounted displays

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US15/799,642 Continuation US10656822B2 (en) 2015-10-01 2017-10-31 Systems, devices, and methods for interacting with content displayed on head-mounted displays

Publications (1)

Publication Number Publication Date
US20170097753A1 true US20170097753A1 (en) 2017-04-06

Family

ID=58427931

Family Applications (2)

Application Number Title Priority Date Filing Date
US15/282,535 Abandoned US20170097753A1 (en) 2015-10-01 2016-09-30 Systems, devices, and methods for interacting with content displayed on head-mounted displays
US15/799,642 Active US10656822B2 (en) 2015-10-01 2017-10-31 Systems, devices, and methods for interacting with content displayed on head-mounted displays

Family Applications After (1)

Application Number Title Priority Date Filing Date
US15/799,642 Active US10656822B2 (en) 2015-10-01 2017-10-31 Systems, devices, and methods for interacting with content displayed on head-mounted displays

Country Status (3)

Country Link
US (2) US20170097753A1 (fr)
CA (1) CA3007196A1 (fr)
WO (1) WO2017059285A1 (fr)

Cited By (40)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9766449B2 (en) 2014-06-25 2017-09-19 Thalmic Labs Inc. Systems, devices, and methods for wearable heads-up displays
US9904051B2 (en) 2015-10-23 2018-02-27 Thalmic Labs Inc. Systems, devices, and methods for laser eye tracking
US9958682B1 (en) 2015-02-17 2018-05-01 Thalmic Labs Inc. Systems, devices, and methods for splitter optics in wearable heads-up displays
US9989764B2 (en) 2015-02-17 2018-06-05 Thalmic Labs Inc. Systems, devices, and methods for eyebox expansion in wearable heads-up displays
USD821391S1 (en) * 2015-12-04 2018-06-26 Samsung Electronics Co., Ltd. Wearable device
US10073268B2 (en) 2015-05-28 2018-09-11 Thalmic Labs Inc. Display with integrated visible light eye tracking
US10126815B2 (en) 2016-01-20 2018-11-13 Thalmic Labs Inc. Systems, devices, and methods for proximity-based eye tracking
US10133075B2 (en) 2015-05-04 2018-11-20 Thalmic Labs Inc. Systems, devices, and methods for angle- and wavelength-multiplexed holographic optical elements
US10151926B2 (en) 2016-01-29 2018-12-11 North Inc. Systems, devices, and methods for preventing eyebox degradation in a wearable heads-up display
US20190050062A1 (en) * 2017-08-10 2019-02-14 Google Llc Context-sensitive hand interaction
US10215987B2 (en) 2016-11-10 2019-02-26 North Inc. Systems, devices, and methods for astigmatism compensation in a wearable heads-up display
US10230929B2 (en) 2016-07-27 2019-03-12 North Inc. Systems, devices, and methods for laser projectors
WO2019119154A1 (fr) 2017-12-22 2019-06-27 North Inc. Conceptions d'antenne destinées à des dispositifs d'affichage tête-haute portables
WO2019136439A1 (fr) * 2018-01-08 2019-07-11 Facebook Technologies, Llc Procédés, dispositifs et systèmes pour créer des stimulations haptiques et suivre un mouvement d'utilisateur
US10365492B2 (en) 2016-12-23 2019-07-30 North Inc. Systems, devices, and methods for beam combining in wearable heads-up displays
US10365550B2 (en) 2016-04-13 2019-07-30 North Inc. Systems, devices, and methods for focusing laser projectors
US10409057B2 (en) 2016-11-30 2019-09-10 North Inc. Systems, devices, and methods for laser eye tracking in wearable heads-up displays
US10437074B2 (en) 2017-01-25 2019-10-08 North Inc. Systems, devices, and methods for beam combining in laser projectors
US10459221B2 (en) 2016-08-12 2019-10-29 North Inc. Systems, devices, and methods for variable luminance in wearable heads-up displays
US10488662B2 (en) 2015-09-04 2019-11-26 North Inc. Systems, articles, and methods for integrating holographic optical elements with eyeglass lenses
US10528135B2 (en) 2013-01-14 2020-01-07 Ctrl-Labs Corporation Wearable muscle interface systems, devices and methods that interact with content displayed on an electronic display
US10657927B2 (en) * 2016-11-03 2020-05-19 Elias Khoury System for providing hands-free input to a computer
US10684692B2 (en) 2014-06-19 2020-06-16 Facebook Technologies, Llc Systems, devices, and methods for gesture identification
US10739869B1 (en) * 2018-05-01 2020-08-11 Facebook Technologies, Llc Apparatus, system, and method for controlling artificial-reality environments
US10802190B2 (en) 2015-12-17 2020-10-13 Covestro Llc Systems, devices, and methods for curved holographic optical elements
US10901216B2 (en) 2017-10-23 2021-01-26 Google Llc Free space multiple laser diode modules
US11055056B1 (en) * 2018-09-25 2021-07-06 Facebook Technologies, Llc Split system for artificial reality
CN113196673A (zh) * 2018-12-18 2021-07-30 美国斯耐普公司 自适应眼戴设备天线
US11079846B2 (en) 2013-11-12 2021-08-03 Facebook Technologies, Llc Systems, articles, and methods for capacitive electromyography sensors
US11080417B2 (en) * 2018-06-26 2021-08-03 Google Llc Private eye-to-eye communications with wearable heads up display
US11467670B2 (en) 2018-03-23 2022-10-11 Meta Platforms Technologies, Llc Methods, devices, and systems for displaying a user interface on a user and detecting touch gestures
US11635736B2 (en) 2017-10-19 2023-04-25 Meta Platforms Technologies, Llc Systems and methods for identifying biological structures associated with neuromuscular source signals
US11644799B2 (en) 2013-10-04 2023-05-09 Meta Platforms Technologies, Llc Systems, articles and methods for wearable electronic devices employing contact sensors
US11666264B1 (en) 2013-11-27 2023-06-06 Meta Platforms Technologies, Llc Systems, articles, and methods for electromyography sensors
US11797087B2 (en) 2018-11-27 2023-10-24 Meta Platforms Technologies, Llc Methods and apparatus for autocalibration of a wearable electrode sensor system
US11822090B2 (en) * 2014-01-24 2023-11-21 Mentor Acquisition One, Llc Haptic systems for head-worn computers
US11868531B1 (en) 2021-04-08 2024-01-09 Meta Platforms Technologies, Llc Wearable device providing for thumb-to-finger-based input gestures detected based on neuromuscular signals, and systems and methods of use thereof
US11907423B2 (en) 2019-11-25 2024-02-20 Meta Platforms Technologies, Llc Systems and methods for contextualized interactions with an environment
US11921471B2 (en) 2013-08-16 2024-03-05 Meta Platforms Technologies, Llc Systems, articles, and methods for wearable devices having secondary power sources in links of a band for providing secondary power in addition to a primary power source
US11961494B1 (en) 2019-03-29 2024-04-16 Meta Platforms Technologies, Llc Electromagnetic interference reduction in extended reality environments

Families Citing this family (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10579099B2 (en) * 2018-04-30 2020-03-03 Apple Inc. Expandable ring device
US11314281B2 (en) 2018-09-10 2022-04-26 Apple Inc. Display system
DE102018215850A1 (de) * 2018-09-18 2020-03-19 Bayerische Motoren Werke Aktiengesellschaft Datenbrille mit Sicherheitsmechanismus
CN111309142A (zh) 2018-12-11 2020-06-19 托比股份公司 用于切换显示设备的输入模态的方法和设备
US11275250B2 (en) 2019-11-19 2022-03-15 Apple Inc. Optical alignment for head-mountable device
US11695758B2 (en) * 2020-02-24 2023-07-04 International Business Machines Corporation Second factor authentication of electronic devices

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6700310B2 (en) * 2000-10-13 2004-03-02 Lear Corporation Self-powered wireless switch
US20120249797A1 (en) * 2010-02-28 2012-10-04 Osterhout Group, Inc. Head-worn adaptive display
US20130169560A1 (en) * 2012-01-04 2013-07-04 Tobii Technology Ab System for gaze interaction
US20130300652A1 (en) * 2011-11-30 2013-11-14 Google, Inc. Unlocking a Screen Using Eye Tracking Information
US20140198035A1 (en) * 2013-01-14 2014-07-17 Thalmic Labs Inc. Wearable muscle interface systems, devices and methods that interact with content displayed on an electronic display
US20150049112A1 (en) * 2013-08-19 2015-02-19 Qualcomm Incorporated Automatic customization of graphical user interface for optical see-through head mounted display with user interaction tracking
US20150277559A1 (en) * 2014-04-01 2015-10-01 Apple Inc. Devices and Methods for a Ring Computing Device
US20170083115A1 (en) * 2014-05-16 2017-03-23 Faindu Gmbh Method for displaying a virtual interaction on at least one screen and input device, system and method for a virtual application by means of a computing unit

Family Cites Families (121)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US3408133A (en) 1964-01-23 1968-10-29 Electro Optical Systems Inc Kerr-cell camera shutter
US3712716A (en) 1971-04-09 1973-01-23 Stanford Research Inst Eye tracker
JPS61198892A (ja) 1985-02-27 1986-09-03 Nec Corp 表示装置
WO1989001756A1 (fr) 1987-08-26 1989-03-09 El Hage Sami G Appareil permettant de determiner le contour de la cornee d'un oeil humain
US5231674A (en) 1989-06-09 1993-07-27 Lc Technologies, Inc. Eye tracking method and apparatus
US5103323A (en) 1990-04-18 1992-04-07 Holographic Optics, Inc. Multi-layer holographic notch filter
JP3151766B2 (ja) 1992-07-31 2001-04-03 キヤノン株式会社 画像表示装置
US5467104A (en) 1992-10-22 1995-11-14 Board Of Regents Of The University Of Washington Virtual retinal display
US6008781A (en) 1992-10-22 1999-12-28 Board Of Regents Of The University Of Washington Virtual retinal display
US5596339A (en) 1992-10-22 1997-01-21 University Of Washington Virtual retinal display with fiber optic point source
JP3141737B2 (ja) 1995-08-10 2001-03-05 株式会社セガ 仮想画像生成装置及びその方法
US6184847B1 (en) 1998-09-22 2001-02-06 Vega Vista, Inc. Intuitive control of portable data displays
US5742421A (en) 1996-03-01 1998-04-21 Reflection Technology, Inc. Split lens video display system
US5959622A (en) * 1996-05-31 1999-09-28 Intel Corporation Still image capture under computer control in response to user-instructed trigger
KR100227179B1 (ko) 1997-04-11 1999-10-15 박호군 고품질 반사형 홀로그래픽 광학 소자 제작 장치
JPH10319240A (ja) 1997-05-22 1998-12-04 Fuji Xerox Co Ltd ヘッドマウンテッドディスプレイ
US6027216A (en) 1997-10-21 2000-02-22 The Johns University School Of Medicine Eye fixation monitor and tracker
US6043799A (en) 1998-02-20 2000-03-28 University Of Washington Virtual retinal display with scanner array for generating multiple exit pupils
US7640007B2 (en) 1999-02-12 2009-12-29 Fisher-Rosemount Systems, Inc. Wireless handheld communicator in a process control environment
US6972734B1 (en) 1999-06-11 2005-12-06 Canon Kabushiki Kaisha Mixed reality apparatus and mixed reality presentation method
CA2375519A1 (fr) 1999-06-21 2000-12-28 The Microoptical Corporation Systeme a lentille de visualisation faisant appel a une configuration optique desaxee
US6924476B2 (en) 2002-11-25 2005-08-02 Microvision, Inc. Resonant beam scanner with raster pinch compensation
JP4168221B2 (ja) 1999-09-06 2008-10-22 株式会社島津製作所 身体装着型表示システム
WO2001059491A1 (fr) 2000-02-10 2001-08-16 Digilens, Inc. Hologramme commutable et procede d'obtention
US6330064B1 (en) 2000-03-13 2001-12-11 Satcon Technology Corporation Doubly-differential interferometer and method for evanescent wave surface detection
US6443900B2 (en) * 2000-03-15 2002-09-03 Olympus Optical Co., Ltd. Ultrasonic wave transducer system and ultrasonic wave transducer
US6813085B2 (en) 2000-06-26 2004-11-02 Angus Duncan Richards Virtual reality display device
US20020093701A1 (en) 2000-12-29 2002-07-18 Xiaoxiao Zhang Holographic multifocal lens
US20020120916A1 (en) 2001-01-16 2002-08-29 Snider Albert Monroe Head-up display system utilizing fluorescent material
KR100495326B1 (ko) 2002-07-13 2005-06-14 삼성아이텍 주식회사 캐스팅 방식에 의한 내구성 편광렌즈의 제조방법 및제조도구
SE524003C2 (sv) 2002-11-21 2004-06-15 Tobii Technology Ab Förfarande och anläggning för att detektera och följa ett öga samt dess blickvinkel
US20040174287A1 (en) * 2002-11-21 2004-09-09 Deak David G. Self-contained switch
CN100374986C (zh) 2003-06-12 2008-03-12 控制仿生学公司 用于交互式通信和分析的方法、系统与软件
GB2407378B (en) 2003-10-24 2006-09-06 Lein Applied Diagnostics Ltd Ocular property measuring apparatus and method therefor
EP3462227A3 (fr) 2004-03-29 2019-06-19 Sony Corporation Dispositif optique et dispositif d'affichage d'images virtuelles
EP1827867A2 (fr) 2004-11-24 2007-09-05 Ver-Tec Security Systems Limited Hologrammes et procedes et appareil de fabrication d'hologrammes
CA2537569C (fr) 2005-02-24 2014-04-29 National Research Council Of Canada Microstores et methode de fabrication
US7773111B2 (en) 2005-03-16 2010-08-10 Lc Technologies, Inc. System and method for perceived image processing in a gaze tracking system
US20070132785A1 (en) 2005-03-29 2007-06-14 Ebersole John F Jr Platform for immersive gaming
KR20090025234A (ko) 2006-05-12 2009-03-10 시리얼 테크놀로지즈 에스.에이. 반사 광학 시스템, 트래킹 시스템 및 홀로그래픽 투영 시스템 및 방법
GB0622325D0 (en) 2006-11-09 2006-12-20 Optos Plc Improvements in or relating to retinal scanning
US20100239776A1 (en) 2007-07-25 2010-09-23 Hoya Corporation Method for producing plastic lens
US7925100B2 (en) 2007-07-31 2011-04-12 Microsoft Corporation Tiled packaging of vector image data
WO2009041055A1 (fr) 2007-09-26 2009-04-02 Panasonic Corporation Dispositif d'affichage de type à balayage de faisceau, son procédé d'affichage, programme et circuit intégré
WO2009054835A1 (fr) 2007-10-25 2009-04-30 Eye Ojo Corp. Verre polarisé et procédé de fabrication de verre polarisé
JP4989417B2 (ja) 2007-10-26 2012-08-01 キヤノン株式会社 画像表示システム、画像表示装置及びその制御方法、コンピュータプログラム
US8355671B2 (en) 2008-01-04 2013-01-15 Kopin Corporation Method and apparatus for transporting video signal over Bluetooth wireless interface
JP5094430B2 (ja) 2008-01-10 2012-12-12 キヤノン株式会社 画像処理方法、画像処理装置、システム
US20100149073A1 (en) 2008-11-02 2010-06-17 David Chaum Near to Eye Display System and Appliance
US8023571B2 (en) 2008-04-15 2011-09-20 Hong Nie Impulse ultra-wideband radio communication system
EP2138886A3 (fr) 2008-06-25 2011-10-05 Samsung Electronics Co., Ltd. Affichage virtuel compact
US9037530B2 (en) 2008-06-26 2015-05-19 Microsoft Technology Licensing, Llc Wearable electromyography-based human-computer interface
US7736000B2 (en) 2008-08-27 2010-06-15 Locarna Systems, Inc. Method and apparatus for tracking eye movement
US7850306B2 (en) 2008-08-28 2010-12-14 Nokia Corporation Visual cognition aware display and visual data transmission architecture
US8922898B2 (en) 2008-09-04 2014-12-30 Innovega Inc. Molded lens with nanofilaments and related methods
JP4780186B2 (ja) 2008-12-09 2011-09-28 ソニー株式会社 ホログラム記録フィルム及びその製造方法、並びに、画像表示装置
GB0902468D0 (en) 2009-02-16 2009-04-01 Light Blue Optics Ltd Optical systems
US20120139817A1 (en) 2009-08-13 2012-06-07 Bae Systems Plc Head up display system
JP2013127489A (ja) 2010-03-29 2013-06-27 Panasonic Corp シースルーディスプレイ
CN102472892B (zh) 2010-04-28 2014-07-09 松下电器产业株式会社 扫描型图像显示装置
US8634119B2 (en) 2010-07-09 2014-01-21 Tipd, Llc System for holography
JP5915981B2 (ja) 2010-08-09 2016-05-11 国立大学法人静岡大学 注視点検出方法及び注視点検出装置
US9406166B2 (en) 2010-11-08 2016-08-02 Seereal Technologies S.A. Display device, in particular a head-mounted display, based on temporal and spatial multiplexing of hologram tiles
US20120182309A1 (en) 2011-01-14 2012-07-19 Research In Motion Limited Device and method of conveying emotion in a messaging application
US8666212B1 (en) 2011-04-28 2014-03-04 Google Inc. Head mounted display using a fused fiber bundle
US8760499B2 (en) 2011-04-29 2014-06-24 Austin Russell Three-dimensional imager and projection device
US8510166B2 (en) 2011-05-11 2013-08-13 Google Inc. Gaze tracking system
KR101252169B1 (ko) * 2011-05-27 2013-04-05 엘지전자 주식회사 휴대 단말기 및 그 동작 제어방법
US20130198694A1 (en) 2011-06-10 2013-08-01 Aliphcom Determinative processes for wearable devices
US20130009853A1 (en) 2011-07-05 2013-01-10 The Board Of Trustees Of The Leland Stanford Junior University Eye-glasses mounted display
US8817379B2 (en) 2011-07-12 2014-08-26 Google Inc. Whole image scanning mirror display system
US8179604B1 (en) 2011-07-13 2012-05-15 Google Inc. Wearable marker for passive interaction
US8471967B2 (en) 2011-07-15 2013-06-25 Google Inc. Eyepiece for near-to-eye display with multi-reflectors
WO2013027004A1 (fr) 2011-08-24 2013-02-28 Milan Momcilo Popovich Affichage de données vestimentaire
US20140202643A1 (en) 2011-08-31 2014-07-24 Koninklijke Philips N.V. Light control panel
US20130088413A1 (en) 2011-10-05 2013-04-11 Google Inc. Method to Autofocus on Near-Eye Display
US8934160B2 (en) 2011-10-25 2015-01-13 National Central University Optical head-mounted display with mechanical one-dimensional scanner
US8704882B2 (en) 2011-11-18 2014-04-22 L-3 Communications Corporation Simulated head mounted display system and method
JP5906692B2 (ja) 2011-11-29 2016-04-20 セイコーエプソン株式会社 表示装置
TWI446896B (zh) 2011-12-23 2014-08-01 Ind Tech Res Inst 肌能參數感測器
US8971023B2 (en) 2012-03-21 2015-03-03 Google Inc. Wearable computing device frame
JP5464219B2 (ja) 2012-02-03 2014-04-09 株式会社デンソー 車両用ヘッドアップディスプレイ装置
US8970571B1 (en) 2012-03-13 2015-03-03 Google Inc. Apparatus and method for display lighting adjustment
CN104169778A (zh) 2012-03-15 2014-11-26 松下电器产业株式会社 光学反射元件和致动器
US8922481B1 (en) 2012-03-16 2014-12-30 Google Inc. Content annotation
US8994672B2 (en) 2012-04-09 2015-03-31 Sony Corporation Content transfer via skin input
KR20130121303A (ko) 2012-04-27 2013-11-06 한국전자통신연구원 원거리 시선 추적 시스템 및 방법
US20130332196A1 (en) 2012-06-07 2013-12-12 The Government Of The United States As Represented By The Secretary Of The Army Diabetes Monitoring Using Smart Device
US9398229B2 (en) 2012-06-18 2016-07-19 Microsoft Technology Licensing, Llc Selective illumination of a region within a field of view
US20140045463A1 (en) 2012-08-10 2014-02-13 Silverplus, Inc. Wearable Communication Device
US9338370B2 (en) 2012-11-05 2016-05-10 Honeywell International Inc. Visual system having multiple cameras
KR101984590B1 (ko) * 2012-11-14 2019-05-31 엘지전자 주식회사 디스플레이 디바이스 및 그 제어 방법
JP2014142423A (ja) 2013-01-22 2014-08-07 Denso Corp ヘッドアップディスプレイ装置
US20150362734A1 (en) 2013-01-28 2015-12-17 Ecole Polytechnique Federale De Lausanne (Epfl) Transflective holographic film for head worn display
US9223139B2 (en) 2013-02-15 2015-12-29 Google Inc. Cascading optics in optical combiners of head mounted displays
US9392129B2 (en) 2013-03-15 2016-07-12 John Castle Simmons Light management for image and data control
JP6449236B2 (ja) 2013-03-25 2019-01-09 インテル コーポレイション 多射出瞳頭部装着型ディスプレイのための方法および装置
KR102043200B1 (ko) 2013-05-07 2019-11-11 엘지전자 주식회사 스마트 워치 및 제어 방법
US20150036221A1 (en) 2013-08-04 2015-02-05 Robert S. Stephenson Wide-field head-up display (HUD) eyeglasses
US9652024B2 (en) * 2013-08-23 2017-05-16 Samsung Electronics Co., Ltd. Mode switching method and apparatus of terminal
EP3042152B1 (fr) 2013-09-04 2022-11-09 Essilor International Procédé de navigation basé sur un dispositif transparent monté sur la tête
CN107315249B (zh) 2013-11-27 2021-08-17 奇跃公司 虚拟和增强现实系统与方法
US8958158B1 (en) 2013-12-03 2015-02-17 Google Inc. On-head detection for head-mounted display
US20150205134A1 (en) 2014-01-17 2015-07-23 Thalmic Labs Inc. Systems, articles, and methods for wearable heads-up displays
WO2015123775A1 (fr) 2014-02-18 2015-08-27 Sulon Technologies Inc. Systèmes et procédés pour incorporer un train d'images réelles dans un train d'images virtuelles
US9804753B2 (en) * 2014-03-20 2017-10-31 Microsoft Technology Licensing, Llc Selection using eye gaze evaluation over time
US20150325202A1 (en) 2014-05-07 2015-11-12 Thalmic Labs Inc. Systems, devices, and methods for wearable computers with heads-up displays
US9477079B2 (en) 2014-06-25 2016-10-25 Thalmic Labs Inc. Systems, devices, and methods for wearable heads-up displays
CA2976898C (fr) 2015-02-17 2023-11-21 Thalmic Labs Inc. Systemes, dispositifs et procedes d'expansion de boitier oculaire dans des affichages tete haute portables
US20160274365A1 (en) 2015-03-17 2016-09-22 Thalmic Labs Inc. Systems, devices, and methods for wearable heads-up displays with heterogeneous display quality
US20160274758A1 (en) 2015-03-20 2016-09-22 Thalmic Labs Inc. Systems, devices, and methods for mitigating false positives in human-electronics interfaces
US10197805B2 (en) 2015-05-04 2019-02-05 North Inc. Systems, devices, and methods for eyeboxes with heterogeneous exit pupils
US10488661B2 (en) 2015-05-28 2019-11-26 North Inc. Systems, devices, and methods that integrate eye tracking and scanning laser projection in wearable heads-up displays
JP2018528475A (ja) 2015-09-04 2018-09-27 サルミック ラブス インコーポレイテッド ホログラフィック光学素子を眼鏡レンズに統合するシステム、製品、及び方法
US9904051B2 (en) 2015-10-23 2018-02-27 Thalmic Labs Inc. Systems, devices, and methods for laser eye tracking
US20170153701A1 (en) 2015-12-01 2017-06-01 Thalmic Labs Inc. Systems, devices, and methods for wearable heads-up displays as wireless controllers
KR20180118107A (ko) 2015-12-17 2018-10-30 탈믹 랩스 인크 곡선의 홀로그램 광학 소자들에 대한 시스템들, 디바이스들 및 방법들
US10303246B2 (en) 2016-01-20 2019-05-28 North Inc. Systems, devices, and methods for proximity-based eye tracking
US10151926B2 (en) 2016-01-29 2018-12-11 North Inc. Systems, devices, and methods for preventing eyebox degradation in a wearable heads-up display
US10365548B2 (en) 2016-04-13 2019-07-30 North Inc. Systems, devices, and methods for focusing laser projectors
CA3029234A1 (fr) 2016-06-30 2018-01-04 Thalmic Labs Inc. Systemes de capture d'images, dispositifs et procedes de mise au point automatique bases sur le suivi de l'oeil

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6700310B2 (en) * 2000-10-13 2004-03-02 Lear Corporation Self-powered wireless switch
US20120249797A1 (en) * 2010-02-28 2012-10-04 Osterhout Group, Inc. Head-worn adaptive display
US20130300652A1 (en) * 2011-11-30 2013-11-14 Google, Inc. Unlocking a Screen Using Eye Tracking Information
US20130169560A1 (en) * 2012-01-04 2013-07-04 Tobii Technology Ab System for gaze interaction
US20140198035A1 (en) * 2013-01-14 2014-07-17 Thalmic Labs Inc. Wearable muscle interface systems, devices and methods that interact with content displayed on an electronic display
US20150049112A1 (en) * 2013-08-19 2015-02-19 Qualcomm Incorporated Automatic customization of graphical user interface for optical see-through head mounted display with user interaction tracking
US20150277559A1 (en) * 2014-04-01 2015-10-01 Apple Inc. Devices and Methods for a Ring Computing Device
US20170083115A1 (en) * 2014-05-16 2017-03-23 Faindu Gmbh Method for displaying a virtual interaction on at least one screen and input device, system and method for a virtual application by means of a computing unit

Cited By (89)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10528135B2 (en) 2013-01-14 2020-01-07 Ctrl-Labs Corporation Wearable muscle interface systems, devices and methods that interact with content displayed on an electronic display
US11009951B2 (en) 2013-01-14 2021-05-18 Facebook Technologies, Llc Wearable muscle interface systems, devices and methods that interact with content displayed on an electronic display
US11921471B2 (en) 2013-08-16 2024-03-05 Meta Platforms Technologies, Llc Systems, articles, and methods for wearable devices having secondary power sources in links of a band for providing secondary power in addition to a primary power source
US11644799B2 (en) 2013-10-04 2023-05-09 Meta Platforms Technologies, Llc Systems, articles and methods for wearable electronic devices employing contact sensors
US11079846B2 (en) 2013-11-12 2021-08-03 Facebook Technologies, Llc Systems, articles, and methods for capacitive electromyography sensors
US11666264B1 (en) 2013-11-27 2023-06-06 Meta Platforms Technologies, Llc Systems, articles, and methods for electromyography sensors
US11822090B2 (en) * 2014-01-24 2023-11-21 Mentor Acquisition One, Llc Haptic systems for head-worn computers
US10684692B2 (en) 2014-06-19 2020-06-16 Facebook Technologies, Llc Systems, devices, and methods for gesture identification
US9766449B2 (en) 2014-06-25 2017-09-19 Thalmic Labs Inc. Systems, devices, and methods for wearable heads-up displays
US9874744B2 (en) 2014-06-25 2018-01-23 Thalmic Labs Inc. Systems, devices, and methods for wearable heads-up displays
US10012829B2 (en) 2014-06-25 2018-07-03 Thalmic Labs Inc. Systems, devices, and methods for wearable heads-up displays
US10067337B2 (en) 2014-06-25 2018-09-04 Thalmic Labs Inc. Systems, devices, and methods for wearable heads-up displays
US10054788B2 (en) 2014-06-25 2018-08-21 Thalmic Labs Inc. Systems, devices, and methods for wearable heads-up displays
US10191283B2 (en) 2015-02-17 2019-01-29 North Inc. Systems, devices, and methods for eyebox expansion displays in wearable heads-up displays
US9958682B1 (en) 2015-02-17 2018-05-01 Thalmic Labs Inc. Systems, devices, and methods for splitter optics in wearable heads-up displays
US9989764B2 (en) 2015-02-17 2018-06-05 Thalmic Labs Inc. Systems, devices, and methods for eyebox expansion in wearable heads-up displays
US10613331B2 (en) 2015-02-17 2020-04-07 North Inc. Systems, devices, and methods for splitter optics in wearable heads-up displays
US10031338B2 (en) 2015-02-17 2018-07-24 Thalmic Labs Inc. Systems, devices, and methods for eyebox expansion in wearable heads-up displays
US10197805B2 (en) 2015-05-04 2019-02-05 North Inc. Systems, devices, and methods for eyeboxes with heterogeneous exit pupils
US10133075B2 (en) 2015-05-04 2018-11-20 Thalmic Labs Inc. Systems, devices, and methods for angle- and wavelength-multiplexed holographic optical elements
US10175488B2 (en) 2015-05-04 2019-01-08 North Inc. Systems, devices, and methods for spatially-multiplexed holographic optical elements
US10114222B2 (en) 2015-05-28 2018-10-30 Thalmic Labs Inc. Integrated eye tracking and laser projection methods with holographic elements of varying optical powers
US10139633B2 (en) 2015-05-28 2018-11-27 Thalmic Labs Inc. Eyebox expansion and exit pupil replication in wearable heads-up display having integrated eye tracking and laser projection
US10488661B2 (en) 2015-05-28 2019-11-26 North Inc. Systems, devices, and methods that integrate eye tracking and scanning laser projection in wearable heads-up displays
US10180578B2 (en) 2015-05-28 2019-01-15 North Inc. Methods that integrate visible light eye tracking in scanning laser projection displays
US10078219B2 (en) 2015-05-28 2018-09-18 Thalmic Labs Inc. Wearable heads-up display with integrated eye tracker and different optical power holograms
US10078220B2 (en) 2015-05-28 2018-09-18 Thalmic Labs Inc. Wearable heads-up display with integrated eye tracker
US10073268B2 (en) 2015-05-28 2018-09-11 Thalmic Labs Inc. Display with integrated visible light eye tracking
US10488662B2 (en) 2015-09-04 2019-11-26 North Inc. Systems, articles, and methods for integrating holographic optical elements with eyeglass lenses
US10705342B2 (en) 2015-09-04 2020-07-07 North Inc. Systems, articles, and methods for integrating holographic optical elements with eyeglass lenses
US10718945B2 (en) 2015-09-04 2020-07-21 North Inc. Systems, articles, and methods for integrating holographic optical elements with eyeglass lenses
US10877272B2 (en) 2015-09-04 2020-12-29 Google Llc Systems, articles, and methods for integrating holographic optical elements with eyeglass lenses
US10890765B2 (en) 2015-09-04 2021-01-12 Google Llc Systems, articles, and methods for integrating holographic optical elements with eyeglass lenses
US10228558B2 (en) 2015-10-23 2019-03-12 North Inc. Systems, devices, and methods for laser eye tracking
US9904051B2 (en) 2015-10-23 2018-02-27 Thalmic Labs Inc. Systems, devices, and methods for laser eye tracking
US10606072B2 (en) 2015-10-23 2020-03-31 North Inc. Systems, devices, and methods for laser eye tracking
USD821391S1 (en) * 2015-12-04 2018-06-26 Samsung Electronics Co., Ltd. Wearable device
US10802190B2 (en) 2015-12-17 2020-10-13 Covestro Llc Systems, devices, and methods for curved holographic optical elements
US10241572B2 (en) 2016-01-20 2019-03-26 North Inc. Systems, devices, and methods for proximity-based eye tracking
US10303246B2 (en) 2016-01-20 2019-05-28 North Inc. Systems, devices, and methods for proximity-based eye tracking
US10126815B2 (en) 2016-01-20 2018-11-13 Thalmic Labs Inc. Systems, devices, and methods for proximity-based eye tracking
US10437067B2 (en) 2016-01-29 2019-10-08 North Inc. Systems, devices, and methods for preventing eyebox degradation in a wearable heads-up display
US10451881B2 (en) 2016-01-29 2019-10-22 North Inc. Systems, devices, and methods for preventing eyebox degradation in a wearable heads-up display
US10151926B2 (en) 2016-01-29 2018-12-11 North Inc. Systems, devices, and methods for preventing eyebox degradation in a wearable heads-up display
US10365550B2 (en) 2016-04-13 2019-07-30 North Inc. Systems, devices, and methods for focusing laser projectors
US10365548B2 (en) 2016-04-13 2019-07-30 North Inc. Systems, devices, and methods for focusing laser projectors
US10365549B2 (en) 2016-04-13 2019-07-30 North Inc. Systems, devices, and methods for focusing laser projectors
US10277874B2 (en) 2016-07-27 2019-04-30 North Inc. Systems, devices, and methods for laser projectors
US10230929B2 (en) 2016-07-27 2019-03-12 North Inc. Systems, devices, and methods for laser projectors
US10250856B2 (en) 2016-07-27 2019-04-02 North Inc. Systems, devices, and methods for laser projectors
US10459223B2 (en) 2016-08-12 2019-10-29 North Inc. Systems, devices, and methods for variable luminance in wearable heads-up displays
US10459222B2 (en) 2016-08-12 2019-10-29 North Inc. Systems, devices, and methods for variable luminance in wearable heads-up displays
US10459221B2 (en) 2016-08-12 2019-10-29 North Inc. Systems, devices, and methods for variable luminance in wearable heads-up displays
US10657927B2 (en) * 2016-11-03 2020-05-19 Elias Khoury System for providing hands-free input to a computer
US10215987B2 (en) 2016-11-10 2019-02-26 North Inc. Systems, devices, and methods for astigmatism compensation in a wearable heads-up display
US10345596B2 (en) 2016-11-10 2019-07-09 North Inc. Systems, devices, and methods for astigmatism compensation in a wearable heads-up display
US10409057B2 (en) 2016-11-30 2019-09-10 North Inc. Systems, devices, and methods for laser eye tracking in wearable heads-up displays
US10459220B2 (en) 2016-11-30 2019-10-29 North Inc. Systems, devices, and methods for laser eye tracking in wearable heads-up displays
US10663732B2 (en) 2016-12-23 2020-05-26 North Inc. Systems, devices, and methods for beam combining in wearable heads-up displays
US10365492B2 (en) 2016-12-23 2019-07-30 North Inc. Systems, devices, and methods for beam combining in wearable heads-up displays
US10718951B2 (en) 2017-01-25 2020-07-21 North Inc. Systems, devices, and methods for beam combining in laser projectors
US10437073B2 (en) 2017-01-25 2019-10-08 North Inc. Systems, devices, and methods for beam combining in laser projectors
US10437074B2 (en) 2017-01-25 2019-10-08 North Inc. Systems, devices, and methods for beam combining in laser projectors
US10782793B2 (en) * 2017-08-10 2020-09-22 Google Llc Context-sensitive hand interaction
US11181986B2 (en) * 2017-08-10 2021-11-23 Google Llc Context-sensitive hand interaction
US20190050062A1 (en) * 2017-08-10 2019-02-14 Google Llc Context-sensitive hand interaction
US11635736B2 (en) 2017-10-19 2023-04-25 Meta Platforms Technologies, Llc Systems and methods for identifying biological structures associated with neuromuscular source signals
US10901216B2 (en) 2017-10-23 2021-01-26 Google Llc Free space multiple laser diode modules
US11300788B2 (en) 2017-10-23 2022-04-12 Google Llc Free space multiple laser diode modules
WO2019119154A1 (fr) 2017-12-22 2019-06-27 North Inc. Conceptions d'antenne destinées à des dispositifs d'affichage tête-haute portables
US11042221B2 (en) 2018-01-08 2021-06-22 Facebook Technologies, Llc Methods, devices, and systems for displaying a user interface on a user and detecting touch gestures
US10678335B2 (en) 2018-01-08 2020-06-09 Facebook Technologies, Llc Methods, devices, and systems for creating haptic stimulations and tracking motion of a user
US10684690B2 (en) 2018-01-08 2020-06-16 Facebook Technologies, Llc Methods, devices, and systems for creating localized haptic stimulations on a user
WO2019136439A1 (fr) * 2018-01-08 2019-07-11 Facebook Technologies, Llc Procédés, dispositifs et systèmes pour créer des stimulations haptiques et suivre un mouvement d'utilisateur
US10824235B2 (en) 2018-01-08 2020-11-03 Facebook Technologies, Llc Methods, devices, and systems for displaying a user interface on a user and detecting touch gestures
US11422628B2 (en) 2018-01-08 2022-08-23 Meta Platforms Technologies, Llc Methods, devices, and systems for modifying perceived haptic stimulations on a user
US10795445B2 (en) 2018-01-08 2020-10-06 Facebook Technologies, Llc Methods, devices, and systems for determining contact on a user of a virtual reality and/or augmented reality device
US11467670B2 (en) 2018-03-23 2022-10-11 Meta Platforms Technologies, Llc Methods, devices, and systems for displaying a user interface on a user and detecting touch gestures
US10739869B1 (en) * 2018-05-01 2020-08-11 Facebook Technologies, Llc Apparatus, system, and method for controlling artificial-reality environments
US11080417B2 (en) * 2018-06-26 2021-08-03 Google Llc Private eye-to-eye communications with wearable heads up display
US11055056B1 (en) * 2018-09-25 2021-07-06 Facebook Technologies, Llc Split system for artificial reality
US11797087B2 (en) 2018-11-27 2023-10-24 Meta Platforms Technologies, Llc Methods and apparatus for autocalibration of a wearable electrode sensor system
US11941176B1 (en) 2018-11-27 2024-03-26 Meta Platforms Technologies, Llc Methods and apparatus for autocalibration of a wearable electrode sensor system
US11509343B2 (en) 2018-12-18 2022-11-22 Snap Inc. Adaptive eyewear antenna
CN113196673A (zh) * 2018-12-18 2021-07-30 美国斯耐普公司 自适应眼戴设备天线
US11949443B2 (en) 2018-12-18 2024-04-02 Snap Inc. Adaptive eyewear antenna
US11961494B1 (en) 2019-03-29 2024-04-16 Meta Platforms Technologies, Llc Electromagnetic interference reduction in extended reality environments
US11907423B2 (en) 2019-11-25 2024-02-20 Meta Platforms Technologies, Llc Systems and methods for contextualized interactions with an environment
US11868531B1 (en) 2021-04-08 2024-01-09 Meta Platforms Technologies, Llc Wearable device providing for thumb-to-finger-based input gestures detected based on neuromuscular signals, and systems and methods of use thereof

Also Published As

Publication number Publication date
US10656822B2 (en) 2020-05-19
US20180067621A1 (en) 2018-03-08
CA3007196A1 (fr) 2017-04-06
WO2017059285A1 (fr) 2017-04-06

Similar Documents

Publication Publication Date Title
US10656822B2 (en) Systems, devices, and methods for interacting with content displayed on head-mounted displays
US11009951B2 (en) Wearable muscle interface systems, devices and methods that interact with content displayed on an electronic display
US11429232B2 (en) Wearable electronic devices having an inward facing input device and methods of use thereof
US20190129676A1 (en) Systems, devices, and methods for wearable computers with heads-up displays
US20180101289A1 (en) Systems, devices, and methods for mitigating false positives in human-electronics interfaces
KR102224481B1 (ko) 이동단말기 및 그 제어방법
US20170336964A1 (en) Wearable electronic device
EP3635511B1 (fr) Dispositif portable permettant des gestes à plusieurs doigts
KR20160120103A (ko) 이동 단말기 및 그것의 제어 방법
KR20160035948A (ko) 이동 단말기와 그것을 포함하는 시스템
US10834304B2 (en) Systems, devices, and methods for a wearable electronic device having a selfie camera
KR102297473B1 (ko) 신체를 이용하여 터치 입력을 제공하는 장치 및 방법
KR20160018163A (ko) 이동 단말기 및 그것의 통신 시스템
US20150277742A1 (en) Wearable electronic device
KR20200094970A (ko) 증강 현실 환경에서 다양한 기능을 수행하는 전자 장치 및 그 동작 방법
US10871837B2 (en) Wearable electronic devices having a rotatable input structure
JP2024026166A (ja) 描画システム
US20230325002A1 (en) Techniques for neuromuscular-signal-based detection of in-air hand gestures for text production and modification, and systems, wearable devices, and methods for using these techniques
Tamaki et al. Gesturenail: Wireless Hand Gesture System
WO2023230354A1 (fr) Systèmes d'interprétation de mouvements de pouce de gestes de la main dans l'air pour commander des interfaces utilisateur sur la base d'orientations spatiales de la main d'un utilisateur, et son procédé d'utilisation
WO2023196671A1 (fr) Techniques de détection fondée sur des signaux neuromusculaires de gestes de la main dans l'air pour la production et la modification de texte, et systèmes, dispositifs à porter sur soi et procédés d'utilisation desdites techniques
KR20150033465A (ko) 전자기기 및 그 제어방법
Bainbridge HCI gesture tracking using wearable passive tags

Legal Events

Date Code Title Description
AS Assignment

Owner name: THALMIC LABS INC., CANADA

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:BAILEY, MATTHEW;ALEXANDER, STEFAN;SIGNING DATES FROM 20161214 TO 20170123;REEL/FRAME:045555/0470

STPP Information on status: patent application and granting procedure in general

Free format text: NON FINAL ACTION MAILED

STPP Information on status: patent application and granting procedure in general

Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER

STPP Information on status: patent application and granting procedure in general

Free format text: FINAL REJECTION MAILED

STCB Information on status: application discontinuation

Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION

AS Assignment

Owner name: GOOGLE LLC, CALIFORNIA

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:NORTH INC.;REEL/FRAME:054113/0907

Effective date: 20200916

AS Assignment

Owner name: NORTH INC., CANADA

Free format text: CHANGE OF NAME;ASSIGNOR:THALMIC LABS INC.;REEL/FRAME:054414/0100

Effective date: 20180830