US20150042680A1 - Method and device for controlling a near eye display - Google Patents
Method and device for controlling a near eye display Download PDFInfo
- Publication number
- US20150042680A1 US20150042680A1 US14/453,733 US201414453733A US2015042680A1 US 20150042680 A1 US20150042680 A1 US 20150042680A1 US 201414453733 A US201414453733 A US 201414453733A US 2015042680 A1 US2015042680 A1 US 2015042680A1
- Authority
- US
- United States
- Prior art keywords
- eye display
- near eye
- axis
- user
- visual indicator
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T19/00—Manipulating 3D models or images for computer graphics
- G06T19/006—Mixed reality
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F1/00—Details not covered by groups G06F3/00 - G06F13/00 and G06F21/00
- G06F1/16—Constructional details or arrangements
- G06F1/1613—Constructional details or arrangements for portable computers
- G06F1/163—Wearable computers, e.g. on a belt
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/017—Gesture based interaction, e.g. based on a set of recognized hand gestures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
- G06F3/0304—Detection arrangements using opto-electronic means
-
- G—PHYSICS
- G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
- G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
- G09G3/00—Control arrangements or circuits, of interest only in connection with visual indicators other than cathode-ray tubes
- G09G3/20—Control arrangements or circuits, of interest only in connection with visual indicators other than cathode-ray tubes for presentation of an assembly of a number of characters, e.g. a page, by composing the assembly by combination of individual elements arranged in a matrix no fixed position being assigned to or needed to be assigned to the individual characters or partial characters
- G09G3/34—Control arrangements or circuits, of interest only in connection with visual indicators other than cathode-ray tubes for presentation of an assembly of a number of characters, e.g. a page, by composing the assembly by combination of individual elements arranged in a matrix no fixed position being assigned to or needed to be assigned to the individual characters or partial characters by control of light from an independent source
- G09G3/3406—Control of illumination source
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2219/00—Indexing scheme for manipulating 3D models or images for computer graphics
- G06T2219/008—Cut plane or projection plane definition
-
- G—PHYSICS
- G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
- G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
- G09G2360/00—Aspects of the architecture of display systems
- G09G2360/14—Detecting light within display terminals, e.g. using a single or a plurality of photosensors
Definitions
- the present invention relates generally to the field of gesture recognition, and more specifically, using same for near eye display applications.
- near eye display as used herein is defined as devices which include augmented reality or virtual reality glasses and wearable projected displays.
- field of view is defined as the extent of the observable world that is seen at any given moment by a user or a sensor.
- AR Augmented Reality
- AR is defined as a live direct or indirect view of a physical, real-world environment whose elements are augmented (or supplemented) by computer-generated sensory input such as sound, video, graphics or any other data.
- VR Virtual Reality
- Virtual reality is defined as a computer-simulated environment that can simulate physical presence in places in the real world or imagined worlds. Virtual reality could recreate sensory experiences, including virtual taste, sight, smell, sound, touch, and the like.
- AR augmented reality
- Another aspect that restricts the functional ability to control the presented content is that it is displayed directly to the user's eye.
- actual touch interfaces are not an option for control, creating a need for separating the actual controlling area from the display while maintaining the extended abilities of touch control.
- portable devices such as near-eye display devices are limited in terms of the acceptable power consumption and weight allowed for the device components. Therefore, using available gestures capturing technologies, based on an IR illumination source and an additional IR sensor is not practical due to the incremental weight and power consumption.
- Some embodiments of the present invention overcome the challenges of the prior art by, inter alia, using existing components of a near-eye device such as the frontal camera with a visible light structured pattern which enables to reduce the additional hardware needed to be embedded into the device as well as limiting the incremental power consumption.
- Some embodiments of the present invention include a transmitter module designed for projecting structured light pattern onto a virtual navigation space, a camera module positioned and oriented for capturing the structured light data, a processor defined for calculating three dimensional data and detecting the user's controlling gestures, and a display system for displaying information relating to the position of the fingertip and performed gestures.
- FIG. 1 illustrates the device and its environment according to some embodiments of the present invention
- FIG. 2A is a block diagram illustrating the device in accordance with one embodiment of the present invention.
- FIG. 2B is a block diagram illustrating the device in accordance with another embodiment of the present invention.
- FIG. 3A is a diagram illustrating a non-limiting gesture and its outcome in accordance with some embodiments of the present invention.
- FIG. 3B is a diagram illustrating another non-limiting gesture and its outcome in accordance with some embodiments of the present invention.
- FIG. 3C is a diagram illustrating yet another non-limiting gesture and its outcome in accordance with some embodiments of the present invention.
- FIG. 3D is a diagram illustrating yet another non-limiting gesture and its outcome in accordance with some embodiments of the present invention.
- FIG. 4 is a diagram illustrating a non-limiting pattern projected onto a hand of a user serving as a controlling object in accordance with some embodiments of the present invention.
- FIG. 5 is a flowchart illustrating a method according to some embodiments of the present invention.
- Some embodiments of the present invention address the challenge of controlling a cursor or any other added content presented to a user wearing a near eye display.
- any computer generated content that is super positioned over the captured scene, to yield an augmented reality or VR may be controlled by gestured and postures of the viewer, using embodiments of the present invention.
- FIG. 1 illustrates the device according to some embodiments of the present invention in operation within its working environment.
- the device may be implemented as a wearable near eye display 100 which directly projects an image onto the eyes of a user 10 .
- the device may include a sensor 110 (e.g., a camera) having a specified field of view 112 , attached to the wearable near eye display 100 and configured to capture a scene at the specified field of view overlapping a field of view of the user 10 .
- device 100 may take the form of eye glasses with some projector and image source coupled to the visor.
- the device may further include a transmitter 120 attached to the wearable near eye display.
- Transmitter 120 may be configured to transmit a specified light pattern onto a navigation space 120 , wherein the sensor 110 is configured to capture reflections of the specified pattern coming from a navigation space 124 .
- the source of light may a laser diode, a light emitting diode (LED), any another element which emits a light beam, as known in the art.
- the light beam emitted by transmitter 120 propagates through a micro structured element (not shown), onto the navigation space.
- the micro structured element modifies the light beam, for generating the light pattern projected onto the navigation space.
- the micro structured element converts the light beam into a light beam of a variable cross sectional intensity profile. Consequently, the cross sectional intensity profile varies along the light beam, thus providing information indicative of a distance of an object (say one of the user's digits) from the source of light.
- the transmitter 120 projects a light pattern having a continuous feature in a first direction and a non-continuous feature in a direction substantially perpendicular to the first direction, onto the navigation space.
- the micro structured element is a diffractive optical element, as known in the art.
- the diffractive optical element may be obtained by a periodic micro structure that splits the light beam into a discrete number of spots having a one dimensional or a two dimensional spatial arrangement, as known in the art.
- an additional element such as a cylindrical micro lens array or an additional diffractive element is used to create a stripe from each of the spots.
- the stripe is marked with phase notations positioned in fixed distances from each other, along the stripe.
- a possible advantage of using an additional microstructure element such as a cylindrical micro lens array is that the additional element may disperse a zero order of a light beam such as a laser beam, as known in the art.
- the zero order is a center of a light spot output from the diffractive element, and may be characterized by relatively high energy. Dispersing the zero order to a light structure such as the stripe, may allow increasing intensity of a laser light beam without crossing eye safety limits.
- transmitter 120 further includes an additional diffractive element, which changes the cross-sectional intensity profile of each stripe in the light pattern with respect to the distance from an object which reflects the projected stripe.
- the cross-sectional intensity profile is an intensity profile perpendicular to the propagation direction of the light projected.
- the intensity profile change is a gradual change from a Gaussian cross-sectional intensity profile to a Top Hat cross-sectional intensity profile, carried out gradually along a distance traversed by the projected light, as the projected light propagates to an object, and is reflected back from surface of the object.
- the intensity profile change is a gradual change from an intensity profile with a single peak to an intensity profile with two or more peaks, etc., as known in the art.
- the change in the intensity profile may help differentiate between stripes reflected from different objects, in different ranges, and thus further helps overcome a segment unification problem.
- the change in the intensity profile may further be used to directly measure distance to the controllable object (e.g., to the user's hand, fingers and thumb).
- Sensor 110 may be a video camera such as a webcam or a cellular phone's camera, positioned next to transmitter 120 .
- FIG. 2A is a block diagram illustrating the device 200 A in accordance with one embodiment of the present invention.
- Device 200 A may include a wearable near eye display 250 , a sensor 210 attached to the wearable near eye display and configured to capture a scene.
- Device 200 A may further include a transmitter 220 attached to the wearable near eye display 250 .
- the transmitter 220 is configured to transmit a structured light pattern onto a navigation space, wherein sensor 210 may further be configured to capture reflections of the specified pattern coming from the navigation space.
- Device 200 A may further include a computer processor 230 configured to analyze the reflections and control a visual indicator (not shown here) presented to a user over the wearable near eye display 250 , based on the analysis.
- Device 200 A may take the form of eye glasses of any head mounted display as depicted in the inset of FIG. 1 .
- FIG. 2B is a block diagram illustrating the device in accordance with another embodiment of the present invention.
- the device 200 B is provided in an add-on or retrofit configuration and apart from the transmitter 220 B and a software application 240 B, all other elements are provided as a workpiece or an environment.
- the system may thus include a transmitter 220 B attachable to a near eye display 250 B, a computer readable code 240 B executable on a computer processor 230 B and configured, when executed, to invoke the computer processor 230 B to: instruct the transmitter to transmit a structured light pattern onto a navigation space, instruct a sensor attached to the wearable near eye display to capture a scene; instruct said sensor to capture reflections of the specified pattern coming from the navigation space; and analyze the reflections and control a visual indicator presented to a user over the wearable near eye display 250 B, based on the analysis.
- the analysis carried out by the computer processor may be achieved by generating 3D depth map based on the reflections of the patterned light coming from the navigational space and further analyzing the 3D depth map.
- the structured light pattern is projected onto the virtual navigation area creating a volume where objects can be detected in 3D.
- the structured light may consist of dots pattern, Lines, curves or any other pattern that enable the ability of 3D depth mapping or object segmentation.
- the virtual navigation area 124 covers only part of the camera field of view 112 .
- Such part may consist of the lower part of the camera image while the majority of the camera field of view is kept free from projected light.
- the needed laser or LED power consumption that produces the structured light is reduced significantly.
- an angle between the transmitter optical axis and the camera optical axis is presented. The effect of such an angle is encapsulated in the method of the depth extraction and the object segmentation.
- the transmitter can be positioned with a shift backwards on the line of the camera optical axis in relation with the sensor (camera) field of view direction. This enables additional flexibility in the device design and is taken into account in the pattern analysis method as well.
- the structured light pattern is projected using a light source in the visible spectral range.
- a light source in the visible spectral range.
- the quantum efficiency of the CMOS sensor enables better sensitivity in the visible spectral range reducing the energy needed for the structured light transmitter.
- Another important advantage of using visible light is that the control object becomes highlighted with the reflected light when entering the virtual navigation space. This indicates to the user on where to position his hand/finger in order to be within the borders of the navigation space.
- the structured light is focused to a specific distance from transmitter 120 utilizing a short depth of focus around that distance. This creates a limited region along the optical axis of the transmitter. Only inside that region the light pattern structure is detectable by the camera. Although it limits the space where the user can operate in, it also blurs the light pattern outside of that region and reduces the unwanted s reflection from objects in the scene in front of the user.
- the structured light pattern may consist of light features that are continues in one direction and non-continues in another direction. These features are used to enable segmentation of the control object from the background scenario and for detecting the contour of the object.
- various intensities may appear on different light feature creating a unique intensity signature between each sub group of the line features in the pattern. This enables detecting the index of each line feature precisely and determining the accurate depth of the object that reflects the pattern back to the camera.
- the light features may consist of notations, variation of intensity, or other feature alternation along the light feature itself to assist the segmentation of the control object as well as locally detect its depth.
- FIG. 3A is a diagram illustrating a non-limiting gesture and its outcome in accordance with some embodiments of the present invention. Responsive to moving the hand from position 310 A to position 312 A along an X axis at the navigation space, the cursor is moving along a corresponding X′ axis from position 330 A to position 332 A. It is understood, however, that, for practical reasons, a similar displacement of the finger only of the hand from one position to another leads to a displacement of the cursor 9 visual indicator) at the display.
- the computer processor of the device is configured to map or translate one form of movement in real life (of the controlling object) at the x-y-z space into another movement at the display. Similar mappings are illustrated in FIGS. 3B-3D where in 3 B Y axis mapping is demonstrated with positions 310 B and 312 B in real life correspond to positions 330 B and 332 B of the cursor over display 320 B. Alternatively, the mapping may convert movement along a Y axis at the navigational space into a movement along a corresponding Z axis at the display, in accordance with the requirements of the application.
- tilting the palm of the hand, a fist or at least one finger along a predefined arc causes moving the cursor in the display 320 C (e.g., from 330 C to 332 C or from 330 D to 332 D), in a gesture that may resemble controlling a joystick.
- One of the key features of some embodiments of the present invention is the ability to separate the pointing direction of the controlling object and the position of the displayed graphics relating the user's perspective, including the pointing indication on the device display. This enables, besides the previously mentioned advantages of reducing the size of the navigation space, also some key controlling schemes.
- One of these schemes includes scaling or normalizing the position of the controlling device to enable larger or smaller movements to cover the entire graphical area presented to the user. In this way, the actual size of the virtual navigation space is not depended on the size and location of the presented object and can be determined based on the use case and user experience.
- the angle and orientation of the controlling object is used as an input to determine the position of the graphical pointing indication on the presented UI. For example, an angle created between the controlling object and the horizontal axis in relation to the transmitter optical axis can be translated to a position on the x-axis of the presented user interface. An angle created between the controlling object and the vertical axis in relation to the transmitter optical axis can be translated to a position on the y-axis of the presented interface.
- selecting gestures are presented.
- a selecting gesture may include one of the following postures: an addition of a controlling object to the virtual navigation space.
- Such controlling object may be an additional finger.
- the additional controlling object may be attached to the first controlling object previously located in the virtual navigation space.
- Another presented selecting gesture includes the rotation of the controlling object. For example, in the case that the controlling object is a hand with a stretched pointing finger, rotating the hand around the finger axis or pushing the finger forward will result in a click event. This enables clicks without changing the apposition and orientation of the controlling object therefore enable stable pointing during the click action.
- the senor and the transmitter are operable only within visible light bandwidth. This way the pattern may also be used as visual feedback to the user—indicating that his or her hand is within the navigational space.
- the senor configured to capture the scene and the reflections is a single element. By configuring a single sensor to do both tasks, a more cost effective and low power consumption device may be achieved.
- control of the visual indicator comprises moving the visual indicator along a first axis of the display responsive to moving the controlling object along a second axis of the navigation space.
- the computer processor is further configured to identify, based on the reflections analysis at least one predefined action gesture, and invoke a corresponding action event associated with the visual indicator and content presented over the wearable near eye display.
- the navigational space is located at a lower side of the field of view. This way it does not obstruct the viewpoint of the user.
- the navigational space is limited in depth by a depth of field of the transmitter, for a depth in which the specified pattern is in focus.
- FIG. 4 is a schematic diagram 400 illustrating a hand projected with a light pattern, according to an exemplary embodiment of the present invention.
- the tracking of the movement of the hand is carried out, using a light pattern designed to enable detection of hand movement, such as fine movements of fingers and thumbs.
- the use of structured light may be, for instance, as disclosed in WIPO PCT Patent Application Publication No. WO 2013/088442, which is incorporated herein by reference in its entirety.
- the specifically designed light pattern allows the tracking of the movement, even in bi-dimensional video data, which unlike three dimensional depth map, does not provide for easy separation of the hands from the rest of the body.
- the light pattern may be specifically designed to track movement of the hand's digits in a bi-dimensional video data (e.g., video images streamed from a regular video camera). More specifically, the light pattern may be designed to enable detection and tracking of digits (i.e., fingers and thumb) as well as palm, in the bi-dimensional video data, according to distortions of the pattern by the digits.
- a bi-dimensional video data e.g., video images streamed from a regular video camera.
- the light pattern may be designed to enable detection and tracking of digits (i.e., fingers and thumb) as well as palm, in the bi-dimensional video data, according to distortions of the pattern by the digits.
- the light pattern has a continuous feature in a first direction (say the X-axis) and a non-continuous (say periodic) feature in a direction substantially perpendicular to the first direction (say the Y-axis).
- the light pattern includes several stripes arranged in parallel (or in near parallel) to each other, as shown in diagram 400 .
- a sensor that may be positioned in a certain Y-axis distance, near a transmitter which projects the stripes pattern on the hand 410 and on the background 420 (say a surface of a table the hand rests on, a wall, etc.).
- the position of the sensor is selected, so as to create a triangulation effect between the camera, the light projector and the light reflected back from the user's hand 410 and the background 420 , as known in the art.
- the triangulation effect causes discontinuities in the pattern at the points along a stripe where there are significant depth shifts from an object projected with a light pattern.
- the discontinuities segment (i.e., divide) the stripe into two or more stripe segments, say a segment 431 positioned on the hand, a segment 432 position to the left of the hand and a segment 433 position to the right of the hand.
- Such depth shift generated stripe segments may be located on the contours of the user's hand's palm or digits, which are positioned between the camera and the user's body. That is to say that the user's digit or palm segments the stripe into two or more stripe segments. Once such a stripe segment is detected, it is easy to follow the stripe segment, to the stripe segment's ends.
- the device may thus analyze bi-dimensional video data, to generate clusters of stripe segments. For example, the device may identify in the light pattern, a cluster of one or more stripe segments created by segmentation of stripes by a digit of the hand, say a cluster 441 of four segments reflected from the hand's central finger. Consequently, the device tracks the movement of the digit, by tracking the cluster of stripe segments created by segmentation of stripes by the digit, or by tracking at least one of the cluster's segments.
- the cluster of stripe segments created by segmentation (i.e., division) of stripes by the digit includes strip segments with an overlap in the X axis.
- the stripe segments in the cluster further have similar lengths (derived from the fingers thickness) or relative proximity in the Y-axis coordinates.
- the segments may have a full overlap for a digit positioned straightly, or a partial overlap for a digit positioned diagonally in the X-Y plane.
- the device further identifies a depth movement of the digit, say by detecting a change in the number of segments in the tracked cluster. For example, if the user stretches the user's central digit, the angle between the digit and the plane of the light projector and camera (X-Y plane) changes. Consequently, the number of segments in the cluster 441 is reduced from four to three.
- the device further identifies in the light pattern, one or more clusters of one or more stripe segments created by segmentation of stripes by a palm of the hand.
- the cluster of stripe segments created by segmentation of stripes by the palm includes an upper strip segment 431 which overlaps with the user hand's fingers stripe segment clusters, in the X axis.
- the upper strip segment 431 overlaps the four finger clusters in the X-axis, but do not exceed beyond the minimum and maximum X value of the four finger clusters' bottom segments.
- the cluster of stripe segments created by segmentation of stripes by the palm further includes, just below segment 431 , a few strip segments in significant overlap with the strip segment 431 .
- the cluster of stripe segments created by segmentation of stripes by the palm further includes longer stripe segments that extend to the base of a stripe segment cluster 451 of the user's thumb. It is understood that the digit and palm cluster's orientation may differ with specific hands positions and rotation.
- FIG. 5 is a flowchart 500 illustrating a high level process for implementing embodiments of the present invention. It is noted that flowchart 500 is not limited to the aforementioned architecture of device 100 .
- the method may include: capturing a scene of a field of view of a user 510 ; transmitting a structured light pattern onto a navigation space contained within said scene 520 ; capturing reflections of the specified pattern coming from the navigation space 530 ; and analyzing said reflections and control a visual indicator presented to a user over a wearable near eye display 540 .
- Methods of the present invention may be implemented by performing or completing manually, automatically, or a combination thereof, selected steps or tasks.
- the present invention may be implemented in the testing or practice with methods and materials equivalent or similar to those described herein.
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Computer Hardware Design (AREA)
- Computer Graphics (AREA)
- Software Systems (AREA)
- User Interface Of Digital Computer (AREA)
Abstract
Description
- This application claims benefit from U.S. Provisional Patent Application No. 61/863,519, filed on Aug. 8, 2013, which is incorporated herein by reference in its entirety.
- The present invention relates generally to the field of gesture recognition, and more specifically, using same for near eye display applications.
- Prior to the background of the invention being set forth, it may be helpful to set forth definitions of certain terms that will be used hereinafter.
- The term ‘near eye display’ as used herein is defined as devices which include augmented reality or virtual reality glasses and wearable projected displays.
- The term ‘field of view’ as used herein is defined as the extent of the observable world that is seen at any given moment by a user or a sensor.
- The term ‘Augmented Reality’ (AR) as used herein is defined as a live direct or indirect view of a physical, real-world environment whose elements are augmented (or supplemented) by computer-generated sensory input such as sound, video, graphics or any other data.
- The term ‘Virtual Reality’ (VR) as used herein is defined as a computer-simulated environment that can simulate physical presence in places in the real world or imagined worlds. Virtual reality could recreate sensory experiences, including virtual taste, sight, smell, sound, touch, and the like.
- Many devices use wearable displays for presenting information in front of the user in everyday activities. An example of such a device is the augmented reality (AR) glasses which positions a screen in front of the user's eye to display an additional layer of information and content. Although the information displayed is diverse and may hold various content categories and flexible graphical user interface, there is no convenient and flexible way for controlling the content displayed.
- Another aspect that restricts the functional ability to control the presented content is that it is displayed directly to the user's eye. In this case, actual touch interfaces are not an option for control, creating a need for separating the actual controlling area from the display while maintaining the extended abilities of touch control.
- In addition, portable devices, such as near-eye display devices are limited in terms of the acceptable power consumption and weight allowed for the device components. Therefore, using available gestures capturing technologies, based on an IR illumination source and an additional IR sensor is not practical due to the incremental weight and power consumption.
- Some embodiments of the present invention overcome the challenges of the prior art by, inter alia, using existing components of a near-eye device such as the frontal camera with a visible light structured pattern which enables to reduce the additional hardware needed to be embedded into the device as well as limiting the incremental power consumption.
- Some embodiments of the present invention include a transmitter module designed for projecting structured light pattern onto a virtual navigation space, a camera module positioned and oriented for capturing the structured light data, a processor defined for calculating three dimensional data and detecting the user's controlling gestures, and a display system for displaying information relating to the position of the fingertip and performed gestures.
- These additional, and/or other aspects and/or advantages of the embodiments of the present invention are set forth in the detailed description which follows; possibly inferable from the detailed description; and/or learnable by practice of the embodiments of the present invention.
- For a better understanding of embodiments of the invention and to show how the same may be carried into effect, reference will now be made, purely by way of example, to the accompanying drawings in which like numerals designate corresponding elements or sections throughout.
-
FIG. 1 illustrates the device and its environment according to some embodiments of the present invention; -
FIG. 2A is a block diagram illustrating the device in accordance with one embodiment of the present invention; -
FIG. 2B is a block diagram illustrating the device in accordance with another embodiment of the present invention; -
FIG. 3A is a diagram illustrating a non-limiting gesture and its outcome in accordance with some embodiments of the present invention; -
FIG. 3B is a diagram illustrating another non-limiting gesture and its outcome in accordance with some embodiments of the present invention; -
FIG. 3C is a diagram illustrating yet another non-limiting gesture and its outcome in accordance with some embodiments of the present invention; -
FIG. 3D is a diagram illustrating yet another non-limiting gesture and its outcome in accordance with some embodiments of the present invention; -
FIG. 4 is a diagram illustrating a non-limiting pattern projected onto a hand of a user serving as a controlling object in accordance with some embodiments of the present invention; and -
FIG. 5 is a flowchart illustrating a method according to some embodiments of the present invention. - With specific reference now to the drawings in detail, it is stressed that the particulars shown are by way of example and for purposes of illustrative discussion of the preferred embodiments of the present technique only, and are presented in the cause of providing what is believed to be the most useful and readily understood description of the principles and conceptual aspects of the present technique. In this regard, no attempt is made to show structural details of the present technique in more detail than is necessary for a fundamental understanding of the present technique, the description taken with the drawings making apparent to those skilled in the art how the several forms of the invention may be embodied in practice.
- Before at least one embodiment of the present technique is explained in detail, it is to be understood that the invention is not limited in its application to the details of construction and the arrangement of the components set forth in the following description or illustrated in the drawings. The present technique is applicable to other embodiments or of being practiced or carried out in various ways. Also, it is to be understood that the phraseology and terminology employed herein is for the purpose of description and should not be regarded as limiting.
- Some embodiments of the present invention address the challenge of controlling a cursor or any other added content presented to a user wearing a near eye display. In other words, any computer generated content that is super positioned over the captured scene, to yield an augmented reality or VR may be controlled by gestured and postures of the viewer, using embodiments of the present invention.
-
FIG. 1 illustrates the device according to some embodiments of the present invention in operation within its working environment. The device may be implemented as a wearable neareye display 100 which directly projects an image onto the eyes of auser 10. The device may include a sensor 110 (e.g., a camera) having a specified field ofview 112, attached to the wearable neareye display 100 and configured to capture a scene at the specified field of view overlapping a field of view of theuser 10. As can be seen in the inset,device 100 may take the form of eye glasses with some projector and image source coupled to the visor. - The device may further include a
transmitter 120 attached to the wearable near eye display.Transmitter 120 may be configured to transmit a specified light pattern onto anavigation space 120, wherein thesensor 110 is configured to capture reflections of the specified pattern coming from anavigation space 124. The source of light may a laser diode, a light emitting diode (LED), any another element which emits a light beam, as known in the art. - The light beam emitted by
transmitter 120 propagates through a micro structured element (not shown), onto the navigation space. The micro structured element modifies the light beam, for generating the light pattern projected onto the navigation space. - Optionally, the micro structured element converts the light beam into a light beam of a variable cross sectional intensity profile. Consequently, the cross sectional intensity profile varies along the light beam, thus providing information indicative of a distance of an object (say one of the user's digits) from the source of light.
- Optionally, the
transmitter 120 projects a light pattern having a continuous feature in a first direction and a non-continuous feature in a direction substantially perpendicular to the first direction, onto the navigation space. Optionally, the micro structured element is a diffractive optical element, as known in the art. - The diffractive optical element may be obtained by a periodic micro structure that splits the light beam into a discrete number of spots having a one dimensional or a two dimensional spatial arrangement, as known in the art. Optionally, an additional element such as a cylindrical micro lens array or an additional diffractive element is used to create a stripe from each of the spots.
- Optionally, the stripe is marked with phase notations positioned in fixed distances from each other, along the stripe.
- A possible advantage of using an additional microstructure element such as a cylindrical micro lens array is that the additional element may disperse a zero order of a light beam such as a laser beam, as known in the art. The zero order is a center of a light spot output from the diffractive element, and may be characterized by relatively high energy. Dispersing the zero order to a light structure such as the stripe, may allow increasing intensity of a laser light beam without crossing eye safety limits.
- Optionally,
transmitter 120 further includes an additional diffractive element, which changes the cross-sectional intensity profile of each stripe in the light pattern with respect to the distance from an object which reflects the projected stripe. The cross-sectional intensity profile is an intensity profile perpendicular to the propagation direction of the light projected. - Optionally, the intensity profile change is a gradual change from a Gaussian cross-sectional intensity profile to a Top Hat cross-sectional intensity profile, carried out gradually along a distance traversed by the projected light, as the projected light propagates to an object, and is reflected back from surface of the object.
- Optionally, the intensity profile change is a gradual change from an intensity profile with a single peak to an intensity profile with two or more peaks, etc., as known in the art.
- The change in the intensity profile, a long distance traversed by light reflected from an object projected by the light with the light pattern, may help differentiate between stripes reflected from different objects, in different ranges, and thus further helps overcome a segment unification problem.
- Further, the change in the intensity profile may further be used to directly measure distance to the controllable object (e.g., to the user's hand, fingers and thumb).
Sensor 110 may be a video camera such as a webcam or a cellular phone's camera, positioned next totransmitter 120. -
FIG. 2A is a block diagram illustrating thedevice 200A in accordance with one embodiment of the present invention.Device 200A may include a wearablenear eye display 250, asensor 210 attached to the wearable near eye display and configured to capture a scene.Device 200A may further include atransmitter 220 attached to the wearablenear eye display 250. As explained in detail above thetransmitter 220 is configured to transmit a structured light pattern onto a navigation space, whereinsensor 210 may further be configured to capture reflections of the specified pattern coming from the navigation space.Device 200A may further include acomputer processor 230 configured to analyze the reflections and control a visual indicator (not shown here) presented to a user over the wearablenear eye display 250, based on the analysis.Device 200A may take the form of eye glasses of any head mounted display as depicted in the inset ofFIG. 1 . -
FIG. 2B is a block diagram illustrating the device in accordance with another embodiment of the present invention. Here thedevice 200B is provided in an add-on or retrofit configuration and apart from thetransmitter 220B and asoftware application 240B, all other elements are provided as a workpiece or an environment. The system may thus include atransmitter 220B attachable to anear eye display 250B, a computerreadable code 240B executable on acomputer processor 230B and configured, when executed, to invoke thecomputer processor 230B to: instruct the transmitter to transmit a structured light pattern onto a navigation space, instruct a sensor attached to the wearable near eye display to capture a scene; instruct said sensor to capture reflections of the specified pattern coming from the navigation space; and analyze the reflections and control a visual indicator presented to a user over the wearablenear eye display 250B, based on the analysis. - As in other structured light techniques known in the art, the analysis carried out by the computer processor may be achieved by generating 3D depth map based on the reflections of the patterned light coming from the navigational space and further analyzing the 3D depth map.
- The structured light pattern is projected onto the virtual navigation area creating a volume where objects can be detected in 3D. The structured light may consist of dots pattern, Lines, curves or any other pattern that enable the ability of 3D depth mapping or object segmentation. Once an object is detected inside the detection volume and is activated as a controlling object, further properties are calculated in regards to the object position, orientation, shape, etc. These properties are then further processed to create interface events and to control a visual feedback of a pointing indication on the user interface of the device. The gestures event can be triggered by the control object's movement, position or a specific shape created by the object.
- According to some embodiments of the present invention, the
virtual navigation area 124 covers only part of the camera field ofview 112. Such part may consist of the lower part of the camera image while the majority of the camera field of view is kept free from projected light. In this way, the needed laser or LED power consumption that produces the structured light is reduced significantly. In addition, there is minor interference between the projected pattern and the standard usage of the device camera in that case. In order to project the structured light pattern on the edge of the camera field of view, an angle between the transmitter optical axis and the camera optical axis is presented. The effect of such an angle is encapsulated in the method of the depth extraction and the object segmentation. In addition, the transmitter can be positioned with a shift backwards on the line of the camera optical axis in relation with the sensor (camera) field of view direction. This enables additional flexibility in the device design and is taken into account in the pattern analysis method as well. - According to some embodiments of the present invention, the structured light pattern is projected using a light source in the visible spectral range. This gives the ability to use the existing frontal camera of the wearable device without removing the IR-cut filter that is standard for such camera. The quantum efficiency of the CMOS sensor enables better sensitivity in the visible spectral range reducing the energy needed for the structured light transmitter. Another important advantage of using visible light is that the control object becomes highlighted with the reflected light when entering the virtual navigation space. This indicates to the user on where to position his hand/finger in order to be within the borders of the navigation space.
- According to some embodiments of the present invention, the structured light is focused to a specific distance from
transmitter 120 utilizing a short depth of focus around that distance. This creates a limited region along the optical axis of the transmitter. Only inside that region the light pattern structure is detectable by the camera. Although it limits the space where the user can operate in, it also blurs the light pattern outside of that region and reduces the unwanted s reflection from objects in the scene in front of the user. - According to some embodiments of the present invention, the structured light pattern may consist of light features that are continues in one direction and non-continues in another direction. These features are used to enable segmentation of the control object from the background scenario and for detecting the contour of the object. In addition, various intensities may appear on different light feature creating a unique intensity signature between each sub group of the line features in the pattern. This enables detecting the index of each line feature precisely and determining the accurate depth of the object that reflects the pattern back to the camera. In another aspect, the light features may consist of notations, variation of intensity, or other feature alternation along the light feature itself to assist the segmentation of the control object as well as locally detect its depth.
-
FIG. 3A is a diagram illustrating a non-limiting gesture and its outcome in accordance with some embodiments of the present invention. Responsive to moving the hand fromposition 310A to position 312A along an X axis at the navigation space, the cursor is moving along a corresponding X′ axis fromposition 330A to position 332A. It is understood, however, that, for practical reasons, a similar displacement of the finger only of the hand from one position to another leads to a displacement of the cursor 9 visual indicator) at the display. - In general, the computer processor of the device is configured to map or translate one form of movement in real life (of the controlling object) at the x-y-z space into another movement at the display. Similar mappings are illustrated in
FIGS. 3B-3D where in 3B Y axis mapping is demonstrated withpositions positions display 320B. Alternatively, the mapping may convert movement along a Y axis at the navigational space into a movement along a corresponding Z axis at the display, in accordance with the requirements of the application. - In
FIGS. 3C and 3D, tilting the palm of the hand, a fist or at least one finger along a predefined arc (e.g., tilting or applying an angular displacement or movement fromposition 310C towards right inposition 312C or fromposition 310D towards up inposition 312D) causes moving the cursor in thedisplay 320C (e.g., from 330C to 332C or from 330D to 332D), in a gesture that may resemble controlling a joystick. - One of the key features of some embodiments of the present invention is the ability to separate the pointing direction of the controlling object and the position of the displayed graphics relating the user's perspective, including the pointing indication on the device display. This enables, besides the previously mentioned advantages of reducing the size of the navigation space, also some key controlling schemes. One of these schemes includes scaling or normalizing the position of the controlling device to enable larger or smaller movements to cover the entire graphical area presented to the user. In this way, the actual size of the virtual navigation space is not depended on the size and location of the presented object and can be determined based on the use case and user experience.
- In another aspect, the angle and orientation of the controlling object is used as an input to determine the position of the graphical pointing indication on the presented UI. For example, an angle created between the controlling object and the horizontal axis in relation to the transmitter optical axis can be translated to a position on the x-axis of the presented user interface. An angle created between the controlling object and the vertical axis in relation to the transmitter optical axis can be translated to a position on the y-axis of the presented interface.
- In another aspect of the invention, selecting gestures are presented. A selecting gesture may include one of the following postures: an addition of a controlling object to the virtual navigation space. Such controlling object may be an additional finger. The additional controlling object may be attached to the first controlling object previously located in the virtual navigation space. Another presented selecting gesture includes the rotation of the controlling object. For example, in the case that the controlling object is a hand with a stretched pointing finger, rotating the hand around the finger axis or pushing the finger forward will result in a click event. This enables clicks without changing the apposition and orientation of the controlling object therefore enable stable pointing during the click action.
- According to some embodiments, the sensor and the transmitter are operable only within visible light bandwidth. This way the pattern may also be used as visual feedback to the user—indicating that his or her hand is within the navigational space.
- According to some embodiments, the sensor configured to capture the scene and the reflections is a single element. By configuring a single sensor to do both tasks, a more cost effective and low power consumption device may be achieved.
- According to some embodiments, the control of the visual indicator comprises moving the visual indicator along a first axis of the display responsive to moving the controlling object along a second axis of the navigation space.
- According to some embodiments, the computer processor is further configured to identify, based on the reflections analysis at least one predefined action gesture, and invoke a corresponding action event associated with the visual indicator and content presented over the wearable near eye display.
- According to some embodiments, the navigational space is located at a lower side of the field of view. This way it does not obstruct the viewpoint of the user.
- According to some embodiments, the navigational space is limited in depth by a depth of field of the transmitter, for a depth in which the specified pattern is in focus.
-
FIG. 4 is a schematic diagram 400 illustrating a hand projected with a light pattern, according to an exemplary embodiment of the present invention. According to an exemplary embodiment, the tracking of the movement of the hand is carried out, using a light pattern designed to enable detection of hand movement, such as fine movements of fingers and thumbs. The use of structured light may be, for instance, as disclosed in WIPO PCT Patent Application Publication No. WO 2013/088442, which is incorporated herein by reference in its entirety. - The specifically designed light pattern allows the tracking of the movement, even in bi-dimensional video data, which unlike three dimensional depth map, does not provide for easy separation of the hands from the rest of the body.
- Optionally, the light pattern may be specifically designed to track movement of the hand's digits in a bi-dimensional video data (e.g., video images streamed from a regular video camera). More specifically, the light pattern may be designed to enable detection and tracking of digits (i.e., fingers and thumb) as well as palm, in the bi-dimensional video data, according to distortions of the pattern by the digits.
- Optionally, the light pattern has a continuous feature in a first direction (say the X-axis) and a non-continuous (say periodic) feature in a direction substantially perpendicular to the first direction (say the Y-axis). In one example for such a pattern, the light pattern includes several stripes arranged in parallel (or in near parallel) to each other, as shown in diagram 400.
- A sensor that may be positioned in a certain Y-axis distance, near a transmitter which projects the stripes pattern on the
hand 410 and on the background 420 (say a surface of a table the hand rests on, a wall, etc.). The position of the sensor is selected, so as to create a triangulation effect between the camera, the light projector and the light reflected back from the user'shand 410 and thebackground 420, as known in the art. The triangulation effect causes discontinuities in the pattern at the points along a stripe where there are significant depth shifts from an object projected with a light pattern. The discontinuities segment (i.e., divide) the stripe into two or more stripe segments, say asegment 431 positioned on the hand, asegment 432 position to the left of the hand and asegment 433 position to the right of the hand. - Such depth shift generated stripe segments may be located on the contours of the user's hand's palm or digits, which are positioned between the camera and the user's body. That is to say that the user's digit or palm segments the stripe into two or more stripe segments. Once such a stripe segment is detected, it is easy to follow the stripe segment, to the stripe segment's ends.
- The device may thus analyze bi-dimensional video data, to generate clusters of stripe segments. For example, the device may identify in the light pattern, a cluster of one or more stripe segments created by segmentation of stripes by a digit of the hand, say a
cluster 441 of four segments reflected from the hand's central finger. Consequently, the device tracks the movement of the digit, by tracking the cluster of stripe segments created by segmentation of stripes by the digit, or by tracking at least one of the cluster's segments. - The cluster of stripe segments created by segmentation (i.e., division) of stripes by the digit includes strip segments with an overlap in the X axis. Optionally, the stripe segments in the cluster further have similar lengths (derived from the fingers thickness) or relative proximity in the Y-axis coordinates.
- On the X-axis, the segments may have a full overlap for a digit positioned straightly, or a partial overlap for a digit positioned diagonally in the X-Y plane. Optionally, the device further identifies a depth movement of the digit, say by detecting a change in the number of segments in the tracked cluster. For example, if the user stretches the user's central digit, the angle between the digit and the plane of the light projector and camera (X-Y plane) changes. Consequently, the number of segments in the
cluster 441 is reduced from four to three. - Optionally, the device further identifies in the light pattern, one or more clusters of one or more stripe segments created by segmentation of stripes by a palm of the hand.
- The cluster of stripe segments created by segmentation of stripes by the palm includes an
upper strip segment 431 which overlaps with the user hand's fingers stripe segment clusters, in the X axis. Theupper strip segment 431 overlaps the four finger clusters in the X-axis, but do not exceed beyond the minimum and maximum X value of the four finger clusters' bottom segments. - The cluster of stripe segments created by segmentation of stripes by the palm further includes, just below
segment 431, a few strip segments in significant overlap with thestrip segment 431. The cluster of stripe segments created by segmentation of stripes by the palm further includes longer stripe segments that extend to the base of astripe segment cluster 451 of the user's thumb. It is understood that the digit and palm cluster's orientation may differ with specific hands positions and rotation. -
FIG. 5 is aflowchart 500 illustrating a high level process for implementing embodiments of the present invention. It is noted thatflowchart 500 is not limited to the aforementioned architecture ofdevice 100. The method may include: capturing a scene of a field of view of auser 510; transmitting a structured light pattern onto a navigation space contained within saidscene 520; capturing reflections of the specified pattern coming from thenavigation space 530; and analyzing said reflections and control a visual indicator presented to a user over a wearablenear eye display 540. - In the above description, an embodiment is an example or implementation of the inventions. The various appearances of “one embodiment,” “an embodiment” or “some embodiments” do not necessarily all refer to the same embodiments.
- Although various features of the invention may be described in the context of a single embodiment, the features may also be provided separately or in any suitable combination. Conversely, although the invention may be described herein in the context of separate embodiments for clarity, the invention may also be implemented in a single embodiment.
- Reference in the specification to “some embodiments”, “an embodiment”, “one embodiment” or “other embodiments” means that a particular feature, structure, or characteristic described in connection with the embodiments is included in at least some embodiments, but not necessarily all embodiments, of the inventions.
- It is to be understood that the phraseology and terminology employed herein is not to be construed as limiting and are for descriptive purpose only.
- The principles and uses of the teachings of the present invention may be better understood with reference to the accompanying description, figures and examples.
- It is to be understood that the details set forth herein do not construe a limitation to an application of the invention.
- Furthermore, it is to be understood that the invention can be carried out or practiced in various ways and that the invention can be implemented in embodiments other than the ones outlined in the description above.
- It is to be understood that the terms “including”, “comprising”, “consisting” and grammatical variants thereof do not preclude the addition of one or more components, features, steps, or integers or groups thereof and that the terms are to be construed as specifying components, features, steps or integers.
- If the specification or claims refer to “an additional” element, that does not preclude there being more than one of the additional element.
- It is to be understood that where the claims or specification refer to “a” or “an” element, such reference is not be construed that there is only one of that element.
- It is to be understood that where the specification states that a component, feature, structure, or characteristic “may”, “might”, “can” or “could” be included, that particular component, feature, structure, or characteristic is not required to be included.
- Where applicable, although state diagrams, flow diagrams or both may be used to describe embodiments, the invention is not limited to those diagrams or to the corresponding descriptions. For example, flow need not move through each illustrated box or state, or in exactly the same order as illustrated and described.
- Methods of the present invention may be implemented by performing or completing manually, automatically, or a combination thereof, selected steps or tasks.
- The descriptions, examples, methods and materials presented in the claims and the specification are not to be construed as limiting but rather as illustrative only.
- Meanings of technical and scientific terms used herein are to be commonly understood as by one of ordinary skill in the art to which the invention belongs, unless otherwise defined.
- The present invention may be implemented in the testing or practice with methods and materials equivalent or similar to those described herein.
- While the invention has been described with respect to a limited number of embodiments, these should not be construed as limitations on the scope of the invention, but rather as exemplifications of some of the preferred embodiments. Other possible variations, modifications, and applications are also within the scope of the invention. Accordingly, the scope of the invention should not be limited by what has thus far been described, but by the appended claims and their legal equivalents.
Claims (26)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US14/453,733 US10019843B2 (en) | 2013-08-08 | 2014-08-07 | Controlling a near eye display |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US201361863510P | 2013-08-08 | 2013-08-08 | |
US14/453,733 US10019843B2 (en) | 2013-08-08 | 2014-08-07 | Controlling a near eye display |
Publications (2)
Publication Number | Publication Date |
---|---|
US20150042680A1 true US20150042680A1 (en) | 2015-02-12 |
US10019843B2 US10019843B2 (en) | 2018-07-10 |
Family
ID=52448242
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US14/453,733 Active 2034-12-27 US10019843B2 (en) | 2013-08-08 | 2014-08-07 | Controlling a near eye display |
Country Status (2)
Country | Link |
---|---|
US (1) | US10019843B2 (en) |
CN (1) | CN104345802B (en) |
Cited By (19)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20120194561A1 (en) * | 2009-09-22 | 2012-08-02 | Nadav Grossinger | Remote control of computer devices |
WO2016138143A1 (en) | 2015-02-25 | 2016-09-01 | Oculus Vr, Llc | Using intensity variations in a light pattern for depth mapping of objects in a volume |
WO2016138145A1 (en) | 2015-02-25 | 2016-09-01 | Oculus Vr, Llc | Identifying an object in a volume based on characteristics of light reflected by the object |
CN106095133A (en) * | 2016-05-31 | 2016-11-09 | 广景视睿科技(深圳)有限公司 | A kind of method and system of alternative projection |
US20160335773A1 (en) * | 2015-05-13 | 2016-11-17 | Oculus Vr, Llc | Augmenting a depth map representation with a reflectivity map representation |
US9552074B2 (en) | 2014-05-21 | 2017-01-24 | Facebook, Inc. | Method and system for generating user feedback of a gesture capturing device |
TWI571131B (en) * | 2016-03-16 | 2017-02-11 | 和碩聯合科技股份有限公司 | Method of reseting shooting direction of near-eye display device, near-eye display device and computer program product |
US20170083114A1 (en) * | 2015-09-18 | 2017-03-23 | Kabushiki Kaisha Toshiba | Electronic apparatus and method |
WO2017084253A1 (en) * | 2015-11-20 | 2017-05-26 | 乐视控股(北京)有限公司 | Control method applied to head-mounted device and head-mounted device |
US9858703B2 (en) | 2014-12-18 | 2018-01-02 | Facebook, Inc. | System, device and method for providing user interface for a virtual reality environment |
US9870068B2 (en) | 2010-09-19 | 2018-01-16 | Facebook, Inc. | Depth mapping with a head mounted display using stereo cameras and structured light |
US10031588B2 (en) | 2015-03-22 | 2018-07-24 | Facebook, Inc. | Depth mapping with a head mounted display using stereo cameras and structured light |
US10091494B2 (en) | 2013-10-23 | 2018-10-02 | Facebook, Inc. | Three dimensional depth mapping using dynamic structured light |
CN110622124A (en) * | 2017-03-08 | 2019-12-27 | 奥斯坦多科技公司 | Compression method and system for near-eye displays |
US10877622B2 (en) * | 2018-08-29 | 2020-12-29 | Facebook Technologies, Llc | Detection of structured light for depth sensing |
US11360551B2 (en) * | 2016-06-28 | 2022-06-14 | Hiscene Information Technology Co., Ltd | Method for displaying user interface of head-mounted display device |
US11398047B2 (en) * | 2019-05-30 | 2022-07-26 | Nvidia Corporation | Virtual reality simulations using surface tracking |
US11507190B2 (en) | 2016-07-26 | 2022-11-22 | Huawei Technologies Co., Ltd. | Gesture control method applied to VR device, and apparatus |
US20230083605A1 (en) * | 2021-09-16 | 2023-03-16 | Globus Medical, Inc. | Extended reality systems for visualizing and controlling operating room equipment |
Families Citing this family (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10019843B2 (en) | 2013-08-08 | 2018-07-10 | Facebook, Inc. | Controlling a near eye display |
CN104834356B (en) * | 2015-05-06 | 2018-04-13 | 百度在线网络技术(北京)有限公司 | The method and device of headset equipment and its visual feedback |
DE112016006840T5 (en) * | 2016-05-11 | 2019-03-21 | Intel Corporation | CONTROLLING A TELEROBOTER BASED ON MOTION MAPPING |
CN106101689B (en) * | 2016-06-13 | 2018-03-06 | 西安电子科技大学 | The method that using mobile phone monocular cam virtual reality glasses are carried out with augmented reality |
CN108073432B (en) * | 2016-11-07 | 2020-12-22 | 亮风台(上海)信息科技有限公司 | User interface display method of head-mounted display equipment |
US10528133B2 (en) * | 2018-03-13 | 2020-01-07 | Facebook Technologies, Llc | Bracelet in a distributed artificial reality system |
KR20220047035A (en) | 2020-10-08 | 2022-04-15 | 삼성전자주식회사 | Device and method for measuring depth of object |
Citations (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20120235906A1 (en) * | 2011-03-16 | 2012-09-20 | Electronics And Telecommunications Research Institute | Apparatus and method for inputting information based on events |
US20130016070A1 (en) * | 2011-07-12 | 2013-01-17 | Google Inc. | Methods and Systems for a Virtual Input Device |
US20130229396A1 (en) * | 2012-03-05 | 2013-09-05 | Kenneth J. Huebner | Surface aware, object aware, and image aware handheld projector |
US20130229499A1 (en) * | 2012-03-05 | 2013-09-05 | Microsoft Corporation | Generation of depth images based upon light falloff |
US20130335302A1 (en) * | 2012-06-18 | 2013-12-19 | Randall T. Crane | Selective illumination |
US20140198017A1 (en) * | 2013-01-12 | 2014-07-17 | Mathew J. Lamb | Wearable Behavior-Based Vision System |
Family Cites Families (9)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN101388114B (en) * | 2008-09-03 | 2011-11-23 | 北京中星微电子有限公司 | Method and system for estimating human body attitudes |
US9901828B2 (en) * | 2010-03-30 | 2018-02-27 | Sony Interactive Entertainment America Llc | Method for an augmented reality character to maintain and exhibit awareness of an observer |
US20130154913A1 (en) * | 2010-12-16 | 2013-06-20 | Siemens Corporation | Systems and methods for a gaze and gesture interface |
CN102193631A (en) * | 2011-05-05 | 2011-09-21 | 上海大学 | Wearable three-dimensional gesture interaction system and using method thereof |
WO2013088442A1 (en) | 2011-12-15 | 2013-06-20 | Pebbles Ltd. | Controllable optical sensing |
CN105378616A (en) * | 2013-04-12 | 2016-03-02 | 爱康尼克斯公司 | Virtual touch screen |
US10019843B2 (en) | 2013-08-08 | 2018-07-10 | Facebook, Inc. | Controlling a near eye display |
US9877080B2 (en) * | 2013-09-27 | 2018-01-23 | Samsung Electronics Co., Ltd. | Display apparatus and method for controlling thereof |
WO2016169221A1 (en) * | 2015-04-20 | 2016-10-27 | 我先有限公司 | Virtual reality device and operating mode |
-
2014
- 2014-08-07 US US14/453,733 patent/US10019843B2/en active Active
- 2014-08-08 CN CN201410389973.9A patent/CN104345802B/en active Active
Patent Citations (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20120235906A1 (en) * | 2011-03-16 | 2012-09-20 | Electronics And Telecommunications Research Institute | Apparatus and method for inputting information based on events |
US20130016070A1 (en) * | 2011-07-12 | 2013-01-17 | Google Inc. | Methods and Systems for a Virtual Input Device |
US20130229396A1 (en) * | 2012-03-05 | 2013-09-05 | Kenneth J. Huebner | Surface aware, object aware, and image aware handheld projector |
US20130229499A1 (en) * | 2012-03-05 | 2013-09-05 | Microsoft Corporation | Generation of depth images based upon light falloff |
US20130335302A1 (en) * | 2012-06-18 | 2013-12-19 | Randall T. Crane | Selective illumination |
US20140198017A1 (en) * | 2013-01-12 | 2014-07-17 | Mathew J. Lamb | Wearable Behavior-Based Vision System |
Cited By (39)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20120194561A1 (en) * | 2009-09-22 | 2012-08-02 | Nadav Grossinger | Remote control of computer devices |
US9507411B2 (en) * | 2009-09-22 | 2016-11-29 | Facebook, Inc. | Hand tracker for device with display |
US9927881B2 (en) | 2009-09-22 | 2018-03-27 | Facebook, Inc. | Hand tracker for device with display |
US9606618B2 (en) | 2009-09-22 | 2017-03-28 | Facebook, Inc. | Hand tracker for device with display |
US9870068B2 (en) | 2010-09-19 | 2018-01-16 | Facebook, Inc. | Depth mapping with a head mounted display using stereo cameras and structured light |
US11962748B2 (en) | 2013-10-23 | 2024-04-16 | Meta Platforms Technologies, Llc | Three dimensional depth mapping using dynamic structured light |
US11057610B2 (en) | 2013-10-23 | 2021-07-06 | Facebook Technologies, Llc | Three dimensional depth mapping using dynamic structured light |
US10687047B2 (en) | 2013-10-23 | 2020-06-16 | Facebook Technologies, Llc | Three dimensional depth mapping using dynamic structured light |
US10091494B2 (en) | 2013-10-23 | 2018-10-02 | Facebook, Inc. | Three dimensional depth mapping using dynamic structured light |
US9552074B2 (en) | 2014-05-21 | 2017-01-24 | Facebook, Inc. | Method and system for generating user feedback of a gesture capturing device |
US10559113B2 (en) | 2014-12-18 | 2020-02-11 | Facebook Technologies, Llc | System, device and method for providing user interface for a virtual reality environment |
US9858703B2 (en) | 2014-12-18 | 2018-01-02 | Facebook, Inc. | System, device and method for providing user interface for a virtual reality environment |
KR20190083002A (en) * | 2015-02-25 | 2019-07-10 | 페이스북, 인크. | Identifying an object in a volume based on characteristics of light reflected by the object |
WO2016138145A1 (en) | 2015-02-25 | 2016-09-01 | Oculus Vr, Llc | Identifying an object in a volume based on characteristics of light reflected by the object |
KR101954855B1 (en) | 2015-02-25 | 2019-03-06 | 페이스북, 인크. | Use of intensity variations of light patterns for depth mapping of objects in a volume |
JP2018508019A (en) * | 2015-02-25 | 2018-03-22 | フェイスブック,インク. | Depth mapping of objects in the volume using intensity variation of light pattern |
JP2018508074A (en) * | 2015-02-25 | 2018-03-22 | フェイスブック,インク. | Identification of objects in the volume based on the characteristics of the light reflected by the objects |
KR20170120655A (en) * | 2015-02-25 | 2017-10-31 | 페이스북, 인크. | Use of intensity variations of light patterns for depth mapping of objects in a volume |
US9934574B2 (en) | 2015-02-25 | 2018-04-03 | Facebook, Inc. | Using intensity variations in a light pattern for depth mapping of objects in a volume |
EP4141793A1 (en) * | 2015-02-25 | 2023-03-01 | Facebook Technologies, LLC | Using intensity variations in a light pattern for depth mapping of objects in a volume |
EP3123288A4 (en) * | 2015-02-25 | 2017-11-22 | Facebook, Inc. | Identifying an object in a volume based on characteristics of light reflected by the object |
US10049460B2 (en) | 2015-02-25 | 2018-08-14 | Facebook, Inc. | Identifying an object in a volume based on characteristics of light reflected by the object |
KR102129376B1 (en) * | 2015-02-25 | 2020-07-02 | 페이스북, 인크. | Identifying an object in a volume based on characteristics of light reflected by the object |
AU2016222716B2 (en) * | 2015-02-25 | 2018-11-29 | Facebook Technologies, Llc | Identifying an object in a volume based on characteristics of light reflected by the object |
WO2016138143A1 (en) | 2015-02-25 | 2016-09-01 | Oculus Vr, Llc | Using intensity variations in a light pattern for depth mapping of objects in a volume |
US10031588B2 (en) | 2015-03-22 | 2018-07-24 | Facebook, Inc. | Depth mapping with a head mounted display using stereo cameras and structured light |
US9947098B2 (en) * | 2015-05-13 | 2018-04-17 | Facebook, Inc. | Augmenting a depth map representation with a reflectivity map representation |
US20160335773A1 (en) * | 2015-05-13 | 2016-11-17 | Oculus Vr, Llc | Augmenting a depth map representation with a reflectivity map representation |
US20170083114A1 (en) * | 2015-09-18 | 2017-03-23 | Kabushiki Kaisha Toshiba | Electronic apparatus and method |
WO2017084253A1 (en) * | 2015-11-20 | 2017-05-26 | 乐视控股(北京)有限公司 | Control method applied to head-mounted device and head-mounted device |
TWI571131B (en) * | 2016-03-16 | 2017-02-11 | 和碩聯合科技股份有限公司 | Method of reseting shooting direction of near-eye display device, near-eye display device and computer program product |
CN106095133A (en) * | 2016-05-31 | 2016-11-09 | 广景视睿科技(深圳)有限公司 | A kind of method and system of alternative projection |
US11360551B2 (en) * | 2016-06-28 | 2022-06-14 | Hiscene Information Technology Co., Ltd | Method for displaying user interface of head-mounted display device |
US11507190B2 (en) | 2016-07-26 | 2022-11-22 | Huawei Technologies Co., Ltd. | Gesture control method applied to VR device, and apparatus |
CN110622124A (en) * | 2017-03-08 | 2019-12-27 | 奥斯坦多科技公司 | Compression method and system for near-eye displays |
US10877622B2 (en) * | 2018-08-29 | 2020-12-29 | Facebook Technologies, Llc | Detection of structured light for depth sensing |
US11398047B2 (en) * | 2019-05-30 | 2022-07-26 | Nvidia Corporation | Virtual reality simulations using surface tracking |
US20230083605A1 (en) * | 2021-09-16 | 2023-03-16 | Globus Medical, Inc. | Extended reality systems for visualizing and controlling operating room equipment |
US20230078919A1 (en) * | 2021-09-16 | 2023-03-16 | Globus Medical, Inc. | Extended reality systems for visualizing and controlling operating room equipment |
Also Published As
Publication number | Publication date |
---|---|
CN104345802A (en) | 2015-02-11 |
US10019843B2 (en) | 2018-07-10 |
CN104345802B (en) | 2019-03-22 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US10019843B2 (en) | Controlling a near eye display | |
JP6348211B2 (en) | Remote control of computer equipment | |
US11816296B2 (en) | External user interface for head worn computing | |
US9972136B2 (en) | Method, system and device for navigating in a virtual reality environment | |
US20210294415A1 (en) | External user interface for head worn computing | |
KR102335132B1 (en) | Multi-modal gesture based interactive system and method using one single sensing system | |
KR101141087B1 (en) | Processing of gesture-based user interactions | |
US20170100664A1 (en) | External user interface for head worn computing | |
US20150205351A1 (en) | External user interface for head worn computing | |
KR20140140095A (en) | Enhanced virtual touchpad and touchscreen | |
US20160139762A1 (en) | Aligning gaze and pointing directions | |
US11144113B2 (en) | System and method for human interaction with virtual objects using reference device with fiducial pattern | |
US10229313B1 (en) | System and method for identifying and tracking a human hand in an interactive space based on approximated center-lines of digits | |
KR100959649B1 (en) | Adaptive user interface system and method |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: PEBBLES LTD., ISRAEL Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:GROSSINGER, NADAV;ALPERN, YAIR;REEL/FRAME:035946/0458 Effective date: 20150627 |
|
AS | Assignment |
Owner name: OCULUS VR, LLC, CALIFORNIA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:PEBBLES LTD.;REEL/FRAME:037545/0701 Effective date: 20160107 |
|
AS | Assignment |
Owner name: FACEBOOK, INC., CALIFORNIA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:OCULUS VR, LLC;REEL/FRAME:040196/0790 Effective date: 20160913 |
|
FEPP | Fee payment procedure |
Free format text: ENTITY STATUS SET TO UNDISCOUNTED (ORIGINAL EVENT CODE: BIG.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |
|
AS | Assignment |
Owner name: FACEBOOK TECHNOLOGIES, LLC, CALIFORNIA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:FACEBOOK, INC.;REEL/FRAME:047687/0942 Effective date: 20181024 |
|
CC | Certificate of correction | ||
MAFP | Maintenance fee payment |
Free format text: PAYMENT OF MAINTENANCE FEE, 4TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1551); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY Year of fee payment: 4 |
|
AS | Assignment |
Owner name: META PLATFORMS TECHNOLOGIES, LLC, CALIFORNIA Free format text: CHANGE OF NAME;ASSIGNOR:FACEBOOK TECHNOLOGIES, LLC;REEL/FRAME:060315/0224 Effective date: 20220318 |