EP3797345A4 - Transmodal input fusion for a wearable system - Google Patents

Transmodal input fusion for a wearable system Download PDF

Info

Publication number
EP3797345A4
EP3797345A4 EP19806507.0A EP19806507A EP3797345A4 EP 3797345 A4 EP3797345 A4 EP 3797345A4 EP 19806507 A EP19806507 A EP 19806507A EP 3797345 A4 EP3797345 A4 EP 3797345A4
Authority
EP
European Patent Office
Prior art keywords
transmodal
wearable system
input fusion
fusion
input
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
EP19806507.0A
Other languages
German (de)
French (fr)
Other versions
EP3797345A1 (en
Inventor
Paul Lacey
Samuel A. Miller
Nicholas Atkinson KRAMER
David Charles Lundmark
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Magic Leap Inc
Original Assignee
Magic Leap Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Magic Leap Inc filed Critical Magic Leap Inc
Publication of EP3797345A1 publication Critical patent/EP3797345A1/en
Publication of EP3797345A4 publication Critical patent/EP3797345A4/en
Pending legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating 3D models or images for computer graphics
    • G06T19/006—Mixed reality
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G06F3/012—Head tracking input arrangements
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G06F3/013—Eye tracking input arrangements
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/017—Gesture based interaction, e.g. based on a set of recognized hand gestures
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/16—Sound input; Sound output
    • G06F3/167—Audio in a user interface, e.g. using voice commands for navigating, audio feedback
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/20—Image enhancement or restoration by the use of local operators
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2203/00—Indexing scheme relating to G06F3/00 - G06F3/048
    • G06F2203/038—Indexing scheme relating to G06F3/038
    • G06F2203/0381—Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2200/00—Indexing scheme for image data processing or generation, in general
    • G06T2200/04—Indexing scheme for image data processing or generation, in general involving 3D image data
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2200/00—Indexing scheme for image data processing or generation, in general
    • G06T2200/24—Indexing scheme for image data processing or generation, in general involving graphical user interfaces [GUIs]
EP19806507.0A 2018-05-22 2019-05-21 Transmodal input fusion for a wearable system Pending EP3797345A4 (en)

Applications Claiming Priority (3)

Application Number Priority Date Filing Date Title
US201862675164P 2018-05-22 2018-05-22
US201862692519P 2018-06-29 2018-06-29
PCT/US2019/033378 WO2019226691A1 (en) 2018-05-22 2019-05-21 Transmodal input fusion for a wearable system

Publications (2)

Publication Number Publication Date
EP3797345A1 EP3797345A1 (en) 2021-03-31
EP3797345A4 true EP3797345A4 (en) 2022-03-09

Family

ID=68614766

Family Applications (1)

Application Number Title Priority Date Filing Date
EP19806507.0A Pending EP3797345A4 (en) 2018-05-22 2019-05-21 Transmodal input fusion for a wearable system

Country Status (5)

Country Link
US (2) US10861242B2 (en)
EP (1) EP3797345A4 (en)
JP (2) JP7341166B2 (en)
CN (1) CN112424727A (en)
WO (1) WO2019226691A1 (en)

Families Citing this family (164)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2016508007A (en) 2013-02-07 2016-03-10 アップル インコーポレイテッド Voice trigger for digital assistant
US20150124566A1 (en) 2013-10-04 2015-05-07 Thalmic Labs Inc. Systems, articles and methods for wearable electronic devices employing contact sensors
US10188309B2 (en) 2013-11-27 2019-01-29 North Inc. Systems, articles, and methods for electromyography sensors
US11921471B2 (en) 2013-08-16 2024-03-05 Meta Platforms Technologies, Llc Systems, articles, and methods for wearable devices having secondary power sources in links of a band for providing secondary power in addition to a primary power source
US10042422B2 (en) 2013-11-12 2018-08-07 Thalmic Labs Inc. Systems, articles, and methods for capacitive electromyography sensors
US9338493B2 (en) 2014-06-30 2016-05-10 Apple Inc. Intelligent automated assistant for TV user interactions
US10768704B2 (en) 2015-03-17 2020-09-08 Whirlwind VR, Inc. System and method for modulating a peripheral device based on an unscripted feed using computer vision
US10747498B2 (en) 2015-09-08 2020-08-18 Apple Inc. Zero latency digital assistant
JP6684559B2 (en) * 2015-09-16 2020-04-22 株式会社バンダイナムコエンターテインメント Program and image generation device
US11106273B2 (en) 2015-10-30 2021-08-31 Ostendo Technologies, Inc. System and methods for on-body gestural interfaces and projection displays
US10691473B2 (en) 2015-11-06 2020-06-23 Apple Inc. Intelligent automated assistant in a messaging environment
US10591988B2 (en) * 2016-06-28 2020-03-17 Hiscene Information Technology Co., Ltd Method for displaying user interface of head-mounted display device
WO2018022602A1 (en) 2016-07-25 2018-02-01 Ctrl-Labs Corporation Methods and apparatus for predicting musculo-skeletal position information using wearable autonomous sensors
US11216069B2 (en) 2018-05-08 2022-01-04 Facebook Technologies, Llc Systems and methods for improved speech recognition using neuromuscular information
CN110337269B (en) 2016-07-25 2021-09-21 脸谱科技有限责任公司 Method and apparatus for inferring user intent based on neuromuscular signals
US10687759B2 (en) 2018-05-29 2020-06-23 Facebook Technologies, Llc Shielding techniques for noise reduction in surface electromyography signal measurement and related systems and methods
WO2020112986A1 (en) 2018-11-27 2020-06-04 Facebook Technologies, Inc. Methods and apparatus for autocalibration of a wearable electrode sensor system
US11337652B2 (en) 2016-07-25 2022-05-24 Facebook Technologies, Llc System and method for measuring the movements of articulated rigid bodies
JP6440910B2 (en) * 2016-07-29 2018-12-19 三菱電機株式会社 Display device, display control device, and display control method
US11507216B2 (en) 2016-12-23 2022-11-22 Realwear, Inc. Customizing user interfaces of binary applications
US10620910B2 (en) 2016-12-23 2020-04-14 Realwear, Inc. Hands-free navigation of touch-based operating systems
US11099716B2 (en) 2016-12-23 2021-08-24 Realwear, Inc. Context based content navigation for wearable display
DK201770429A1 (en) 2017-05-12 2018-12-14 Apple Inc. Low-latency intelligent automated assistant
DK179496B1 (en) 2017-05-12 2019-01-15 Apple Inc. USER-SPECIFIC Acoustic Models
US11373650B2 (en) * 2017-10-17 2022-06-28 Sony Corporation Information processing device and information processing method
CN112040858A (en) 2017-10-19 2020-12-04 脸谱科技有限责任公司 System and method for identifying biological structures associated with neuromuscular source signals
CN107977586B (en) * 2017-12-22 2021-04-13 联想(北京)有限公司 Display content processing method, first electronic device and second electronic device
US11961494B1 (en) 2019-03-29 2024-04-16 Meta Platforms Technologies, Llc Electromagnetic interference reduction in extended reality environments
US11481030B2 (en) 2019-03-29 2022-10-25 Meta Platforms Technologies, Llc Methods and apparatus for gesture detection and classification
US11493993B2 (en) 2019-09-04 2022-11-08 Meta Platforms Technologies, Llc Systems, methods, and interfaces for performing inputs based on neuromuscular control
CN112074870A (en) 2018-01-25 2020-12-11 脸谱科技有限责任公司 Visualization of reconstructed hand state information
US10937414B2 (en) 2018-05-08 2021-03-02 Facebook Technologies, Llc Systems and methods for text input using neuromuscular information
US11907423B2 (en) * 2019-11-25 2024-02-20 Meta Platforms Technologies, Llc Systems and methods for contextualized interactions with an environment
US11150730B1 (en) 2019-04-30 2021-10-19 Facebook Technologies, Llc Devices, systems, and methods for controlling computing devices via neuromuscular signals of users
US10928918B2 (en) 2018-05-07 2021-02-23 Apple Inc. Raise to speak
US10592001B2 (en) 2018-05-08 2020-03-17 Facebook Technologies, Llc Systems and methods for improved speech recognition using neuromuscular information
CN112424727A (en) 2018-05-22 2021-02-26 奇跃公司 Cross-modal input fusion for wearable systems
WO2019241701A1 (en) 2018-06-14 2019-12-19 Ctrl-Labs Corporation User identification and authentication with neuromuscular signatures
US10976999B1 (en) * 2018-06-15 2021-04-13 Chosen Realities, LLC Mixed reality sensor suite and interface for physical region enhancement
US11169601B2 (en) * 2018-07-12 2021-11-09 Toyota Research Institute, Inc. Methods and systems for determining teleoperating user intent via eye tracking
US11045137B2 (en) 2018-07-19 2021-06-29 Facebook Technologies, Llc Methods and apparatus for improved signal robustness for a wearable neuromuscular recording device
WO2020036958A1 (en) 2018-08-13 2020-02-20 Ctrl-Labs Corporation Real-time spike detection and identification
US10842407B2 (en) 2018-08-31 2020-11-24 Facebook Technologies, Llc Camera-guided interpretation of neuromuscular signals
EP3853698A4 (en) 2018-09-20 2021-11-17 Facebook Technologies, LLC Neuromuscular text entry, writing and drawing in augmented reality systems
JP2020047062A (en) * 2018-09-20 2020-03-26 Ｄｙｎａｂｏｏｋ株式会社 Electronic device and control method
WO2020069181A1 (en) * 2018-09-26 2020-04-02 Ctrl-Labs Corporation Neuromuscular control of physical objects in an environment
US11462215B2 (en) 2018-09-28 2022-10-04 Apple Inc. Multi-modal inputs for voice commands
USD916723S1 (en) * 2018-10-05 2021-04-20 Google Llc Display screen with animated graphical user interface
CN112822992A (en) 2018-10-05 2021-05-18 脸谱科技有限责任公司 Providing enhanced interaction with physical objects using neuromuscular signals in augmented reality environments
US11351671B2 (en) * 2018-10-29 2022-06-07 Toyota Research Institute, Inc. Intent based control of a robotic device
US11442549B1 (en) * 2019-02-07 2022-09-13 Apple Inc. Placement of 3D effects based on 2D paintings
US11059582B2 (en) 2019-02-11 2021-07-13 Cnh Industrial Canada, Ltd. Systems for acquiring field condition data
US11001380B2 (en) * 2019-02-11 2021-05-11 Cnh Industrial Canada, Ltd. Methods for acquiring field condition data
US20200387220A1 (en) * 2019-02-18 2020-12-10 Tobii Ab Combined gaze-based and scanning-based control of an apparatus
US11507019B2 (en) * 2019-02-23 2022-11-22 Microsoft Technology Licensing, Llc Displaying holograms via hand location
US10905383B2 (en) 2019-02-28 2021-02-02 Facebook Technologies, Llc Methods and apparatus for unsupervised one-shot machine learning for classification of human gestures and estimation of applied forces
WO2020176965A1 (en) * 2019-03-01 2020-09-10 Scorched Ice Inc. Systems and methods for recreating or augmenting real-time events using sensor-based virtual reality, augmented reality, or extended reality
CN111752425B (en) * 2019-03-27 2022-02-15 北京外号信息技术有限公司 Method for selecting an interactive object on a display medium of a device
CN110347305A (en) * 2019-05-30 2019-10-18 华为技术有限公司 A kind of VR multi-display method and electronic equipment
US11468890B2 (en) 2019-06-01 2022-10-11 Apple Inc. Methods and user interfaces for voice-based control of electronic devices
US10825163B1 (en) 2019-06-13 2020-11-03 Viavi Solutions Inc. Automatically executing a test to inspect an end face of an optical fiber when an image is in focus
JP2021005157A (en) * 2019-06-25 2021-01-14 株式会社ソニー・インタラクティブエンタテインメント Image processing apparatus and image processing method
KR102582407B1 (en) * 2019-07-28 2023-09-26 구글 엘엘씨 Methods, systems, and media for rendering immersive video content with foveated meshes
KR102570009B1 (en) * 2019-07-31 2023-08-23 삼성전자주식회사 Electronic device and method for generating argument reality object
US11042259B2 (en) * 2019-08-18 2021-06-22 International Business Machines Corporation Visual hierarchy design governed user interface modification via augmented reality
US10943388B1 (en) * 2019-09-06 2021-03-09 Zspace, Inc. Intelligent stylus beam and assisted probabilistic input to element mapping in 2D and 3D graphical user interfaces
US11275443B1 (en) * 2019-09-23 2022-03-15 Facebook Technologies, Llc Variable-resistance actuator
CN110908504B (en) * 2019-10-10 2021-03-23 浙江大学 Augmented reality museum collaborative interaction method and system
US11054923B2 (en) * 2019-10-11 2021-07-06 Finch Technologies Ltd. Automatic switching between different modes of tracking user motions to control computer applications
US11315326B2 (en) * 2019-10-15 2022-04-26 At&T Intellectual Property I, L.P. Extended reality anchor caching based on viewport prediction
US11289086B2 (en) * 2019-11-01 2022-03-29 Microsoft Technology Licensing, Llc Selective response rendering for virtual assistants
US20210157394A1 (en) 2019-11-24 2021-05-27 XRSpace CO., LTD. Motion tracking system and method
CN112904997B (en) * 2019-12-04 2023-05-26 Oppo广东移动通信有限公司 Equipment control method and related product
CN111103976B (en) * 2019-12-05 2023-05-02 深圳职业技术学院 Gesture recognition method and device and electronic equipment
US11308698B2 (en) * 2019-12-05 2022-04-19 Facebook Technologies, Llc. Using deep learning to determine gaze
US11175730B2 (en) 2019-12-06 2021-11-16 Facebook Technologies, Llc Posture-based virtual space configurations
US11816757B1 (en) * 2019-12-11 2023-11-14 Meta Platforms Technologies, Llc Device-side capture of data representative of an artificial reality environment
CN113124852B (en) * 2019-12-31 2023-07-25 北京凌宇智控科技有限公司 Multi-sensor fusion positioning system and positioning method
US11687778B2 (en) 2020-01-06 2023-06-27 The Research Foundation For The State University Of New York Fakecatcher: detection of synthetic portrait videos using biological signals
WO2021173839A1 (en) 2020-02-26 2021-09-02 Magic Leap, Inc. Hand gesture input for wearable system
US11455078B1 (en) 2020-03-31 2022-09-27 Snap Inc. Spatial navigation and creation interface
US20210373649A1 (en) * 2020-04-01 2021-12-02 Rajesh Dayal Moorjani Attaching Extended Reality (XR) Content To Physical Points Of Interest
US11481965B2 (en) * 2020-04-10 2022-10-25 Samsung Electronics Co., Ltd. Electronic device for communicating in augmented reality and method thereof
US11061543B1 (en) 2020-05-11 2021-07-13 Apple Inc. Providing relevant data items based on context
US20210365104A1 (en) * 2020-05-22 2021-11-25 XRSpace CO., LTD. Virtual object operating method and virtual object operating system
US11257280B1 (en) 2020-05-28 2022-02-22 Facebook Technologies, Llc Element-based switching of ray casting rules
US11436804B2 (en) * 2020-06-10 2022-09-06 Arm Limited Augmented reality system
US11256336B2 (en) 2020-06-29 2022-02-22 Facebook Technologies, Llc Integration of artificial reality interaction modes
US11582409B2 (en) * 2020-06-29 2023-02-14 Snap Inc. Visual-inertial tracking using rolling shutter cameras
US20210407203A1 (en) * 2020-06-29 2021-12-30 Ilteris Canberk Augmented reality experiences using speech and text captions
US11490204B2 (en) 2020-07-20 2022-11-01 Apple Inc. Multi-device audio adjustment coordination
US11438683B2 (en) 2020-07-21 2022-09-06 Apple Inc. User identification using headphones
US11320896B2 (en) * 2020-08-03 2022-05-03 Facebook Technologies, Llc. Systems and methods for object tracking using fused data
US11249314B1 (en) * 2020-08-04 2022-02-15 Htc Corporation Method for switching input devices, head-mounted display and computer readable storage medium
EP3958095A1 (en) * 2020-08-21 2022-02-23 Deutsche Telekom AG A mobile computer-tethered virtual reality/augmented reality system using the mobile computer as a man machine interface
US11694413B2 (en) * 2020-08-25 2023-07-04 Spatial Systems Inc. Image editing and sharing in an augmented reality system
US11178376B1 (en) 2020-09-04 2021-11-16 Facebook Technologies, Llc Metering for display modes in artificial reality
US11228750B1 (en) * 2020-09-08 2022-01-18 Rovi Guides, Inc. Systems and methods for generating virtual reality scenes
WO2022055820A1 (en) * 2020-09-11 2022-03-17 Sterling Labs Llc Method of displaying selectable options
WO2022066472A1 (en) 2020-09-23 2022-03-31 Sterling Labs Llc Targeted drop of a computer-generated object
CN116917850A (en) * 2020-09-23 2023-10-20 苹果公司 Arranging virtual objects
KR20230050466A (en) * 2020-09-25 2023-04-14 애플 인크. Methods for navigating user interfaces
CN116490250A (en) * 2020-09-25 2023-07-25 苹果公司 Asymmetric presentation of environments
US10989783B1 (en) * 2020-10-26 2021-04-27 Listen Technologies Corporation False positive immunity in beacon systems, and related systems, methods, and devices
US11587055B1 (en) * 2020-12-02 2023-02-21 Wells Fargo Bank, N.A. Systems and methods for generating a user expression map configured to enable contactless human to device interactions
US11803831B1 (en) * 2020-12-02 2023-10-31 Wells Fargo Bank, N.A. Systems and methods for utilizing a user expression map configured to enable contactless human to device interactions
US11416079B2 (en) * 2020-12-03 2022-08-16 Motorola Mobility Llc Snap motion gesture detection and response
US20220180888A1 (en) * 2020-12-08 2022-06-09 International Business Machines Corporation Directional voice command identification
WO2022159214A2 (en) * 2020-12-17 2022-07-28 Trustees Of Tufts College Fusion-based sensing intelligence and reporting
EP4016252A1 (en) * 2020-12-17 2022-06-22 XRSpace CO., LTD. System and method related to motion tracking
EP4016253A1 (en) * 2020-12-17 2022-06-22 XRSpace CO., LTD. System and method related to data fusing
CN112631424A (en) * 2020-12-18 2021-04-09 上海影创信息科技有限公司 Gesture priority control method and system and VR glasses thereof
US11782577B2 (en) 2020-12-22 2023-10-10 Snap Inc. Media content player on an eyewear device
US11797162B2 (en) 2020-12-22 2023-10-24 Snap Inc. 3D painting on an eyewear device
US11687165B2 (en) 2020-12-30 2023-06-27 Meta Platforms Technologies, Llc Stabilizing gestures in artificial reality environments
CA3206042A1 (en) 2021-01-22 2022-07-28 Sarah Nancy STAAB Visual data management system and method
JP2024506630A (en) 2021-02-08 2024-02-14 サイトフル コンピューターズ リミテッド Extended reality for productivity
US11294475B1 (en) 2021-02-08 2022-04-05 Facebook Technologies, Llc Artificial reality multi-modal input switching model
CN117043709A (en) * 2021-02-08 2023-11-10 视域电脑有限公司 Augmented reality for productivity
EP4288950A1 (en) 2021-02-08 2023-12-13 Sightful Computers Ltd User interactions in extended reality
JP2024507749A (en) 2021-02-08 2024-02-21 サイトフル コンピューターズ リミテッド Content sharing in extended reality
WO2022170222A1 (en) * 2021-02-08 2022-08-11 Multinarity Ltd Content sharing in extended reality
US20220253146A1 (en) * 2021-02-09 2022-08-11 Finch Technologies Ltd. Combine Inputs from Different Devices to Control a Computing Device
CN117178245A (en) * 2021-04-01 2023-12-05 苹果公司 Gaze and head pose interactions
US11868531B1 (en) 2021-04-08 2024-01-09 Meta Platforms Technologies, Llc Wearable device providing for thumb-to-finger-based input gestures detected based on neuromuscular signals, and systems and methods of use thereof
CN113181619A (en) * 2021-04-09 2021-07-30 青岛小鸟看看科技有限公司 Exercise training method, device and system
CN113205074B (en) * 2021-05-29 2022-04-26 浙江大学 Gesture recognition method fusing multi-mode signals of myoelectricity and micro-inertia measurement unit
WO2023283145A1 (en) * 2021-07-05 2023-01-12 Dathomir Laboratories Llc Method and device for dynamically selecting an operation modality for an object
WO2023009580A2 (en) 2021-07-28 2023-02-02 Multinarity Ltd Using an extended reality appliance for productivity
US20230041814A1 (en) * 2021-08-06 2023-02-09 Lenovo (Singapore) Pte. Ltd. System and method for demonstrating objects at remote locations
CN113419636B (en) * 2021-08-23 2021-11-30 北京航空航天大学 Gesture recognition method and tool automatic matching method in virtual maintenance
WO2023026700A1 (en) * 2021-08-26 2023-03-02 株式会社Ｎｔｔドコモ Display control apparatus
CN113822174B (en) * 2021-09-02 2022-12-16 合肥的卢深视科技有限公司 Sight line estimation method, electronic device and storage medium
CN115809672A (en) * 2021-09-14 2023-03-17 北京小米移动软件有限公司 Translation method, translation device, AR glasses, storage medium and computer program product
US20230085330A1 (en) * 2021-09-15 2023-03-16 Neural Lab, Inc. Touchless image-based input interface
US20230081605A1 (en) * 2021-09-16 2023-03-16 Apple Inc. Digital assistant for moving and copying graphical elements
WO2023059087A1 (en) * 2021-10-08 2023-04-13 Samsung Electronics Co., Ltd. Augmented reality interaction method and apparatus
US20230128658A1 (en) * 2021-10-26 2023-04-27 Sony Interactive Entertainment LLC Personalized vr controls and communications
CN113743544A (en) * 2021-11-05 2021-12-03 中科智为科技(天津)有限公司 Cross-modal neural network construction method, pedestrian retrieval method and system
WO2023079537A1 (en) * 2021-11-08 2023-05-11 Tobii Dynavox Ab Methods of input and interaction with an augmentative and alternative communications (aac) device
WO2023091525A1 (en) * 2021-11-19 2023-05-25 Apple Inc. Intention-based user interface control for electronic devices
US20230162531A1 (en) * 2021-11-22 2023-05-25 Microsoft Technology Licensing, Llc Interpretation of resonant sensor data using machine learning
US20230168736A1 (en) * 2021-11-29 2023-06-01 Sony Interactive Entertainment LLC Input prediction for pre-loading of rendering data
CN114373064A (en) * 2021-12-14 2022-04-19 聚好看科技股份有限公司 VRAR content editing method, device, equipment and medium
CN114461063B (en) * 2022-01-18 2022-09-20 深圳时空科技集团有限公司 Man-machine interaction method based on vehicle-mounted screen
US20230334795A1 (en) 2022-01-25 2023-10-19 Multinarity Ltd Dual mode presentation of user interface elements
US11948263B1 (en) 2023-03-14 2024-04-02 Sightful Computers Ltd Recording the complete physical and extended reality environments of a user
EP4220356A1 (en) * 2022-01-27 2023-08-02 Volkswagen Ag Vehicle, apparatus, method and computer program for obtaining user input information
US20230267693A1 (en) * 2022-02-18 2023-08-24 Htc Corporation Method for determining physical characteristics of objects, host, and computer readable storage medium
TW202336689A (en) * 2022-03-11 2023-09-16 緯創資通股份有限公司 Virtual window configuration device, virtual window configuration method and virtual window configuration system
WO2023192047A1 (en) * 2022-03-29 2023-10-05 Apple Inc. Energy efficient context relevant processing for content
US11620000B1 (en) * 2022-03-31 2023-04-04 Microsoft Technology Licensing, Llc Controlled invocation of a precision input mode
WO2023196671A1 (en) * 2022-04-08 2023-10-12 Meta Platforms Technologies, Llc Techniques for neuromuscular-signal-based detection of in-air hand gestures for text production and modification, and systems, wearable devices, and methods for using these techniques
US20230377223A1 (en) * 2022-05-18 2023-11-23 Snap Inc. Hand-tracked text selection and modification
FR3136081A1 (en) * 2022-05-31 2023-12-01 Orange Augmented reader and method for augmented text reading
US20230396752A1 (en) * 2022-06-03 2023-12-07 Apple Inc. Electronic Device that Displays Virtual Objects
DE102022118795A1 (en) 2022-07-27 2024-02-01 Bayerische Motoren Werke Aktiengesellschaft Method for detecting the viewing direction of a user of data glasses for execution in a vehicle
CN115237255B (en) * 2022-07-29 2023-10-31 天津大学 Natural image co-pointing target positioning system and method based on eye movement and voice
CN115329578A (en) * 2022-08-19 2022-11-11 南京邮电大学 Three-dimensional modeling system and modeling method based on multi-mode fusion
WO2024059319A1 (en) * 2022-09-16 2024-03-21 Apple Inc. Gesture recognition with hand-object interaction
CN115951787B (en) * 2023-03-15 2023-07-25 北京亮亮视野科技有限公司 Interaction method of near-eye display device, storage medium and near-eye display device
CN116975318B (en) * 2023-08-03 2024-01-23 四川大学 Half-pairing image-text retrieval method based on cross-correlation mining
CN117348737A (en) * 2023-12-06 2024-01-05 之江实验室 Data processing system and method based on multi-channel interaction
CN117406867B (en) * 2023-12-15 2024-02-09 小芒电子商务有限责任公司 Webpage-based augmented reality interaction method and device

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP3046006A1 (en) * 2013-09-11 2016-07-20 Clarion Co., Ltd. Information processing device, gesture detection method, and gesture detection program
US20160357266A1 (en) * 2014-01-27 2016-12-08 Google Inc. Methods And Systems For Hands-Free Browsing In A Wearable Computing Device
WO2017115618A1 (en) * 2015-12-28 2017-07-06 Sony Corporation Information processing apparatus, information processing method, and program

Family Cites Families (55)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6222525B1 (en) 1992-03-05 2001-04-24 Brad A. Armstrong Image controllers with sheet connected sensors
US5670988A (en) 1995-09-05 1997-09-23 Interlink Electronics, Inc. Trigger operated electronic device
US7889244B2 (en) * 2005-12-27 2011-02-15 Panasonic Corporation Image processing apparatus
US9104312B2 (en) 2010-03-12 2015-08-11 Nuance Communications, Inc. Multimodal text input system, such as for use with touch screens on mobile phones
US9292973B2 (en) 2010-11-08 2016-03-22 Microsoft Technology Licensing, Llc Automatic variable virtual focus for augmented reality displays
US9304319B2 (en) 2010-11-18 2016-04-05 Microsoft Technology Licensing, Llc Automatic focus improvement for augmented reality displays
KR101890328B1 (en) 2010-12-24 2018-08-21 매직 립, 인코포레이티드 An ergonomic head mounted display device and optical system
US10156722B2 (en) 2010-12-24 2018-12-18 Magic Leap, Inc. Methods and systems for displaying stereoscopy with a freeform optical system with addressable focus for virtual and augmented reality
US9785242B2 (en) 2011-03-12 2017-10-10 Uday Parshionikar Multipurpose controllers and methods
JP6126076B2 (en) 2011-03-29 2017-05-10 クアルコム，インコーポレイテッド A system for rendering a shared digital interface for each user's perspective
EP2705435B8 (en) 2011-05-06 2017-08-23 Magic Leap, Inc. Massive simultaneous remote digital presence world
US10795448B2 (en) 2011-09-29 2020-10-06 Magic Leap, Inc. Tactile glove for human-computer interaction
US20130104085A1 (en) 2011-10-21 2013-04-25 Digital Artforms, Inc. Systems and methods for human-computer interaction using a two handed interface
KR101917630B1 (en) 2011-10-28 2018-11-13 매직 립, 인코포레이티드 System and method for augmented and virtual reality
CN108391033B (en) 2012-04-05 2020-10-30 奇跃公司 Wide field of view (FOV) imaging device with active foveal capability
US9671566B2 (en) 2012-06-11 2017-06-06 Magic Leap, Inc. Planar waveguide apparatus with diffraction element(s) and system employing same
KR102217788B1 (en) 2012-06-11 2021-02-18 매직 립, 인코포레이티드 Multiple depth plane three-dimensional display using a wave guide reflector array projector
US9740006B2 (en) 2012-09-11 2017-08-22 Magic Leap, Inc. Ergonomic head mounted display device and optical system
IL293789B2 (en) 2013-01-15 2023-08-01 Magic Leap Inc System for scanning electromagnetic imaging radiation
EP2967322A4 (en) 2013-03-11 2017-02-08 Magic Leap, Inc. System and method for augmented and virtual reality
CN108427504B (en) 2013-03-15 2021-06-11 奇跃公司 Display system and method
US10262462B2 (en) 2014-04-18 2019-04-16 Magic Leap, Inc. Systems and methods for augmented and virtual reality
US9874749B2 (en) 2013-11-27 2018-01-23 Magic Leap, Inc. Virtual and augmented reality systems and methods
EP2843507A1 (en) * 2013-08-26 2015-03-04 Thomson Licensing Display method through a head mounted device
IL302408A (en) 2013-10-16 2023-06-01 Magic Leap Inc Virtual or augmented reality headsets having adjustable interpupillary distance
CN107203045B (en) 2013-11-27 2023-10-20 奇跃公司 Virtual and augmented reality systems and methods
US9857591B2 (en) 2014-05-30 2018-01-02 Magic Leap, Inc. Methods and system for creating focal planes in virtual and augmented reality
CN106233189B (en) 2014-01-31 2020-06-26 奇跃公司 Multi-focus display system and method
EP3100099B1 (en) 2014-01-31 2020-07-01 Magic Leap, Inc. Multi-focal display system and method
US10203762B2 (en) 2014-03-11 2019-02-12 Magic Leap, Inc. Methods and systems for creating virtual and augmented reality
KR102230172B1 (en) 2014-05-09 2021-03-19 아이플루언스, 인크. Systems and methods for biomechanically-based eye signals for interacting with real and virtual objects
NZ727350A (en) 2014-05-30 2020-08-28 Magic Leap Inc Methods and systems for generating virtual content display with a virtual or augmented reality apparatus
KR20170041720A (en) * 2014-08-07 2017-04-17 포브, 아이엔씨. Algorithm for identifying three-dimensional point of gaze
WO2016174659A1 (en) * 2015-04-27 2016-11-03 Snapaid Ltd. Estimating and using relative head pose and camera field-of-view
US10761599B2 (en) * 2015-08-04 2020-09-01 Artilux, Inc. Eye gesture tracking
US10101803B2 (en) 2015-08-26 2018-10-16 Google Llc Dynamic switching and merging of head, gesture and touch input in virtual reality
AU2016324039B2 (en) 2015-09-16 2021-09-30 Magic Leap, Inc. Head pose mixing of audio files
CA3002369A1 (en) 2015-10-20 2017-04-27 Magic Leap, Inc. Selecting virtual objects in a three-dimensional space
US9818031B2 (en) 2016-01-06 2017-11-14 Orcam Technologies Ltd. Crowd-sourced vision-based information collection
JP2017129898A (en) * 2016-01-18 2017-07-27 ソニー株式会社 Information processing apparatus, information processing method, and program
US10373381B2 (en) 2016-03-30 2019-08-06 Microsoft Technology Licensing, Llc Virtual object manipulation within physical environment
CN109196447B (en) 2016-03-31 2022-06-17 奇跃公司 Interaction with 3D virtual objects using gestures and multi-DOF controllers
KR102520143B1 (en) * 2016-07-25 2023-04-11 매직 립, 인코포레이티드 Light field processor system
WO2018022602A1 (en) 2016-07-25 2018-02-01 Ctrl-Labs Corporation Methods and apparatus for predicting musculo-skeletal position information using wearable autonomous sensors
CN110337269B (en) 2016-07-25 2021-09-21 脸谱科技有限责任公司 Method and apparatus for inferring user intent based on neuromuscular signals
US11000211B2 (en) 2016-07-25 2021-05-11 Facebook Technologies, Llc Adaptive system for deriving control signals from measurements of neuromuscular activity
US11337652B2 (en) 2016-07-25 2022-05-24 Facebook Technologies, Llc System and method for measuring the movements of articulated rigid bodies
US10345925B2 (en) * 2016-08-03 2019-07-09 Google Llc Methods and systems for determining positional data for three-dimensional interactions inside virtual reality environments
IL264690B (en) 2016-08-11 2022-06-01 Magic Leap Inc Automatic placement of a virtual object in a three-dimensional space
US20180052512A1 (en) * 2016-08-16 2018-02-22 Thomas J. Overly Behavioral rehearsal system and supporting software
US20180082482A1 (en) * 2016-09-22 2018-03-22 Apple Inc. Display system having world and user sensors
JP7149289B2 (en) 2017-04-19 2022-10-06 マジック リープ， インコーポレイテッド Multimode execution and text editing for wearable systems
CN112040858A (en) 2017-10-19 2020-12-04 脸谱科技有限责任公司 System and method for identifying biological structures associated with neuromuscular source signals
US10706300B2 (en) * 2018-01-23 2020-07-07 Toyota Research Institute, Inc. Vehicle systems and methods for determining a target based on a virtual eye position and a pointing direction
CN112424727A (en) 2018-05-22 2021-02-26 奇跃公司 Cross-modal input fusion for wearable systems

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP3046006A1 (en) * 2013-09-11 2016-07-20 Clarion Co., Ltd. Information processing device, gesture detection method, and gesture detection program
US20160357266A1 (en) * 2014-01-27 2016-12-08 Google Inc. Methods And Systems For Hands-Free Browsing In A Wearable Computing Device
WO2017115618A1 (en) * 2015-12-28 2017-07-06 Sony Corporation Information processing apparatus, information processing method, and program
US20180356636A1 (en) * 2015-12-28 2018-12-13 Sony Corporation Information processing apparatus, information processing method, and program

Also Published As

Publication number Publication date
JP2023001328A (en) 2023-01-04
JP7341166B2 (en) 2023-09-08
CN112424727A (en) 2021-02-26
US20210056764A1 (en) 2021-02-25
US20190362557A1 (en) 2019-11-28
WO2019226691A1 (en) 2019-11-28
EP3797345A1 (en) 2021-03-31
US10861242B2 (en) 2020-12-08
JP2021524629A (en) 2021-09-13

Similar Documents

Publication Publication Date Title
EP3797345A4 (en) Transmodal input fusion for a wearable system
EP3837674A4 (en) A cross reality system
EP3844722A4 (en) Computer vision system
EP3487666A4 (en) Controls optimization for wearable systems
EP3813323A4 (en) Cross-blockchain interaction system
EP3820410A4 (en) Wearable joint augmentation system
EP3580695A4 (en) System for identifying a defined object
EP3311212A4 (en) A zoom lens system
EP3612881A4 (en) Fiber routing systems
EP3724701A4 (en) Optical system
EP3861533A4 (en) A cross reality system
EP3835007A4 (en) Robot system
EP3737994A4 (en) Advanced femtoprojector optical systems
EP3895023A4 (en) Macro-op fusion
EP3714315A4 (en) A conical optical combiner
EP3848164A4 (en) Robot system
EP3779625A4 (en) Visualization system
EP3526863A4 (en) A laser system having a dual pulse-length regime
EP4081448A4 (en) A monowheel system
EP3881452A4 (en) Secured fiber link system
EP3896506A4 (en) Fusion splicing device
EP3714254A4 (en) A laser system
EP3259878A4 (en) System for general purpose input/output functionality over a network
AU2018900829A0 (en) A System
EP3869942A4 (en) A control system

Legal Events

Date Code Title Description
STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE

PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

17P Request for examination filed

Effective date: 20201127

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

AX Request for extension of the european patent

Extension state: BA ME

DAV Request for validation of the european patent (deleted)
DAX Request for extension of the european patent (deleted)
A4 Supplementary search report drawn up and despatched

Effective date: 20220203

RIC1 Information provided on ipc code assigned before grant

Ipc: G06F 3/01 20060101AFI20220128BHEP

P01 Opt-out of the competence of the unified patent court (upc) registered

Effective date: 20230920