EP3612878A4 - Multimodal task execution and text editing for a wearable system - Google Patents

Multimodal task execution and text editing for a wearable system Download PDF

Info

Publication number
EP3612878A4
EP3612878A4 EP18788499.4A EP18788499A EP3612878A4 EP 3612878 A4 EP3612878 A4 EP 3612878A4 EP 18788499 A EP18788499 A EP 18788499A EP 3612878 A4 EP3612878 A4 EP 3612878A4
Authority
EP
European Patent Office
Prior art keywords
task execution
wearable system
text editing
multimodal task
multimodal
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
EP18788499.4A
Other languages
German (de)
French (fr)
Other versions
EP3612878B1 (en
EP3612878A1 (en
Inventor
James M. Powderly
Savannah NILES
Jennifer M. R. Devine
Adam C. Carlson
Jeffrey Sommers
Praveen BABU JD
Ajoy Savio FERNANDES
Anthony Robert Sheeder
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Magic Leap Inc
Original Assignee
Magic Leap Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Magic Leap Inc filed Critical Magic Leap Inc
Priority to EP23177854.9A priority Critical patent/EP4220258A1/en
Publication of EP3612878A1 publication Critical patent/EP3612878A1/en
Publication of EP3612878A4 publication Critical patent/EP3612878A4/en
Application granted granted Critical
Publication of EP3612878B1 publication Critical patent/EP3612878B1/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/017—Gesture based interaction, e.g. based on a set of recognized hand gestures
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/017—Head mounted
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/017—Head mounted
    • G02B27/0172—Head mounted characterised by optical features
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/0179—Display position adjusting means not related to the information to be displayed
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B6/00—Light guides; Structural details of arrangements comprising light guides and other optical elements, e.g. couplings
    • G02B6/0001—Light guides; Structural details of arrangements comprising light guides and other optical elements, e.g. couplings specially adapted for lighting devices or systems
    • G02B6/0011—Light guides; Structural details of arrangements comprising light guides and other optical elements, e.g. couplings specially adapted for lighting devices or systems the light guides being planar or of plate-like form
    • G02B6/0075—Arrangements of multiple light guides
    • G02B6/0076—Stacked arrangements of multiple light guides of the same or different cross-sectional area
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F1/00—Details not covered by groups G06F3/00 - G06F13/00 and G06F21/00
    • G06F1/16—Constructional details or arrangements
    • G06F1/1613—Constructional details or arrangements for portable computers
    • G06F1/163—Wearable computers, e.g. on a belt
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G06F3/013—Eye tracking input arrangements
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/16—Sound input; Sound output
    • G06F3/167—Audio in a user interface, e.g. using voice commands for navigating, audio feedback
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating 3D models or images for computer graphics
    • G06T19/003—Navigation within 3D models or images
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating 3D models or images for computer graphics
    • G06T19/006—Mixed reality
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/0101—Head-up displays characterised by optical features
    • G02B2027/0127—Head-up displays characterised by optical features comprising devices increasing the depth of field
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/0101—Head-up displays characterised by optical features
    • G02B2027/0138—Head-up displays characterised by optical features comprising image capture systems, e.g. camera
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/0101—Head-up displays characterised by optical features
    • G02B2027/014—Head-up displays characterised by optical features comprising information/image processing systems
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/0179—Display position adjusting means not related to the information to be displayed
    • G02B2027/0187—Display position adjusting means not related to the information to be displayed slaved to motion of at least a part of the body of the user, e.g. head, eye
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2203/00—Indexing scheme relating to G06F3/00 - G06F3/048
    • G06F2203/038—Indexing scheme relating to G06F3/038
    • G06F2203/0381—Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
EP18788499.4A 2017-04-19 2018-04-17 Multimodal task execution and text editing for a wearable system Active EP3612878B1 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
EP23177854.9A EP4220258A1 (en) 2017-04-19 2018-04-17 Multimodal task execution and text editing for a wearable system

Applications Claiming Priority (3)

Application Number Priority Date Filing Date Title
US201762487364P 2017-04-19 2017-04-19
US201762609647P 2017-12-22 2017-12-22
PCT/US2018/028002 WO2018195099A1 (en) 2017-04-19 2018-04-17 Multimodal task execution and text editing for a wearable system

Related Child Applications (1)

Application Number Title Priority Date Filing Date
EP23177854.9A Division EP4220258A1 (en) 2017-04-19 2018-04-17 Multimodal task execution and text editing for a wearable system

Publications (3)

Publication Number Publication Date
EP3612878A1 EP3612878A1 (en) 2020-02-26
EP3612878A4 true EP3612878A4 (en) 2020-09-23
EP3612878B1 EP3612878B1 (en) 2023-06-28

Family

ID=63854461

Family Applications (2)

Application Number Title Priority Date Filing Date
EP23177854.9A Pending EP4220258A1 (en) 2017-04-19 2018-04-17 Multimodal task execution and text editing for a wearable system
EP18788499.4A Active EP3612878B1 (en) 2017-04-19 2018-04-17 Multimodal task execution and text editing for a wearable system

Family Applications Before (1)

Application Number Title Priority Date Filing Date
EP23177854.9A Pending EP4220258A1 (en) 2017-04-19 2018-04-17 Multimodal task execution and text editing for a wearable system

Country Status (9)

Country Link
US (3) US10768693B2 (en)
EP (2) EP4220258A1 (en)
JP (3) JP7149289B2 (en)
KR (2) KR20240046613A (en)
CN (2) CN113608617A (en)
AU (1) AU2018256365A1 (en)
CA (1) CA3059234A1 (en)
IL (1) IL270002B2 (en)
WO (1) WO2018195099A1 (en)

Families Citing this family (149)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10768704B2 (en) 2015-03-17 2020-09-08 Whirlwind VR, Inc. System and method for modulating a peripheral device based on an unscripted feed using computer vision
WO2017068926A1 (en) * 2015-10-21 2017-04-27 ソニー株式会社 Information processing device, control method therefor, and computer program
US11416714B2 (en) 2017-03-24 2022-08-16 Revealit Corporation Method, system, and apparatus for identifying and revealing selected objects from video
EP4220258A1 (en) 2017-04-19 2023-08-02 Magic Leap, Inc. Multimodal task execution and text editing for a wearable system
EP3542252B1 (en) * 2017-08-10 2023-08-02 Google LLC Context-sensitive hand interaction
US10373332B2 (en) * 2017-12-08 2019-08-06 Nvidia Corporation Systems and methods for dynamic facial analysis using a recurrent neural network
CN107977586B (en) * 2017-12-22 2021-04-13 联想(北京)有限公司 Display content processing method, first electronic device and second electronic device
BR112020010376A2 (en) * 2017-12-22 2020-11-24 Telefonaktiebolaget Lm Ericsson (Publ) method for initiating voice control by looking at detection, device for initiating voice control by looking at detection, and, computer-readable media
JP7272278B2 (en) * 2018-01-04 2023-05-12 ソニーグループ株式会社 Information processing device, information processing method and program
KR20200110771A (en) * 2018-01-18 2020-09-25 삼성전자주식회사 Augmented Reality Content Adjustment Method and Device
EP3750032A4 (en) * 2018-02-06 2021-11-17 Wal-Mart Apollo, LLC Customized augmented reality item filtering system
WO2019181153A1 (en) * 2018-03-20 2019-09-26 ソニー株式会社 Information processing device, information processing method, and recording medium
US10540015B2 (en) * 2018-03-26 2020-01-21 Chian Chiu Li Presenting location related information and implementing a task based on gaze and voice detection
US11157075B2 (en) * 2018-05-01 2021-10-26 Dell Products, L.P. Gaze-activated voice services for interactive workspaces
US11688417B2 (en) 2018-05-04 2023-06-27 Google Llc Hot-word free adaptation of automated assistant function(s)
US10916065B2 (en) * 2018-05-04 2021-02-09 Facebook Technologies, Llc Prevention of user interface occlusion in a virtual reality environment
CN112236738A (en) 2018-05-04 2021-01-15 谷歌有限责任公司 Invoking automated assistant functionality based on detected gestures and gaze
US20190339837A1 (en) * 2018-05-04 2019-11-07 Oculus Vr, Llc Copy and Paste in a Virtual Reality Environment
JP7471279B2 (en) * 2018-05-04 2024-04-19 グーグル エルエルシー Adapting an automated assistant based on detected mouth movements and/or gaze
US10504290B2 (en) * 2018-05-04 2019-12-10 Facebook Technologies, Llc User interface security in a virtual reality environment
US11169668B2 (en) * 2018-05-16 2021-11-09 Google Llc Selecting an input mode for a virtual assistant
EP3797345A4 (en) 2018-05-22 2022-03-09 Magic Leap, Inc. Transmodal input fusion for a wearable system
US11733824B2 (en) * 2018-06-22 2023-08-22 Apple Inc. User interaction interpreter
US10937243B2 (en) * 2018-08-29 2021-03-02 Dell Products, L.P. Real-world object interface for virtual, augmented, and mixed reality (xR) applications
US11348316B2 (en) * 2018-09-11 2022-05-31 Apple Inc. Location-based virtual element modality in three-dimensional content
US11232635B2 (en) 2018-10-05 2022-01-25 Magic Leap, Inc. Rendering location specific virtual content in any location
US11288733B2 (en) * 2018-11-14 2022-03-29 Mastercard International Incorporated Interactive 3D image projection systems and methods
JP2022510843A (en) * 2018-11-30 2022-01-28 マジック リープ， インコーポレイテッド Multimode hand location and orientation for avatar movement
WO2020116001A1 (en) * 2018-12-03 2020-06-11 ソニー株式会社 Information processing device and information processing method
US11036838B2 (en) 2018-12-05 2021-06-15 Bank Of America Corporation Processing authentication requests to secured information systems using machine-learned user-account behavior profiles
US11048793B2 (en) 2018-12-05 2021-06-29 Bank Of America Corporation Dynamically generating activity prompts to build and refine machine learning authentication models
US11159510B2 (en) 2018-12-05 2021-10-26 Bank Of America Corporation Utilizing federated user identifiers to enable secure information sharing
US11120109B2 (en) 2018-12-05 2021-09-14 Bank Of America Corporation Processing authentication requests to secured information systems based on machine-learned event profiles
US11113370B2 (en) 2018-12-05 2021-09-07 Bank Of America Corporation Processing authentication requests to secured information systems using machine-learned user-account behavior profiles
US11176230B2 (en) 2018-12-05 2021-11-16 Bank Of America Corporation Processing authentication requests to secured information systems based on user behavior profiles
US10930275B2 (en) * 2018-12-18 2021-02-23 Microsoft Technology Licensing, Llc Natural language input disambiguation for spatialized regions
KR20200076325A (en) * 2018-12-19 2020-06-29 삼성전자주식회사 Wearable device and method for using external object as controller
US10789952B2 (en) * 2018-12-20 2020-09-29 Microsoft Technology Licensing, Llc Voice command execution from auxiliary input
US11024074B2 (en) * 2018-12-27 2021-06-01 Facebook Technologies, Llc Virtual spaces, mixed reality spaces, and combined mixed reality spaces for improved interaction and collaboration
KR20200092464A (en) * 2019-01-07 2020-08-04 삼성전자주식회사 Electronic device and method for providing assistant service using the electronic device
JP7468506B2 (en) * 2019-03-13 2024-04-16 ソニーグループ株式会社 Information processing device, information processing method, and recording medium
CN113597594A (en) * 2019-03-28 2021-11-02 索尼集团公司 Information processing apparatus, information processing method, and program
WO2020218634A1 (en) * 2019-04-23 2020-10-29 엘지전자 주식회사 Answering device determination method and apparatus
US11321436B2 (en) 2019-05-01 2022-05-03 Samsung Electronics Co., Ltd. Human ID for mobile authentication
EP4300430A3 (en) 2019-05-06 2024-03-06 Apple Inc. Device, method, and graphical user interface for composing cgr files
CN113508361A (en) 2019-05-06 2021-10-15 苹果公司 Apparatus, method and computer-readable medium for presenting computer-generated reality files
US10969863B2 (en) * 2019-05-08 2021-04-06 International Business Machines Corporation Configurable sensor array for a multi-target environment
WO2020236171A1 (en) 2019-05-22 2020-11-26 Google Llc Methods, systems, and media for object grouping and manipulation in immersive environments
CN110334330A (en) * 2019-05-27 2019-10-15 努比亚技术有限公司 A kind of information edit method, wearable device and computer readable storage medium
US10937192B2 (en) * 2019-05-29 2021-03-02 Dell Products, L.P. Resolving incorrect distributed simultaneous localization and mapping (SLAM) data in edge cloud architectures
JP7336266B2 (en) * 2019-06-04 2023-08-31 キヤノン株式会社 ELECTRONIC DEVICE, CONTROL METHOD FOR ELECTRONIC DEVICE, PROGRAM, STORAGE MEDIUM
JP2021005157A (en) * 2019-06-25 2021-01-14 株式会社ソニー・インタラクティブエンタテインメント Image processing apparatus and image processing method
US11422638B2 (en) 2019-07-08 2022-08-23 Apple Inc. Input devices that use self-mixing interferometry to determine movement within an enclosure
US11017231B2 (en) * 2019-07-10 2021-05-25 Microsoft Technology Licensing, Llc Semantically tagged virtual and physical objects
JP7451110B2 (en) * 2019-08-27 2024-03-18 ソニーグループ株式会社 Ranging systems and electronic equipment
CN114341747A (en) * 2019-09-03 2022-04-12 光场实验室公司 Light field display of mobile device
US11848000B2 (en) * 2019-09-06 2023-12-19 Microsoft Technology Licensing, Llc Transcription revision interface for speech recognition system
US10943388B1 (en) 2019-09-06 2021-03-09 Zspace, Inc. Intelligent stylus beam and assisted probabilistic input to element mapping in 2D and 3D graphical user interfaces
US11409365B2 (en) * 2019-09-06 2022-08-09 Apple Inc. Self-mixing interferometry-based gesture input system including a wearable or handheld device
US11086406B1 (en) 2019-09-20 2021-08-10 Facebook Technologies, Llc Three-state gesture virtual controls
US11189099B2 (en) 2019-09-20 2021-11-30 Facebook Technologies, Llc Global and local mode virtual object interactions
US10991163B2 (en) 2019-09-20 2021-04-27 Facebook Technologies, Llc Projection casting in virtual environments
US11170576B2 (en) 2019-09-20 2021-11-09 Facebook Technologies, Llc Progressive display of virtual objects
US11176745B2 (en) 2019-09-20 2021-11-16 Facebook Technologies, Llc Projection casting in virtual environments
US11580869B2 (en) * 2019-09-23 2023-02-14 Revealit Corporation Computer-implemented interfaces for identifying and revealing selected objects from video
US11419546B2 (en) 2019-09-24 2022-08-23 Apple Inc. Wearable self-mixing interferometry device used to sense physiological conditions
KR20210136116A (en) 2019-09-27 2021-11-16 애플 인크. virtual objects control
EP4046401A4 (en) 2019-10-15 2023-11-01 Magic Leap, Inc. Cross reality system with wireless fingerprints
US11567788B1 (en) 2019-10-18 2023-01-31 Meta Platforms, Inc. Generating proactive reminders for assistant systems
US11861674B1 (en) 2019-10-18 2024-01-02 Meta Platforms Technologies, Llc Method, one or more computer-readable non-transitory storage media, and a system for generating comprehensive information for products of interest by assistant systems
US11086476B2 (en) * 2019-10-23 2021-08-10 Facebook Technologies, Llc 3D interactions with web content
US11289086B2 (en) * 2019-11-01 2022-03-29 Microsoft Technology Licensing, Llc Selective response rendering for virtual assistants
US11493989B2 (en) * 2019-11-08 2022-11-08 Magic Leap, Inc. Modes of user interaction
US11175730B2 (en) 2019-12-06 2021-11-16 Facebook Technologies, Llc Posture-based virtual space configurations
CN114762008A (en) 2019-12-09 2022-07-15 奇跃公司 Simplified virtual content programmed cross reality system
US11816757B1 (en) * 2019-12-11 2023-11-14 Meta Platforms Technologies, Llc Device-side capture of data representative of an artificial reality environment
US11042222B1 (en) 2019-12-16 2021-06-22 Microsoft Technology Licensing, Llc Sub-display designation and sharing
US11487423B2 (en) 2019-12-16 2022-11-01 Microsoft Technology Licensing, Llc Sub-display input areas and hidden inputs
US11093046B2 (en) 2019-12-16 2021-08-17 Microsoft Technology Licensing, Llc Sub-display designation for remote content source device
US20210192681A1 (en) * 2019-12-18 2021-06-24 Ati Technologies Ulc Frame reprojection for virtual reality and augmented reality
US11475639B2 (en) 2020-01-03 2022-10-18 Meta Platforms Technologies, Llc Self presence in artificial reality
US11687778B2 (en) 2020-01-06 2023-06-27 The Research Foundation For The State University Of New York Fakecatcher: detection of synthetic portrait videos using biological signals
EP3851939A1 (en) * 2020-01-14 2021-07-21 Apple Inc. Positioning a user-controlled spatial selector based on extremity tracking information and eye tracking information
CN115380236A (en) 2020-01-24 2022-11-22 奇跃公司 Content movement and interaction using a single controller
WO2021163300A1 (en) 2020-02-13 2021-08-19 Magic Leap, Inc. Cross reality system with map processing using multi-resolution frame descriptors
CN115380264A (en) * 2020-02-13 2022-11-22 奇跃公司 Cross reality system for large-scale environments
EP4103910A4 (en) 2020-02-13 2024-03-06 Magic Leap Inc Cross reality system with accurate shared maps
CN115398484A (en) 2020-02-13 2022-11-25 奇跃公司 Cross reality system with geolocation information priority for location
JP2021157277A (en) * 2020-03-25 2021-10-07 ソニーグループ株式会社 Information processing apparatus, information processing method, and program
US11789688B1 (en) 2020-03-26 2023-10-17 Apple Inc. Content presentation based on environmental data
CN111526083B (en) * 2020-04-15 2022-04-15 上海幂方电子科技有限公司 Method, device, system and storage medium for instant messaging through head action
CN111459288B (en) * 2020-04-23 2021-08-03 捷开通讯(深圳)有限公司 Method and device for realizing voice input by using head control
JP7402322B2 (en) * 2020-05-15 2023-12-20 株式会社Ｎｔｔドコモ information processing system
CN111782031A (en) * 2020-05-26 2020-10-16 北京理工大学 Text input system and method based on head movement and finger micro-gestures
CN111831112A (en) * 2020-05-26 2020-10-27 北京理工大学 Text input system and method based on eye movement and finger micro-gesture
US11257280B1 (en) 2020-05-28 2022-02-22 Facebook Technologies, Llc Element-based switching of ray casting rules
US11340707B2 (en) * 2020-05-29 2022-05-24 Microsoft Technology Licensing, Llc Hand gesture-based emojis
WO2021245886A1 (en) * 2020-06-04 2021-12-09 三菱電機株式会社 Traffic management system for building
US20210407203A1 (en) * 2020-06-29 2021-12-30 Ilteris Canberk Augmented reality experiences using speech and text captions
GB202009874D0 (en) * 2020-06-29 2020-08-12 Microsoft Technology Licensing Llc Visual interface for a computer system
US11256336B2 (en) 2020-06-29 2022-02-22 Facebook Technologies, Llc Integration of artificial reality interaction modes
CN111782053B (en) * 2020-08-10 2023-04-28 Oppo广东移动通信有限公司 Model editing method, device, equipment and storage medium
WO2022046367A1 (en) * 2020-08-25 2022-03-03 Sterling Labs Llc Displaying an environment from a selected point-of-view
US11176755B1 (en) 2020-08-31 2021-11-16 Facebook Technologies, Llc Artificial reality augments and surfaces
US11227445B1 (en) 2020-08-31 2022-01-18 Facebook Technologies, Llc Artificial reality augments and surfaces
US11178376B1 (en) 2020-09-04 2021-11-16 Facebook Technologies, Llc Metering for display modes in artificial reality
US11606364B2 (en) 2020-09-15 2023-03-14 Meta Platforms Technologies, Llc Artificial reality collaborative working environments
CN117555417A (en) * 2020-09-25 2024-02-13 苹果公司 Method for adjusting and/or controlling immersion associated with a user interface
CN112379799B (en) * 2020-10-16 2022-04-01 杭州易现先进科技有限公司 Control method and device for input box in three-dimensional rendering and electronic device
US11113893B1 (en) 2020-11-17 2021-09-07 Facebook Technologies, Llc Artificial reality environment with glints displayed by an extra reality device
KR102258531B1 (en) * 2020-11-23 2021-06-01 (주)펜타유니버스 Analysis unit for integrated recognition for multiple input recognition system
US11854230B2 (en) 2020-12-01 2023-12-26 Meta Platforms Technologies, Llc Physical keyboard tracking
US11928263B2 (en) * 2020-12-07 2024-03-12 Samsung Electronics Co., Ltd. Electronic device for processing user input and method thereof
KR20220080399A (en) * 2020-12-07 2022-06-14 삼성전자주식회사 Electronic device and system for processing user input and method thereof
US20220180888A1 (en) * 2020-12-08 2022-06-09 International Business Machines Corporation Directional voice command identification
US11409405B1 (en) 2020-12-22 2022-08-09 Facebook Technologies, Llc Augment orchestration in an artificial reality environment
US11461973B2 (en) 2020-12-22 2022-10-04 Meta Platforms Technologies, Llc Virtual reality locomotion via hand gesture
US11610363B2 (en) 2020-12-31 2023-03-21 Oberon Technologies, Inc. Systems and methods for virtual reality environments
KR20220105893A (en) * 2021-01-21 2022-07-28 삼성전자주식회사 Wearable electronic device receiving information from external electronic device and operating method thereof
US11294475B1 (en) 2021-02-08 2022-04-05 Facebook Technologies, Llc Artificial reality multi-modal input switching model
JP2024507749A (en) 2021-02-08 2024-02-21 サイトフル コンピューターズ リミテッド Content sharing in extended reality
WO2022212916A1 (en) * 2021-04-01 2022-10-06 Giant.Ai, Inc. Hybrid computing architectures with specialized processors to encode/decode latent representations for controlling dynamic mechanical systems
WO2022215347A1 (en) * 2021-04-08 2022-10-13 株式会社Ｎｔｔドコモ Content display control system
WO2022245629A1 (en) * 2021-05-19 2022-11-24 Snap Inc. Contextual visual and voice search from electronic eyewear device
CN113286163B (en) * 2021-05-21 2022-07-08 成都威爱新经济技术研究院有限公司 Timestamp error calibration method and system for virtual shooting live broadcast
US11295503B1 (en) 2021-06-28 2022-04-05 Facebook Technologies, Llc Interactive avatars in artificial reality
US11762952B2 (en) 2021-06-28 2023-09-19 Meta Platforms Technologies, Llc Artificial reality application lifecycle
WO2023009580A2 (en) 2021-07-28 2023-02-02 Multinarity Ltd Using an extended reality appliance for productivity
CN113627312A (en) * 2021-08-04 2021-11-09 东南大学 System for assisting paralyzed speaker to output language through eye movement tracking
US20230055819A1 (en) * 2021-08-18 2023-02-23 Target Brands, Inc. Virtual reality system for retail store design
US20230267933A1 (en) * 2021-09-27 2023-08-24 International Business Machines Corporation Selective inclusion of speech content in documents
US11810242B2 (en) * 2021-10-12 2023-11-07 Htc Corporation Method for stabilizing raycast and electronic device using the same
US11748944B2 (en) 2021-10-27 2023-09-05 Meta Platforms Technologies, Llc Virtual object structures and interrelationships
US11798247B2 (en) 2021-10-27 2023-10-24 Meta Platforms Technologies, Llc Virtual object structures and interrelationships
US11726587B2 (en) * 2021-11-03 2023-08-15 Htc Corporation Virtual image display system and pointing direction control method of control device thereof
US11948263B1 (en) 2023-03-14 2024-04-02 Sightful Computers Ltd Recording the complete physical and extended reality environments of a user
US20230266817A1 (en) * 2022-02-23 2023-08-24 International Business Machines Corporation Gaze based text manipulation
US11579747B1 (en) * 2022-03-14 2023-02-14 Snap Inc. 3D user interface depth forgiveness
US20230315193A1 (en) * 2022-03-31 2023-10-05 Lenovo (Singapore) Pte. Ltd. Direction of user input to virtual objects based on command metadata
US20230326144A1 (en) * 2022-04-08 2023-10-12 Meta Platforms Technologies, Llc Triggering Field Transitions for Artificial Reality Objects
US20230367611A1 (en) * 2022-05-10 2023-11-16 Meta Platforms Technologies, Llc World-Controlled and Application-Controlled Augments in an Artificial-Reality Environment
US20240036578A1 (en) * 2022-07-26 2024-02-01 Zebra Technologies Corporation Generation of Navigational Paths Using Recursive Partitioning of Unoccupied Space
CN115237255B (en) * 2022-07-29 2023-10-31 天津大学 Natural image co-pointing target positioning system and method based on eye movement and voice
CN115344121A (en) * 2022-08-10 2022-11-15 北京字跳网络技术有限公司 Method, device, equipment and storage medium for processing gesture event
US11947862B1 (en) 2022-12-30 2024-04-02 Meta Platforms Technologies, Llc Streaming native application content to artificial reality devices
US11880936B1 (en) * 2023-01-26 2024-01-23 Intuit Inc. Generating and displaying text in a virtual reality environment
CN116300092A (en) * 2023-03-09 2023-06-23 北京百度网讯科技有限公司 Control method, device and equipment of intelligent glasses and storage medium
CN116301388B (en) * 2023-05-11 2023-08-01 环球数科集团有限公司 Man-machine interaction scene system for intelligent multi-mode combined application
CN117348737A (en) * 2023-12-06 2024-01-05 之江实验室 Data processing system and method based on multi-channel interaction

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP3118722A1 (en) * 2015-07-14 2017-01-18 Nokia Technologies Oy Mediated reality
EP3665550A1 (en) * 2017-09-29 2020-06-17 Apple Inc. Gaze-based user interactions

Family Cites Families (75)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH04372012A (en) * 1991-06-20 1992-12-25 Fuji Xerox Co Ltd Input device
US6222525B1 (en) 1992-03-05 2001-04-24 Brad A. Armstrong Image controllers with sheet connected sensors
JP3461980B2 (en) 1995-08-25 2003-10-27 株式会社東芝 High-speed drawing method and apparatus
US5670988A (en) 1995-09-05 1997-09-23 Interlink Electronics, Inc. Trigger operated electronic device
JP3822357B2 (en) * 1998-02-09 2006-09-20 株式会社東芝 Interface device and method for multimodal input / output device
US7257575B1 (en) * 2002-10-24 2007-08-14 At&T Corp. Systems and methods for generating markup-language based expressions from multi-modal and unimodal inputs
USD514570S1 (en) 2004-06-24 2006-02-07 Microsoft Corporation Region of a fingerprint scanning device with an illuminated ring
US7561717B2 (en) 2004-07-09 2009-07-14 United Parcel Service Of America, Inc. System and method for displaying item information
US11428937B2 (en) 2005-10-07 2022-08-30 Percept Technologies Enhanced optical and perceptual digital eyewear
US8696113B2 (en) 2005-10-07 2014-04-15 Percept Technologies Inc. Enhanced optical and perceptual digital eyewear
US20070081123A1 (en) 2005-10-07 2007-04-12 Lewis Scott W Digital eyewear
JP5145569B2 (en) 2007-12-19 2013-02-20 株式会社国際電気通信基礎技術研究所 Object identification method and apparatus
JP5642809B2 (en) * 2010-03-12 2014-12-17 ニュアンス コミュニケーションズ， インコーポレイテッド Multi-modal text input system for use with mobile phone touchscreen etc.
US8645136B2 (en) * 2010-07-20 2014-02-04 Intellisist, Inc. System and method for efficiently reducing transcription error using hybrid voice transcription
US9292973B2 (en) * 2010-11-08 2016-03-22 Microsoft Technology Licensing, Llc Automatic variable virtual focus for augmented reality displays
US9304319B2 (en) 2010-11-18 2016-04-05 Microsoft Technology Licensing, Llc Automatic focus improvement for augmented reality displays
JP5472056B2 (en) 2010-11-19 2014-04-16 コニカミノルタ株式会社 Display system, display processing apparatus, display method, and display program
JP6185844B2 (en) 2010-12-24 2017-08-30 マジック リープ， インコーポレイテッド Ergonomic head-mounted display device and optical system
US10156722B2 (en) 2010-12-24 2018-12-18 Magic Leap, Inc. Methods and systems for displaying stereoscopy with a freeform optical system with addressable focus for virtual and augmented reality
CN103460256B (en) * 2011-03-29 2016-09-14 高通股份有限公司 In Augmented Reality system, virtual image is anchored to real world surface
CN103635891B (en) 2011-05-06 2017-10-27 奇跃公司 The world is presented in a large amount of digital remotes simultaneously
USD658094S1 (en) 2011-06-18 2012-04-24 Dunn Scott E Rounded square display device for a cord
US9417754B2 (en) 2011-08-05 2016-08-16 P4tents1, LLC User interface system, method, and computer program product
JP5762892B2 (en) 2011-09-06 2015-08-12 ビッグローブ株式会社 Information display system, information display method, and information display program
EP2760363A4 (en) 2011-09-29 2015-06-24 Magic Leap Inc Tactile glove for human-computer interaction
US20130104085A1 (en) 2011-10-21 2013-04-25 Digital Artforms, Inc. Systems and methods for human-computer interaction using a two handed interface
BR112014010230A8 (en) 2011-10-28 2017-06-20 Magic Leap Inc system and method for augmented and virtual reality
CA3024054C (en) 2011-11-23 2020-12-29 Magic Leap, Inc. Three dimensional virtual and augmented reality display system
WO2013152205A1 (en) 2012-04-05 2013-10-10 Augmented Vision Inc. Wide-field of view (fov) imaging devices with active foveation capability
CN104737061B (en) 2012-06-11 2018-01-16 奇跃公司 Use more depth plane three dimensional displays of the waveguided reflector arrays projector
US9671566B2 (en) * 2012-06-11 2017-06-06 Magic Leap, Inc. Planar waveguide apparatus with diffraction element(s) and system employing same
US8754829B2 (en) 2012-08-04 2014-06-17 Paul Lapstun Scanning light field camera and display
EP2895910A4 (en) 2012-09-11 2016-04-20 Magic Leap Inc Ergonomic head mounted display device and optical system
JP2014085954A (en) 2012-10-25 2014-05-12 Kyocera Corp Portable terminal device, program and input operation accepting method
US10151875B2 (en) 2013-01-15 2018-12-11 Magic Leap, Inc. Ultra-high resolution scanning fiber display
KR102387314B1 (en) 2013-03-11 2022-04-14 매직 립, 인코포레이티드 System and method for augmented and virtual reality
NZ751602A (en) 2013-03-15 2020-01-31 Magic Leap Inc Display system and method
JP5900393B2 (en) 2013-03-21 2016-04-06 ソニー株式会社 Information processing apparatus, operation control method, and program
US9367960B2 (en) * 2013-05-22 2016-06-14 Microsoft Technology Licensing, Llc Body-locked placement of augmented reality objects
US10262462B2 (en) 2014-04-18 2019-04-16 Magic Leap, Inc. Systems and methods for augmented and virtual reality
US9874749B2 (en) 2013-11-27 2018-01-23 Magic Leap, Inc. Virtual and augmented reality systems and methods
US9129430B2 (en) 2013-06-25 2015-09-08 Microsoft Technology Licensing, Llc Indicating out-of-view augmented reality images
KR20150037254A (en) 2013-09-30 2015-04-08 엘지전자 주식회사 Wearable display device and method of controlling layer
EP3058418B1 (en) 2013-10-16 2023-10-04 Magic Leap, Inc. Virtual or augmented reality headsets having adjustable interpupillary distance
JPWO2015059976A1 (en) 2013-10-24 2017-03-09 ソニー株式会社 Information processing apparatus, information processing method, and program
US9857591B2 (en) 2014-05-30 2018-01-02 Magic Leap, Inc. Methods and system for creating focal planes in virtual and augmented reality
KR102268462B1 (en) 2013-11-27 2021-06-22 매직 립, 인코포레이티드 Virtual and augmented reality systems and methods
JP2015114757A (en) 2013-12-10 2015-06-22 ソニー株式会社 Information processing apparatus, information processing method, and program
EP4071537A1 (en) 2014-01-31 2022-10-12 Magic Leap, Inc. Multi-focal display system
EP3100098B8 (en) 2014-01-31 2022-10-05 Magic Leap, Inc. Multi-focal display system and method
JP2015145934A (en) 2014-02-03 2015-08-13 セイコーエプソン株式会社 projector
US9377626B2 (en) * 2014-02-18 2016-06-28 Merge Labs, Inc. Remote control augmented motion data capture
US10203762B2 (en) * 2014-03-11 2019-02-12 Magic Leap, Inc. Methods and systems for creating virtual and augmented reality
KR20150110257A (en) * 2014-03-21 2015-10-02 삼성전자주식회사 Method and wearable device for providing a virtual input interface
US20150324568A1 (en) * 2014-05-09 2015-11-12 Eyefluence, Inc. Systems and methods for using eye signals with secure mobile communications
USD759657S1 (en) 2014-05-19 2016-06-21 Microsoft Corporation Connector with illumination region
KR102193052B1 (en) 2014-05-30 2020-12-18 매직 립, 인코포레이티드 Methods and systems for generating virtual content display with a virtual or augmented reality apparatus
US9971492B2 (en) * 2014-06-04 2018-05-15 Quantum Interface, Llc Dynamic environment for object and attribute display and interaction
USD752529S1 (en) 2014-06-09 2016-03-29 Comcast Cable Communications, Llc Electronic housing with illuminated region
USD758367S1 (en) 2015-05-14 2016-06-07 Magic Leap, Inc. Virtual reality headset
JP6780642B2 (en) 2015-06-15 2020-11-04 ソニー株式会社 Information processing equipment, information processing methods and programs
US10101803B2 (en) 2015-08-26 2018-10-16 Google Llc Dynamic switching and merging of head, gesture and touch input in virtual reality
EP4254145A3 (en) * 2015-09-16 2023-11-01 Magic Leap, Inc. Head pose mixing of audio files
NZ741866A (en) 2015-10-20 2019-07-26 Magic Leap Inc Selecting virtual objects in a three-dimensional space
US10921979B2 (en) * 2015-12-07 2021-02-16 Huawei Technologies Co., Ltd. Display and processing methods and related apparatus
CN106997235B (en) * 2016-01-25 2018-07-13 亮风台（上海）信息科技有限公司 For realizing method, the equipment of augmented reality interaction and displaying
US10229541B2 (en) 2016-01-28 2019-03-12 Sony Interactive Entertainment America Llc Methods and systems for navigation within virtual reality space using head mounted display
USD805734S1 (en) 2016-03-04 2017-12-26 Nike, Inc. Shirt
USD794288S1 (en) 2016-03-11 2017-08-15 Nike, Inc. Shoe with illuminable sole light sequence
EP3436863A4 (en) 2016-03-31 2019-11-27 Magic Leap, Inc. Interactions with 3d virtual objects using poses and multiple-dof controllers
US10068134B2 (en) * 2016-05-03 2018-09-04 Microsoft Technology Licensing, Llc Identification of objects in a scene using gaze tracking techniques
CN115202482A (en) 2016-08-11 2022-10-18 奇跃公司 System and method for automatic placement of virtual objects in three-dimensional space
US10536691B2 (en) * 2016-10-04 2020-01-14 Facebook, Inc. Controls and interfaces for user interactions in virtual spaces
CN108664201B (en) * 2017-03-29 2021-12-28 北京搜狗科技发展有限公司 Text editing method and device and electronic equipment
EP4220258A1 (en) 2017-04-19 2023-08-02 Magic Leap, Inc. Multimodal task execution and text editing for a wearable system

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP3118722A1 (en) * 2015-07-14 2017-01-18 Nokia Technologies Oy Mediated reality
EP3665550A1 (en) * 2017-09-29 2020-06-17 Apple Inc. Gaze-based user interactions

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
DANA SLAMBEKOVA ET AL: "Gaze and gesture based object manipulation in virtual worlds", VIRTUAL REALITY SOFTWARE AND TECHNOLOGY, ACM, 2 PENN PLAZA, SUITE 701 NEW YORK NY 10121-0701 USA, 10 December 2012 (2012-12-10), pages 203 - 204, XP058010169, ISBN: 978-1-4503-1469-5, DOI: 10.1145/2407336.2407380 *

Also Published As

Publication number Publication date
KR20240046613A (en) 2024-04-09
US20200363865A1 (en) 2020-11-19
US20220171453A1 (en) 2022-06-02
EP3612878B1 (en) 2023-06-28
IL270002B2 (en) 2023-11-01
AU2018256365A1 (en) 2019-10-31
IL270002A (en) 2019-12-31
EP3612878A1 (en) 2020-02-26
IL270002B1 (en) 2023-07-01
US20180307303A1 (en) 2018-10-25
JP7336005B2 (en) 2023-08-30
CA3059234A1 (en) 2018-10-25
KR20190141696A (en) 2019-12-24
KR102652922B1 (en) 2024-03-29
JP2023144096A (en) 2023-10-06
WO2018195099A1 (en) 2018-10-25
JP2022121592A (en) 2022-08-19
EP4220258A1 (en) 2023-08-02
CN110785688A (en) 2020-02-11
CN113608617A (en) 2021-11-05
US11960636B2 (en) 2024-04-16
CN110785688B (en) 2021-08-27
JP2020519986A (en) 2020-07-02
US10768693B2 (en) 2020-09-08
US11237623B2 (en) 2022-02-01
JP7149289B2 (en) 2022-10-06

Similar Documents

Publication Publication Date Title
EP3612878A4 (en) Multimodal task execution and text editing for a wearable system
EP3642743A4 (en) A method and system for computer-aided triage
EP3627290A4 (en) Device-facing human-computer interaction method and system
EP3555857A4 (en) A locating method and system
EP3649558B8 (en) Method and system for a distributed computing system
EP3701665A4 (en) System and method for generating a blockchain application for different blockchain technologies
EP3595872A4 (en) Systems and methods for printing a fiber structure
EP3688697A4 (en) A tracking system and method
EP3371683A4 (en) A design system and method
EP3676823A4 (en) Wearable vibrotactile speech aid
EP3646694A4 (en) Work management system and work management method
EP3603114A4 (en) A wearable device
EP3304343A4 (en) Systems and methods for providing a comment-centered news reader
EP3635486A4 (en) A photography system and method
GB201720957D0 (en) Systems and method for querying a data repository
EP3591486A4 (en) Work system and work machine
EP4069081A4 (en) A wearable device and associated methods and systems
EP3642733A4 (en) System and method for segmenting a sentence
EP3360054A4 (en) Systems and methods for creating and using a data structure for parallel programming
EP3717957A4 (en) System and method for providing synthetic information on a see-through device
EP3479317A4 (en) A task management system and a method thereof
EP3348897B8 (en) A lighting system and corresponding method
EP3605441A4 (en) Information processing system and program
EP3554783A4 (en) Method for operating a sawline and a sawline
EP3393960A4 (en) Device for a jib-carried tool and a system thereof

Legal Events

Date Code Title Description
STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE

PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

17P Request for examination filed

Effective date: 20191010

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

AX Request for extension of the european patent

Extension state: BA ME

DAV Request for validation of the european patent (deleted)
DAX Request for extension of the european patent (deleted)
A4 Supplementary search report drawn up and despatched

Effective date: 20200825

RIC1 Information provided on ipc code assigned before grant

Ipc: G06F 3/01 20060101ALI20200819BHEP

Ipc: G06F 3/16 20060101ALI20200819BHEP

Ipc: G02F 1/01 20060101ALI20200819BHEP

Ipc: G02B 6/26 20060101AFI20200819BHEP

Ipc: G06F 3/0346 20130101ALI20200819BHEP

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: EXAMINATION IS IN PROGRESS

17Q First examination report despatched

Effective date: 20210812

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: GRANT OF PATENT IS INTENDED

INTG Intention to grant announced

Effective date: 20230123

GRAS Grant fee paid

Free format text: ORIGINAL CODE: EPIDOSNIGR3

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE PATENT HAS BEEN GRANTED

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

REG Reference to a national code

Ref country code: CH

Ref legal event code: EP

REG Reference to a national code

Ref country code: AT

Ref legal event code: REF

Ref document number: 1583207

Country of ref document: AT

Kind code of ref document: T

Effective date: 20230715

REG Reference to a national code

Ref country code: IE

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: DE

Ref legal event code: R096

Ref document number: 602018052502

Country of ref document: DE

P01 Opt-out of the competence of the unified patent court (upc) registered

Effective date: 20230629

REG Reference to a national code

Ref country code: NL

Ref legal event code: FP

REG Reference to a national code

Ref country code: LT

Ref legal event code: MG9D

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230628

Ref country code: NO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230928

REG Reference to a national code

Ref country code: AT

Ref legal event code: MK05

Ref document number: 1583207

Country of ref document: AT

Kind code of ref document: T

Effective date: 20230628

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: RS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230628

Ref country code: LV

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230628

Ref country code: LT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230628

Ref country code: HR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230628

Ref country code: GR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230929

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: FI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230628

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230628

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: ES

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230628

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20231028

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SM

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230628

Ref country code: SK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230628

Ref country code: RO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230628

Ref country code: PT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20231030

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20231028

Ref country code: ES

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230628

Ref country code: EE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230628

Ref country code: CZ

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230628

Ref country code: AT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230628

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: PL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230628

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: NL

Payment date: 20240320

Year of fee payment: 7