KR102926290B1 - 증강 현실 환경을 위한 시선 기반 인터페이스 - Google Patents

증강 현실 환경을 위한 시선 기반 인터페이스

Info

Publication number
KR102926290B1
KR102926290B1 KR1020217002660A KR20217002660A KR102926290B1 KR 102926290 B1 KR102926290 B1 KR 102926290B1 KR 1020217002660 A KR1020217002660 A KR 1020217002660A KR 20217002660 A KR20217002660 A KR 20217002660A KR 102926290 B1 KR102926290 B1 KR 102926290B1
Authority
KR
South Korea
Prior art keywords
augmented reality
determining
user
hmd
display
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
KR1020217002660A
Other languages
English (en)
Korean (ko)
Other versions
KR20210059697A (ko
Inventor
월터 블럼
마이클 케이. 사우스워스
제니퍼 엔. 아바리 실바
조나단 알. 실바
Original Assignee
센티에이알, 인코포레이티드
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 센티에이알, 인코포레이티드 filed Critical 센티에이알, 인코포레이티드
Publication of KR20210059697A publication Critical patent/KR20210059697A/ko
Application granted granted Critical
Publication of KR102926290B1 publication Critical patent/KR102926290B1/ko
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/017—Head mounted
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G06F3/012—Head tracking input arrangements
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G06F3/013—Eye tracking input arrangements
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/017—Gesture based interaction, e.g. based on a set of recognized hand gestures
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0481—Interaction techniques based on graphical user interfaces [GUI] based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
    • G06F3/04815—Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0484—Interaction techniques based on graphical user interfaces [GUI] for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating three-dimensional [3D] models or images for computer graphics
    • G06T19/006—Mixed reality
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/0179—Display position adjusting means not related to the information to be displayed
    • G02B2027/0187—Display position adjusting means not related to the information to be displayed slaved to motion of at least a part of the body of the user, e.g. head, eye

Landscapes

  • Engineering & Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Human Computer Interaction (AREA)
  • Software Systems (AREA)
  • Computer Graphics (AREA)
  • Computer Hardware Design (AREA)
  • Optics & Photonics (AREA)
  • User Interface Of Digital Computer (AREA)
  • Processing Or Creating Images (AREA)
  • Position Input By Displaying (AREA)
KR1020217002660A 2018-06-27 2019-06-25 증강 현실 환경을 위한 시선 기반 인터페이스 Active KR102926290B1 (ko)

Applications Claiming Priority (3)

Application Number Priority Date Filing Date Title
US201862690834P 2018-06-27 2018-06-27
US62/690,834 2018-06-27
PCT/US2019/039076 WO2020006002A1 (en) 2018-06-27 2019-06-25 Gaze based interface for augmented reality environment

Publications (2)

Publication Number Publication Date
KR20210059697A KR20210059697A (ko) 2021-05-25
KR102926290B1 true KR102926290B1 (ko) 2026-02-11

Family

ID=68985564

Family Applications (1)

Application Number Title Priority Date Filing Date
KR1020217002660A Active KR102926290B1 (ko) 2018-06-27 2019-06-25 증강 현실 환경을 위한 시선 기반 인터페이스

Country Status (8)

Country Link
US (2) US11199898B2 (enExample)
EP (1) EP3814829A4 (enExample)
JP (1) JP7213899B2 (enExample)
KR (1) KR102926290B1 (enExample)
AU (1) AU2019294617B2 (enExample)
CA (1) CA3105192A1 (enExample)
IL (1) IL279705B2 (enExample)
WO (1) WO2020006002A1 (enExample)

Families Citing this family (48)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2018156809A1 (en) * 2017-02-24 2018-08-30 Masimo Corporation Augmented reality system for displaying patient data
EP3585254B1 (en) 2017-02-24 2024-03-20 Masimo Corporation Medical device cable and method of sharing data between connected medical devices
CN107223082B (zh) * 2017-04-21 2020-05-12 深圳前海达闼云端智能科技有限公司 一种机器人控制方法、机器人装置及机器人设备
JP7159208B2 (ja) 2017-05-08 2022-10-24 マシモ・コーポレイション ドングルを使用することによって医療システムをネットワークコントローラとペアリングするためのシステム
JP2021182174A (ja) * 2018-08-07 2021-11-25 ソニーグループ株式会社 情報処理装置、情報処理方法およびプログラム
US10891792B1 (en) * 2019-01-31 2021-01-12 Splunk Inc. Precise plane detection and placement of virtual objects in an augmented reality environment
WO2021104300A1 (zh) 2019-11-25 2021-06-03 京东方科技集团股份有限公司 健康管理系统以及用于其的人体信息显示方法和人体模型生成方法
US11816757B1 (en) * 2019-12-11 2023-11-14 Meta Platforms Technologies, Llc Device-side capture of data representative of an artificial reality environment
US10921882B1 (en) * 2019-12-26 2021-02-16 Jie Li Human-machine interaction method, system and apparatus for controlling an electronic device
US11922581B2 (en) * 2020-05-08 2024-03-05 Coviden Lp Systems and methods of controlling an operating room display using an augmented reality headset
AU2021349381B2 (en) 2020-09-25 2024-02-22 Apple Inc. Methods for interacting with virtual controls and/or an affordance for moving virtual objects in virtual environments
EP4697149A3 (en) 2020-09-25 2026-03-11 Apple Inc. Methods for adjusting and/or controlling immersion associated with user interfaces
CN117032450B (zh) 2020-09-25 2024-11-08 苹果公司 用于操纵环境中的对象的方法
EP4213482A4 (en) * 2020-12-16 2024-03-20 Samsung Electronics Co., Ltd. Method and device for transmitting plurality of pieces of application data with low latency
CN116670627A (zh) 2020-12-31 2023-08-29 苹果公司 对环境中的用户界面进行分组的方法
US11995230B2 (en) 2021-02-11 2024-05-28 Apple Inc. Methods for presenting and sharing content in an environment
EP4388397A1 (en) 2021-09-25 2024-06-26 Apple Inc. Devices, methods, and graphical user interfaces for presenting virtual objects in virtual environments
US12456271B1 (en) 2021-11-19 2025-10-28 Apple Inc. System and method of three-dimensional object cleanup and text annotation
CN119556830A (zh) 2022-01-12 2025-03-04 苹果公司 用于在环境中显示、选择以及移动对象和容器的方法
JP7843357B2 (ja) 2022-01-19 2026-04-09 アップル インコーポレイテッド 環境内にオブジェクトを表示及び再位置決めするための方法
EP4398593A4 (en) 2022-02-11 2025-02-26 Samsung Electronics Co., Ltd. METHOD AND DEVICE FOR OBTAINING AN IMAGE OF AN OBJECT
WO2023153611A1 (ko) * 2022-02-11 2023-08-17 삼성전자주식회사 객체의 영상을 획득하는 방법 및 장치
US12541280B2 (en) 2022-02-28 2026-02-03 Apple Inc. System and method of three-dimensional placement and refinement in multi-user communication sessions
US12272005B2 (en) 2022-02-28 2025-04-08 Apple Inc. System and method of three-dimensional immersive applications in multi-user communication sessions
JP2023143437A (ja) * 2022-03-25 2023-10-06 富士フイルム株式会社 制御装置、制御方法、及び制御プログラム
US12321666B2 (en) 2022-04-04 2025-06-03 Apple Inc. Methods for quick message response and dictation in a three-dimensional environment
US12548206B2 (en) 2022-04-18 2026-02-10 Lyv Technologies Inc. Mixed-reality beacons
EP4511722A1 (en) 2022-04-20 2025-02-26 Apple Inc. Obstructed objects in a three-dimensional environment
WO2023205457A1 (en) 2022-04-21 2023-10-26 Apple Inc. Representations of messages in a three-dimensional environment
US12394167B1 (en) 2022-06-30 2025-08-19 Apple Inc. Window resizing and virtual object rearrangement in 3D environments
US12112011B2 (en) 2022-09-16 2024-10-08 Apple Inc. System and method of application-based three-dimensional refinement in multi-user communication sessions
US12169598B2 (en) * 2022-09-19 2024-12-17 Snap Inc. AR glasses as IoT remote control
US12099653B2 (en) 2022-09-22 2024-09-24 Apple Inc. User interface response based on gaze-holding event assessment
US12405704B1 (en) 2022-09-23 2025-09-02 Apple Inc. Interpreting user movement as direct touch user interface interactions
CN120266083A (zh) 2022-09-24 2025-07-04 苹果公司 用于基于注意力与用户界面交互的方法
KR20250075620A (ko) 2022-09-24 2025-05-28 애플 인크. 3차원 환경을 제어하고 그와 상호작용하기 위한 방법들
JP2024052410A (ja) * 2022-09-30 2024-04-11 富士フイルム株式会社 プロセッサ、情報処理方法、及び情報処理プログラム
US12361661B1 (en) 2022-12-21 2025-07-15 Meta Platforms Technologies, Llc Artificial reality (XR) location-based displays and interactions
US12501099B2 (en) 2023-01-17 2025-12-16 Snap Inc. AR glasses as IoT device for enhanced screen experience
CN121165940A (zh) 2023-01-30 2025-12-19 苹果公司 用于响应于注视和/或手势输入来显示多组控件的设备、方法和图形用户界面
WO2024186838A2 (en) * 2023-03-05 2024-09-12 Walker Stanley A system for an independent recursive virtual world with a voice controlled interface
US12443286B2 (en) 2023-06-02 2025-10-14 Apple Inc. Input recognition based on distinguishing direct and indirect user interactions
US12118200B1 (en) 2023-06-02 2024-10-15 Apple Inc. Fuzzy hit testing
CN121187445A (zh) 2023-06-04 2025-12-23 苹果公司 用于管理重叠窗口和应用视觉效果的方法
US12099695B1 (en) 2023-06-04 2024-09-24 Apple Inc. Systems and methods of managing spatial groups in multi-user communication sessions
US12399550B2 (en) 2024-01-25 2025-08-26 Depuy Ireland Unlimited Company Triggering an acquisition during mixed-reality headset-based navigation
US12585131B2 (en) * 2024-04-09 2026-03-24 DISTANCE TECHNOLOGIES Oy Hybrid motion stabilisation for augmented reality displays
US20250378645A1 (en) 2024-06-09 2025-12-11 Apple Inc. Methods of interacting with content in a virtual environment

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20140375544A1 (en) * 2013-06-25 2014-12-25 Morgan Kolya Venable User interface navigation
US20150169070A1 (en) * 2013-12-17 2015-06-18 Google Inc. Visual Display of Interactive, Gesture-Controlled, Three-Dimensional (3D) Models for Head-Mountable Displays (HMDs)
US20170270715A1 (en) * 2016-03-21 2017-09-21 Megan Ann Lindsay Displaying three-dimensional virtual objects based on field of view
US20170277256A1 (en) * 2016-03-22 2017-09-28 Aaron Mackay Burns Virtual-reality navigation

Family Cites Families (23)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102067180B (zh) * 2008-04-14 2014-07-09 谷歌公司 使用虚拟表面摇摄
US9977496B2 (en) * 2010-07-23 2018-05-22 Telepatheye Inc. Eye-wearable device user interface and augmented reality method
US20130154913A1 (en) 2010-12-16 2013-06-20 Siemens Corporation Systems and methods for a gaze and gesture interface
US20120257035A1 (en) * 2011-04-08 2012-10-11 Sony Computer Entertainment Inc. Systems and methods for providing feedback by tracking user gaze and gestures
US8611015B2 (en) * 2011-11-22 2013-12-17 Google Inc. User interface
US20160140766A1 (en) * 2012-12-12 2016-05-19 Sulon Technologies Inc. Surface projection system and method for augmented reality
US11278353B2 (en) * 2016-03-16 2022-03-22 Synaptive Medical Inc. Trajectory alignment system and methods
US9094677B1 (en) * 2013-07-25 2015-07-28 Google Inc. Head mounted display device with automated positioning
US10466858B2 (en) 2013-12-01 2019-11-05 Upskill, Inc. Systems and methods for interacting with a virtual menu
US9696798B2 (en) * 2014-04-09 2017-07-04 Lenovo Enterprise Solutions (Singapore) Pte. Ltd. Eye gaze direction indicator
US10416760B2 (en) * 2014-07-25 2019-09-17 Microsoft Technology Licensing, Llc Gaze-based object placement within a virtual reality environment
US10186086B2 (en) * 2015-09-02 2019-01-22 Microsoft Technology Licensing, Llc Augmented reality control of computing device
CA2997965C (en) * 2015-10-14 2021-04-27 Surgical Theater LLC Augmented reality surgical navigation
US10667981B2 (en) * 2016-02-29 2020-06-02 Mentor Acquisition One, Llc Reading assistance system for visually impaired
EP3440495A1 (en) * 2016-04-08 2019-02-13 Google LLC Encoding image data at a head mounted display device based on pose information
US10429647B2 (en) * 2016-06-10 2019-10-01 Facebook Technologies, Llc Focus adjusting virtual reality headset
JP6238381B1 (ja) 2016-06-30 2017-11-29 株式会社コナミデジタルエンタテインメント 端末装置、及びプログラム
US20180033198A1 (en) * 2016-07-29 2018-02-01 Microsoft Technology Licensing, Llc Forward direction determination for augmented reality and virtual reality
JP6939801B2 (ja) 2016-09-14 2021-09-22 ソニーグループ株式会社 情報処理装置、情報処理方法およびプログラム
US10664049B2 (en) * 2016-12-09 2020-05-26 Nvidia Corporation Systems and methods for gaze tracking
US10692287B2 (en) * 2017-04-17 2020-06-23 Microsoft Technology Licensing, Llc Multi-step placement of virtual objects
WO2018217951A1 (en) * 2017-05-24 2018-11-29 Camplex, Inc. Surgical visualization systems and displays
US10861236B2 (en) * 2017-09-08 2020-12-08 Surgical Theater, Inc. Dual mode augmented reality surgical system and method

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20140375544A1 (en) * 2013-06-25 2014-12-25 Morgan Kolya Venable User interface navigation
US20150169070A1 (en) * 2013-12-17 2015-06-18 Google Inc. Visual Display of Interactive, Gesture-Controlled, Three-Dimensional (3D) Models for Head-Mountable Displays (HMDs)
US20170270715A1 (en) * 2016-03-21 2017-09-21 Megan Ann Lindsay Displaying three-dimensional virtual objects based on field of view
US20170277256A1 (en) * 2016-03-22 2017-09-28 Aaron Mackay Burns Virtual-reality navigation

Also Published As

Publication number Publication date
US20220075449A1 (en) 2022-03-10
WO2020006002A1 (en) 2020-01-02
US20200004328A1 (en) 2020-01-02
JP7213899B2 (ja) 2023-01-27
EP3814829A1 (en) 2021-05-05
KR20210059697A (ko) 2021-05-25
AU2019294617A1 (en) 2021-02-18
US11829526B2 (en) 2023-11-28
IL279705B2 (en) 2025-04-01
EP3814829A4 (en) 2022-03-23
IL279705B1 (en) 2024-12-01
US11199898B2 (en) 2021-12-14
IL279705A (en) 2021-03-01
CA3105192A1 (en) 2020-01-02
JP2021528786A (ja) 2021-10-21
AU2019294617B2 (en) 2025-02-06

Similar Documents

Publication Publication Date Title
KR102926290B1 (ko) 증강 현실 환경을 위한 시선 기반 인터페이스
US11507336B2 (en) Augmented reality display sharing
JP7731362B2 (ja) ウェアラブルシステムのための手のジェスチャ入力
US12057037B2 (en) User interface systems for sterile fields and other working environments
EP3380031B1 (en) Method and system for interacting with medical information
US11475642B2 (en) Methods and systems for selection of objects
WO2013028268A1 (en) Method and system for use in providing three dimensional user interface
EP4135583B1 (en) System and method for augmented reality data interaction for ultrasound imaging
JP7631274B2 (ja) 制御装置、制御方法、プログラム
US20160004318A1 (en) System and method of touch-free operation of a picture archiving and communication system

Legal Events

Date Code Title Description
PA0105 International application

St.27 status event code: A-0-1-A10-A15-nap-PA0105

P11-X000 Amendment of application requested

St.27 status event code: A-2-2-P10-P11-nap-X000

P13-X000 Application amended

St.27 status event code: A-2-2-P10-P13-nap-X000

PG1501 Laying open of application

St.27 status event code: A-1-1-Q10-Q12-nap-PG1501

A201 Request for examination
P11-X000 Amendment of application requested

St.27 status event code: A-2-2-P10-P11-nap-X000

P13-X000 Application amended

St.27 status event code: A-2-2-P10-P13-nap-X000

PA0201 Request for examination

St.27 status event code: A-1-2-D10-D11-exm-PA0201

E902 Notification of reason for refusal
PE0902 Notice of grounds for rejection

St.27 status event code: A-1-2-D10-D21-exm-PE0902

T11-X000 Administrative time limit extension requested

St.27 status event code: U-3-3-T10-T11-oth-X000

E13-X000 Pre-grant limitation requested

St.27 status event code: A-2-3-E10-E13-lim-X000

P11-X000 Amendment of application requested

St.27 status event code: A-2-2-P10-P11-nap-X000

P13-X000 Application amended

St.27 status event code: A-2-2-P10-P13-nap-X000

E90F Notification of reason for final refusal
PE0902 Notice of grounds for rejection

St.27 status event code: A-1-2-D10-D21-exm-PE0902

P11-X000 Amendment of application requested

St.27 status event code: A-2-2-P10-P11-nap-X000

D22 Grant of ip right intended

Free format text: ST27 STATUS EVENT CODE: A-1-2-D10-D22-EXM-PE0701 (AS PROVIDED BY THE NATIONAL OFFICE)

PE0701 Decision of registration

St.27 status event code: A-1-2-D10-D22-exm-PE0701

PR1002 Payment of registration fee

St.27 status event code: A-2-2-U10-U12-oth-PR1002

Fee payment year number: 1

U12 Designation fee paid

Free format text: ST27 STATUS EVENT CODE: A-2-2-U10-U12-OTH-PR1002 (AS PROVIDED BY THE NATIONAL OFFICE)

Year of fee payment: 1

PG1601 Publication of registration

St.27 status event code: A-4-4-Q10-Q13-nap-PG1601

Q13 Ip right document published

Free format text: ST27 STATUS EVENT CODE: A-4-4-Q10-Q13-NAP-PG1601 (AS PROVIDED BY THE NATIONAL OFFICE)