CN107408314B - 混合现实系统 - Google Patents

混合现实系统 Download PDF

Info

Publication number
CN107408314B
CN107408314B CN201680006581.7A CN201680006581A CN107408314B CN 107408314 B CN107408314 B CN 107408314B CN 201680006581 A CN201680006581 A CN 201680006581A CN 107408314 B CN107408314 B CN 107408314B
Authority
CN
China
Prior art keywords
camera
image
mixed reality
display device
reality environment
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Expired - Fee Related
Application number
CN201680006581.7A
Other languages
English (en)
Chinese (zh)
Other versions
CN107408314A (zh
Inventor
A·C·汤姆林
E·M·圭布勒
N·G·法杰特
B·J·芒特
G·L·阿尔特
J·托萨尔
J·M·里昂斯
A·J·安布鲁斯
C·Q·埃格伯特
W·盖曼
J·W·麦格林
J·汉斯
R·S-K·叙尔文
A·G·普法夫
D·克罗伊曼
E·A·索尔特威尔
C·沃德
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Microsoft Technology Licensing LLC
Original Assignee
Microsoft Technology Licensing LLC
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Microsoft Technology Licensing LLC filed Critical Microsoft Technology Licensing LLC
Publication of CN107408314A publication Critical patent/CN107408314A/zh
Application granted granted Critical
Publication of CN107408314B publication Critical patent/CN107408314B/zh
Expired - Fee Related legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating three-dimensional [3D] models or images for computer graphics
    • G06T19/006—Mixed reality
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/0093—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/017—Head mounted
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/50—Constructional details
    • H04N23/51—Housings
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/0101—Head-up displays characterised by optical features
    • G02B2027/0138—Head-up displays characterised by optical features comprising image capture systems, e.g. camera
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/017—Head mounted
    • G02B2027/0178—Eyeglass type

Landscapes

  • Physics & Mathematics (AREA)
  • Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Optics & Photonics (AREA)
  • General Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Graphics (AREA)
  • Software Systems (AREA)
  • Computer Hardware Design (AREA)
  • Human Computer Interaction (AREA)
  • Signal Processing (AREA)
  • Multimedia (AREA)
  • User Interface Of Digital Computer (AREA)
  • Controls And Circuits For Display Device (AREA)
  • Processing Or Creating Images (AREA)
  • Studio Devices (AREA)
CN201680006581.7A 2015-01-20 2016-01-13 混合现实系统 Expired - Fee Related CN107408314B (zh)

Applications Claiming Priority (5)

Application Number Priority Date Filing Date Title
US201562105701P 2015-01-20 2015-01-20
US62/105,701 2015-01-20
US14/728,650 2015-06-02
US14/728,650 US9846968B2 (en) 2015-01-20 2015-06-02 Holographic bird's eye view camera
PCT/US2016/013127 WO2016118371A1 (en) 2015-01-20 2016-01-13 Mixed reality system

Publications (2)

Publication Number Publication Date
CN107408314A CN107408314A (zh) 2017-11-28
CN107408314B true CN107408314B (zh) 2022-01-25

Family

ID=56408228

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201680006581.7A Expired - Fee Related CN107408314B (zh) 2015-01-20 2016-01-13 混合现实系统

Country Status (5)

Country Link
US (1) US9846968B2 (enExample)
EP (1) EP3248176A1 (enExample)
JP (1) JP6860488B2 (enExample)
CN (1) CN107408314B (enExample)
WO (1) WO2016118371A1 (enExample)

Families Citing this family (62)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9986225B2 (en) * 2014-02-14 2018-05-29 Autodesk, Inc. Techniques for cut-away stereo content in a stereoscopic display
US10838207B2 (en) 2015-03-05 2020-11-17 Magic Leap, Inc. Systems and methods for augmented reality
US10180734B2 (en) 2015-03-05 2019-01-15 Magic Leap, Inc. Systems and methods for augmented reality
CN112764536B (zh) 2015-03-05 2025-04-01 奇跃公司 用于增强现实的系统和方法
US10133532B2 (en) * 2015-09-25 2018-11-20 Seiko Epson Corporation Display system, display device, information display method, and program
JP2018536244A (ja) 2015-12-04 2018-12-06 マジック リープ， インコーポレイテッドＭａｇｉｃ Ｌｅａｐ，Ｉｎｃ． 再位置特定システムおよび方法
JP7133470B2 (ja) * 2016-02-18 2022-09-08 イーディーエックス テクノロジーズ インコーポレイテッド ネットワークの拡張現実表現のためのシステムおよび方法
JPWO2017159063A1 (ja) * 2016-03-14 2019-01-17 ソニー株式会社 表示装置並びに情報処理端末装置
US10210661B2 (en) * 2016-04-25 2019-02-19 Microsoft Technology Licensing, Llc Location-based holographic experience
IL264489B (en) * 2016-08-02 2022-07-01 Magic Leap Inc Virtual and augmented reality systems at a fixed distance and methods
JP2018081543A (ja) * 2016-11-17 2018-05-24 セイコーエプソン株式会社 表示装置、頭部装着型表示装置、表示装置の制御方法、及び、プログラム
CA3046399A1 (en) * 2016-12-13 2018-06-21 Magic Leap, Inc. 3d object rendering using detected features
JP6849430B2 (ja) * 2016-12-27 2021-03-24 キヤノン株式会社 画像処理装置、画像処理方法、及びプログラム
US10812936B2 (en) 2017-01-23 2020-10-20 Magic Leap, Inc. Localization determination for mixed reality systems
CN110249631B (zh) * 2017-01-31 2022-02-11 株式会社尼康 显示控制系统及显示控制方法
US10628950B2 (en) * 2017-03-01 2020-04-21 Microsoft Technology Licensing, Llc Multi-spectrum illumination-and-sensor module for head tracking, gesture recognition and spatial mapping
IL303275B2 (en) 2017-03-17 2024-08-01 Magic Leap Inc A mixed reality system with the assembly of multi-source virtual content and a method for creating virtual content using it
KR102366781B1 (ko) 2017-03-17 2022-02-22 매직 립, 인코포레이티드 컬러 가상 콘텐츠 와핑을 갖는 혼합 현실 시스템 및 이를 사용하여 가상 콘텐츠를 생성하는 방법
US10769752B2 (en) 2017-03-17 2020-09-08 Magic Leap, Inc. Mixed reality system with virtual content warping and method of generating virtual content using same
CN110402415A (zh) * 2017-03-17 2019-11-01 奇跃公司 记录增强现实数据的技术
US12498895B2 (en) 2017-06-16 2025-12-16 Apple Inc. Head-mounted device with publicly viewable display
US11861255B1 (en) 2017-06-16 2024-01-02 Apple Inc. Wearable device for facilitating enhanced interaction
WO2019015261A1 (en) * 2017-07-17 2019-01-24 Chengdu Topplusvision Technology Co., Ltd. DEVICES AND METHODS FOR SCENE DETERMINATION
CN107976811B (zh) * 2017-12-25 2023-12-29 河南诺控信息技术有限公司 一种基于虚拟现实混合的仿真实验室及其仿真方法
WO2019136588A1 (zh) * 2018-01-09 2019-07-18 深圳前海达闼云端智能科技有限公司 基于云端计算的标定方法、装置、电子设备和计算机程序产品
JP2019152899A (ja) * 2018-02-28 2019-09-12 株式会社バンダイナムコスタジオ シミュレーションシステム及びプログラム
WO2019204161A1 (en) 2018-04-20 2019-10-24 Pcms Holdings, Inc. Method and system for gaze-based control of mixed reality content
JP6688337B2 (ja) * 2018-06-06 2020-04-28 任天堂株式会社 情報処理プログラム、情報処理システム、情報処理装置および情報処理方法
US10964053B2 (en) * 2018-07-02 2021-03-30 Microsoft Technology Licensing, Llc Device pose estimation using 3D line clouds
GB201811249D0 (en) * 2018-07-09 2018-08-29 Digitalbridge System and method for virtual image alignment
EP3827584A4 (en) 2018-07-23 2021-09-08 Magic Leap, Inc. DISPLAY INTRA FIELD SUBCODE TIMING IN FIELD SEQUENTIAL DISPLAYS
EP3827299B1 (en) 2018-07-23 2025-07-09 Magic Leap, Inc. Mixed reality system with virtual content warping and method of generating virtual content using same
US10839556B2 (en) 2018-10-23 2020-11-17 Microsoft Technology Licensing, Llc Camera pose estimation using obfuscated features
WO2020101892A1 (en) 2018-11-12 2020-05-22 Magic Leap, Inc. Patch tracking image sensor
WO2020132243A1 (en) * 2018-12-21 2020-06-25 Magic Leap, Inc. Multi-camera cross reality device
US12013979B2 (en) 2019-02-07 2024-06-18 Magic Leap, Inc. Lightweight and low power cross reality device with high temporal resolution
CN113711587B (zh) 2019-02-07 2025-08-08 奇跃公司 具有无源深度提取的轻型交叉显示设备
US11372474B2 (en) * 2019-07-03 2022-06-28 Saec/Kinetic Vision, Inc. Systems and methods for virtual artificial intelligence development and testing
CN110456905A (zh) * 2019-07-23 2019-11-15 广东虚拟现实科技有限公司 定位跟踪方法、装置、系统及电子设备
CN120295462A (zh) 2019-09-09 2025-07-11 苹果公司 虚拟和物理相机的位置同步
CN111260793B (zh) * 2020-01-10 2020-11-24 中国电子科技集团公司第三十八研究所 面向增强和混合现实的远程虚实高精度匹配定位的方法
US11561613B2 (en) * 2020-05-29 2023-01-24 Magic Leap, Inc. Determining angular acceleration
US11475242B2 (en) * 2020-07-27 2022-10-18 Seiko Epson Corporation Domain adaptation losses
CN116981978A (zh) * 2020-09-21 2023-10-31 苹果公司 用于动态确定呈现和过渡区域的方法和设备
CN112669671B (zh) * 2020-12-28 2022-10-25 北京航空航天大学江西研究院 一种基于实物交互的混合现实飞行仿真系统
US11527047B2 (en) 2021-03-11 2022-12-13 Quintar, Inc. Augmented reality system for viewing an event with distributed computing
US11657578B2 (en) 2021-03-11 2023-05-23 Quintar, Inc. Registration for augmented reality system for viewing an event
US12244782B2 (en) 2021-03-11 2025-03-04 Quintar, Inc. Augmented reality system for remote presentation for viewing an event
US12028507B2 (en) 2021-03-11 2024-07-02 Quintar, Inc. Augmented reality system with remote presentation including 3D graphics extending beyond frame
US12586305B2 (en) 2021-03-11 2026-03-24 Quintar, Inc. 3D reference point detection for survey for venue model construction
US12159359B2 (en) 2021-03-11 2024-12-03 Quintar, Inc. Use of multiple registrations for augmented reality system for viewing an event
US12675962B2 (en) * 2021-03-11 2026-07-07 Quintar, Inc. Alignment of 3D graphics extending beyond frame in augmented reality system with remote presentation
US12003806B2 (en) * 2021-03-11 2024-06-04 Quintar, Inc. Augmented reality system for viewing an event with multiple coordinate systems and automatically generated model
US12542946B2 (en) 2021-03-11 2026-02-03 Quintar, Inc. Remote presentation with augmented reality content synchronized with separately displayed video content
US11645819B2 (en) 2021-03-11 2023-05-09 Quintar, Inc. Augmented reality system for viewing an event with mode based on crowd sourced images
US12482192B2 (en) * 2021-11-03 2025-11-25 The Regents Of The University California Collaborative mixed-reality system for immersive surgical telementoring
US12390104B2 (en) 2022-05-25 2025-08-19 Microsoft Technology Licensing, Llc Enhanced eye tracking using lens inverse transform
CN117234281A (zh) * 2022-06-07 2023-12-15 北京字跳网络技术有限公司 数据处理方法、装置、电子设备、头戴设备及介质
US12639885B2 (en) * 2022-07-27 2026-05-26 Apple Inc. Stereo depth markers
US12217372B2 (en) 2022-10-17 2025-02-04 T-Mobile Usa, Inc. Generating mixed reality content based on data from a wireless device
JP2024092551A (ja) * 2022-12-26 2024-07-08 国立研究開発法人情報通信研究機構 仮想物体操作装置およびそのプログラム、ならびに、仮想物体表示システム
TWI897136B (zh) * 2023-12-07 2025-09-11 宏達國際電子股份有限公司 虛擬影像顯示裝置以及立體影像的建立方法

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102346660A (zh) * 2011-10-13 2012-02-08 苏州梦想人软件科技有限公司 一种混合现实与虚拟物体的系统及方法
CN102540464A (zh) * 2010-11-18 2012-07-04 微软公司 提供环绕视频的头戴式显示设备
CN102591449A (zh) * 2010-10-27 2012-07-18 微软公司 虚拟内容和现实内容的低等待时间的融合
CN103847640A (zh) * 2012-12-05 2014-06-11 现代自动车株式会社 提供增强现实的方法和设备

Family Cites Families (29)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6711293B1 (en) 1999-03-08 2004-03-23 The University Of British Columbia Method and apparatus for identifying scale invariant features in an image and use of same for locating an object in an image
JP3363861B2 (ja) * 2000-01-13 2003-01-08 キヤノン株式会社 複合現実感提示装置及び複合現実感提示方法並びに記憶媒体
US7015831B2 (en) 2002-12-17 2006-03-21 Evolution Robotics, Inc. Systems and methods for incrementally updating a pose of a mobile device calculated by visual simultaneous localization and mapping techniques
US7401920B1 (en) 2003-05-20 2008-07-22 Elbit Systems Ltd. Head mounted eye tracking and display system
IL157837A (en) 2003-09-10 2012-12-31 Yaakov Amitai Substrate-guided optical device particularly for three-dimensional displays
CN101273368A (zh) 2005-08-29 2008-09-24 埃韦里克斯技术股份有限公司 经由移动图像识别的交互
US8160400B2 (en) 2005-11-17 2012-04-17 Microsoft Corporation Navigating images using image based geometric alignment and object based controls
KR100809479B1 (ko) 2006-07-27 2008-03-03 한국전자통신연구원 혼합 현실 환경을 위한 얼굴 착용형 디스플레이 장치
EP2193825B1 (en) 2008-12-03 2017-03-22 Alcatel Lucent Mobile device for augmented reality applications
KR20100138700A (ko) 2009-06-25 2010-12-31 삼성전자주식회사 가상 세계 처리 장치 및 방법
US20110317871A1 (en) 2010-06-29 2011-12-29 Microsoft Corporation Skeletal joint recognition and tracking system
US8941559B2 (en) 2010-09-21 2015-01-27 Microsoft Corporation Opacity filter for display device
US8884984B2 (en) * 2010-10-15 2014-11-11 Microsoft Corporation Fusing virtual content into real content
US20120113223A1 (en) 2010-11-05 2012-05-10 Microsoft Corporation User Interaction in Augmented Reality
US20120306850A1 (en) 2011-06-02 2012-12-06 Microsoft Corporation Distributed asynchronous localization and mapping for augmented reality
US8929612B2 (en) 2011-06-06 2015-01-06 Microsoft Corporation System for recognizing an open or closed hand
WO2013028908A1 (en) 2011-08-24 2013-02-28 Microsoft Corporation Touch and social cues as inputs into a computer
CA2750287C (en) 2011-08-29 2012-07-03 Microsoft Corporation Gaze detection in a see-through, near-eye, mixed reality display
US8611015B2 (en) 2011-11-22 2013-12-17 Google Inc. User interface
US9229231B2 (en) 2011-12-07 2016-01-05 Microsoft Technology Licensing, Llc Updating printed content with personalized virtual data
US20130326364A1 (en) * 2012-05-31 2013-12-05 Stephen G. Latta Position relative hologram interactions
US20130335405A1 (en) 2012-06-18 2013-12-19 Michael J. Scavezze Virtual object generation within a virtual environment
US9417692B2 (en) 2012-06-29 2016-08-16 Microsoft Technology Licensing, Llc Deep augmented reality tags for mixed reality
US20140168261A1 (en) 2012-12-13 2014-06-19 Jeffrey N. Margolis Direct interaction system mixed reality environments
JP6313432B2 (ja) * 2013-06-09 2018-04-18 株式会社ソニー・インタラクティブエンタテインメント 頭部装着ディスプレイ
US10955665B2 (en) 2013-06-18 2021-03-23 Microsoft Technology Licensing, Llc Concurrent optimal viewing of virtual objects
US10175483B2 (en) 2013-06-18 2019-01-08 Microsoft Technology Licensing, Llc Hybrid world/body locked HUD on an HMD
US9329682B2 (en) 2013-06-18 2016-05-03 Microsoft Technology Licensing, Llc Multi-step virtual object selection
US9824498B2 (en) * 2014-12-30 2017-11-21 Sony Interactive Entertainment Inc. Scanning display system in head-mounted display for virtual reality

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102591449A (zh) * 2010-10-27 2012-07-18 微软公司 虚拟内容和现实内容的低等待时间的融合
CN102540464A (zh) * 2010-11-18 2012-07-04 微软公司 提供环绕视频的头戴式显示设备
CN102346660A (zh) * 2011-10-13 2012-02-08 苏州梦想人软件科技有限公司 一种混合现实与虚拟物体的系统及方法
CN103847640A (zh) * 2012-12-05 2014-06-11 现代自动车株式会社 提供增强现实的方法和设备

Also Published As

Publication number Publication date
US9846968B2 (en) 2017-12-19
WO2016118371A1 (en) 2016-07-28
US20160210783A1 (en) 2016-07-21
EP3248176A1 (en) 2017-11-29
JP2018511098A (ja) 2018-04-19
JP6860488B2 (ja) 2021-04-14
CN107408314A (zh) 2017-11-28

Similar Documents

Publication Publication Date Title
JP6860488B2 (ja) 複合現実システム
EP3000020B1 (en) Hologram anchoring and dynamic positioning
US9710973B2 (en) Low-latency fusing of virtual and real content
US10269139B2 (en) Computer program, head-mounted display device, and calibration method
CN114761909B (zh) 头戴式设备及调整对虚拟内容项的渲染的方法
US20140168261A1 (en) Direct interaction system mixed reality environments
US10620779B2 (en) Navigating a holographic image
US12299933B2 (en) Collaborative augmented reality eyewear with ego motion alignment
US20140176591A1 (en) Low-latency fusing of color image data
US20160210780A1 (en) Applying real world scale to virtual content
US20130326364A1 (en) Position relative hologram interactions
US20160131902A1 (en) System for automatic eye tracking calibration of head mounted display device
CN105393158A (zh) 共享的和私有的全息物体
HK1171532B (en) Low-latency fusing of virtual and real content

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant
CF01 Termination of patent right due to non-payment of annual fee
CF01 Termination of patent right due to non-payment of annual fee

Granted publication date: 20220125