CN107408314B - 混合现实系统 - Google Patents

混合现实系统 Download PDF

Info

Publication number
CN107408314B
CN107408314B CN201680006581.7A CN201680006581A CN107408314B CN 107408314 B CN107408314 B CN 107408314B CN 201680006581 A CN201680006581 A CN 201680006581A CN 107408314 B CN107408314 B CN 107408314B
Authority
CN
China
Prior art keywords
camera
image
mixed reality
display device
reality environment
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201680006581.7A
Other languages
English (en)
Chinese (zh)
Other versions
CN107408314A (zh
Inventor
A·C·汤姆林
E·M·圭布勒
N·G·法杰特
B·J·芒特
G·L·阿尔特
J·托萨尔
J·M·里昂斯
A·J·安布鲁斯
C·Q·埃格伯特
W·盖曼
J·W·麦格林
J·汉斯
R·S-K·叙尔文
A·G·普法夫
D·克罗伊曼
E·A·索尔特威尔
C·沃德
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Microsoft Technology Licensing LLC
Original Assignee
Microsoft Technology Licensing LLC
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Microsoft Technology Licensing LLC filed Critical Microsoft Technology Licensing LLC
Publication of CN107408314A publication Critical patent/CN107408314A/zh
Application granted granted Critical
Publication of CN107408314B publication Critical patent/CN107408314B/zh
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating 3D models or images for computer graphics
    • G06T19/006—Mixed reality
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/0093—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/017—Head mounted
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/50—Constructional details
    • H04N23/51—Housings
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/0101—Head-up displays characterised by optical features
    • G02B2027/0138—Head-up displays characterised by optical features comprising image capture systems, e.g. camera
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/017—Head mounted
    • G02B2027/0178—Eyeglass type

Landscapes

  • Physics & Mathematics (AREA)
  • Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Optics & Photonics (AREA)
  • Theoretical Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Computer Hardware Design (AREA)
  • Software Systems (AREA)
  • Computer Graphics (AREA)
  • Human Computer Interaction (AREA)
  • Signal Processing (AREA)
  • Multimedia (AREA)
  • User Interface Of Digital Computer (AREA)
  • Studio Devices (AREA)
  • Processing Or Creating Images (AREA)
  • Controls And Circuits For Display Device (AREA)
CN201680006581.7A 2015-01-20 2016-01-13 混合现实系统 Active CN107408314B (zh)

Applications Claiming Priority (5)

Application Number Priority Date Filing Date Title
US201562105701P 2015-01-20 2015-01-20
US62/105,701 2015-01-20
US14/728,650 2015-06-02
US14/728,650 US9846968B2 (en) 2015-01-20 2015-06-02 Holographic bird's eye view camera
PCT/US2016/013127 WO2016118371A1 (en) 2015-01-20 2016-01-13 Mixed reality system

Publications (2)

Publication Number Publication Date
CN107408314A CN107408314A (zh) 2017-11-28
CN107408314B true CN107408314B (zh) 2022-01-25

Family

ID=56408228

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201680006581.7A Active CN107408314B (zh) 2015-01-20 2016-01-13 混合现实系统

Country Status (5)

Country Link
US (1) US9846968B2 (enExample)
EP (1) EP3248176A1 (enExample)
JP (1) JP6860488B2 (enExample)
CN (1) CN107408314B (enExample)
WO (1) WO2016118371A1 (enExample)

Families Citing this family (58)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9986225B2 (en) * 2014-02-14 2018-05-29 Autodesk, Inc. Techniques for cut-away stereo content in a stereoscopic display
KR102331164B1 (ko) 2015-03-05 2021-11-24 매직 립, 인코포레이티드 증강 현실을 위한 시스템들 및 방법들
US10838207B2 (en) 2015-03-05 2020-11-17 Magic Leap, Inc. Systems and methods for augmented reality
US10180734B2 (en) 2015-03-05 2019-01-15 Magic Leap, Inc. Systems and methods for augmented reality
US10133532B2 (en) * 2015-09-25 2018-11-20 Seiko Epson Corporation Display system, display device, information display method, and program
KR20180090355A (ko) 2015-12-04 2018-08-10 매직 립, 인코포레이티드 리로컬리제이션 시스템들 및 방법들
JP7133470B2 (ja) * 2016-02-18 2022-09-08 イーディーエックス テクノロジーズ インコーポレイテッド ネットワークの拡張現実表現のためのシステムおよび方法
WO2017159063A1 (ja) * 2016-03-14 2017-09-21 ソニー株式会社 表示装置並びに情報処理端末装置
US10210661B2 (en) * 2016-04-25 2019-02-19 Microsoft Technology Licensing, Llc Location-based holographic experience
WO2018026737A1 (en) * 2016-08-02 2018-02-08 Magic Leap, Inc. Fixed-distance virtual and augmented reality systems and methods
JP2018081543A (ja) * 2016-11-17 2018-05-24 セイコーエプソン株式会社 表示装置、頭部装着型表示装置、表示装置の制御方法、及び、プログラム
US10922887B2 (en) * 2016-12-13 2021-02-16 Magic Leap, Inc. 3D object rendering using detected features
JP6849430B2 (ja) * 2016-12-27 2021-03-24 キヤノン株式会社 画像処理装置、画像処理方法、及びプログラム
US10812936B2 (en) 2017-01-23 2020-10-20 Magic Leap, Inc. Localization determination for mixed reality systems
CN110249631B (zh) * 2017-01-31 2022-02-11 株式会社尼康 显示控制系统及显示控制方法
US10628950B2 (en) * 2017-03-01 2020-04-21 Microsoft Technology Licensing, Llc Multi-spectrum illumination-and-sensor module for head tracking, gesture recognition and spatial mapping
AU2018236457B2 (en) 2017-03-17 2021-04-29 Magic Leap, Inc. Mixed reality system with virtual content warping and method of generating virtual content using same
WO2018170490A1 (en) * 2017-03-17 2018-09-20 Magic Leap, Inc. Technique for recording augmented reality data
IL298822B2 (en) 2017-03-17 2025-05-01 Magic Leap Inc Mixed reality system with color virtual content distortion and method for creating virtual content therewith
EP3596702A4 (en) 2017-03-17 2020-07-22 Magic Leap, Inc. MIXED REALITY SYSTEM WITH MULTI-SOURCE VIRTUAL CONTENT COMPOSITION AND ASSOCIATED VIRTUAL CONTENT GENERATION PROCESS
US11861255B1 (en) 2017-06-16 2024-01-02 Apple Inc. Wearable device for facilitating enhanced interaction
WO2019015261A1 (en) * 2017-07-17 2019-01-24 Chengdu Topplusvision Technology Co., Ltd. DEVICES AND METHODS FOR SCENE DETERMINATION
CN107976811B (zh) * 2017-12-25 2023-12-29 河南诺控信息技术有限公司 一种基于虚拟现实混合的仿真实验室及其仿真方法
WO2019136588A1 (zh) * 2018-01-09 2019-07-18 深圳前海达闼云端智能科技有限公司 基于云端计算的标定方法、装置、电子设备和计算机程序产品
JP2019152899A (ja) * 2018-02-28 2019-09-12 株式会社バンダイナムコスタジオ シミュレーションシステム及びプログラム
CN112262361A (zh) * 2018-04-20 2021-01-22 Pcms控股公司 用于对混合现实内容进行基于注视的控制的方法和系统
JP6688337B2 (ja) * 2018-06-06 2020-04-28 任天堂株式会社 情報処理プログラム、情報処理システム、情報処理装置および情報処理方法
US10964053B2 (en) * 2018-07-02 2021-03-30 Microsoft Technology Licensing, Llc Device pose estimation using 3D line clouds
GB201811249D0 (en) * 2018-07-09 2018-08-29 Digitalbridge System and method for virtual image alignment
EP3827299B1 (en) 2018-07-23 2025-07-09 Magic Leap, Inc. Mixed reality system with virtual content warping and method of generating virtual content using same
CN112470464B (zh) 2018-07-23 2023-11-28 奇跃公司 场顺序显示器中的场内子码时序
US10839556B2 (en) 2018-10-23 2020-11-17 Microsoft Technology Licensing, Llc Camera pose estimation using obfuscated features
WO2020101892A1 (en) 2018-11-12 2020-05-22 Magic Leap, Inc. Patch tracking image sensor
CN118921450A (zh) * 2018-12-21 2024-11-08 奇跃公司 多相机交叉现实设备
CN113711587B (zh) 2019-02-07 2025-08-08 奇跃公司 具有无源深度提取的轻型交叉显示设备
US12013979B2 (en) 2019-02-07 2024-06-18 Magic Leap, Inc. Lightweight and low power cross reality device with high temporal resolution
US11372474B2 (en) * 2019-07-03 2022-06-28 Saec/Kinetic Vision, Inc. Systems and methods for virtual artificial intelligence development and testing
CN110456905A (zh) * 2019-07-23 2019-11-15 广东虚拟现实科技有限公司 定位跟踪方法、装置、系统及电子设备
CN114207557B (zh) * 2019-09-09 2025-04-11 苹果公司 虚拟和物理相机的位置同步
CN111260793B (zh) * 2020-01-10 2020-11-24 中国电子科技集团公司第三十八研究所 面向增强和混合现实的远程虚实高精度匹配定位的方法
US11288877B2 (en) 2020-01-10 2022-03-29 38th Research Institute, China Electronics Technology Group Corp. Method for matching a virtual scene of a remote scene with a real scene for augmented reality and mixed reality
EP4158615A4 (en) * 2020-05-29 2023-11-15 Magic Leap, Inc. DETERMINATION OF ANGULAR ACCELERATION
US11475242B2 (en) * 2020-07-27 2022-10-18 Seiko Epson Corporation Domain adaptation losses
CN116981978A (zh) * 2020-09-21 2023-10-31 苹果公司 用于动态确定呈现和过渡区域的方法和设备
CN112669671B (zh) * 2020-12-28 2022-10-25 北京航空航天大学江西研究院 一种基于实物交互的混合现实飞行仿真系统
US11657578B2 (en) 2021-03-11 2023-05-23 Quintar, Inc. Registration for augmented reality system for viewing an event
US12028507B2 (en) 2021-03-11 2024-07-02 Quintar, Inc. Augmented reality system with remote presentation including 3D graphics extending beyond frame
US11527047B2 (en) 2021-03-11 2022-12-13 Quintar, Inc. Augmented reality system for viewing an event with distributed computing
US11645819B2 (en) 2021-03-11 2023-05-09 Quintar, Inc. Augmented reality system for viewing an event with mode based on crowd sourced images
US12159359B2 (en) 2021-03-11 2024-12-03 Quintar, Inc. Use of multiple registrations for augmented reality system for viewing an event
US12244782B2 (en) 2021-03-11 2025-03-04 Quintar, Inc. Augmented reality system for remote presentation for viewing an event
US12003806B2 (en) * 2021-03-11 2024-06-04 Quintar, Inc. Augmented reality system for viewing an event with multiple coordinate systems and automatically generated model
US20230260240A1 (en) * 2021-03-11 2023-08-17 Quintar, Inc. Alignment of 3d graphics extending beyond frame in augmented reality system with remote presentation
US12482192B2 (en) * 2021-11-03 2025-11-25 The Regents Of The University California Collaborative mixed-reality system for immersive surgical telementoring
US12390104B2 (en) 2022-05-25 2025-08-19 Microsoft Technology Licensing, Llc Enhanced eye tracking using lens inverse transform
US12299827B2 (en) 2022-10-17 2025-05-13 T-Mobile Usa, Inc. Generating mixed reality content based on a location of a wireless device
JP2024092551A (ja) * 2022-12-26 2024-07-08 国立研究開発法人情報通信研究機構 仮想物体操作装置およびそのプログラム、ならびに、仮想物体表示システム
TWI897136B (zh) * 2023-12-07 2025-09-11 宏達國際電子股份有限公司 虛擬影像顯示裝置以及立體影像的建立方法

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102346660A (zh) * 2011-10-13 2012-02-08 苏州梦想人软件科技有限公司 一种混合现实与虚拟物体的系统及方法
CN102540464A (zh) * 2010-11-18 2012-07-04 微软公司 提供环绕视频的头戴式显示设备
CN102591449A (zh) * 2010-10-27 2012-07-18 微软公司 虚拟内容和现实内容的低等待时间的融合
CN103847640A (zh) * 2012-12-05 2014-06-11 现代自动车株式会社 提供增强现实的方法和设备

Family Cites Families (29)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6711293B1 (en) 1999-03-08 2004-03-23 The University Of British Columbia Method and apparatus for identifying scale invariant features in an image and use of same for locating an object in an image
JP3363861B2 (ja) * 2000-01-13 2003-01-08 キヤノン株式会社 複合現実感提示装置及び複合現実感提示方法並びに記憶媒体
US7015831B2 (en) 2002-12-17 2006-03-21 Evolution Robotics, Inc. Systems and methods for incrementally updating a pose of a mobile device calculated by visual simultaneous localization and mapping techniques
US7401920B1 (en) 2003-05-20 2008-07-22 Elbit Systems Ltd. Head mounted eye tracking and display system
IL157837A (en) 2003-09-10 2012-12-31 Yaakov Amitai Substrate-guided optical device particularly for three-dimensional displays
CN101273368A (zh) 2005-08-29 2008-09-24 埃韦里克斯技术股份有限公司 经由移动图像识别的交互
US8160400B2 (en) 2005-11-17 2012-04-17 Microsoft Corporation Navigating images using image based geometric alignment and object based controls
KR100809479B1 (ko) 2006-07-27 2008-03-03 한국전자통신연구원 혼합 현실 환경을 위한 얼굴 착용형 디스플레이 장치
EP2193825B1 (en) 2008-12-03 2017-03-22 Alcatel Lucent Mobile device for augmented reality applications
KR20100138700A (ko) 2009-06-25 2010-12-31 삼성전자주식회사 가상 세계 처리 장치 및 방법
US20110317871A1 (en) 2010-06-29 2011-12-29 Microsoft Corporation Skeletal joint recognition and tracking system
US8941559B2 (en) 2010-09-21 2015-01-27 Microsoft Corporation Opacity filter for display device
US8884984B2 (en) * 2010-10-15 2014-11-11 Microsoft Corporation Fusing virtual content into real content
US20120113223A1 (en) 2010-11-05 2012-05-10 Microsoft Corporation User Interaction in Augmented Reality
US20120306850A1 (en) 2011-06-02 2012-12-06 Microsoft Corporation Distributed asynchronous localization and mapping for augmented reality
US8929612B2 (en) 2011-06-06 2015-01-06 Microsoft Corporation System for recognizing an open or closed hand
WO2013028908A1 (en) 2011-08-24 2013-02-28 Microsoft Corporation Touch and social cues as inputs into a computer
CA2750287C (en) 2011-08-29 2012-07-03 Microsoft Corporation Gaze detection in a see-through, near-eye, mixed reality display
US8611015B2 (en) 2011-11-22 2013-12-17 Google Inc. User interface
US9229231B2 (en) 2011-12-07 2016-01-05 Microsoft Technology Licensing, Llc Updating printed content with personalized virtual data
US20130326364A1 (en) * 2012-05-31 2013-12-05 Stephen G. Latta Position relative hologram interactions
US20130335405A1 (en) 2012-06-18 2013-12-19 Michael J. Scavezze Virtual object generation within a virtual environment
US9417692B2 (en) 2012-06-29 2016-08-16 Microsoft Technology Licensing, Llc Deep augmented reality tags for mixed reality
US20140168261A1 (en) 2012-12-13 2014-06-19 Jeffrey N. Margolis Direct interaction system mixed reality environments
CN105359063B (zh) * 2013-06-09 2018-08-17 索尼电脑娱乐公司 利用追踪的头戴式显示器
US9329682B2 (en) 2013-06-18 2016-05-03 Microsoft Technology Licensing, Llc Multi-step virtual object selection
US10175483B2 (en) 2013-06-18 2019-01-08 Microsoft Technology Licensing, Llc Hybrid world/body locked HUD on an HMD
US10955665B2 (en) 2013-06-18 2021-03-23 Microsoft Technology Licensing, Llc Concurrent optimal viewing of virtual objects
US9824498B2 (en) * 2014-12-30 2017-11-21 Sony Interactive Entertainment Inc. Scanning display system in head-mounted display for virtual reality

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102591449A (zh) * 2010-10-27 2012-07-18 微软公司 虚拟内容和现实内容的低等待时间的融合
CN102540464A (zh) * 2010-11-18 2012-07-04 微软公司 提供环绕视频的头戴式显示设备
CN102346660A (zh) * 2011-10-13 2012-02-08 苏州梦想人软件科技有限公司 一种混合现实与虚拟物体的系统及方法
CN103847640A (zh) * 2012-12-05 2014-06-11 现代自动车株式会社 提供增强现实的方法和设备

Also Published As

Publication number Publication date
CN107408314A (zh) 2017-11-28
US20160210783A1 (en) 2016-07-21
EP3248176A1 (en) 2017-11-29
US9846968B2 (en) 2017-12-19
JP6860488B2 (ja) 2021-04-14
JP2018511098A (ja) 2018-04-19
WO2016118371A1 (en) 2016-07-28

Similar Documents

Publication Publication Date Title
CN107408314B (zh) 混合现实系统
EP3000020B1 (en) Hologram anchoring and dynamic positioning
US9710973B2 (en) Low-latency fusing of virtual and real content
US10269139B2 (en) Computer program, head-mounted display device, and calibration method
US20140168261A1 (en) Direct interaction system mixed reality environments
US10620779B2 (en) Navigating a holographic image
US20140176591A1 (en) Low-latency fusing of color image data
US20160210780A1 (en) Applying real world scale to virtual content
US12299933B2 (en) Collaborative augmented reality eyewear with ego motion alignment
US20130326364A1 (en) Position relative hologram interactions
US20160131902A1 (en) System for automatic eye tracking calibration of head mounted display device
CN105393158A (zh) 共享的和私有的全息物体
HK1171532B (en) Low-latency fusing of virtual and real content

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant