CN107408314B - 混合现实系统 - Google Patents

混合现实系统 Download PDF

Info

Publication number
CN107408314B
CN107408314B CN201680006581.7A CN201680006581A CN107408314B CN 107408314 B CN107408314 B CN 107408314B CN 201680006581 A CN201680006581 A CN 201680006581A CN 107408314 B CN107408314 B CN 107408314B
Authority
CN
China
Prior art keywords
camera
image
mixed reality
display device
reality environment
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Expired - Fee Related
Application number
CN201680006581.7A
Other languages
English (en)
Chinese (zh)
Other versions
CN107408314A (zh
Inventor
A·C·汤姆林
E·M·圭布勒
N·G·法杰特
B·J·芒特
G·L·阿尔特
J·托萨尔
J·M·里昂斯
A·J·安布鲁斯
C·Q·埃格伯特
W·盖曼
J·W·麦格林
J·汉斯
R·S-K·叙尔文
A·G·普法夫
D·克罗伊曼
E·A·索尔特威尔
C·沃德
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Microsoft Technology Licensing LLC
Original Assignee
Microsoft Technology Licensing LLC
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Microsoft Technology Licensing LLC filed Critical Microsoft Technology Licensing LLC
Publication of CN107408314A publication Critical patent/CN107408314A/zh
Application granted granted Critical
Publication of CN107408314B publication Critical patent/CN107408314B/zh
Expired - Fee Related legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating three-dimensional [3D] models or images for computer graphics
    • G06T19/006—Mixed reality
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/0093—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/017—Head mounted
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/50—Constructional details
    • H04N23/51—Housings
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/0101—Head-up displays characterised by optical features
    • G02B2027/0138—Head-up displays characterised by optical features comprising image capture systems, e.g. camera
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/017—Head mounted
    • G02B2027/0178—Eyeglass type

Landscapes

  • Physics & Mathematics (AREA)
  • Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Optics & Photonics (AREA)
  • General Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Graphics (AREA)
  • Computer Hardware Design (AREA)
  • Software Systems (AREA)
  • Human Computer Interaction (AREA)
  • Signal Processing (AREA)
  • Multimedia (AREA)
  • User Interface Of Digital Computer (AREA)
  • Controls And Circuits For Display Device (AREA)
  • Processing Or Creating Images (AREA)
  • Studio Devices (AREA)
CN201680006581.7A 2015-01-20 2016-01-13 混合现实系统 Expired - Fee Related CN107408314B (zh)

Applications Claiming Priority (5)

Application Number Priority Date Filing Date Title
US201562105701P 2015-01-20 2015-01-20
US62/105,701 2015-01-20
US14/728,650 2015-06-02
US14/728,650 US9846968B2 (en) 2015-01-20 2015-06-02 Holographic bird's eye view camera
PCT/US2016/013127 WO2016118371A1 (en) 2015-01-20 2016-01-13 Mixed reality system

Publications (2)

Publication Number Publication Date
CN107408314A CN107408314A (zh) 2017-11-28
CN107408314B true CN107408314B (zh) 2022-01-25

Family

ID=56408228

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201680006581.7A Expired - Fee Related CN107408314B (zh) 2015-01-20 2016-01-13 混合现实系统

Country Status (5)

Country Link
US (1) US9846968B2 (https=)
EP (1) EP3248176A1 (https=)
JP (1) JP6860488B2 (https=)
CN (1) CN107408314B (https=)
WO (1) WO2016118371A1 (https=)

Families Citing this family (62)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9986225B2 (en) * 2014-02-14 2018-05-29 Autodesk, Inc. Techniques for cut-away stereo content in a stereoscopic display
AU2016225963B2 (en) 2015-03-05 2021-05-13 Magic Leap, Inc. Systems and methods for augmented reality
US10180734B2 (en) 2015-03-05 2019-01-15 Magic Leap, Inc. Systems and methods for augmented reality
US10838207B2 (en) 2015-03-05 2020-11-17 Magic Leap, Inc. Systems and methods for augmented reality
US10133532B2 (en) * 2015-09-25 2018-11-20 Seiko Epson Corporation Display system, display device, information display method, and program
US10909711B2 (en) 2015-12-04 2021-02-02 Magic Leap, Inc. Relocalization systems and methods
WO2017143239A1 (en) * 2016-02-18 2017-08-24 Edx Wireless, Inc. Systems and methods for augmented reality representations of networks
WO2017159063A1 (ja) * 2016-03-14 2017-09-21 ソニー株式会社 表示装置並びに情報処理端末装置
US10210661B2 (en) * 2016-04-25 2019-02-19 Microsoft Technology Licensing, Llc Location-based holographic experience
US10649211B2 (en) * 2016-08-02 2020-05-12 Magic Leap, Inc. Fixed-distance virtual and augmented reality systems and methods
JP2018081543A (ja) * 2016-11-17 2018-05-24 セイコーエプソン株式会社 表示装置、頭部装着型表示装置、表示装置の制御方法、及び、プログラム
KR102497707B1 (ko) * 2016-12-13 2023-02-08 매직 립, 인코포레이티드 검출된 특징들을 사용한 3d 객체 렌더링
JP6849430B2 (ja) * 2016-12-27 2021-03-24 キヤノン株式会社 画像処理装置、画像処理方法、及びプログラム
US10812936B2 (en) 2017-01-23 2020-10-20 Magic Leap, Inc. Localization determination for mixed reality systems
JP6965896B2 (ja) * 2017-01-31 2021-11-10 株式会社ニコン 表示制御システム、及び、表示制御方法
US10628950B2 (en) * 2017-03-01 2020-04-21 Microsoft Technology Licensing, Llc Multi-spectrum illumination-and-sensor module for head tracking, gesture recognition and spatial mapping
AU2018236457B2 (en) 2017-03-17 2021-04-29 Magic Leap, Inc. Mixed reality system with virtual content warping and method of generating virtual content using same
WO2018170482A1 (en) 2017-03-17 2018-09-20 Magic Leap, Inc. Mixed reality system with color virtual content warping and method of generating virtual content using same
WO2018170490A1 (en) 2017-03-17 2018-09-20 Magic Leap, Inc. Technique for recording augmented reality data
KR102359978B1 (ko) 2017-03-17 2022-02-07 매직 립, 인코포레이티드 다중-소스 가상 콘텐츠 합성을 갖는 혼합 현실 시스템 및 이를 사용하여 가상 콘텐츠를 생성하는 방법
US11861255B1 (en) 2017-06-16 2024-01-02 Apple Inc. Wearable device for facilitating enhanced interaction
US12498895B2 (en) 2017-06-16 2025-12-16 Apple Inc. Head-mounted device with publicly viewable display
WO2019015261A1 (en) * 2017-07-17 2019-01-24 Chengdu Topplusvision Technology Co., Ltd. DEVICES AND METHODS FOR SCENE DETERMINATION
CN107976811B (zh) * 2017-12-25 2023-12-29 河南诺控信息技术有限公司 一种基于虚拟现实混合的仿真实验室及其仿真方法
CN108235778B (zh) * 2018-01-09 2021-09-10 达闼机器人有限公司 基于云端计算的标定方法、装置、电子设备和计算机程序产品
JP2019152899A (ja) * 2018-02-28 2019-09-12 株式会社バンダイナムコスタジオ シミュレーションシステム及びプログラム
WO2019204161A1 (en) 2018-04-20 2019-10-24 Pcms Holdings, Inc. Method and system for gaze-based control of mixed reality content
JP6688337B2 (ja) * 2018-06-06 2020-04-28 任天堂株式会社 情報処理プログラム、情報処理システム、情報処理装置および情報処理方法
US10964053B2 (en) * 2018-07-02 2021-03-30 Microsoft Technology Licensing, Llc Device pose estimation using 3D line clouds
GB201811249D0 (en) * 2018-07-09 2018-08-29 Digitalbridge System and method for virtual image alignment
WO2020023523A1 (en) 2018-07-23 2020-01-30 Magic Leap, Inc. Intra-field sub code timing in field sequential displays
US11379948B2 (en) 2018-07-23 2022-07-05 Magic Leap, Inc. Mixed reality system with virtual content warping and method of generating virtual content using same
US10839556B2 (en) 2018-10-23 2020-11-17 Microsoft Technology Licensing, Llc Camera pose estimation using obfuscated features
US11902677B2 (en) 2018-11-12 2024-02-13 Magic Leap, Inc. Patch tracking image sensor
US20220051441A1 (en) * 2018-12-21 2022-02-17 Magic Leap, Inc. Multi-camera cross reality device
JP7720783B2 (ja) 2019-02-07 2025-08-08 マジック リープ， インコーポレイテッド 受動深度抽出を伴う軽量のクロスリアリティデバイス
EP3922011A4 (en) 2019-02-07 2022-10-26 Magic Leap, Inc. LIGHTWEIGHT, LOW POWER, HIGH TIME RESOLUTION VIRTUAL REALITY DEVICE
US11372474B2 (en) * 2019-07-03 2022-06-28 Saec/Kinetic Vision, Inc. Systems and methods for virtual artificial intelligence development and testing
CN110456905A (zh) * 2019-07-23 2019-11-15 广东虚拟现实科技有限公司 定位跟踪方法、装置、系统及电子设备
JP7385733B2 (ja) * 2019-09-09 2023-11-22 アップル インコーポレイテッド 仮想カメラと物理的カメラとの位置同期
CN111260793B (zh) * 2020-01-10 2020-11-24 中国电子科技集团公司第三十八研究所 面向增强和混合现实的远程虚实高精度匹配定位的方法
US11288877B2 (en) 2020-01-10 2022-03-29 38th Research Institute, China Electronics Technology Group Corp. Method for matching a virtual scene of a remote scene with a real scene for augmented reality and mixed reality
JP2023527357A (ja) * 2020-05-29 2023-06-28 マジック リープ， インコーポレイテッド 角加速の決定
US11475242B2 (en) * 2020-07-27 2022-10-18 Seiko Epson Corporation Domain adaptation losses
CN116981978A (zh) * 2020-09-21 2023-10-31 苹果公司 用于动态确定呈现和过渡区域的方法和设备
CN112669671B (zh) * 2020-12-28 2022-10-25 北京航空航天大学江西研究院 一种基于实物交互的混合现实飞行仿真系统
US20230260240A1 (en) * 2021-03-11 2023-08-17 Quintar, Inc. Alignment of 3d graphics extending beyond frame in augmented reality system with remote presentation
US12542946B2 (en) 2021-03-11 2026-02-03 Quintar, Inc. Remote presentation with augmented reality content synchronized with separately displayed video content
US11645819B2 (en) 2021-03-11 2023-05-09 Quintar, Inc. Augmented reality system for viewing an event with mode based on crowd sourced images
US12003806B2 (en) * 2021-03-11 2024-06-04 Quintar, Inc. Augmented reality system for viewing an event with multiple coordinate systems and automatically generated model
US11527047B2 (en) 2021-03-11 2022-12-13 Quintar, Inc. Augmented reality system for viewing an event with distributed computing
US12244782B2 (en) 2021-03-11 2025-03-04 Quintar, Inc. Augmented reality system for remote presentation for viewing an event
US12159359B2 (en) 2021-03-11 2024-12-03 Quintar, Inc. Use of multiple registrations for augmented reality system for viewing an event
US11657578B2 (en) 2021-03-11 2023-05-23 Quintar, Inc. Registration for augmented reality system for viewing an event
US12028507B2 (en) 2021-03-11 2024-07-02 Quintar, Inc. Augmented reality system with remote presentation including 3D graphics extending beyond frame
US12586305B2 (en) 2021-03-11 2026-03-24 Quintar, Inc. 3D reference point detection for survey for venue model construction
US12482192B2 (en) * 2021-11-03 2025-11-25 The Regents Of The University California Collaborative mixed-reality system for immersive surgical telementoring
US12390104B2 (en) * 2022-05-25 2025-08-19 Microsoft Technology Licensing, Llc Enhanced eye tracking using lens inverse transform
US12217372B2 (en) 2022-10-17 2025-02-04 T-Mobile Usa, Inc. Generating mixed reality content based on data from a wireless device
CN118259740A (zh) * 2022-12-26 2024-06-28 北京字跳网络技术有限公司 交互界面显示跟随方法和装置、电子设备、介质、芯片和程序产品
JP2024092551A (ja) * 2022-12-26 2024-07-08 国立研究開発法人情報通信研究機構 仮想物体操作装置およびそのプログラム、ならびに、仮想物体表示システム
TWI897136B (zh) * 2023-12-07 2025-09-11 宏達國際電子股份有限公司 虛擬影像顯示裝置以及立體影像的建立方法

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102346660A (zh) * 2011-10-13 2012-02-08 苏州梦想人软件科技有限公司 一种混合现实与虚拟物体的系统及方法
CN102540464A (zh) * 2010-11-18 2012-07-04 微软公司 提供环绕视频的头戴式显示设备
CN102591449A (zh) * 2010-10-27 2012-07-18 微软公司 虚拟内容和现实内容的低等待时间的融合
CN103847640A (zh) * 2012-12-05 2014-06-11 现代自动车株式会社 提供增强现实的方法和设备

Family Cites Families (29)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6711293B1 (en) 1999-03-08 2004-03-23 The University Of British Columbia Method and apparatus for identifying scale invariant features in an image and use of same for locating an object in an image
JP3363861B2 (ja) * 2000-01-13 2003-01-08 キヤノン株式会社 複合現実感提示装置及び複合現実感提示方法並びに記憶媒体
US7015831B2 (en) 2002-12-17 2006-03-21 Evolution Robotics, Inc. Systems and methods for incrementally updating a pose of a mobile device calculated by visual simultaneous localization and mapping techniques
US7401920B1 (en) 2003-05-20 2008-07-22 Elbit Systems Ltd. Head mounted eye tracking and display system
IL157837A (en) 2003-09-10 2012-12-31 Yaakov Amitai Substrate-guided optical device particularly for three-dimensional displays
KR101019569B1 (ko) 2005-08-29 2011-03-08 에브릭스 테크놀로지스, 인코포레이티드 모바일 이미지 인식을 통한 상호작용
US8160400B2 (en) 2005-11-17 2012-04-17 Microsoft Corporation Navigating images using image based geometric alignment and object based controls
KR100809479B1 (ko) 2006-07-27 2008-03-03 한국전자통신연구원 혼합 현실 환경을 위한 얼굴 착용형 디스플레이 장치
EP2193825B1 (en) 2008-12-03 2017-03-22 Alcatel Lucent Mobile device for augmented reality applications
KR20100138700A (ko) 2009-06-25 2010-12-31 삼성전자주식회사 가상 세계 처리 장치 및 방법
US20110317871A1 (en) 2010-06-29 2011-12-29 Microsoft Corporation Skeletal joint recognition and tracking system
US8941559B2 (en) 2010-09-21 2015-01-27 Microsoft Corporation Opacity filter for display device
US8884984B2 (en) * 2010-10-15 2014-11-11 Microsoft Corporation Fusing virtual content into real content
US20120113223A1 (en) 2010-11-05 2012-05-10 Microsoft Corporation User Interaction in Augmented Reality
US20120306850A1 (en) 2011-06-02 2012-12-06 Microsoft Corporation Distributed asynchronous localization and mapping for augmented reality
US8929612B2 (en) 2011-06-06 2015-01-06 Microsoft Corporation System for recognizing an open or closed hand
WO2013028908A1 (en) 2011-08-24 2013-02-28 Microsoft Corporation Touch and social cues as inputs into a computer
CA2750287C (en) 2011-08-29 2012-07-03 Microsoft Corporation Gaze detection in a see-through, near-eye, mixed reality display
US8611015B2 (en) 2011-11-22 2013-12-17 Google Inc. User interface
US9229231B2 (en) 2011-12-07 2016-01-05 Microsoft Technology Licensing, Llc Updating printed content with personalized virtual data
US20130326364A1 (en) * 2012-05-31 2013-12-05 Stephen G. Latta Position relative hologram interactions
US20130335405A1 (en) 2012-06-18 2013-12-19 Michael J. Scavezze Virtual object generation within a virtual environment
US9417692B2 (en) 2012-06-29 2016-08-16 Microsoft Technology Licensing, Llc Deep augmented reality tags for mixed reality
US20140168261A1 (en) 2012-12-13 2014-06-19 Jeffrey N. Margolis Direct interaction system mixed reality environments
EP3008549B1 (en) * 2013-06-09 2021-03-17 Sony Interactive Entertainment Inc. Head mounted display
US9329682B2 (en) 2013-06-18 2016-05-03 Microsoft Technology Licensing, Llc Multi-step virtual object selection
US10175483B2 (en) 2013-06-18 2019-01-08 Microsoft Technology Licensing, Llc Hybrid world/body locked HUD on an HMD
US10955665B2 (en) 2013-06-18 2021-03-23 Microsoft Technology Licensing, Llc Concurrent optimal viewing of virtual objects
US9824498B2 (en) * 2014-12-30 2017-11-21 Sony Interactive Entertainment Inc. Scanning display system in head-mounted display for virtual reality

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102591449A (zh) * 2010-10-27 2012-07-18 微软公司 虚拟内容和现实内容的低等待时间的融合
CN102540464A (zh) * 2010-11-18 2012-07-04 微软公司 提供环绕视频的头戴式显示设备
CN102346660A (zh) * 2011-10-13 2012-02-08 苏州梦想人软件科技有限公司 一种混合现实与虚拟物体的系统及方法
CN103847640A (zh) * 2012-12-05 2014-06-11 现代自动车株式会社 提供增强现实的方法和设备

Also Published As

Publication number Publication date
WO2016118371A1 (en) 2016-07-28
JP2018511098A (ja) 2018-04-19
JP6860488B2 (ja) 2021-04-14
US20160210783A1 (en) 2016-07-21
US9846968B2 (en) 2017-12-19
CN107408314A (zh) 2017-11-28
EP3248176A1 (en) 2017-11-29

Similar Documents

Publication Publication Date Title
CN107408314B (zh) 混合现实系统
EP3000020B1 (en) Hologram anchoring and dynamic positioning
US9710973B2 (en) Low-latency fusing of virtual and real content
US10269139B2 (en) Computer program, head-mounted display device, and calibration method
US20140168261A1 (en) Direct interaction system mixed reality environments
US10620779B2 (en) Navigating a holographic image
US20140176591A1 (en) Low-latency fusing of color image data
US20160210780A1 (en) Applying real world scale to virtual content
US12299933B2 (en) Collaborative augmented reality eyewear with ego motion alignment
US20130326364A1 (en) Position relative hologram interactions
US20160131902A1 (en) System for automatic eye tracking calibration of head mounted display device
CN105393158A (zh) 共享的和私有的全息物体
HK1171532B (en) Low-latency fusing of virtual and real content

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant
CF01 Termination of patent right due to non-payment of annual fee
CF01 Termination of patent right due to non-payment of annual fee

Granted publication date: 20220125