JP6860488B2 - 複合現実システム - Google Patents

複合現実システム Download PDF

Info

Publication number
JP6860488B2
JP6860488B2 JP2017538330A JP2017538330A JP6860488B2 JP 6860488 B2 JP6860488 B2 JP 6860488B2 JP 2017538330 A JP2017538330 A JP 2017538330A JP 2017538330 A JP2017538330 A JP 2017538330A JP 6860488 B2 JP6860488 B2 JP 6860488B2
Authority
JP
Japan
Prior art keywords
image
camera
camera assembly
head
display device
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Expired - Fee Related
Application number
JP2017538330A
Other languages
English (en)
Japanese (ja)
Other versions
JP2018511098A (ja
JP2018511098A5 (https=
Inventor
チャールズ トムリン，アーサー
チャールズ トムリン，アーサー
マイケル キーブラー，エヴァン
マイケル キーブラー，エヴァン
ガーバス ファイト，ニコラス
ガーバス ファイト，ニコラス
ジェイ． マウント，ブライアン
ジェイ． マウント，ブライアン
ローウェル アルト，グレゴリー
ローウェル アルト，グレゴリー
トサール，ジョージ
マイケル ライオンズ，ジョナサン
マイケル ライオンズ，ジョナサン
ジェイ． アンブラス，アンソニー
ジェイ． アンブラス，アンソニー
クイン エグバート，キャメロン
クイン エグバート，キャメロン
グイマン，ウィル
ダブリュー． マクグリン，ジェフ
ダブリュー． マクグリン，ジェフ
ハンス，ジェレミー
セバスチャン−ケヴィン シルヴァン，ロジャー
セバスチャン−ケヴィン シルヴァン，ロジャー
ゲオルク プファッフェ，アレクサンダー
ゲオルク プファッフェ，アレクサンダー
クロイマン，ダン
アンドリュー サルトウェル，エリック
アンドリュー サルトウェル，エリック
ワード，クリス
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Microsoft Corp
Microsoft Technology Licensing LLC
Original Assignee
Microsoft Corp
Microsoft Technology Licensing LLC
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Microsoft Corp, Microsoft Technology Licensing LLC filed Critical Microsoft Corp
Publication of JP2018511098A publication Critical patent/JP2018511098A/ja
Publication of JP2018511098A5 publication Critical patent/JP2018511098A5/ja
Application granted granted Critical
Publication of JP6860488B2 publication Critical patent/JP6860488B2/ja
Expired - Fee Related legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating three-dimensional [3D] models or images for computer graphics
    • G06T19/006—Mixed reality
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/0093—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/017—Head mounted
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/50—Constructional details
    • H04N23/51—Housings
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/0101—Head-up displays characterised by optical features
    • G02B2027/0138—Head-up displays characterised by optical features comprising image capture systems, e.g. camera
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/017—Head mounted
    • G02B2027/0178—Eyeglass type

Landscapes

  • Physics & Mathematics (AREA)
  • Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Optics & Photonics (AREA)
  • General Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Graphics (AREA)
  • Computer Hardware Design (AREA)
  • Software Systems (AREA)
  • Human Computer Interaction (AREA)
  • Signal Processing (AREA)
  • Multimedia (AREA)
  • User Interface Of Digital Computer (AREA)
  • Controls And Circuits For Display Device (AREA)
  • Processing Or Creating Images (AREA)
  • Studio Devices (AREA)
JP2017538330A 2015-01-20 2016-01-13 複合現実システム Expired - Fee Related JP6860488B2 (ja)

Applications Claiming Priority (5)

Application Number Priority Date Filing Date Title
US201562105701P 2015-01-20 2015-01-20
US62/105,701 2015-01-20
US14/728,650 2015-06-02
US14/728,650 US9846968B2 (en) 2015-01-20 2015-06-02 Holographic bird's eye view camera
PCT/US2016/013127 WO2016118371A1 (en) 2015-01-20 2016-01-13 Mixed reality system

Publications (3)

Publication Number Publication Date
JP2018511098A JP2018511098A (ja) 2018-04-19
JP2018511098A5 JP2018511098A5 (https=) 2019-01-31
JP6860488B2 true JP6860488B2 (ja) 2021-04-14

Family

ID=56408228

Family Applications (1)

Application Number Title Priority Date Filing Date
JP2017538330A Expired - Fee Related JP6860488B2 (ja) 2015-01-20 2016-01-13 複合現実システム

Country Status (5)

Country Link
US (1) US9846968B2 (https=)
EP (1) EP3248176A1 (https=)
JP (1) JP6860488B2 (https=)
CN (1) CN107408314B (https=)
WO (1) WO2016118371A1 (https=)

Families Citing this family (62)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9986225B2 (en) * 2014-02-14 2018-05-29 Autodesk, Inc. Techniques for cut-away stereo content in a stereoscopic display
AU2016225963B2 (en) 2015-03-05 2021-05-13 Magic Leap, Inc. Systems and methods for augmented reality
US10180734B2 (en) 2015-03-05 2019-01-15 Magic Leap, Inc. Systems and methods for augmented reality
US10838207B2 (en) 2015-03-05 2020-11-17 Magic Leap, Inc. Systems and methods for augmented reality
US10133532B2 (en) * 2015-09-25 2018-11-20 Seiko Epson Corporation Display system, display device, information display method, and program
US10909711B2 (en) 2015-12-04 2021-02-02 Magic Leap, Inc. Relocalization systems and methods
WO2017143239A1 (en) * 2016-02-18 2017-08-24 Edx Wireless, Inc. Systems and methods for augmented reality representations of networks
WO2017159063A1 (ja) * 2016-03-14 2017-09-21 ソニー株式会社 表示装置並びに情報処理端末装置
US10210661B2 (en) * 2016-04-25 2019-02-19 Microsoft Technology Licensing, Llc Location-based holographic experience
US10649211B2 (en) * 2016-08-02 2020-05-12 Magic Leap, Inc. Fixed-distance virtual and augmented reality systems and methods
JP2018081543A (ja) * 2016-11-17 2018-05-24 セイコーエプソン株式会社 表示装置、頭部装着型表示装置、表示装置の制御方法、及び、プログラム
KR102497707B1 (ko) * 2016-12-13 2023-02-08 매직 립, 인코포레이티드 검출된 특징들을 사용한 3d 객체 렌더링
JP6849430B2 (ja) * 2016-12-27 2021-03-24 キヤノン株式会社 画像処理装置、画像処理方法、及びプログラム
US10812936B2 (en) 2017-01-23 2020-10-20 Magic Leap, Inc. Localization determination for mixed reality systems
JP6965896B2 (ja) * 2017-01-31 2021-11-10 株式会社ニコン 表示制御システム、及び、表示制御方法
US10628950B2 (en) * 2017-03-01 2020-04-21 Microsoft Technology Licensing, Llc Multi-spectrum illumination-and-sensor module for head tracking, gesture recognition and spatial mapping
AU2018236457B2 (en) 2017-03-17 2021-04-29 Magic Leap, Inc. Mixed reality system with virtual content warping and method of generating virtual content using same
WO2018170482A1 (en) 2017-03-17 2018-09-20 Magic Leap, Inc. Mixed reality system with color virtual content warping and method of generating virtual content using same
WO2018170490A1 (en) 2017-03-17 2018-09-20 Magic Leap, Inc. Technique for recording augmented reality data
KR102359978B1 (ko) 2017-03-17 2022-02-07 매직 립, 인코포레이티드 다중-소스 가상 콘텐츠 합성을 갖는 혼합 현실 시스템 및 이를 사용하여 가상 콘텐츠를 생성하는 방법
US11861255B1 (en) 2017-06-16 2024-01-02 Apple Inc. Wearable device for facilitating enhanced interaction
US12498895B2 (en) 2017-06-16 2025-12-16 Apple Inc. Head-mounted device with publicly viewable display
WO2019015261A1 (en) * 2017-07-17 2019-01-24 Chengdu Topplusvision Technology Co., Ltd. DEVICES AND METHODS FOR SCENE DETERMINATION
CN107976811B (zh) * 2017-12-25 2023-12-29 河南诺控信息技术有限公司 一种基于虚拟现实混合的仿真实验室及其仿真方法
CN108235778B (zh) * 2018-01-09 2021-09-10 达闼机器人有限公司 基于云端计算的标定方法、装置、电子设备和计算机程序产品
JP2019152899A (ja) * 2018-02-28 2019-09-12 株式会社バンダイナムコスタジオ シミュレーションシステム及びプログラム
WO2019204161A1 (en) 2018-04-20 2019-10-24 Pcms Holdings, Inc. Method and system for gaze-based control of mixed reality content
JP6688337B2 (ja) * 2018-06-06 2020-04-28 任天堂株式会社 情報処理プログラム、情報処理システム、情報処理装置および情報処理方法
US10964053B2 (en) * 2018-07-02 2021-03-30 Microsoft Technology Licensing, Llc Device pose estimation using 3D line clouds
GB201811249D0 (en) * 2018-07-09 2018-08-29 Digitalbridge System and method for virtual image alignment
WO2020023523A1 (en) 2018-07-23 2020-01-30 Magic Leap, Inc. Intra-field sub code timing in field sequential displays
US11379948B2 (en) 2018-07-23 2022-07-05 Magic Leap, Inc. Mixed reality system with virtual content warping and method of generating virtual content using same
US10839556B2 (en) 2018-10-23 2020-11-17 Microsoft Technology Licensing, Llc Camera pose estimation using obfuscated features
US11902677B2 (en) 2018-11-12 2024-02-13 Magic Leap, Inc. Patch tracking image sensor
US20220051441A1 (en) * 2018-12-21 2022-02-17 Magic Leap, Inc. Multi-camera cross reality device
JP7720783B2 (ja) 2019-02-07 2025-08-08 マジック リープ， インコーポレイテッド 受動深度抽出を伴う軽量のクロスリアリティデバイス
EP3922011A4 (en) 2019-02-07 2022-10-26 Magic Leap, Inc. LIGHTWEIGHT, LOW POWER, HIGH TIME RESOLUTION VIRTUAL REALITY DEVICE
US11372474B2 (en) * 2019-07-03 2022-06-28 Saec/Kinetic Vision, Inc. Systems and methods for virtual artificial intelligence development and testing
CN110456905A (zh) * 2019-07-23 2019-11-15 广东虚拟现实科技有限公司 定位跟踪方法、装置、系统及电子设备
JP7385733B2 (ja) * 2019-09-09 2023-11-22 アップル インコーポレイテッド 仮想カメラと物理的カメラとの位置同期
CN111260793B (zh) * 2020-01-10 2020-11-24 中国电子科技集团公司第三十八研究所 面向增强和混合现实的远程虚实高精度匹配定位的方法
US11288877B2 (en) 2020-01-10 2022-03-29 38th Research Institute, China Electronics Technology Group Corp. Method for matching a virtual scene of a remote scene with a real scene for augmented reality and mixed reality
JP2023527357A (ja) * 2020-05-29 2023-06-28 マジック リープ， インコーポレイテッド 角加速の決定
US11475242B2 (en) * 2020-07-27 2022-10-18 Seiko Epson Corporation Domain adaptation losses
CN116981978A (zh) * 2020-09-21 2023-10-31 苹果公司 用于动态确定呈现和过渡区域的方法和设备
CN112669671B (zh) * 2020-12-28 2022-10-25 北京航空航天大学江西研究院 一种基于实物交互的混合现实飞行仿真系统
US20230260240A1 (en) * 2021-03-11 2023-08-17 Quintar, Inc. Alignment of 3d graphics extending beyond frame in augmented reality system with remote presentation
US12542946B2 (en) 2021-03-11 2026-02-03 Quintar, Inc. Remote presentation with augmented reality content synchronized with separately displayed video content
US11645819B2 (en) 2021-03-11 2023-05-09 Quintar, Inc. Augmented reality system for viewing an event with mode based on crowd sourced images
US12003806B2 (en) * 2021-03-11 2024-06-04 Quintar, Inc. Augmented reality system for viewing an event with multiple coordinate systems and automatically generated model
US11527047B2 (en) 2021-03-11 2022-12-13 Quintar, Inc. Augmented reality system for viewing an event with distributed computing
US12244782B2 (en) 2021-03-11 2025-03-04 Quintar, Inc. Augmented reality system for remote presentation for viewing an event
US12159359B2 (en) 2021-03-11 2024-12-03 Quintar, Inc. Use of multiple registrations for augmented reality system for viewing an event
US11657578B2 (en) 2021-03-11 2023-05-23 Quintar, Inc. Registration for augmented reality system for viewing an event
US12028507B2 (en) 2021-03-11 2024-07-02 Quintar, Inc. Augmented reality system with remote presentation including 3D graphics extending beyond frame
US12586305B2 (en) 2021-03-11 2026-03-24 Quintar, Inc. 3D reference point detection for survey for venue model construction
US12482192B2 (en) * 2021-11-03 2025-11-25 The Regents Of The University California Collaborative mixed-reality system for immersive surgical telementoring
US12390104B2 (en) * 2022-05-25 2025-08-19 Microsoft Technology Licensing, Llc Enhanced eye tracking using lens inverse transform
US12217372B2 (en) 2022-10-17 2025-02-04 T-Mobile Usa, Inc. Generating mixed reality content based on data from a wireless device
CN118259740A (zh) * 2022-12-26 2024-06-28 北京字跳网络技术有限公司 交互界面显示跟随方法和装置、电子设备、介质、芯片和程序产品
JP2024092551A (ja) * 2022-12-26 2024-07-08 国立研究開発法人情報通信研究機構 仮想物体操作装置およびそのプログラム、ならびに、仮想物体表示システム
TWI897136B (zh) * 2023-12-07 2025-09-11 宏達國際電子股份有限公司 虛擬影像顯示裝置以及立體影像的建立方法

Family Cites Families (33)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6711293B1 (en) 1999-03-08 2004-03-23 The University Of British Columbia Method and apparatus for identifying scale invariant features in an image and use of same for locating an object in an image
JP3363861B2 (ja) * 2000-01-13 2003-01-08 キヤノン株式会社 複合現実感提示装置及び複合現実感提示方法並びに記憶媒体
US7015831B2 (en) 2002-12-17 2006-03-21 Evolution Robotics, Inc. Systems and methods for incrementally updating a pose of a mobile device calculated by visual simultaneous localization and mapping techniques
US7401920B1 (en) 2003-05-20 2008-07-22 Elbit Systems Ltd. Head mounted eye tracking and display system
IL157837A (en) 2003-09-10 2012-12-31 Yaakov Amitai Substrate-guided optical device particularly for three-dimensional displays
KR101019569B1 (ko) 2005-08-29 2011-03-08 에브릭스 테크놀로지스, 인코포레이티드 모바일 이미지 인식을 통한 상호작용
US8160400B2 (en) 2005-11-17 2012-04-17 Microsoft Corporation Navigating images using image based geometric alignment and object based controls
KR100809479B1 (ko) 2006-07-27 2008-03-03 한국전자통신연구원 혼합 현실 환경을 위한 얼굴 착용형 디스플레이 장치
EP2193825B1 (en) 2008-12-03 2017-03-22 Alcatel Lucent Mobile device for augmented reality applications
KR20100138700A (ko) 2009-06-25 2010-12-31 삼성전자주식회사 가상 세계 처리 장치 및 방법
US20110317871A1 (en) 2010-06-29 2011-12-29 Microsoft Corporation Skeletal joint recognition and tracking system
US8941559B2 (en) 2010-09-21 2015-01-27 Microsoft Corporation Opacity filter for display device
US8884984B2 (en) * 2010-10-15 2014-11-11 Microsoft Corporation Fusing virtual content into real content
US9348141B2 (en) * 2010-10-27 2016-05-24 Microsoft Technology Licensing, Llc Low-latency fusing of virtual and real content
US20120113223A1 (en) 2010-11-05 2012-05-10 Microsoft Corporation User Interaction in Augmented Reality
US8576276B2 (en) * 2010-11-18 2013-11-05 Microsoft Corporation Head-mounted display device which provides surround video
US20120306850A1 (en) 2011-06-02 2012-12-06 Microsoft Corporation Distributed asynchronous localization and mapping for augmented reality
US8929612B2 (en) 2011-06-06 2015-01-06 Microsoft Corporation System for recognizing an open or closed hand
WO2013028908A1 (en) 2011-08-24 2013-02-28 Microsoft Corporation Touch and social cues as inputs into a computer
CA2750287C (en) 2011-08-29 2012-07-03 Microsoft Corporation Gaze detection in a see-through, near-eye, mixed reality display
CN102346660A (zh) * 2011-10-13 2012-02-08 苏州梦想人软件科技有限公司 一种混合现实与虚拟物体的系统及方法
US8611015B2 (en) 2011-11-22 2013-12-17 Google Inc. User interface
US9229231B2 (en) 2011-12-07 2016-01-05 Microsoft Technology Licensing, Llc Updating printed content with personalized virtual data
US20130326364A1 (en) * 2012-05-31 2013-12-05 Stephen G. Latta Position relative hologram interactions
US20130335405A1 (en) 2012-06-18 2013-12-19 Michael J. Scavezze Virtual object generation within a virtual environment
US9417692B2 (en) 2012-06-29 2016-08-16 Microsoft Technology Licensing, Llc Deep augmented reality tags for mixed reality
KR101360061B1 (ko) * 2012-12-05 2014-02-12 현대자동차 주식회사 증강 현실 제공 방법 및 그 장치
US20140168261A1 (en) 2012-12-13 2014-06-19 Jeffrey N. Margolis Direct interaction system mixed reality environments
EP3008549B1 (en) * 2013-06-09 2021-03-17 Sony Interactive Entertainment Inc. Head mounted display
US9329682B2 (en) 2013-06-18 2016-05-03 Microsoft Technology Licensing, Llc Multi-step virtual object selection
US10175483B2 (en) 2013-06-18 2019-01-08 Microsoft Technology Licensing, Llc Hybrid world/body locked HUD on an HMD
US10955665B2 (en) 2013-06-18 2021-03-23 Microsoft Technology Licensing, Llc Concurrent optimal viewing of virtual objects
US9824498B2 (en) * 2014-12-30 2017-11-21 Sony Interactive Entertainment Inc. Scanning display system in head-mounted display for virtual reality

Also Published As

Publication number Publication date
WO2016118371A1 (en) 2016-07-28
JP2018511098A (ja) 2018-04-19
US20160210783A1 (en) 2016-07-21
US9846968B2 (en) 2017-12-19
CN107408314A (zh) 2017-11-28
EP3248176A1 (en) 2017-11-29
CN107408314B (zh) 2022-01-25

Similar Documents

Publication Publication Date Title
JP6860488B2 (ja) 複合現実システム
EP3000020B1 (en) Hologram anchoring and dynamic positioning
EP3467585B1 (en) Head-mounted display tracking system
US10740971B2 (en) Augmented reality field of view object follower
US9710973B2 (en) Low-latency fusing of virtual and real content
US10269139B2 (en) Computer program, head-mounted display device, and calibration method
CN107209950B (zh) 从现实世界材料自动生成虚拟材料
EP3011411B1 (en) Hybrid world/body locked hud on an hmd
US20140168261A1 (en) Direct interaction system mixed reality environments
US20130326364A1 (en) Position relative hologram interactions
US20140176591A1 (en) Low-latency fusing of color image data
US20160210780A1 (en) Applying real world scale to virtual content
CN110377148B (zh) 计算机可读介质、训练对象检测算法的方法及训练设备
HK1171532B (en) Low-latency fusing of virtual and real content

Legal Events

Date Code Title Description
A529 Written submission of copy of amendment under article 34 pct

Free format text: JAPANESE INTERMEDIATE CODE: A529

Effective date: 20170915

A521 Request for written amendment filed

Free format text: JAPANESE INTERMEDIATE CODE: A523

Effective date: 20181213

A621 Written request for application examination

Free format text: JAPANESE INTERMEDIATE CODE: A621

Effective date: 20181213

A977 Report on retrieval

Free format text: JAPANESE INTERMEDIATE CODE: A971007

Effective date: 20200120

A131 Notification of reasons for refusal

Free format text: JAPANESE INTERMEDIATE CODE: A131

Effective date: 20200128

A521 Request for written amendment filed

Free format text: JAPANESE INTERMEDIATE CODE: A523

Effective date: 20200420

A131 Notification of reasons for refusal

Free format text: JAPANESE INTERMEDIATE CODE: A131

Effective date: 20200930

A521 Request for written amendment filed

Free format text: JAPANESE INTERMEDIATE CODE: A523

Effective date: 20201223

TRDD Decision of grant or rejection written
A01 Written decision to grant a patent or to grant a registration (utility model)

Free format text: JAPANESE INTERMEDIATE CODE: A01

Effective date: 20210305

A61 First payment of annual fees (during grant procedure)

Free format text: JAPANESE INTERMEDIATE CODE: A61

Effective date: 20210326

R150 Certificate of patent or registration of utility model

Ref document number: 6860488

Country of ref document: JP

Free format text: JAPANESE INTERMEDIATE CODE: R150

R250 Receipt of annual fees

Free format text: JAPANESE INTERMEDIATE CODE: R250

LAPS Cancellation because of no payment of annual fees