WO2019017818A1 - Système de réalité virtuelle basé sur un téléphone intelligent et un miroir incliné - Google Patents

Système de réalité virtuelle basé sur un téléphone intelligent et un miroir incliné Download PDF

Info

Publication number
WO2019017818A1
WO2019017818A1 PCT/RU2018/000452 RU2018000452W WO2019017818A1 WO 2019017818 A1 WO2019017818 A1 WO 2019017818A1 RU 2018000452 W RU2018000452 W RU 2018000452W WO 2019017818 A1 WO2019017818 A1 WO 2019017818A1
Authority
WO
WIPO (PCT)
Prior art keywords
camera
marker
smartphone
user
virtual reality
Prior art date
Application number
PCT/RU2018/000452
Other languages
English (en)
Inventor
Viktor Sergeevich LEMPITSKY
Original Assignee
Autonomous Non-Profit Organization For Higher Education "Skolkovo Institute Of Science And Technology"
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Autonomous Non-Profit Organization For Higher Education "Skolkovo Institute Of Science And Technology" filed Critical Autonomous Non-Profit Organization For Higher Education "Skolkovo Institute Of Science And Technology"
Priority to EA201992443A priority Critical patent/EA038022B1/ru
Publication of WO2019017818A1 publication Critical patent/WO2019017818A1/fr

Links

Classifications

    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G06F3/012—Head tracking input arrangements
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06K—GRAPHICAL DATA READING; PRESENTATION OF DATA; RECORD CARRIERS; HANDLING RECORD CARRIERS
    • G06K1/00—Methods or arrangements for marking the record carrier in digital fashion
    • G06K1/12—Methods or arrangements for marking the record carrier in digital fashion otherwise than by punching
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T15/00—3D [Three Dimensional] image rendering
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating 3D models or images for computer graphics
    • G06T19/006—Mixed reality
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/10—Image acquisition

Definitions

  • the invention relates to systems for mobile virtual reality and specifically to systems for mobile virtual reality performing six-degrees-of-freedom tracking of the user position using a smartphone camera as the only imaging device.
  • the virtual reality (VR) systems are in the state of active development.
  • the VR systems based on smartphones (so called “mobile VR") show particular promise and outnumber all other devices in the market.
  • These systems comprise a smartphone and a low-cost headmount into which the smartphone is inserted.
  • the headmount itself usually does not have any additional sensors or, in fact, any electronic hardware.
  • Mobile VR setups thus enjoy considerably lower cost and complexity compared to standalone systems (as long as the cost of the smartphone is not considered).
  • the main downside of mobile VR systems compared to higher-end VR systems is the lack of positional tracking.
  • mobile VR systems are usually unable to track the headset absolute position and only estimate the angular orientation of the headmount (which is known as three degrees of freedom (3DoF) tracking).
  • 3DoF degrees of freedom
  • higher end systems such as HTC Vive or Occulus Rift use external cameras for positional tracking of the headmount and therefore are able to track all six degrees of freedom (6D0F) associated with the motion of a body (i.e. a headmount) in 3D space.
  • 6D0F tracking enables much richer and more immersive VR experience, since it allows the user to move naturally in the virtual world.
  • the virtual view observed by the user changes with the head absolute motion in a manner anticipated by the user's brain.
  • marker-based methods enjoy higher robustness, higher accuracy, lower latency, and lower CPU load. This however comes at the cost of the need to embed markers into the environment so that a sufficient number of those remain in the camera field of view at all time. On top of that a complex calibration procedure is often needed before marker-based tracking is possible. Overall, the need to cover the environment with markers and to calibrate them is regarded as too big of an obstacle, and marker-based positional tracking is generally regarded to be not viable commercially, with most efforts concentrated on developing markerless solutions.
  • the system includes a camera mounted on a user and a floor marker.
  • the floor marker may be provided on an unreliable carpet.
  • the floor marker may denote a safe walkable area for the user.
  • a slanted mirror may be mounted in front of the camera so that the view of the camera is redirected downwards.
  • the camera may be a smartphone camera.
  • the slanted mirror may be attached to a headmount or a smartphone.
  • the marker may include a grid of dark square-shaped patterns on white background (or vice versa).
  • system is configured to track user hand positions and reproduce the user hand positions in virtual reality. Also, the system is configured to detect gestures using the camera, wherein the detected gestures are used for a gesture-based interface.
  • Fig. 1 shows a general view of the mobile VR system in one embodiment.
  • Fig. 2 shows a view of a floor marker as seen by the camera of the system of Fig. 1 after reflecting from a slanted mirror.
  • the new marker-based positional tracking system claimed herein avoids the major drawbacks of other marker-based solutions by using a single floor marker (that is called a magic carpet).
  • the marker can be easily “deployed” by simply unrolling the carpet.
  • the smartphone camera is oriented in a horizontal direction that is roughly parallel to the ground, and therefore would not be able to observe the marker (the carpet). Therefore, a slanted mirror may be mounted in front of the smartphone camera so that the view of the camera is redirected downwards. In practice, the mirror can be attached to either the smartphone or to the headmount. After such modification a camera is able to observe parts of the marker as long as the user stays on the carpet and looks roughly horizontally.
  • the system has the following hardware components: a floor marker 1 ("magic carpet"), a smartphone 2, a VR headmount 3, into which the smartphone is inserted, and a slanted mirror 4, in this embodiment attached to the smartphone in front of the smartphone camera. While the user is facing in the horizontal direction, the camera observes the marker thanks to the reflection in the attached mirror.
  • a floor marker 1 (“magic carpet")
  • a smartphone 2 a VR headmount 3 into which the smartphone is inserted
  • a slanted mirror 4 in this embodiment attached to the smartphone in front of the smartphone camera. While the user is facing in the horizontal direction, the camera observes the marker thanks to the reflection in the attached mirror.
  • Computer vision algorithms can then be used to estimate the position of the camera with respect to the marker. This can be done efficiently (quickly and in robust way) as long as the appearance of the marker is optimized for part localization and identification. Marker-based pose estimation is a well researched area, and popular marker families along with associated tracking algorithms are available (e.g. AprilTags or ArucoMarkers).
  • the camera of the smartphone can see the floor marker 1 reflected from the slanted mirror.
  • the floor marker may be composed of square- shaped distinguishable parts. Learnable marker technology described in Grinchuk et al. NIPS2016 may be used to design aesthetically-pleasant and distinguishable grid elements.
  • the frame from the smartphone camera is matched to the marker by detecting dark quadrangles and matching them to the marker parts.
  • adaptive thresholding followed by connected component analysis identifies quadrangular-shaped dark regions. Each quadrangular region is then transformed geometrically into a square and its appearance is then matched against the square patterns that are known to be on the marker using a simple pixel-based matching measure. This allows to establish correspondences between detected regions and parts of the marker.
  • a standard pose estimation algorithms are then used to estimate the location of the camera with reference to the magic carpet.
  • a pose estimation algorithm would estimate the 6DoF position of the virtual camera that is obtained by reflecting the position of the real smartphone camera in the mounted mirror.
  • the virtual camera can be reflected back and the position of the real camera with respect to the marker (carpet) can be estimated, thus accomplishing positional tracking.
  • the camera view also clearly shows the pose of the user hands, which can be reproduced in virtual reality or to design gesture-based interfaces. As the camera in the proposed system observes the user hands most of the time, the reflected view can be used to track the user hand positions and to reproduce them in virtual reality.
  • the image stream from the camera can be used to detect certain gestures, which can be used for gesture-based interfaces.
  • gesture-based interfaces No known systems that can perform hand tracking without the use of external imaging devices have been suggested previously. Note that such interfaces can be devised even in the absence of the floor marker (carpet).
  • the proposed system comprises the following hardware components: a smartphone, a headset with the slanted mirror (alternatively the mirror can be attached to the smartphone directly), and a marker in the form of a floor carpet.
  • the system includes a pose estimation method that is able to detect and match parts of the marker in the reflected image, and then use pose estimation to estimate six-DoF camera position.
  • the system according to the invention enjoys the benefits of the marker-based solutions. Knowing the marker in advance allows us to use simpler, more efficient, more precise, more robust algorithms for position estimation compared to markerless technologies that have to build a map of the environment on the fly (using so-called simultaneous localization and mapping (SLAM) algorithms).
  • SLAM simultaneous localization and mapping
  • the system enables positional tracking as long as the user walks over the carpet.
  • a standard VR interface (chaperoning) can then signal to the user as he/she is leaving the carpet as is done in the high-end VR systems that allow the user to walk.
  • the system according to the invention allows to observe the body of the user using the smartphone camera. This can be used to augment the VR experience with a coarse representation of the user body thus improving the sense of perception. Furthermore this can enable gesture- based interfaces without additional controllers. • The system is completely inside the headset and does not need any wired connection to other devices, so the user can safely move around a carpet.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Computer Graphics (AREA)
  • Software Systems (AREA)
  • Computer Hardware Design (AREA)
  • Human Computer Interaction (AREA)
  • Optics & Photonics (AREA)
  • Multimedia (AREA)
  • User Interface Of Digital Computer (AREA)
  • Position Input By Displaying (AREA)
  • Studio Devices (AREA)

Abstract

L'invention concerne des systèmes de réalité virtuelle mobiles et spécifiquement des systèmes de réalité virtuelle mobiles effectuant un suivi à six degrés de liberté de la position de l'utilisateur à l'aide d'une caméra de téléphone intelligent en tant que seul dispositif d'imagerie. L'invention concerne un système de VR à base marquée dans lequel des marqueurs peuvent être facilement déployés et une procédure d'étalonnage peut être effectuée d'une manière moins complexe. L'invention concerne également un système de suivi de position basé sur un marqueur pour réalité virtuelle. Le système comprend une caméra montée sur un utilisateur; et un marqueur de sol. Le marqueur de sol peut être fourni sur un tapis non enroulable et désigner une zone sans danger pour l'utilisateur. Un miroir incliné peut être monté devant la caméra de sorte que la vue de la caméra soit redirigée vers le bas. En outre, la caméra peut être une caméra de téléphone intelligent.
PCT/RU2018/000452 2017-07-19 2018-07-10 Système de réalité virtuelle basé sur un téléphone intelligent et un miroir incliné WO2019017818A1 (fr)

Priority Applications (1)

Application Number Priority Date Filing Date Title
EA201992443A EA038022B1 (ru) 2017-07-19 2018-07-10 Система виртуальной реальности на основе смартфона и наклонного зеркала

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
RU2017125876 2017-07-19
RU2017125876A RU2686029C2 (ru) 2017-07-19 2017-07-19 Система виртуальной реальности на основе смартфона и наклонного зеркала

Publications (1)

Publication Number Publication Date
WO2019017818A1 true WO2019017818A1 (fr) 2019-01-24

Family

ID=63490656

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/RU2018/000452 WO2019017818A1 (fr) 2017-07-19 2018-07-10 Système de réalité virtuelle basé sur un téléphone intelligent et un miroir incliné

Country Status (3)

Country Link
EA (1) EA038022B1 (fr)
RU (1) RU2686029C2 (fr)
WO (1) WO2019017818A1 (fr)

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP2955610A1 (fr) * 2014-06-11 2015-12-16 Samsung Electronics Co., Ltd Procédé de contrôle de fonctionnement avec un visiocasque
WO2016187474A1 (fr) * 2015-05-20 2016-11-24 Daqri, Llc Dispositif d'affichage acoustico-optique pour réalité augmentée
WO2017014733A1 (fr) * 2015-07-17 2017-01-26 Ivd Mining Apprentissage de réalité virtuelle

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20080252746A1 (en) * 2007-04-13 2008-10-16 Newton Eliot Mack Method and apparatus for a hybrid wide area tracking system
GB0804274D0 (en) * 2008-03-07 2008-04-16 Virtually Live Ltd A media sysyem and method
US8923622B2 (en) * 2012-12-10 2014-12-30 Symbol Technologies, Inc. Orientation compensation using a mobile device camera and a reference marker
JP6641122B2 (ja) * 2015-08-27 2020-02-05 キヤノン株式会社 表示装置及び情報処理装置及びその制御方法
RU2617557C1 (ru) * 2015-11-18 2017-04-25 Виталий Витальевич Аверьянов Способ воздействия на виртуальные объекты дополненной реальности

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP2955610A1 (fr) * 2014-06-11 2015-12-16 Samsung Electronics Co., Ltd Procédé de contrôle de fonctionnement avec un visiocasque
WO2016187474A1 (fr) * 2015-05-20 2016-11-24 Daqri, Llc Dispositif d'affichage acoustico-optique pour réalité augmentée
WO2017014733A1 (fr) * 2015-07-17 2017-01-26 Ivd Mining Apprentissage de réalité virtuelle

Also Published As

Publication number Publication date
RU2017125876A (ru) 2019-01-22
RU2686029C2 (ru) 2019-04-23
EA201992443A1 (ru) 2020-02-18
EA038022B1 (ru) 2021-06-24
RU2017125876A3 (fr) 2019-01-22

Similar Documents

Publication Publication Date Title
US11796309B2 (en) Information processing apparatus, information processing method, and recording medium
CN111465886B (zh) 头戴式显示器的选择性跟踪
JP6110573B2 (ja) 仮想現実システムの較正
US9524021B2 (en) Imaging surround system for touch-free display control
US9625994B2 (en) Multi-camera depth imaging
WO2017172984A3 (fr) Casque de réalité virtuelle avec dispositif de suivi de tête à mouvement relatif
WO2014156706A1 (fr) Dispositif et méthode de traitement d'image, et programme
WO2014071254A4 (fr) Dispositif informatique et de commande de type montre sans fil et procédé pour imagerie en 3d, cartographie, réseau social et interfaçage
KR20160003233A (ko) 컴퓨터 비전 애플리케이션 초기화를 용이하게 하는 방법들
US10140507B2 (en) Apparatus and method for recognizing hand gestures in a virtual reality headset
US20170206666A1 (en) Information processing apparatus, information processing method, and program
EP3188075B1 (fr) Appareil et procédé permettant de reconnaître les gestes de la main dans un casque de réalité virtuelle
JP2015536507A (ja) 対象とのオブジェクトの関連付け
JP2011189066A (ja) ゲーム装置、ゲーム装置の制御方法、及びプログラム
JP2021060627A (ja) 情報処理装置、情報処理方法、およびプログラム
US10859831B1 (en) Systems and methods for safely operating a mobile virtual reality system
US20240054780A1 (en) Object detections for virtual reality
US9773350B1 (en) Systems and methods for greater than 360 degree capture for virtual reality
JP2023502239A (ja) 広視野角のステレオカメラ装置およびこれを利用した深度画像処理方法
KR20140009900A (ko) 로봇 제어 시스템 및 그 동작 방법
US11073902B1 (en) Using skeletal position to predict virtual boundary activation
US20210134039A1 (en) Using a handheld device to recreate a human pose or align an object in an augmented reality or virtual reality environment
WO2019017818A1 (fr) Système de réalité virtuelle basé sur un téléphone intelligent et un miroir incliné
US20140043327A1 (en) Method and system for superimposing content to have a fixed pose
Carozza et al. Robust 6-DOF immersive navigation using commodity hardware

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 18765210

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 18765210

Country of ref document: EP

Kind code of ref document: A1