EA038022B1 - Virtual reality system based on a smartphone and a slanted mirror - Google Patents

Abstract

The invention relates to systems for mobile virtual reality and specifically to systems for mobile virtual reality performing six-degrees-of-freedom tracking of the user position using a smartphone camera as the only imaging device. The invention provides a marked-based VR system where markers can be easily deployed and a calibration procedure may be performed in a less complex way. A marker-based positional tracking system for virtual reality is provided. The system includes a camera mounted on a user; and a floor marker. The floor marker may be provided on an unrollable carpet and denote a safe walkable area for the user. A slanted mirror may be mounted in front of the camera so that the view of the camera is redirected downwards. Also, the camera may be a smartphone camera.

Description

Technology area

The present invention relates to mobile virtual reality systems, and in particular to mobile virtual reality systems that track the position of a user with six degrees of freedom using a smartphone camera as the sole imaging device.

State of the art

Virtual reality (VR) systems are currently under active development. Smartphone-based VR systems (called mobile VR) are particularly promising and outnumber all other devices on the market. These systems include a smartphone and an inexpensive helmet holder into which the smartphone is inserted. The helmet holder itself, as a rule, does not contain any additional sensors or any other electronic equipment. This makes mobile VR kits significantly less expensive and less complex than standalone systems (assuming we don't factor in the cost of a smartphone). The main disadvantage of mobile VR systems compared to higher-end VR systems is the lack of position tracking.

In more detail, during a VR session, mobile VR systems generally cannot track the absolute position of the helmet holder and only estimate its angular orientation (which is known as tracking with 3 degrees of freedom). In contrast, higher-end systems such as the HTC Vive or Oculus Rift use external cameras to track the position of the helmet, and thus these systems can track the movement of the body (i.e. helmet) in three-dimensional space with six degrees of freedom. Tracking with six degrees of freedom provides a much more complete and deep immersion in virtual reality, as it allows the user to move naturally in the virtual space. As a result, the virtual picture perceived by the user changes when the absolute position of the head changes, as the user's brain expects.

To implement tracking with six degrees of freedom in mobile VR, a number of experimental sets of mobile VR were developed, which solved the problem of using a smartphone camera to assess the position of a smartphone (and a helmet-holdera, in which it is inserted) with six degrees of freedom. This task (known as pose estimation in machine vision and inside-out tracking in VR) is challenging, especially if it needs to be done at high frame rates and using a mobile device. Existing technologies in this area are divided into two groups: markerless technologies, which do not assume any prior knowledge of the environment in which the user navigates, and marker technologies, in which some visual markers of a certain, known in advance appearance are introduced in a certain way. into the environment.

Of the two groups above, marker methods provide higher reliability, higher accuracy, lower latency, and lower CPU utilization. However, this is achieved at the expense of the need to embed markers into the environment so that a sufficient number of markers are always located in the camera's field of view. In addition, a complex calibration procedure is often required for marker tracking to be possible. In general, the need to implement markers into the environment and their calibration is too much of an obstacle, and therefore marker position tracking is usually considered uncompetitive and most of the efforts of developers are aimed at creating marker-free solutions.

Thus, there is a need for a BP marker system that allows markers to be placed as easily as possible in the surrounding space, and in which the calibration procedure can be performed in the simplest possible way.

Disclosure of invention

To achieve the above goal, a self-positioning marker system for virtual reality is proposed. The system includes a user-mounted camera and a floor marker.

In one possible embodiment, the floor marker can be embedded in the roll-out carpet.

In another embodiment, the floor marker may indicate an area that is safe for the user to move.

In yet another embodiment, an oblique mirror may be positioned in front of the camera so that the camera's field of view is redirected downward.

In addition, a smartphone camera can be used as a camera.

In yet another embodiment, the tilting mirror can be attached to a helmet holder or smartphone.

In one possible embodiment, the marker can be a grid of dark squares on a white background (or vice versa).

In yet another possible embodiment, the system monitors the position of the user's hands and

- 1 038022 reproduces the position of the user's hands in virtual reality. In addition, the camera can recognize gestures with the camera, with the recognized gestures being used as a gesture interface.

Brief Description of Drawings

FIG. 1 shows a general view of the mobile VR system in one of the possible implementations; in fig. 2 is a camera side view of the floor marker of the system shown in FIG. 1 after reflection from an inclined mirror.

Implementation of the invention

The new marker position tracking system stated in this document avoids the main disadvantages inherent in other marker solutions in the field of virtual reality, through the use of a floor marker (called magic carpet). This marker can be easily unrolled by simply rolling out the carpet. During a VR session, the user walks on the carpet, and in the real world, the carpet limits a safe area that should not be obstructed during the VR session. When the user moves, the smartphone camera is oriented horizontally, that is, approximately parallel to the floor surface, and, therefore, the marker (carpet) cannot fall into its field of view. Therefore, an oblique mirror can be installed in front of the smartphone camera so that the camera's field of view is redirected downward. In practice, the mirror can be attached to a smartphone or helmet holder. After such a modification, the marker elements already fall into the field of view of the camera, provided that the user is on the carpet and his gaze is directed approximately horizontally.

As shown in FIG. 1, in one possible embodiment, the system comprises the following hardware components: a floor marker 1 (magic carpet), a smartphone 2, a VR helmet holder 3 into which the said smartphone is inserted, and an inclined mirror 4, in this embodiment, attached to the smartphone in front of the smartphone camera ... When the user looks horizontally, the marker is brought into the camera's field of view by the reflection off the attached mirror.

Computer vision algorithms can then be used to estimate the position of the camera relative to the marker. This can be done efficiently (quickly and reliably) provided that the appearance of the marker is optimized to locate and identify marker elements. Position estimation using marker technology is a well-studied area, and there are popular marker families with associated tracking algorithms (eg AprilTags or ArucoMarkers).

As shown in FIG. 2, the smartphone camera can see the floor marker 1 reflected from the tilting mirror. In one possible embodiment, the floor marker can be composed of highly visible square elements. The teachable marker technology described by Grinchuk et al. NIPS2016 can be used to create aesthetically pleasing and distinguishable mesh elements. The position of the smartphone camera frame is snapped to the marker by detecting dark squares and matching them to marker elements.

During self-positioning, adaptive thresholding and subsequent analysis of the corresponding components reveal the dark areas of the quadrangular shape. Each dark area is geometrically transformed into a square, and then its appearance is matched with the square elements that should be on the marker through a simple pixel-by-pixel comparison. This allows you to establish a correspondence between the detected areas and marker elements.

Standard position estimation algorithms are then used to determine the position of the camera relative to the magic carpet. In more detail, the position estimation algorithm provides a six-degree-of-freedom position estimate of a virtual camera obtained by reflecting the position of a real smartphone camera in a mounted mirror. Since the position of the mirror relative to the smartphone camera is known, the virtual camera can be reflected back and the position of the real camera relative to the marker (carpet) can be estimated, which provides position tracking.

As shown in FIG. 2, the user's hand position is clearly visible in the camera's field of view, which can be reproduced in virtual reality or used to implement a gesture interface. Since the user's hands are in the camera's field of view most of the time, reflection can be used to track the user's hand positions and reproduce them in virtual reality. As you know, this greatly enhances the sense of immersion in virtual reality. In addition, the stream from the camera can be used to detect certain gestures that can be used within the gesture interface. Until now, there have been no systems that could perform hand tracking without using an external imaging device. It should be noted that such interfaces can be developed even in the absence of a floor marker (carpet).

In general, the proposed system includes the following hardware components: a smartphone, a helmet holder with an inclined mirror (as an option, the mirror can be attached directly to the smartphone) and a marker in the form of a floor carpet. In terms of software, the system uses a position estimation method that allows detecting and establishing correspondence between pixels of the reflected image and marker elements, followed by applying position estimation to determine the position of a camera with six degrees of freedom.

The mobile VR marker system according to the present invention achieves the following technical results.

The system according to the present invention has the advantages of marker technical solutions. By knowing the marker in advance, simpler, more efficient, accurate and reliable position estimation algorithms can be used to determine position compared to markerless technologies, which require an on-the-fly environmental map (so-called simultaneous positioning and mapping algorithms (SLAM)).

This system can be easily deployed simply by rolling the carpet on the floor. No additional calibration procedure is required after this.

The system provides position tracking as long as the user remains on the carpet. In the event that a user leaves the carpet, the standard VR interface can signal the user that he has stepped outside the carpet, as is the case with higher-end VR systems in which the user can walk.

Before a VR session begins, the carpet naturally limits the area that must be cleared of any objects in order to avoid collisions during the session. This ability to clearly delineate the critical area is not found in other VR systems (even in higher-grade VR systems that use external tracking technology).

The system according to the present invention allows the user's body position to be monitored using a smartphone camera. It can be used to enhance immersion in virtual reality with a rough representation of the user's body, thus providing an enhanced immersion experience. In addition, it can provide a gesture interface implementation without additional controllers.

The system is located completely inside the helmet and does not require any wires to connect to other devices, so the user can move safely and freely on the carpet.

Claims (8)

CLAIM 1. A marker system for tracking a position for virtual reality, containing a camera installed on the user, and the camera is a smartphone camera, and a floor marker, while an oblique mirror is installed in front of the camera so that the field of view of the camera is redirected downward. 2. The system of claim 1, wherein the floor marker is embedded in the expandable carpet. 3. The system of claim 1, wherein the floor marker designates an area that is safe for the user to move. 4. The system of claim 1, wherein the tilting mirror is attached to the helmet holder. 5. The system of claim 1, wherein the tilting mirror is attached to the smartphone. 6. The system of claim 1, wherein the marker comprises a grid of dark squares on a white background. 7. The system of claim 1, configured to track the position of the user's hands and reproduce the positions of the user's hands in virtual reality. 8. The system of claim 7, configured to recognize gestures with a camera, the recognized gestures being used as the gesture interface.