WO2020185115A1

WO2020185115A1 - Method and system for real-time data collection for a mixed reality device

Info

Publication number: WO2020185115A1
Application number: PCT/RU2019/050231
Authority: WO
Inventors: Ильнур Зямилевич ХАРИСОВ
Original assignee: Общество с ограниченной ответственностью "ТрансИнжКом"
Priority date: 2019-03-13
Filing date: 2019-11-28
Publication date: 2020-09-17
Also published as: RU2702495C1

Abstract

The present technical solution relates to the field of digital data processing, and more particularly to a method and system for real-time data collection for a mixed reality device. The technical result consists in providing more accurate display of mixed reality. A method for real-time data collection for a mixed reality device consists in carrying out the following steps using a computing device: capturing real-world images in a wide range of angles using at least one camera; scanning a real-world space in order to determine the dimensions and arrangement of real-world objects; determining data pertaining to the position and orientation of a mixed reality device in said space; carrying out real-time tracking and 3D recognition of hand gestures in order to determine the specific location of objects in the space in relation to the hand being tracked; saving in a memory unit of the mixed reality device the data pertaining to the captured real-world images, the data pertaining to the dimensions and arrangement of the real-world objects, the data pertaining to the position and orientation of a data collection unit, and the data pertaining to the tracking and recognition of hand gestures; processing the saved data and using built-in mini projectors to project onto a holographic screen of the mixed reality device, said screen merging virtual and augmented reality to produce mixed reality.

Description

METHOD AND SYSTEM FOR COLLECTING INFORMATION FOR ALIGNED REALITY DEVICE IN REAL TIME MODE

FIELD OF TECHNOLOGY

The present technical solution relates to the field of digital data processing, in particular, to a method and a system for collecting information for combined reality devices in real time.

LEVEL OF TECHNOLOGY

Mixed reality, sometimes called hybrid, mixed reality or augmented virtual reality, is a mixture of real and virtual reality, when believable virtual objects are superimposed on our familiar environment. At the same time, the depth of immersion in virtual reality is higher than that of augmented reality, which does not change the human vision of the surrounding world and its perception, but only supplements the real world with artificial elements and new information. The combined reality does not take us away from the real world, but supplements the real world with information, images and sounds stored inside information systems, while in the combined reality there is a point of interaction between the real and the virtual world

Combined reality devices are aimed at bringing a piece of virtuality into our everyday world and ensuring interaction with digital elements as if they were real objects. A person can freely move in space without fear of bumping into real objects or falling off a virtual staircase. Natural application of combined reality systems is a means of "improving" the real world, i.e. virtual prototyping, when we expand the real world with objects that we would like to see there, but for a number of reasons, for example, the cost of manufacturing, we cannot put them there. This is especially true in design-related work, when new objects must be optimally fit into the existing space. For example, you can select and arrange furniture in an empty room, analyze an architectural project, present a landscape design option, etc. Also, according to the technology of combined reality, the system has the ability to compare a real and a virtual object and display some kind of hint on a real object, without displaying the virtual object in full or in part. The combined reality finds its application in completely different areas of our life:

Construction - Visualization of three-dimensional drawings directly at the construction site, plan-fact analysis of the objects under construction and constructed, display virtual prompts to existing objects.

Military - Conducting combat training in various terrain conditions. In this case, there will be no need to fly anywhere on purpose. There are also ample opportunities for operators of military equipment.

Sports - Virtual objects provide a completely new approach to sports training, and wearable gadgets allow to display the physical condition of athletes.

Gaming - Mixed reality gaming is undoubtedly opening up a whole new market for game development.

Medicine - Mixed reality will make examinations and complex surgeries much safer and more reliable.

Retail - Online sales of goods will move to a new level with the ability to “touch” the purchased items in 3D models.

They also predict the huge impact of mixed reality on other areas, such as education, architecture, business communications.

There are solutions in the field of virtual reality, in which only images from the computer's memory are provided to the user by means of a virtual reality helmet.

There are solutions in the field of augmented reality, in which the user sees additional images superimposed on images of the real world without reference to the geographic location of objects in the real world.

There are solutions in the field of combined reality, in which virtual reality images are snapped to the geographical coordinates of the real world, which additionally increases the realism of virtual reality images and gives more opportunities to interact with it.

There is a known solution (US2014210947A1, publ. 07/31/2014), which describes the process of forming augmented reality using coordinate geometry. Embodiments of this invention include a method, system, and mobile device that incorporate augmented reality technology in land surveying, 3D laser scanning, and digital modeling processes. Using augmented reality technology, a mobile device can display an augmented reality image that represents a real-world view of a physical structure in a real-world environment, and a 3D digital model of an unbuilt design element overlaid on top of the physical structure at its intended location. In one embodiment, a marker may be placed at a predetermined set of coordinates on or around a location of interest as determined by surveying equipment so that a 3D digital model of an unbuilt design element can be rendered in a geometrically correct orientation relative to the physical structure. Embodiments of this solution can also be applied to a reduced 3D printed object representing a physical structure if visiting the project site is not possible.

However, in this solution, in order to build an image, it is necessary to place markers in the place of interest to orient the augmented image forming device in space.

A solution is known (US2014270477A1, publ. 09/18/2014), which describes systems and methods for displaying a three-dimensional model from a photogrammetric scan. In one embodiment, a computer-implemented method is provided for displaying a three-dimensional (3D) model from a photogrammetric scan. The object image and scan marker can be obtained in the first place. The relationship between the object image and the scan marker image at the first location can be determined. The geometric property of an object can be determined based on the relationship between the object image and the scan marker image. The ZO-model of an object can be generated based on a specific geometric property of the object. The 3D model of the object can be displayed for scaling in augmented reality environment at the second location based on the scan marker at the second location.

However, in this solution, to build an image, it is necessary to place markers in the place of interest to form an augmented image.

Known solution (US2016104323A1, publ. 04.14.2016), which describes a device for displaying an image and a method for displaying an image. According to one embodiment, an apparatus for displaying an image includes a data acquisition unit and a display processing unit. The data collection unit is configured to acquire a received image, which is acquired by the camera, and which includes an optical recognition code representing identification information, by generating a plurality of line-shaped elements. The display processing unit is configured to superimpose and display an image of a three-dimensional object corresponding to the identification information on the acquired image. The orientation of the 3D object image superimposed on and displayed on the captured image is determined based on the orientation of the OCR code in the captured image and the tilt of the camera.

However, in this solution, identification information must be placed in order to build the image.

SUMMARY OF THE INVENTION

The technical problem to be solved by the claimed technical solution is the creation of a computer-implemented method and system for collecting information for a combined reality device in real time, which are characterized in independent claims. Additional embodiments of the present invention are presented in the dependent claims.

The technical result consists in improving the accuracy of displaying the combined reality.

In a preferred embodiment, a computer-implemented method of collecting information for a combined reality device in real time is claimed, which consists in performing stages using a computing device, on which:

• capture images of the real world in a wide range of angles using at least one camera;

• scan the space of the real world to determine the size and location of objects in the real world;

• determine the data on the location and orientation of the combined reality device in space;

• carry out tracking and three-dimensional recognition of hand gestures in real time to determine exactly where objects are in space relative to the tracked hand; • store data of captured images of the real world, data of dimensions and location of objects of the real world, data of location and orientation of the data collection unit, tracking data and recognition of hand gestures in the memory unit of the combined reality device;

• process the stored information and project through the built-in mini projectors onto the holographic screen of the combined reality device, which merges virtual and augmented reality to obtain a combined reality.

In a private version, hand gesture tracking and recognition is carried out using a machine learning algorithm.

In another particular embodiment, audio data is additionally captured.

In another particular embodiment, the combined reality device is a helmet, tablet, or glasses.

In another particular embodiment, the camera may be a stereoscopic camera and / or a depth-effect camera for machine vision and / or gesture recognition.

In another particular embodiment, data is additionally collected from the inertial navigation system to further determine the location data of the aligned reality device.

The claimed solution is also implemented through a system for collecting information for a combined reality device in real time. This system includes:

- a combined reality device that contains:

• at least one camera capable of capturing images of the real world in a wide range of angles;

• built-in mini projectors;

• holographic screen;

• real world space scanning unit;

• positioning unit;

• a computing device capable of performing the above method.

In another particular embodiment, the system further includes an inertial navigation system for additional determination of data on the location of the combined reality device, while the inertial navigation system is based on gyroscopes and accelerometers. DESCRIPTION OF DRAWINGS

The implementation of the invention will be described in the following in accordance with the accompanying drawings, which are presented to clarify the essence of the invention and in no way limit the scope of the invention. The following drawings are attached to the application:

FIG. 1 illustrates a computer-implemented method for collecting information for a combined reality device in real time;

FIG. 2 illustrates a scheme for locating a user;

FIG. 3 illustrates an example of a general arrangement of a computing device.

DETAILED DESCRIPTION OF THE INVENTION

In the following detailed description of an implementation of the invention, numerous implementation details are set forth to provide a thorough understanding of the present invention. However, it will be obvious to those skilled in the art how the present invention can be used, with or without these implementation details. In other instances, well-known techniques, procedures, and components have not been described in detail so as not to obscure the details of the present invention.

In addition, it will be clear from the above description that the invention is not limited to the above implementation. Numerous possible modifications, changes, variations and substitutions, while retaining the spirit and form of the present invention, will be apparent to those skilled in the art.

The present invention is aimed at providing a computer-implemented method and system for collecting information for a combined reality device in real time.

As shown in FIG. 1, the claimed computer-implemented method of collecting information for a combined reality device in real time (100) is implemented as follows:

In step (101), at least one camera captures images of the real world over a wide range of angles. The camera can be a stereoscopic camera and / or a depth-effect camera for machine vision and / or gesture recognition. The minimum capture angle range for images is 30 degrees.

Next, at step (102), the real world space is scanned to determine the dimensions and location of real world objects. In this case, the real-world space scanning unit can be made in the form of ultrasonic, optical or laser scanning means.

In step (103), data on the location and orientation of the aligned reality device in space is determined.

To build images of the combined reality, it is necessary to determine the location of the user by determining the location of the combined reality device that he is wearing.

At the same time, a combined reality device can be either a helmet, a tablet, or glasses, which are completely autonomous devices powered by a battery. But at the same time, they can additionally be connected to a computer or tablet.

Determining the location of a combined reality device in space is possible using GPS, Glonas, but the accuracy in this case can be about 6-8 meters, which is unacceptable for most applications.

A common disadvantage of using any radio navigation system is that under certain conditions the signal may not reach the receiver, or arrive with significant distortion or delays. For example, it is almost impossible to determine your exact location in the depths of an apartment inside a reinforced concrete building, in a basement or in a tunnel, even with professional geodetic receivers. Since the operating frequency of GPS / Glonas lies in the decimeter range of radio waves, the signal level from satellites can seriously decrease under dense foliage of trees or due to very large clouds. The normal reception of GPS / Glonas signals can be damaged by interference from many terrestrial radio sources, as well as from magnetic storms.

In addition to GPS / Glonas-based methods, there are a number of location determination methods that are more suitable for indoor implementation, some of which are presented below:

Cellular Positioning - Accuracy is poor, even in areas with high base station density. Determination of location using inertial systems, where a model of human movement is used: if we know where we were, in which direction and how fast we were moving, then we can calculate where we ended up after a while. This is now achieved with gyroscopes, accelerometers, Hall sensors, or other suitable means.

Determination of the location using optical systems, which are based on preliminary scanning of the room, and then on the picture, for example, of the ceiling from the front camera of a smartphone, it is possible to determine the location.

Positioning using magnetometry - determining the location by the magnetic field using the compass of your smartphone. The solution requires preliminary calibration in the room and is highly susceptible to metal and magnets.

Location based trilateration based on Wi-Fi / Bluetooth transmitters. For implementation, common equipment is used, both for infrastructure and for location determination. It is also possible to use already deployed Wi-Fi / Bluetooth networks.

Wi-Fi / Bluetooth radio map or “digital fingerprints” - location is calculated by comparing real-time signal strengths from surrounding Wi-Fi / BLE points with previously measured values tied to a room map.

To improve the accuracy of determining the location of the combined reality device in space, it was proposed to place in real space, where it is supposed to form images of the combined reality, stations transmitting radio signals (as shown in Fig. 2). Using the radio signals of these stations located in previously known coordinates, the trilateration method can be used to determine the location of the receiver with high accuracy. To implement this approach, at least three transmitters are required, the signal from which is confidently received by the receiver.

The claimed solution uses a radar system based on ultra-wideband signals to determine the exact location of the user in real space, as well as an inertial navigation system to determine the user's orientation (indoor positioning system). As a result Using these two systems, it is possible to obtain information about the location and orientation of the user in real space, which makes it possible to place geo-referenced virtual reality objects with a minimum error sufficient to use the latter as a reference point when performing, for example, construction work.

Also discussed below are embodiments that combine the location capabilities of different methods. These solutions provide acceptable quality for accurate alignment of real-world and virtual reality images.

In the proposed solution, hardware and software for the implementation of the location determination functionality is installed in a combined reality device in space, which contains at least one camera configured to capture an image of the real world in a wide range of angles (preferably a stereoscopic camera) around the user who it is a combined reality device.

In general, the hardware and software for positioning is standard and well known in the art, features differing from known implementations will be described separately. In general, the combined reality device is configured to determine its location using a GPS / Glonas positioning unit and / or an indoor positioning unit, with the ability to receive images from a camera, transmit all data to a processing unit, with the ability to provide processed data (data combined reality) to the user.

In step (104), real-time tracking and 3D recognition of hand gestures is performed to determine where exactly the objects are in space with respect to the tracked hand. The hand tracking system is based on computer vision and works with a self-optimizing machine learning algorithm.

Next, at step (105), the data of the captured images of the real world, the data of the dimensions and location of the objects of the real world, the data of the location and orientation of the information collection unit, the tracking data and recognition of hand gestures are stored in the memory unit of the combined reality device. At step (106), the stored information is processed and projected by means of built-in mini projectors onto the holographic screen of the combined reality device, which merges virtual and augmented reality to obtain a combined reality. The virtual 3D hologram is formed by means of built-in mini projectors. The visualization method can be any, at this stage of technological development there are: projection image construction, LCD, TFT, IPS, AMOLED, P-OLED, QLED images.

Then the combined reality is sent to the eyes of the user.

A combined reality device can be equipped with two speakers - thus, projection sound is added to the sounds of the outside world. Alternatively, bone conduction technology may be used to transmit sound.

In addition, a function of tracking the direction of a user's gaze can be implemented in a combined reality device, namely, the system tracks the position of a person's pupils. As a result, you can move the cursor in the direction you are looking. And the hologram at which the cursor is directed will be highlighted. And also calibrate the visualization system for the user's eyes.

Information about the position of the pupils of a person allows you to determine the direction of gaze, highlight the object that the person is looking at, and synthesize images with the correct vergence on the selected object. As a result, information about the position of a person's pupils can help to form an image with the correct accommodation and thereby provide the most comfortable conditions for perception of a virtual image, indistinguishable from the image of the real world.

Unlike virtual reality helmets, in the declared solution, the combined reality device allows the user to simultaneously remain in the real world, maintaining full interaction with him.

Holographic projections created by a combined reality device have three dimensions, which allows you to walk around a virtual object from all sides, examine and enlarge its details, see how it fits into reality, which opens up endless possibilities for various types of 3D modeling.

FIG. 3, a general diagram of a computing device (300) will be presented below, which provides data processing necessary for the implementation of the claimed solution. In the general case, the device (300) contains such components as: one or more processors (301), at least one memory (302), data storage means (303), input / output interfaces (304), I / O means ( 305), networking tools (306).

The device processor (301) performs the basic computational operations necessary for the operation of the device (300) or the functionality of one or more of its components. The processor (301) executes the necessary computer-readable instructions contained in the main memory (302).

Memory (302), as a rule, is made in the form of RAM and contains the necessary program logic to provide the required functionality.

The data storage medium (303) can be performed in the form of HDD, SSD disks, raid array, network storage, flash memory, optical information storage devices (CD, DVD, MD, Blue-Ray disks), etc. The tool (303) allows performing long-term storage of various types of information, for example, the history of processing transactional requests (logs), user identifiers, etc.

Interfaces (304) are standard means for connecting and working with the server side, for example, USB, RS232, RJ45, LPT, COM, HDMI, PS / 2, Lightning, FireWire, etc.

The choice of interfaces (304) depends on the specific implementation of the device (300), which can be a personal computer, mainframe, server cluster, thin client, smartphone, laptop, cloud space, etc.

As means of I / O data (305) in any embodiment of the system that implements the described method, a keyboard should be used. In addition to it, the following can also be used: joystick, display (touch screen), projector, touchpad, mouse, trackball, light pen, speakers, microphone, etc.

Networking means (306) are selected from a device that provides network reception and transmission of data, for example, Ethernet card, WLAN / Wi-Fi module, Bluetooth module, BLE module, NFC module, IrDa, RFID module, GSM modem, etc. The means (305) provide the organization of data exchange via a wired or wireless data transmission channel, for example, WAN, PAN, LAN, Intranet, Internet, WLAN, WMAN or GSM. The components of the device (300) are interfaced via a common data bus (310).

In the present application materials, the preferred disclosure of the implementation of the claimed technical solution was presented, which should not be used as limiting other, particular embodiments of its implementation, which do not go beyond the scope of the claimed scope of legal protection and are obvious to specialists in the relevant field of technology.

Claims

CLAIM

1. A method of collecting information for a device of combined reality, in real time, which consists in performing stages using a computing device, on which:

• carry out tracking and three-dimensional recognition of hand gestures in real time to determine exactly where objects are in space relative to the tracked hand;

• store data of captured images of the real world, data of dimensions and location of objects of the real world, data of location and orientation of the data collection unit, tracking data and recognition of hand gestures in the memory unit of the combined reality device;

2. The method according to claim 1, characterized in that the tracking and recognition of hand gestures is performed using a machine learning algorithm.

3. The method according to claim 1, further comprising capturing audio data.

4. A method according to claim 1, characterized in that the combined reality device is a helmet, tablet or glasses.

5. A method according to claim 1, characterized in that the camera can be a stereoscopic camera and / or a camera with a depth effect for machine vision and / or gesture recognition.

6. The method according to claim 1, further comprising collecting data from the inertial navigation system to further determine location data of the aligned reality device.

7. A system for collecting information for a combined reality device, in real time, containing:

- a combined reality device that contains:

• built-in mini projectors;

• holographic screen;

• real world space scanning unit;

• positioning unit;

• a computing device capable of performing the method according to any one of claims. 1-6.

8. The system according to claim 7, further comprising an inertial navigation system for further determining the location data of the combined reality device, wherein the inertial navigation system is based on gyroscopes and accelerometers.