JP6785025B1 - Information processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To enable a plurality of information processing devices to share a space. SOLUTION: In the information processing method, a first information processing device sets a first coordinate axis in a predetermined space, a second information processing device sets a second coordinate axis in a predetermined space, and a second The information processing device generates a conversion rule for converting the second coordinate axis to the first coordinate axis, and each of the first and second information processing devices is represented by the first coordinate axis from an image of a predetermined space. It is characterized by acquiring a group of points, sharing the acquired point group, and creating an environmental map using the shared point group. [Selection diagram] Fig. 1

Description

The present invention relates to an information processing method.

Conventionally, with the remarkable development of information processing, it has become possible to execute various information processing such as games using AR (Augmented Reality) on a mobile terminal (see, for example, Patent Document 1).

Japanese Patent No. 6417467

However, in the conventional techniques including Patent Document 1, it is difficult for a plurality of people to share a space and execute a game using AR by using each of the plurality of mobile terminals.

The present invention has been made in view of such a background, and an object of the present invention is to enable a plurality of information processing devices to share a space.

The main invention of the present invention for solving the above problems is an information processing method, in which a first information processing device sets a first coordinate axis in a predetermined space and a second information processing device sets the first coordinate axis in the predetermined space. A second coordinate axis is set, the second information processing device generates a conversion rule for converting the second coordinate axis into the first coordinate axis, and each of the first and second information processing devices It is characterized in that a point group represented by the first coordinate axis is acquired from an image obtained by capturing the predetermined space, the acquired point group is shared, and an environment map is created using the shared point group. And.

Other problems disclosed in the present application and solutions thereof will be clarified in the columns and drawings of the embodiments of the invention.

According to the present invention, a plurality of information processing devices can share a space.

It is a figure which shows the example of the sharing method of AR space. It is a figure which shows the example of the error correction method. It is a figure explaining the outline of the error correction method. It is a figure explaining the flow of the initial setting process. It is a figure explaining the flow of processing at the time of participation in the middle.

<Outline of the invention>
The contents of the embodiments of the present invention will be described in a list. The present invention includes, for example, the following configuration.
[Item 1]
The first information processing device sets the first coordinate axis in a predetermined space,
The second information processing device sets the second coordinate axis in the predetermined space,
The second information processing apparatus generates a conversion rule for converting the second coordinate axis into the first coordinate axis.
Each of the first and second information processing devices
A point cloud represented by the first coordinate axis is acquired from an image obtained by capturing the predetermined space.
Share the acquired point cloud and
Creating an environmental map using the shared point cloud,
An information processing method characterized by.
[Item 2]
The information processing method according to item 1.
The third information processing device
The first point cloud, which is the point cloud, is shared by at least one of the first and second information processing devices.
An image of the predetermined space is acquired and
The second point cloud was identified from the acquired image, and
To set the first coordinate axis by comparing with the first and second point clouds.
An information processing method characterized by.
[Item 3]
The information processing method according to item 1.
At least one of the first and second information processing
After sharing the first point cloud, the second point cloud is acquired and
When the similarity between the first feature amount extracted from the first point cloud and the second feature amount extracted from the second point group is equal to or more than a predetermined value, the first point group To calculate the distance between the position and the position of the second point cloud and move at least one of the origins of the first and second coordinate axes by the distance.
An information processing method characterized by.

In the augmented reality system according to the embodiment of the present invention, the augmented reality system provides augmented reality (AR) in which an image of a virtual space is superimposed on a real image of a real space. It should be noted that mixed reality (MR; Mixed Reality) in which a real image is reflected in a virtual space or virtual reality (VR; Virtual Reality) in which only an image of a virtual space is displayed may be provided.

The augmented reality system of the present embodiment can be configured to include, for example, two or more information processing devices such as a smartphone having a communication function and a shooting function. The information processing device includes a CPU, a memory, a storage device, a communication interface, an input device, an output device, and a camera. The storage device stores various data and programs, such as a hard disk drive, a solid state drive, and a flash memory. The communication interface is an interface for connecting to a communication network, for example, an adapter for connecting to Ethernet (registered trademark), a modem for connecting to a public telephone network, a wireless communication device for wireless communication, and a serial. It is a USB (Universal Serial Bus) connector or RS232C connector for communication. A computer can communicate with other computers via a communication network. The input device is, for example, a keyboard, a mouse, a touch panel, a button, a microphone, or the like for inputting data. The output device is, for example, a display, a printer, a speaker, or the like that outputs data. The camera may be a camera that captures a two-dimensional image, a rider that measures depth, or a compound eye camera. Each functional unit of the information processing device, which will be described later, is realized by the CPU reading the program stored in the storage device into the memory and executing the program, and each storage unit of the information processing device is the memory and the storage provided by the storage device. Realized as part of the realm.

By collaborating with various hardwares of such information processing devices and various software, it becomes possible to execute seed information processing such as a game using AR (Augmented Reality) between two or more information processing devices.

Here, when two or more information processing devices execute various information processing such as a game using AR, it is necessary to share the AR space. The AR space is a space processed by the information processing device, and is a space in which some information such as the arrangement of objects is added to the real space (a space constructed in the information processing device based on the captured image). ). The position and orientation of an object or the like in the AR space are defined based on the coordinate axes (including the origin) used in the AR space recognized by the information processing device. Aligning the coordinate axes used in the AR space in each of the plurality of information processing devices is "sharing the AR space".

In the present embodiment, a plurality of information processing devices (for example, master unit 1-P and slave unit 1-C) share the AR space by aligning (combining) the coordinate axes used in the AR space. I am trying to. Here, the master unit 1-P refers to an information processing device in which a reference coordinate axis (coordinate axis used in the shared AR space) is set. Further, the slave unit 1-C refers to an information processing device other than the master unit 1-P.

FIG. 1 is a diagram showing an example of a method of sharing an AR space. As a method of sharing the AR space, a marker is used (marker base), a general image that is not limited to the marker is used (image base), and a point cloud extracted from the image is used (point cloud base). ) Exists. In each case, there are cases where an object is not placed in the real space and is realized only by an information processing device such as a smartphone (smartphone), and there are cases where an object placed in the real space is used. In the present embodiment, the initial alignment (spatial sharing) is performed on a marker basis or an image basis, and the position error correction after the spatial sharing and the spatial sharing when an additional user participates are performed on a point cloud basis.

The information processing device (master unit 1-P and slave unit 1-C, respectively)
A space recognition unit that recognizes AR space,
A space sharing section that shares AR space,
An information sharing unit that shares the position of the information processing device with other information processing devices,
A point cloud sharing unit that shares the point cloud acquired by the space recognition unit with other information processing devices,
An environmental map storage unit that stores point clouds and
An error correction unit that corrects the deviation of the position (origin of the coordinate axes) of the information processing device,
To be equipped.

The space recognition unit can perform space recognition by moving SLAM (Simultaneus Localization and Mapping). In the present embodiment, the space recognition unit analyzes the image taken by the camera, acquires a three-dimensional point cloud, creates an environmental map using the acquired point cloud, and determines its own position on the coordinate axes of the environmental map. Perform a so-called Visual SLAM to estimate. The axis axp in the AR space is set on the master unit 1-P side, while the axis axc (different from the axis app) in the AR space is set on the slave unit 1-C side.

The environment map storage unit stores the coordinate axes set by the space recognition unit and the point cloud. The point cloud may store only each point (X, Y, Z), but in the present embodiment, the feature amount extracted from the point cloud is also stored in the environment map storage unit. The feature quantity can be analyzed based on, for example, a group of point clouds.

The space sharing unit can share space on a marker basis, for example. For example, the space sharing unit of the master unit 1-P generates data of an image as a marker (hereinafter, referred to as “marker image”) including information that can identify the axis axp, and displays the marker image on the output device. Let me. The space sharing unit of the slave unit 1-C reads a marker image using a camera. When the marker image is read, the space sharing portion of the slave unit 1-C is the position and orientation of the master unit 1-P as seen from the slave unit 1-C (with the axis axc set by the slave unit 1-C). And the vector starting from that coordinate) is estimated. Based on the estimation result, the space sharing unit of the slave unit 1-C aligns the origin of the axis axc with the origin of the axis axp and aligns the direction of the axis axc with that of the axis axp. In other words, the space sharing unit of the slave unit 1-C generates a matrix XC (hereinafter, referred to as "transformation matrix XC") for converting the coordinates and the posture on the axis axc into the coordinates and the posture on the axis axp. As a result, the slave unit 1-C and the master unit 1-P can share the AR space.

The master unit 1-P can process various information in the AR space by using the coordinates and directions of the coordinate axis app set at the beginning. On the other hand, the slave unit 1-C uses various coordinates and directions in the AR space by using the coordinates and directions of the coordinate axes (same as the coordinate axis axp) converted by the transformation matrix XC with respect to the coordinates and directions of the coordinate axis axc initially set. Information can be processed.

The information sharing unit can exchange various information subjected to various processing in this way to and from each other by P2P (peer-to-peer). In addition to the position information, the various information includes, for example, the direction in which the attack was issued by the game.

The point cloud sharing unit can share the point cloud by transmitting and receiving the point cloud to and from another information processing device by, for example, P2P communication. The point cloud sharing unit can also share features.

Here, when the position of the information processing apparatus changes and the self-position is estimated using SLAM, the estimation error accumulates, and it is difficult to adjust the position error in the middle of play. In the present embodiment, the error correction unit corrects the position error using the point cloud. As described above, in the present embodiment, since the point cloud is shared by other information processing devices, it is possible to use the point cloud acquired from an image other than the image collected by itself. Therefore, it is expected that the accuracy of error correction will be improved.

FIG. 2 is a diagram showing an example of an error correction method. In the case of the marker base, it is necessary to display the marker image again on the information processing device and read it again, or to recognize the marker in the real space again. Similarly, in the image base, it is necessary to display the image and reload it, or to recognize the image in the real space again. However, such an approach is not good for UX. In the case of point cloud base, if UWB is available, it can be corrected if the smartphones are facing each other within the range without obstacles, but it is difficult to correct under other conditions. In the real space, if the ARCloud is constructed, the error can be corrected by collating with the ARCloud's environment map, but if the ARCloud is not constructed, a specific point cloud (feature amount) is found. You can make corrections when you can. In the present embodiment, the shared point cloud can be used to detect a specific feature amount and correct the error.

FIG. 3 is a diagram illustrating an outline of an error correction method.

(1) Initial setting Obtain the relative position and align the origin.
At that time, save the position of the surrounding point cloud and share it with each other. When the point cloud is recognized, the self-position is determined based on the coordinates and the origin is adjusted.

(2) Participation in the middle If you want to participate in the middle of the play, ask them to share the point cloud stored in the information processing device. When looking at the point cloud, the self-position can be determined and the origin can be adjusted.

FIG. 4 is a diagram illustrating a flow of initial setting processing.
1. 1. The space sharing unit acquires a relative position and aligns the origin as described above.
2. After that, the spatial recognition unit continuously collects and stores the point cloud.
3. 3. The point cloud stores the position information (X, Y, Z) of the feature points and the feature amount in the environment map storage unit.
4. The point cloud sharing unit communicates the point cloud by P2P and stores it in each information processing device.
5. The space recognition unit then recognizes the point cloud in real time.
6. When the position error is small (for example, the reliability of the position estimation by the space recognition unit may be equal to or higher than a predetermined threshold value).
6-1. The error correction unit can determine at which position and in which direction in space. Therefore, when the point cloud is recognized, the error correction unit can determine whether or not the point cloud may be preserved.
6-2. The error correction unit can pick up only the point cloud that is likely to match from the point cloud stored in the terminal. For example, it is possible to pick up only a point cloud included within a predetermined angle in the vertical and horizontal directions from the direction in which the information processing device is facing.
6-3. The error correction unit measures the similarity between the feature amount of the point cloud acquired in real time and the stored point cloud.
6-4. When the similarity is close (less than or equal to a predetermined value), the error correction unit measures the distance between the position of the point cloud acquired in real time and the position of the stored point cloud.
6-5. When the average position between the point clouds exceeds the constant R, the error correction unit adjusts the position of the origin of the terminal by the distance of the error.
7. When the position error is large (for example, the reliability of the position estimation by the space recognition unit may be less than a predetermined threshold value).
7-1. The error correction unit measures the similarity between the feature amount of the point cloud acquired in real time and the stored point cloud.
7-2. When the similarity is close, the error correction unit measures the distance between the position of the point cloud acquired in real time and the position of the stored point cloud.
7-3. The error correction unit adjusts the position of the origin of the terminal by the average distance between the point clouds.

FIG. 5 is a diagram for explaining the flow of processing at the time of participation in the middle. It is premised that the point cloud is continuously collected and saved in the existing information processing device that acquires the relative position and aligns the origin.
6. Method of determining self-position by newly participating information processing device 6-1. The error correction unit measures the similarity between the feature amount of the point cloud acquired in real time and the stored point cloud.
6-2. When the similarity is close, the error correction unit measures the distance between the position of the point cloud acquired in real time and the position of the stored point cloud.
6-3. The error correction unit adjusts the position of the origin of the terminal by the average distance between the point clouds.

As described above, according to the augmented reality system of the present embodiment, the slave unit 1-C captures the image displayed by the master unit 1-P using only the information processing device, so that the master unit 1- The coordinate axes of the slave units 1-C can be converted to the coordinate axes of P. Then, after such position (origin) alignment, the point group collected by each information processing device can be shared with each other. A larger number of points are collected, and the accumulated points can be used to correct the self-position error. Also, other than the master unit 1-P and the slave unit 1-C, which will participate later. The third information processing device of No. 3 uses the point group (environmental map) shared by the master unit 1-P and the slave unit 1-C to adjust its own coordinate axis to that of the master unit 1-P, and also concerned. The self-position on the coordinate axes can be estimated.

Although the present embodiment has been described above, the above embodiment is for facilitating the understanding of the present invention, and is not for limiting and interpreting the present invention. The present invention can be modified and improved without departing from the spirit thereof, and the present invention also includes equivalents thereof.

Claims (3)

The first information processing device sets the first coordinate axis in a predetermined space,
The second information processing device sets the second coordinate axis in the predetermined space,
The second information processing apparatus generates a conversion rule for converting the second coordinate axis into the first coordinate axis.
Each of the first and second information processing devices
A point cloud represented by the first coordinate axis is acquired from an image obtained by capturing the predetermined space.
Share the acquired point cloud and
Creating an environmental map using the shared point cloud,
An information processing method characterized by. The information processing method according to claim 1.
The third information processing device
The first point cloud, which is the point cloud, is shared by at least one of the first and second information processing devices.
An image of the predetermined space is acquired and
The second point cloud was identified from the acquired image, and
To set the first coordinate axis by comparing with the first and second point clouds.
An information processing method characterized by. The information processing method according to claim 1.
At least one of the first and second information processing devices
After sharing the first point cloud, the second point cloud is acquired and
When the similarity between the first feature amount extracted from the first point cloud and the second feature amount extracted from the second point group is equal to or more than a predetermined value, the first point group To calculate the distance between the position and the position of the second point cloud and move at least one of the origins of the first and second coordinate axes by the distance.
An information processing method characterized by.

Images