JPWO2021024479A1 - Information processing equipment, information processing methods, and programs - Google Patents

Abstract

Multiple information processing devices can appropriately execute various information processing such as games using AR by sharing a space. The slave unit 1-C shares a predetermined space with the master unit 1-P to exchange information. That is, the coordinate setting unit 71 sets the first coordinate axis in the predetermined space. The transformation matrix generation unit 73 generates a rule for converting the first coordinate axis into a second coordinate axis used by the other information processing apparatus in the predetermined space.

Description

The present invention relates to an information processing device, an information processing method, and a program.

Conventionally, due to the remarkable development of information processing, it has become possible to execute various information processing such as games using AR (Augmented Reality) on a mobile terminal (see, for example, Patent Document 1).

Japanese Patent No. 6417467

However, in recent years, there has been a demand for a plurality of people to share a space and execute a game using AR by using each of a plurality of mobile terminals. The situation is such that such requests cannot be fully met.

The present invention has been made in view of such a situation, and an object of the present invention is to enable a plurality of information processing devices to appropriately execute various information processing such as games using AR by sharing a space. do.

In order to achieve the above object, the information processing device of one aspect of the present invention is
In an information processing device that shares a predetermined space with other information processing devices and exchanges information.
A coordinate setting means for setting the first coordinate axis in the predetermined space, and
A conversion rule generating means for generating a rule for converting the first coordinate axis into a second coordinate axis used by the other information processing apparatus in the predetermined space.
To be equipped.

A reading means for reading an image when the image is displayed on the other information processing device in which the second coordinate axis is set in the predetermined space.
With more
The conversion rule generating means is
Based on the read image, the position of the other information processing device on the first coordinate axis is estimated, and based on the estimation result, the coordinate axis in the predetermined space is set from the first coordinate axis as the rule. Generate a transformation matrix to convert to the second coordinate axis,
be able to.

A coordinate conversion means for converting a coordinate axis in the predetermined space from the first coordinate axis to the second coordinate axis using the transformation matrix.
Can be further prepared.

Each of the information processing methods and programs of one aspect of the present invention is each of the above-mentioned information processing methods and programs of one aspect of the present invention.

Further, in order to achieve the above object, the information processing device of another aspect of the present invention may be used.
In an information processing device that shares a predetermined space with other information processing devices and exchanges information.
A coordinate setting means for setting the first coordinate axis in the predetermined space, and
In the other information processing device in which the second coordinate axis is set in the predetermined space, the position of the information processing device on the second coordinate axis is estimated by being read, and the second coordinate axis is estimated based on the estimation result. An image generation means for generating image data used for generating a rule for converting the first coordinate axis into the first coordinate axis, and an image generation means for generating the image data based on the first coordinate axis.
A display control means for executing control to display the image generated as the data by the image generation means, and a display control means.
To be equipped.

Each of the information processing methods and programs of another aspect of the present invention is each of the above-mentioned information processing methods and programs of another aspect of the present invention.

According to the present invention, a plurality of information processing devices can appropriately execute various information processing such as games using AR by sharing a space.

It is a block diagram which shows an example of the hardware composition which concerns on one Embodiment of the information processing apparatus of this invention. FIG. 5 is a diagram illustrating an example of a method of sharing an AR space between a master unit and a slave unit when the plurality of information processing devices of FIG. 1 function as a master unit or a slave unit, respectively. It is a functional block diagram which shows an example of the functional configuration at the time of executing a series of processing until the AR space is shared among the functional configurations in each of the master unit and the slave unit of FIG. FIG. 5 is an arrow chart illustrating an example of a series of processes up to sharing the AR space, which are executed between the master unit and the slave unit having the functional configuration of FIG. It is an example of the method of sharing an AR space when a plurality of information processing devices of FIG. 1 exist, and is a diagram for explaining an example different from that of FIG. It is an example of the method of sharing an AR space when a plurality of information processing devices of FIG. 1 exist, and is a diagram for explaining an example different from those of FIGS. 4 and 5.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.

FIG. 1 is a block diagram showing a hardware configuration of an embodiment according to the information processing apparatus of the present invention.

The information processing device 1 is composed of a smartphone or the like.
The information processing device 1 includes a CPU (Central Processing Unit) 21, a ROM (Read Only Memory) 22, a RAM (Random Access Memory) 23, a bus 24, an input / output interface 25, a touch operation input unit 26, and the like. It includes a display unit 27, an image pickup unit 28, a storage unit 29, a communication unit 30, and a drive 31.

The CPU 21 executes various processes according to the program recorded in the ROM 22 or the program loaded from the storage unit 29 into the RAM 23.
Data and the like necessary for the CPU 21 to execute various processes are also appropriately stored in the RAM 23.

The CPU 21, ROM 22 and RAM 23 are connected to each other via the bus 24. An input / output interface 25 is also connected to the bus 24. A touch operation input unit 26, a display unit 27, an imaging unit 28, a storage unit 29, a communication unit 30, and a drive 31 are connected to the input / output interface 25.

The touch operation input unit 26 is composed of, for example, a capacitance type or resistance film type (pressure sensitive type) position input sensor laminated on the display unit 27, and detects the coordinates of the position where the touch operation is performed.
Here, the touch operation means an operation of contacting or approaching an object with respect to the touch operation input unit 26. An object that comes into contact with or is close to the touch operation input unit 26 is, for example, a player's finger or a touch pen. Hereinafter, the position where the touch operation is performed is referred to as a "touch position", and the coordinates of the touch position are referred to as "touch coordinates".
The display unit 27 is composed of a display such as a liquid crystal display, and displays various images such as an image related to a game.
As described above, in the present embodiment, the touch panel is composed of the touch operation input unit 26 and the display unit 27.

The imaging unit 28 images a subject in response to an instruction operation by the player, and outputs data of an captured image (for example, a marker image described later) obtained as a result.
The storage unit 29 is composed of a DRAM (Dynamic Random Access Memory) or the like, and stores various data.
The communication unit 30 controls communication with other devices (a server (not shown) or another player terminal (not shown) via a network including the Internet (not shown).

The drive 31 is provided as needed. A removable media 32 made of a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is appropriately mounted on the drive 31. The program read from the removable media 32 by the drive 31 is installed in the storage unit 29 as needed. Further, the removable media 32 can also store various data stored in the storage unit 29 in the same manner as the storage unit 29.

Execution of various information processing such as games using AR (Augmented Reality) between two or more information processing devices 1 by the cooperation between various hardware of the information processing device 1 of FIG. 1 and various software. Becomes possible.

Here, when two or more information processing devices 1 execute various information processing such as a game using AR, it is necessary to share the AR space.
The AR space is a space processed by the information processing device 1, and is a space to which some information such as the arrangement of objects is added to the real space (constructed in the information processing device 1 based on the captured image). Space).
The position and orientation of an object or the like in the AR space are defined based on the coordinate axes (including the origin) used in the AR space recognized by the information processing device 1.
Aligning the coordinate axes used in the AR space in each of the plurality of information processing devices 1 means "sharing the AR space".

In the present embodiment, as shown in FIG. 2, the plurality of information processing devices 1 (master unit 1-P and slave unit 1-C described later in the example of FIG. 2) have the same coordinate axes used in the AR space. By matching (making them together), we are trying to share the AR space.
Here, the master unit 1-P refers to an information processing device 1 in which a reference coordinate axis (coordinate axis used in a shared AR space) is set. Further, the slave unit 1-C refers to an information processing device 1 other than the master unit 1-P.

As shown in FIG. 2, each of the master unit 1-P and the slave unit 1-C moves SLAM (Simultaneus Localization and Mapping) to perform spatial recognition. As a result, the axis axp in the AS space SP is set on the master unit 1-P side, while the axis axc (different from the axis app) in the AS space SP is set on the slave unit 1-C side. NS.

Next, the master unit 1-P generates data of an image GM (hereinafter, referred to as “marker image GM”) as a marker including information capable of identifying the axis axp, and displays the marker image GM on the display unit 27 (hereinafter, referred to as “marker image GM”). It is displayed in Fig. 1).
The slave unit 1-C reads the marker image GM using the imaging unit 28 (FIG. 1).

When the slave unit 1-C reads the marker image GM, the position and orientation of the master unit 1-P as seen from the slave unit 1-C (coordinates on the axis axc set by the slave unit 1-C and Estimate the vector starting from that coordinate.
Based on the estimation result, the slave unit 1-C aligns the origin of the axis axc with the origin of the axis axp and aligns the direction of the axis axc with that of the axis axp. In other words, the slave unit 1-C generates a matrix XC (hereinafter, referred to as "transformation matrix XC") for converting the coordinates and the posture on the axis axc into the coordinates and the posture on the axis axp.

The slave unit 1-C and the master unit 1-P share the AR space SP. That is, the master unit 1-P processes various information in the AR space SP by using the coordinates and directions of the coordinate axis app set first. On the other hand, the slave unit 1-C uses the coordinates and directions of the coordinate axes (same as the coordinate axis axp) converted by the transformation matrix XC with respect to the coordinates and directions of the coordinate axis axc initially set in the AR space SP. Process various information.
The slave unit 1-C and the master unit 1-P mutually exchange various information subjected to various processing in this way by P2P (peer-to-peer).

The series of processes up to sharing the AR space SP between the master unit 1-P and the slave unit 1-C described above is the cooperation of hardware and software between the master unit 1-P and the slave unit 1-C, respectively. Realized by work. In this case, the master unit 1-P and the slave unit 1-C can have, for example, the functional configuration shown in FIG.
FIG. 3 is a functional block diagram showing an example of the functional configuration when a series of processes up to sharing the AR space are executed among the functional configurations of the master unit and the slave unit of FIG. 2. ..

As shown in FIG. 3, in the CPU 21 of the master unit 1-P, the master unit processing unit 52 and the communication control unit 53 function. The master unit 1-P may also function as the slave unit 1-C depending on the situation, and in this case, the slave unit processing unit 51 functions.
The master unit processing unit 52 is provided with a coordinate setting unit 61, a marker image generation unit 62, and a marker image display control unit 63.

In the CPU 21 of the slave unit 1-C, the slave unit processing unit 51 and the communication control unit 53 function. The slave unit 1-C may also function as the master unit 1-P depending on the situation, and in this case, the master unit processing unit 52 functions.
The slave unit processing unit 52 is provided with a coordinate setting unit 71, a marker image reading unit 72, a conversion matrix generation unit 73, and a coordinate conversion unit 74.

Hereinafter, details of each functional block of the master unit 1-P and the slave unit 1-C will be described with reference to FIG. 4 as appropriate.
FIG. 4 is an arrow chart illustrating an example of a series of processes executed between the master unit and the slave unit having the functional configuration of FIG. 3 until the AR space is shared.

In step S11, the coordinate setting unit 61 of the master unit 1-P performs spatial recognition by SLAM based on the data of the captured image in the real space captured by the imaging unit 28, and the coordinate axis axp in the AR space SP (FIG. 2). To set.

Similarly, in step S21, the coordinate setting unit 71 of the slave unit 1-C performs spatial recognition by SLAM based on the data of the captured image in the real space captured by the imaging unit 28, and the coordinate axis axc in the AR space SP ( Figure 2) is set.

In step S12, the marker image generation unit 62 of the master unit 1-P generates the data of the marker image GM (FIG. 2) based on the coordinate axis axp set in the process of step S11.
In step S13, the marker image display control unit 63 of the master unit 1-P executes control to display the marker image GM corresponding to the data generated in the process of step S12 on the display unit 27.

When the marker image GM displayed on the master unit 1-P is imaged by the imaging unit 28 of the slave unit 1-C, the process on the slave unit 1-C side proceeds to step S22.
In step S22, the marker image reading unit 72 of the slave unit 1-C reads the marker image GM (data of the captured image of the imaging unit 28).

In step S23, the transformation matrix generation unit 73 of the slave unit 1-C has the position and orientation of the master unit 1-P as seen from the slave unit 1-C at the timing of step S22 (the timing when the marker image GM is read). Estimate the coordinates on the axis axc set by the slave unit 1-C and the vector starting from those coordinates).
In step S24, the transformation matrix generation unit 73 of the slave unit 1-C generates the transformation matrix XC (FIG. 2) based on the position and orientation of the master unit 1-P estimated in the process of step S23.
In step S25, the coordinate conversion unit 74 of the slave unit 1-C converts the coordinate axes (matches the coordinate axis axp) using the conversion matrix XC generated in the process of step S24.
As a result, the AR space SP is shared between the slave unit 1-C and the master unit 1-P.

In step S14, the communication control unit 53 of the master unit 1-P uses the coordinate axis axp (set in the process of step S11) set by the coordinate setting unit 61 to transmit information in the shared AR space SP. , Transfers to and from the slave unit 1-C via the communication unit 30.
In step S26, the communication control unit 53 of the slave unit 1-C uses the coordinate axis app converted by the coordinate conversion unit 74 (converted in the process of step S25) to transmit information in the shared AR space SP. , Transfers to and from the master unit 1-P via the communication unit 30.

Although one embodiment of the present invention has been described above, the present invention is not limited to the above-described embodiment, and modifications, improvements, and the like within the range in which the object of the present invention can be achieved are included in the present invention. be.

For example, as a method of sharing the AR space SP, a method of matching the coordinate axes axp of the master unit 1-P (the method shown in FIG. 2) was adopted in the above-described embodiment, but the method is not particularly limited to this, and for example, FIG. And each method shown in FIG. 6 can be adopted.

FIG. 5 is an example of a method of sharing an AR space when a plurality of information processing devices of FIG. 1 exist, and is a diagram for explaining an example different from that of FIG.
In the example of FIG. 5, unlike the example of FIG. 4, a method of sharing only the transformation matrix is adopted.
That is, the example of FIG. 5 is the same as the example of FIG. 4 in that the marker image GM is read on the slave unit 1-C side. However, in the example of FIG. 5, the transformation matrix XCC for converting the coordinate axis app on the master unit 1-P side to the coordinate axis axc on the slave unit 1-C side and the coordinate axis axc on the slave unit 1-C side are used as the master unit 1. The transformation matrix XCP (inverse matrix of the transformation matrix XCC) for converting to the coordinate axis axp on the −P side is shared between the master unit 1-P and the slave unit 1-C, so that the AR space SP Sharing is planned.

FIG. 6 is an example of a method of sharing an AR space when a plurality of information processing devices of FIG. 1 exist, and is a diagram for explaining an example different from those of FIGS. 4 and 5.
In the example of FIG. 6, unlike the examples of FIGS. 4 and 5, the information processing apparatus 1 is equal without distinguishing between the master unit 1-P and the slave unit 1-C.
That is, the reference coordinate axis in the AR space SP is adopted as the reference axis axp on the master unit 1-P side in the examples of FIGS. 4 and 5, but is not particularly limited to this, and is set in advance in the example of FIG. The coordinate axis axes held on the cloud (Internet) is adopted.
Specifically, for example, if the real space SF corresponding to the AR space SP is known, as shown in FIG. 6, the point cloud data PG obtained as a result of restoring the real space SF in three dimensions and the data thereof. It is assumed that the coordinate axes ax used in the PG are associated with each other and are stored in the cloud (on the Internet) in advance. This coordinate axis axes is a reference coordinate axis in the AR space SP corresponding to the real space SF.
When it is necessary to share the AR space SP corresponding to the real space SF, each of the information processing devices 1-1 and 1-2 images the real space SF, and based on the data of the captured image obtained as a result, the information processing devices 1-1 and 1-2 are used. The coordinate axis axc1 or the coordinate axis axc2 is set respectively. Each of the information processing devices 1-1 and 1-2 extracts the feature points of the captured image and matches them with the data PG held in the cloud, so that the coordinate axis axc1 or the coordinate axis axc2 is used as a reference coordinate axis. Match to axes. As a result, the AR space SP corresponding to the real space SF can be shared between the information processing devices 1-1 and 1-2.

Here, the series of processes described above can be executed by hardware or software.
In other words, the functional configuration of FIG. 3 is merely an example and is not particularly limited. That is, it suffices if the information processing system is provided with a function capable of executing the above-mentioned series of processes as a whole, and what kind of functional block is used to realize this function is not particularly limited to the example of FIG. Further, the location of the functional block is not particularly limited to FIG. 3, and may be arbitrary.
Specifically, for example, each functional block shown in FIG. 3 is provided in the information processing device 1 as a native application in the above-described embodiment, but is implemented as a Web application using HTML and Javascript (registered trademark). Therefore, it is possible to prepare for a server or the like (not shown).
Further, one functional block may be configured by a single piece of hardware, a single piece of software, or a combination thereof.

When a series of processes are executed by software, the programs constituting the software are installed on a computer or the like from a network or a recording medium.
The computer may be a computer embedded in dedicated hardware. Further, the computer may be a computer capable of executing various functions by installing various programs, for example, a general-purpose smartphone or a personal computer in addition to a server.

The recording medium containing such a program is not only composed of a removable medium (not shown) distributed separately from the device main body in order to provide the program to the user, but also is preliminarily incorporated in the device main body to the user. It is composed of the provided recording medium and the like.

In the present specification, the steps for describing a program recorded on a recording medium are not necessarily processed in chronological order, but also in parallel or individually, even if they are not necessarily processed in chronological order. It also includes the processing to be executed.
Further, in the present specification, the term of the system means an overall device composed of a plurality of devices, a plurality of means, and the like.

In other words, the information processing apparatus to which the present invention is applied takes various embodiments having the following configurations, including the information processing apparatus as the embodiment of FIG. 3, FIG. 5 or FIG. 6 described above. be able to.
That is, the information processing device to which the present invention is applied (for example, the information processing device 1 in FIG. 1, specifically, the slave unit 1-C in FIG. 2, etc.) is
In an information processing device that exchanges information by sharing a predetermined space (for example, AR space SP as shown in FIG. 2) with another information processing device (for example, the master unit 1-P as shown in FIG. 2).
A coordinate setting means (for example, the coordinate setting unit 71 of FIG. 3) for setting the first coordinate axis (for example, the coordinate axis axc of FIGS. 2 and 4, the coordinate axis axc1 of FIG. 6 or the coordinate axis axc2) in the predetermined space.
A rule for converting the first coordinate axis into a second coordinate axis used by the other information processing apparatus in the predetermined space (for example, the coordinate axis axp in FIGS. 2 and 4, and the coordinate axis ax in FIG. 5) (for example, FIG. 2). A transformation rule generating means (for example, a transformation matrix generation unit 73 in FIG. 3) for generating a transformation matrix XC, a transformation matrix XCC in FIG. 4 or a transformation matrix XCP), and
To be equipped.

in this case,
When an image (for example, the marker image GM of FIG. 2) is displayed on the other information processing device in which the second coordinate axis is set in the predetermined space, a reading means for reading the image (for example, the marker image reading unit 72 of FIG. 3). ),
With more
The conversion rule generating means is
Based on the read image, the position of the other information processing device on the first coordinate axis is estimated, and based on the estimation result, the coordinate axis in the predetermined space is set from the first coordinate axis as the rule. Generate a transformation matrix to be converted to the second coordinate axis (for example, the transformation matrix XC in FIG. 2).
be able to.
Further, a coordinate conversion means (for example, the coordinate conversion unit 74 in FIG. 3) that converts the coordinate axes in the predetermined space from the first coordinate axis to the second coordinate axis by using the conversion matrix can be provided.

Further, the information processing device to which the present invention is applied (for example, the information processing device 1 in FIG. 1, specifically, the master unit 1-P in FIG. 2, etc.) is
In an information processing device that shares a predetermined space with another information processing device (for example, slave unit 1-C shown in FIG. 2) to exchange information.
A coordinate setting means (for example, the coordinate setting unit 61 in FIG. 3) for setting the first coordinate axis (for example, the coordinate axis axp in FIG. 2) in the predetermined space, and
In the other information processing device in which the second coordinate axis (for example, the coordinate axis axc in FIG. 2) is set in the predetermined space, the information processing device is read and the position of the information processing device on the second coordinate axis is estimated, and the estimation result thereof. Based on the data of the image (for example, the marker image GM of FIG. 2) used for generating the rule for converting the second coordinate axis to the first coordinate axis (for example, the transformation matrix XC of FIG. 2). An image generation means (for example, a marker image generation unit 62 in FIG. 3) generated based on the first coordinate axis, and
A display control means (for example, the marker image display control unit 63 in FIG. 3) that executes control for displaying the image generated as the data by the image generation means, and
To be equipped.

1 ... Information processing device, 1-P ... Master unit, 1-C ... Slave unit, 21 ... CPU, 51 ... Slave unit processing unit, Master unit processing unit, 52 ... Slave unit processing unit, master unit processing unit, 53 ... communication control unit, 61 ... coordinate setting unit, 62 ... marker image generation unit, 63 ... marker image display control unit, 71 ... coordinates Setting unit, 72 ... Marker image reading unit, 73 ... Conversion matrix generation unit, 74 ... Coordinate conversion unit

Claims (8)

In an information processing device that shares a predetermined space with other information processing devices and exchanges information.
A coordinate setting means for setting the first coordinate axis in the predetermined space, and
A conversion rule generating means for generating a rule for converting the first coordinate axis into a second coordinate axis used by the other information processing apparatus in the predetermined space.
Information processing device equipped with. A reading means for reading an image when the image is displayed on the other information processing device in which the second coordinate axis is set in the predetermined space.
With more
The conversion rule generating means is
Based on the read image, the position of the other information processing device on the first coordinate axis is estimated, and based on the estimation result, the coordinate axis in the predetermined space is set from the first coordinate axis as the rule. Generate a transformation matrix to convert to the second coordinate axis,
The information processing device according to claim 1. A coordinate conversion means for converting a coordinate axis in the predetermined space from the first coordinate axis to the second coordinate axis using the transformation matrix.
The information processing apparatus according to claim 2. In the information processing method executed by an information processing device that exchanges information by sharing a predetermined space with another information processing device.
A coordinate setting step for setting the first coordinate axis in the predetermined space, and
A conversion rule generation step for generating a rule for converting the first coordinate axis into a second coordinate axis used by the other information processing apparatus in the predetermined space, and a conversion rule generation step.
Information processing methods including. A computer that controls an information processing device that exchanges information by sharing a predetermined space with other information processing devices.
A coordinate setting step for setting the first coordinate axis in the predetermined space, and
A conversion rule generation step for generating a rule for converting the first coordinate axis into a second coordinate axis used by the other information processing apparatus in the predetermined space, and a conversion rule generation step.
A program that executes control processing including. In an information processing device that shares a predetermined space with other information processing devices and exchanges information.
A coordinate setting means for setting the first coordinate axis in the predetermined space, and
In the other information processing device in which the second coordinate axis is set in the predetermined space, the position of the information processing device on the second coordinate axis is estimated by being read, and the second coordinate axis is estimated based on the estimation result. An image generation means for generating image data used for generating a rule for converting the first coordinate axis into the first coordinate axis, and an image generation means for generating the image data based on the first coordinate axis.
A display control means for executing control to display the image generated as the data by the image generation means, and a display control means.
Information processing device equipped with. In the information processing method executed by an information processing device that exchanges information by sharing a predetermined space with another information processing device.
A coordinate setting step for setting the first coordinate axis in the predetermined space, and
In the other information processing device in which the second coordinate axis is set in the predetermined space, the position of the information processing device on the second coordinate axis is estimated by being read, and the second coordinate axis is estimated based on the estimation result. An image generation step of generating image data used for generating a rule (for example, a transformation matrix) for converting the first coordinate axis based on the first coordinate axis.
A display control step that executes control for displaying the image generated as the data in the processing of the image generation step, and a display control step.
Information processing methods including. A computer that controls an information processing device that exchanges information by sharing a predetermined space with other information processing devices.
A coordinate setting step for setting the first coordinate axis in the predetermined space, and
In the other information processing device in which the second coordinate axis is set in the predetermined space, the position of the information processing device on the second coordinate axis is estimated by being read, and the second coordinate axis is estimated based on the estimation result. The image generation step of generating the image data used for generating the rule for converting the first coordinate axis into the first coordinate axis based on the first coordinate axis.
A display control step that executes control for displaying the image generated as the data in the processing of the image generation step, and a display control step.
A program that executes control processing including.

Images