JP2021047516A - Information processing device, coordinate conversion system, coordinate conversion method, and coordinate conversion program - Google Patents

Abstract

To provide an information processing device, a coordinate conversion system, a coordinate conversion method, and a coordinate conversion program capable of accurately aligning a coordinate system in a real space and a coordinate system possessed by a device by using a marker arranged in the real space.SOLUTION: In a system where a plurality of AR devices 12 are connected with a server via a network, the AR device estimates a position and an attitude of a coordinate marker viewed from the AR device 12 which has photographed the coordinate marker, on the basis of the photographed image obtained by a camera 24 photographing the coordinate marker as a set of plurality of markers 32A and 32B arranged in a real space with respectively different directions. The AR device 12 converts the coordinate system in the real space to a unique coordinate system possessed by the AR device on the basis of the position of the coordinate marker in the real space and the position and the attitude of the coordinate marker viewed from the AR device 12.SELECTED DRAWING: Figure 4

Description

The present invention relates to an information processing device, a coordinate conversion system, a coordinate conversion method, and a coordinate conversion program.

Nowadays, an image showing a real space (hereinafter referred to as "real space") captured by a camera and a virtual object existing in a virtual space (hereinafter referred to as "virtual space") are superimposed to be virtual in the real space. Technological development related to Augmented Reality (AR), which makes an object appear on the display as if it exists, is in progress.

In order to realize AR, it is necessary to match the coordinate system of the real space where the AR device exists and the coordinate system of the virtual space where the virtual object exists. The coordinate system of this virtual space is, that is, the coordinate system of the AR device (hereinafter referred to as "unique coordinate system").

However, when the same AR is executed simultaneously by a plurality of AR devices, it is difficult to easily share the coordinate system between different AR devices and platforms. For example, on a specific platform, image feature points in real space (hereinafter referred to as "spatial recognition feature points") are defined and registered on the server, and other AR devices that use the same platform are recognized by themselves. The spatial coordinate system for performing AR is shared by determining whether or not the spatial recognition feature point matches the registered spatial recognition feature point.

However, methods for sharing the spatial coordinate system, such as detection methods for these spatial recognition feature points, registration processing, and search for similar data, cannot be commonly used by any device. Since different AR devices and platforms have different unique coordinate systems, the above methods may not be used in common.

Here, there is a method of arranging a marker in the real space as a space recognition feature point and matching the coordinate system of the real space with the original coordinate system with the marker as a reference.

For example, as disclosed in Patent Document 1, a marker arranged at the origin of the real space and indicating the direction of the origin and the coordinate axis is detected, and the coordinate value and the orientation of the marker in the coordinate system in the virtual space of the detected marker are determined. A method of converting a coordinate system in spatial data in real space into a coordinate system in virtual space (original coordinate system) is disclosed.

Japanese Unexamined Patent Publication No. 2019-79439

In the method disclosed in Patent Document 1, a marker arranged in a real space is imaged by a camera mounted on an AR device, and its position and posture are estimated. However, a general camera mounted on an AR device is a monocular camera, and depending on the positional relationship between the marker and the AR device (camera), there are positions and orientations of the markers that are difficult to accurately detect with the monocular camera. For example, it is difficult for a monocular camera to detect a distance in the depth direction of a subject. Further, depending on the angle formed by the marker and the AR device, it may be difficult for the monocular camera to correctly detect the posture (roll angle, pitch angle, yaw angle) of the marker.

For this reason, the accuracy of estimating the position and orientation of the marker by imaging with a monocular camera is not sufficient, and there are cases where the alignment between the coordinate system in the real space and the original coordinate system cannot be made highly accurate. In addition, if the markers placed in the real space can be used to accurately align the coordinate system in the real space with the original coordinate system, it is possible to execute the same AR on a plurality of AR devices at the same time. ..

Therefore, in view of the above background, the present invention makes it possible to accurately align the coordinate system of the real space with the coordinate system of the device by using the markers arranged in the real space. It is an object of the present invention to provide a coordinate conversion system, a coordinate conversion method, and a coordinate conversion program.

The information processing apparatus of the present invention is viewed from an apparatus that has imaged the coordinate markers based on an image captured by the imaging means with a set of coordinate markers arranged in different directions in a real space. Based on the estimation means for estimating the position and orientation of the coordinate marker, the position of the coordinate marker in the coordinate system in real space, and the position and orientation of the coordinate marker as seen from the device, the coordinate system in real space is determined. A conversion means for converting to the coordinate system of the device is provided.

According to this configuration, a plurality of markers are arranged in the real space in different directions to form a set of coordinate markers. Then, the coordinate marker is imaged by the imaging means in order to perform coordinate conversion between the coordinate system of the device and the coordinate system of the real space. Therefore, even if there is a marker whose posture or position is difficult to be accurately detected from the captured image, the posture or position which is difficult to detect can be detected by another marker by capturing another marker having a different orientation. Is possible. That is, the plurality of markers that serve as coordinate markers are in a complementary relationship with each other in order to accurately estimate the position and orientation of the markers. Therefore, according to this configuration, it is possible to more accurately align the coordinate system of the device with the coordinate system of the real space by using the markers arranged in the real space.

According to the information processing apparatus of the present invention, it may be determined for each of the markers which of the parameters of the position and orientation of the coordinate marker is used for estimation.

Parameters such as the distance, roll angle, pitch angle, and yaw angle that define the position and orientation of the coordinate marker differ in ease of estimation depending on the orientation of the marker. According to this configuration, which parameter is used for estimation is determined for each extracted marker, so that the position and orientation of the coordinate marker can be estimated with higher accuracy.

According to the information processing apparatus of the present invention, the coordinate marker may include the marker arranged in a horizontal plane and the marker arranged in a vertical plane orthogonal to the horizontal plane.

According to this configuration, by arranging the markers on the horizontal plane and the vertical plane, the orientations differ by about 90 °, and the markers have an appropriate complementary relationship when estimating the position and orientation of the coordinate markers, so that they are arranged in the real space. It is possible to more accurately align the coordinate system of the device using the marker with the coordinate system in the real space.

According to the information processing apparatus of the present invention, the marker may be represented by an identifier indicating the top-bottom direction.

According to this configuration, the top and bottom of the marker can be easily recognized.

According to the information processing apparatus of the present invention, the marker may be represented by an identifier indicating a position in the coordinate system in the real space.

According to this configuration, since the position of the marker in the real space can be determined by recognizing the marker, it is not necessary to store the position of the marker in another information processing device such as a server in order to determine the position of the marker. ..

According to the coordinate conversion system of the present invention, a coordinate marker in which a plurality of markers arranged in different directions in a real space are set, and a device including an imaging means for imaging the coordinate marker are provided, and the imaging means is provided. An estimation means for estimating the position and orientation of the coordinate marker as seen from the device based on the captured image captured by the device, the position of the coordinate marker in the coordinate system in real space, and the coordinate marker as seen from the device. A conversion means for converting the coordinate system in the real space into the coordinate system possessed by the device based on the position and the orientation is provided.

The coordinate conversion method of the present invention is viewed from a device that has imaged the coordinate markers based on an image captured by the imaging means with a set of a plurality of markers arranged in different directions in the real space. The coordinate system in the real space is based on the first step of estimating the position and orientation of the coordinate marker, the position of the coordinate marker in the coordinate system in real space, and the position and orientation of the coordinate marker as seen from the device. Has a second step of converting the device into the coordinate system of the device.

The coordinate conversion program of the present invention is a device that captures the coordinate markers based on an image captured by the imaging means with the computer as a set of a plurality of markers arranged in different directions in the real space. Based on the estimation means for estimating the position and orientation of the coordinate marker as seen from the above, the position of the coordinate marker in the coordinate system in real space, and the position and orientation of the coordinate marker as seen from the device, the real space It functions as a conversion means for converting the coordinate system into the coordinate system possessed by the device.

According to the present invention, it is possible to accurately align the coordinate system in the real space with the coordinate system possessed by the device by using the markers arranged in the real space.

It is a schematic block diagram of the AR system of this embodiment. It is a schematic diagram which shows the coordinate marker of this embodiment. It is a figure which shows the marker of this embodiment. It is a schematic diagram which shows the position and posture of the horizontal marker and the vertical marker with respect to the camera of this embodiment. It is a functional block diagram concerning the coordinate conversion function of this embodiment. It is a flowchart which shows the flow of the coordinate conversion process of this embodiment. It is a schematic diagram which shows the arrangement position of the coordinate marker of another embodiment.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. It should be noted that the embodiments described below show an example of the case where the present invention is carried out, and the present invention is not limited to the specific configuration described below. In carrying out the present invention, a specific configuration according to the embodiment may be appropriately adopted.

FIG. 1 is a schematic configuration diagram of the AR (Augmented Reality) system 10 of the present embodiment.

As shown in FIG. 1, the AR system 10 is composed of a plurality of AR devices 12 and a server 14. Information can be transmitted and received between the plurality of AR devices 12 and the server 14 via the communication line 16. The communication line 16 is a wide area communication line provided by an electric power company, a premises communication network such as a LAN (Local Area Network), or the like, and may be either a wired line or a wireless line.

The AR device 12 is a portable information processing device including a calculation processing unit 20, a display 22, and a camera 24, and is, for example, a smartphone, a laptop computer, a head-mounted display, or the like. The plurality of AR devices 12 may be different information processing devices. Further, the AR system 10 may have one AR device 12.

The arithmetic processing unit 20 is composed of a CPU (Central Processing Unit), a RAM (Random Access Memory), a ROM (Read Only Memory), a computer-readable storage medium (for example, a semiconductor memory), and the like. Then, as an example, a series of processes for realizing various functions are stored in a storage medium or the like in the form of a program, and the CPU reads this program into a RAM or the like to execute information processing / arithmetic processing. As a result, various functions are realized. The program is installed in a ROM or other storage medium in advance, is provided in a state of being stored in a computer-readable storage medium, or is distributed via a wired or wireless communication means. Etc. may be applied. The storage medium of the present embodiment stores a program for realizing AR (hereinafter referred to as "AR application").

The display 22 displays an image corresponding to the image processing by the arithmetic processing unit 20, and may be, for example, a touch panel display.

The camera 24 is, for example, a monocular camera, and images a subject according to an operation by a user of the AR device 12. The image captured by the camera 24 is appropriately stored in the RAM or storage medium included in the arithmetic processing unit 20 and displayed on the display 22. Further, while the AR device 12 is executing AR, the camera 24 continuously captures images, and the captured images are continuously displayed on the display 22.

Further, the AR device 12 includes various sensors such as an acceleration sensor and a GPS (Global Positioning System) sensor. Then, the AR device 12 estimates its own posture and position (motion estimation) by these sensors, and stores and updates its own position and posture based on the estimation result. Further, the AR device 12 includes an operation input means for accepting an operation by the user. The operation input means may be, for example, the touch panel display described above.

The server 14 transmits and receives various data for realizing AR to and from the AR device 12. The server 14 transmits, for example, data indicating a virtual object or the like displayed on the display 22 of the AR device 12 to the AR device 12.

In the AR system 10 of the present embodiment, the same virtual object is displayed on the display 22 of each AR device 12 at the same timing for the plurality of AR devices 12 constituting the system, and the displayed virtual object is displayed by the user. Allows you to operate. Then, when a certain user operates a virtual object displayed on the AR device 12, the operation on the virtual object is also reflected in the AR device 12 of another user.

Here, when AR is executed, since the virtual object is superimposed on the image captured by the camera 24 and displayed on the display 22, it is necessary to perform coordinate conversion between the virtual space and the real space. Therefore, in the AR system 10 of the present embodiment, the AR marker, which is an image marker, is arranged in the real space, and the coordinates of the virtual space and the real space are converted based on the coordinates indicating the position of the AR marker.

Here, when the camera 24 is a monocular camera, there is a position or orientation (orientation) of the AR marker that is difficult to accurately detect with the monocular camera depending on the positional relationship between the AR marker and the AR device (camera 24). Therefore, in the present embodiment, the position and orientation of the AR marker are estimated with high accuracy by using a plurality of markers that complement each other as a set.

FIG. 2 is a schematic view showing a coordinate marker 30 which is an AR marker of the present embodiment. The coordinate marker 30 of the present embodiment is a set of a plurality of markers 32A and 32B arranged in different directions in the real space. As an example, the coordinate marker 30 is composed of a horizontal marker 32A arranged on a horizontal plane and a vertical marker 32B arranged on a vertical plane orthogonal to the horizontal plane. Hereinafter, when it is not necessary to distinguish between the markers 32A and 32B, it is simply referred to as "marker 32".

When the coordinate marker 30 is arranged indoors, for example, the horizontal plane is a plane parallel to the floor surface 34 or the floor surface 34, and the vertical plane is a wall surface 36 or a plane parallel to the wall surface 36 erected on the floor surface 34. The horizontal marker 32A of the present embodiment is arranged on the floor surface 34 as an example, and the vertical marker 32B is arranged by being supported by a support portion 38 standing vertically with respect to the floor surface 34 as an example.

The arrangement of the plurality of markers 32 in different directions means that the normals 39 with respect to the surface of the markers 32 are arranged so as to intersect each other. As an example, the normal line 39 of the horizontal marker 32A and the normal line 39 of the vertical marker 32B intersect at 90 °.

The horizontal marker 32A and the vertical marker 32B of the present embodiment each have the same image as an example, but as will be described in detail later, their positions and postures can be determined by image recognition when they are arranged in the real space. A different image may be displayed as long as it is a simple image. It should be noted that an image having point symmetry, an image having a mirror image vertically or horizontally, or the like is not preferable because it is difficult to determine the posture (orientation). That is, each marker 32 is represented by an image in which the top-bottom direction (vertical direction or horizontal direction) can be recognized when an image is taken.

FIG. 3 is a diagram showing an example of the marker 32. The marker 32 has a black frame portion 40 having a rectangular black border on the periphery. The AR device 12 estimates the position and orientation of the coordinate marker 30 by imaging the marker 32 and determining the size and shape of the black frame portion 40. The number written inside the black frame portion 40 is a marker identifier 42 for recognizing that it is a marker 32. The marker identifier 42 shown in FIG. 3 is an example, and if the marker 32 can be identified, it does not have to be a number, and the number may be one. Further, the center position of the marker 32 is represented by, for example, the center identifier 43 indicated by a cross.

Further, on the marker 32 of the present embodiment, a direction identifier 44 indicating the vertical direction thereof is written. In the example of FIG. 3, a triangular shape is represented as a direction identifier 44 at one of the inner corners of the black frame portion 40. In the example of FIG. 3, the direction in which the direction identifier 44 is written is recognized as the upper left direction of the marker 32. The shape of the direction identifier 44 shown in FIG. 3 is an example, and is not limited to this.

Further, in the coordinate marker 30 of the present embodiment, a position identifier 46, which is an identifier indicating a position in the coordinate system in the real space (hereinafter, referred to as “real space position”), is described. The real space position is, for example, the position of the coordinate marker 30 based on a predetermined reference origin (0, 0, 0), and the arrangement position of the coordinate marker 30 may be used as the reference origin. The position identifier may be expressed as a two-dimensional code such as a QR code (registered trademark) as long as it can be recognized based on the captured image of the coordinate marker 30, or the coordinates (x) in the real space. , Y, z) may be written.

In the coordinate marker 30 of the present embodiment, although the position identifier 46 is indicated for each of the plurality of markers 32, the real space position indicated by the position identifier 46 is the same. That is, although the coordinate marker 30 is composed of a plurality of markers 32, there is only one real space position set as the coordinate marker 30.

Further, in the real space where the coordinate marker 30 is arranged, the reference direction is predetermined. The coordinate marker 30 is set to have an orientation with respect to the reference direction. More specifically, in the coordinate marker 30, the direction perpendicular to the surface of the vertical marker 32B is set as the front direction, and the direction (the angle formed by the upper surface direction and the reference direction) with respect to the reference direction in the front direction is set.

As described above, the arrangement position of the coordinate marker 30 is specified by the distance and the direction from the reference position in the real space, and these are indicated on the coordinate marker 30 as the position identifier 46.

FIG. 4 is a schematic view showing the positions and orientations of the horizontal marker 32A and the vertical marker 32B with respect to the camera 24. In FIG. 4, the direction perpendicular to the floor surface is the z-axis, the direction perpendicular to the z-axis and from the coordinate marker 30 toward the camera 24 is the x-axis, and the direction perpendicular to both the x-axis and the z-axis and toward the back of the paper is the y-axis. Is.

The position and orientation of the camera 24 with respect to the marker 32 are specified by how much the marker 32 is rotated about the z-axis and imaged, and how much the marker 32 is rotated about the x-axis or the y-axis. The rotation around the x-axis is also called a roll rotation, the rotation around the y-axis is also called a pitch rotation, and the rotation around the z-axis is also called a yaw rotation. Further, the relative position of the coordinate marker 30 with respect to the camera 24 is specified by the distance L between the camera 24 and the coordinate marker 30.

That is, the position and orientation of the coordinate marker 30 of the present embodiment are specified by, for example, the x-coordinate position, the y-coordinate position, the z-coordinate position, the roll angle, the pitch angle, and the yaw angle. Assuming that the floor surface 34 is horizontal without distortion, for example, the roll rotation and pitch rotation of the horizontal marker 32A and the pitch rotation of the vertical marker 32B may not occur.

Then, in the present embodiment, as will be described in detail later, the position and orientation of the coordinate marker 30 as seen from the AR device 12 (camera 24) that has imaged the coordinate marker 30 are estimated.

Here, it is difficult for the monocular camera to accurately detect the position (depth direction) of the subject existing in front. That is, it is difficult to accurately estimate the distance L based on the vertical marker 32B located in front of the camera 24. On the other hand, the estimated distance L based on the horizontal marker 32A arranged at an angle with respect to the camera 24 is more accurate. The height H is the length from the horizontal plane to the center position of the vertical marker 32B, and is estimated by the vertical marker 32B.

Further, regarding the posture of the marker 32 with respect to the camera 24, the yaw angle is more accurate in the estimation based on the horizontal marker 32A, while the pitch angle is more accurate in the estimation based on the vertical marker 32B. Further, the roll angle is more accurate when estimated based on the horizontal marker 32A. As described above, the parameters such as the distance L, the roll angle, the pitch angle, and the yaw angle extracted for estimating the position and orientation of the coordinate marker 30 differ in their suitability depending on the arrangement position of the marker 32.

Therefore, in the present embodiment, for each of the markers 32 arranged in different directions, which of the parameters of the position and orientation of the coordinate marker 30 is used for estimation is determined. The marker 32 to be extracted may be determined depending on the place where the coordinate marker 30 is arranged (the degree of horizontality, the color scheme and the feature of the background).

FIG. 5 is a functional block diagram relating to the coordinate conversion function of the AR device 12. The AR device 12 includes a camera control unit 50, a marker recognition unit 52, a position / orientation estimation unit 54, a coordinate conversion unit 56, and an image display control unit 58. In the following description, the coordinate system of the virtual space is also referred to as a unique coordinate system which is a coordinate system of each AR device 12.

The camera control unit 50 executes the activation of the camera 24 mounted on the AR device 12 and the acquisition of the captured image based on a user operation or an instruction from an application installed on the AR device 12.

The marker recognition unit 52 recognizes the marker 32 by image processing from the captured image acquired by the camera 24. The recognition method of the marker 32 may be a conventionally known method. For example, the captured image is binarized and the image region recognized as the marker 32 is extracted based on various identifiers. Further, the marker recognition unit 52 recognizes the marker 32 and recognizes the position and orientation of the marker 32 from the reference origin in the real space based on the position identifier 46 indicated on the marker 32.

Further, the marker recognition unit 52 recognizes whether the marker 32 included in the captured image is the horizontal marker 32A or the vertical marker 32B based on the direction identifier 44. For example, the marker recognition unit 52 recognizes that the marker 32 whose direction identifier 44 is located upward is the vertical marker 32B, and the marker 32 whose direction identifier 44 is located in the depth direction of the captured image is the horizontal marker 32A. Recognize that.

The position / orientation estimation unit 54 estimates the position and orientation of the coordinate marker 30 as seen from the AR device 12 in which the coordinate marker 30 is imaged, based on the image captured by the camera 24 of the horizontal marker 32A and the vertical marker 32B. The estimation method may be a conventionally known method. For example, the relative position and the relative posture of the coordinate marker 30 and the camera 24 are estimated based on the size and shape of the black frame portion 40 of the horizontal marker 32A and the vertical marker 32B.

In the present embodiment, as described above, the parameters (x-coordinate position, y-coordinate position, z-coordinate position, roll angle) to be extracted for estimating the position and orientation of the coordinate marker 30 for each of the horizontal marker 32A and the vertical marker 32B , Pitch angle, and yaw angle). Therefore, the position / orientation estimation unit 54 of the present embodiment extracts the parameters defined from the horizontal marker 32A and the vertical marker 32B. The position / orientation estimation unit 54 acquires the relative position and relative orientation of the AR device 12 and the coordinate marker 30 in its own coordinate system.

The coordinate conversion unit 56 has a coordinate system (original) in which the AR device has a real space coordinate system based on the position of the coordinate marker 30 in the real space coordinate system and the position and orientation of the coordinate marker 30 as seen from the AR device 12. Convert to coordinate system). More specifically, the coordinate conversion unit 56 converts the real space coordinate system into the original coordinate system based on the coordinates of the coordinate marker 30 in the original coordinate system and the real space position set in the coordinate marker 30. (Hereinafter, "coordinate conversion setting value") is calculated.

The image display control unit 58 controls the image display on the display 22. The image display control unit 58 of the present embodiment has a function of superimposing and displaying a virtual object on a real space image captured by the camera 24. At this time, the image display control unit 58 converts the coordinate system in the real space into the original coordinate system using the coordinate conversion set value, and superimposes the virtual object on the display 22 with respect to the image in the real space captured by the camera 24. indicate.

By such processing, the virtual object is superimposed and displayed on the image captured by the camera 24 on the display 22 of the AR device 12, and the instruction for the user to operate the displayed virtual object is replaced with the original coordinate system for each AR device 12. It becomes possible to perform operations on virtual objects. Further, even if different AR devices 12 are used, the same virtual object can be handled at the same time in the same virtual space by making the reference origin in the real space the same.

FIG. 6 is a flowchart showing the flow of the coordinate conversion process of the present embodiment. The coordinate conversion process is executed by the arithmetic processing unit 20 when, for example, the AR application is started by the AR device 12.

First, in step 100, the camera control unit 50 activates the camera 24 and starts imaging by the camera 24. The image captured by the camera 24 is displayed on the display 22 of the AR device 12, and can be confirmed by the user.

In the next step 102, the marker recognition unit 52 determines whether or not the coordinate marker 30 is included in the angle of view (see FIG. 4) imaged by the camera 24, and in the case of an affirmative determination, the process proceeds to step 104. In the case of a negative determination, the state waits until the coordinate marker 30 is included in the angle of view.

When the affirmative determination is made in step 102, the horizontal marker 32A and the vertical marker 32B are both included in the same angle of view. Therefore, the user moves so that the horizontal marker 32A and the vertical marker 32B are included in the same angle of view. Further, even if the horizontal marker 32A and the vertical marker 32B are included in the same angle of view, if the distance L from the user is long, the marker recognition unit 52 recognizes the horizontal marker 32A and the vertical marker 32B. It may not be possible. In such a case, the user moves in a direction approaching the horizontal marker 32A and the vertical marker 32B so that the marker recognition unit 52 can recognize the horizontal marker 32A and the vertical marker 32B.

In step 104, the camera 24 is controlled so that the camera control unit 50 acquires the captured image, and the captured image is stored in the storage unit.

In the next step 106, the position / orientation estimation unit 54 acquires the relative position and relative orientation of the AR device 12 and the coordinate marker 30 in the original coordinate system based on the stored captured image.

In the next step 108, coordinate conversion for converting the coordinates in the real space to the original coordinate system based on the coordinates in the original coordinate system of the coordinate marker 30 and the reference origin in the real space set in the coordinate marker 30. The coordinate conversion unit 56 calculates the set value.

As described above, in the coordinate marker 30 of the present embodiment, a plurality of markers 32 are arranged in different directions in the real space. Then, the AR device 12 of the present embodiment captures the coordinate marker 30 by the camera 24 in order to perform coordinate conversion between the original coordinate system of the AR device 12 and the coordinate system in the real space. Therefore, even if there is a marker 32 whose posture or position is difficult to be accurately detected from the captured image, another marker 32 having a different orientation is imaged, so that the posture or position which is difficult to detect can be detected by the other marker 32. It becomes possible to detect. Therefore, according to the AR device 12 of the present embodiment, the coordinate marker 30 arranged in the real space can be used to more accurately align the original coordinate system of the AR device 12 with the coordinate system in the real space. it can.

Further, as a result, each AR device 12 constituting the AR system 10 performs coordinate conversion between the real space and the original coordinate space based on the common coordinate marker 30, which results in accurate coordinate conversion. Therefore, the AR system Even if each AR device 12 constituting the 10 is a different device, the user of each AR device 12 can share the same virtual space regardless of the device configuration.

Although the present invention has been described above using the above-described embodiment, the technical scope of the present invention is not limited to the scope described in the above-described embodiment. Various changes or improvements can be made to the above embodiments without departing from the gist of the invention, and the modified or improved forms are also included in the technical scope of the present invention.

For example, the horizontal marker 32A and the vertical marker 32B may be represented by an identifier indicating a parameter to be extracted for each marker 32.

Further, in the above-described embodiment, the mode in which the position identifier 46 indicating the real space position or the orientation in the real space is indicated on the coordinate marker 30 has been described, but the present invention is not limited thereto. For example, when the server 14 stores the position identification information indicating the real space position of the coordinate marker 30 and the orientation in the real space and the AR device 12 recognizes the coordinate marker 30, the position identification information may be read from the server 14.

Further, in the above embodiment, the mode of recognizing whether the marker 32A is the horizontal marker or the vertical marker 32B is described based on the direction identifier 44 written on the marker 32, but the present invention is not limited thereto. For example, each marker 32 may be labeled with identification information indicating whether it is a horizontal marker 32A or a vertical marker 32B.

Further, the placement position of the marker 32 may be estimated by the movement of the AR device 12. For example, when the marker 32 is imaged with the AR device 12 facing downward, the marker 32 is presumed to be the horizontal marker 32A. Further, when the marker 32 is imaged with the AR device 12 oriented in the horizontal direction, the marker 32 is presumed to be the vertical marker 32B.

Further, in the present embodiment, the form in which the AR device 12 includes the marker recognition unit 52, the position / orientation estimation unit 54, and the coordinate conversion unit 56 has been described, but the present invention is not limited thereto. For example, the server 14 may include a marker recognition unit 52, a position / orientation estimation unit 54, and a coordinate conversion unit 56. In the case of this form, the captured image including the coordinate marker 30 captured by the camera 24 included in the AR device 12 is transmitted from the AR device 12 to the server 14. Then, the marker recognition unit 52, the position / orientation estimation unit 54, and the coordinate conversion unit 56 included in the server 14 calculate the coordinate conversion set value, and the coordinate conversion set value is transmitted from the server 14 to the AR device 12.

Further, the AR device 12 is a head-mounted display including a display 22 and a camera 24, the head-mounted display is connected to another information processing device (for example, a desktop personal computer or the like), and the information processing device is the marker recognition unit 52. , The position / orientation estimation unit 54, the coordinate conversion unit 56, and the like may be provided. In this form, the information processing device and the server 14 appropriately transmit and receive data.

Further, in the above embodiment, the embodiment in which the coordinate marker 30 is composed of the horizontal marker 32A and the vertical marker 32B has been described, but the present invention is not limited thereto. For example, as shown in FIG. 7, the coordinate marker 30 may be composed of three markers 32 such as the horizontal marker 32A, the vertical marker 32B, and the vertical marker 32C. The vertical marker 32C is arranged on a surface orthogonal to the horizontal plane on which the horizontal marker 32A is arranged and the vertical surface on which the vertical marker 32B is arranged, for example, a wall surface 36B orthogonal to the wall surface 36A at 90 °. As a result, an appropriate marker 32 can be selected for each parameter used for estimating the position and orientation of the coordinate marker 30, and the estimation accuracy of the position and orientation of the coordinate marker 30 can be further improved.

Further, in the present embodiment, the embodiment in which the present invention is applied to AR has been described, but the present invention is not limited to this, and may be applied to MR (Mixed Reality).

For example, it is useful as a technology applied to AR technology and MR technology in which a real space and a virtual space are combined and displayed on a display.

12 AR equipment (information processing equipment)
24 camera (imaging means)
30 Coordinate marker 32A Horizontal marker 32B Vertical marker 54 Position / orientation estimation unit (estimation means)
56 Coordinate conversion unit (conversion means)

Claims (8)

Based on the image captured by the imaging means with a set of coordinate markers arranged in different directions in the real space, the position and orientation of the coordinate markers as seen from the device that imaged the coordinate markers. Estimating means to estimate and
A conversion means for converting the real space coordinate system into the coordinate system possessed by the device based on the position of the coordinate marker in the real space coordinate system and the position and orientation of the coordinate marker as seen from the device.
Information processing device equipped with. The information processing apparatus according to claim 1, wherein for each of the markers, which of the parameters of the position and orientation of the coordinate marker is used for estimation is determined. The information processing apparatus according to claim 1 or 2, wherein the coordinate marker includes the marker arranged on a horizontal plane and the marker arranged on a vertical plane orthogonal to the horizontal plane. The information processing device according to any one of claims 1 to 3, wherein the marker is an identifier indicating the top-bottom direction. The information processing apparatus according to any one of claims 1 to 4, wherein the marker is represented by an identifier indicating a position in the coordinate system in the real space. A coordinate marker that is a set of multiple markers arranged in different directions in the real space,
A device including an imaging means for imaging the coordinate marker, and
With
An estimation means for estimating the position and orientation of the coordinate marker as seen from the device based on the captured image captured by the imaging means, and an estimation means.
A conversion means for converting the real space coordinate system into the coordinate system possessed by the device based on the position of the coordinate marker in the real space coordinate system and the position and orientation of the coordinate marker as seen from the device.
Coordinate conversion system with. Based on the image captured by the imaging means with a set of coordinate markers arranged in different directions in the real space, the position and orientation of the coordinate markers as seen from the device that imaged the coordinate markers. The first step to estimate and
A second step of converting the real space coordinate system into the coordinate system of the device based on the position of the coordinate marker in the real space coordinate system and the position and orientation of the coordinate marker as seen from the device.
Coordinate transformation method having. Computer,
Based on the image captured by the imaging means with a set of coordinate markers arranged in different directions in the real space, the position and orientation of the coordinate markers as seen from the device that imaged the coordinate markers. Estimating means to estimate and
A conversion means for converting the real space coordinate system into the coordinate system possessed by the device based on the position of the coordinate marker in the real space coordinate system and the position and orientation of the coordinate marker as seen from the device.
A coordinate conversion program that works.

Images