WO2021177132A1 - Information processing device, information processing system, information processing method, and program - Google Patents

Abstract

An information processing device according to the present technology is provided with a control unit. The control unit estimates an own position in global coordinates corresponding to a real space, acquires coordinate information about the location of a first AR-displayable virtual object that is set in the global coordinates for a real object in the real space by another device sharing the global coordinates, sets the location of the first virtual object in the global coordinates on the basis of the own location and the coordinate information, calculates the location of the real object in the global coordinates on the basis of image information, and corrects the location of a second virtual object, which is commonly AR-displayed with the other device, on the basis of a location relationship between the first virtual object and the real object.

Description

Information processing equipment, information processing system, information processing method and program

This technology relates to a technology for correcting the position of a virtual object that is commonly displayed in AR among a plurality of AR (Augmented Reality) devices.

Conventionally, a technique for displaying a common virtual object among a plurality of AR devices in AR at the same position has been known (see, for example, Patent Document 1 below).

In order to accurately display a common virtual object in the same position among a plurality of AR devices, it is necessary for each of the plurality of AR devices to accurately estimate the self-position. In the self-position estimation, the self-position of the AR device is estimated by comparing the feature point cloud extracted from the image information captured by the AR device with the information of the feature point group included in the map information. The map information used for self-position estimation includes a method of creating it in advance and a method of creating it at the same time as self-position estimation without creating it in advance. The method of creating map information at the same time as self-position estimation is generally called SLAM (Simultaneous Localization and Mapping).

In either of the above two methods, the estimated self-position may deviate from the actual position in the real space, resulting in an error. Such an error causes a misalignment of a common virtual object displayed in AR in a plurality of AR devices.

Japanese Unexamined Patent Publication No. 2012-168646

There is a need for a technology that can accurately display a common virtual object in AR at the same position among multiple AR devices.

In view of the above circumstances, the purpose of this technology is to provide a technology that can accurately display a common virtual object in AR at the same position among a plurality of AR devices.

The information processing device according to the present technology includes a control unit.
The control unit estimates its own position in the global coordinate system corresponding to the real space.
Another device sharing the global coordinate system acquires the coordinate information of the position of the first virtual object that can be AR-displayed set for the real object in the real space in the global coordinate system.
Based on the self-position and the coordinate information, the position of the first virtual object is set in the global coordinate system.
Based on the image information, the position of the real object in the global coordinate system is calculated.
Based on the positional relationship between the first virtual object and the real object, the position of the second virtual object that is AR-displayed in common with the other device is corrected.

This makes it possible to AR-display a common virtual object at the same position among a plurality of AR devices.

In the information processing device, the control unit may correct the position of the second virtual object based on the difference between the position of the first virtual object and the position of the real object.

In the information processing device, the control unit sets the position of the first virtual object with respect to the real object in the global coordinate system based on the image information, and the first virtual object based on the coordinate information. The position of the second virtual object may be corrected based on the difference between the position of the virtual object and the position of the first virtual object based on the image information.

In the information processing device, the control unit may calculate a correction value based on the difference and correct the position of the second virtual object by the correction value.

In the information processing device, the control unit may move the position of the second virtual object according to the correction value to correct the position of the second virtual object.

In the information processing device, the control unit may rotate the second virtual object to correct the position of the second virtual object according to the correction value.

In the information processing device, the control unit may change the degree of correction by the correction value.

In the information processing device, the control unit may change the degree of correction by the correction value according to the distance between the information processing device and the real object when the correction value is calculated. good.

In the information processing device, the control unit determines the distance between the other device and the real object when the other device sets the position of the first virtual object with respect to the real object. The degree of correction by the correction value may be changed accordingly.

In the information processing device, the other device may set the position of the first virtual object so that the first virtual object overlaps the real object and can be AR-displayed.

In the information processing device, the other device may set the position of the first virtual object so that the first virtual object can be AR-displayed in the vicinity of the real object.

In the information processing device, the other device may display the first virtual object in AR.

In the information processing device, the control unit may display the first virtual object in AR.

In the information processing device, the other device is an object in which the first virtual object is located from among a plurality of real objects existing in the real space based on the image information acquired by the other device. You may select the real object that becomes.

In the information processing device, the other device may select the real object satisfying a predetermined condition as the real object to which the first virtual object is located.

In the information processing device, the predetermined condition may be that the real object has a specific shape.

In the information processing apparatus, the predetermined condition is that the real object has a three-dimensional shape that is substantially uniquely specified regardless of the direction in which the real object is viewed. good.

The information processing system according to the present technology includes an information processing device and another device.
The information processing device has a control unit.
The control unit estimates its own position in the global coordinate system corresponding to the real space.
Another device sharing the global coordinate system acquires the coordinate information of the position of the first virtual object that can be AR-displayed set for the real object in the real space in the global coordinate system.
Based on the self-position and the coordinate information, the position of the first virtual object is set in the global coordinate system.
Based on the image information, the position of the real object in the global coordinate system is calculated.
Based on the positional relationship between the first virtual object and the real object, the position of the second virtual object that is AR-displayed in common with the other device is corrected.

The information processing method related to this technology estimates the self-position in the global coordinate system corresponding to the real space, and
Another device sharing the global coordinate system acquires the coordinate information of the position of the first virtual object that can be AR-displayed set for the real object in the real space in the global coordinate system.
Based on the self-position and the coordinate information, the position of the first virtual object is set in the global coordinate system.
Based on the image information, the position of the real object in the global coordinate system is calculated.
This includes correcting the position of the second virtual object that is AR-displayed in common with the other device based on the positional relationship between the first virtual object and the real object.

The program related to this technology estimates the self-position in the global coordinate system corresponding to the real space, and
Another device sharing the global coordinate system acquires the coordinate information of the position of the first virtual object that can be AR-displayed set for the real object in the real space in the global coordinate system.
Based on the self-position and the coordinate information, the position of the first virtual object is set in the global coordinate system.
Based on the image information, the position of the real object in the global coordinate system is calculated.
Based on the positional relationship between the first virtual object and the real object, the computer is made to execute a process of correcting the position of the second virtual object that is AR-displayed in common with the other device.

It is a figure which shows the information processing system which concerns on 1st Embodiment of this technology. It is a perspective view which shows HMD in an information processing system. It is a block diagram which shows the internal structure of an HMD. It is a timing chart which shows the processing of an information processing system. It is a timing chart which shows the processing of an information processing system. It is a figure which shows the state when the user who wears a 1st HMD and the user who wears a 2nd HMD are looking at a real object for correction. In the example shown in FIG. 6, it is a figure which shows an example of the image information when the correction real object was imaged by the image pickup unit of the 1st HMD. It is a figure which shows the information of the feature point cloud extracted from the image information shown in FIG. 7. It is a figure which shows an example when the coordinate position of the correction virtual object is set with respect to the correction real object. It is a figure which shows the state when the position of the correction virtual object is set (AR is displayed) by the 2nd HMD. It is a figure which shows the state when the correction real object was imaged by the image pickup part of the 2nd HMD. It is a figure which shows the information of the feature point cloud extracted from the image information shown in FIG. It is a figure which shows the difference between the position and posture of a virtual object for correction, and the position and posture of a real object for correction. It is a figure which shows an example when the coordinate position of the correction virtual object is set (when AR is displayed) with respect to the correction real object. It is a figure which shows the difference between the position and posture of the correction virtual object based on the coordinate information, and the position and posture of a correction virtual object based on image information.

Hereinafter, embodiments relating to the present technology will be described with reference to the drawings.

<< First Embodiment >>
<Overall system configuration and configuration of each part>
FIG. 1 is a diagram showing an information processing system 100 according to a first embodiment of the present technology. As shown in FIG. 1, the information processing system 100 includes a plurality of HMDs (Head Mounted Display) 10s and a server device 20.

In the first embodiment, as an example of an AR device (information processing device) capable of displaying a virtual object in AR, HMD10, which is used by being attached to a head, will be described as an example. In the example shown in FIG. 1, the number of HMD10s is two, but the number of HMD10s is not particularly limited as long as it is two or more.

"HMD10"
FIG. 2 is a perspective view showing the HMD 10. FIG. 3 is a block diagram showing an internal configuration of the HMD.

As shown in these figures, the HMD 10 includes an HMD main body 11, a control unit 1, a storage unit 2, a display unit 3, an imaging unit 4, an inertial sensor 5, an operation unit 6, and a communication unit 7. It has.

The HMD main body 11 is attached to the user's head and used. The HMD main body 11 is attached to the front portion 12, the right temple portion 13 provided on the right side of the front portion 12, the left temple portion 14 provided on the left side of the front portion 12, and the lower side of the front portion 12. It has a glass portion 15.

At least a part of the display unit 3 is provided in the glass unit 15. The display unit 3 is a display having light transmission (optical see-through display), and includes, for example, an OLED (Organic Light Emitting Diode) as a light source and a light guide plate. The display unit 3 can AR-display the virtual object according to the control of the control unit 1. The display unit 3 may adopt various forms such as a configuration using a half mirror and a retinal operation display. The light source of the display unit 3 may be provided on the front unit 12, the right temple unit 13, the left temple unit 14, or the like.

AR display means that the virtual object is displayed so as to be perceived as if it were a real object existing in the real space from the user's point of view. The display unit 3 may be a video see-through display. In this case, an image in which a virtual object is superimposed on the image captured by the imaging unit 4 is displayed on the display unit 3.

The image pickup unit 4 is, for example, a camera, and includes an image pickup element such as a CCD (Charge Coupled Device) sensor and a CMOS (Complemented Metal Oxide Semiconductor) sensor, and an optical system such as an imaging lens. The imaging unit 4 is provided outward on the outer surface of the front unit 12, images an actual object existing in the user's line-of-sight direction, and transfers the image information (depth information) obtained by the imaging to the control unit 1. Is output.

Two imaging units 4 are provided in the front unit 12 at predetermined intervals in the lateral direction. The location and number of the imaging units 4 can be changed as appropriate. As the image pickup unit 4, an infrared sensor may be used instead of the camera, or a combination of the camera and the infrared sensor may be used.

The inertial sensor 5 includes a 3-axis acceleration sensor that detects acceleration in the 3-axis direction and an angular velocity sensor that detects an angular velocity around the 3-axis. The inertial sensor 5 outputs the acceleration in the three-axis direction and the angular velocity around the three axes obtained by the detection to the control unit 1 as inertial information.

In the present embodiment, the detection axes of the inertial sensor 5 are three axes, but the detection axes may be one axis or two axes. Further, in the present embodiment, two types of sensors are used as the inertial sensor 5, but one type or three or more types of sensors may be used as the inertial sensor 5. Other examples of the inertial sensor 5 include a speed sensor, an angle sensor, and the like.

The operation unit 6 is, for example, various types of operation units such as a pressing type and a contact type, and detects an operation by the user and outputs the operation to the control unit 1. In the example shown in FIG. 2, the operation unit 6 is provided on the front side of the left temple unit 14, but the position where the operation unit 6 is provided is any position as long as it is easy for the user to operate. May be good.

The communication unit 7 is capable of wirelessly or wired communication with another HMD 10 and the server device 20.

The control unit 1 executes various operations based on various programs stored in the storage unit 2 and comprehensively controls each unit of the HMD 10. The control unit 1 includes a CPU (Central Processing Unit) 16, a VPU (Vision Processing Unit) 17, and a GPU (Graphics Processing Unit) 18.

The VPU 17 executes a process related to self-position estimation, a process of analyzing image information acquired by the imaging unit 4, and the like. Self-position estimation includes relocalization and motion tracking.

The relocalization estimates the current self-position and attitude in the global coordinate system based on the image and map information captured by the image pickup unit 4 immediately after the power is turned on to the HMD 10 or at a predetermined timing thereafter. It is a technology.

Motion tracking calculates the amount of change (movement) of self-position and posture for each minute time based on image information (and inertial information), and by sequentially adding this amount of change, the current self in the global coordinate system. It is a technique for estimating the position and posture.

In the relocalization, the VPU 17 first performs image processing on the image information acquired by the imaging unit 4 and extracts a feature point cloud from the image information. Then, the VPU 17 compares the extracted feature point cloud with the feature point cloud (or mesh information in which the feature point cloud is connected) included in the map information, thereby displaying its own position and its own position in the global coordinate system. Estimate the posture.

Relocation is executed immediately after the power is turned on, or when self-position estimation based on motion tracking fails. Further, the process of comparing the feature point cloud from the image information by the imaging unit 4 with the feature point cloud included in the map information is constantly executed, and when the matching of these feature point groups is successful, the relocalization is performed. It may be executed.

In motion tracking, the VPU 17 first performs image processing on the image information acquired by the imaging unit 4 and extracts a feature point cloud from the image information. Then, the VPU 17 calculates the change amount of the previous self-position and posture and the current self-position and posture by comparing the feature point group of the image information in the previous time with the feature point group of the image information in this time. The VPU 17 estimates the current self-position and attitude in the global coordinate system by adding this amount of change to the previous self-position and attitude.

Immediately after the relocalization is executed, the original self-position and posture to which the amount of change is added becomes the self-position and posture estimated by the re-localization.

In the description here, the case where the image information from the imaging unit 4 is used for motion tracking has been described, but the inertia information from the inertial sensor 5 may be used instead of the image information. Alternatively, both image information and inertial information may be used.

The CPU 16 executes a process of setting the position and display contents of the virtual object, a delay compensation process, and the like. In addition, the CPU 16 executes various processes related to the application commonly executed by each HMD.

The VPU 17 executes a process of controlling the AR display of the virtual object by the display unit 3. Specifically, the VPU 17 executes a process of calculating an image of a virtual object to be AR-displayed according to the user's viewpoint position and outputting it to the display unit 3.

In the present embodiment, the display of the display unit 3 is controlled so that the common virtual object is AR-displayed at the same position in the global coordinate system in the plurality of HMD10s.

As an example, a case where, for example, a survival game application in which a plurality of users shoot each other with a simulated gun is executed in a plurality of HMD10s will be described. In this case, when the user shoots a simulated gun, the virtual object of the ammunition is AR-displayed at the same position in the global coordinate system in each HMD 10.

In this embodiment, there are two types of virtual objects as virtual objects. The first type is a correction virtual object 31 (first virtual object), and the second type is a normal virtual object (second virtual object).

A normal virtual object is a common virtual object that is AR-displayed at the same position in the global coordinate system when a common application is executed in a plurality of HMD10s. For example, in the survival game example, the ammunition virtual object corresponds to this regular virtual object.

The correction virtual object 31 is a virtual object used to correct the AR display position so that a normal virtual object is displayed in the same position and posture in a plurality of HMD10s. The correction virtual object 31 may be AR-displayed, or may not be AR-displayed simply by setting the position in the global coordinate system. However, in the first embodiment, the correction virtual object 31 is AR-displayed. The case where it is displayed will be described.

Note that when the term is simply referred to as a virtual object in the description of each embodiment, it means a general term for a normal virtual object and a correction virtual object 31.

The storage unit 2 includes various programs required for processing of the control unit 1, a non-volatile memory for storing various data, and a volatile memory used as a work area of the control unit 1. The various programs may be read from a portable recording medium such as an optical disk or a semiconductor memory, or may be downloaded from the server device 20 on the network.

In the present embodiment, in particular, the storage unit 2 stores the map information commonly used in each HMD 10. There are a method of creating map information in advance and a method of creating map information at the same time as self-position estimation without creating it in advance (SLAM). Any of these two methods may be used, but in the description of the present embodiment, the case where the map information is created in advance will be described.

Map information is three-dimensional information corresponding to the global coordinate system, and is information used for self-position estimation (relocalization). This map information includes information on a feature point cloud (or mesh information in which the feature point clouds are connected) corresponding to each real object in the real space. Each feature point included in this feature point cloud is associated with position information in the global coordinate system.

The map information is generated based on the image information acquired in advance by the operator with a camera or the like in the real space where the application is executed in each HMD 10, for example.

In this embodiment, the position and orientation of each HMD 10 are represented by a common global coordinate system by using common map information in the self-position estimation in each HMD 10.

"Server device 20"
The server device 20 is configured to be able to communicate with each HMD 10. The server device 20 includes a control unit, a storage unit, a communication unit, and the like (each portion is not shown).

The control unit executes various operations based on various programs stored in the storage unit, and controls each part of the server device in an integrated manner.

The storage unit includes various programs required for processing of the control unit, a non-volatile memory for storing various data, and a volatile memory used as a work area of the control unit 1. Various programs may be read from a portable recording medium such as an optical disk or a semiconductor memory.

The communication unit is configured to be able to communicate with each HMD.

<Operation explanation>
Next, the processing of the information processing system 100 will be described. 4 and 5 are timing charts showing the processing of the information processing system 100.

In the description of the process, the two HMDs 10 are referred to as a first HMD10a (another device) and a second HMD10b (information processing device), respectively, for convenience.

As shown in FIG. 4, first, the server device 20 transmits the map information created in advance to the first HMD10a and the second HMD10b, respectively. The control unit 1 (VPU17) of the first HMD10a and the second HMD10b receives the map information and stores it in the storage unit 2, respectively, whereby the map information is shared by the first HMD10a and the second HMD10b. And the global coordinate system is shared.

The control unit 1 (VPU17) of the first HMD10a and the second HMD10b has a feature point group in the image information acquired by its own imaging unit 4 after receiving the map information, and a feature point group in the map information. Estimate the self-position and orientation in the global coordinate system based on the comparison of. That is, the control unit 1 (VPU17) of the first HMD10a and the second HMD10b executes self-position estimation based on relocalization when the map information is received.

After that, the control unit 1 (VPU17) of the first HMD10a and the second HMD10b compares the feature point group of the image information in the previous time with the feature point group of the image information in the current time, and determines the previous self-position and posture. Calculate the amount of change in self-position and posture this time. The control unit 1 (VPU17) of the first HMD10a and the second HMD10b determines the amount of change in the self-position and posture in the previous time (immediately after the re-localization, the self-position and posture estimated by the re-localization). By adding to, the current self-position and attitude in the global coordinate system are estimated. That is, the control unit 1 (VPU17) of the first HMD10a and the second HMD10b executes the self-position estimation based on the motion tracking after the self-position estimation based on the relocalization.

After that, self-position estimation based on motion tracking is continuously executed. While the self-position estimation based on this motion tracking is being continued, the self-position estimation based on the relocation may be executed at a predetermined timing.

The timing at which the self-position estimation based on relocalization is executed is, for example, when the self-position estimation based on motion tracking fails or when the self-position estimation based on relocalization is always executed, the image information For example, when matching by the feature point cloud and the feature point cloud of the map information is successful.

In this way, the control unit 1 (VPU17) of the first HMD10a and the second HMD10b estimates its own position and attitude, respectively, in the common global coordinate system.

The control unit 1 (VPU17) of the first HMD10a and the second HMD10b transmits the self-position and the posture to the server device 20 each time the self-position and the posture are estimated. Further, the server device 20 transmits the self-position and the posture received from one HMD10 of the first HMD10a and the second HMD10b to the other HMD10. Thereby, the first HMD10a and the second HMD10b can recognize the position and orientation of the other HMD10 in the global coordinate system.

Here, in order to AR-display a common (normal) virtual object at exactly the same position in the first HMD10a and the second HMD10b, the first HMD10a and the second HMD10b, respectively, are in the global coordinate system. It is necessary to accurately estimate the self-position of each.

On the other hand, the first HMD10a and the second HMD10b cannot completely and accurately estimate their own positions. That is, the estimated self-position and posture may deviate from the actual position and posture in the real space, resulting in an error. Such an error causes a misalignment of a common (normal) virtual object that can be AR-displayed at the same position in the global coordinate system in the first HMD10a and the second HMD10b.

The following are examples of common (normal) virtual object misalignments.
-Misrecognition of self-position due to detection of multiple locations in real space where similar feature point cloud patterns exist (for example, multiple windows of the same shape or walls with repeating patterns) -The lens unit of the imaging unit 4 Misrecognition of self-position due to dirt in When an area where map information does not exist is detected, or when the feature point at the time when the map information is created by a shadow or a dynamic object has changed from the current feature point, etc.) Lost position or drift self-position

The position of a common (normal) virtual object is represented as three-dimensional coordinate information in the global coordinate system. If one of the first HMD10a and the second HMD10b AR displays a common virtual object at the position (x, y, z) in the global coordinate system, the other HMD10 also has global coordinates. The common virtual object in the system is AR-displayed at the position (x, y, z).

In this case, it is assumed that there is an error between the estimated self-position and posture and the actual position and posture in any of the HMD10s. In this case, the position and orientation seen by the user in the common (normal) virtual object AR displayed in one HMD10 and the position and orientation seen by the user in the common virtual object AR displayed in the other HMD10. Will be different.

Explaining with the example of the survival game, from the user who wears one HMD10, it seems that the virtual object of the ammunition hits the user who wears the other HMD10, whereas the user who wears the other HMD10. From the perspective, there can be situations where the virtual object of the ammunition does not appear to hit itself.

Therefore, in the present embodiment, the subsequent processing is executed in order to correct the misalignment of such a common (normal) virtual object.

After the start of self-position estimation, the control unit 1 (VPU17) of the first HMD 10a is used for correction within a certain field of view and distance ahead of the line-of-sight direction seen by the user, based on the image information from the image pickup unit 4. It is determined whether or not the real object 30 (see FIGS. 6, 7, etc.) exists. That is, the control unit 1 (VPU17) of the first HMD 10a executes a process of selecting (finding) the correction real object 30 from the plurality of real objects existing in the real space based on the image information.

The correction real object 30 is a real object for which the correction virtual object 31 is positioned (displayed in AR). The correction real object 30 is a real object that satisfies a predetermined condition among a plurality of real objects existing in the real space, and the first HMD10a and the second HMD10b are real objects that satisfy the predetermined condition. Is selected as the correction real object 30.

In the present embodiment, the condition selected as the correction real object 30 is that the real object has a specific shape. Typically, this condition is that the real object has a three-dimensional shape that is substantially uniquely identified when the real object is viewed from any direction. For example, the correction real object 30 has a regular shape such as a sphere, a rectangular parallelepiped (including a cube), a cylinder, a polygonal prism, a cone, and a polygonal pyramid.

FIG. 6 is a diagram showing a state when a user wearing the first HMD10a and a user wearing the second HMD10b are looking at the actual object 30 for correction. In the example shown in FIG. 6, the shape of the correction real object 30 is a cube.

Further, in the example shown in FIG. 6, the user wearing the first HMD10a is looking at the correction real object 30 from an oblique left direction, and the user wearing the second HMD10b is looking at the correction real object 30 from an oblique right direction. An example is shown when looking at the object 30.

FIG. 7 is a diagram showing an example of image information when the correction real object 30 is imaged by the image capturing unit 4 of the first HMD 10a in the example shown in FIG. FIG. 8 is a diagram showing information of a feature point cloud extracted from the image information shown in FIG. 7.

The control unit 1 (VPU17) of the first HMD 10a selects the actual object 30 for correction based on the information of the feature point cloud as shown in FIG. 8 extracted from the image information as shown in FIG. 7, for example (VPU17). Execute the process (find).

Also, the number of correction real objects 30 selected is not always one. That is, every time the correction real object 30 is detected by the first HMD 10a due to the movement of the user wearing the first HMD 10a or the operation of looking around, a new correction real object 30 is sequentially added. ..

After selecting the correction real object 30, the control unit 1 (CPU16) of the first HMD 10a sets the correction real object 30 in its own global coordinate system based on the information of the feature point cloud of the correction real object 30 (see FIG. 8). The coordinate position (AR display position) of the correction virtual object 31 is set with respect to the correction real object 30.

The coordinate positions of the correction virtual object 31 set at this time are the position of the center of gravity of the correction virtual object 31, the position of the vertices (corners) of each part (face) constituting the correction virtual object 31, and the correction virtual object. Includes information such as the orientation of 31.

FIG. 9 is a diagram showing an example when the coordinate position of the correction virtual object 31 is set (when AR is displayed) with respect to the correction real object 30.

After setting the coordinate position of the correction virtual object 31, the control unit 1 (GPU18) of the first HMD 10a causes the display unit 3 to AR-display the correction virtual object 31 at the coordinate position. The correction virtual object 31 does not necessarily have to be AR-displayed in the first HMD 10a, and its coordinate position may only be set.

In the present embodiment, the shape and size of the correction virtual object 31 are not determined in advance, but are determined based on the shape and size of the correction real object 30. Specifically, in the present embodiment, the shape and size of the correction virtual object 31 are related to the correction real object 30, and are the same as the shape (cube) and size of the correction real object 30. There is.

Further, when the correction virtual object 31 is AR-displayed, the position of the correction virtual object 31 is set so that the correction virtual object 31 overlaps with the correction real object 30 and is displayed in AR. Will be done.

The shape and size of the correction virtual object 31 may be determined in advance (for example, when the correction virtual object 31 is a character). Further, the position of the correction virtual object 31 may be set in the vicinity of the correction virtual object 31 (for example, when the correction virtual object 31 of the character is on the correction real object 30).

After setting the position of the correction virtual object 31 or displaying the correction virtual object 31 in AR, the control unit 1 (CPU 16) of the first HMD 10a outputs the coordinate information of the position of the correction virtual object 31 to the server. It transmits to the device 20.

For example, as shown in FIG. 9, in the case of the cube correction virtual object 31, the control unit 1 (CPU 16) of the first HMD 10a uses the corners (each angle) on the three surfaces as the coordinate information of the correction virtual object 31. The coordinate information of a) to (g) is transmitted to the server device 20.
Front: [{x _a , y _a , z _a }, {x _b , y _b , z _b }, {x _c , y _c , z _c }, {x _d , y _d , z _d }]
Top surface: [{x _e , y _e , z _e }, {x _f , y _f , z _f }, {x _b , y _b , z _b }, {x _a , y _a , z _a }]
Left _{side: [{x e, y e} , z e}, {x a, y a, z a}, {x d, y d, z d}, {x g, y g, z g}]

When the server device 20 receives the coordinate information of the correction virtual object 31 from the first HMD 10a, the server device 20 transmits this coordinate information to the second HMD 10b. In the example here, the coordinate information of the correction virtual object 31 is transmitted from the first HMD 10a to the second HMD 10b via the server device 20, but this coordinate information is directly from the first HMD 10a. May be transmitted to the second HMD10b.

When the control unit 1 (CPU16) of the second HMD 10b receives the coordinate information of the correction virtual object 31, in its own global coordinate system, the control unit 1 (CPU 16) obtains the self-position and orientation and the acquired coordinate information of the correction virtual object 31. Based on this, it is determined whether or not the correction real object 30 selected by the first HMD 10a exists within a certain field of view and distance ahead of the user's line-of-sight direction.

Here, it is assumed that the user wearing the first HMD10a and the second HMD10b are located at a distance from each other when the correction real object 30 is selected by the first HMD10a. In this case, the second HMD 10b approaches a position that is equal to or less than a certain distance with respect to the correction real object 30 selected by the first HMD 10a, and the direction of the user's line of sight of the correction real object 30 is selected. It is determined that the correction real object 30 exists in the user's field of view when facing.

In order to correct the position of the shared (normal) virtual object, the user wearing the second HMD10b approaches the correction real object 30 selected by the first HMD10a and looks in the direction thereof. There is a need. On the other hand, since the user wearing the first HMD10a basically moves while looking around, the correction real objects 30 are sequentially increased according to the movement of the user of the first HMD10a and moved to a plurality of locations. It will be scattered. Therefore, the possibility that the user wearing the second HMD 10b approaches and faces the correction real object 30 selected by the first HMD 10a is increased every time the correction real object 30 is newly selected and increases. Will increase to.

When the correction real object 30 selected by the first HMD 10a is at a certain distance from the user wearing the second HMD 10b and is within the user's field of view, the control unit 1 of the second HMD 10b (CPU 16) executes the following processing. That is, the control unit 1 (CPU16) of the second HMD 10b is the position of the correction virtual object 31 based on the self position and the posture and the acquired coordinate information of the correction virtual object 31 in its own global coordinate system. To set.

FIG. 10 is a diagram showing a state when the position of the correction virtual object 31 is set (AR displayed) by the second HMD 10b based on the coordinate information.

Next, the control unit 1 (GPU18) of the second HMD 10b causes the display unit 3 to AR-display the correction virtual object 31 at the set position. The control unit 1 of the second HMD 10b only sets the position of the correction virtual object 31, and does not have to actually display the correction virtual object 31 in AR.

The process when the control unit 1 (CPU 16) of the second HMD 10b sets the position of the correction virtual object 31 will be specifically described with an example.

The control unit 1 (CPU16) of the second HMD10b receives the coordinate information of the angles (a) to (g) on the front surface, the upper surface, and the left surface described above ([{x _a , y _a , z _a }, {x _b , y _b , z _b }, {x _c , y _c , z _c }, {x _d , y _d , z _d }], [{x _e , y _e , z _{e}, {x f, y} f, z f}, {x b, y b, z b}, {x a, y a, z a}], [{x e, y e, z e}, { x _a , y _a , z _a }, {x _d , y _d , z _d }, {x _g , y _g , z _g }]) has been acquired. The control unit 1 (CPU 16) of the second HMD 10b obtains what the virtual object 31 for correcting the cube based on this information looks like when viewed from the current self-position and posture.

As shown in FIG. 6, when the correction real object 30 is selected by the first HMD 10a, the user wearing the first HMD 10a is looking at the correction real object 30 from an oblique left direction. On the other hand, when the user wearing the second HMD 10b approaches the correction real object 30 and looks at the correction real object 30, the user is looking at the correction real object 30 from an oblique right direction.

Therefore, the front surface, the upper surface, and the left side surface of the correction virtual object 31 can be seen from the first HMD10a side, but the front surface, the upper surface surface, and the right side surface of the correction object can be seen from the second HMD10b side. You can see three sides.

Control unit 1 of the second HMD10b (CPU16) is a front, a surface other than the right side surface of the three faces of the top and right side, i.e., coordinate information of the front and top _{([{x a, y a} , z a} , {X _b , y _b , z _b }, {x _c , y _c , z _c }, {x _d , y _d , z _d }], [{x _e , y _e , z _e }, {x _f , Y _f , z _f }, {x _b , y _b , z _b }, {x _a , y _a , z _a }]). Therefore, the control unit 1 (CPU16) of the second HMD10b sets the coordinates at the position in its own global coordinate system, and calculates how the front surface and the upper surface look from the self position and the posture. ..

On the other hand, since the control unit 1 (CPU16) of the second HMD 10b does not have the coordinate information of the right side surface, the coordinate information of the right side surface ([{x _f , y _f , z) is obtained from the coordinate information of the other surface. _f }, {x _b , y _b , z _b }, {x _c , y _c , z _c }, {x _h , y _h , z _h }]) (H) Coordinates need to be predicted). Then, the control unit 1 (CPU16) of the second HMD10b sets the coordinates at the predicted position in its own global coordinate system, and obtains what the right side surface looks like when viewed from its own position and posture.

In the present embodiment, as described above, a real object having a regular shape such that the three-dimensional shape is uniquely specified when viewed from any direction is selected as the correction real object 30. Therefore, the control unit 1 (CPU16) of the second HMD 10b also has a portion of the correction virtual object 31 corresponding to the portion of the correction real object 30 that cannot be seen from the first HMD 10a when the correction real object 30 is selected. , The coordinates of that part can be predicted accurately.

Note that the prediction of the coordinates in the part of the virtual object corresponding to the portion of the invisible real object 30 for correction may be executed on the first HMD10a side instead of the second HMD10b. In this case, the predicted coordinate information is transmitted to the first HMD 10a together with the coordinate information corresponding to the visible part (via the server).

Here, when the self-position and the posture in the first HMD 10a and the second HMD 10b are accurate, the relative positional relationship between the correction virtual object 31 and the correction real object 30 in the first HMD 10a (see FIG. 9). And the relative positional relationship (see FIG. 10) between the correction virtual object 31 and the correction real object 30 in the second HMD 10b should be the same. Therefore, when the self-position and posture in the first HMD 10a and the second HMD 10b are accurate, the position of the correction virtual object 31 is set to completely overlap the correction real object 30 in the second HMD 10b. Should be done.

On the other hand, if the self-position and orientation in at least one of the first HMD10a or the second HMD10b are inaccurate, the relative positions of the correction virtual object 31 and the correction real object 30 in the first HMD10a. The relationship (see FIG. 9) and the relative positional relationship (see FIG. 10) between the correction virtual object 31 and the correction real object 30 in the second HMD 10b are different. In this case, as shown in FIG. 10, the position of the correction virtual object 31 is set to a position deviated from the correction real object 30.

In the present technology, this relationship is utilized, and in the second HMD 10b, the AR display position of the common (normal) virtual object is corrected based on the positional relationship between the correction virtual object 31 and the correction real object 30. ..

After setting the coordinate position of the correction virtual object 31 or displaying the correction virtual object 31 in AR, the control unit 1 (VPU17) of the second HMD 10b acquires image information from the image pickup unit 4 and obtains image information. A feature point cloud is extracted from the image information.

FIG. 11 is a diagram showing a state when the correction real object 30 is imaged by the imaging unit 4 of the second HMD 10b. FIG. 12 is a diagram showing information of a feature point cloud extracted from the image information shown in FIG.

Next, the control unit 1 (CPU16) of the second HMD10b is a feature point cloud corresponding to the correction real object 30 among the feature point cloud included in the image information based on the coordinate information acquired from the first HMD10a. Determine which feature point cloud the group belongs to. Next, the control unit 1 (CPU16) of the second HMD 10b determines the position and orientation of the correction real object 30 in its own global coordinate system based on the information of the feature point cloud corresponding to the correction real object 30. calculate.

When the position and orientation of the correction real object 30 are calculated, the control unit 1 (CPU16) of the second HMD10b has the position and orientation of the correction virtual object 31 set based on the coordinate information from the first HMD10a. , The difference between the position and the posture of the actual object 30 for correction is obtained.

FIG. 13 is a diagram showing the difference between the position and orientation of the correction virtual object 31 and the position and orientation of the correction real object 30.

When the control unit 1 (CPU16) of the second HMD10b obtains the difference, the control unit 1 (CPU16) stores the difference as a correction value in the storage unit 2. Then, when the control unit 1 (CPU16) of the second HMD10b AR-displays the (normal) virtual object common to the first HMD10a, the control unit 1 (CPU16) uses this difference as a correction value and AR-displays the common virtual object. Correct the position. When the correction virtual object 31 is displayed in AR, the correction virtual object 31 may also be corrected by the correction value.

As a method of obtaining the correction value, for example, there is a method of obtaining the movement amount and the rotation angle. The difference of the movement amount, for example, the center of gravity of the correction for the virtual object _{31 (x G, y G,} z G) and center of gravity of the correction the real object _{30 (x 'G, y'} G, z 'G) and It is calculated from.

The position of the center of gravity (x _G , y _G , z _G ) of the correction virtual object 31 is calculated from, for example, the positions of the angles (a) to (h) in the correction virtual object 31. Further, the center of gravity of the correction the real object _{30 (x 'G, y'} G, z 'G) , the angular correction real object 30 (a' is calculated from the position of) ~ (h ').

In this case, the movement amount (correction _{value) (T x, T y,} T z) _{is, (T x, T y,} T z) = (x G -x 'G, y G -y' G, z G - It is calculated by _z'G).

The rotation angles (correction values) (θ _z , θ _z , θ _z ) are the vector A connecting two specific angles (a) to (h) of the correction virtual object 31 and the correction actual. Calculated by cosθ = A · A'/ | A || A'| using the vector A'connecting two corresponding specific angles among the angles (a') to (h') of the object 30. .. This formula is used to calculate the rotation angles around the X-axis, Y-axis, and Z-axis, respectively.

Then, the calculated movement amount and rotation angle are used as correction values, and the corrections represented by the following formulas are executed for the coordinate points of the common (normal) virtual object. In each of the following equations, the coordinates before correction are P (x, y, z), and the coordinates after correction are P'(x', y', z').

[Move]

[Rotation around X-axis]

[Rotation around Y axis]

[Rotation around Z axis]

By this correction, the common (normal) virtual object will be displayed in the same position and orientation in the first HMD10a and the second HMD10b.

Note that the correction value may be transmitted from the second HMD 10b to the server device 20, and the server device 20 may use this correction value to correct the position and orientation of a common (normal) virtual object. Further, the calculation of the correction value may be performed by the server device 20 or the first HMD10a instead of the second HMD10b (for example, when the processing performance of the second HMD10b is low). In this case, the information necessary for calculating the correction value (information on the position and posture of the correction virtual object 31 in the second HMD 10b, information on the position and posture of the correction real object 30) is transmitted from the second HMD 10b to the server. It is transmitted to the device 20 or the second HMD 10b.

After the correction value is calculated, when the user wearing the second HMD10b approaches the correction real object 30 selected by the first HMD10a and looks in the direction, the correction value is newly calculated. And the correction value is updated. When self-position estimation based on relocalization is executed by one of the first HMD10a and the second HMD10b, the current correction value may be reset to 0.

Here, the correction value calculated when the distance between the correction real object 30 and the second HMD 10b is long is calculated when the distance between the correction real object 30 and the second HMD 10b is short. It may be less reliable than the corrected correction value. This is because the recognition of the position and shape of the correction real object 30 in the second HMD 10b may become inaccurate as the distance between the second HMD 10b and the correction real object 30 increases. be.

Therefore, the degree of correction by the correction value of the common (normal) virtual object is changed according to the distance between the second HMD 10b when the correction value is calculated and the real object 30 for correction. May be good. In this case, the closer the distance between the second HMD 10b when the correction value is calculated and the actual object 30 for correction is, the higher the degree of correction by the correction value is.

As an example, suppose that a common (normal) virtual object is AR-displayed at a certain distance from the second HMD10b. In this case, when the distance between the second HMD 10b when the correction value is calculated and the correction real object 30 is equal to or less than a certain distance, the correction value is multiplied by 1. On the other hand, when the distance between the second HMD 10b when the correction value is calculated and the correction real object 30 exceeds a certain distance, 0.9 with respect to the correction value as the distance increases. Values such as 0.8 ... are multiplied.

Regarding this, when the first HMD 10a sets the coordinate position of the correction virtual object 31 with respect to the correction real object 30, the distance between the correction real object 30 and the first HMD 10a is set. The same is true when it is far away. That is, the correction value calculated when the distance between the correction real object 30 and the first HMD 10a is long is calculated when the distance between the correction real object 30 and the first HMD 10a is short. The reliability may be lower than the corrected value.

This is because as the distance between the first HMD 10a and the correction real object 30 increases, the recognition of the position and shape of the correction real object 30 in the first HMD 10a becomes inaccurate, and the first HMD 10a transmits. This is because the coordinate information of the correction virtual object 31 to be performed may be inaccurate.

Therefore, the degree of correction by the correction value of the common (normal) virtual object is the first HMD 10a when the coordinate position of the correction virtual object 31 is set by the first HMD 10a, and the correction real object 30. It may vary depending on the distance between them. In this case, the closer the distance between the first HMD 10a when the coordinate position of the correction virtual object 31 is set by the first HMD 10a and the correction real object 30, the higher the degree of correction by the correction value. Be crushed.

For example, suppose that a common (normal) virtual object is AR-displayed at a certain distance from the second HMD10b. In this case, when the distance between the first HMD 10a when the coordinate position of the correction virtual object 31 is set by the first HMD 10a and the correction real object 30 is equal to or less than a certain distance, the correction value is set. Is multiplied by 1. On the other hand, when the distance between the first HMD 10a and the correction real object 30 when the coordinate position of the correction virtual object 31 is set by the first HMD 10a exceeds a certain distance, the distance becomes long. Therefore, the correction value is multiplied by a value such as 0.9, 0.8, and so on.

In the description here, the case where the correction value is calculated each time the correction real object 30 selected by the first HMD10a is detected by the second HMD10b has been described. On the other hand, when the correction real objects 30 selected by the first HMD 10a are densely packed, the correction real objects 30 are frequently detected by the second HMD 10b, and the correction value is frequently calculated. Will be done. In this case, the processing load on the second HMD 10b may increase. Therefore, for example, the second HMD 10b may execute the calculation of a new correction value when the correction real object 30 is detected after a certain period of time has elapsed from the processing of the previous calculation of the correction value.

Further, in the example here, the case where the HMD10 on the transmitting side for transmitting the coordinate information of the correction virtual object 31 and the HMD10 on the receiving side for receiving the coordinate information are predetermined has been described. On the other hand, the HMD10 on the transmitting side and the HMD10 on the receiving side may not be determined in advance.

In this case, for example, both the first HMD10a and the second HMD10b execute the process of selecting (finding) the correction real object 30, and the one HMD10 that first finds the correction real object 30 is for correction. The coordinate information of the virtual object 31 is transmitted. The other HMD 10 calculates a correction value based on the positional relationship between the correction virtual object 31 and the correction real object 30 when approaching the correction real object 30 selected by one HMD 10 and looking in the direction thereof. .. Then, one HMD10 corrects the AR display position with the common (normal) virtual object by using the correction value.

<Action, etc.>
As described above, in the first embodiment, the positions of the common (normal) virtual objects are corrected based on the positional relationship between the correction virtual object 31 and the correction real object 30.

As a result, in the first HMD10a and the second HMD10b, it is possible to eliminate the apparent deviation of the common (normal) virtual object, and the common (normal) virtual object is AR-displayed at the same position. It becomes possible to do. Further, in the present embodiment, the position of a common (normal) virtual object can be corrected by a relatively simple method.

Further, in the first embodiment, the position of the common (normal) virtual object is corrected based on the difference between the position of the correction virtual object 31 and the position of the correction real object 30. As a result, the AR display position of the common virtual object can be appropriately corrected.

Further, in the first embodiment, the correction value is calculated based on the difference between the position of the correction virtual object 31 and the position of the correction real object 30, and the position of the common (normal) virtual object is calculated according to the correction value. Is corrected. Also, the correction value moves and rotates a common (normal) virtual object. As a result, the AR display position of the common (normal) virtual object can be corrected more appropriately.

Further, as described above, in the first embodiment, the degree of correction by the correction value is changed according to the distance between the second HMD 10b when the correction value is calculated and the actual object 30 for correction. You may. Thereby, the degree of correction by the correction value can be appropriately changed.

Further, as described above, in the first embodiment, the degree of correction by the correction value is the first HMD10a when the coordinate position of the correction virtual object 31 is set by the first HMD10a, and the correction real object. It may be varied depending on the distance to 30. Thereby, the degree of correction by the correction value can be appropriately changed.

Further, in the first embodiment, the first HMD10a selects (finds) the correction real object 30 from the plurality of real objects existing in the real space based on the image information acquired by the first HMD10a. ). As a result, in the first embodiment, the real object existing in the real space can be selected as the correction real object 30, so that it is not necessary to specially install a marker or the like in the real space, which saves time and effort. be able to.

Further, in the first embodiment, the condition selected as the correction real object 30 is that the real object has a specific shape. In particular, this condition is that the real object has a three-dimensional shape that is substantially uniquely specified regardless of the direction in which the real object is viewed. As a result, it is possible to accurately predict the coordinates of the part of the correction virtual object 31 corresponding to the part of the correction real object 30 that cannot be seen from the first HMD 10a when the correction real object 30 is selected. It becomes.

Here, in the first embodiment, a method is used in which correction is performed individually for each common (normal) virtual object according to the correction value. On the other hand, instead of this method, a method of correcting the self-position and the posture itself by the correction value may be used.

However, with this method, the correction will be applied to all common (normal) virtual objects. At this time, if the distance between the above-mentioned second HMD 10b and the correction real object 30 and the distance between the first HMD 10a and the correction real object 30 become long, the correction using the correction value is also common. Unintended misalignment of (normal) virtual objects can occur. Therefore, it is necessary to individually correct each common (normal) virtual object by the correction value and share it with each HMD10 without using the method of correcting the self-position and the posture itself by the correction value. A method such as performing correction by the correction value only for a relatively high common virtual object may be used.

<< Second Embodiment >>
Next, a second embodiment of the present technology will be described. In the second embodiment, another method of obtaining the positional relationship between the correction virtual object 31 and the correction real object 30 in the second HMD 10b will be described.

In the second embodiment, the control unit 1 (CPU16) of the second HMD10b sets the position of the correction virtual object 31 based on the coordinate information acquired from the first HMD10a (see FIG. 10), or After displaying the correction virtual object 31 in AR, the following processing is executed.

First, the control unit 1 (VPU17) of the second HMD10b acquires image information from the image pickup unit 4 and extracts a feature point cloud from the image information. Then, the control unit 1 (CPU16) of the second HMD10b is a feature point cloud corresponding to the correction real object 30 among the feature point cloud included in the image information based on the coordinate information acquired from the first HMD10a. Determines which feature point cloud is.

Next, the control unit 1 (CPU 16) of the second HMD 10b corrects the correction real object 30 in its own global coordinate system based on the information of the feature point group corresponding to the correction real object 30. Set the coordinate position (AR display position) of the virtual object 31.

FIG. 14 is a diagram showing an example when the coordinate position of the correction virtual object 31 is set (when AR is displayed) with respect to the correction real object 30.

In the second embodiment, there are two types of virtual objects as the correction virtual objects 31 in the second HMD10b. The first type is a correction virtual object 31a whose coordinate position is set based on the coordinate information acquired from the first HMD 10a (see FIG. 10). The second type is a correction virtual object 31b whose coordinate position is set based on the image information acquired in the second HMD 10b (see FIG. 14). In the following description, the first type of correction virtual object 31 is referred to as a correction virtual object 31a based on coordinate information, and the second type of correction virtual object 31 is referred to as a correction virtual object 31b based on image information. ..

When setting the coordinate position of the correction virtual object 31b based on the image information, the control unit 1 (CPU16) of the second HMD10b determines that the first HMD10a is the correction real object 30 based on the image information of the first HMD10a. The same conditions as when the coordinate position of the correction virtual object 31 is set are used. Then, the control unit 1 (CPU16) of the second HMD10b uses this same condition to make a correction virtual object based on the image information with respect to the correction real object 30 based on the image information of the second HMD10b. The coordinate position (AR display position) of 31b is set.

For example, in the control unit 1 (CPU16) of the second HMD 10b, when the correction virtual object 31b based on the image information is AR-displayed, the correction virtual object 31b based on the image information is displayed with respect to the correction real object 30. The coordinate position of the correction virtual object 31b based on the image information is set so that the AR is displayed so as to overlap the same position. When the shape and size of the correction virtual object 31 are not determined in advance like the cube correction virtual object 31, the image information is used when setting the coordinate position of the correction virtual object 31b based on the image information. Based on this, the shape and size of the correction virtual object 31b based on the image information are determined.

Next, the control unit 1 (GPU18) of the first HMD 10a AR-displays the correction virtual object 31b based on the image information at the position corresponding to the set coordinate position on the display unit 3. The correction virtual object 31b based on the image information does not necessarily have to be AR-displayed in the second HMD10b, and its coordinate position may only be set.

After setting the coordinate position of the correction virtual object 31b based on the image information, or after displaying the correction virtual object 31b based on the image information in AR, the control unit 1 (CPU16) of the second HMD 10b uses the coordinate information. The difference between the position and orientation of the correction virtual object 31a based on the image information and the position and orientation of the correction virtual object 31b based on the image information is obtained.

FIG. 15 is a diagram showing the difference between the position and orientation of the correction virtual object 31a based on the coordinate information and the position and orientation of the correction virtual object 31b based on the image information.

When the control unit 1 (CPU16) of the second HMD10b obtains the difference, the control unit 1 (CPU16) stores the difference as a correction value in the storage unit 2. Then, when the control unit 1 (CPU16) of the second HMD10b AR-displays the (normal) virtual object common to the first HMD10a, the control unit 1 (CPU16) uses this difference as a correction value and AR-displays the common virtual object. Correct the position.

As a specific calculation method of the correction value and a specific correction method using this correction value, the same method as in the above-described first embodiment can be used. That is, in the above-described first embodiment, the wording of the "correction virtual object 31" in the place where the calculation method of the correction value and the correction method using the correction value are explained is "the correction virtual based on the coordinate information". It may be read as "object 31a", and the wording of "correction real object 30" may be read as "correction virtual object 31b based on image information". Further, "angle (a') to (h')" may be read as "angle (a") to (h ")".

In the second embodiment as well as in the first embodiment, it is possible to eliminate the apparent deviation of the common (normal) virtual object in the first HMD10a and the second HMD10b, and it is exactly common in the same position. (Normal) virtual object can be AR-displayed.

≪Various deformation examples≫
In the above description, HMD10 has been taken as an example of the AR device (information processing device). On the other hand, the AR device is not limited to the HMD 10. Other examples of the AR device include wearable devices other than the HMD10 such as a wristband type (watch type), a ring type, and a pendant type. Further, as another example of the AR device, a mobile phone (including a smartphone), a tablet PC, a portable game machine, a portable music player, and the like can be mentioned. Typically, the AR device may be any device as long as it can display a virtual object in AR (and can be mounted or grasped by the user and moved together with the user).

The present technology can also have the following configurations.
(1) Estimate the self-position in the global coordinate system corresponding to the real space,
Another device sharing the global coordinate system acquires the coordinate information of the position of the first virtual object that can be AR-displayed set for the real object in the real space in the global coordinate system.
Based on the self-position and the coordinate information, the position of the first virtual object is set in the global coordinate system.
Based on the image information, the position of the real object in the global coordinate system is calculated.
An information processing device including a control unit that corrects the position of a second virtual object that is AR-displayed in common with the other device based on the positional relationship between the first virtual object and the real object.
(2) The information processing device according to (1) above.
The control unit is an information processing device that corrects the position of the second virtual object based on the difference between the position of the first virtual object and the position of the real object.
(3) The information processing device according to (1) above.
The control unit sets the position of the first virtual object with respect to the real object in the global coordinate system based on the image information, and sets the position of the first virtual object based on the coordinate information. An information processing device that corrects the position of the second virtual object based on the difference from the position of the first virtual object based on the image information.
(4) The information processing device according to (2) or (3) above.
The control unit is an information processing device that calculates a correction value based on the difference and corrects the position of the second virtual object based on the correction value.
(5) The information processing device according to (4) above.
The control unit is an information processing device that corrects the position of the second virtual object by moving the position of the second virtual object according to the correction value.
(6) The information processing device according to (4) or (5) above.
The control unit is an information processing device that corrects the position of the second virtual object by rotating the second virtual object according to the correction value.
(7) The information processing device according to any one of (4) to (6) above.
The control unit is an information processing device that changes the degree of correction by the correction value.
(8) The information processing device according to (7) above.
The control unit is an information processing device that changes the degree of correction by the correction value according to the distance between the information processing device when the correction value is calculated and the real object.
(9) The information processing device according to (7) or (8) above.
The control unit determines the correction value according to the distance between the other device and the real object when the other device sets the position of the first virtual object with respect to the real object. An information processing device that changes the degree of correction by.
(10) The information processing device according to any one of (1) to (9) above.
The other device is an information processing device that sets the position of the first virtual object so that the first virtual object overlaps the real object and can be AR-displayed.
(11) The information processing device according to any one of (1) to (9) above.
The other device is an information processing device that sets the position of the first virtual object so that the first virtual object can be AR-displayed in the vicinity of the real object.
(12) The information processing device according to any one of (1) to (11) above.
The other device is an information processing device that AR-displays the first virtual object.
(13) The information processing device according to any one of (1) to (12) above.
The control unit is an information processing device that AR-displays the first virtual object.
(14) The information processing apparatus according to any one of (1) to (13) above.
Based on the image information acquired by the other device, the other device selects the real object to which the first virtual object is located from among a plurality of real objects existing in the real space. Information processing device to select.
(15) The information processing device according to (14) above.
The other device is an information processing device that selects the real object satisfying a predetermined condition as the real object to which the first virtual object is located.
(16) The information processing device according to (15) above.
The information processing device that the predetermined condition is that the real object has a specific shape.
(17) The information processing device according to (16) above.
The predetermined condition is that the real object has a three-dimensional shape that is substantially uniquely specified regardless of the direction in which the real object is viewed.
(18) Estimate the self-position in the global coordinate system corresponding to the real space, and
Another device sharing the global coordinate system acquires the coordinate information of the position of the first virtual object that can be AR-displayed set for the real object in the real space in the global coordinate system.
Based on the self-position and the coordinate information, the position of the first virtual object is set in the global coordinate system.
Based on the image information, the position of the real object in the global coordinate system is calculated.
An information processing device having a control unit that corrects the position of a second virtual object that is AR-displayed in common with the other device based on the positional relationship between the first virtual object and the real object, and the other device. Information processing system equipped with.
(19) Estimate the self-position in the global coordinate system corresponding to the real space, and
Another device sharing the global coordinate system acquires the coordinate information of the position of the first virtual object that can be AR-displayed set for the real object in the real space in the global coordinate system.
Based on the self-position and the coordinate information, the position of the first virtual object is set in the global coordinate system.
Based on the image information, the position of the real object in the global coordinate system is calculated.
An information processing method for correcting the position of a second virtual object that is AR-displayed in common with the other device based on the positional relationship between the first virtual object and the real object.
(20) Estimate the self-position in the global coordinate system corresponding to the real space, and
Another device sharing the global coordinate system acquires the coordinate information of the position of the first virtual object that can be AR-displayed set for the real object in the real space in the global coordinate system.
Based on the self-position and the coordinate information, the position of the first virtual object is set in the global coordinate system.
Based on the image information, the position of the real object in the global coordinate system is calculated.
A program that causes a computer to execute a process of correcting the position of a second virtual object that is AR-displayed in common with the other device based on the positional relationship between the first virtual object and the real object.

1 ... Control unit 10 ... HMD
20 ... Server device 30 ... Real object for correction 31 ... Virtual object for correction 100 ... Information processing system

Claims (20)

1. Estimate the self-position in the global coordinate system corresponding to the real space,
   Another device sharing the global coordinate system acquires the coordinate information of the position of the first virtual object that can be AR-displayed set for the real object in the real space in the global coordinate system.
   Based on the self-position and the coordinate information, the position of the first virtual object is set in the global coordinate system.
   Based on the image information, the position of the real object in the global coordinate system is calculated.
   An information processing device including a control unit that corrects the position of a second virtual object that is AR-displayed in common with the other device based on the positional relationship between the first virtual object and the real object.
2. The information processing device according to claim 1.
   The control unit is an information processing device that corrects the position of the second virtual object based on the difference between the position of the first virtual object and the position of the real object.
3. The information processing device according to claim 1.
   The control unit sets the position of the first virtual object with respect to the real object in the global coordinate system based on the image information, and sets the position of the first virtual object based on the coordinate information. An information processing device that corrects the position of the second virtual object based on the difference from the position of the first virtual object based on the image information.
4. The information processing device according to claim 2.
   The control unit is an information processing device that calculates a correction value based on the difference and corrects the position of the second virtual object based on the correction value.
5. The information processing device according to claim 4.
   The control unit is an information processing device that corrects the position of the second virtual object by moving the position of the second virtual object according to the correction value.
6. The information processing device according to claim 4.
   The control unit is an information processing device that corrects the position of the second virtual object by rotating the second virtual object according to the correction value.
7. The information processing device according to claim 4.
   The control unit is an information processing device that changes the degree of correction by the correction value.
8. The information processing device according to claim 7.
   The control unit is an information processing device that changes the degree of correction by the correction value according to the distance between the information processing device when the correction value is calculated and the real object.
9. The information processing device according to claim 4.
   The control unit determines the correction value according to the distance between the other device and the real object when the other device sets the position of the first virtual object with respect to the real object. An information processing device that changes the degree of correction by.
10. The information processing device according to claim 1.
    The other device is an information processing device that sets the position of the first virtual object so that the first virtual object overlaps the real object and can be AR-displayed.
11. The information processing device according to claim 1.
    The other device is an information processing device that sets the position of the first virtual object so that the first virtual object can be AR-displayed in the vicinity of the real object.
12. The information processing device according to claim 1.
    The other device is an information processing device that AR-displays the first virtual object.
13. The information processing device according to claim 1.
    The control unit is an information processing device that AR-displays the first virtual object.
14. The information processing apparatus according to claim 1.
    Based on the image information acquired by the other device, the other device selects the real object to which the first virtual object is located from among a plurality of real objects existing in the real space. Information processing device to select.
15. The information processing device according to claim 14.
    The other device is an information processing device that selects the real object satisfying a predetermined condition as the real object to which the first virtual object is located.
16. The information processing device according to claim 15.
    The information processing device that the predetermined condition is that the real object has a specific shape.
17. The information processing device according to claim 16.
    The predetermined condition is that the real object has a three-dimensional shape that is substantially uniquely specified regardless of the direction in which the real object is viewed.
18. Estimate the self-position in the global coordinate system corresponding to the real space,
    Another device sharing the global coordinate system acquires the coordinate information of the position of the first virtual object that can be AR-displayed set for the real object in the real space in the global coordinate system.
    Based on the self-position and the coordinate information, the position of the first virtual object is set in the global coordinate system.
    Based on the image information, the position of the real object in the global coordinate system is calculated.
    An information processing device having a control unit that corrects the position of a second virtual object that is AR-displayed in common with the other device based on the positional relationship between the first virtual object and the real object, and the other device. Information processing system equipped with.
19. Estimate the self-position in the global coordinate system corresponding to the real space,
    Another device sharing the global coordinate system acquires the coordinate information of the position of the first virtual object that can be AR-displayed set for the real object in the real space in the global coordinate system.
    Based on the self-position and the coordinate information, the position of the first virtual object is set in the global coordinate system.
    Based on the image information, the position of the real object in the global coordinate system is calculated.
    An information processing method for correcting the position of a second virtual object that is AR-displayed in common with the other device based on the positional relationship between the first virtual object and the real object.
20. Estimate the self-position in the global coordinate system corresponding to the real space,
    Another device sharing the global coordinate system acquires the coordinate information of the position of the first virtual object that can be AR-displayed set for the real object in the real space in the global coordinate system.
    Based on the self-position and the coordinate information, the position of the first virtual object is set in the global coordinate system.
    Based on the image information, the position of the real object in the global coordinate system is calculated.
    A program that causes a computer to execute a process of correcting the position of a second virtual object that is AR-displayed in common with the other device based on the positional relationship between the first virtual object and the real object.

Images