WO2022044900A1 - Information processing device, information processing method, and recording medium - Google Patents

Abstract

The present disclosure relates to an information processing device, an information processing method, and a recording medium that make it possible to more appropriately present a virtual object corresponding to a real object. A detection range setting unit sets, on the basis of object location information of a real object and self-location information in a three-dimensional coordinate system corresponding to a real space, a detection range which is smaller than an imaging angle of view at the self-location and corresponds to the location of the real object. An object detection unit detects the real object in the detection range. The present disclosure is applicable, for example, to an AR terminal that superimposes information onto the real space.

Description

Information processing equipment, information processing methods, and recording media

The present disclosure relates to an information processing device, an information processing method, and a recording medium, and more particularly to an information processing device, an information processing method, and a recording medium that enable more suitable presentation of a virtual object corresponding to a real object.

In recent years, AR (Augmented Reality) technology has been known in which virtual objects of various modes such as text, icons, and animations are superimposed on an object in real space (hereinafter referred to as a real object) and presented to a user. There is.

In Patent Document 1, as a configuration using AR technology, when the position or posture of the user's viewpoint changes, the position shift of the virtual object that occurs between the time when the virtual object is drawn and the time when the virtual object is presented to the user is described. Information processing devices that suppress are disclosed.

Japanese Unexamined Patent Publication No. 2020-3898

In an AR terminal that superimposes a virtual object on a real object that moves at a relatively distant position, there is a risk that the position of the virtual object will shift.

This disclosure is made in view of such a situation, and makes it possible to more preferably present a virtual object corresponding to a real object.

The information processing apparatus of the present disclosure corresponds to the position of the real object, which is smaller than the imaging angle at the self-position, based on the object position information and the self-position information of the real object in the three-dimensional coordinate system corresponding to the real space. It is an information processing apparatus including a detection range setting unit for setting a detection range to be processed and an object detection unit for detecting the real object in the detection range.

In the information processing method of the present disclosure, the information processing apparatus is smaller than the image pickup angle at the self-position and is smaller than the real object based on the object position information and the self-position information of the real object in the three-dimensional coordinate system corresponding to the real space. This is an information processing method in which a detection range corresponding to a position of a body is set and the real object is detected in the detection range.

The recording medium of the present disclosure is smaller than the imaging angle at the self-position and corresponds to the position of the real object based on the object position information and the self-position information of the real object in the three-dimensional coordinate system corresponding to the real space. A computer-readable recording medium on which a program for setting a detection range and executing a process for detecting the real object in the detection range is recorded.

In the present disclosure, based on the object position information and the self-position information of the real object in the three-dimensional coordinate system corresponding to the real space, the detection range smaller than the imaging angle of view at the self-position and corresponding to the position of the real object. Is set, and the real object is detected in the detection range.

It is a figure which shows the example of the network configuration which applied the technique which concerns on this disclosure. It is a figure explaining the presentation of a virtual object. It is a block diagram which shows the configuration example of the real object position estimation system and the distribution server. It is a flowchart explaining operation of a real object position estimation system. It is a flowchart explaining operation of a distribution server. It is a block diagram which shows the configuration example of the real object position estimation system and the distribution server. It is a flowchart explaining operation of a real object position estimation system. It is a block diagram which shows the configuration example of the real object position estimation system and the distribution server. It is a flowchart explaining operation of a real object position estimation system. It is a flowchart explaining operation of a distribution server. It is a block diagram which shows the configuration example of the real object position estimation system and the distribution server. It is a flowchart explaining operation of a real object position estimation system. It is a flowchart explaining operation of a distribution server. It is a block diagram which shows the configuration example of the real object position estimation system and the distribution server. It is a flowchart explaining operation of a distribution server. It is a block diagram which shows the configuration example of the real object position estimation system and the distribution server. It is a flowchart explaining operation of a real object position estimation system. It is a block diagram which shows the structural example of the AR terminal which concerns on 1st Embodiment. It is a figure explaining the setting of the detection range. It is a figure which shows the example of the bounding box set in a three-dimensional coordinate system. It is a figure explaining the projection of the position of a real object on an image plane. It is a flowchart explaining the operation of an AR terminal. It is a flowchart explaining the detail of 3D position acquisition processing. It is a block diagram which shows the structural example of the AR terminal which concerns on 2nd Embodiment. It is a flowchart explaining the detail of 3D position acquisition processing. It is a block diagram which shows the configuration example of a computer.

Hereinafter, a mode for implementing the present disclosure (hereinafter referred to as an embodiment) will be described. The explanation will be given in the following order.

1. 1. Issues of conventional technology 2. Outline of the technology related to this disclosure 3. Configuration and operation of the real object position estimation system and distribution server 3-1. GPS method 3-2. Inside-out method 1
3-3. Inside-out method 2
3-4. Outside-in method 1
3-5. Outside-in method 2
3-6. Outside-in method 3
4. Configuration and operation of AR terminal 4-1. First Embodiment 4-2. Second embodiment 5. Computer configuration example

<1. Issues of conventional technology ＞
For example, consider a case where information is superimposed on a player's head reflected on an AR terminal owned by an spectator in a stadium such as athletics or soccer. In this case, when superimposing a virtual object on a player who moves at a position several hundred meters or more away from the spectator, the AR terminal can identify the player with the camera of the AR terminal, measure the distance to the player with a depth sensor, and so on. Then, it is necessary to grasp the position of the player. However, it is not easy to realize these with a sensor that can be mounted on a mobile terminal that is required to be compact and have low power consumption.

On the other hand, by attaching a sensor that can recognize the player's position to the player, or by recognizing the position of the player by a large number of high-precision sensors installed in the stadium, the distribution server can use wireless communication to communicate with the player. It becomes possible to deliver the location information to the AR terminal of the audience. However, there is a possibility that the position of the virtual object may be displaced due to the transmission delay between the sensor and the distribution server and between the distribution server and the AR terminal.

Therefore, in the technique according to the present disclosure, it is possible to eliminate the misalignment of the virtual object due to the transmission delay described above based on the position of the athlete (real object) detected in the captured image captured by the AR terminal of the spectator. Realize.

<2. Outline of the technology related to this disclosure>
FIG. 1 is a diagram showing an example of a network configuration to which the technique according to the present disclosure is applied.

FIG. 1 shows a real object position estimation system 10, a distribution server 20, and an AR terminal 30. The real object position estimation system 10 and the distribution server 20, the distribution server 20 and the AR terminal 30 each perform wireless communication with each other.

The real object position estimation system 10 senses the position of a real object such as a player RO1 or a formula car RO2 in a three-dimensional coordinate system corresponding to a real space such as a stadium such as land or soccer or a circuit such as F1. .. The real object position estimation system 10 aggregates information such as the position of the real object and the sensing time obtained by sensing in the distribution server 20.

Based on the information aggregated by the real object position estimation system 10, the distribution server 20 has a real object identifier unique to the real object, object position information indicating the position of the real object, sensing time corresponding to the object position information, and the real object. Additional information regarding the above is sequentially delivered to the AR terminal 30.

The AR terminal 30 is configured as a mobile terminal such as a smartphone owned by the above-mentioned stadium or circuit spectator (user U), an HMD (Head Mounted Display), AR goggles, or the like. The AR terminal 30 superimposes a virtual object on the player RO1 in the stadium and the formula car RO2 in the circuit. The AR terminal 30 shares a time axis with the same three-dimensional coordinate system as the real object position estimation system 10, and recognizes the position (self-position) of the own terminal in the three-dimensional coordinate system in real time. The AR terminal 30 uses various information distributed from the distribution server 20 and its own position to acquire the position of the real object at the current time, thereby transmitting the transmission between the real object position estimation system 10 and the AR terminal 30. Compensates for delays and realizes the presentation of virtual objects without misalignment.

The presentation of the virtual object by the AR terminal 30 will be described with reference to FIG.

(1) The AR terminal 30 receives the sensing time t-1 and the object position information representing the position P (t-1) of the real object in the three-dimensional coordinate system at the time t-1 from the distribution server 20.

(2) The AR terminal 30 sets an imaging angle of view including a plane that passes through the position P (t-1) and faces the front of the camera of the AR terminal 30 based on the self-position in the three-dimensional coordinate system. The captured image CI is imaged.

(3) The AR terminal 30 sets a detection range DR (t) capable of detecting a real object at the current time t, which is the current time, in the captured image CI.

(4) The AR terminal 30 converts the position p (t) on the captured image CI of the real object into the position P (t) in the three-dimensional coordinate system by detecting the real object in the detection range DR (t). do.

(5) In the captured image CI, the AR terminal 30 corresponds to the position P (t) of the real object at the current time t in the three-dimensional coordinate system and does not overlap the real object (for example, the real object in the direction of gravity). A virtual object VO corresponding to the additional information from the distribution server 20 is superimposed on the position above the body).

As described above, by superimposing the virtual object VO based on the position of the real object detected in the captured image CI captured by the AR terminal 30, transmission between the real object position estimation system 10 and the AR terminal 30 is performed. It is possible to eliminate the positional shift of the virtual object VO due to the delay.

In the following, the configuration and operation of each part in the network configuration of FIG. 1 will be described.

<3. Configuration and operation of real object position estimation system and distribution server>
As the configuration of the real object position estimation system 10 and the distribution server 20, the following six methods can be adopted.

(3-1. GPS method)
FIG. 3 is a block diagram showing a configuration example of a real object position estimation system 10 and a distribution server 20 that employ a GPS (Global Positioning System) method as a tracking method for a real object. In the example of FIG. 3, the real object position estimation system 10 is configured as, for example, a wearable device worn by each athlete as a real object.

The real object position estimation system 10 of FIG. 3 includes a GPS sensor 51, a coordinate conversion unit 52, a time measurement unit 53, and a transmission unit 54.

The GPS sensor 51 receives GPS position information from GPS satellites and supplies it to the coordinate conversion unit 52. The GPS position information represents a position (latitude, longitude, altitude) in the GPS coordinate system. Instead of the GPS sensor 51, positioning by BLE (Bluetooth (registered trademark) Low Energy), UWB (Ultra Wide Band), or the like may be used, or these positioning techniques may be used in combination.

The coordinate conversion unit 52 converts the GPS position information from the GPS sensor 51 into the position information in the three-dimensional coordinate system, so that the sensor position information indicating the position of the GPS sensor 51 in the three-dimensional coordinate system is transmitted to the transmission unit 54. Supply. In a three-dimensional coordinate system, it is assumed that one of the axes is aligned in the direction of gravity or the direction of gravity is determined. The transformation of coordinates is performed using a predetermined transformation logic or transformation matrix.

The time measuring unit 53 acquires the time (sensing time) when the GPS sensor 51 receives the GPS position information by measuring the time with an accuracy of milliseconds or more, and supplies the time to the transmitting unit 54.

The transmission unit 54 transmits the sensor position information from the coordinate conversion unit 52 and the sensing time from the time measurement unit 53 to the distribution server 20 together with the identifier (sensor identifier) unique to the GPS sensor 51.

The distribution server 20 of FIG. 3 includes a receiving unit 61, a real object identifier conversion unit 62, an additional information acquisition unit 63, and a transmitting unit 64.

The receiving unit 61 receives the sensor identifier, the sensor position information, and the sensing time transmitted from the real object position estimation system 10. The sensor identifier is supplied to the real object identifier conversion unit 62, and the sensor position information and the sensing time are supplied to the transmission unit 64.

The real object identifier conversion unit 62 converts the sensor identifier from the reception unit 61 into an identifier (real object identifier) unique to the real object to which the real object position estimation system 10 is mounted, and transmits it to the additional information acquisition unit 63. Supply to unit 64. The conversion from the sensor identifier to the real object identifier is performed, for example, based on a correspondence table showing which player is wearing which sensor device (real object position estimation system 10).

The additional information acquisition unit 63 acquires the additional information added to the real object as a virtual object, which is the information corresponding to the real object identifier from the real object identifier conversion unit 62, and supplies it to the transmission unit 64. The additional information acquired by the additional information acquisition unit 63 may be, for example, fixed information about a real object such as a player's name, affiliation, or uniform number, or another system such as a player's ranking or score. It may be real-time changing information about a real object obtained from.

The transmission unit 64 transmits the sensor position information and sensing time from the reception unit 61, the real object identifier from the real object identifier conversion unit 62, and the additional information from the additional information acquisition unit 63 to the AR terminal 30. Since the position of the GPS sensor 51 represented by the sensor position information is equal to the position of the real object to which the real object position estimation system 10 is mounted, the sensor position information is AR as the object position information representing the position of the real object. It is transmitted to the terminal 30.

Next, the operations of the real object position estimation system 10 and the distribution server 20 in FIG. 3 will be described.

FIG. 4 is a flowchart illustrating the operation of the real object position estimation system 10 of FIG.

In step S11, the GPS sensor 51 receives GPS position information from GPS satellites.

In step S12, the time measuring unit 53 acquires the time when the GPS sensor 51 receives the GPS position information as the sensing time.

In step S13, the coordinate conversion unit 52 converts the GPS position information received by the GPS sensor 51 into position information (sensor position information) in the three-dimensional coordinate system.

In step S14, the transmission unit 54 transmits the sensor identifier of the GPS sensor 51, the sensor position information, and the sensing time to the distribution server 20.

FIG. 5 is a flowchart illustrating the operation of the distribution server 20 of FIG.

In step S21, the receiving unit 61 receives the sensor identifier, the sensor position information, and the sensing time transmitted from the real object position estimation system 10.

In step S22, the real object identifier conversion unit 62 converts the sensor identifier received by the reception unit 61 into a real object identifier.

In step S23, the additional information acquisition unit 63 acquires additional information corresponding to the converted real object identifier.

In step S24, the transmission unit 64 transmits the real object identifier, the object position information (sensor position information), the sensing time, and the additional information to the AR terminal 30.

As described above, in the configuration of FIG. 3, the position of the real object is acquired based on the GPS position information.

(3-2. Inside-out method 1)
FIG. 6 is a block diagram showing a configuration example of a real object position estimation system 10 and a distribution server 20 that employ an inside-out method as a tracking method for a real object. In the inside-out method, the position of the object is measured by using the sensor mounted on the object itself to be measured. Therefore, in the example of FIG. 6, the real object position estimation system 10 is configured as, for example, a wearable device worn by each athlete as a real object.

The real object position estimation system 10 of FIG. 6 includes a sensor unit 71, a self-position estimation unit 72, a time measurement unit 53, and a transmission unit 54. Since the time measuring unit 53 and the transmitting unit 54 have the same configuration as shown in FIG. 3, the description thereof will be omitted.

The sensor unit 71 is composed of a stereo camera, a depth sensor, and the like, senses the environment around the real object, and supplies the sensing result to the self-position estimation unit 72.

The self-position estimation unit 72 estimates the position of the sensor unit 71 in the three-dimensional coordinate system based on the sensing result from the sensor unit 71, and thereby transmits the sensor position information indicating the position of the sensor unit 71 to the transmission unit 54. Supply. In addition to the sensor unit 71, an IMU (Inertial Measurement Unit) is provided so that the position of the sensor unit 71 in the three-dimensional coordinate system can be estimated based on the sensing result by the sensor unit 71 and the angular velocity and acceleration detected by the IMU. You may do it.

Since the distribution server 20 of FIG. 6 has the same configuration as that shown in FIG. 3, the description thereof will be omitted.

Next, the operations of the real object position estimation system 10 and the distribution server 20 of FIG. 6 will be described.

FIG. 7 is a flowchart illustrating the operation of the real object position estimation system 10 of FIG.

In step S31, the sensor unit 71 senses the environment around the real object.

In step S32, the time measuring unit 53 acquires the time sensed by the sensor unit 71 as the sensing time.

In step S33, the self-position estimation unit 72 estimates the position of the sensor unit 71 in the three-dimensional coordinate system based on the sensing result of the sensing by the sensor unit 71.

In step S34, the transmission unit 54 transmits the sensor identifier of the sensor unit 71, the sensor position information indicating the estimated position of the sensor unit 71, and the sensing time to the distribution server 20.

Since the operation of the distribution server 20 in FIG. 6 is the same as the operation of the distribution server 20 in FIG. 3 described with reference to FIG. 5, the description thereof will be omitted.

As described above, in the configuration of FIG. 6, the position of the real object is estimated by the real object position estimation system 10 mounted on the real object.

(3-3. Inside-out method 2)
FIG. 8 is a block diagram showing a configuration example of a real object position estimation system 10 and a distribution server 20 that employ an inside-out method as a tracking method for a real object. Therefore, even in the example of FIG. 8, the real object position estimation system 10 is configured as, for example, a wearable device worn by each athlete as a real object.

In the real object position estimation system 10 and the distribution server 20 of FIG. 8, the self-position estimation unit 72 is provided in the distribution server 20 instead of the real object position estimation system 10, and the real object position estimation system 10 of FIG. 6 is provided. And different from the distribution server 20.

That is, in the real object position estimation system 10 of FIG. 8, the sensor unit 71 supplies the sensing result to the transmitting unit 54, and the transmitting unit 54 transmits the sensing result as it is to the distribution server 20 instead of the sensor position information. do.

In the distribution server 20 of FIG. 8, the self-position estimation unit 72 estimates the position of the sensor unit 71 in the three-dimensional coordinate system based on the sensing result from the real object position estimation system 10, so that the sensor unit 71 The sensor position information indicating the position is supplied to the transmission unit 54.

Next, the operations of the real object position estimation system 10 and the distribution server 20 of FIG. 8 will be described.

FIG. 9 is a flowchart illustrating the operation of the real object position estimation system 10 of FIG.

In step S41, the sensor unit 71 senses the environment around the real object.

In step S42, the time measuring unit 53 acquires the time sensed by the sensor unit 71 as the sensing time.

In step S43, the transmission unit 54 transmits the sensor identifier of the sensor unit 71, the sensing result, and the sensing time to the distribution server 20.

FIG. 10 is a flowchart illustrating the operation of the distribution server 20 of FIG.

In step S51, the receiving unit 61 receives the sensor identifier, the sensing result, and the sensing time transmitted from the real object position estimation system 10.

In step S52, the real object identifier conversion unit 62 converts the sensor identifier received by the reception unit 61 into a real object identifier.

In step S53, the self-position estimation unit 72 is equipped with the position of the sensor unit 71 in the three-dimensional coordinate system, that is, the real object position estimation system 10 based on the sensing result from the real object position estimation system 10. Estimate the position of the body.

In step S54, the additional information acquisition unit 63 acquires additional information corresponding to the converted real object identifier.

In step S55, the transmission unit 64 transmits the real object identifier, the object position information, the sensing time, and the additional information to the AR terminal 30.

As described above, in the configuration of FIG. 8, the position of the real object is estimated by the distribution server 20 instead of the real object position estimation system 10 mounted on the real object.

(3-4. Outside-in method 1)
FIG. 11 is a block diagram showing a configuration example of a real object position estimation system 10 and a distribution server 20 that employ an outside-in method as a tracking method for a real object. In the outside-in method, the position of the object is measured using a sensor installed outside. A method of attaching a marker to an object and observing it with an external camera can be mentioned. Therefore, in the example of FIG. 11, the real object position estimation system 10 is arranged around the real object, and is configured as, for example, a plurality of high-precision sensor devices installed so as to surround the stadium.

The real object position estimation system 10 of FIG. 11 includes a sensor unit 71, a real object position estimation unit 81, a real object identification unit 82, a time measurement unit 53, and a transmission unit 83. Since the sensor unit 71 has the same configuration as shown in FIG. 6 and the time measuring unit 53 has the same configuration as shown in FIG. 3, the description thereof will be omitted.

The real object position estimation unit 81 estimates the position of the real object in the three-dimensional coordinate system based on the sensing result from the sensor unit 71, and supplies the object position information indicating the position of the real object to the transmission unit 83. do. Here, since the position and orientation of the sensor unit 71 (for example, the depth sensor) in the three-dimensional coordinate system are known, the position of the real object can be estimated based on the depth information from the depth sensor. When the sensor unit 71 is composed of a camera, the position of the real object may be estimated by image recognition for the captured image captured by the camera. The object position information may represent the position of one point such as the center of gravity of the real object or the center of the bounding box that recognizes the real object.

The real object identification unit 82 identifies the real object based on the sensing result from the sensor unit 71, and supplies the real object identifier unique to the identified real object to the transmission unit 83. Here, each player is identified by recognizing the player's face and uniform number in the captured image as the sensing result. Athletes as real objects may be equipped with markers and infrared lamps necessary for identification.

The transmission unit 83 transmits the object position information from the real object position estimation unit 81, the real object identifier from the real object identification unit 82, and the sensing time from the time measurement unit 53 to the distribution server 20.

The distribution server 20 of FIG. 11 includes a receiving unit 61, an additional information acquisition unit 63, and a transmitting unit 64. Since the receiving unit 61, the additional information acquisition unit 63, and the transmitting unit 64 have the same configuration as shown in FIG. 3, the description thereof will be omitted.

Next, the operations of the real object position estimation system 10 and the distribution server 20 in FIG. 11 will be described.

FIG. 12 is a flowchart illustrating the operation of the real object position estimation system 10 of FIG.

In step S61, the sensor unit 71 senses the environment in which the real object exists.

In step S62, the time measuring unit 53 acquires the time sensed by the sensor unit 71 as the sensing time.

In step S63, the real object position estimation unit 81 estimates the position of the real object in the three-dimensional coordinate system from the known position of the sensor unit 71 (sensor position) and the sensing result from the sensor unit 71.

In step S64, the real object identification unit 82 identifies the real object from the sensing result from the sensor unit 71.

In step S65, the transmission unit 83 transmits the real object identifier, the object position information, and the sensing time to the distribution server 20.

FIG. 13 is a flowchart illustrating the operation of the distribution server 20 of FIG.

In step S71, the receiving unit 61 receives the real object identifier, the object position information, and the sensing time transmitted from the real object position estimation system 10.

In step S72, the additional information acquisition unit 63 acquires additional information corresponding to the real object identifier received by the reception unit 61.

In step S73, the transmission unit 64 transmits the real object identifier, the object position information, the sensing time, and the additional information to the AR terminal 30.

As described above, in the configuration of FIG. 11, the position of the real object is acquired by the plurality of real object position estimation systems 10 installed outside the real object.

(3-5. Outside-in method 2)
FIG. 14 is a block diagram showing a configuration example of a real object position estimation system 10 and a distribution server 20 that employ an outside-in method as a tracking method for a real object. Therefore, even in the example of FIG. 14, the real object position estimation system 10 is configured as, for example, a plurality of high-precision sensor devices installed so as to surround the stadium.

In the real object position estimation system 10 and the distribution server 20 of FIG. 14, the real object position estimation unit 81 and the real object identification unit 82 are provided in the distribution server 20 instead of the real object position estimation system 10. It is different from the real object position estimation system 10 and the distribution server 20 of the above. The distribution server 20 of FIG. 14 is further provided with a sensor position acquisition unit 91.

That is, in the real object position estimation system 10 of FIG. 14, the sensor unit 71 supplies the sensing result to the transmission unit 83, and the transmission unit 83 replaces the real object identifier and the object position information with the sensor identifier and the sensing result. Is transmitted to the distribution server 20 as it is.

In the distribution server 20 of FIG. 14, the sensor position acquisition unit 91 acquires the position / orientation of the sensor unit 71 in the three-dimensional coordinate system based on the sensor identifier from the actual object position estimation system 10, and estimates the actual object position. It is supplied to the unit 81. The acquisition of the position / posture of the sensor unit 71 is performed, for example, based on a correspondence table showing the correspondence between the position / posture of each sensor unit 71 measured in advance and the sensor identifier.

The real object position estimation unit 81 is a real object in the three-dimensional coordinate system based on the sensing result from the real object position estimation system 10 and the position / orientation of the sensor unit 71 in the three-dimensional coordinate system from the sensor position acquisition unit 91. The position of the object is estimated, and the object position information is supplied to the transmission unit 64.

The real object identification unit 82 identifies the real object based on the sensing result from the real object position estimation system 10, and assigns the real object identifier unique to the identified real object to the additional information acquisition unit 63 and the transmission unit 64. Supply.

Next, the operations of the real object position estimation system 10 and the distribution server 20 of FIG. 14 will be described.

Since the operation of the real object position estimation system 10 of FIG. 14 is the same as the operation of the real object position estimation system 10 of FIG. 8 described with reference to FIG. 9, the description thereof will be omitted.

FIG. 15 is a flowchart illustrating the operation of the distribution server 20 of FIG.

In step S81, the receiving unit 61 receives the sensor identifier, the sensing result, and the sensing time transmitted from the real object position estimation system 10.

In step S82, the sensor position acquisition unit 91 acquires the position (sensor position) of the sensor unit 71 in the three-dimensional coordinate system based on the sensor identifier received by the reception unit 61.

In step S83, the real object position estimation unit 81 estimates the position of the real object in the three-dimensional coordinate system from the sensor position acquired by the sensor position acquisition unit 91 and the sensing result received by the reception unit 61.

In step S84, the real object identification unit 82 identifies the real object from the sensing result received by the reception unit 61.

In step S85, the additional information acquisition unit 63 acquires additional information corresponding to the real object identifier unique to the identified real object.

In step S86, the transmission unit 64 transmits the real object identifier, the object position information, the sensing time, and the additional information to the AR terminal 30.

As described above, in the configuration of FIG. 15, the real object is identified and the position of the real object is estimated by the distribution server 20 instead of the real object position estimation system 10 mounted on the real object.

(3-6. Outside-in method 3)
FIG. 16 is a block diagram showing a configuration example of a real object position estimation system 10 and a distribution server 20 that employ an outside-in method as a tracking method for a real object. However, in the example of FIG. 16, the real object position estimation system 10 is configured as a moving body (for example, a drone flying over the stadium) that moves around the real object.

The real object position estimation system 10 of FIG. 16 is different from the real object position estimation system 10 of FIG. 11 in that a self-position estimation unit 72 and a control unit 101 are further provided in the distribution server 20. The distribution server 20 of FIG. 16 is configured in the same manner as the distribution server 20 of FIG.

In the real object position estimation system 10 of FIG. 16, the self-position estimation unit 72 estimates the position of the sensor unit 71 (real object position estimation system 10) in the three-dimensional coordinate system based on the sensing result from the sensor unit 71. Then, it is supplied to the real object position estimation unit 81 and the control unit 101.

The real object position estimation unit 81 estimates the position of the real object in the three-dimensional coordinate system based on the sensing result from the sensor unit 71 and the position of the sensor unit 71 in the three-dimensional coordinate system from the self-position estimation unit 72. , The object position information is supplied to the transmission unit 83.

The control unit 101 determines that the real object position estimation system 10 as a drone is localized or moved to a predetermined position based on the position of the sensor unit 71 in the three-dimensional coordinate system from the self-position estimation unit 72. Control actuators not shown.

Next, the operations of the real object position estimation system 10 and the distribution server 20 of FIG. 16 will be described.

FIG. 17 is a flowchart illustrating the operation of the real object position estimation system 10 of FIG.

In step S91, the sensor unit 71 senses the environment in which the real object exists, in which the real object position estimation system 10 as a drone is flying.

In step S92, the time measuring unit 53 acquires the time sensed by the sensor unit 71 as the sensing time.

In step S93, the self-position estimation unit 72 estimates the position (sensor position) of the sensor unit 71 (real object position estimation system 10) in the three-dimensional coordinate system based on the sensing result of the sensing by the sensor unit 71.

In step S94, the control unit 101 controls an actuator for flying the real object position estimation system 10 based on the sensor position estimated by the self-position estimation unit 72.

In step S95, the real object position estimation unit 81 estimates the position of the real object in the three-dimensional coordinate system from the sensor position estimated by the self-position estimation unit 72 and the sensing result from the sensor unit 71.

In step S96, the real object identification unit 82 identifies the real object from the sensing result from the sensor unit 71.

In step S97, the transmission unit 83 transmits the real object identifier, the object position information, and the sensing time to the distribution server 20.

Since the operation of the distribution server 20 of FIG. 16 is the same as the operation of the distribution server 20 of FIG. 11 described with reference to FIG. 13, the description thereof will be omitted.

As described above, in the configuration of FIG. 16, the position of the real object is acquired by the real object position estimation system 10 flying around the real object.

<4. Configuration and operation of AR terminal>
The configuration of the AR terminal 30 will be described.

(4-1. First Embodiment)
FIG. 18 is a block diagram showing a configuration example of the AR terminal 30 according to the first embodiment. In the example of FIG. 18, the AR terminal 30 acquires the position of the real object at the current time based only on the information distributed from the distribution server 20, and arranges the corresponding virtual object.

The AR terminal 30 of FIG. 18 includes a receiving unit 111, a time measuring unit 112, a sensor unit 113, a self-position estimation unit 114, a moving range prediction unit 115, a detection range setting unit 116, an object detection unit 117, and a virtual object arrangement unit 118. It includes a drawing unit 119 and a display unit 120.

The receiving unit 111 receives the real object identifier, the object position information, the sensing time, and the additional information distributed from the distribution server 20. The object position information representing the position of the real object is not the information acquired by the AR terminal 30, but the information based on the sensing result by the external sensor acquired by wireless communication. The object position information and the sensing time are associated with the real object identifier and supplied to the movement range prediction unit 115, and the additional information is associated with the real object identifier and supplied to the virtual object arrangement unit 118.

The time measurement unit 112 acquires the current time (current time) by measuring the time with an accuracy of milliseconds or more, and supplies it to the movement range prediction unit 115.

The sensor unit 113 is composed of a stereo camera, a depth sensor, or the like, senses the environment around the AR terminal 30, and supplies the sensing result to the self-position estimation unit 114 and the detection range setting unit 116.

The self-position estimation unit 114 estimates the position of the sensor unit 113 (AR terminal 30) in the three-dimensional coordinate system based on the sensing result from the sensor unit 113, and thereby obtains self-position information indicating the position of the AR terminal 30. , Supply to the detection range setting unit 116 and the drawing unit 119. In addition to the sensor unit 113, an IMU may be provided so that the position of the sensor unit 113 in the three-dimensional coordinate system can be estimated based on the sensing result by the sensor unit 113 and the angular velocity and acceleration detected by the IMU.

The movement range prediction unit 115 predicts the movement range in which the real object can move in the three-dimensional coordinate system based on the object position information from the reception unit 111. Specifically, the movement range prediction unit 115 predicts the movement range of the actual object from the sensing time from the reception unit 111 corresponding to the object position information to the current time from the time measurement unit 112. When there are a plurality of real objects in the three-dimensional coordinate system, the movement range is predicted for each real object based on the real object identifier unique to each real object.

Here, the moving range of the real object is predicted by estimating the predicted position of the real object at the current time from the moving speed and moving direction of the real object based on the object position information. For example, the velocity vector (traveling direction and velocity) of the real object is estimated based on the difference of the object position information for the last two times received for a certain real object. The predicted position of the real object at the current time is estimated using the estimated velocity vector with the travel time between the received sensing time and the current time as the travel time. The movement range is between the predicted positions estimated from the positions represented by the latest object position information.

Further, the movement range of the real object may be predicted using the context of the real object. The context here is the maximum speed at which the real object can move, or the plane or direction in which the real object can move. When the real object is a human, the maximum speed is, for example, 12.5 m / s based on the world record of land (sprinting). When the real object is a car, the maximum speed is, for example, 378 km / h, which is the world record of an F1 car. In addition, the real object can move on a plane perpendicular to the direction of gravity. Further, in a track and field stadium or a circuit such as F1, the traveling direction of the real object is uniquely determined depending on the position of the real object on the course. However, in ball games such as soccer, when it is difficult to estimate the traveling direction of a real object (player), all directions should be included in the movement range on a plane perpendicular to the direction of gravity. .. Given such a context, it is possible to limit the range of movement according to the real object.

Further, the movement range predicted as described above may be provided with a margin such as 1.2 times, or may include a positioning error or a distance measurement error of the GPS sensor 51 or the sensor unit 71. .. For example, when the GPS method is adopted as the tracking method for a real object, the movement range is predicted including a positioning error of about several meters.

The detection range setting unit 116 is smaller than the imaging angle of view at the self-position and corresponds to the position of the real object based on the object position information of the real object and the self-position information from the self-position estimation unit 114. To set. The detection range is set as a range in which a real object can be detected in the captured image captured at the imaging angle of view at the self-position. Specifically, the detection range setting unit 116 uses the self-position information from the self-position estimation unit 114 to capture an image angle of view including the movement range of the real object in the three-dimensional coordinate system predicted by the movement range prediction unit 115. To set. The detection range setting unit 116 sets the movement range predicted by the movement range prediction unit 115 based on the object position information in the image captured by the sensor unit 113 at the set angle of view (angle of view including the real object). The range including the corresponding area is set as the detection range.

For example, it is assumed that the movement range from the sensing time t-1 corresponding to the object position information to the current time t is predicted by the movement range prediction unit 115. In this case, as shown in FIG. 19, the detection range setting unit 116 is based on the self-position in the captured image CI captured at the imaging angle of view including the moving range from time t-1 to time t. , The detection range DR (t) capable of detecting a real object at the current time t is set.

More specifically, based on the predicted range of movement, the smallest rectangular region in the captured image CI that is predicted to capture the shape of a person between time t-1 and time t is determined. .. First, a bounding box BB is set on the three-dimensional coordinate system as shown in FIG. 20 for the shapes of people at time t-1 and time t. When the positions of real objects in the three-dimensional coordinate system are aggregated to one point, the bounding box BB is set so that the height of the person is parallel to the direction of gravity and the aggregated one point is the center of gravity. The height of the person here may be a world record such as 2.5 m, or may be included in the additional information from the distribution server 20.

Next, based on the camera model shown in FIG. 21, the eight vertices of the bounding box BB for the real object at time t-1 and the eight vertices of the bounding box BB for the real object at time t on the world coordinate system. Is projected onto the uv coordinate system. In FIG. 21, the point P on the world coordinate system (three-dimensional coordinate system) (Xw-Yw-Zw coordinate system) is an image of the captured image CI via the camera coordinate system (Xc-Yc-Zc coordinate system). It is projected onto the point p on the coordinate system (uv coordinate system). Then, a rectangular area surrounded by the minimum value of u, the maximum value of u, the minimum value of v, and the maximum value of v in the uv coordinate system is set in the detection range. The detection range described here is an example, and its shape is not limited to a rectangle such as a circle.

The object detection unit 117 detects a real object within the detection range set in the captured image, and converts the position of the detected real object on the captured image into a position in the three-dimensional coordinate system.

First, the object detection unit 117 acquires the center of gravity of the rectangular detection frame in which the real object is detected in the detection range as the position on the captured image of the real object. If the real object is limited to humans, only humans will be detected. Further, when a plurality of people are detected, the position closest to the position on the captured image corresponding to the predicted position at the current time is adopted. Semantic segmentation, which estimates the subject based on the attributes of each pixel of the captured image, may be used to detect the real object.

Next, the object detection unit 117 converts the position on the captured image (position on the uv coordinate system) of the detected real object to the position on the three-dimensional coordinate system. Here, the position on the three-dimensional coordinate system corresponding to the Xc—Yc coordinates excluding the depth direction in the camera coordinate system is obtained by the back projection in the above-mentioned camera model. It is assumed that the predicted position at the current time t estimated by the movement range prediction unit 115 is applied to the position on the three-dimensional coordinate system corresponding to the position in the depth direction in the camera coordinate system. This is because the position in the depth direction is not accurate, but the difference from the actual position is considered to be small, and human beings are insensitive to the difference in the position in the depth direction. This is because it does not give a sense of discomfort.

In this way, the object detection unit 117 acquires the position of the real object at the current time in the three-dimensional coordinate system.

The virtual object arranging unit 118 virtually generates additional information from the receiving unit 111 on the three-dimensional coordinate system corresponding to the position of the real object at the current time in the three-dimensional coordinate system acquired by the object detection unit 117. Place it as an object. The virtual object arranging unit 118 arranges the virtual object at a position that does not overlap with the real object, for example, at a position several tens of centimeters above the real object in the direction of gravity.

The drawing unit 119 renders a virtual object arranged on the three-dimensional coordinate system by the virtual object arrangement unit 118 based on the self-position information from the self-position estimation unit 114.

The display unit 120 is composed of a display or the like, and displays a virtual object rendered by the drawing unit 119.

(Operation of AR terminal)
The operation of the AR terminal 30 will be described with reference to the flowchart of FIG.

In step S111, the sensor unit 113 senses the environment around the AR terminal 30 (for example, the stadium or circuit in which the user U is watching).

In step S112, the time measuring unit 112 acquires the time sensed by the sensor unit 113 as the current time.

In step S113, the self-position estimation unit 114 estimates the self-position (position of the AR terminal 30) in the three-dimensional coordinate system based on the sensing result of the sensing by the sensor unit 113.

In step S114, the AR terminal 30 executes a three-dimensional position acquisition process for acquiring the position of the real object at the current time in the three-dimensional coordinate system. The details of the three-dimensional position acquisition process will be described later.

In step S115, the virtual object arranging unit 118 arranges the virtual object on the three-dimensional coordinate system corresponding to the position of the real object at the current time in the three-dimensional coordinate system acquired by the three-dimensional position acquisition process. ..

In step S116, the drawing unit 119 renders a virtual object arranged on the three-dimensional coordinate system by the virtual object arrangement unit 118 based on the self-position information representing the self-position estimated by the self-position estimation unit 114.

In step S117, the display unit 120 displays the virtual object rendered by the drawing unit 119.

FIG. 23 is a flowchart illustrating details of the three-dimensional position acquisition process executed in step S114 of FIG. 22.

In step S121, the receiving unit 111 receives the real object identifier, the object position information, the sensing time, and the additional information distributed from the distribution server 20.

In step S122, the movement range prediction unit 115 predicts the movement range of the real object in the three-dimensional coordinate system based on the corresponding object position information for all the received real object identifiers.

In step S123, the detection range setting unit 116 is predicted for all real object identifiers in the captured image captured at the self-position using the self-position information representing the self-position estimated by the self-position estimation unit 114. Set the detection range corresponding to the movement range.

In step S124, the object detection unit 117 detects the corresponding real object in each of the detection ranges set for all the real object identifiers.

In step S125, the object detection unit 117 converts the positions of all the detected real objects on the captured image into the positions in the three-dimensional coordinate system. As a result, the position of the real object at the current time in the three-dimensional coordinate system is acquired.

According to the above processing, in the AR terminal 30, the detection range smaller than the imaging angle of view and corresponding to the position of the real object is set, so that the real object can be detected with less processing. Further, as the position of the real object to be detected, the position of the real object at the current time detected on the captured image is adopted instead of the position distributed from the distribution server 20, so that the position of the real object is estimated from the real object position estimation system 10. It is possible to compensate for the transmission delay up to the AR terminal 30. As a result, it is possible to eliminate the positional deviation of the virtual object while reducing the processing load of the AR terminal 30, and it is possible to more preferably present the virtual object corresponding to the real object.

(4-2. Second embodiment)
FIG. 24 is a block diagram showing a configuration example of the AR terminal 30 according to the second embodiment. In the example of FIG. 24, the AR terminal 30 acquires the position of the real object at the current time by using the information distributed from the distribution server 20 and the distance measurement result from the own terminal to the real object together. Place the corresponding virtual object.

The AR terminal 30 of FIG. 24 includes a real object position estimation unit 131, a real object identification unit 132, and a real object position selection unit 133, in addition to the reception unit 111 to the display unit 120 of the AR terminal 30 of FIG.

The real object position estimation unit 131 estimates the position of the real object in the three-dimensional coordinate system based on the sensing result from the sensor unit 113 and the self-position information from the self-position estimation unit 114, and the estimation result is the actual object. It is supplied to the body position selection unit 133. The sensing result from the sensor unit 113 is, for example, the distance to the actual object measured by the depth sensor.

The real object identification unit 132 identifies the real object based on the sensing result from the sensor unit 113, and supplies the real object identifier unique to the identified real object to the real object position selection unit 133.

When the position of the real object can be estimated by the real object position estimation unit 131, the real object position selection unit 133 selects the estimation result as the position of the real object, and the real object from the real object identification unit 132. It is supplied to the virtual object arrangement unit 118 in association with the identifier. When the position of the real object cannot be estimated by the real object position estimation unit 131, the position of the real object acquired by the object detection unit 117 is selected as the position of the real object and supplied to the virtual object arrangement unit 118. ..

(Operation of AR terminal)
The operation of the AR terminal 30 of FIG. 24 is basically the same as the operation of the AR terminal 30 of FIG. 18 described with reference to the flowchart of FIG. 22 except for step S114.

FIG. 25 is a flowchart illustrating details of the three-dimensional position acquisition process executed in step S114 of FIG. 22 by the AR terminal 30 of FIG. 24.

In step S131, the real object identification unit 132 identifies the real object from the sensing result from the sensor unit 113.

In step S132, the real object position estimation unit 131 is the position of the real object in the three-dimensional coordinate system from the self-position (position of the AR terminal 30) estimated by the self-position estimation unit 114 and the sensing result from the sensor unit 113. To estimate.

In step S133, the AR terminal 30 of FIG. 24 acquires the positions of all real objects at the current time in the three-dimensional coordinate system by executing the process described with reference to the flowchart of FIG. 23.

In step S134, the real object position selection unit 133 determines whether or not the position of the real object can be estimated by the real object position estimation unit 131 for all the real object identifiers from the real object identification unit 132. If it is determined that the position of the real object can be estimated, the process proceeds to step S135.

In step S135, the real object position selection unit 133 replaces the position of the real object acquired in step S133 with the position of the real object estimated in step S132 for all the real object identifiers. Here, the position of the real object acquired in step S133 may be replaced with the position of the real object estimated in step S132 only for the real object whose position is determined to be estimable.

On the other hand, if it is not determined in step S134 that the position of the real object could be estimated, step S135 is skipped and the positions of the real objects acquired in step S133 are adopted as the positions of all the real objects. Here, the position of the real object acquired in step S133 may be adopted only for the real object whose position is not determined to be estimable.

According to the above processing, when the position of the real object can be estimated within the range measuring range such as the depth sensor provided in the AR terminal 30, such as when the real object exists at a close distance from the AR terminal 30, the estimated real object is obtained. The position of the body is adopted. In this case, it is not necessary to compensate for the transmission delay between the real object position estimation system 10 and the AR terminal 30, and it is possible to more preferably present a virtual object corresponding to the real object.

In the above, an example in which the technology according to the present disclosure is applied to a configuration in which a virtual object is superimposed on a player in a stadium such as athletics or soccer or a formula car in a circuit such as F1 has been described. Not limited to these, the technology according to the present disclosure may be applied to a configuration in which a virtual object is superimposed on a vehicle arranged by a user in, for example, a vehicle allocation application in which a taxi, a hire, or the like can be arranged. Further, the technique according to the present disclosure may be applied to a configuration in which a virtual object is superimposed on a player around the user in an AR shooting game that can be played from a first-person viewpoint such as FPS (First Person Shooter).

<5. Computer configuration example>
The series of processes described above can be executed by hardware or software. When a series of processes is executed by software, the programs constituting the software are installed on the computer. Here, the computer includes a computer embedded in dedicated hardware and, for example, a general-purpose personal computer capable of executing various functions by installing various programs.

FIG. 26 is a block diagram showing a configuration example of computer hardware that executes the above-mentioned series of processes programmatically.

In the computer, the CPU 501, the ROM (ReadOnlyMemory) 502, and the RAM (RandomAccessMemory) 503 are connected to each other by the bus 504.

An input / output interface 505 is further connected to the bus 504. An input unit 506, an output unit 507, a storage unit 508, a communication unit 509, and a drive 510 are connected to the input / output interface 505.

The input unit 506 includes a keyboard, a mouse, a microphone, and the like. The output unit 507 includes a display, a speaker, and the like. The storage unit 508 includes a hard disk, a non-volatile memory, and the like. The communication unit 509 includes a network interface and the like. The drive 510 drives a removable medium 511 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

In the computer configured as described above, the CPU 501 loads the program stored in the storage unit 508 into the RAM 503 via the input / output interface 505 and the bus 504 and executes the above-mentioned series. Is processed.

The program executed by the computer (CPU 501) can be recorded and provided on the removable media 511 as a package media or the like, for example. The program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

In the computer, the program can be installed in the storage unit 508 via the input / output interface 505 by mounting the removable media 511 in the drive 510. Further, the program can be received by the communication unit 509 and installed in the storage unit 508 via a wired or wireless transmission medium. In addition, the program can be installed in the ROM 502 or the storage unit 508 in advance.

The program executed by the computer may be a program in which processing is performed in chronological order according to the order described in the present specification, in parallel, or at a necessary timing such as when a call is made. It may be a program in which processing is performed.

The embodiments of the present disclosure are not limited to the embodiments described above, and various changes can be made without departing from the gist of the present disclosure.

The effects described in this specification are merely examples and are not limited, and other effects may be used.

Further, the present disclosure may have the following structure.
(1)
A detection range that sets a detection range that is smaller than the imaging angle of view at the self-position and corresponds to the position of the real object, based on the object position information and the self-position information of the real object in the three-dimensional coordinate system corresponding to the real space. Setting part and
An information processing device including an object detection unit that detects the real object in the detection range.
(2)
The detection range setting unit sets the detection range capable of detecting the real object in the captured image captured at the imaging angle of view.
The information processing apparatus according to (1), wherein the object detection unit converts a position of the real object detected in the detection range on the captured image into a position in the three-dimensional coordinate system.
(3)
Further, a movement range prediction unit for predicting a movement range in which the real object can move in the three-dimensional coordinate system based on the object position information is provided.
The information processing apparatus according to (2), wherein the detection range setting unit sets the detection range in the captured image captured at the imaging angle of view including the moving range by using the self-position information.
(4)
The information processing apparatus according to (3), wherein the detection range setting unit sets the detection range including a region corresponding to the movement range in the captured image.
(5)
The information processing apparatus according to (4), wherein the detection range setting unit sets the minimum region in which the real object is predicted to be captured in the captured image based on the movement range in the detection range.
(6)
The movement range prediction unit predicts the movement range from the sensing time when the real object is sensed by an external sensor to the current time.
The detection range setting unit sets the detection range at the current time corresponding to the movement range, and sets the detection range.
The information processing device according to (4) or (5), wherein the object detection unit acquires the position of the real object at the current time in the three-dimensional coordinate system.
(7)
The movement range prediction unit predicts the movement range by estimating the predicted position of the real object at the current time from the movement speed and the movement direction of the real object based on the object position information (6). The information processing device described in.
(8)
The detection range setting unit projects the position of the real object at the sensing time in the three-dimensional coordinate system and the predicted position of the real object at the current time onto the image coordinate system of the captured image. , The information processing apparatus according to (7), which sets the detection range at the current time.
(9)
The information processing apparatus according to any one of (3) to (8), wherein the movement range prediction unit predicts the movement range using the context of the real object.
(10)
9. The information processing apparatus according to (9), wherein the context includes at least one of the maximum velocity, plane, and direction in which the real object can move.
(11)
When a plurality of the real objects exist, the movement range prediction unit predicts the movement range for each real object based on the real object identifier unique to the real object, whichever is (3) to (10). Information processing device described in Crab.
(12)
The information processing apparatus according to any one of (1) to (11), wherein the object position information is information acquired by wireless communication and based on a sensing result by an external sensor.
(13)
The sensor is a first sensor mounted on the real object, a second sensor arranged around the real object, and a third sensor mounted on a moving body moving around the real object. The information processing apparatus according to (12), which is configured as at least one of.
(14)
Any of (1) to (13) further including a virtual object arranging unit for arranging a virtual object corresponding to the position of the real object at the current time in the three-dimensional coordinate system detected in the detection range. The information processing device described in.
(15)
The information processing apparatus according to (14), wherein the virtual object arranging unit arranges the virtual object at a position that does not overlap with the real object.
(16)
The information processing apparatus according to (14) or (15), wherein the virtual object arranging unit arranges additional information about the real object acquired together with the object position information as the virtual object.
(17)
A drawing unit that renders the virtual object based on the self-position information,
The information processing apparatus according to any one of (14) to (16), further comprising a display unit for displaying the rendered virtual object.
(18)
Further provided with a real object position estimation unit that estimates the position of the real object in the three-dimensional coordinate system based on the self-position information and the distance to the real object.
When the real object position estimation unit can estimate the position of the real object, the virtual object placement unit arranges the virtual object based on the estimated position of the real object (14) to (17). The information processing device described in any of the above.
(19)
Information processing equipment
Based on the object position information and self-position information of the real object in the three-dimensional coordinate system corresponding to the real space, a detection range smaller than the imaging angle of view at the self-position and corresponding to the position of the real object is set.
An information processing method for detecting the real object in the detection range.
(20)
Based on the object position information and self-position information of the real object in the three-dimensional coordinate system corresponding to the real space, a detection range smaller than the imaging angle of view at the self-position and corresponding to the position of the real object is set.
A computer-readable recording medium on which a program for executing a process for detecting the real object in the detection range is recorded.

10 Real object position estimation system, 20 distribution server, 30 AR terminal, 111 receiver, 112 time measurement unit, 113 sensor unit, 114 self-position estimation unit, 115 movement range prediction unit, 116 detection range setting unit, 117 object detection unit , 118 virtual object placement unit, 119 drawing unit, 120 display unit, 131 real object position estimation unit, 132 real object identification unit, 133 real object position selection unit

Claims (20)

1. A detection range that sets a detection range that is smaller than the imaging angle of view at the self-position and corresponds to the position of the real object, based on the object position information and the self-position information of the real object in the three-dimensional coordinate system corresponding to the real space. Setting part and
   An information processing device including an object detection unit that detects the real object in the detection range.
2. The detection range setting unit sets the detection range capable of detecting the real object in the captured image captured at the imaging angle of view.
   The information processing apparatus according to claim 1, wherein the object detection unit converts a position of the real object detected in the detection range on the captured image into a position in the three-dimensional coordinate system.
3. Further, a movement range prediction unit for predicting a movement range in which the real object can move in the three-dimensional coordinate system based on the object position information is provided.
   The information processing apparatus according to claim 2, wherein the detection range setting unit uses the self-position information to set the detection range in the captured image captured at the imaging angle of view including the moving range.
4. The information processing apparatus according to claim 3, wherein the detection range setting unit sets the detection range including a region corresponding to the movement range in the captured image.
5. The information processing apparatus according to claim 4, wherein the detection range setting unit sets the minimum region in which the real object is predicted to be captured in the captured image based on the movement range in the detection range.
6. The movement range prediction unit predicts the movement range from the sensing time when the real object is sensed by an external sensor to the current time.
   The detection range setting unit sets the detection range at the current time corresponding to the movement range, and sets the detection range.
   The information processing device according to claim 4, wherein the object detection unit acquires the position of the real object at the current time in the three-dimensional coordinate system.
7. The moving range prediction unit predicts the moving range by estimating the predicted position of the real object at the current time from the moving speed and the moving direction of the real object based on the object position information. The information processing device described in.
8. The detection range setting unit projects the position of the real object at the sensing time in the three-dimensional coordinate system and the predicted position of the real object at the current time onto the image coordinate system of the captured image. The information processing apparatus according to claim 7, wherein the detection range at the current time is set.
9. The information processing device according to claim 3, wherein the movement range prediction unit predicts the movement range by using the context of the real object.
10. The information processing apparatus according to claim 9, wherein the context includes at least one of a maximum speed, a plane, and a direction in which the real object can move.
11. The information processing device according to claim 3, wherein the movement range prediction unit predicts the movement range for each real object based on the real object identifier unique to the real object when a plurality of the real objects exist. ..
12. The information processing device according to claim 1, wherein the object position information is information based on a sensing result by an external sensor acquired by wireless communication.
13. The sensor is a first sensor mounted on the real object, a second sensor arranged around the real object, and a third sensor mounted on a moving body moving around the real object. The information processing apparatus according to claim 12, which is configured as at least one of the above.
14. The information processing apparatus according to claim 1, further comprising a virtual object arranging unit for arranging a virtual object corresponding to the position of the real object at the current time in the three-dimensional coordinate system detected in the detection range.
15. The information processing device according to claim 14, wherein the virtual object arranging unit arranges the virtual object at a position that does not overlap with the real object.
16. The information processing device according to claim 14, wherein the virtual object arranging unit arranges additional information about the real object acquired together with the object position information as the virtual object.
17. A drawing unit that renders the virtual object based on the self-position information,
    The information processing apparatus according to claim 14, further comprising a display unit for displaying the rendered virtual object.
18. Further provided with a real object position estimation unit that estimates the position of the real object in the three-dimensional coordinate system based on the self-position information and the distance to the real object.
    The information according to claim 14, wherein the virtual object arranging unit arranges the virtual object based on the estimated position of the real object when the real object position estimation unit can estimate the position of the real object. Processing equipment.
19. Information processing equipment
    Based on the object position information and self-position information of the real object in the three-dimensional coordinate system corresponding to the real space, a detection range smaller than the imaging angle of view at the self-position and corresponding to the position of the real object is set.
    An information processing method for detecting the real object in the detection range.
20. Based on the object position information and self-position information of the real object in the three-dimensional coordinate system corresponding to the real space, a detection range smaller than the imaging angle of view at the self-position and corresponding to the position of the real object is set.
    A computer-readable recording medium on which a program for executing a process for detecting the real object in the detection range is recorded.

Images