WO2022044900A1 - Dispositif de traitement d'informations, procédé de traitement d'informations et support d'enregistrement - Google Patents

Dispositif de traitement d'informations, procédé de traitement d'informations et support d'enregistrement Download PDF

Info

Publication number
WO2022044900A1
WO2022044900A1 PCT/JP2021/030110 JP2021030110W WO2022044900A1 WO 2022044900 A1 WO2022044900 A1 WO 2022044900A1 JP 2021030110 W JP2021030110 W JP 2021030110W WO 2022044900 A1 WO2022044900 A1 WO 2022044900A1
Authority
WO
WIPO (PCT)
Prior art keywords
real object
unit
real
detection range
information processing
Prior art date
Application number
PCT/JP2021/030110
Other languages
English (en)
Japanese (ja)
Inventor
俊逸 小原
誠 ダニエル 徳永
春香 藤澤
一 若林
優生 武田
Original Assignee
ソニーグループ株式会社
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by ソニーグループ株式会社 filed Critical ソニーグループ株式会社
Publication of WO2022044900A1 publication Critical patent/WO2022044900A1/fr

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0481—Interaction techniques based on graphical user interfaces [GUI] based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating 3D models or images for computer graphics
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras

Definitions

  • the present disclosure relates to an information processing device, an information processing method, and a recording medium, and more particularly to an information processing device, an information processing method, and a recording medium that enable more suitable presentation of a virtual object corresponding to a real object.
  • AR Augmented Reality
  • virtual objects of various modes such as text, icons, and animations are superimposed on an object in real space (hereinafter referred to as a real object) and presented to a user.
  • Patent Document 1 as a configuration using AR technology, when the position or posture of the user's viewpoint changes, the position shift of the virtual object that occurs between the time when the virtual object is drawn and the time when the virtual object is presented to the user is described. Information processing devices that suppress are disclosed.
  • This disclosure is made in view of such a situation, and makes it possible to more preferably present a virtual object corresponding to a real object.
  • the information processing apparatus of the present disclosure corresponds to the position of the real object, which is smaller than the imaging angle at the self-position, based on the object position information and the self-position information of the real object in the three-dimensional coordinate system corresponding to the real space. It is an information processing apparatus including a detection range setting unit for setting a detection range to be processed and an object detection unit for detecting the real object in the detection range.
  • the information processing apparatus is smaller than the image pickup angle at the self-position and is smaller than the real object based on the object position information and the self-position information of the real object in the three-dimensional coordinate system corresponding to the real space.
  • This is an information processing method in which a detection range corresponding to a position of a body is set and the real object is detected in the detection range.
  • the recording medium of the present disclosure is smaller than the imaging angle at the self-position and corresponds to the position of the real object based on the object position information and the self-position information of the real object in the three-dimensional coordinate system corresponding to the real space.
  • a computer-readable recording medium on which a program for setting a detection range and executing a process for detecting the real object in the detection range is recorded.
  • the detection range smaller than the imaging angle of view at the self-position and corresponding to the position of the real object. Is set, and the real object is detected in the detection range.
  • Issues of conventional technology For example, consider a case where information is superimposed on a player's head reflected on an AR terminal owned by an spectator in a stadium such as athletics or soccer. In this case, when superimposing a virtual object on a player who moves at a position several hundred meters or more away from the spectator, the AR terminal can identify the player with the camera of the AR terminal, measure the distance to the player with a depth sensor, and so on. Then, it is necessary to grasp the position of the player. However, it is not easy to realize these with a sensor that can be mounted on a mobile terminal that is required to be compact and have low power consumption.
  • the distribution server can use wireless communication to communicate with the player. It becomes possible to deliver the location information to the AR terminal of the audience. However, there is a possibility that the position of the virtual object may be displaced due to the transmission delay between the sensor and the distribution server and between the distribution server and the AR terminal.
  • FIG. 1 is a diagram showing an example of a network configuration to which the technique according to the present disclosure is applied.
  • FIG. 1 shows a real object position estimation system 10, a distribution server 20, and an AR terminal 30.
  • the real object position estimation system 10 and the distribution server 20, the distribution server 20 and the AR terminal 30 each perform wireless communication with each other.
  • the real object position estimation system 10 senses the position of a real object such as a player RO1 or a formula car RO2 in a three-dimensional coordinate system corresponding to a real space such as a stadium such as land or soccer or a circuit such as F1. ..
  • the real object position estimation system 10 aggregates information such as the position of the real object and the sensing time obtained by sensing in the distribution server 20.
  • the distribution server 20 Based on the information aggregated by the real object position estimation system 10, the distribution server 20 has a real object identifier unique to the real object, object position information indicating the position of the real object, sensing time corresponding to the object position information, and the real object. Additional information regarding the above is sequentially delivered to the AR terminal 30.
  • the AR terminal 30 is configured as a mobile terminal such as a smartphone owned by the above-mentioned stadium or circuit spectator (user U), an HMD (Head Mounted Display), AR goggles, or the like.
  • the AR terminal 30 superimposes a virtual object on the player RO1 in the stadium and the formula car RO2 in the circuit.
  • the AR terminal 30 shares a time axis with the same three-dimensional coordinate system as the real object position estimation system 10, and recognizes the position (self-position) of the own terminal in the three-dimensional coordinate system in real time.
  • the AR terminal 30 uses various information distributed from the distribution server 20 and its own position to acquire the position of the real object at the current time, thereby transmitting the transmission between the real object position estimation system 10 and the AR terminal 30. Compensates for delays and realizes the presentation of virtual objects without misalignment.
  • the AR terminal 30 receives the sensing time t-1 and the object position information representing the position P (t-1) of the real object in the three-dimensional coordinate system at the time t-1 from the distribution server 20.
  • the AR terminal 30 sets an imaging angle of view including a plane that passes through the position P (t-1) and faces the front of the camera of the AR terminal 30 based on the self-position in the three-dimensional coordinate system.
  • the captured image CI is imaged.
  • the AR terminal 30 sets a detection range DR (t) capable of detecting a real object at the current time t, which is the current time, in the captured image CI.
  • the AR terminal 30 converts the position p (t) on the captured image CI of the real object into the position P (t) in the three-dimensional coordinate system by detecting the real object in the detection range DR (t). do.
  • the AR terminal 30 corresponds to the position P (t) of the real object at the current time t in the three-dimensional coordinate system and does not overlap the real object (for example, the real object in the direction of gravity).
  • a virtual object VO corresponding to the additional information from the distribution server 20 is superimposed on the position above the body).
  • FIG. 3 is a block diagram showing a configuration example of a real object position estimation system 10 and a distribution server 20 that employ a GPS (Global Positioning System) method as a tracking method for a real object.
  • the real object position estimation system 10 is configured as, for example, a wearable device worn by each athlete as a real object.
  • the real object position estimation system 10 of FIG. 3 includes a GPS sensor 51, a coordinate conversion unit 52, a time measurement unit 53, and a transmission unit 54.
  • the GPS sensor 51 receives GPS position information from GPS satellites and supplies it to the coordinate conversion unit 52.
  • the GPS position information represents a position (latitude, longitude, altitude) in the GPS coordinate system.
  • positioning by BLE (Bluetooth (registered trademark) Low Energy), UWB (Ultra Wide Band), or the like may be used, or these positioning techniques may be used in combination.
  • the coordinate conversion unit 52 converts the GPS position information from the GPS sensor 51 into the position information in the three-dimensional coordinate system, so that the sensor position information indicating the position of the GPS sensor 51 in the three-dimensional coordinate system is transmitted to the transmission unit 54. Supply.
  • a three-dimensional coordinate system it is assumed that one of the axes is aligned in the direction of gravity or the direction of gravity is determined.
  • the transformation of coordinates is performed using a predetermined transformation logic or transformation matrix.
  • the time measuring unit 53 acquires the time (sensing time) when the GPS sensor 51 receives the GPS position information by measuring the time with an accuracy of milliseconds or more, and supplies the time to the transmitting unit 54.
  • the transmission unit 54 transmits the sensor position information from the coordinate conversion unit 52 and the sensing time from the time measurement unit 53 to the distribution server 20 together with the identifier (sensor identifier) unique to the GPS sensor 51.
  • the distribution server 20 of FIG. 3 includes a receiving unit 61, a real object identifier conversion unit 62, an additional information acquisition unit 63, and a transmitting unit 64.
  • the receiving unit 61 receives the sensor identifier, the sensor position information, and the sensing time transmitted from the real object position estimation system 10.
  • the sensor identifier is supplied to the real object identifier conversion unit 62, and the sensor position information and the sensing time are supplied to the transmission unit 64.
  • the real object identifier conversion unit 62 converts the sensor identifier from the reception unit 61 into an identifier (real object identifier) unique to the real object to which the real object position estimation system 10 is mounted, and transmits it to the additional information acquisition unit 63. Supply to unit 64.
  • the conversion from the sensor identifier to the real object identifier is performed, for example, based on a correspondence table showing which player is wearing which sensor device (real object position estimation system 10).
  • the additional information acquisition unit 63 acquires the additional information added to the real object as a virtual object, which is the information corresponding to the real object identifier from the real object identifier conversion unit 62, and supplies it to the transmission unit 64.
  • the additional information acquired by the additional information acquisition unit 63 may be, for example, fixed information about a real object such as a player's name, affiliation, or uniform number, or another system such as a player's ranking or score. It may be real-time changing information about a real object obtained from.
  • the transmission unit 64 transmits the sensor position information and sensing time from the reception unit 61, the real object identifier from the real object identifier conversion unit 62, and the additional information from the additional information acquisition unit 63 to the AR terminal 30. Since the position of the GPS sensor 51 represented by the sensor position information is equal to the position of the real object to which the real object position estimation system 10 is mounted, the sensor position information is AR as the object position information representing the position of the real object. It is transmitted to the terminal 30.
  • FIG. 4 is a flowchart illustrating the operation of the real object position estimation system 10 of FIG.
  • step S11 the GPS sensor 51 receives GPS position information from GPS satellites.
  • step S12 the time measuring unit 53 acquires the time when the GPS sensor 51 receives the GPS position information as the sensing time.
  • step S13 the coordinate conversion unit 52 converts the GPS position information received by the GPS sensor 51 into position information (sensor position information) in the three-dimensional coordinate system.
  • step S14 the transmission unit 54 transmits the sensor identifier of the GPS sensor 51, the sensor position information, and the sensing time to the distribution server 20.
  • FIG. 5 is a flowchart illustrating the operation of the distribution server 20 of FIG.
  • step S21 the receiving unit 61 receives the sensor identifier, the sensor position information, and the sensing time transmitted from the real object position estimation system 10.
  • step S22 the real object identifier conversion unit 62 converts the sensor identifier received by the reception unit 61 into a real object identifier.
  • step S23 the additional information acquisition unit 63 acquires additional information corresponding to the converted real object identifier.
  • step S24 the transmission unit 64 transmits the real object identifier, the object position information (sensor position information), the sensing time, and the additional information to the AR terminal 30.
  • the position of the real object is acquired based on the GPS position information.
  • FIG. 6 is a block diagram showing a configuration example of a real object position estimation system 10 and a distribution server 20 that employ an inside-out method as a tracking method for a real object.
  • the position of the object is measured by using the sensor mounted on the object itself to be measured. Therefore, in the example of FIG. 6, the real object position estimation system 10 is configured as, for example, a wearable device worn by each athlete as a real object.
  • the real object position estimation system 10 of FIG. 6 includes a sensor unit 71, a self-position estimation unit 72, a time measurement unit 53, and a transmission unit 54. Since the time measuring unit 53 and the transmitting unit 54 have the same configuration as shown in FIG. 3, the description thereof will be omitted.
  • the sensor unit 71 is composed of a stereo camera, a depth sensor, and the like, senses the environment around the real object, and supplies the sensing result to the self-position estimation unit 72.
  • the self-position estimation unit 72 estimates the position of the sensor unit 71 in the three-dimensional coordinate system based on the sensing result from the sensor unit 71, and thereby transmits the sensor position information indicating the position of the sensor unit 71 to the transmission unit 54. Supply.
  • an IMU Inertial Measurement Unit
  • the position of the sensor unit 71 in the three-dimensional coordinate system can be estimated based on the sensing result by the sensor unit 71 and the angular velocity and acceleration detected by the IMU. You may do it.
  • FIG. 7 is a flowchart illustrating the operation of the real object position estimation system 10 of FIG.
  • step S31 the sensor unit 71 senses the environment around the real object.
  • step S32 the time measuring unit 53 acquires the time sensed by the sensor unit 71 as the sensing time.
  • step S33 the self-position estimation unit 72 estimates the position of the sensor unit 71 in the three-dimensional coordinate system based on the sensing result of the sensing by the sensor unit 71.
  • step S34 the transmission unit 54 transmits the sensor identifier of the sensor unit 71, the sensor position information indicating the estimated position of the sensor unit 71, and the sensing time to the distribution server 20.
  • the position of the real object is estimated by the real object position estimation system 10 mounted on the real object.
  • FIG. 8 is a block diagram showing a configuration example of a real object position estimation system 10 and a distribution server 20 that employ an inside-out method as a tracking method for a real object. Therefore, even in the example of FIG. 8, the real object position estimation system 10 is configured as, for example, a wearable device worn by each athlete as a real object.
  • the self-position estimation unit 72 is provided in the distribution server 20 instead of the real object position estimation system 10, and the real object position estimation system 10 of FIG. 6 is provided. And different from the distribution server 20.
  • the sensor unit 71 supplies the sensing result to the transmitting unit 54, and the transmitting unit 54 transmits the sensing result as it is to the distribution server 20 instead of the sensor position information. do.
  • the self-position estimation unit 72 estimates the position of the sensor unit 71 in the three-dimensional coordinate system based on the sensing result from the real object position estimation system 10, so that the sensor unit 71 The sensor position information indicating the position is supplied to the transmission unit 54.
  • FIG. 9 is a flowchart illustrating the operation of the real object position estimation system 10 of FIG.
  • step S41 the sensor unit 71 senses the environment around the real object.
  • step S42 the time measuring unit 53 acquires the time sensed by the sensor unit 71 as the sensing time.
  • step S43 the transmission unit 54 transmits the sensor identifier of the sensor unit 71, the sensing result, and the sensing time to the distribution server 20.
  • FIG. 10 is a flowchart illustrating the operation of the distribution server 20 of FIG.
  • step S51 the receiving unit 61 receives the sensor identifier, the sensing result, and the sensing time transmitted from the real object position estimation system 10.
  • step S52 the real object identifier conversion unit 62 converts the sensor identifier received by the reception unit 61 into a real object identifier.
  • step S53 the self-position estimation unit 72 is equipped with the position of the sensor unit 71 in the three-dimensional coordinate system, that is, the real object position estimation system 10 based on the sensing result from the real object position estimation system 10. Estimate the position of the body.
  • step S54 the additional information acquisition unit 63 acquires additional information corresponding to the converted real object identifier.
  • step S55 the transmission unit 64 transmits the real object identifier, the object position information, the sensing time, and the additional information to the AR terminal 30.
  • the position of the real object is estimated by the distribution server 20 instead of the real object position estimation system 10 mounted on the real object.
  • FIG. 11 is a block diagram showing a configuration example of a real object position estimation system 10 and a distribution server 20 that employ an outside-in method as a tracking method for a real object.
  • the position of the object is measured using a sensor installed outside.
  • a method of attaching a marker to an object and observing it with an external camera can be mentioned. Therefore, in the example of FIG. 11, the real object position estimation system 10 is arranged around the real object, and is configured as, for example, a plurality of high-precision sensor devices installed so as to surround the stadium.
  • the real object position estimation system 10 of FIG. 11 includes a sensor unit 71, a real object position estimation unit 81, a real object identification unit 82, a time measurement unit 53, and a transmission unit 83. Since the sensor unit 71 has the same configuration as shown in FIG. 6 and the time measuring unit 53 has the same configuration as shown in FIG. 3, the description thereof will be omitted.
  • the real object position estimation unit 81 estimates the position of the real object in the three-dimensional coordinate system based on the sensing result from the sensor unit 71, and supplies the object position information indicating the position of the real object to the transmission unit 83. do.
  • the position and orientation of the sensor unit 71 for example, the depth sensor
  • the position of the real object can be estimated based on the depth information from the depth sensor.
  • the sensor unit 71 is composed of a camera
  • the position of the real object may be estimated by image recognition for the captured image captured by the camera.
  • the object position information may represent the position of one point such as the center of gravity of the real object or the center of the bounding box that recognizes the real object.
  • the real object identification unit 82 identifies the real object based on the sensing result from the sensor unit 71, and supplies the real object identifier unique to the identified real object to the transmission unit 83.
  • each player is identified by recognizing the player's face and uniform number in the captured image as the sensing result. Athletes as real objects may be equipped with markers and infrared lamps necessary for identification.
  • the transmission unit 83 transmits the object position information from the real object position estimation unit 81, the real object identifier from the real object identification unit 82, and the sensing time from the time measurement unit 53 to the distribution server 20.
  • the distribution server 20 of FIG. 11 includes a receiving unit 61, an additional information acquisition unit 63, and a transmitting unit 64. Since the receiving unit 61, the additional information acquisition unit 63, and the transmitting unit 64 have the same configuration as shown in FIG. 3, the description thereof will be omitted.
  • FIG. 12 is a flowchart illustrating the operation of the real object position estimation system 10 of FIG.
  • step S61 the sensor unit 71 senses the environment in which the real object exists.
  • step S62 the time measuring unit 53 acquires the time sensed by the sensor unit 71 as the sensing time.
  • step S63 the real object position estimation unit 81 estimates the position of the real object in the three-dimensional coordinate system from the known position of the sensor unit 71 (sensor position) and the sensing result from the sensor unit 71.
  • step S64 the real object identification unit 82 identifies the real object from the sensing result from the sensor unit 71.
  • step S65 the transmission unit 83 transmits the real object identifier, the object position information, and the sensing time to the distribution server 20.
  • FIG. 13 is a flowchart illustrating the operation of the distribution server 20 of FIG.
  • step S71 the receiving unit 61 receives the real object identifier, the object position information, and the sensing time transmitted from the real object position estimation system 10.
  • step S72 the additional information acquisition unit 63 acquires additional information corresponding to the real object identifier received by the reception unit 61.
  • step S73 the transmission unit 64 transmits the real object identifier, the object position information, the sensing time, and the additional information to the AR terminal 30.
  • the position of the real object is acquired by the plurality of real object position estimation systems 10 installed outside the real object.
  • FIG. 14 is a block diagram showing a configuration example of a real object position estimation system 10 and a distribution server 20 that employ an outside-in method as a tracking method for a real object. Therefore, even in the example of FIG. 14, the real object position estimation system 10 is configured as, for example, a plurality of high-precision sensor devices installed so as to surround the stadium.
  • the real object position estimation unit 81 and the real object identification unit 82 are provided in the distribution server 20 instead of the real object position estimation system 10. It is different from the real object position estimation system 10 and the distribution server 20 of the above.
  • the distribution server 20 of FIG. 14 is further provided with a sensor position acquisition unit 91.
  • the sensor unit 71 supplies the sensing result to the transmission unit 83, and the transmission unit 83 replaces the real object identifier and the object position information with the sensor identifier and the sensing result. Is transmitted to the distribution server 20 as it is.
  • the sensor position acquisition unit 91 acquires the position / orientation of the sensor unit 71 in the three-dimensional coordinate system based on the sensor identifier from the actual object position estimation system 10, and estimates the actual object position. It is supplied to the unit 81.
  • the acquisition of the position / posture of the sensor unit 71 is performed, for example, based on a correspondence table showing the correspondence between the position / posture of each sensor unit 71 measured in advance and the sensor identifier.
  • the real object position estimation unit 81 is a real object in the three-dimensional coordinate system based on the sensing result from the real object position estimation system 10 and the position / orientation of the sensor unit 71 in the three-dimensional coordinate system from the sensor position acquisition unit 91.
  • the position of the object is estimated, and the object position information is supplied to the transmission unit 64.
  • the real object identification unit 82 identifies the real object based on the sensing result from the real object position estimation system 10, and assigns the real object identifier unique to the identified real object to the additional information acquisition unit 63 and the transmission unit 64. Supply.
  • FIG. 15 is a flowchart illustrating the operation of the distribution server 20 of FIG.
  • step S81 the receiving unit 61 receives the sensor identifier, the sensing result, and the sensing time transmitted from the real object position estimation system 10.
  • step S82 the sensor position acquisition unit 91 acquires the position (sensor position) of the sensor unit 71 in the three-dimensional coordinate system based on the sensor identifier received by the reception unit 61.
  • step S83 the real object position estimation unit 81 estimates the position of the real object in the three-dimensional coordinate system from the sensor position acquired by the sensor position acquisition unit 91 and the sensing result received by the reception unit 61.
  • step S84 the real object identification unit 82 identifies the real object from the sensing result received by the reception unit 61.
  • step S85 the additional information acquisition unit 63 acquires additional information corresponding to the real object identifier unique to the identified real object.
  • step S86 the transmission unit 64 transmits the real object identifier, the object position information, the sensing time, and the additional information to the AR terminal 30.
  • the real object is identified and the position of the real object is estimated by the distribution server 20 instead of the real object position estimation system 10 mounted on the real object.
  • FIG. 16 is a block diagram showing a configuration example of a real object position estimation system 10 and a distribution server 20 that employ an outside-in method as a tracking method for a real object.
  • the real object position estimation system 10 is configured as a moving body (for example, a drone flying over the stadium) that moves around the real object.
  • the real object position estimation system 10 of FIG. 16 is different from the real object position estimation system 10 of FIG. 11 in that a self-position estimation unit 72 and a control unit 101 are further provided in the distribution server 20.
  • the distribution server 20 of FIG. 16 is configured in the same manner as the distribution server 20 of FIG.
  • the self-position estimation unit 72 estimates the position of the sensor unit 71 (real object position estimation system 10) in the three-dimensional coordinate system based on the sensing result from the sensor unit 71. Then, it is supplied to the real object position estimation unit 81 and the control unit 101.
  • the real object position estimation unit 81 estimates the position of the real object in the three-dimensional coordinate system based on the sensing result from the sensor unit 71 and the position of the sensor unit 71 in the three-dimensional coordinate system from the self-position estimation unit 72. , The object position information is supplied to the transmission unit 83.
  • the control unit 101 determines that the real object position estimation system 10 as a drone is localized or moved to a predetermined position based on the position of the sensor unit 71 in the three-dimensional coordinate system from the self-position estimation unit 72. Control actuators not shown.
  • FIG. 17 is a flowchart illustrating the operation of the real object position estimation system 10 of FIG.
  • step S91 the sensor unit 71 senses the environment in which the real object exists, in which the real object position estimation system 10 as a drone is flying.
  • step S92 the time measuring unit 53 acquires the time sensed by the sensor unit 71 as the sensing time.
  • step S93 the self-position estimation unit 72 estimates the position (sensor position) of the sensor unit 71 (real object position estimation system 10) in the three-dimensional coordinate system based on the sensing result of the sensing by the sensor unit 71.
  • step S94 the control unit 101 controls an actuator for flying the real object position estimation system 10 based on the sensor position estimated by the self-position estimation unit 72.
  • step S95 the real object position estimation unit 81 estimates the position of the real object in the three-dimensional coordinate system from the sensor position estimated by the self-position estimation unit 72 and the sensing result from the sensor unit 71.
  • step S96 the real object identification unit 82 identifies the real object from the sensing result from the sensor unit 71.
  • step S97 the transmission unit 83 transmits the real object identifier, the object position information, and the sensing time to the distribution server 20.
  • the position of the real object is acquired by the real object position estimation system 10 flying around the real object.
  • FIG. 18 is a block diagram showing a configuration example of the AR terminal 30 according to the first embodiment.
  • the AR terminal 30 acquires the position of the real object at the current time based only on the information distributed from the distribution server 20, and arranges the corresponding virtual object.
  • the AR terminal 30 of FIG. 18 includes a receiving unit 111, a time measuring unit 112, a sensor unit 113, a self-position estimation unit 114, a moving range prediction unit 115, a detection range setting unit 116, an object detection unit 117, and a virtual object arrangement unit 118. It includes a drawing unit 119 and a display unit 120.
  • the receiving unit 111 receives the real object identifier, the object position information, the sensing time, and the additional information distributed from the distribution server 20.
  • the object position information representing the position of the real object is not the information acquired by the AR terminal 30, but the information based on the sensing result by the external sensor acquired by wireless communication.
  • the object position information and the sensing time are associated with the real object identifier and supplied to the movement range prediction unit 115, and the additional information is associated with the real object identifier and supplied to the virtual object arrangement unit 118.
  • the time measurement unit 112 acquires the current time (current time) by measuring the time with an accuracy of milliseconds or more, and supplies it to the movement range prediction unit 115.
  • the sensor unit 113 is composed of a stereo camera, a depth sensor, or the like, senses the environment around the AR terminal 30, and supplies the sensing result to the self-position estimation unit 114 and the detection range setting unit 116.
  • the self-position estimation unit 114 estimates the position of the sensor unit 113 (AR terminal 30) in the three-dimensional coordinate system based on the sensing result from the sensor unit 113, and thereby obtains self-position information indicating the position of the AR terminal 30. , Supply to the detection range setting unit 116 and the drawing unit 119.
  • an IMU may be provided so that the position of the sensor unit 113 in the three-dimensional coordinate system can be estimated based on the sensing result by the sensor unit 113 and the angular velocity and acceleration detected by the IMU.
  • the movement range prediction unit 115 predicts the movement range in which the real object can move in the three-dimensional coordinate system based on the object position information from the reception unit 111. Specifically, the movement range prediction unit 115 predicts the movement range of the actual object from the sensing time from the reception unit 111 corresponding to the object position information to the current time from the time measurement unit 112. When there are a plurality of real objects in the three-dimensional coordinate system, the movement range is predicted for each real object based on the real object identifier unique to each real object.
  • the moving range of the real object is predicted by estimating the predicted position of the real object at the current time from the moving speed and moving direction of the real object based on the object position information.
  • the velocity vector (traveling direction and velocity) of the real object is estimated based on the difference of the object position information for the last two times received for a certain real object.
  • the predicted position of the real object at the current time is estimated using the estimated velocity vector with the travel time between the received sensing time and the current time as the travel time.
  • the movement range is between the predicted positions estimated from the positions represented by the latest object position information.
  • the movement range of the real object may be predicted using the context of the real object.
  • the context here is the maximum speed at which the real object can move, or the plane or direction in which the real object can move.
  • the maximum speed is, for example, 12.5 m / s based on the world record of land (sprinting).
  • the maximum speed is, for example, 378 km / h, which is the world record of an F1 car.
  • the real object can move on a plane perpendicular to the direction of gravity. Further, in a track and field stadium or a circuit such as F1, the traveling direction of the real object is uniquely determined depending on the position of the real object on the course.
  • the movement range predicted as described above may be provided with a margin such as 1.2 times, or may include a positioning error or a distance measurement error of the GPS sensor 51 or the sensor unit 71. ..
  • the movement range is predicted including a positioning error of about several meters.
  • the detection range setting unit 116 is smaller than the imaging angle of view at the self-position and corresponds to the position of the real object based on the object position information of the real object and the self-position information from the self-position estimation unit 114. To set.
  • the detection range is set as a range in which a real object can be detected in the captured image captured at the imaging angle of view at the self-position.
  • the detection range setting unit 116 uses the self-position information from the self-position estimation unit 114 to capture an image angle of view including the movement range of the real object in the three-dimensional coordinate system predicted by the movement range prediction unit 115. To set.
  • the detection range setting unit 116 sets the movement range predicted by the movement range prediction unit 115 based on the object position information in the image captured by the sensor unit 113 at the set angle of view (angle of view including the real object).
  • the range including the corresponding area is set as the detection range.
  • the detection range setting unit 116 is based on the self-position in the captured image CI captured at the imaging angle of view including the moving range from time t-1 to time t.
  • the detection range DR (t) capable of detecting a real object at the current time t is set.
  • a bounding box BB is set on the three-dimensional coordinate system as shown in FIG. 20 for the shapes of people at time t-1 and time t.
  • the bounding box BB is set so that the height of the person is parallel to the direction of gravity and the aggregated one point is the center of gravity.
  • the height of the person here may be a world record such as 2.5 m, or may be included in the additional information from the distribution server 20.
  • the eight vertices of the bounding box BB for the real object at time t-1 and the eight vertices of the bounding box BB for the real object at time t on the world coordinate system. Is projected onto the uv coordinate system.
  • the point P on the world coordinate system three-dimensional coordinate system
  • Xw-Yw-Zw coordinate system is an image of the captured image CI via the camera coordinate system (Xc-Yc-Zc coordinate system). It is projected onto the point p on the coordinate system (uv coordinate system).
  • a rectangular area surrounded by the minimum value of u, the maximum value of u, the minimum value of v, and the maximum value of v in the uv coordinate system is set in the detection range.
  • the detection range described here is an example, and its shape is not limited to a rectangle such as a circle.
  • the object detection unit 117 detects a real object within the detection range set in the captured image, and converts the position of the detected real object on the captured image into a position in the three-dimensional coordinate system.
  • the object detection unit 117 acquires the center of gravity of the rectangular detection frame in which the real object is detected in the detection range as the position on the captured image of the real object. If the real object is limited to humans, only humans will be detected. Further, when a plurality of people are detected, the position closest to the position on the captured image corresponding to the predicted position at the current time is adopted. Semantic segmentation, which estimates the subject based on the attributes of each pixel of the captured image, may be used to detect the real object.
  • the object detection unit 117 converts the position on the captured image (position on the uv coordinate system) of the detected real object to the position on the three-dimensional coordinate system.
  • the position on the three-dimensional coordinate system corresponding to the Xc—Yc coordinates excluding the depth direction in the camera coordinate system is obtained by the back projection in the above-mentioned camera model.
  • the predicted position at the current time t estimated by the movement range prediction unit 115 is applied to the position on the three-dimensional coordinate system corresponding to the position in the depth direction in the camera coordinate system. This is because the position in the depth direction is not accurate, but the difference from the actual position is considered to be small, and human beings are insensitive to the difference in the position in the depth direction. This is because it does not give a sense of discomfort.
  • the object detection unit 117 acquires the position of the real object at the current time in the three-dimensional coordinate system.
  • the virtual object arranging unit 118 virtually generates additional information from the receiving unit 111 on the three-dimensional coordinate system corresponding to the position of the real object at the current time in the three-dimensional coordinate system acquired by the object detection unit 117. Place it as an object.
  • the virtual object arranging unit 118 arranges the virtual object at a position that does not overlap with the real object, for example, at a position several tens of centimeters above the real object in the direction of gravity.
  • the drawing unit 119 renders a virtual object arranged on the three-dimensional coordinate system by the virtual object arrangement unit 118 based on the self-position information from the self-position estimation unit 114.
  • the display unit 120 is composed of a display or the like, and displays a virtual object rendered by the drawing unit 119.
  • step S111 the sensor unit 113 senses the environment around the AR terminal 30 (for example, the stadium or circuit in which the user U is watching).
  • step S112 the time measuring unit 112 acquires the time sensed by the sensor unit 113 as the current time.
  • step S113 the self-position estimation unit 114 estimates the self-position (position of the AR terminal 30) in the three-dimensional coordinate system based on the sensing result of the sensing by the sensor unit 113.
  • step S114 the AR terminal 30 executes a three-dimensional position acquisition process for acquiring the position of the real object at the current time in the three-dimensional coordinate system.
  • the details of the three-dimensional position acquisition process will be described later.
  • step S115 the virtual object arranging unit 118 arranges the virtual object on the three-dimensional coordinate system corresponding to the position of the real object at the current time in the three-dimensional coordinate system acquired by the three-dimensional position acquisition process. ..
  • step S116 the drawing unit 119 renders a virtual object arranged on the three-dimensional coordinate system by the virtual object arrangement unit 118 based on the self-position information representing the self-position estimated by the self-position estimation unit 114.
  • step S117 the display unit 120 displays the virtual object rendered by the drawing unit 119.
  • FIG. 23 is a flowchart illustrating details of the three-dimensional position acquisition process executed in step S114 of FIG. 22.
  • step S121 the receiving unit 111 receives the real object identifier, the object position information, the sensing time, and the additional information distributed from the distribution server 20.
  • step S122 the movement range prediction unit 115 predicts the movement range of the real object in the three-dimensional coordinate system based on the corresponding object position information for all the received real object identifiers.
  • step S123 the detection range setting unit 116 is predicted for all real object identifiers in the captured image captured at the self-position using the self-position information representing the self-position estimated by the self-position estimation unit 114. Set the detection range corresponding to the movement range.
  • step S124 the object detection unit 117 detects the corresponding real object in each of the detection ranges set for all the real object identifiers.
  • step S125 the object detection unit 117 converts the positions of all the detected real objects on the captured image into the positions in the three-dimensional coordinate system. As a result, the position of the real object at the current time in the three-dimensional coordinate system is acquired.
  • the detection range smaller than the imaging angle of view and corresponding to the position of the real object is set, so that the real object can be detected with less processing.
  • the position of the real object to be detected the position of the real object at the current time detected on the captured image is adopted instead of the position distributed from the distribution server 20, so that the position of the real object is estimated from the real object position estimation system 10. It is possible to compensate for the transmission delay up to the AR terminal 30. As a result, it is possible to eliminate the positional deviation of the virtual object while reducing the processing load of the AR terminal 30, and it is possible to more preferably present the virtual object corresponding to the real object.
  • FIG. 24 is a block diagram showing a configuration example of the AR terminal 30 according to the second embodiment.
  • the AR terminal 30 acquires the position of the real object at the current time by using the information distributed from the distribution server 20 and the distance measurement result from the own terminal to the real object together. Place the corresponding virtual object.
  • the AR terminal 30 of FIG. 24 includes a real object position estimation unit 131, a real object identification unit 132, and a real object position selection unit 133, in addition to the reception unit 111 to the display unit 120 of the AR terminal 30 of FIG.
  • the real object position estimation unit 131 estimates the position of the real object in the three-dimensional coordinate system based on the sensing result from the sensor unit 113 and the self-position information from the self-position estimation unit 114, and the estimation result is the actual object. It is supplied to the body position selection unit 133.
  • the sensing result from the sensor unit 113 is, for example, the distance to the actual object measured by the depth sensor.
  • the real object identification unit 132 identifies the real object based on the sensing result from the sensor unit 113, and supplies the real object identifier unique to the identified real object to the real object position selection unit 133.
  • the real object position selection unit 133 selects the estimation result as the position of the real object, and the real object from the real object identification unit 132. It is supplied to the virtual object arrangement unit 118 in association with the identifier.
  • the position of the real object cannot be estimated by the real object position estimation unit 131, the position of the real object acquired by the object detection unit 117 is selected as the position of the real object and supplied to the virtual object arrangement unit 118. ..
  • the operation of the AR terminal 30 of FIG. 24 is basically the same as the operation of the AR terminal 30 of FIG. 18 described with reference to the flowchart of FIG. 22 except for step S114.
  • FIG. 25 is a flowchart illustrating details of the three-dimensional position acquisition process executed in step S114 of FIG. 22 by the AR terminal 30 of FIG. 24.
  • step S131 the real object identification unit 132 identifies the real object from the sensing result from the sensor unit 113.
  • step S132 the real object position estimation unit 131 is the position of the real object in the three-dimensional coordinate system from the self-position (position of the AR terminal 30) estimated by the self-position estimation unit 114 and the sensing result from the sensor unit 113. To estimate.
  • step S133 the AR terminal 30 of FIG. 24 acquires the positions of all real objects at the current time in the three-dimensional coordinate system by executing the process described with reference to the flowchart of FIG. 23.
  • step S134 the real object position selection unit 133 determines whether or not the position of the real object can be estimated by the real object position estimation unit 131 for all the real object identifiers from the real object identification unit 132. If it is determined that the position of the real object can be estimated, the process proceeds to step S135.
  • step S135 the real object position selection unit 133 replaces the position of the real object acquired in step S133 with the position of the real object estimated in step S132 for all the real object identifiers.
  • the position of the real object acquired in step S133 may be replaced with the position of the real object estimated in step S132 only for the real object whose position is determined to be estimable.
  • step S135 is skipped and the positions of the real objects acquired in step S133 are adopted as the positions of all the real objects.
  • the position of the real object acquired in step S133 may be adopted only for the real object whose position is not determined to be estimable.
  • the position of the real object when the position of the real object can be estimated within the range measuring range such as the depth sensor provided in the AR terminal 30, such as when the real object exists at a close distance from the AR terminal 30, the estimated real object is obtained.
  • the position of the body is adopted. In this case, it is not necessary to compensate for the transmission delay between the real object position estimation system 10 and the AR terminal 30, and it is possible to more preferably present a virtual object corresponding to the real object.
  • the technology according to the present disclosure may be applied to a configuration in which a virtual object is superimposed on a player in a stadium such as athletics or soccer or a formula car in a circuit such as F1.
  • the technology according to the present disclosure may be applied to a configuration in which a virtual object is superimposed on a vehicle arranged by a user in, for example, a vehicle allocation application in which a taxi, a hire, or the like can be arranged.
  • the technique according to the present disclosure may be applied to a configuration in which a virtual object is superimposed on a player around the user in an AR shooting game that can be played from a first-person viewpoint such as FPS (First Person Shooter).
  • FPS First Person Shooter
  • the series of processes described above can be executed by hardware or software.
  • the programs constituting the software are installed on the computer.
  • the computer includes a computer embedded in dedicated hardware and, for example, a general-purpose personal computer capable of executing various functions by installing various programs.
  • FIG. 26 is a block diagram showing a configuration example of computer hardware that executes the above-mentioned series of processes programmatically.
  • the CPU 501 the CPU 501, the ROM (ReadOnlyMemory) 502, and the RAM (RandomAccessMemory) 503 are connected to each other by the bus 504.
  • An input / output interface 505 is further connected to the bus 504.
  • An input unit 506, an output unit 507, a storage unit 508, a communication unit 509, and a drive 510 are connected to the input / output interface 505.
  • the input unit 506 includes a keyboard, a mouse, a microphone, and the like.
  • the output unit 507 includes a display, a speaker, and the like.
  • the storage unit 508 includes a hard disk, a non-volatile memory, and the like.
  • the communication unit 509 includes a network interface and the like.
  • the drive 510 drives a removable medium 511 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.
  • the CPU 501 loads the program stored in the storage unit 508 into the RAM 503 via the input / output interface 505 and the bus 504 and executes the above-mentioned series. Is processed.
  • the program executed by the computer (CPU 501) can be recorded and provided on the removable media 511 as a package media or the like, for example.
  • the program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.
  • the program can be installed in the storage unit 508 via the input / output interface 505 by mounting the removable media 511 in the drive 510. Further, the program can be received by the communication unit 509 and installed in the storage unit 508 via a wired or wireless transmission medium. In addition, the program can be installed in the ROM 502 or the storage unit 508 in advance.
  • the program executed by the computer may be a program in which processing is performed in chronological order according to the order described in the present specification, in parallel, or at a necessary timing such as when a call is made. It may be a program in which processing is performed.
  • the present disclosure may have the following structure.
  • a detection range that sets a detection range that is smaller than the imaging angle of view at the self-position and corresponds to the position of the real object, based on the object position information and the self-position information of the real object in the three-dimensional coordinate system corresponding to the real space.
  • Setting part and An information processing device including an object detection unit that detects the real object in the detection range.
  • the detection range setting unit sets the detection range capable of detecting the real object in the captured image captured at the imaging angle of view.
  • the information processing apparatus according to (1), wherein the object detection unit converts a position of the real object detected in the detection range on the captured image into a position in the three-dimensional coordinate system.
  • a movement range prediction unit for predicting a movement range in which the real object can move in the three-dimensional coordinate system based on the object position information.
  • the detection range setting unit sets the detection range including a region corresponding to the movement range in the captured image.
  • the detection range setting unit sets the minimum region in which the real object is predicted to be captured in the captured image based on the movement range in the detection range.
  • the movement range prediction unit predicts the movement range from the sensing time when the real object is sensed by an external sensor to the current time.
  • the detection range setting unit sets the detection range at the current time corresponding to the movement range, and sets the detection range.
  • the information processing device according to (4) or (5), wherein the object detection unit acquires the position of the real object at the current time in the three-dimensional coordinate system.
  • the movement range prediction unit predicts the movement range by estimating the predicted position of the real object at the current time from the movement speed and the movement direction of the real object based on the object position information (6).
  • the detection range setting unit projects the position of the real object at the sensing time in the three-dimensional coordinate system and the predicted position of the real object at the current time onto the image coordinate system of the captured image.
  • the information processing apparatus according to (7) which sets the detection range at the current time.
  • the context includes at least one of the maximum velocity, plane, and direction in which the real object can move.
  • the movement range prediction unit predicts the movement range for each real object based on the real object identifier unique to the real object, whichever is (3) to (10).
  • the information processing apparatus according to any one of (1) to (11), wherein the object position information is information acquired by wireless communication and based on a sensing result by an external sensor.
  • the sensor is a first sensor mounted on the real object, a second sensor arranged around the real object, and a third sensor mounted on a moving body moving around the real object.
  • the information processing apparatus which is configured as at least one of.
  • Any of (1) to (13) further including a virtual object arranging unit for arranging a virtual object corresponding to the position of the real object at the current time in the three-dimensional coordinate system detected in the detection range.
  • the virtual object placement unit arranges the virtual object based on the estimated position of the real object (14) to (17).
  • the information processing device described in any of the above.
  • (19) Information processing equipment Based on the object position information and self-position information of the real object in the three-dimensional coordinate system corresponding to the real space, a detection range smaller than the imaging angle of view at the self-position and corresponding to the position of the real object is set. An information processing method for detecting the real object in the detection range.
  • a detection range smaller than the imaging angle of view at the self-position and corresponding to the position of the real object is set.
  • a computer-readable recording medium on which a program for executing a process for detecting the real object in the detection range is recorded.
  • Real object position estimation system 20 distribution server, 30 AR terminal, 111 receiver, 112 time measurement unit, 113 sensor unit, 114 self-position estimation unit, 115 movement range prediction unit, 116 detection range setting unit, 117 object detection unit , 118 virtual object placement unit, 119 drawing unit, 120 display unit, 131 real object position estimation unit, 132 real object identification unit, 133 real object position selection unit

Abstract

La présente divulgation concerne un dispositif de traitement d'informations, un procédé de traitement d'informations et un support d'enregistrement qui permettent de présenter de manière plus appropriée un objet virtuel correspondant à un objet réel. Sur la base d'informations d'emplacement d'objet d'un objet réel et d'informations d'emplacement automatique dans un système de coordonnées tridimensionnelles correspondant à un espace réel, une unité de définition de plage de détection définit une plage de détection qui est inférieure à un angle de champ d'imagerie à l'emplacement automatique et correspond à l'emplacement de l'objet réel. Une unité de détection d'objet détecte l'objet réel dans la plage de détection. La présente divulgation peut être appliquée, par exemple, à un terminal de RA qui superpose des informations sur l'espace réel.
PCT/JP2021/030110 2020-08-31 2021-08-18 Dispositif de traitement d'informations, procédé de traitement d'informations et support d'enregistrement WO2022044900A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2020-145344 2020-08-31
JP2020145344 2020-08-31

Publications (1)

Publication Number Publication Date
WO2022044900A1 true WO2022044900A1 (fr) 2022-03-03

Family

ID=80354261

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2021/030110 WO2022044900A1 (fr) 2020-08-31 2021-08-18 Dispositif de traitement d'informations, procédé de traitement d'informations et support d'enregistrement

Country Status (1)

Country Link
WO (1) WO2022044900A1 (fr)

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2017131071A1 (fr) * 2016-01-28 2017-08-03 日本電信電話株式会社 Dispositif de construction d'environnement virtuel, dispositif de présentation de vidéo, dispositif d'apprentissage de modèle, dispositif de détermination de profondeur optimale, procédé associé et programme
JP2019036346A (ja) * 2017-08-14 2019-03-07 キヤノン株式会社 画像処理装置、画像処理方法およびプログラム
WO2019219423A1 (fr) * 2018-05-18 2019-11-21 Valeo Comfort And Driving Assistance Environnement partagé pour occupant de véhicule et utilisateur distant

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2017131071A1 (fr) * 2016-01-28 2017-08-03 日本電信電話株式会社 Dispositif de construction d'environnement virtuel, dispositif de présentation de vidéo, dispositif d'apprentissage de modèle, dispositif de détermination de profondeur optimale, procédé associé et programme
JP2019036346A (ja) * 2017-08-14 2019-03-07 キヤノン株式会社 画像処理装置、画像処理方法およびプログラム
WO2019219423A1 (fr) * 2018-05-18 2019-11-21 Valeo Comfort And Driving Assistance Environnement partagé pour occupant de véhicule et utilisateur distant

Similar Documents

Publication Publication Date Title
CN112567201B (zh) 距离测量方法以及设备
US9747697B2 (en) System and method for tracking
US10462406B2 (en) Information processing apparatus and information processing method
US10481679B2 (en) Method and system for optical-inertial tracking of a moving object
CN109658435A (zh) 用于视频捕获和创建的无人机云
US20160299505A1 (en) Autonomous moving device and control method of autonomous moving device
WO2016031105A1 (fr) Dispositif de traitement d'informations, procédé de traitement d'informations et programme
JPWO2018179644A1 (ja) 情報処理装置、情報処理方法及び記録媒体
US11181376B2 (en) Information processing device and information processing method
WO2003092291A1 (fr) Dispositif de detection d'objet, serveur de detection d'objet et procede de detection d'objet
JP2013537618A (ja) オブジェクト追跡及び認識方法及び装置
WO2018113759A1 (fr) Système et procédé de détection basés sur un système de positionnement et l'ar/mr
JP2017129904A (ja) 情報処理装置、情報処理方法、及び記録媒体
WO2019051832A1 (fr) Procédé de commande d'objet mobile, dispositif et système
KR102190743B1 (ko) 로봇과 인터랙션하는 증강현실 서비스 제공 장치 및 방법
WO2021250914A1 (fr) Dispositif de traitement d'informations, dispositif de mouvement, système de traitement d'informations, procédé et programme
US20210157396A1 (en) System and method related to data fusing
US20170359671A1 (en) Positioning arrangement
WO2022044900A1 (fr) Dispositif de traitement d'informations, procédé de traitement d'informations et support d'enregistrement
US20230103650A1 (en) System and method for providing scene information
CN113272864A (zh) 信息处理装置、信息处理方法和程序
CN112788443A (zh) 基于光通信装置的交互方法和系统
US10735902B1 (en) Method and computer program for taking action based on determined movement path of mobile devices
JP2005258792A (ja) 画像生成装置、画像生成方法、および画像生成プログラム
US20230120092A1 (en) Information processing device and information processing method

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21861328

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21861328

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: JP