WO2024057779A1

WO2024057779A1 - Information processing device, program, and information processing system

Info

Publication number: WO2024057779A1
Application number: PCT/JP2023/028915
Authority: WO
Inventors: 優生武田; 一若林; 富士夫荒井; ダニエル誠徳永; 諒介村田; 春香藤澤
Original assignee: ソニーグループ株式会社
Priority date: 2022-09-14
Filing date: 2023-08-08
Publication date: 2024-03-21

Abstract

This information processing device is provided with: an estimation accuracy determination unit that determines the estimation accuracy of the position and orientation of a client as estimated on the basis of a 3D map of a real space and a position and orientation estimation image captured of the real space and transmitted from the client; and a client identification unit that, on the basis of the determination result of the estimation accuracy, identifies another client who is to be requested to transmit a 3D map update image for updating the 3D map.

Description

Information processing equipment, programs and information processing systems

The present technology relates to an information processing device, a program, and an information processing system.

In AR (Augmented Reality), the same 3D object (virtual object) may be displayed at a specific location in real space for multiple clients. To do this, it is necessary to specify the position and orientation of each client using a common coordinate system. As a means for realizing this, there is a method called VPS (Visual Positioning System) that estimates the position and orientation of each terminal from camera images.

VPS uses a 3D map of the real space, and uses methods such as Perspective-n-Point to determine the position of the client that captured the camera image based on the correspondence between the camera image sent from each client and the features of the 3D map. and the pose are estimated in the 3D map coordinate system. The 3D map includes 3D feature points (landmarks) that model objects in real space, feature points and features used for pose estimation, images used for 3D shape estimation (Key Frame: KF), etc. contains data.

The accuracy of estimating the position and orientation of each client depends on the accuracy of the positions of landmarks included in the 3D map. Therefore, when a landmark with low positional accuracy is used, there is a problem that the accuracy of the estimation result of the client's position and orientation becomes low.

Therefore, a method has been proposed in which low-precision landmarks included in the 3D map are detected and updated to high-precision landmarks. For example, Patent Document 1 proposes a method in which when a landscape image acquired by a certain client differs from a landscape image recorded in a database, the database is updated by presenting a command prompting the client to request additional data acquisition. are doing.

JP2021-170341A

However, the method of Patent Document 1 works only when the 3D map can be updated using only sensor data that can be obtained by the client that performs pose estimation. A client located at a point where orientation is to be estimated cannot obtain highly accurate sensor data of an object far from that point, so there is a problem in that a 3D map cannot be updated for the purpose of increasing the accuracy of landmarks.

The present technology was developed in view of these problems, and aims to provide an information processing device, a program, and an information processing system that can update a 3D map by increasing the accuracy of landmarks.

In order to solve the above-mentioned problems, a first technique involves estimating the position and orientation of the client based on a position and orientation estimation image of the real space transmitted from the client and a 3D map of the real space. Information processing comprising: an estimation accuracy determination unit that determines accuracy; and a client identification unit that identifies another client that requests transmission of a 3D map update image for updating a 3D map based on the estimation accuracy determination result. It is a device.

In addition, the second technology determines the estimation accuracy of the client's position and orientation, which is estimated based on a position and orientation estimation image captured from the real space and a 3D map of the real space, and determines the estimation accuracy. This is a program that causes a computer to execute an information processing method for identifying another client that requests transmission of a 3D map update image for updating a 3D map based on the determination result.

Furthermore, the third technique is a client transmitting an image for position and orientation estimation captured in real space, and the estimation accuracy of the position and orientation of the client estimated based on the image for position and orientation estimation and a 3D map of the real space. an information processing device that includes an estimation accuracy determination unit that determines the estimation accuracy; and a client identification unit that identifies another client that requests transmission of a 3D map update image for updating a 3D map based on the estimation accuracy determination result. and another client that transmits a 3D map update image in response to a request from an information processing device to transmit a 3D map update image.

1 is a block diagram showing the configuration of a client 10. FIG. 2 is a block diagram showing the configuration of a server 20. FIG. 1 is a block diagram showing the configuration of an information processing system 1000. FIG. 1 is a block diagram showing the configurations of an information processing system 1000 and an information processing device 100. FIG. FIG. 3 is an explanatory diagram of a 3D map, key frames, and landmarks. FIG. 7 is an explanatory diagram of posture estimation results when landmarks are highly accurate and when landmarks are low accurate. FIG. 3 is a diagram showing the positions of a client 10 and landmarks in a specific example of the present embodiment. FIG. 8A is a flowchart showing processing in the client 10, and FIG. 8B is a flowchart showing processing in the information processing apparatus 100. 9A is a flowchart showing processing in the client 10, and FIG. 9B is a flowchart showing processing in the information processing apparatus 100. It is a diagram showing the positions of landmarks, KFs, and clients 10 in a specific example. FIG. 11A is a flowchart showing processing in the information processing apparatus 100, and FIG. 11B is a flowchart showing processing in the client 10. It is an explanatory diagram of a search range. It is a figure which shows the 1st example of UI for image acquisition. FIG. 13 is a diagram showing a second example of a UI for image acquisition. It is a figure which shows the 3D map update image transmitted from client 10B in a specific example. 2 is a diagram showing the positions of a client 10A and a client 10B specified by the information processing device 100. FIG.

Embodiments of the present technology will be described below with reference to the drawings. Note that the explanation will be given in the following order.
<1. Embodiment>
[1-1. Configuration of client 10 and server 20]
[1-2. Configuration of information processing system 1000 and information processing device 100]
[1-3. About 3D maps, key frames, and landmarks]
[1-4. Processing in information processing system 1000 and information processing device 100]
<2. Modified example>

<1. Embodiment>
[1-1. Configuration of client 10 and server 20]
The client 10 included in the information processing system 1000 will be described with reference to FIG. 1. The client 10 includes at least a control section 11, a storage section 12, a communication section 13, an input section 14, a display section 15, and a camera section 16.

The control unit 11 includes a CPU (Central Processing Unit), a RAM (Random Access Memory), a ROM (Read Only Memory), and the like. The CPU controls the entire client 10 and each part by executing various processes and issuing commands according to programs stored in the ROM.

The storage unit 12 is, for example, a large capacity storage medium such as a hard disk or flash memory. The storage unit 12 stores AR applications and various data used by the client 10.

The communication unit 13 is a communication module that communicates with the server 20, the Internet, etc. Communication methods can be wired or wireless, including cellular communication, Wi-Fi, Bluetooth (registered trademark), NFC (Near Field Communication), Ethernet (registered trademark), and HDMI (registered trademark) (High-Definition Multimedia). interface), USB (Universal Serial Bus), etc.

The input unit 14 allows the user to input information and various instructions to the client 10. When the user makes an input to the input unit 14, a control signal corresponding to the input is generated and supplied to the control unit 11. The control unit 11 then performs various processes corresponding to the control signal. The input unit 14 may be a physical button, a touch panel, or a touch screen integrated with a monitor.

The display unit 15 is a display device such as a liquid crystal display or an organic EL display that displays images, videos, and a UI (User Interface) for the user to use the client 10.

The camera unit 16 is composed of a lens, an image sensor, a video signal processing circuit, etc., and captures a camera image of the real space to be transmitted to the information processing device 100. Note that the camera unit 16 may be provided in the client 10 itself, or may be a single device such as a camera or other device having a camera function. In that case, the device transmits the camera image to the client 10 via wired or wireless communication, and the client 10 transmits the camera image captured by the device to the information processing apparatus 100.

The client 10 is configured as described above. Examples of the client 10 include a smartphone, a tablet terminal, a digital camera, a wearable device, a head-mounted display, and a personal computer. A control program that performs processing according to the present technology may be installed in the client 10 in advance, or may be downloaded, distributed on a storage medium, etc., and installed by the user himself.

Next, the server 20 constituting the information processing system 1000 will be described with reference to FIG. 2. The server 20 is configured to include at least a control unit 21, a storage unit 22, and a communication unit 23. The control unit 21, the storage unit 22, and the communication unit 23 have the same functions as those included in the client 10. The server 20 may be, for example, a cloud server.

[1-2. Configuration of information processing system 1000 and information processing device 100]
Next, the configurations of the information processing system 1000 and the information processing apparatus 100 will be described with reference to FIGS. 3 and 4.

As shown in FIG. 3, the information processing system 1000 includes a plurality of

clients

10, 10, . . . and a server 20 that performs processing as the information processing device 100. A plurality of clients 10 and servers 20 are connected via a network. The client 10 sends a camera image used for position and orientation estimation (referred to as an image for position and orientation estimation) to the information processing apparatus 100 to request position and orientation estimation, and in response to a request from the information processing apparatus 100. There is a device (hereinafter referred to as another client 10) that transmits a camera image used for 3D map update (referred to as 3D map update image) to information processing apparatus 100. Note that although the client 10 functions as an AR device by executing an AR application, a device that transmits a 3D map update image in response to a request from the information processing device 100 may not function as an AR device. There is no particular limitation on the number of clients 10.

FIG. 4 shows the information processing apparatus 100 constituting the information processing system 1000 and a client A (denoted as client 10A) and client B (denoted as client 10B) out of the plurality of clients 10. Note that FIG. 4 shows only processing blocks related to information and signals related to the present technology.

In FIG. 4, data transmission and reception between the client 10A and the information processing device 100 and data transmission and reception between the client 10B and the information processing device 100 are performed by communication functions provided in the client 10A, the client 10B, and the server 20.

The client 10A functions as an AR device by executing an AR application. The client 10A requests the information processing apparatus 100 to estimate the position and orientation of the client 10A by transmitting the image for position and orientation estimation captured by the camera unit 16.

In addition to the configuration shown in FIG. 1, the client 10A includes an application control section 17A and a drawing section 18A.

The application control unit 17A performs processing based on an application for using the information processing system 1000. The application control unit 17A performs processing related to the AR application executed by the client 10A, and acquires the position and orientation estimation result transmitted from the information processing device 100 and outputs it to the drawing unit 18A.

The drawing unit 18A draws a virtual object based on the position and orientation estimation results transmitted from the information processing device 100 to generate an output image, and outputs the output image to the display unit 15A. As a result, the output image is displayed on the display section 15A. The virtual object may be, for example, a character, a figure, an object that indicates information in a map service, a character placed on an image of real space, but is not limited to these, and may be of any kind.

The application control unit 17A and the drawing unit 18A are realized by the client 10A executing an AR application, and the application may be installed in the client 10A in advance, or may be downloaded or distributed on a storage medium, so that the user can You may also install it yourself. Further, the client 10A may be provided with the functions of the application control section 17A and the drawing section 18A in advance.

The client 10B receives a request to send a 3D map update image from the information processing device 100, and transmits the camera image captured by the camera unit 16B to the information processing device 100 as a 3D map update image.

In addition to the configuration shown in FIG. 1, the client 10B includes a coarse-grained position sensor section 19B, an application control section 17B, and a drawing section 18B.

The coarse-grained position sensor unit 19B is a sensor that acquires position information such as GNSS (Global Navigation Satellite System) or GPS (Global Positioning System), and indicates the position of the client 10B with an accuracy of, for example, several tens to several hundred meters. Obtain coarse-grained position information. The coarse-grain position sensor unit 19B periodically acquires coarse-grain position information, and the client 10B transmits the coarse-grain position information to the information processing device 100. Furthermore, the coarse-grained position sensor unit 19B may acquire coarse-grained positional information using mobile line base station information, Wi-Fi access point names, and the like. Note that since the coarse-grained position information is used to specify the approximate position of the client 10B, it does not need to be highly accurate position information. However, the client 10B may transmit highly accurate position information, and the coarse-grained position sensor section 19B does not exclude the acquisition of highly accurate position information. Further, the client 10B may use a geomagnetic sensor or the like to transmit information indicating the direction in which the client 10B is facing to the information processing apparatus 100 together with the coarse-grained position information.

The application control unit 17B performs processing based on an application for using the information processing system 1000. The application control unit 17B controls the drawing of the UI by the drawing unit 18B, the display on the display unit 15B, etc. in order to present to the user a request to send a 3D map update image from the information processing device 100. It also controls the capturing of camera images by the camera unit 16.

The drawing unit 18B draws a UI etc. for presenting a request to the user to send a 3D map update image, and outputs the drawing result to the display unit 15A. The drawing result is displayed on the display section 15B.

Note that the client 10B is equipped with IMU (Inertial Measurement Unit), LiDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging), ToF (Time of Flight) sensors, etc., and the sensing data acquired by these sensors is used as a 3D map. It may also be transmitted to the information processing apparatus 100 as update data or coarse-grained position information. Furthermore, information such as GNSS or mobile line base station information, Wi-Fi access point name, etc. may also be used as 3D map update data.

The client 10B may function as an AR device by executing an AR application, or may not function as an AR device.

The information processing device 100 includes a 3D map DB (Database) 101, a position/orientation estimation section 102, an estimation accuracy determination section 103, a 3D map accuracy determination section 104, a client position DB 105, a client identification section 106, and a 3D map update section 107. In this embodiment, the storage unit 22 of the server 20 functions as the 3D map DB 101 and the client position DB 105, but each DB may be configured with a different storage medium.

The 3D map DB 101 is a database that stores 3D maps that model real space. It is assumed that a plurality of 3D maps exist for each point in real space, and a plurality of 3D maps that have been generated in advance are stored in the 3D map DB 101. 3D maps can be generated by simultaneously estimating the three-dimensional shape and imaging point based on multiple camera images from different viewpoints using technologies such as SfM (Structure from Motion) and VIO (Visual Inertial Odometry).

The position and orientation estimating unit 102 estimates the position and orientation of the client 10A using VPS based on the correspondence between the position and orientation estimation image transmitted from the client 10A and the 3D map acquired from the 3D map DB 101. In VPS, the position and orientation of the client 10A is estimated in the 3D map coordinate system using a method such as Perspective-n-Point based on the correspondence between the image for position and orientation estimation and the feature points and feature amounts of the 3D map.

The position and orientation estimation result by the position and orientation estimation unit 102 is transmitted to the client 10 that made the position and orientation estimation request. In the example of FIG. 4, the position and orientation estimating unit 102 estimates the position and orientation of the client 10A based on the correspondence between the position and orientation estimation image transmitted from the client 10A and the 3D map, and the information processing device 100 estimates the position and orientation of the client 10A. The results are sent to client 10A. The drawing unit 18A of the client 10A uses this position and orientation estimation result to draw a virtual object in AR.

The estimation accuracy determination unit 103 evaluates the accuracy of the position and orientation estimation result by the position and orientation estimation unit 102. If the accuracy of the position and orientation estimation result is low, the 3D map accuracy determination unit 104 determines the accuracy of the 3D map.

The 3D map accuracy determination unit 104 determines the accuracy of the landmarks included in the 3D map that are used to estimate the position and orientation of the client 10A. If the accuracy of the landmark is low, the client specifying unit 106 performs a process of specifying the client 10 requesting transmission of the 3D map update image.

The client location DB 105 is a database that stores coarse-grained location information sent from multiple clients 10 connected to the server 20.

The client specifying unit 106 refers to the coarse-grained position information stored in the client position DB 105 to specify the client 10 that requests transmission of the 3D map update image. In the example of FIG. 4, it is assumed that the client 10B is identified by the client identifying unit 106 and receives the request to send the 3D map update image.

The 3D map update unit 107 uses the 3D map update image sent from the client 10B and the computer resources of the server 20 to update the 3D map used by the position and orientation estimation unit 102 to estimate the position and orientation of the client 10A. . Computer resources refer to the amount of hardware occupied by the server 20, such as the execution time of the server 20, the usage rate of the processor, and the amount of storage 22 occupied.

The information processing system 1000 and the information processing device 100 are configured as described above. In this embodiment, the information processing device 100 operates on the server 20, but the server 20 may have the function of the information processing device 100 in advance, or the server 20, which has the function of a computer, may execute the program. The information processing device 100 and the information processing method may be realized by the following. Further, the control unit 21 may function as the information processing device 100 by executing a program. The program may be installed in advance on the server 20, or may be downloaded, distributed on a storage medium, etc., and installed by a user or the like. Further, the information processing device 100 may be configured as a single device. Further, the information processing device 100 is not limited to the server 20, and may operate on a personal computer, a smartphone, a tablet terminal, or the like.

[1-3. About 3D maps, key frames, and landmarks]
Here, the 3D map and the key frames and landmarks included in the 3D map will be explained.

VPS uses a 3D map of the real space to determine the position using a method such as Perspective-n-Point based on the correspondence between the camera image as an image for position and orientation estimation sent from the client 10 and the features of the 3D map. The position and orientation of the client 10 that captured the orientation estimation image are estimated using the 3D map coordinate system. The 3D map includes three-dimensional feature points (landmarks) that are modeled by estimating objects in real space, feature points and feature amounts used for pose estimation, and images used for three-dimensional shape estimation (Key Frame: KF ), coarse-grained position information, and other data. In the landmark, KF imaging position information used for landmark estimation is recorded. Examples of landmarks include the position of the vertex of an acute angle portion of an object.

A 3D map can be generated by estimating the three-dimensional shape and the imaging position and orientation of the KF based on multiple KFs from different viewpoints using techniques such as SfM and VIO. As shown in Figure 5, these techniques use multiple KFs with different viewpoints (KF1 to KF4 in Figure 5), and use the positional relationship of feature points commonly seen in each KF to create a triangular shape. Landmark positions, KF imaging positions, and orientations are estimated by surveying and overall optimization of the results.

Since the landmark position is estimated using triangulation, the accuracy of the estimated landmark position is determined by the accuracy of the position of the feature point in the KF, the distance between the two KFs used for estimation (baseline length) Depends on. The accuracy of the position of the feature point in the KF is inversely proportional to the distance of the object from which the landmark is detected from the position of the KF. This is because as the distance from the KF imaging point to the object increases, the size of the object projected onto the image (KF) becomes smaller, so the size of the object represented by 1 pixel increases in proportion to the distance. It's for a reason. Therefore, when estimating a landmark on an object far from the KF imaging position using a short baseline length, the accuracy of the estimated landmark position becomes low.

The position and orientation of the client 10A is estimated using the position and orientation estimation image sent by the client 10A and the 3D map acquired from the 3D map DB 101. The estimation accuracy of the position and orientation of the client 10A depends on the positional accuracy of landmarks used for estimation in the 3D map. Therefore, as shown in FIG. 6A, if landmarks with high positional accuracy (indicated by six white dots) are used, the estimation accuracy of the position and orientation of the client 10A becomes high, but as shown in FIG. 6B, the positional accuracy is If a low landmark (indicated by six white dots) is used, the accuracy of estimating the position and orientation of the client 10A will also be low.

[1-4. Processing in information processing system 1000 and information processing device 100]
Next, processing in the information processing system 1000 and the information processing apparatus 100 will be explained. Here, as shown in FIG. 7, it is assumed that there are

clients

10A, 10B, and 10C, which are smartphones, in the real space, and that the landmark _Li is a corner of an object P that exists in the real space. The client 10A requests the information processing device 100 to estimate the position and orientation, and the client 10B and the client 10C may be requested by the information processing device to send a 3D map update image. Suppose there is.

As shown in FIG. 8A, first in step S101, the coarse-grained position sensor unit 19B of the client 10B acquires coarse-grained position information.

Next, in step 102, the client 10B transmits coarse-grained position information to the information processing device 100.

The client 10C also performs the process shown in FIG. 8A in the same way as the client 10B. However, the client 10B and the client 10C do not need to perform the processing in FIG. 8A at the same timing.

Next, as shown in FIG. 8B, in step S201, the information processing device 100 receives coarse-grained position information transmitted from the client 10B.

Next, in step S202, the information processing device 100 stores the coarse-grained position information in the client position DB 105.

The information processing apparatus 100 similarly performs the process shown in FIG. 8B on the coarse-grained position information transmitted from the client 10C.

The clients 10B, 10C, and the information processing apparatus 100 periodically perform the processes from step S101 to step S202 at predetermined time intervals. As a result, the latest coarse-grained position information of the clients 10B and 10C is periodically stored in the client position DB 105.

Although only the clients 10B and 10C are shown as an example, if there are multiple clients 10 connected to the server 20 and using the information processing system 1000, all the clients 10 will process the coarse-grained position information. The coarse-grained position information of the plurality of clients 10 is stored in the client position DB 105. Further, the client 10A may also transmit coarse-grained position information to the information processing device 100. This is because if another client 10 makes a request for position and orientation estimation, the client 10A may also receive a request from the information processing device 100 to send a 3D map update image.

Next, a posture estimation request from the client 10A using the AR application and processing in the information processing device will be explained.

First, as shown in FIG. 9A, in step S301, the camera unit 16A images the real space and acquires an image for position and orientation estimation based on the input operation of the user A using the client 10A.

Next, in step 302, the client 10A transmits the position and orientation estimation image to the information processing device 100.

In step S303, the client 10A enters a waiting state until the position and orientation estimation result is received from the information processing device 100 (No in step S303).

Next, processing in the information processing device 100 will be described with reference to FIG. 9B. In step S401, the information processing apparatus 100 receives the position and orientation estimation image transmitted from the client 10A.

Next, in step S402, the position and orientation estimating unit 102 estimates the position and orientation of the client 10A using the image for position and orientation estimation and the 3D map acquired from the 3D map DB 101.

The position and orientation estimating unit 102 extracts feature points and feature amounts from the received image for position and orientation estimation, and combines the feature points and feature amounts with the feature points and features of each of the plurality of 3D maps stored in the 3D map DB 101. The amounts and the like are compared, and a 3D map that matches or approximates them is set as a 3D map to be used for estimating the position and orientation of the client 10A. In addition, a 3D map covering the range where the client 10A exists is created based on the coarse-grained position information such as GPS data transmitted by the client 10A and the coarse-grained position information included in the 3D map. There is also a method of determining a 3D map to be used for estimation.

Here, as shown in FIG. 10, it is assumed that the client 10A has captured an image for position and orientation estimation necessary for the orientation estimation request at the current point _a1 . Note that in FIG. 10, a method using a frustum called Viewing Frustum is used to express the direction of the camera and the direction of the KF. In the KF, the short side of the triangle represents the imaging surface on which the KF is imaged, and in the client 10, it represents the imaging surface of the camera section 16 (the direction in which the lens is facing). The same applies to FIGS. 12 and 16, which will be described later.

Here, it is assumed that the 3D map includes two KFs, KF _x and KF _y , and a landmark _Li , and the positional relationship between KF _x , KF _y , and the current location _a1 of the client 10A is shown in FIG. Suppose there is. In this case, the position and orientation estimating unit 102 of the information processing device 100 estimates the position and orientation of the client 10A using VPS based on the image for position and orientation estimation transmitted from the client 10A and the landmark _Li included in the 3D map.

Next, in step S403, the estimation accuracy determination unit 103 determines the accuracy of the position and orientation estimation result by the position and orientation estimation unit 102. The accuracy of the position and orientation estimation results can be determined by calculating the estimation accuracy of the position and orientation estimation results and comparing the estimation accuracy with a predetermined threshold (first threshold).

As a method for calculating the estimation accuracy, for example, there is a method that uses the average value of the distances between all the landmarks used for position and orientation estimation and the position of the client 10A estimated by the position and orientation estimation unit 102. If this average value is less than the first threshold, it is determined that the estimation accuracy is high, and if the average value is greater than the first threshold, it is determined that the estimation accuracy is low. The first threshold value may be set in advance to a specific value by the operator of the information processing system 1000 or the developer of the AR application, or may be set by the operator or the developer depending on the operational status of the information processing system 1000 or the AR application. It may be possible to make changes by a developer or the like.

If the estimation accuracy is high, the process proceeds to step S404 (Yes in step S403). Then, in step S404, the information processing device 100 transmits the position and orientation estimation results to the client 10A.

On the other hand, if the estimation accuracy is low, the process proceeds to step S405 (No in step S403), and the information processing device 100 requests transmission of the 3D map update image and updates the 3D map. The process of step S405 will be described later with reference to FIG.

The processing of the client 10A after the information processing device 100 transmits the position and orientation estimation result to the client 10A in step S404 will be described with reference to FIG. 9A. When the information processing device 100 transmits the position and orientation estimation result to the client 10A, the process proceeds from step S303 to step S304 (Yes in step S303), and the client 10A receives the position and orientation estimation result.

Then, in step S305, the client 10A executes the AR application using the position and orientation estimation results. The application control unit 17A outputs the position and orientation estimation results sent from the server 20 to the drawing unit 18A, and the drawing unit 18A draws a virtual object based on the position and orientation estimation results to generate an output image. It is output to the display section 15A. As a result, the output image is displayed on the display section 15A.

Next, with reference to FIG. 11, the request to send the 3D map update image and the 3D map update in step S405 described above will be explained. When the estimation accuracy of the position and orientation estimation result of the client 10A is low, the process shown in FIG. 11A is performed for each landmark included in the 3D map used for estimating the position and orientation of the client 10A. The process shown in FIG. 11A is a process of updating the 3D map in order to improve the estimation accuracy of the position and orientation of the client 10A.

First, in step S501, the 3D map accuracy determining unit 104 determines the estimation accuracy of the landmark _Li included in the 3D map and used to estimate the position and orientation of the client 10. Here, the estimation accuracy of the landmark L _i is assumed to be E _i .

As shown in FIG. 10, let D _xy be the distance between the imaging positions of two KFs (KF _x , KF _y ) used for triangulation, and let l _xi be the distance between the imaging position of KF _x and landmark _Li . , KF _y , and the distance between the landmark L _i is _lyi , the estimation accuracy E _i can be calculated using the following equation 1.

[Formula 1]

If this estimation accuracy E _i is less than a predetermined threshold (second threshold), it is determined that the estimation accuracy of the landmark L _i is low and the process proceeds to step S502 (No in step S501).
The second threshold value may be set to a specific value in advance by the operator of the information processing system 1000 or the developer of the AR application, or may be set by the operator or the like depending on the operational status of the information processing system 1000 or the AR application. It may be possible to make changes by a developer or the like. Note that although the estimation accuracy E i of the landmark L _i is calculated based on two KFs in FIG. 10, the estimation accuracy E _i of the landmark L i is calculated based on three or more KFs using the same method. good.

Next, in step S502, the client specifying unit 106 uses a 3D map necessary for updating the landmark _Li based on the coarse-grained position information in a search range that is a predetermined range based on the landmark _Li . Other clients 10 that can request transmission of update images are identified. The other client 10 is a client 10 that is different from the client that transmitted the position and orientation estimation image and made the position and orientation estimation request. To update the landmark L _i , a 3D map update image obtained by capturing the landmark L _i from another position is required. Therefore, the client specifying unit 106 uses the client position DB 105 to search for other clients based on the coarse-grained position information transmitted from other clients 10 within the search range based on the landmark _Li , as shown in FIG. The client 10 is specified. The search range is a range determined based on the distance between the imaging position of the key frame included in the 3D map and the landmark. More specifically, the search range can be defined as a circle with a radius _Ri based on the landmark _Li , and the radius _Ri is, for example, a circle from the landmark _Li to the imaging of KF _x included in the 3D map. It can be calculated using the following equation 2 using the distance l _xi between the positions and the distance l _yi from the landmark L _i to the KF _y included in the 3D map. However, the search range is not limited to a circular shape, and may be a rectangular range or a range of any shape.

[Formula 2]

Note that the denominator value "2" in Equation 2 is not a fixed value but is merely an example, and the denominator value is not limited to 2. The larger the value of the denominator in Equation 2, the smaller the radius R _i becomes, so the size of the search range can be adjusted by adjusting the value of the denominator.

If another client 10 exists in the search range, the process proceeds to step S503 (Yes in step S502). Here, as an example, it is assumed that the client 10B is detected from the search range as shown in FIG. Note that, since the client 10C exists outside the search range, the client specifying unit 106 does not specify the client 10C as the client 10 that can request transmission of the 3D map update image.

Next, in step S503, the information processing device 100 requests the client 10B identified by the client identifying unit 106 to send a 3D map update image.

Then, in step S504, the information processing device 100 enters a waiting state until the 3D map update image arrives from the client 10B (No in step S504).

The processing of client 10B after information processing device 100 requests client 10B to send an image for updating the 3D map in step S503 will be described with reference to FIG. 11B.

When the information processing device 100 requests the transmission of the 3D map update image, the client 10B receives the request to transmit the 3D map update image in step S601.

Next, in step S602, under the control of the application control unit 17B, the client 10B displays a UI (image acquisition UI) on the display unit 15B for prompting the user B who uses the client 10B to capture an image for updating the 3D map. do. Although the client 10B exists within the search range based on the landmark _Li , the user B who uses the client 10B does not know where the landmark _Li is located. Therefore, by presenting KF _x and KF _y used for estimating landmark _Li to user B, user B can search for landmark _Li and capture camera images around landmark _Li . Make it. As the image acquisition UI, for example, the one shown in FIG. 13 can be considered.

The image acquisition UI shown in FIG. 13 displays KF (KF _x , KF _y ) included in the 3D map to be updated and a message prompting the user to capture a camera image as the 3D map update image. Since an object P having _a landmark Li is displayed on both KF _x and KF _y displayed on the image acquisition UI, user B uses this object P as a landmark to update the 3D map update image. can be imaged.

Furthermore, if the client 10B is equipped with a sensor that detects the direction, an icon IC indicating the direction in which an object P having a landmark _Li is present may be displayed together with KF by being superimposed on the camera image as shown in Fig. 14. This camera image is an image currently being captured by the client 10B. The icon IC is displayed superimposed on the camera image, with the current camera image always being displayed on the display unit 15B, like a through image when shooting a video.

Next, in step S603, the client 10B images the surrounding real space and acquires a 3D map update image in response to an input operation by user B with reference to the image acquisition UI. Here, it is assumed that a camera image as shown in FIG. 15 is captured at the position shown in FIG. 12.

Then, in step S604, the client 10B transmits the camera image shown in FIG. 15 to the information processing apparatus 100 as a 3D map update image.

The processing of the information processing device 100 after the client 10B transmits the 3D map update image to the information processing device 100 in step S604 will be described with reference to FIG. 11A.

When the client 10B transmits the 3D map update image to the information processing device 100, the process proceeds from step S504 to step S505 (Yes in step S504), and in step S505, the information processing device 100 receives the 3D map update image.

Next, in step S506, the 3D map update unit 107 updates the 3D map using the 3D map update image. Here, as shown in FIG. 16, a case will be considered in which the 3D map update image sent by the client 10B is used as a new KF, KF _z . In this case, the baseline length between KF _x and KF _z is longer than the baseline length between KF _x and KF _y , so when using _{KF x} _and _{KF z} _, when using _KF Landmark L _i can be estimated with higher accuracy than when using _z .

The above processing of steps S501 to S505 is performed for all landmarks used for position and orientation estimation.

Note that if the estimation accuracy of the landmark is high in step S501 (Yes in step S501), the process ends without requesting the transmission of the 3D map update image or updating the 3D map. This is because if the landmark estimation accuracy is high, there is no need to update the 3D map.

Further, in step S502, if the client specifying unit 106 cannot detect a client 10 capable of transmitting the 3D map update image necessary for updating the landmark _Li within the search range (No in step S502), the 3D map The process ends without requesting to send an update image or updating the 3D map. This is because if the client 10 does not exist, a request to send a 3D map update image cannot be made.

Then, after updating the 3D map, the client 10A captures a new image for position and orientation estimation in step S301 of FIG. 9A, transmits the image for position and orientation estimation to the information processing device 100 again in step S302, and requests position and orientation estimation. If so, the information processing apparatus 100 receives a position and orientation estimation image in step S401, and performs position and orientation estimation again in step S402.

At this time, the information processing apparatus 100 estimates the position and orientation of the client 10A using the 3D map updated in step S505. Since the position and orientation estimation is performed using the 3D map updated by increasing the precision of the landmark L _i , the position and orientation estimation result should be higher than before the 3D map was updated.

If a plurality of clients 10 exist within the search range, the information processing device 100 may request the plurality of clients 10 to transmit a 3D map update image. Then, when 3D map update images are transmitted from a plurality of clients 10, the 3D map update unit 107 estimates the landmark L _i using all of the 3D map update images and updates the 3D map. do. After the update process is completed, the position and orientation estimating unit 102 performs position estimation using the updated 3D map, and the position and orientation estimation result is transmitted to the client 10 that has made the position and orientation estimation request.

Note that the client 10B, which has received a request to send a 3D map update image, does not necessarily transmit an appropriate 3D map update image in which the landmark _Li is reflected to the information processing apparatus 100. There is also a possibility that a camera image that does not include the landmark _Li is transmitted, and in that case, the 3D map updating unit 107 cannot update the 3D map. Therefore, the information processing device 100 may continue to issue requests to all clients within the search range to send 3D map update images until the 3D map is updated by receiving the 3D map update image. .

The processing in this technology is performed as described above. According to the present technology, another client 10 that can send a 3D map update image for increasing the precision of landmarks included in a 3D map is identified, and a request is made to the other client to send the 3D map update image. . Thereby, the 3D map can be updated using the 3D map update image that cannot be obtained by the client that has made the position and orientation estimation request. Since the landmarks included in the updated 3D map have increased accuracy, the accuracy of the position and orientation estimation results of the client that has requested position and orientation estimation can also be improved. Furthermore, by using the position and orientation estimation result with improved accuracy, it is possible to improve the accuracy of drawing a virtual object in an AR application.

<2. Modified example>
Although the embodiments of the present technology have been specifically described above, the present technology is not limited to the above-described embodiments, and various modifications based on the technical idea of the present technology are possible.

In the embodiment, one information processing device 100 performs processing on the server 20, but the information processing device 100 may be divided into a plurality of information processing devices according to their functions, and each information processing device may perform processing on a different server. Good too.

Although the embodiment has been described assuming that there is one server 20, there is no limit to the number of servers 20. The present technology may be combined with a 3D map update method that uses multiple servers 20. This can reduce the time required to update the 3D map.

The present technology can also have the following configuration.
(1)
an estimation accuracy determination unit that determines the estimation accuracy of the position and orientation of the client, which is estimated based on a position and orientation estimation image captured in real space transmitted from the client and a 3D map of the real space;
An information processing apparatus comprising: a client specifying unit that specifies another client that requests transmission of a 3D map update image for updating the 3D map, based on a determination result of the estimation accuracy.
(2)
As described in (1), when the estimation accuracy determination unit determines that the estimation accuracy of the position and orientation of the client is low, the client identification unit identifies the other client that requests transmission of the 3D map update image. information processing equipment.
(3)
If the estimation accuracy determining unit determines that the estimation accuracy of the position and orientation of the client is low, the accuracy of the landmark included in the 3D map and used to estimate the position and orientation of the client is determined, and the accuracy of the landmark used to estimate the position and orientation of the client is determined. The information processing apparatus according to (2), wherein the client specifying unit specifies the other client requesting transmission of the 3D map update image when the mark accuracy is low.
(4)
The accuracy of the landmark is based on the distance between a plurality of key frames included in the 3D map and used for estimating the landmark, and the distance between each of the plurality of key frames and the landmark. The information processing device according to (3), wherein the information processing device is calculated.
(5)
The estimation accuracy determination unit determines a predetermined threshold value and an average distance between the position of a landmark used for estimating the position and orientation of the client included in the 3D map and the estimated position of the client. The information processing device according to any one of (1) to (4), wherein the estimation accuracy of the position and orientation is determined by comparison.
(6)
The client specifying unit specifies the other client that requests transmission of the 3D map update image within the search range, based on the position information sent from the other client. The information processing device according to any one of the above.
(7)
The search range is based on a landmark included in the 3D map and used for estimating the position and orientation of the client, and is based on a distance between the imaging position of a key frame included in the 3D map and the landmark. The information processing device according to (6), which is a determined range.
(8)
The information processing device according to any one of (1) to (7), further comprising a 3D map update unit that updates the 3D map using the 3D map update image transmitted from the other client.
(9)
The 3D map updating unit updates the 3D map by estimating the position of the landmark by triangulation using the positional relationship between the feature points in the 3D map update image and key frames included in the 3D map ( 8) The information processing device according to item 8).
(10)
The information processing according to any one of (1) to (9), including a position and orientation estimation unit that estimates the position and orientation of the client based on the position and orientation estimation image transmitted from the client and the 3D map. Device.
(11)
(1) to (10) in which key frames included in the 3D map and used to estimate landmarks are displayed on a display unit included in the other client that has received the request to send the 3D map update image; The information processing device according to any one of.
(12)
The information processing device according to (11), wherein information about a direction in which the landmark exists is displayed on the display unit.
(13)
The information processing apparatus according to any one of (1) to (12), wherein the information processing apparatus requests another client different from the client that has transmitted the position and orientation estimation image to send the 3D map update image.
(14)
Determining the estimation accuracy of the position and orientation of the client estimated based on a position and orientation estimation image captured from the real space transmitted from the client and a 3D map of the real space;
A program that causes a computer to execute an information processing method for identifying another client that requests transmission of a 3D map update image for updating the 3D map based on a determination result of the estimation accuracy.
(15)
A client that sends a position and orientation estimation image captured in real space and requests position and orientation estimation,
an estimation accuracy determination unit that determines the estimation accuracy of the position and orientation of the client estimated based on the position and orientation estimation image and the 3D map of the real space;
an information processing device that specifies another client that requests transmission of a 3D map update image for updating the 3D map based on a determination result of the estimation accuracy;
An information processing system comprising: another client that transmits the 3D map update image in response to a request from the information processing device to transmit the 3D map update image.

10...Client 100...Information processing device 102...Position/orientation estimating section 103...Estimation accuracy determining section 104...3D map accuracy determining section 106...Client specifying section 107...3D map Update section 1000...information processing system

Claims

an estimation accuracy determination unit that determines the estimation accuracy of the position and orientation of the client, which is estimated based on a position and orientation estimation image captured in real space transmitted from the client and a 3D map of the real space;
An information processing apparatus comprising: a client specifying unit that specifies another client that requests transmission of a 3D map update image for updating the 3D map, based on a determination result of the estimation accuracy.
When the estimation accuracy determination unit determines that the estimation accuracy of the position and orientation of the client is low, the client identification unit identifies the other client that requests transmission of the 3D map update image. information processing equipment.
If the estimation accuracy determining unit determines that the estimation accuracy of the position and orientation of the client is low, the accuracy of the landmark included in the 3D map and used to estimate the position and orientation of the client is determined, and the accuracy of the landmark used to estimate the position and orientation of the client is determined. The information processing apparatus according to claim 2, wherein the client specifying unit specifies the other client requesting transmission of the 3D map update image when the accuracy of the mark is low.
The accuracy of the landmark is based on the distance between a plurality of key frames included in the 3D map and used for estimating the landmark, and the distance between each of the plurality of key frames and the landmark. The information processing device according to claim 3, wherein the information processing device is calculated.
The estimation accuracy determination unit determines a predetermined threshold value and an average distance between the position of a landmark used for estimating the position and orientation of the client included in the 3D map and the estimated position of the client. The information processing apparatus according to claim 1, wherein the estimation accuracy of the position and orientation is determined by comparison.
The information processing according to claim 1, wherein the client specifying unit specifies the other client that requests transmission of the 3D map update image within a search range based on position information transmitted from the other client. Device.
The search range is based on a landmark included in the 3D map and used for estimating the position and orientation of the client, and based on the distance between the imaging position of the key frame included in the 3D map and the landmark. The information processing device according to claim 6, wherein the information processing device is a determined range.
The information processing apparatus according to claim 1, further comprising a 3D map update unit that updates the 3D map using the 3D map update image transmitted from the other client.
The 3D map updating unit updates the 3D map by estimating the position of the landmark by triangulation using the positional relationship between the feature points in the 3D map updating image and key frames included in the 3D map. Item 8. Information processing device according to item 8.
The information processing apparatus according to claim 1, further comprising a position and orientation estimation unit that estimates the position and orientation of the client based on the position and orientation estimation image transmitted from the client and the 3D map.
The information according to claim 1, wherein a key frame included in the 3D map and used for estimating a landmark is displayed on a display unit included in the other client that has received the request to send the 3D map update image. Processing equipment.
The information processing apparatus according to claim 11, wherein information about a direction in which the landmark exists is displayed on the display unit.
The information processing apparatus according to claim 1, wherein the information processing apparatus requests another client different from the client that has transmitted the position and orientation estimation image to send the 3D map update image.
Determining the estimation accuracy of the position and orientation of the client estimated based on a position and orientation estimation image captured from the real space transmitted from the client and a 3D map of the real space;
A program that causes a computer to execute an information processing method for identifying another client that requests transmission of a 3D map update image for updating the 3D map based on a determination result of the estimation accuracy.
A client that sends a position and orientation estimation image captured in real space and requests position and orientation estimation,
an estimation accuracy determination unit that determines the estimation accuracy of the position and orientation of the client estimated based on the position and orientation estimation image and the 3D map of the real space;
an information processing device that specifies another client that requests transmission of a 3D map update image for updating the 3D map based on a determination result of the estimation accuracy;
An information processing system comprising: another client that transmits the 3D map update image in response to a request from the information processing device to transmit the 3D map update image.