WO2023243558A1

WO2023243558A1 - Information processing device, program, and information processing system

Info

Publication number: WO2023243558A1
Application number: PCT/JP2023/021513
Authority: WO
Inventors: 優生武田; 一若林; 富士夫荒井; ダニエル誠徳永; 諒介村田; 春香藤澤
Original assignee: ソニーグループ株式会社
Priority date: 2022-06-15
Filing date: 2023-06-09
Publication date: 2023-12-21

Abstract

An information processing device comprising a priority setting unit that sets a priority for update for each of a plurality of 3D maps which are targets of an update request and a 3D map update unit that uses sensing data to update the 3D maps on the basis of the priorities, the 3D maps being the targets of the update request.

Description

Information processing equipment, programs and information processing systems

The present technology relates to an information processing device, a program, and an information processing system.

In AR (Augmented Reality), the same 3D object (virtual object) may be displayed on multiple devices at a specific location in real space. To do this, it is necessary to specify the position and orientation of each device using a common coordinate system. As a means for realizing this, there is a method called VPS (Visual Positioning System) that estimates the position and orientation of each terminal from camera images.

In VPS, a 3D map of the real space is used, and by associating the camera images sent from each device with the features of the 3D map, the position and orientation at which the camera image was captured is estimated in the 3D map coordinate system. 3D maps can be generated by simultaneously estimating the three-dimensional shape and imaging point for multiple images from different viewpoints using technologies such as SfM (Structure from Motion) and VIO (Visual Inertial Odometry). . When estimating the position and orientation of a device from the correspondence between the camera image captured during use and the 3D map generated from past data, there is a discrepancy between the real space landscape at the time the 3D map was generated and the current real space landscape. It is desirable that you do not.

However, since many objects in real space can move and deform, the environment may have changed between the past 3D map generation time and the present. There is a problem that this change in the environment has a negative effect on the success rate and accuracy of position and orientation estimation, and the virtual object is not displayed in the intended position.

Such problems can be solved by updating the 3D map of the real space every time the object moves or deforms, but updating the 3D map takes a long processing time, so the 3D map update is completed. Previously, past 3D maps were used, which resulted in a problem that the accuracy and success rate of position and orientation estimation were low.

Therefore, by comparing the image with the stored dataset in the database and requesting the retrieval of additional data if it is determined that the stored dataset is missing one or more details of the environment. Techniques have been proposed to obtain additional data of the environment.

JP2021-170341A

Although the technology of Patent Document 1 can be used to collect data for creating a 3D map of the environment, there is an unresolved problem that it takes time to create a 3D map.

This technology was developed in view of these problems, and provides an information processing device, a program, and an information processing system that can efficiently update a 3D map by setting update priorities for the 3D map. The purpose is to provide.

In order to solve the above-mentioned problems, a first technique includes a priority setting unit that sets a priority when updating each of a plurality of 3D maps that are the targets of an update request, The information processing apparatus includes a 3D map update unit that uses sensing data to update a 3D map that is the subject of an update request.

In addition, the second technology sets a priority when updating each of a plurality of 3D maps that are the subject of an update request, and uses sensing data based on the priority to update the 3D map that is the subject of an update request. This is a program that causes a computer to execute an information processing method for updating a map.

Furthermore, the third technology is a client that sends sensing data acquired by a sensor unit and requests an update, and a priority that sets a priority when updating each of a plurality of 3D maps that are the targets of an update request. This information processing system includes an information processing apparatus including a degree setting section and a 3D map updating section that updates a 3D map that is the subject of an update request using sensing data based on the priority.

1 is a block diagram showing the configuration of a client 10. FIG. 2 is a block diagram showing the configuration of a server 20. FIG. 1 is a block diagram showing the configuration of an information processing system 1000. FIG. 1 is a block diagram showing the configurations of an information processing system 1000 and an information processing device 100. FIG. 3 is a sequence diagram showing processing in the client 10 and the information processing device 100. FIG. 3 is a flowchart showing processing in the information processing device 100. FIG. 3 is a sequence diagram showing processing in the client 10 and the information processing device 100. FIG. 12 is a flowchart showing processing in a specific example of the present embodiment. It is a figure which shows the camera image imaged by client 10A in the specific example of this Embodiment. It is a figure which shows the camera image imaged by the client 10B in the specific example of this Embodiment. It is a figure which shows the camera image used for 3D map generation in the specific example of this Embodiment. It is a block diagram showing the composition of information processing system 1000 in a modification of this technique.

Embodiments of the present technology will be described below with reference to the drawings. Note that the explanation will be given in the following order.
<1. Embodiment>
[1-1. Configuration of client 10 and server 20]
[1-2. Configuration of information processing system 1000 and information processing device 100]
[1-3. Processing in information processing system 1000 and information processing device 100]
[1-4. Specific example of processing in information processing system 1000 and information processing device 100]
<2. Modified example>

<1. Embodiment>
[1-1. Configuration of client 10 and server 20]
The client 10 included in the information processing system 1000 will be described with reference to FIG. 1. The client 10 includes at least a control section 11, a storage section 12, a communication section 13, an input section 14, a display section 15, and a sensor section 16.

The control unit 11 includes a CPU (Central Processing Unit), a RAM (Random Access Memory), a ROM (Read Only Memory), and the like. The CPU controls the entire client 10 and each part by executing various processes and issuing commands according to programs stored in the ROM.

The storage unit 12 is, for example, a large capacity storage medium such as a hard disk or flash memory. The storage unit 12 stores AR applications and various data used by the client 10.

The communication unit 13 is a communication module that communicates with the server 20, the Internet, etc. Communication methods can be wired or wireless, including cellular communication, Wi-Fi, Bluetooth (registered trademark), NFC (Near Field Communication), Ethernet (registered trademark), and HDMI (registered trademark) (High-Definition Multimedia). interface), USB (Universal Serial Bus), etc.

The input unit 14 is used by the provider to input information and give various instructions to the client 10. When a user makes an input to the input unit 104, a control signal corresponding to the input is generated and supplied to the control unit 11. The control unit 11 then performs various processes corresponding to the control signals. In addition to physical buttons, the input unit 104 includes a touch panel, a touch screen integrated with a monitor, and the like.

The display unit 15 is a display device such as a display that displays images, videos, a UI (User Interface) for using the client 10, and the like.

The sensor unit 16 is various sensors for acquiring sensing data. As the sensor unit 16, a camera, IMU (Inertial Measurement Unit), LiDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging), ToF (Time of Flight) sensor, etc. can be used. Furthermore, sensing data may also be information that can be used to specify a location, such as location information obtained by GPS (Global Positioning System), network base station information, Wi-Fi access point name, etc.

In this embodiment, a camera is used as the sensor unit 16. The camera includes a lens, an image sensor, a video signal processing circuit, and the like, and can capture a camera image as sensing data for capturing an image of the real world and transmitting it to the server 20.

The client 10 is configured as described above. Examples of the client 10 include a smartphone, a tablet terminal, a digital camera, a wearable device, a head-mounted display, and a personal computer. The control program that performs the processing according to the present technology may be installed in the client 10 in advance, or may be downloaded, distributed on a storage medium, etc., and installed by the provider himself.

Next, the server 20 that constitutes the information processing system 1000 will be explained with reference to FIG. 2. The server 20 includes at least a control section 21, a storage section 22, and a communication section 23. The control unit 21, storage unit 22, and communication unit 23 have the same functions as those provided in the client 10. The server 20 may be a cloud server, for example.

[1-2. Configuration of information processing system 1000 and information processing device 100]
Next, the configurations of the information processing system 1000 and the information processing apparatus 100 will be described with reference to FIGS. 3 and 4.

As shown in FIG. 3, the information processing system 1000 includes a plurality of clients 10 and a server 20 that performs processing as the information processing device 100. A plurality of clients 10 and servers 20 are connected via a network. The client 10 makes one or both of a 3D map update request and a position and orientation estimation request to the information processing apparatus 100. Note that the client 10 functions as an AR device by executing an AR application, but a client that only requests a 3D map update and does not request position and orientation estimation may not function as an AR device. There is no particular limitation on the number of clients 10, and the clients 10 that utilize 3D map updating and position and orientation estimation by the information processing apparatus 100 are connected to the information processing apparatus 100 via a network.

In FIG. 4, the information processing apparatus 100 constituting the information processing system 1000 and two clients, client A (denoted as client 10A) and client B (denoted as client 10B), which are two of the plurality of clients, are extracted and shown. . Note that FIG. 4 shows only processing blocks related to information and signals related to the present technology.

In FIG. 4, data transmission and reception between the client 10A and the information processing device 100 and data transmission and reception between the client 10B and the information processing device 100 are performed by communication functions provided in the client 10A, the client 10B, and the server 20.

The client 10A requests a 3D map update by transmitting sensing data acquired by the sensor unit 16A to the server 20 as 3D map update data. Note that the client 10A may function as an AR device by executing an AR application, or may not function as an AR device.

Furthermore, the client 10B functions as an AR device by executing an AR application. The client 10B makes a request to estimate the position and orientation of the client 10B by transmitting sensing data acquired by the sensor unit 16B to the information processing device 100 as position and orientation estimation data.

In addition to the configuration shown in FIG. 1, the client 10B includes an application control section 17B and a drawing section 18B.

The application control unit 17B performs processing related to the AR application executed by the client 10, and acquires the position and orientation estimation results sent from the server 20 and outputs them to the drawing unit 18B.

The drawing unit 18B draws a virtual object based on the position and orientation estimation results transmitted from the information processing device 100 to generate an output image, and outputs the output image to the display unit 15B. As a result, the output image is displayed on the display section 15B. Virtual objects include, for example, characters, figures, objects, and characters that represent information in a map service, and are arranged on images of the real world.

The application control unit 17B and the drawing unit 18B are realized by the client 10B executing an AR application, and the application may be installed in the client 10B in advance, or may be downloaded or distributed on a storage medium, so that the user can You may also install it yourself. Further, the client 10B may be provided with the functions of the application control section 17B and the drawing section 18B in advance.

The information processing device 100 includes an update request DB (Database) control unit 101, an update request DB 102, a 3D map usage record DB 103, a 3D map DB 104, a priority setting unit 105, a computer resource allocation unit 106, a 3D map update unit 107, and a 3D map selection unit. section 108 and position/orientation estimating section 109. In this embodiment, the storage unit 22 of the server 20 functions as the update request DB 102, the 3D map usage record DB 103, and the 3D map DB 104, but each DB may be configured with a different storage medium.

The update request DB control unit 101 receives 3D map update data sent from the client 10 and stores it in the update request DB 102. In the example of FIG. 4, the update request DB control unit 101 stores the 3D map update data sent from the client 10A in the update request DB 102.

The update request DB 102 is a database that stores 3D map update data sent from the client 10.

The 3D map usage record DB 103 is a database that records which 3D map was used to perform position and orientation estimation based on a position and orientation estimation request from the client 10.

The 3D map DB 104 is a database that stores 3D maps that model real space. Multiple 3D maps exist for each point in real space. A 3D map can be generated by simultaneously estimating the three-dimensional shape and imaging point of multiple images from different viewpoints using techniques such as SfM (Structure from Motion) and VIO (Visual Inertial Odometry). The 3D map contains 3D shapes and feature points (Landmarks) that model objects in real space, feature points and features used for position and orientation estimation, images used for 3D shape estimation (Key Frames), etc. Contains data. Examples of feature points include the position of the object, the position of the vertex of an acute angle part of the object, and the outline of the object. In the present technology, it is assumed that a plurality of generated 3D maps are stored in the 3D map DB 104 in advance.

The priority setting unit 105 sets priorities for each of the plurality of 3D maps that are the targets of update requests based on information regarding the 3D maps.

The computer resource allocation unit 106 allocates computer resources to be used for updating the 3D map based on the priority. In this embodiment, the computer is the server 20 that updates the 3D map, and the computer resources are the hardware of the server 20, such as the execution time on the server 20, the usage rate of the processor, and the amount of space occupied by the storage unit 22. It refers to the amount of occupancy.

The 3D map update unit 107 updates the 3D map that is the target of the update request, using sensing data as 3D map update data and the allocated computer resources.

The 3D map selection unit 108 selects a 3D map to be used for position and orientation estimation from among the plurality of 3D maps stored in the 3D map DB 104, based on sensing data that is data for position and orientation estimation transmitted from the client 10. select. In the example of FIG. 4, the 3D map selection unit 108 selects a 3D map based on the position and orientation estimation data transmitted from the client 10B.

The position and orientation estimating unit 109 estimates the position and orientation of the client 10 based on the correspondence between the camera image, which is position and orientation estimation data, transmitted from the client 10 by the VPS and the 3D map selected by the 3D map selection unit 108. presume. In VPS, the position and orientation of the client 10 is estimated in the 3D map coordinate system using a method such as Perspective-n-Point based on the correspondence between feature points and feature amounts of the camera image and the 3D map.

The position and orientation estimation result by the position and orientation estimation unit 109 is transmitted to the client 10 that made the position and orientation estimation request. In the example of FIG. 4, the position and orientation estimating unit 109 estimates the position and orientation of the client 10B based on the correspondence between the position and orientation estimation data transmitted from the client 10B and the 3D map, and the information processing device 100 estimates the position and orientation of the client 10B. Send the result to client 10B. The drawing unit 18B of the client 10B draws a virtual object in AR using the position and orientation estimation results.

The information processing system 1000 and the information processing device 100 are configured as described above. In this embodiment, the information processing device 100 operates on the server 20, but the server 20 may have the function of the information processing device 100 in advance, or the server 20, which has the function of a computer, may execute the program. The information processing device 100 and the information processing method may be realized by the following. Further, the control unit 21 may function as the information processing device 100 by executing a program. The program may be installed in advance on the server 20, or may be downloaded, distributed on a storage medium, etc., and installed by a user or the like. Further, the information processing device 100 may be configured as a single device. Further, the information processing device 100 is not limited to the server 20, and may operate on a personal computer, a smartphone, a tablet terminal, or the like.

[1-3. Processing in information processing system 1000 and information processing device 100]
Next, processing in the information processing system 1000 and the information processing apparatus 100 will be described with reference to FIGS. 5 to 7. In the following description, it is assumed that the sensing data acquired by the sensor unit 16 is a camera image captured by a camera.

As shown in FIG. 5, first, in step S101, the sensor unit 16A of the client 10A captures a surrounding camera image to obtain sensing data based on the user's input operation.

Next, in step 102, the client 10A transmits the camera image, which is sensing data, to the information processing device 100 as 3D map update data.

Next, in step S201, the information processing device 100 receives 3D map update data sent from the client 10A.

Next, in step S202, the update request DB control unit 101 stores the 3D map update data in the update request DB 102, and further records the reception time of the 3D map update data.

Next, in step S203, the priority setting unit 105 identifies the 3D map that is the target of the update request from among the plurality of 3D maps stored in advance in the 3D map DB 104 based on the 3D map update data. The priority setting unit 105 extracts the feature points and feature amounts of the camera image as the received 3D map update data, and uses the feature points and feature amounts and the features of each of the plurality of 3D maps stored in the 3D map DB 104. Points, feature amounts, etc. are compared, and a 3D map that matches or approximates them is identified as a 3D map that is the subject of an update request.

It is assumed that there are requests to update the 3D map from a plurality of clients 10 in addition to the client 10A, and each of these clients 10 requests a different 3D map.

Next, priority setting, computer resource allocation, and 3D map updating in the information processing device 100 will be described with reference to FIG. 6.

First, in step S301, the priority setting unit 105 sets priorities for all 3D maps that are the targets of update requests.

Here, the priority setting will be explained. Specifically, a score is set for the x-th 3D map among the n 3D maps that are the targets of the update request based on information regarding the 3D map. In this embodiment, it is assumed that there are seven types of information regarding the 3D map, and the scores calculated from each piece of information regarding the 3D map are a _x , b _x , c _x , d _x , e _x , f _x , and g _x, respectively. Then, the priority level p _x of the x-th 3D map is set from the total score.

The information regarding the first 3D map is "the number of times position and orientation estimation requests have been made since the time when the 3D map update data was saved." This is the number of times that position and orientation estimation was performed based on the 3D map before the 3D map that is the target of the update request was updated, even though the information processing device 100 received the 3D map update data. It is. The time when the 3D map update data is received is, for example, the time when the update request DB control unit 101 saves the 3D map update data in the update request DB 102, but it can also be the time when the server 20 receives the 3D map update data. good. This number of times can be determined by comparing the time of the position/orientation estimation request stored in the 3D map usage record DB 103 with the time of receiving the update request. This number of times is defined as the score _ax . Even though the 3D map update data has been received, the position and orientation estimation is performed based on the 3D map before the update, so it is thought that the accuracy of position and orientation estimation using the 3D map has deteriorated. . Therefore, since the 3D map should be updated with priority, the priority is increased by the score _ax .

The information regarding the second 3D map is "the number of times it has been used for position and orientation estimation in the past." This number of times is recorded in the 3D map usage record DB 103. This number of times is defined as the score _bx . It is considered that the more times a 3D map has been used for position and orientation estimation in the past, the more times it will be used for position and orientation estimation in the future. Therefore, since that 3D map should be updated with priority, the priority is increased by the score b _x .

The information regarding the third 3D map is "the ratio of information that can be used for updating the 3D map among the data for updating the 3D map." Let this ratio be the score _cx . The 3D map update data sent from the client 10 may include information that cannot be used to update the 3D map. For example, there are camera images with insufficient brightness (light intensity) and blacked-out images, camera images that cannot be used for static 3D maps because they include many moving objects, and the like. Therefore, the higher the ratio of information that can be used to update the 3D map among the information included in the 3D map update data, the higher the priority is given by the score c _x .

Detection of areas with insufficient brightness in the camera image as 3D map update data may be performed, for example, based on whether the average brightness value of the entire specific area in the camera image is below a specific threshold value. Can be done. Furthermore, detection of a moving body region in an image as data for updating a 3D map can be performed based on the results of semantic segmentation or the like. The number of pixels in the area in the camera image that cannot be used for updating such a 3D map is calculated for each camera image, and the total value s is determined. The more update data that can be used to update the 3D map, the higher the priority, so by subtracting s from the number of pixels in the entire camera image and dividing the value of the subtraction result by the number of pixels in the entire camera image, the 3D map can be updated. Calculate the percentage of usable data in the update data. Let this ratio be _cx .

The information regarding the fourth 3D map is "the degree of similarity between the 3D map update data and the existing 3D map." For example, the rate at which the semantic segmentation result of a camera image as 3D map update data matches the semantic segmentation result of a camera image included in an existing 3D map can be used as the degree of similarity. The degree of similarity expressed by the ratio is defined as the score 1/d _x . There is a possibility that camera images similar to the existing 3D map may exist as data for updating the 3D map. For example, a camera image taken of a place where the scenery has not changed corresponds to this. Since it is more necessary to update the 3D map in places where the scenery is changing, the lower the similarity between the 3D map update data and the existing 3D map, the higher the priority is given by the score 1/d _x .

The information regarding the fifth 3D map is "the number of update requests for the 3D map." This number can be obtained by referring to the update request DB 102. This number of times is defined as the score e _x . It is considered that the more updates are requested for the same 3D map, the higher the need for updating is, so the higher the number of update requests, the higher the priority is given by the score _ex .

The information regarding the sixth 3D map is "the priority level for the 3D map set by the person concerned with the AR application". When displaying a virtual object using AR using position and orientation estimation results, the accuracy of position and orientation estimation required may vary depending on the type of virtual object to be displayed. For example, if a virtual object is floating in the air, the user will not be able to perceive it even if the position is slightly shifted, but if the virtual object is placed in contact with a real object, the user will not be able to perceive it if the position is shifted slightly. Therefore, highly accurate position and orientation estimation is required to draw the virtual object. Therefore, it is possible for those involved in the AR application to set the update priority of the 3D map according to the superimposition accuracy of the virtual object. Let this value be _fx . Note that the person involved in the AR application is the developer of the AR application, but may also be an operator, an AR service provider, or the like.

The information regarding the seventh 3D map is "priority for the 3D map set by the user". If a user perceives a superimposition position shift of a virtual object due to deterioration in the accuracy of position and orientation estimation while using an AR application, it is considered that the 3D map used for the position and orientation estimation needs to be updated. Therefore, the user is allowed to set the update priority of the 3D map in hopes of increasing the superimposition accuracy of virtual objects. Let this value be _gx .

Then, the priority is calculated based on the score obtained from each piece of information regarding these first to seventh 3D maps. For example, the priority p _x for the xth 3D map can be calculated using Equation 1 below.

[Formula 1]

Note that in Equation 1, the score obtained as the number of times is converted into a percentage to calculate the priority.

The information regarding the first to seventh 3D maps described above is just an example, and the priority does not always have to be set based on the information regarding these seven 3D maps. The priority may be set based on any one of them, or the priority may be set based on a plurality of them. Therefore, the priority setting unit 105 sets the score obtained from the information regarding one 3D map or the total score obtained from the information regarding a plurality of 3D maps as the priority. Furthermore, the priority may be set based on information regarding 3D maps other than the seven. It is thought that the 3D map can be updated more quickly by reducing the amount of information related to the 3D map used for priority setting and reducing the time required for priority setting.

Note that the information regarding the plurality of 3D maps described above may be weighted according to various conditions. For example, an AR application developer's idea is to weight information regarding the sixth 3D map as being the most important.

Return to the explanation of the flowchart. Next, in step S302, the computer resource allocation unit 106 sets the allocation of computer resources to each of the plurality of 3D maps that are the targets of the update request by performing proportional allocation based on the priority. The proportional distribution coefficient r _x can be calculated using Equation 2 below.

[Formula 2]

For example, when the available computer resources of the server 20 are V, the computer resources V _x that can be used to update the x-th 3D map can be calculated as shown in Equation 3 below.

V _x = V・r _x

For example, if the proportional allocation coefficient r _x of 3D map α is 0.7 and the proportional allocation coefficient r _x of 3D map β is 0.3, 70% of computer resources will be allocated to update 3D map α. , 30% is allocated to updating the 3D map β.

Next, in step S303, the 3D map updating unit 107 uses the allocated computer resources to update the 3D map that is the target of the update request. Note that if the update of the 3D map was interrupted in the past, the update is restarted, and if not, a new update of the 3D map is executed. Similar to 3D map generation, the 3D map can be updated using a camera image as 3D map update data and techniques such as SfM and VIO.

Next, in step S304, if the update of the 3D map is completed before the allocated computer resources are used up, the process proceeds to step S305 (Yes in step S304).

Then, in step S305, the 3D map updating unit 107 replaces the 3D map that is the target of the update request and is stored in the 3D map DB 104 with the updated 3D map.

Further, in step S306, the update request DB control unit 101 deletes the 3D map update data used for updating the 3D map from the update request DB 102.

On the other hand, if the update of the 3D map is not completed before the computer resources allocated in step S305 are used up, the process proceeds to step S307 (No in step S305).

Then, in step S307, the 3D map update unit 107 saves the progress of the 3D map update in the 3D map DB 104 and interrupts the update midway.

The information processing apparatus 100 executes steps S303 to S307 for each 3D map that is the subject of an update request.

Next, a request for position and orientation estimation from the client 10B and execution of position and orientation estimation in the information processing device 100 will be described with reference to FIG.

First, in step S401, the sensor unit 16B of the client 10B captures a surrounding camera image and obtains sensing data based on the user's input operation.

Next, in step 402, the client 10B transmits the camera image, which is sensing data, to the information processing device 100 as position and orientation estimation data.

Next, in step S501, the information processing device 100 receives position and orientation estimation data transmitted from the client 10B.

Next, in step S502, the 3D map selection unit 108 selects one of the plurality of 3D maps stored in the 3D map DB 104 for position and orientation estimation based on the characteristics of the camera image that is the received data for position and orientation estimation. Identify the 3D map. At this time, coarse position information such as GPS or mobile line base station information may be used to narrow down the 3D maps to be used for position and orientation estimation. The 3D map selection unit 108 acquires the selected 3D map from the 3D map DB 104 and outputs it to the position and orientation estimation unit 109. The 3D map selection unit 108 extracts feature points and feature amounts from the camera image as position and orientation estimation data, and compares the feature points and feature amounts with the feature points and feature amounts included in the 3D map. A 3D map that matches or approximates them is identified as a 3D map to be used for position and orientation estimation.

Next, in step S503, the position and orientation estimating unit 109 estimates the position and orientation of the client 10B using the camera image and the 3D map, which are position and orientation estimation data.

Next, in step S504, the information processing device 100 records information (ID, etc.) that identifies the 3D map used for position and orientation estimation and the time when position and orientation estimation was performed in the 3D map usage record DB 103.

Next, in step S505, the information processing device 100 transmits the position and orientation estimation results to the client 10B.

Next, in step S403, the client 10B receives the position and orientation estimation results transmitted from the information processing device 100.

Then, in step 404, the client 10B executes the AR application using the position and orientation estimation results. The application control unit 17B acquires the position and orientation estimation results transmitted from the server 20 and outputs them to the drawing unit 18B, and the drawing unit 18B draws a virtual object based on the position and orientation estimation results to generate an output image, The output image is output to the display section 15B. As a result, the output image is displayed on the display section 15B.

Processing in the information processing system 1000 and the information processing device 100 is performed as described above.

[1-4. Specific example of processing in information processing system 1000 and information processing device 100]
Next, as a specific example, consider a case where user A who uses client 10A and user B who uses client 10B exist in the same space. Here, for convenience of explanation, as a simple example, we will consider a case where a 3D map update request and a position/orientation estimation request are made in the order shown in FIG.

First, in step S601, the client 10A transmits a camera image as 3D map update data to the information processing device 100 based on an input instruction from user A to request an update of the 3D map. Here, it is assumed that the client 10A has transmitted the camera image shown in FIG. 9 to the information processing apparatus 100.

Furthermore, before the information processing apparatus 100 updates the 3D map in response to the 3D map update request, the client 10B transmits a camera image as sensing data to the information processing apparatus 100 based on the input instruction from the user B in step S602. Assume that a position and orientation estimation request is made in order to use an AR application.

Note that user B may or may not know that user A exists or that user A has requested an update of the 3D map. Here, it is assumed that the client 10B transmits the camera image B shown in FIG. 10 to the information processing apparatus 100 as position and orientation estimation data.

It is assumed that camera image A shown in FIG. 9 and camera image B shown in FIG. 10 are images taken from the same position and in the same direction. It is assumed that camera image A shown in FIG. 9 and camera image B shown in FIG. 10 were taken at close dates and times, although not at the same time, and the objects other than the moving vehicle, ie, the automobile, are the same. Therefore, the 3D map that is the target of the 3D map update request from the client 10A and the 3D map used for position and orientation estimation performed in response to the position and orientation estimation request from the client 10B are the same. Note that the same position is not limited to the completely same position, but includes an error of several meters or tens of meters. The same direction is not limited to the completely same direction, but includes an angular error of several degrees or tens of degrees. Moreover, the imaging times of camera image A and camera image B do not need to be the same.

The information processing device 100 performs position and orientation estimation in response to a position and orientation estimation request from the client 10B, but at this time, the 3D map used for position and orientation estimation may be old. For example, if the past camera images used for 3D map generation were as shown in FIG. 11, that is, the past scenery at the time of 3D map generation was in the state shown in FIG. In this case, the scenery has changed to something like this. In the past camera image in FIG. 11 and the current camera image in FIGS. 9 and 10, the building in the area surrounded by the broken line in FIG. 11 has changed. In such a case, it is not possible to estimate the position and orientation using the characteristics of the changed area, so the position and orientation estimation will fail or an estimation result with low accuracy will be obtained.

Next, in step S603, the information processing device 100 updates the 3D map in response to the update request from the client 10A.

Then, in step S604, based on the input instruction from user B, the client 10B transmits the camera image to the information processing device 100 as data for position and orientation estimation, and requests position and orientation estimation again. It is assumed that the camera image at this time is an image taken at the same position and direction as the position and direction in which camera image B shown in FIG. 10 was taken.

Note that in step S604, user B is only using the AR application and is not aware that the 3D map has been updated, and the client 10B is not notified that the 3D map has been updated. . User B simply used the AR application twice in step S602 and step S604.

In response to this position and orientation estimation request, the information processing device 100 estimates the position and orientation of the client 10B using the received camera image and 3D map. At this time, the 3D map is updated with the camera image A, which is the current scenery, in step S603 between the position and orientation estimation request in step S602 and the position and orientation estimation request in step S604. Since camera image A is in a state close to camera image B, which is the current scenery, the position and orientation of the client 10B is estimated using the 3D map updated by camera image A and camera image B. The position and orientation estimation result for the estimation request has higher accuracy than the position and orientation estimation result for the position and orientation estimation request in step S602. By using this highly accurate position and orientation estimation result, the client 10B can draw a virtual object with higher accuracy.

The processing in this technology is performed as described above. According to the present technology, computer resources for updating a 3D map are allocated based on the priority, so the 3D map with a higher priority can be updated more quickly. Furthermore, the 3D map can be updated while suppressing a decrease in accuracy and an increase in generation cost. Then, using the updated 3D map, highly accurate position and orientation estimation becomes possible. Furthermore, the accuracy of position and orientation estimation in an environment with changing scenery can also be improved.

<2. Modified example>
Although the embodiments of the present technology have been specifically described above, the present technology is not limited to the above-described embodiments, and various modifications based on the technical idea of the present technology are possible.

Although camera images were used as sensing data in the embodiment, sensing data is not limited to camera images, and can include acceleration information, distance information, position information, etc. acquired by IMU, LiDAR, ToF sensor, GPS (Global Positioning System), etc. It may be used in place of an image or in combination with a camera image.

In the embodiment, one information processing apparatus 100 performs processing in the server 20, but as shown in FIG. It may be divided into 100C and each may be processed by a different server. Note that the division of information processing devices and servers shown in FIG. 12 is just an example, and the division is not limited to this example.

In the example of FIG. 4, the client 10A that makes a 3D map update request and the client 10B that makes a position and orientation estimation request are described as different clients, but one client 10 makes both a 3D map update request and a position and orientation estimation request. Good too.

Although the embodiment has been described as having one server 20, the number of servers 20 is not limited. The present technology may be combined with a 3D map update method that uses multiple servers 20. This can reduce the time required to update the 3D map.

Additionally, the present technology may be used in conjunction with existing low-precision 3D map generation methods. For example, upon receiving the update request, the information processing apparatus 100 first generates a low-precision 3D map at high speed and updates the 3D map. Although the accuracy of the position and orientation estimation results will not improve because it has been updated with a low-precision 3D map, the success rate of position and orientation estimation can be expected to improve because the position and orientation can be estimated using a 3D map based on the new scenery. After that, a high-precision 3D map is generated using the present technology, and the low-precision 3D map is updated with the high-precision 3D map.

In the embodiment, the position and orientation estimating unit 109 estimates the position and orientation of the client 10, but the information processing device 100 can use VPS to estimate only the position or only the orientation of the client 10. be.

The present technology can also have the following configuration.
(1)
a priority setting unit that sets a priority when updating each of the plurality of 3D maps that are the targets of the update request;
An information processing device comprising: a 3D map update unit that updates the 3D map that is the target of an update request using sensing data based on the priority.
(2)
The information processing device according to (1), wherein the priority setting unit sets the priority based on information regarding the 3D map.
(3)
The information processing device according to (2), wherein the priority setting unit sets a value obtained from information regarding one of the 3D maps or a total of values obtained from information regarding a plurality of the 3D maps as the priority.
(4)
The information regarding the 3D map is the information according to (2) or (3), which is the number of times a position/orientation estimation request using the 3D map that is the subject of the update request has been made after the update request has been made. Processing equipment.
(5)
The information processing device according to any one of (2) to (4), wherein the information regarding the 3D map is the number of times the 3D map has been used for the position and orientation estimation in the past.
(6)
The information processing device according to any one of (2) to (5), wherein the information regarding the 3D map is a ratio of information that can be used to update the 3D map among the sensing data.
(7)
The information processing device according to any one of (2) to (6), wherein the information regarding the 3D map is a degree of similarity between the sensing data and the 3D map.
(8)
The information processing device according to any one of (2) to (7), wherein the information regarding the 3D map is the number of past update requests for the 3D map.
(9)
The information processing device according to any one of (2) to (8), wherein the information regarding the 3D map is a priority set by a person involved in an application that uses the result of position and orientation estimation using the 3D map.
(10)
The information processing device according to any one of (2) to (9), wherein the information regarding the 3D map is a priority set by a user of an application that uses the result of position and orientation estimation using the 3D map.
(11)
The information processing device according to any one of (1) to (10), wherein the priority setting unit identifies the 3D map that is the target of an update request from among the plurality of 3D maps based on the sensing data.
(12)
The information processing apparatus according to any one of (1) to (11), including a computer resource allocation unit that allocates computer resources to be used for updating the 3D map for each of the 3D maps to be updated based on the priority. .
(13)
The information processing device according to (12), wherein the computer resource allocation unit allocates the computer resources to each 3D map by proportionally allocating the computer resources based on the priority.
(14)
The information processing device according to any one of (1) to (13), wherein the sensing data is an image of real space captured by a camera.
(15)
The information processing apparatus according to any one of (1) to (14), including a 3D map selection unit that selects a 3D map to be used for estimating the position and orientation of the device based on sensing data.
(16)
The information processing apparatus according to (15), further comprising a position and orientation estimation unit that estimates the position and orientation of the device using the sensing data and the selected 3D map.
(17)
The information processing apparatus according to (15) or (16), wherein the device is an AR device.
(18)
Set the priority when updating each of the multiple 3D maps that are the subject of the update request,
A program that causes a computer to execute an information processing method for updating the 3D map that is a target of an update request using sensing data based on the priority.
(19)
A client that sends sensing data acquired by the sensor unit and requests an update,
a priority setting unit that sets a priority when updating each of the plurality of 3D maps that are the targets of the update request;
an information processing device comprising: a 3D map update unit that updates the 3D map that is the target of the update request using sensing data based on the priority;
An information processing system consisting of

10...Client 16...Sensor unit 100...Information processing device 105...Priority setting unit 106...Computer resource allocation unit 107...3D map update unit 108...3D map Selection unit 109...Position and orientation estimation unit 1000...Information processing system

Claims

a priority setting unit that sets a priority when updating each of the plurality of 3D maps that are the targets of the update request;
An information processing device comprising: a 3D map update unit that updates the 3D map that is the target of an update request using sensing data based on the priority.
The information processing apparatus according to claim 1, wherein the priority setting unit sets the priority based on information regarding the 3D map.
The information processing apparatus according to claim 2, wherein the priority setting unit sets a value obtained from information regarding one of the 3D maps or a total of values obtained from information regarding a plurality of the 3D maps as the priority.
3. The information processing apparatus according to claim 2, wherein the information regarding the 3D map is the number of times a position and orientation estimation request using the 3D map that is the target of the update request has been made after the update request has been made.
The information processing apparatus according to claim 2, wherein the information regarding the 3D map is the number of times the 3D map has been used for the position and orientation estimation in the past.
The information processing apparatus according to claim 2, wherein the information regarding the 3D map is a ratio of information that can be used for updating the 3D map, out of the sensing data.
The information processing apparatus according to claim 2, wherein the information regarding the 3D map is a degree of similarity between the sensing data and the 3D map.
3. The information processing apparatus according to claim 2, wherein the information regarding the 3D map is the number of past update requests for the 3D map.
The information processing apparatus according to claim 2, wherein the information regarding the 3D map is a priority set by a person involved in an application that uses the result of position and orientation estimation using the 3D map.
The information processing apparatus according to claim 2, wherein the information regarding the 3D map is a priority set by a user of an application that uses a result of position and orientation estimation using the 3D map.
The information processing apparatus according to claim 1, wherein the priority setting unit identifies the 3D map that is the target of an update request from among the plurality of 3D maps based on the sensing data.
The information processing apparatus according to claim 1, further comprising a computer resource allocation unit that allocates computer resources to be used for updating the 3D map for each of the 3D maps to be updated based on the priority.
The information processing apparatus according to claim 12, wherein the computer resource allocation unit allocates the computer resources to each of the 3D maps by proportionally allocating the computer resources based on the priority.
The information processing device according to claim 1, wherein the sensing data is an image of real space captured by a camera.
The information processing apparatus according to claim 1, further comprising a 3D map selection unit that selects a 3D map to be used for estimating the position and orientation of the device based on sensing data.
The information processing apparatus according to claim 15, further comprising a position and orientation estimation unit that estimates the position and orientation of the device using the sensing data and the selected 3D map.
The information processing apparatus according to claim 15, wherein the device is an AR device.
Set the priority when updating each of the multiple 3D maps that are the subject of the update request,
A program that causes a computer to execute an information processing method for updating the 3D map that is a target of an update request using sensing data based on the priority.
A client that sends sensing data acquired by the sensor unit and requests an update,
a priority setting unit that sets a priority when updating each of the plurality of 3D maps that are the targets of the update request;
an information processing device comprising: a 3D map update unit that updates the 3D map that is the target of the update request using the sensing data based on the priority;
An information processing system consisting of