JP2022109524A - Information processing system and information processing method - Google Patents

Abstract

To display the same augmented reality image at equal timing on a plurality of information terminals.SOLUTION: A first information terminal belonging to a predetermined space includes the steps of: creating display coordinate data indicating coordinates in space when a first AR image is displayed on a display section in accordance with the reception of an operation for superimposing the first AR image on a first space image to display it on the display section; transmitting the display coordinate data to an information processor with the first information terminal; registering the display coordinate data received from the first information terminal in an in-memory database with the information processor; transmitting the display coordinate data to a second information terminal with the information processor in accordance with the registration of the display coordinate data in the in-memory database; receiving the display coordinate data transmitted by the information processor with the second information terminal; and displaying a second AR image at a position in the display section corresponding to the coordinates in the space indicated by the display coordinate data with the second information terminal.SELECTED DRAWING: Figure 2

Description

The present invention relates to an information processing system and information processing method for displaying an augmented reality image.

Conventionally, an augmented reality image to be provided to an information terminal is specified based on the feature points of the subject included in the captured image generated by capturing the subject by the information terminal, and the augmented reality image is superimposed on the captured image and displayed on the display unit. A system for displaying is known (see Patent Document 1, for example).

JP 2019-175384 A

There is a need to display the same augmented reality image on multiple information terminals at the same timing so that multiple users of multiple information terminals in the same space can view the same augmented reality image. For example, when a plurality of users play a game while capturing images of a subject in the same space, the information terminals used by each user may display an augmented reality image of a common character. In the conventional technology, an augmented reality image is displayed only on the information terminal that specifies the feature points of the subject, and it is difficult to display the same augmented reality image at the same timing on a plurality of information terminals.

Accordingly, the present invention has been made in view of these points, and an object of the present invention is to enable a plurality of information terminals to display the same augmented reality image at the same timing.

An information processing system according to a first aspect of the present invention includes: one or more information terminal groups including a plurality of information terminals that display an augmented reality image in a space of the same coordinate system; a first information terminal among the plurality of information terminals displays a first augmented reality image on a display unit superimposed on an image of a space viewed by a user of the first information terminal. and a first display processing unit that displays the first augmented reality image on the display unit in response to receiving an operation for displaying the first augmented reality image on the display unit. and a first transmission processing unit configured to transmit the display coordinate data to the information processing device, wherein the information processing device receives the first information a coordinate receiving unit for receiving the display coordinate data transmitted by the terminal and registering the received display coordinate data in an in-memory database; and a coordinate transmission unit configured to transmit the display coordinate data to a second information terminal different from the first information terminal among the information terminals, wherein the second information terminal transmits the display transmitted by the coordinate transmission unit a second reception processing unit that receives the coordinate data, and an image of the space viewed by the user of the second information terminal at a position on the display unit that corresponds to the coordinates in the space indicated by the display coordinate data, and a second display processing unit that displays a second augmented reality image.

The information processing device has a plurality of in-memory databases corresponding to each of the plurality of spaces, and the coordinate receiving unit corresponds to the space including the first information terminal that transmitted the display coordinate data. The display coordinate data may be registered in the in-memory database.

The information processing device has a plurality of in-memory databases corresponding to each of the plurality of spaces, and the coordinate transmission unit is registered in the in-memory database corresponding to the space containing the second information terminal. The display coordinate data may be transmitted to the second information terminal.

The information processing device further has a storage unit that stores a plurality of terminal identification information for identifying the plurality of information terminals included in the space in association with each of the plurality of spaces, and the coordinate transmission unit includes: , by referring to the storage unit, the second information terminal having second terminal identification information associated with the space corresponding to the first terminal identification information of the first information terminal that transmitted the display coordinate data; The display coordinate data may be transmitted.

The first operation receiving unit creates operation content data indicating a type of the first augmented reality image based on the received operation, and the first transmission processing unit transmits the operation content data to the information processing device. and the coordinate transmission unit, in response to registration of the operation content data in the in-memory database, data corresponding to the second augmented reality image corresponding to the operation content data, or the second augmented reality image Data for identifying the real image may be transmitted to the second information terminal.

The first information terminal stores a plurality of spatial data indicating feature points of a subject included in a plurality of images previously created by imaging the space from a plurality of positions and orientations in the space. and a terminal storage unit for storing in association with the orientation, and characteristics of a subject included in a captured image created by capturing the space by the first information terminal among the plurality of space data stored in the terminal storage unit. a detailed position specifying unit that specifies a position and orientation corresponding to spatial data that is most similar to the feature point cloud data indicating the point, wherein the first operation reception unit specifies the position specified by the detailed position specifying unit; And the display coordinate data may be created based on the relationship between the direction and the position of the accepted operation.

The first operation reception unit performs the first operation when displaying the first augmented reality image on the display unit based on the relationship between the position and orientation identified by the detailed position identification unit and the position of the received operation. display orientation data indicating the orientation of one augmented reality image, the first transmission processing unit further transmits the display orientation data to the information processing device, and the coordinate transmission unit transmits the display orientation data to the in-memory database; The display orientation data may be transmitted to the second information terminal in response to registration of the orientation data.

The first information terminal acquires the plurality of spatial data from the information processing device in response to the first operation accepting unit accepting an operation for acquiring the plurality of spatial data from the information processing device. and a spatial data acquiring unit that stores the plurality of acquired spatial data in the terminal storage unit.

The first information terminal acquires from the information processing device the plurality of spatial data corresponding to the positions detected by the position detection unit that detects the position of the first information terminal, and the acquired It may further include a spatial data acquisition unit that stores a plurality of pieces of spatial data in the terminal storage unit.

The information processing device further includes a device storage unit that stores the first information terminal and the second information terminal belonging to a space of the same coordinate system as the plurality of information terminals that display an augmented reality image in the space. may have.

In the information processing method according to the second aspect of the present invention, a first information terminal belonging to a predetermined space superimposes a first augmented reality image on an image of a space visually recognized by a user of the first information terminal and displays the first information. creating display coordinate data indicating coordinates in the space when the first augmented reality image is displayed on the display unit in response to receiving an operation for displaying it on the display unit of the terminal; a step in which a first information terminal transmits the display coordinate data to an information processing device; a step in which the information processing device registers the display coordinate data received from the first information terminal in an in-memory database; a step of transmitting the display coordinate data to a second information terminal belonging to the space different from the first information terminal in response to the registration of the display coordinate data in the in-memory database; the terminal receiving the display coordinate data transmitted by the information processing device; and displaying a second augmented reality image corresponding to the first augmented reality image at a position on a display unit of the second information terminal.

ADVANTAGE OF THE INVENTION According to this invention, it is effective in being able to display the same augmented-reality image at the same timing on several information terminals.

It is a figure which shows the outline|summary of an information processing system. It is a figure for demonstrating the basic operation|movement of an information processing system. FIG. 3 is a diagram for explaining screens displayed on a plurality of information terminals; FIG. It is a figure which shows the feature point group data corresponding to the to-be-photographed object in 1st space. FIG. 4 is a diagram showing an outline of processing by which an information terminal acquires spatial data corresponding to a space to which the information terminal belongs; FIG. 4 is a diagram showing an outline of processing for updating spatial data; It is a figure which shows the structure of an information terminal. It is a figure which shows the structure of an information processing apparatus. 3 is a diagram showing a configuration of computers included in a computer group; FIG. FIG. 4 is a sequence diagram showing a flow until a plurality of information terminals belonging to the same space acquire spatial data; FIG. 4 is a sequence diagram showing a flow until a plurality of information terminals belonging to different spaces acquire spatial data; FIG. 10 is a sequence diagram showing a flow of processing for displaying AR images by a plurality of information terminals belonging to the same space; FIG. 10 is a sequence diagram showing a flow of processing in which an information terminal registers feature point group data as spatial data;

[Overview of information processing system S]
(System configuration)
FIG. 1 is a diagram showing an outline of an information processing system S. As shown in FIG. The information processing system S is a system for displaying an augmented reality image (hereinafter referred to as an “AR image”) on a plurality of information terminals 1 . An information processing system S includes a plurality of information terminals 1 , an information processing device 2 , a computer group 3 and a storage device 4 .

The information terminal 1 is a terminal having an imaging function, and is, for example, a smart phone, a tablet, or a glasses-type terminal. A user using the information terminal 1 can move to any place. In FIG. 1, a plurality of information terminals 1a (1a-1 to 1a-m, m is a natural number) exist in a first space, and a plurality of information terminals 1b (1b-1 to 1b-n, n is a natural number) It shows the state existing in the second space. The first space and the second space are, for example, one room partitioned by a wall, one floor in a building, one building, or one town. It corresponds to an area set to display the same AR image. A plurality of information terminals 1 displaying the same AR image in the space of the same coordinate system constitute an information terminal group belonging to the space.

The information processing device 2 is a server that provides various data to the information terminal 1 . The information processing device 2 provides, for example, spatial data used by the information terminal 1 to identify its own position and orientation, and AR image data to be displayed on the information terminal 1 . The spatial data includes a plurality of pre-created registered feature point group data composed of feature point groups representing one or more features of an object that can be seen when viewing the space at a plurality of positions in the space. In the spatial data, a plurality of pieces of registered feature point group data are associated with coordinate data indicating the visually recognized position and direction in the space. Further, pixel values (R value, G value, B value) indicating colors and coordinates indicating positions in space may be associated with a plurality of feature points included in registered feature point group data.

The computer group 3 executes matching processing for identifying the space to which the information terminal 1 belongs. The computer group 3 acquires the captured image data generated by the information terminal 1 or the feature point group data indicating the feature points of the subject included in the captured image data, and acquires the acquired captured image data or the acquired feature point group. The space to which the information terminal 1 belongs is specified by matching the data with a plurality of spatial data composed of a plurality of registered feature point group data corresponding to each of the plurality of spaces. A plurality of feature points included in the feature point group data may be associated with pixel values (R value, G value, B value) indicating colors.

Computer group 3 selects spatial data including a plurality of registered feature point cloud data corresponding to the space to which information terminal 1 belongs, and provides information terminal 1 with the selected spatial data. Although the details will be described later, the computer group 3 has a plurality of computers 30 that store spatial data corresponding to each of a plurality of spaces. A plurality of computers 30 execute matching processing in parallel.

The storage device 4 is a device that stores spatial data corresponding to various spaces. The storage device 4 stores, for example, spatial data corresponding to a large number of spaces all over the world. provide to

(basic action)
FIG. 2 is a diagram for explaining the basic operation of the information processing system S. As shown in FIG. In FIG. 2, in response to the operation of the user U1 using the information terminal 1a-1 existing in the first space, the user U1 of the information terminal 1a-1 and the user of the information terminal 1a-2 existing in the same first space It shows the flow of operation in which the same AR image is displayed superimposed on the image of the space that U2 is visually recognizing. Since the information terminals 1a-1 and 1a-2 face different directions at different positions in the first space, the same AR image is displayed in different directions at different positions on the respective screens being displayed. be.

The image of the space viewed by the user U1 and the user U2 is, for example, a captured image generated by the information terminal 1 capturing the space where the user U1 and the user U2 are present. When the information terminal 1 is a spectacles-type terminal, the image of the space viewed by the user U1 and the user U2 is generated by the light incident on the eyes of the user U1 and the user U2 through the transparent lenses of the spectacles-type terminal. It may be an image seen by the eyes of U1 and user U2. In the following description, the case where the images of the space viewed by the user U1 and the user U2 are mainly captured images will be described as an example. , the following configuration and processing are applied even when the image is seen by the user U1 and the user U2 through the lens.

The information terminal 1 acquires, from the information processing device 2, spatial data indicating positions of feature points of a subject visible to the user U at a plurality of positions in the space where the user U exists, and at least the user U , stores spatial data corresponding to a given space. A feature point group is a group of points that indicate the characteristics of the shape of a subject, and is composed of a plurality of points on the outline of the subject that are included in captured image data generated by photographing the space. . In spatial data, a combination of position and orientation in a three-dimensional space defined by latitude, longitude, altitude, and direction is associated with feature point group data.

FIG. 3 is a diagram for explaining screens displayed on a plurality of information terminals 1. FIG. FIG. 3(a) is a view showing the space photographed by the information terminals 1a-1 and 1a-2 from above. The horizontal direction in FIG. 2A is the X direction, the vertical direction is the Y direction, and the depth direction is the Z direction.

In FIG. 3, it is assumed that the user U1 using the information terminal 1a-1 and the user U2 using the information terminal 1a-2 are in the positions shown in FIG. 3(a). Subjects T1 to T4 in FIG. 3 are subjects present in the first space to which the information terminals 1a-1 and 1a-2 belong. Image A (person), image B (bird), and image C (arrow) in FIG. 3 are AR images displayed on the information terminals 1a-1 and 1a-2.

FIG. 3(b) shows an example of a screen displayed on the information terminal 1a-1. FIG. 3B shows a state in which the user U1 performs an operation of shooting an image C (arrow) from the image A (person) toward the image B (bird) with the hand H of the user U1. FIG. 3(c) shows an image displayed on the information terminal 1a-2 when the operation shown in FIG. 3(b) is performed on the information terminal 1a-1. In FIG. 3(c), the image C is displayed in a different orientation at a position different from that in FIG. 3(b). By displaying such images on the information terminal 1a-1 and the information terminal 1a-2, the user U1 and the user U2 can superimpose the subject seen at different positions in the same space on the screen of the information terminal 1 used by each user. , the same AR image can be viewed from different orientations.

The flow of processing for displaying the image shown in FIG. 3 by the information terminals 1a-1 and 1a-2 will be described below with reference to FIG. The information terminal 1 creates feature point cloud data based on captured image data generated by a built-in camera by executing application software such as a game or tourist guide having a function of displaying an AR image.

FIG. 4 is a diagram showing feature point group data corresponding to the subjects T1 to T4 in the first space shown in FIG. FIG. 4(a) shows feature point cloud data of subjects T1 to T4 photographed by the information terminal 1a-1, and FIG. feature point cloud data. As described above, the positions of the plurality of feature points included in the feature point group data differ depending on the position and orientation of the image taken by the information terminal 1 .

The information terminal 1 compares the created feature point group data with a plurality of registered feature point group data included in pre-stored spatial data, and compares the created feature point group data among the plurality of registered feature point group data. Identify registered feature point cloud data that is most similar to the data. The information terminal 1 identifies the position and orientation of the information terminal 1 by identifying the position and orientation corresponding to the identified registered feature point cloud data.

When the information terminal 1a-1 receives a user's operation that serves as a trigger for displaying an AR image, as shown in FIG. is notified to the information processing device 2 . The information terminal 1a-1 acquires the AR image corresponding to the position and orientation from the information processing device 2, and displays the acquired AR image as shown in FIG. 3B, for example.

The information processing device 2 has a plurality of memory databases corresponding to a plurality of spaces, and the position coordinate data notified from the information terminal 1a-1 is stored in the memory database corresponding to the space to which the information terminal 1a-1 belongs. In response to the writing, the AR image and the display coordinate data indicating the position to display the AR image and the direction to move the AR image to the information terminal 1a-2, which is another information terminal 1 belonging to the same space. , send. The information terminal 1a-2 displays an AR image corresponding to the position and orientation indicated by the received display coordinate data, for example, as shown in FIG. 3(c).

As in the present embodiment, a system that displays AR images requires a high-speed response. An accessible in-memory database is desirable. In the following description, a case where the database is an in-memory database is exemplified.

(Provision of spatial data)
FIG. 5 is a diagram showing an overview of the process by which the information terminal 1 acquires spatial data corresponding to the space to which the information terminal 1 belongs. The information terminal 1 acquires spatial data triggered by activation of predetermined application software, login of a user U to a predetermined site, or movement of the information terminal 1 to a predetermined position.

For example, upon detecting that predetermined application software has been activated, the information terminal 1 creates feature point group data based on the captured image data generated by the built-in camera. The information terminal 1 transmits activation information indicating that the application software has been activated and the created feature point group data to the information processing device 2 . The information processing device 2 transmits the received feature point group data to the computer group 3 in response to receiving the activation information.

The computer group 3 inputs the received feature point group data to the plurality of computers 30, and the plurality of computers 30 execute matching processing. Among the plurality of computers 30, the computer 30 that detects that the pattern of the feature point group indicated by the received feature point group data matches the pattern of the registered feature point group indicated by the spatial data stored in itself is , notifies the information processing device 2 of the match, and transmits the spatial data stored in itself to the information processing device 2 . The information processing device 2 transmits the received spatial data to the information terminal 1 .

The information terminal 1 may transmit the approximate position data indicating the approximate position detected by the GPS receiver to the information processing device 2 together with the activation information. In this case, the information processing device 2 transmits the feature point group data to the computer 30 corresponding to the space including the position indicated by the received approximate position data, thereby reducing the number of computers 30 executing the matching process. Therefore, the load on the computer group 3 is reduced.

(Update spatial data)
Objects such as buildings, articles, and plants in space change over time. Therefore, the spatial data stored in the storage device 4 does not always correspond to the latest spatial state. Also, there may be a space in which no spatial data has been created. However, it is not easy to collect spatial data every time the spatial conditions change. Therefore, the information processing system S uses a large amount of feature point group data created by the information terminal 1 to update the spatial data to the latest state.

FIG. 6 is a diagram showing an outline of processing for updating spatial data. The information terminal 1 includes registered feature point cloud data that matches the feature point cloud data created by itself, among a plurality of registered feature point cloud data included in the spatial data acquired via the information processing device 2. If not, it is determined that there is an error in the spatial data, or that the spatial data of the space to which the information terminal 1 belongs does not exist. In this case, the information terminal 1 transmits to the information processing apparatus 2 an update request for updating the registered feature point cloud data together with the feature point cloud data created by itself and the position data indicating its own position. Position data is, for example, data indicating latitude, longitude, altitude and direction.

It is desirable that the position data be as accurate as possible. Therefore, for example, the information terminal 1 issues an update request based on the moving distance and moving direction from the position where the registered feature point cloud data included in the space data immediately before and the feature point cloud data created by itself match. A position corresponding to the feature point cloud data transmitted together with is calculated. The information terminal 1 issues an update request on the condition that the distance from the position where the registered feature point cloud data included in the spatial data immediately before and the feature point cloud data created by itself match is within a predetermined threshold. may be sent. The predetermined threshold is determined based on at least one of the accuracy of the position indicated by the spatial data and the accuracy with which the information terminal 1 can specify the moving distance and moving direction.

Upon receiving the update request, the information processing device 2 transmits the received feature point group data and position data to the storage device 4 . When receiving feature point group data corresponding to the same position and orientation from a plurality of information terminals 1, the information processing apparatus 2 transmits feature point group data that is considered to be more accurate. The feature point cloud data that is considered to be more accurate is the feature point cloud data with a larger number of feature points included in the feature point cloud data, or the information terminal with a higher accuracy sensor used to identify the position. 1, and the like.

When the storage device 4 receives the feature point group data and the position data from the information terminal 1, the storage device 4 stores the feature point group data as spatial data in association with the position and orientation (that is, latitude, longitude, altitude and azimuth) indicated by the received position data. stored as registered feature point cloud data included in . By updating the registered feature point group data included in the spatial data by the storage device 4 in this manner, the information terminal 1 can specify its own position and orientation using the spatial data corresponding to the latest space. AR images can be displayed at appropriate positions.

[Structure of each part]
(Configuration of information terminal)
FIG. 7 is a diagram showing the configuration of the information terminal 1. As shown in FIG. Information terminal 1 includes communication unit 11 , operation unit 12 , position detection unit 13 , imaging unit 14 , display unit 15 , storage unit 16 , and control unit 17 . The control unit 17 includes a transmission processing unit 171, a reception processing unit 172, an operation reception unit 173, a general position specifying unit 174, an image processing unit 175, a detailed position specifying unit 176, a display processing unit 177, a spatial and a data acquisition unit 178 .

The communication unit 11 has a communication interface for connecting to a communication network. The communication unit 11 transmits and receives various data to and from the information processing device 2 via a mobile wireless communication network or a wireless communication network such as Wi-Fi (registered trademark). The communication unit 11 transmits data input from the transmission processing unit 171 and inputs data received via the communication network to the reception processing unit 172 .

The operation unit 12 has a device for detecting user U's operation of the information terminal 1 . The operation unit 12 has, for example, a touch panel, a trackball, or a mouse. When the operation unit 12 is a touch panel, the operation unit 12 is provided over the display unit 15 . The operation unit 12 notifies the operation reception unit 173 of operation data indicating at least one of the position where the operation was performed and the amount of operation.

The position detection unit 13 has a device for detecting the approximate position of the information terminal 1 . The position detection unit 13 has, for example, a GPS (Global Positioning System) receiver, and notifies the approximate position specifying unit 174 of position data indicating latitude and longitude detected based on radio waves received from GPS satellites. The position detection unit 13 has at least one of an altimeter, a compass, and a gyro sensor, and notifies the approximate position specifying unit 174 of the altitude of the information terminal 1 and the direction in which the information terminal 1 is facing as position data. good too.

The image capturing unit 14 is a device, such as a camera, that captures a subject around the information terminal 1 and generates captured image data. The imaging unit 14 generates captured image data by capturing an image of at least a partial area of the space to which the information terminal 1 belongs. The imaging unit 14 may continuously generate captured image data at predetermined time intervals (frame rate). Also, the imaging unit 14 may be a stereo camera that has a plurality of imaging elements and generates distance data indicating the distance to the subject. In this case, the feature point group data may include information indicating the distance from the information terminal 1 to the subject corresponding to each feature point.

The display unit 15 is a device for displaying various information, such as a display. The display unit 15 displays, for example, a screen for the user U of the information terminal 1 to operate, or displays an image based on captured image data generated by the imaging unit 14 . In addition, the display unit 15 displays an AR image provided from the information processing device 2 superimposed on an image of the space visually recognized by the user U (for example, a captured image).

The storage unit 16 is a terminal storage unit having storage media such as ROM (Read Only Memory) and RAM (Random Access Memory). The storage unit 16 stores programs executed by the control unit 17 . The storage unit 16 also temporarily stores image data displayed by the display unit 15, such as captured image data and AR image data. The storage unit 16 stores a plurality of feature point group data representing feature point groups of a subject included in a plurality of images created by imaging a space from a plurality of positions and orientations in the space. Store in association.

The control unit 17 has, for example, a CPU (Central Processing Unit). By executing the programs stored in the storage unit 16, the control unit 17 controls a transmission processing unit 171, a reception processing unit 172, an operation reception unit 173, a general position specifying unit 174, an image processing unit 175, and a detailed position specifying unit 176. , a display processing unit 177 and a spatial data acquisition unit 178 .

The transmission processing unit 171 transmits various data to the information processing device 2 or other external devices via the communication unit 11 . The transmission processing unit 171 , for example, transmits operation data received from the operation reception unit 173 to the information processing device 2 . The operation data includes operation content data indicating the content of the operation, display coordinate data indicating the position of the operation (that is, the position where the AR image is displayed), and display orientation data indicating the orientation in which the AR image is displayed. ing. The operation data may further include speed data indicating the operation speed of the user U corresponding to the speed of moving the AR image.

When the user U performs an operation for displaying an AR image on the display unit 15, the transmission processing unit 171 transmits display coordinates indicating the coordinates in the space when the AR image is displayed on the display unit 15 together with the operation content data. data, display orientation data, and speed data are transmitted to the information processing device 2; The transmission processing unit 171 also transmits the approximate position data received from the approximate position specifying unit 174 to the information processing device 2 .

In order to transmit and receive data asynchronously with the in-memory database of the information processing device 2 in real time, the transmission processing unit 171 transmits operation data using Pub/Sub messages for accessing the in-memory database. good.

Further, the transmission processing unit 171 acquires feature point cloud data composed of a plurality of feature points of the subject included in the captured image data from the image processing unit 175 and transmits the acquired feature point cloud data to the information processing device 2 . do. The transmission processing unit 171 may transmit the captured image data received from the image processing unit 175 to the information processing device 2 instead of the feature point group data or together with the feature point group data. The transmission processing unit 171 transmits the Pub message to, for example, an in-memory database corresponding to the space to which the approximate position specified by the approximate position specifying unit 174 belongs, among the plurality of in-memory databases stored in the information processing device 2 . By doing so, the feature point group data is written.

Further, when there is no registered feature point cloud data whose difference from the feature point cloud data is equal to or less than a threshold among the plurality of pieces of registered feature point cloud data stored in the storage unit 16, the transmission processing unit 171 The feature point group data acquired from the image processing unit 175 is transmitted to the information processing device 2 together with an update request for updating the data.

The reception processing unit 172 receives various data from the information processing device 2 or other external devices via the communication unit 11 . The reception processing unit 172 receives display coordinate data transmitted to the information processing device 2 by another information terminal 1 belonging to the same space, for example.

When the information terminal 1a-1 transmits the display coordinate data to the information processing device 2 in response to the user U1 using the information terminal 1a-1 performing an operation to display an AR image, the information terminal 1a-2 receives The processing unit 172 receives display coordinate data transmitted by a coordinate transmission unit 232 included in the information processing device 2, which will be described later. The reception processing unit 172 inputs the received display coordinate data to the detailed position specifying unit 176 . The reception processing unit 172 also receives spatial data corresponding to the space to which the information terminal 1 belongs from the information processing device 2 . The reception processing unit 172 inputs the received spatial data to the spatial data acquiring unit 178 .

The operation reception unit 173 identifies the content of the operation performed by the user U based on the operation data notified from the operation unit 12 . The operation accepting unit 173 creates operation content data indicating the type of AR image to be displayed based on the accepted operation. The operation reception unit 173 notifies the information processing device 2 of the operation content by inputting the created operation content data to the transmission processing unit 171 . For example, the operation reception unit 173 receives an operation for displaying the AR image on the display unit 15 while superimposing it on an image of the space (for example, a captured image). Create display coordinate data that indicates coordinates in space.

The operation reception unit 173 acquires detailed position data indicating the detailed position specified by the detailed position specifying unit 176, and determines the position and orientation of the information terminal 1 indicated by the acquired detailed position data and the operation performed by the user U on the operation unit 12. The display coordinate data may be created based on the position. That is, the operation reception unit 173 may create the display coordinate data based on the relationship between the position and orientation identified by the detailed position identification unit 176 and the position of the operation received by the operation reception unit 173 .

In the case of the example shown in FIG. 3B, the operation reception unit 173 creates display coordinate data including the X and Z coordinates of the position where the hand H touches the operation unit 12 . The operation reception unit 173 creates display coordinate data at predetermined time intervals when the position of the hand H moves. The operation accepting unit 173 may create display coordinate data including speed data indicating the speed at which the hand H moves based on display coordinate data created at a plurality of times. When the Y coordinate of the subject touched by the hand H in FIG. Display coordinate data including the Y coordinate may be created. The operation reception unit 173 inputs the created display coordinate data to the transmission processing unit 171 .

When the operation reception unit 173 causes the display unit 15 to display the AR image based on the relationship between the position and orientation of the information terminal 1 identified by the detailed position identification unit 176 and the position of the operation received by the operation reception unit 173. Display orientation data indicating the orientation of the AR image may be created. The operation accepting portion 173 inputs the display orientation data to the transmission processing portion 171 .

The approximate position specifying unit 174 generates approximate position data indicating the approximate position of the information terminal 1 based on the received radio waves. The general position identifying section 174 identifies the general position of the information terminal 1 based on the position data input from the position detecting section 13 . The rough position data is data indicating a position with accuracy corresponding to the accuracy of GPS, and is position data with an error of 10 m or more, for example.

The approximate position specifying unit 174 notifies the information processing device 2 of the approximate position of the information terminal 1 by inputting the specified position data to the transmission processing unit 171 . The approximate position specifying unit 174 may periodically notify the information processing device 2 of the approximate position data, and when the user U performs a predetermined operation or when it becomes necessary to acquire registered feature point cloud data. Alternatively, the approximate position data may be notified to the information processing device 2 .

The image processing unit 175 creates feature point group data composed of a plurality of feature points of the subject included in the captured image data. Specifically, the image processing unit 175 first identifies pixels in the captured image data in which the amount of change in pixel value from adjacent pixels is equal to or greater than a threshold value, thereby determining the outline of the subject and the contour of the uneven portion of the subject. Identify lines. The image processing unit 175 creates feature point group data composed of a plurality of feature points on the identified contour line.

As in the example shown in FIG. 4, the image processing unit 175 may create feature point group data by specifying points at which the direction of the outline changes as feature points. The image processing unit 175 creates feature point group data each time new captured image data is input from the imaging unit 14 . The image processing unit 175 inputs the created feature point group data to the detailed position specifying unit 176 . The image processing unit 175 inputs feature point group data to the transmission processing unit 171 when it is necessary to acquire spatial data from the information processing device 2 (that is, when it is necessary to specify the space to which the information terminal 1 belongs). .

Further, when the image processing unit 175 receives a notification that the detailed position specifying unit 176 cannot specify the position and orientation of the information terminal 1 based on the registered feature point cloud data acquired from the information processing device 2, In order to register the created feature point cloud data in the storage device 4, the feature point cloud data is input to the transmission processing unit 171, and an update request for updating the registered feature point cloud data is sent to the information processing device 2. The transmission processing unit 171 is instructed.

The detailed position specifying unit 176 determines the position and position corresponding to the registered feature point group data most similar to the feature point group data input from the image processing unit 175 among the plurality of registered feature point group data received from the information processing device 2 . Identify orientation. The detailed position specifying unit 176 selects the registered feature point group data that is most similar to the feature point group data representing the feature point group of the subject included in the captured image data, among the plurality of registered feature point group data stored in the storage unit 16 . Identify the position and orientation corresponding to .

Specifically, first, the detailed position specifying unit 176 first uses the feature point group data input from the image processing unit 175 and a plurality of points included in the spatial data acquired from the information processing device 2 and stored in the storage unit 16. Compare with registered feature point cloud data. The detailed position specifying unit 176 calculates the correlation value between the feature point group data input from the image processing unit 175 and each of the plurality of registered feature point group data included in the spatial data, and the calculated correlation value is the maximum. Identify the registered feature point cloud data that becomes

The detailed position specifying unit 176 specifies the position and orientation of the information terminal 1 based on coordinate data (for example, data indicating latitude, longitude, altitude, and direction) associated with the specified registered feature point group data. In this way, the detailed position specifying section 176 can specify the position and orientation of the information terminal 1 with higher accuracy than the general position specifying section 174, for example. The detailed position specifying unit 176 specifies the position and orientation of the information terminal 1 with an accuracy of 1 mm, for example. The detailed position specifying unit 176 notifies the reception processing unit 172 and the display processing unit 177 of detailed position data indicating the specified position and orientation.

The display processing unit 177 causes the display unit 15 to display various information or images. For example, the display processing unit 177 causes the display unit 15 to display a captured image based on captured image data generated by capturing an image of the space to which the information terminal 1 belongs. Also, the display processing unit 177 determines the position and orientation of the AR image to be displayed based on the display coordinate data. The display processing unit 177 displays an image of the space viewed by the user U (for example, the space to which the information terminal 1 belongs, captured by the imaging unit 14, at a position on the display unit 15 corresponding to the coordinates in the space indicated by the display coordinate data. The captured image based on the captured image data generated by the AR image is superimposed on the AR image.

When the user U performs an operation to display an AR image, the display processing unit 177 performs the following operations based on the detailed position data input from the detailed position specifying unit 176 and the operation position data notified from the operation receiving unit 173. Create display coordinate data indicating the position and orientation of the AR image to be displayed. When a user U of another information terminal 1 belonging to the same space as the information terminal 1 performs an operation to display an AR image, the display processing unit 177 generates display coordinate data used for displaying an AR image corresponding to the operation. is obtained from the information processing device 2 .

As shown in FIG. 2, when the information terminal 1a-1 and the information terminal 1a-2 exist in the first space, the display processing unit 177 of the information terminal 1a-1 receives the display coordinates notified from the operation reception unit 173. Display the AR image at a position and orientation based on the data. On the other hand, the display processing unit 177 of the information terminal 1a-2 displays the AR image at the position and orientation based on the display coordinate data notified from the information processing device 2. FIG. As the display processing unit 177 of each of the information terminals 1a-1 and 1a-2 operates in this manner, the user U1 using the information terminal 1a-1 and the user U2 using the information terminal 1a-2 can display the information Depending on the position and orientation of the terminal 1, AR images can be viewed simultaneously at different positions and in different directions.

The spatial data acquisition unit 178 acquires spatial data including registered feature point cloud data from the information processing device 2 via the communication unit 11 and the reception processing unit 172 . The spatial data acquisition unit 178 allows the operation reception unit 173 to perform an operation for acquiring a plurality of pieces of registered feature point cloud data from the information processing apparatus (for example, an operation to start predetermined application software, an operation to log in to a predetermined site). In response to the reception, spatial data including a plurality of registered feature point group data corresponding to the space to which the information terminal 1 belongs is acquired from the information processing device, and the acquired spatial data is stored in 16 .

The spatial data acquisition unit 178 notifies the detailed position specifying unit 176 of the acquired plurality of registered feature point group data. For example, the spatial data acquisition unit 178 notifies the detailed position specifying unit 176 of the plurality of registered feature point group data via the storage unit 16 by storing the acquired plurality of registered feature point group data in the storage unit 16. do.

(Configuration of information processing device 2)
FIG. 8 is a diagram showing the configuration of the information processing device 2. As shown in FIG. The information processing device 2 has a communication section 21 , a storage section 22 and a control section 23 . The control unit 23 includes a coordinate receiving unit 231, a coordinate transmitting unit 232, a terminal data receiving unit 233, a feature point group transmitting unit 234, an information receiving unit 235, a spatial data transmitting unit 236, and a spatial data managing unit 237. , and a number management unit 238 .

The communication unit 21 has a communication interface for connecting to a communication network. The communication unit 21 transmits and receives various data to and from the information terminal 1 , the computer group 3 and the storage device 4 .

The storage unit 22 is a device storage unit having ROM, RAM and SSD (Solid State Drive). The storage unit 22 stores programs executed by the control unit 23 . The storage unit 22 also stores a first space database 221 and a second space database 222 as multiple in-memory databases corresponding to multiple spaces. The storage unit 22 stores, for example, a plurality of in-memory databases (for example, the first space database 221 and the second 2 space database 222) is stored.

As described above, the in-memory database is a database built on the main memory, and is a database that can be accessed at a higher speed than databases stored in a disk storage mechanism. In FIG. 8, the storage unit 22 exemplifies the first space database 221 and the second space database 222, but the storage unit 22 stores databases corresponding to more spaces.

The storage unit 22 stores the first information terminal and the second information terminal belonging to the space of the same coordinate system as the plurality of information terminals 1 that display AR images in the same space. The first information terminal is the information terminal 1 operated by the user U to display the AR image, and is the information terminal 1a-1 in the example shown in FIG. The second information terminal is another information terminal 1 belonging to the same space as the first information terminal, and is the information terminal 1a-2 in the example shown in FIG. The storage unit 22 stores a plurality of pieces of terminal identification information for identifying a plurality of information terminals 1 included in each space in association with each of the plurality of spaces.

Specifically, in the in-memory database stored in the storage unit 22, information for identifying the space to which each information terminal 1 belongs is associated with terminal identification information for identifying each of the plurality of information terminals 1. Spatial identification information is registered. In addition, in the in-memory database, in association with the space identification information, AR image identification information for identifying the AR image to be displayed on the information terminal 1 belonging to the space corresponding to the space identification information, and the AR image to be displayed Display coordinate data indicating the position and orientation are registered.

A plurality of in-memory databases may store the feature point group data in order to transfer the feature point group data received from the information terminal 1 belonging to the corresponding space to the computer group 3 . When the first space database 221 corresponding to the first space receives feature point group data from one or more information terminals 1 belonging to the first space, the feature point group data is registered feature point group data corresponding to the first space. The received feature point group data is transferred to the computer 30 that analyzes whether or not it matches with . The in-memory database relays feature point cloud data between the information terminal 1 and the computer group 3 based on Pub/Sub messages transmitted from the information terminal 1 or the computer group 3 without going through the control unit 23. may

The control unit 23 has, for example, a CPU. By executing the programs stored in the storage unit 22, the control unit 23 controls a coordinate receiving unit 231, a coordinate transmitting unit 232, a terminal data receiving unit 233, a feature point group transmitting unit 234, an information receiving unit 235, and a spatial data transmitting unit. It functions as a unit 236 , a spatial data management unit 237 and a number management unit 238 .

The coordinate receiving unit 231 receives the operation content data and the display coordinate data transmitted by the information terminal 1 according to the operation of the user U who uses the information terminal 1, and stores the received operation content data and the display coordinate data in the in-memory database. sign up. The coordinate receiving unit 231 registers the display coordinate data in the in-memory database corresponding to the space containing the information terminal 1 that transmitted the display coordinate data. For example, when the coordinate receiving unit 231 receives the display coordinate data from the information terminal 1a-1 belonging to the first space, the coordinate receiving unit 231 stores the display coordinate data in the database 221 for the first space.

In response to the display coordinate data being registered in the in-memory database by the coordinate receiving unit 231, the coordinate transmitting unit 232 transmits the display coordinate data to the second information terminal, which is different from the first information terminal among the plurality of information terminals 1. Send. The coordinate transmission unit 232 transmits the display coordinate data registered in the in-memory database corresponding to the space containing the second information terminal to the second information terminal.

The coordinate transmission unit 232 refers to the storage unit 22 to transmit the second information terminal having the second terminal identification information associated with the space corresponding to the first terminal identification information of the first information terminal that transmitted the display coordinate data. Send display coordinate data to The coordinate transmission unit 232 may transmit the display orientation data to the second information terminal in response to registration of the display orientation data transmitted by the first information terminal in the in-memory database. As a result, a plurality of information terminals 1 belonging to the same space share the position and orientation of displaying the AR image.

Further, in response to registration of operation content data received from the first information terminal in the in-memory database, the coordinate transmission unit 232 transmits data corresponding to the second AR image corresponding to the operation content data, or the second AR image corresponding to the operation content data. Data for identifying the 2AR image is transmitted to the second information terminal. The second AR image is an image corresponding to the first AR image that shows the same target as the first AR image, but has a different orientation or size. As a result, multiple information terminals 1 belonging to the same space share the same AR image.

The coordinate transmission unit 232 transmits the display coordinate data and the AR image stored in the first spatial database 221 to the information terminal 1a-2. When the coordinate transmission unit 232 has already transmitted the AR image to the information processing device 2, instead of transmitting the AR image, the coordinate transmission unit 232 displays one of the already transmitted AR images on the information terminal 1a-2. AR image identification information for identifying the AR image to be displayed may be transmitted.

The terminal data receiving unit 233 receives feature point group data from the information terminal 1 via the communication unit 21 . In addition, the terminal data receiving unit 233 receives approximate position data indicating the approximate position of the information terminal 1 from the information terminal 1 that has transmitted the feature point group data. The terminal data reception unit 233 notifies the feature point group transmission unit 234 of the received feature point group data and approximate position data. The terminal data reception unit 233 stores the feature point group data and the approximate position data in the storage unit 22, and notifies the feature point group data and the approximate position data to the feature point group transmission unit 234 via the storage unit 22. good.

The feature point group transmission unit 234 transmits the feature point group data received from the information terminal 1 to the computer group 3 . For example, as shown in FIG. 5, when the terminal data receiving unit 233 receives the feature point cloud data transmitted by the information terminal 1 together with the activation information, the feature point cloud transmitting unit 234 transmits a message requesting spatial data together with the feature point cloud data. , the feature point cloud data is transmitted to the computer group 3 .

When the feature point group transmission unit 234 acquires the general position data from the terminal data reception unit 233, the feature point group transmission unit 234 transmits one or more computers corresponding to the space including the general position indicated by the general position data from the plurality of computers 30 included in the computer group 3. 30 may be selected and the feature point cloud data may be transmitted to one or more selected computers 30 . That is, the feature point cloud transmission unit 234 transmits the feature point cloud data to the computer 30 associated with the space corresponding to the data to which the approximate position indicated by the approximate position data transmitted by the information terminal 1 together with the feature point cloud data belongs. good too.

The feature point group transmission unit 234 may relay the feature point group data between the information terminal 1 and the computer group 3 via in-memory databases corresponding to each of the plurality of spaces. The feature point cloud transmission unit 234 transmits the feature point cloud data to the computer 30 associated with the space corresponding to the in-memory database in which the feature point cloud data is written among the plurality of in-memory databases.

From one or more computers 30 to which the feature point cloud transmission unit 234 has transmitted the feature point cloud data, the information reception unit 235 specifies the space to which the information terminal 1 corresponding to the transmitted feature point cloud data belongs, Predetermined information indicating that the space to which the information terminal 1 belongs is specified is received. Specifically, the information receiving unit 235 receives from one computer 30 among the one or more computers 30 that have transmitted the feature point group data any one of a plurality of registered feature point group data stored in the computer 30 . Also, it receives predetermined information indicating that the correlation value with the feature point cloud data transmitted by the feature point cloud transmission unit 234 is equal to or greater than the threshold. That is, the information receiving unit 235 converts the feature point cloud data transmitted by the feature point cloud transmitting unit 234 from the computer 30 into registered feature point cloud data of a subject included in the space in which the computer 30 is in charge of matching processing. receives predetermined information indicating that it is equivalent to

The information receiving unit 235 may acquire space data including a plurality of registered feature point group data corresponding to the space in which the computer 30 is in charge of matching processing, along with predetermined information. The information receiving unit 235 inputs the acquired spatial data to the spatial data transmitting unit 236 together with the terminal identification information of the information terminal 1 that created the feature point cloud data transmitted by the feature point cloud transmitting unit 234 to the computer group 3 . Further, the information receiving unit 235 acquires space identification information for identifying the space in which the computer 30 is in charge of matching processing, and transmits the feature point cloud data transmitted by the feature point cloud transmitting unit 234 to the computer group 3. The space identification information may be stored in the storage unit 22 in association with the created terminal identification information of the information terminal 1 .

The spatial data transmission unit 236 transmits to the information terminal 1 a plurality of registered feature point group data included in the spatial data corresponding to the computer 30 that transmitted the predetermined information. The spatial data transmission unit 236 transmits a plurality of registered feature point group data to the information terminal 1 corresponding to the terminal identification information acquired from the information reception unit 235 .

The spatial data management unit 237 performs management to keep the spatial data stored in the computer group 3 or the storage device 4 up-to-date. The spatial data management unit 237 keeps the spatial data up-to-date by, for example, storing the feature point group data received from the information terminal 1 in the storage device 4 in association with the position in the space.

When the spatial data management unit 237 determines that the feature point cloud data received from the information terminal 1 is the latest feature point cloud data, the spatial data management unit 237 stores the feature point cloud data received from the information terminal 1 as registered feature point cloud data. stored in the device 4; For example, when the spatial data management unit 237 receives feature point cloud data from a plurality of information terminals 1 together with an update request requesting updating of the registered feature point cloud data, the spatial data management unit 237 receives, for example, By referring to the time stamp, it is determined that the feature point group data based on the captured image data generated later is the latest feature point group data.

When the device reception processing unit receives a plurality of feature point group data from a plurality of information terminals 1 that have transmitted detailed position data indicating the same position and orientation, the spatial data management unit 237 selects the feature point group data having a larger amount of data. may be stored in the storage device 4 as registered feature point cloud data. The data amount is, for example, the number of point groups included in the feature point group data. As the number of point clouds increases, it is considered that more detailed features of the subject belonging to the space appear in the feature point cloud data. The accuracy is improved when the information terminal 1 specifies the position and orientation using the feature point group data.

When the feature point group data associated with the position and orientation indicated by the detailed position data transmitted by the information terminal 1 together with the feature point group data is not stored in the storage device 4, the spatial data management unit 237 The received feature point group data may be stored in the storage device 4 as registered feature point group data. The storage unit 22 stores the latitude, longitude, altitude, and direction corresponding to each of the plurality of registered feature point cloud data stored in the storage device 4. The spatial data management unit 237 stores the latitude indicated by the detailed position data. , longitude, altitude and direction are not stored in the storage unit 22, the feature point group data is stored in the storage unit 4. FIG.

When registered feature point cloud data corresponding to the position and orientation indicated by the position data is stored in the storage device 4, the spatial data management unit 237 determines that the data amount of the stored registered feature point group data is larger than that of the information terminal 1. The feature point group data received from the information terminal 1 may be stored in the storage device 4 as registered feature point group data on condition that the data amount of the feature point group data received from the information terminal 1 is larger.

The storage unit 22 stores the data amount of each of the plurality of registered feature point cloud data stored in the storage device 4 together with the latitude, longitude, altitude and direction. The data amount of the feature point group data thus obtained is compared with the data amount of the registered feature point group data stored in the storage unit 22 in association with the position and orientation indicated by the detailed position data transmitted from the information terminal 1 . As a result of the comparison, if the data amount of the feature point group data received from the information terminal 1 is larger, the spatial data management unit 237 stores the feature point group data received from the information terminal 1 in the storage device 4 . By operating the spatial data management unit 237 in this way, it is possible to prevent deterioration of the registered feature point group data due to updating of the registered feature point group data.

The spatial data management unit 237 stores the feature point group data in the storage device 4 on the condition that the feature point group data received from the information terminal 1 contains data indicating an object having a pre-registered shape. may A subject with a pre-registered shape is, for example, a newly constructed building. By operating the spatial data management unit 237 in this manner, it is possible to quickly register the registered feature point cloud data when it is known in advance that the registered feature point group data does not exist.

The number management unit 238 determines the number of computers 30 that execute matching processing for specifying the space to which the information terminal 1 belongs, based on preset conditions. The number management unit 238 changes the number of computers 30 based on the amount of registered feature point group data stored in the storage device 4, for example. The feature point group transmission unit 234 transmits the feature point group data received from the plurality of information terminals 1 to the number of computers 30 determined by the number management unit 238 . By operating the number management unit 238 in this way, even if the amount of registered feature point cloud data increases with the passage of time, it is possible to suppress an increase in the time required for matching processing.

The number management unit 238 determines the time required from when the feature point group transmission unit 234 transmits the feature point group data to the plurality of computers 30 to when it acquires predetermined information indicating that the space to which the information terminal 1 belongs has been specified. (that is, the time required for matching processing), the number of computers 30 may be changed. If it takes a long time from when the feature point cloud transmission unit 234 transmits the feature point cloud data to the plurality of computers 30 to when the predetermined information is acquired, it is considered that the number of computers 30 is insufficient. . Therefore, the number management unit 238 prevents the time required for the matching process from becoming too long by increasing the number of computers 30 that execute the matching process when the time is equal to or greater than the threshold.

By the way, when the number of information terminals 1 from which the information processing device 2 receives feature point group data increases, the amount of feature point group data received by the terminal data receiving unit 233 increases. As the amount of feature point cloud data increases, the time required for matching processing increases. Therefore, the number management unit 238 may change the number of computers 30 based on the amount of feature point group data received by the terminal data reception unit 233 . The number management unit 238 may change the number of computers based on the number of information terminals 1 that transmit feature point group data.

When the amount of feature point cloud data received by the terminal data receiving unit 233 within a predetermined time reaches or exceeds a threshold value, or when the number of information terminals 1 that receive feature point cloud data within a predetermined time When the number increases, by increasing the number of computers 30 that execute the matching process, it is possible to prevent the time required for the matching process from becoming too long. The number management unit 238 may increase the number of computers 30 to execute the matching process when the number of the information terminals 1 from which the terminal data reception unit 233 receives the feature point group data exceeds the threshold.

When the information terminal 1 is moving, it is assumed that the information terminal 1 moves to a different space, and the frequency of matching processing for identifying the space to which the information terminal 1 belongs increases. Therefore, the number management unit 238 counts the number of computers 30 based on whether or not the approximate location data received from the information terminal 1 by the terminal data reception unit 233 indicates that the information terminal 1 is moving. You can change it.

When the number of moving information terminals 1 is specified to be equal to or greater than a predetermined number, the number management unit 238 increases the number of computers 30 that execute the matching process, thereby increasing the time required for the matching process. Avoid overdoing it. Conversely, when the number management unit 238 identifies that the number of moving information terminals 1 is less than the predetermined number, the number management unit 238 reduces the number of computers 30 that execute the matching process, thereby reducing the number of computers 30 to be used. can reduce costs.

The number management unit 238 identifies the moving speed of the information terminals 1 based on a plurality of pieces of approximate position data transmitted at a plurality of times, and the number of the information terminals 1 whose identified moving speed is equal to or higher than the threshold is equal to or greater than a predetermined number. In some cases, the number of computers 30 may be increased. By operating the number management unit 238 in this way, the number of computers 30 can be changed based on the number of information terminals 1 with a high probability that the space will be switched. It can be a more appropriate number.

(Configuration of computer group 3)
FIG. 9 is a diagram showing the configuration of the computer 30 included in the computer group 3. As shown in FIG. The communication unit 31 has a communication unit 31 , a storage unit 32 and a control unit 33 . The control unit 33 has a feature point group acquisition unit 331 , a registered data acquisition unit 332 and a data analysis unit 333 .

The communication unit 31 has a communication interface for connecting to a communication network. The communication unit 31 transmits and receives various data to and from the information processing device 2 and the storage device 4 .

The storage unit 32 has ROM, RAM and SSD. The storage unit 32 stores programs executed by the control unit 33 .

The control unit 33 has, for example, a CPU. The control unit 33 functions as a feature point group acquisition unit 331 , a registered data acquisition unit 332 and a data analysis unit 333 by executing programs stored in the storage unit 32 .

The feature point group acquisition unit 331 acquires feature point group data from the information processing device 2 . The feature point group acquisition unit 331 inputs the acquired feature point group data to the data analysis unit 333 .

The registered data acquisition unit 332 reads out a plurality of registered feature point cloud data corresponding to the spaces assigned to the computer 30 from the storage device 4 storing the registered feature point cloud data corresponding to a plurality of spaces. Acquire registered feature point cloud data. The registered data acquisition unit 332 inputs the acquired plurality of registered feature point cloud data to the data analysis unit 333 . The registered data acquisition unit 332 may cause the storage unit 32 to store the acquired plurality of pieces of registered feature point cloud data.

Note that the space identification information corresponding to the space assigned to the computer 30 is stored, for example, in the storage unit 32, and the registration data acquisition unit 332 acquires a plurality of data stored in the storage device 4 in association with the space identification information. Registered feature point cloud data is acquired from the storage device 4 .

The data analysis unit 333 analyzes any of the feature point cloud data acquired by the feature point cloud acquisition unit 331 and a plurality of registered feature point cloud data corresponding to the space assigned to each computer 30 acquired by the registration data acquisition unit 332 . predetermined information is transmitted to the information processing device 2 when the difference between the two is equal to or less than the threshold; The predetermined information is, as described above, that the feature point group data received by the registration data acquisition unit 332 is composed of the feature point group of the subject included in the space in which the computer 30 is in charge of matching processing. This information indicates that

[Flow of processing in information processing system S]
(Processing to acquire spatial data)
FIG. 10 is a sequence diagram showing a flow until a plurality of information terminals 1 belonging to the same space acquire spatial data. The sequence diagram shown in FIG. 10 starts when the information terminal 1 needs to acquire spatial data. When it becomes necessary to acquire the spatial data, for example, the user U activates a predetermined application software on the information terminal 1, logs in to a predetermined site, or moves a predetermined distance or more from the point where the spatial data was previously acquired. or when a predetermined period of time or more has passed since the last acquisition of spatial data.

A first information terminal and a second information terminal in FIG. 10 belong to the same first space, and are, for example, the information terminal 1a-1 and the information terminal 1a-2 shown in FIG. The first information terminal transmits first approximate position data indicating the position of the first information terminal and first feature point group data generated based on the captured image data to the information processing device 2 . The information processing device 2 transmits the first feature point cloud data to the first computer that executes matching processing using registered feature point cloud data in the first space including the approximate position indicated by the received first approximate position data. Here, the information processing device 2 may transmit the first feature point group data to a plurality of first computers.

When the first computer determines that the first feature point cloud data matches any one of the plurality of registered feature point group data, the first information terminal is in the first space corresponding to the matching registered feature data. and transmits first space data including a plurality of registered feature point group data corresponding to the first space to the information processing device 2 . The information processing device 2 transmits the received first spatial data to the first information terminal. After that, the first information terminal identifies its own detailed position by comparing the feature point group data generated by itself with the plurality of registered feature point group data received.

The second information terminal also operates in the same manner as the first information terminal. Send. Since the second information terminal also belongs to the same first space as the first information terminal, the information processing device 2 transmits the second feature point group data to the first computer and receives the first spatial data from the first computer. The information processing device 2 transmits the first spatial data to the second information terminal, and the second information terminal converts the feature point group data generated by itself into a plurality of registered feature point groups included in the received first spatial data. It identifies its own detailed position by comparing it with the data.

FIG. 11 is a sequence diagram showing a flow until a plurality of information terminals 1 belonging to different spaces acquire spatial data. A first information terminal and a third information terminal in FIG. 11 belong to different spaces, and are, for example, the information terminal 1a-1 and the information terminal 1b-1 shown in FIG. The operation of the first information terminal is the same as the operation shown in FIG.

The third information terminal transmits third approximate position data indicating the position of the third information terminal and third feature point group data generated based on the captured image data to the information processing device 2 . Since the third information terminal belongs to the second space different from the first information terminal, the information processing device 2 transmits the third feature point cloud data to the second computer in charge of the second space corresponding to the third approximate position data. and receive second spatial data from a third computer. The information processing device 2 transmits the second spatial data to the second information terminal, and the second information terminal converts the feature point group data generated by itself into a plurality of registered feature point groups included in the received second spatial data. It identifies its own detailed position by comparing it with the data.

(Processing for displaying AR image)
FIG. 12 is a sequence diagram showing the flow of processing in which a plurality of information terminals 1 belonging to the same space display AR images. The sequence diagram shown in FIG. 12 starts when the user U1 using the first information terminal performs an operation to display an AR image. The first information terminal specifies the content of the operation performed by the user U1 (S11), and specifies the operation position coordinates indicating the position where the operation was performed (S12). The first information terminal transmits to the information processing device 2 operation content data indicating the content of the operation and coordinate data indicating the operation position.

The information processing device 2 selects AR image data to be displayed on the first information terminal based on the received operation content data, and transmits the selected AR image data to the first information terminal. Note that when the first information terminal stores AR image data, the information processing device 2 does not have to transmit the AR image data to the first information terminal.

The information processing device 2 transmits display coordinate data corresponding to the coordinate data received from the first information terminal to the second information terminal belonging to the same space as the first information terminal. The information processing device 2 also transmits the AR image data transmitted to the first information terminal to the second information terminal. The first information terminal and the second information terminal display the AR image based on the received AR image data at the position and orientation corresponding to the display coordinate data (S13, S14). By operating the first information terminal and the second information terminal in this manner, a plurality of information terminals 1 belonging to the same space can display the same AR image at positions and orientations corresponding to their respective positions and orientations. .

(Registration of spatial data)
FIG. 13 is a sequence diagram showing the flow of processing in which the information terminal 1 registers feature point cloud data as spatial data. The sequence diagram shown in FIG. 13 starts when the first information terminal compares the feature point group data created by itself with the registered feature point group data.

When the first information terminal determines that there is no spatial data including registered feature point cloud data that matches the first feature point cloud data created by itself (YES in S21), it determines that the spatial data needs to be updated. (S22), and transmits the first feature point group data to the information processing device 2 along with the update request. Similarly, when the second information terminal determines that there is no spatial data including registered feature point cloud data that matches the second feature point cloud data created by itself (YES in S23), The second feature point group data is transmitted to the information processing device 2 together with the update request.

When receiving the first feature point group data and the second feature point group data corresponding to the same position and orientation, the information processing apparatus 2 determines which of the feature point group data is to be registered, and one of the feature point group data is selected (S24). The information processing device 2 transmits the selected feature point group data to the storage device 4, and the storage device 4 registers the received feature point group data as spatial data (S25).

[Modification]
In the above description, the information processing device 2 relays the feature point group data and registered feature point group data between the information terminal 1 and the computer group 3, but the computer group 3 has the functions of the information processing device 2. may be

Further, in the above description, the information processing device 2 provides the AR image to the information terminal 1 in response to the information terminal 1 transmitting the display coordinate data to the information processing device 2 as an example. may store a plurality of AR images in advance. In this case, the information terminal 1 selects an AR image to be displayed from a plurality of stored AR images based on an instruction from the information processing device 2, and displays the selected AR image at the position and orientation based on the received display coordinate data. Display the AR image.

Further, in the above description, the information processing device 2 has an in-memory database, and various display coordinate data and feature point group data are exchanged via the in-memory database. 2 does not have an in-memory database, and a plurality of information terminals 1 may exchange these data via other storage media.

[Effects of information processing system S]
The information processing device 2 according to the present embodiment receives the display coordinate data transmitted by the first information terminal, registers the received display coordinate data in the in-memory database, and registers the display coordinate data in the in-memory database. In response, the display coordinate data is transmitted to the second information terminal belonging to the same space as the first information terminal. The information processing device 2 transmits and receives display coordinate data to and from a plurality of information terminals 1 via an in-memory database, thereby shortening the time required to display an AR image on the information terminals 1 .

Further, the computer group 3 acquires feature point cloud data based on the captured image data generated by the information terminal 1, and a plurality of computers 30, each in charge of a different space, acquires a plurality of registered feature point cloud data and the acquired feature point cloud data. The space to which the information terminal 1 belongs is specified by comparing with the point cloud data. Then, the computer group 3 provides the information terminal 1 with a plurality of registered feature point cloud data corresponding to the specified space. With the computer group 3 operating in this manner, the information terminal 1 stores a plurality of registered feature point cloud data to be compared with the feature point cloud data created by the information terminal 1 as a plurality of registered feature point cloud data corresponding to the space to which the information terminal 1 belongs. Since it can be limited to feature point cloud data, it is possible to specify its own accurate position and orientation in a short time.

Further, the information processing device 2 changes the number of computers 30 that execute matching processing with the feature point group data generated by the information terminal 1 based on preset conditions. By operating the information processing device 2 in this way, even if the number of information terminals 1 increases or decreases, or the amount of feature point cloud data transmitted by the information terminal 1 increases or decreases, appropriate registration in the information terminal 1 can be performed. The time required to transmit the feature point group data can be maintained within a certain range.

Further, the information terminal 1 can register the feature point group data generated by itself in the storage device 4 when there is no registered feature point group data corresponding to the feature point group data generated by itself. By registering a large amount of feature point cloud data generated by the information terminal 1 in the storage device 4, the registered feature point cloud data stored in the storage device 4 can be maintained in the latest state. Furthermore, when the information processing apparatus 2 receives update requests for feature point group data corresponding to the same position and orientation from a plurality of information terminals 1 , the information processing apparatus 2 registers feature point group data with higher reliability in the storage device 4 . By operating the information processing device 2 in this way, the reliability of the registered feature point group data stored in the storage device 4 is improved.

Although the present invention has been described above using the embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments, and various modifications and changes are possible within the scope of the gist thereof. be. For example, all or part of the device can be functionally or physically distributed and integrated in arbitrary units. Further, new embodiments resulting from arbitrary combinations of multiple embodiments are also included in the embodiments of the present invention. The effect of the new embodiment caused by the combination has the effect of the original embodiment.

1 Information terminal 2 Information processing device 3 Computer group 4 Storage device 11 Communication unit 12 Operation unit 13 Position detection unit 14 Imaging unit 15 Display unit 16 Storage unit 17 Control unit 21 Communication unit 22 Storage unit 23 Control unit 30 Computer 31 Communication unit 32 Storage unit 33 control unit 171 transmission processing unit 172 reception processing unit 173 operation reception unit 174 general position specifying unit 175 image processing unit 176 detailed position specifying unit 177 display processing unit 178 spatial data acquisition unit 221 first space database 222 second space database 231 coordinate receiving unit 232 coordinate transmitting unit 233 terminal data receiving unit 234 feature point group transmitting unit 235 information receiving unit 236 spatial data transmitting unit 237 spatial data managing unit 238 number managing unit 331 feature point group acquiring unit 332 registered data acquiring unit 333 Data Analysis Department

Claims (11)

One or more information terminal groups including a plurality of information terminals that display an augmented reality image in the same coordinate system space, and an information processing device that provides the augmented reality image to the information terminal,
A first information terminal among the plurality of information terminals,
a first display processing unit that causes a display unit to display a first augmented reality image superimposed on an image of a space viewed by a user of the first information terminal;
display coordinate data indicating coordinates in the space when the first augmented reality image is displayed on the display unit in response to receiving an operation for displaying the first augmented reality image on the display unit; a first operation reception unit to be created;
a first transmission processing unit that transmits the display coordinate data to the information processing device;
has
The information processing device is
a coordinate receiving unit that receives the display coordinate data transmitted by the first information terminal and registers the received display coordinate data in an in-memory database;
a coordinate transmission unit configured to transmit the display coordinate data to a second information terminal, which is different from the first information terminal among the plurality of information terminals, in response to registration of the display coordinate data in the in-memory database;
has
The second information terminal is
a second reception processing unit that receives the display coordinate data transmitted by the coordinate transmission unit;
Second display processing for displaying a second augmented reality image superimposed on an image of the space viewed by the user of the second information terminal at a position on the display unit corresponding to the coordinates in the space indicated by the display coordinate data. Department and
An information processing system having The information processing device has a plurality of in-memory databases corresponding to each of the plurality of spaces,
The coordinate receiving unit registers the display coordinate data in the in-memory database corresponding to the space containing the first information terminal that transmitted the display coordinate data.
The information processing system according to claim 1. The information processing device has a plurality of in-memory databases corresponding to each of the plurality of spaces,
The coordinate transmission unit transmits the display coordinate data registered in the in-memory database corresponding to the space containing the second information terminal to the second information terminal.
The information processing system according to claim 1 or 2. The information processing device further has a storage unit that stores a plurality of terminal identification information for identifying the plurality of information terminals included in the space in association with each of the plurality of spaces,
The coordinate transmission unit has, by referring to the storage unit, second terminal identification information associated with the space corresponding to the first terminal identification information of the first information terminal that transmitted the display coordinate data. transmitting the display coordinate data to the second information terminal;
The information processing system according to any one of claims 1 to 3. The first operation receiving unit creates operation content data indicating a type of the first augmented reality image based on the received operation,
The first transmission processing unit further transmits the operation content data to the information processing device,
The coordinate transmission unit transmits data corresponding to the second augmented reality image corresponding to the operation content data or the second augmented reality image in response to the operation content data being registered in the in-memory database. transmitting data for identification to the second information terminal;
The information processing system according to any one of claims 1 to 3. The first information terminal stores a plurality of spatial data indicating feature points of a subject included in a plurality of images previously created by imaging the space from a plurality of positions and orientations in the space. and a terminal storage unit that stores in association with the orientation;
A space most similar to feature point group data indicating feature points of a subject included in a captured image created by capturing the space by the first information terminal, among the plurality of space data stored in the terminal storage unit. a fine localization unit that identifies a position and orientation corresponding to the data;
further having
The first operation reception unit creates the display coordinate data based on the relationship between the position and orientation identified by the detailed position identification unit and the position of the received operation.
The information processing system according to any one of claims 1 to 5. The first operation reception unit performs the first operation when displaying the first augmented reality image on the display unit based on the relationship between the position and orientation identified by the detailed position identification unit and the position of the received operation. 1 Create display orientation data indicating the orientation of the augmented reality image,
The first transmission processing unit further transmits the display orientation data to the information processing device,
The coordinate transmission unit transmits the display orientation data to the second information terminal in response to registration of the display orientation data in the in-memory database.
The information processing system according to claim 6. The first information terminal acquires the plurality of spatial data from the information processing device in response to the first operation accepting unit accepting an operation for acquiring the plurality of spatial data from the information processing device. , further comprising a spatial data acquisition unit that stores the plurality of acquired spatial data in the terminal storage unit;
The information processing system according to claim 6 or 7. The first information terminal includes a position detection unit that detects the position of the first information terminal;
a spatial data acquisition unit that acquires the plurality of spatial data corresponding to the position detected by the position detection unit from the information processing device and stores the acquired plurality of spatial data in the terminal storage unit;
The information processing system according to claim 6 or 7. The information processing device further includes a device storage unit that stores the first information terminal and the second information terminal belonging to a space of the same coordinate system as the plurality of information terminals that display an augmented reality image in the space. have
The information processing system according to any one of claims 6 to 9. A first information terminal belonging to a predetermined space receives an operation for displaying a first augmented reality image on a display unit of the first information terminal superimposed on an image of the space viewed by a user of the first information terminal. a step of generating display coordinate data indicating coordinates in the space when the first augmented reality image is displayed on a display unit;
a step in which the first information terminal transmits the display coordinate data to an information processing device;
a step in which the information processing device registers the display coordinate data received from the first information terminal in an in-memory database;
a step in which the information processing device transmits the display coordinate data to a second information terminal belonging to the space, different from the first information terminal, in response to registration of the display coordinate data in the in-memory database;
a step in which the second information terminal receives the display coordinate data transmitted by the information processing device;
The second information terminal corresponds to the first augmented reality image at a position on the display unit of the second information terminal corresponding to the coordinates in the space viewed by the user of the second information terminal indicated by the display coordinate data. displaying a second augmented reality image;
An information processing method comprising:

Images