JP7075090B1 - Information processing system and information processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To shorten the time required for a process of estimating the position of a terminal by matching the data obtained by scanning the real space with the data imitating the real space.
In an information processing system, a terminal measures the position of the terminal in the real space, scans the real space in 3D, and displays the terminal position / orientation information including the measurement result of the position of the terminal and the result of the 3D scan in the real space. The including 3D scan data is transmitted to the server, and the server receives the terminal position / orientation information and the 3D scan data from the terminal, and based on the terminal position / orientation information, the spatial model data which is the data obtained by modeling the real space in 3D. By extracting a part of the above as subspace model data and detecting the part that matches the 3D scan data from the subspace model data, the position of the terminal in the space model data is specified, and the position of the terminal in the space model data. In-model terminal position / orientation information indicating is transmitted to the terminal.
[Selection diagram] Fig. 1

Description

The present disclosure relates to an information processing system and an information processing method.

AR (Augmented Reality) or MR (Mixed Reality) technologies that make users visually recognize virtual objects as if they exist in real space are known. In order to visually recognize the virtual object without shifting to the real space, it is required to estimate the position and posture of the terminal that provides the user with the experience of AR or MR with high accuracy.

Patent Document 1 discloses the following terminals. That is, the terminal is in the real space of the terminal by matching each feature point based on the current real image taken by the camera that takes the surrounding real image and the 3D point group data acquired from the server. The position of is estimated, and the three-dimensional image of the equipment information acquired from the server is superimposed and displayed on the actual image of the equipment taken by the camera using AR technology.

Japanese Unexamined Patent Publication No. 2016-170060

However, the process of estimating the position of the terminal in the real space by matching the data obtained by imaging or scanning the real space with the data imitating the real space is processed when the amount of data to be matched becomes large. The time tends to be longer.

An object of the present disclosure is to reduce the time required for the process of estimating the position of the terminal by matching the data obtained by scanning the real space with the data imitating the real space.

One aspect of the present disclosure is an information processing system including a terminal and a server, in which the terminal measures the position of the terminal in the real space, scans the real space in 3D, and obtains the measurement result of the position of the terminal. The terminal position / orientation information including the terminal position / orientation information and the 3D scan data including the result of the 3D scan in the real space are transmitted to the server, and the server receives the terminal position / orientation information and the 3D scan data from the terminal, and the terminal. Based on the position / orientation information, a part of the spatial model data which is the data obtained by modeling the real space in 3D is extracted as the partial spatial model data, and the portion matching with the 3D scan data is detected from the partial spatial model data. This provides an information processing system that identifies the position of the terminal in the spatial model data and transmits the terminal position / orientation information in the model indicating the position of the terminal in the spatial model data to the terminal.

One aspect of the present disclosure is an information processing method using a terminal and a server, wherein the terminal measures the position of the terminal in a real space, scans the real space in 3D, and includes a measurement result of the position of the terminal. The terminal position / orientation information and 3D scan data including the result of the 3D scan in the real space are transmitted to the server, and the server receives the terminal position / attitude information and the 3D scan data from the terminal, and the terminal. Based on the position / orientation information, a part of the spatial model data which is the data obtained by modeling the real space in 3D is extracted as the partial spatial model data, and the portion matching with the 3D scan data is specified from the partial spatial model data. This provides an information processing method for specifying the position of the terminal in the spatial model data and transmitting the terminal position / orientation information in the model indicating the position of the terminal in the specified spatial model data to the terminal. ..

It should be noted that these comprehensive or specific embodiments may be realized by a system, an apparatus, a method, an integrated circuit, a computer program or a recording medium, and any of the system, an apparatus, a method, an integrated circuit, a computer program and a recording medium. It may be realized by various combinations.

According to the present disclosure, it is possible to shorten the time required for the process of estimating the position of the terminal by matching the data obtained by scanning the real space with the data imitating the real space.

It is a block diagram which shows the structural example of the information processing system which concerns on this embodiment. It is a schematic diagram which shows an example of the spatial model data which concerns on this embodiment. It is a schematic diagram which shows an example of 3D (three dimensions) scan data which concerns on this embodiment. It is a block diagram which shows the hardware configuration example of the terminal which concerns on this embodiment. It is a block diagram which shows the hardware configuration example of the server which concerns on this embodiment. It is a flowchart which shows the operation example of the information processing system which concerns on this embodiment. It is a flowchart which shows the detailed example of the subspace model extraction process. It is a flowchart which shows the detailed example of the moving body removal process. It is a flowchart which shows the detailed example of a matching process. It is a flowchart which shows the detailed example of the terminal position attitude calculation process in a model.

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. However, explanations may be omitted for publicly known matters or overlapping matters. In addition, the drawings and the following description are merely examples, and are not intended to limit the subject matter of the description of the claims.

(Implementation)
<Information processing system configuration>
FIG. 1 is a block diagram showing a configuration example of an information processing system according to the present embodiment. FIG. 2 is a schematic diagram showing an example of spatial model data according to the present embodiment. FIG. 3 is a schematic diagram showing an example of 3D scan data according to the present embodiment.

The information processing system 1 according to the present embodiment includes a terminal 100 and a server 200. The server 200 may be read as an information processing device. The terminal 100 and the server 200 can send and receive data via a predetermined communication network 3. Examples of the communication network 3 include a mobile communication network (for example, LTE (Long Term Evolution), 4G, 5G, Wi-Fi (registered trademark), etc.), an Internet network, and the like. Examples of the terminal 100 include smartphones, tablet terminals, VR (Virtual Reality) devices, AR glasses, and the like.

In the information processing system 1, the terminal 100 scans the real space in 3D to generate 3D scan data 120 (see FIG. 3), and the server 200 matches the 3D scan data 120 with the spatial model data 220 (see FIG. 2). By performing processing and calculating the position, altitude, and posture of the terminal 100 in the spatial model data 220, the position, altitude, and posture of the terminal 100 in the real space are specified with high accuracy. Hereinafter, the information indicating the position, altitude, and attitude of the terminal 100 specified with high accuracy is referred to as high-precision terminal position / attitude information 122. Thereby, for example, when the terminal 100 makes the user visually recognize that a predetermined virtual object exists at a predetermined position in the real space, the position of the virtual object with respect to the real space is used by using the high-precision terminal position / orientation information 122. Matching can be done with high accuracy.

As shown in FIG. 3, the 3D scan data 120 may be 3D point cloud data obtained by 3D scanning a real space with a LiDAR (Light Detection And Ranging), a stereo camera, or the like.

As shown in FIG. 2, the space model data 220 may be data obtained by modeling an object in the real space in 3D. The real space may include both outdoor and indoor. Examples of objects that are 3D modeled in outdoor real space include buildings, roads, terrain, road signs, and the like. Examples of objects that are 3D modeled in an indoor real space include room floors, ceilings, walls, doors, furniture, furnishings, and the like. The spatial model data 220 may be read as digital twin data. The spatial model data 220 may be 3D point cloud data, texture-mapped 3D data, or other 3D data. The 3D point cloud data may be data configured as a collection of 3D points (that is, a 3D point cloud) indicating the three-dimensional coordinates of one point on the surface of the object.

The spatial model data 220 may be data obtained by modeling the entire country, the entire region, the entire city, the entire road, the entire building, or the like in 3D, and the amount of data is very large. Therefore, the time required for the matching process between the 3D scan data 120 and the spatial model data 220 tends to be long. If the time required for the matching process becomes long, it becomes difficult for the terminal 100 to superimpose the virtual object on the real space in real time. Therefore, in the present embodiment, a method of shortening the time required for the matching process between the 3D scan data 120 and the spatial model data 220 will be described.

<Terminal configuration>
FIG. 4 is a block diagram showing a hardware configuration example of the terminal 100 according to the present embodiment.

The terminal 100 includes a position measuring unit 101, a posture measuring unit 102, a 3D scanning unit 103, an imaging unit 104, an input unit 105, a display unit 106, a memory 107, a storage 108, a communication unit 109, and a processor 110.

The position measuring unit 101 receives a GNSS (Global Navigation Satellite System) signal and / or an RTK (Real Time Kinematic) signal, and measures the position of the terminal 100 in the real space. The position of the terminal 100 may be represented by longitude and latitude.

The posture measuring unit 102 measures the posture of the terminal 100 in the real space by at least one of an electronic compass, a gyro sensor, an acceleration sensor, and the like. The posture of the terminal 100 may be represented by an azimuth angle and an elevation / depression angle. In the present embodiment, the information including the position and posture of the terminal 100 measured in the real space is referred to as the terminal position / posture information 121.

The 3D scanning unit 103 is composed of a LiDAR, a stereo camera, or the like. The 3D scanning unit 103 scans the real space in 3D and generates 3D scan data 120 (see FIG. 3) in the real space. The 3D scan data 120 may be composed of 3D point cloud data.

The image pickup unit 104 includes a lens and an image sensor. The image pickup unit 104 may be read as a camera. The imaging unit 104 captures the direction of the posture of the terminal 100 in the real space and generates an captured image.

The input unit 105 is composed of, for example, a touch panel, buttons, and / or a microphone. The input unit 105 receives an input operation from the user.

The display unit 106 is composed of, for example, a display device. When the terminal 100 is AR glasses, the display device may be a transparent display device that allows the user wearing AR glasses to visually recognize the real space. The display unit 106 displays, for example, a captured image, a virtual object image, or the like.

The memory 107 includes a volatile storage medium and / or a non-volatile storage medium. The memory 107 may be configured to include a ROM (Read Only Memory) and a RAM (Random Access Memory). The memory 107 may store computer programs and data that realize the functions of the terminal 100.

The storage 108 is configured to include a non-volatile storage medium. The storage 108 may be composed of, for example, at least one of a flash memory, an SSD (Solid State Drive), and an HDD (Hard Disk Drive). The storage 108 may store computer programs and data that realize the functions of the terminal 100.

The communication unit 109 controls the transmission and reception of data via the communication network 3. The communication unit 109 may be read as another term such as a communication interface, a communication circuit, or a communication module.

The processor 110 cooperates with each component 101 to 109 to realize the function of the terminal 100. For example, the processor 110 realizes the function of the terminal 100 by reading a computer program from the memory 107 or the storage 108 and executing the program. Therefore, in the present embodiment, the process described mainly by the terminal 100 may be read as the process mainly by the processor 110 of the terminal 100. The processor 110 is read as another term such as CPU (Central Processing Unit), LSI (Large Scale Integrated Circuit), ASIC (Application Specific Integrated Circuit), FPGA (Field Programmable Gate Array), controller, and control circuit. May be good.

<Server configuration>
FIG. 5 is a block diagram showing a hardware configuration example of the server 200 according to the present embodiment.

The server 200 includes an input unit 201, a display unit 202, a memory 203, a storage 204, a communication unit 205, and a processor 206.

The input unit 201 is composed of, for example, a keyboard, a mouse, a microphone, and the like. The input unit 201 receives an input operation from the administrator of the server 200.

The display unit 202 is configured by, for example, a display device. The display unit 202 displays various information for the administrator.

The memory 203 includes a volatile storage medium and / or a non-volatile storage medium. The memory may be configured to include a ROM and a RAM. The memory 203 may store a computer program and data that realize the functions of the server 200.

The storage 204 is configured to include a non-volatile storage medium. The storage 204 may be composed of, for example, at least one of a flash memory, an SSD and an HDD. The storage 204 may store computer programs and data that realize the functions of the server 200.

The communication unit 205 controls the transmission / reception of data via the communication network 3.

The processor 206 cooperates with each component 201 to 205 to realize the function of the server 200. For example, the processor 206 realizes the function of the server 200 by reading a computer program from the memory 203 or the storage 204 and executing the program. Therefore, in the present embodiment, the process described mainly by the server 200 may be read as the process mainly by the processor 206 of the server 200.

<Operation of information processing system>
FIG. 6 is a flowchart showing an operation example of the information processing system 1 according to the present embodiment.

The terminal 100 measures the position (longitude and latitude) of the terminal 100 in the real space by the position measuring unit 101, and measures the posture (azimuth and elevation / depression angle) of the terminal 100 in the real space by the posture measuring unit 102 (the attitude measuring unit 102). Step S101).

The terminal 100 generates terminal position / posture information 121 including the position and posture of the terminal 100 measured in step S101, and transmits the terminal position / posture information 121 to the server 200 (step S102).

The server 200 receives the terminal position / orientation information 121 from the terminal 100 (step S103).

The server 200 sets the extraction area 230 in the space model data 220 based on the terminal position / orientation information 121, and extracts the subspace model data 221 from the extraction area 230 (see FIG. 2) (step S104). The process of step S104 is referred to as a subspace model extraction process. The details of the subspace model extraction process will be described later.

Further, the terminal 100 captures the direction of the posture of the terminal 100 in the real space by the imaging unit 104, and generates an captured image (step S105).

The terminal 100 3D scans the direction of the posture of the terminal 100 in the real space by the 3D scanning unit 103, and generates 3D scan data 120 (3D point cloud data) (step S106).

The terminal 100 transmits the 3D scan data 120 to the server 200 (step S107).

The server 200 receives the 3D scan data 120 from the terminal 100 (step S108).

The server 200 removes the moving body 240 from the 3D scan data 120, and generates 3D scan data 120 (hereinafter, referred to as 3D scan data after removing the moving body) from which the moving body 240 is removed (step S109). The process of step S109 is referred to as a moving body removal process. The details of the moving body removal process will be described later.

The server 200 detects a portion (hereinafter referred to as a matching portion) that matches the 3D scan data 120 after removing the moving object from the subspace model data 221 (step S110). The process of step S110 is referred to as a matching process. The details of the matching process will be described later.

The server 200 specifies the position, altitude, and attitude of the terminal 100 in the spatial model data 220 based on the detected matching portion (step S111). The process of step S111 is referred to as a terminal position / orientation calculation process in the model. The details of the terminal position / orientation calculation process in the model will be described later.

The server 200 generates the terminal position / attitude information 222 in the model including the position, altitude, and attitude of the terminal 100 in the spatial model data 220 specified in step S111, and transmits it to the terminal 100 (step S112).

The terminal 100 receives the terminal position / attitude information 222 in the model from the server 200 (step S113).

The terminal 100 specifies the position, altitude, and attitude of the terminal 100 with high accuracy in the real space based on the terminal position / attitude information 222 in the model (step S114). That is, the terminal 100 generates high-precision terminal position / orientation information 122 including information indicating the position, altitude, and attitude of the high-precision terminal 100 in the real space.

The terminal 100 specifies a superposed position of the virtual object image on the captured image based on the high-precision terminal position / orientation information 122, superimposes the virtual object image on the specified superposed position, and displays the virtual object image on the display unit (step S115). ..

As a result, the terminal 100 can display an image on the display unit 106 as if a virtual object exists at a predetermined position of the captured image captured in the real space. Further, since the superposed position of the virtual object image is specified by using the high-precision terminal position / orientation information 122, the terminal 100 is a virtual object with respect to the real space even when the measurement accuracy in step S101 is insufficient. Image alignment can be performed with high accuracy. For example, even in a place where the GNSS signal cannot be received accurately, such as indoors, the terminal 100 can perform the alignment of the virtual object image with high accuracy by using the spatial model data 220 in the room.

When the terminal 100 is AR glasses, the terminal 100 may perform the next step S115a as step S115.

The terminal 100 specifies the display position of the virtual object image with respect to the real space that can be seen through the display unit 106 based on the high-precision terminal position / orientation information 122, and displays the virtual object image at the specified display position of the display unit 106. Display (step S115a).

As a result, the terminal 100 allows the user wearing the AR glasses to visually recognize the virtual object as if it exists at a predetermined position in the real space seen through the display unit 106. can.

In FIG. 6, for the sake of clarity, the processing of the terminal 100 and the processing of the server 200 are shown as serial processing, but at least a part of the processing of the terminal 100 and the processing of the server 200 is parallel processing. May be good. For example, the terminal 100 may perform the processing of steps S101 and S102, the processing of step S105, and the processing of step S106 in parallel. In this case, the server 200 may perform the processing of steps S103 and S104 and the processing of steps S108 to S112 in parallel.

<Details of subspace model extraction process>
FIG. 7 is a flowchart showing a detailed example of the subspace model extraction process. Next, with reference to FIG. 7, the subspace model extraction process of step S104 shown in FIG. 6 will be described in detail.

(Step S201) The server 200 sets the extraction area 230 (see FIG. 2) in the spatial model data 220 based on the terminal position / orientation information 121 received from the terminal 100.

For example, the server 200 specifies the position of the terminal 100 (hereinafter referred to as the terminal position 131 in the model) indicated by the terminal position / orientation information 121 in the spatial model data 220, and has a predetermined size including the specified terminal position 131 in the model. The extraction area 230 is set for the spatial model data 220.

The extraction area 230 may be set horizontally. In that case, the shape of the extraction region 230 may be a square, a rectangle, a circle, an ellipse, or the like. When the extraction area 230 is square, the length of one side of the extraction area 230 may be about 100 m. However, the length of one side of the extraction region 230 may be any length, and may change depending on whether the spatial model data 220 in which the extraction region 230 is set is outdoor or indoor. ..

Further, as shown in FIG. 2, the server 200 has a direction of the azimuth angle of the terminal 100 indicated by the terminal position / orientation information 121 from the terminal position 131 in the model (that is, in front of the terminal 100; hereinafter, the azimuth angle direction of the terminal in the model). The extraction area is such that the distance d1 to the boundary of (referred to as) is longer than the distance d2 from the terminal position in the model to the boundary in the direction opposite to the azimuth direction of the terminal in the model (that is, behind the terminal 100). 230 may be set. This is because in the matching process for 3D scanning the front of the terminal 100, the spatial model data 220 in front of the terminal 100 is mainly used, and the spatial model data 220 behind the terminal 100 is hardly used. As a result, the server 200 can extract the subspace model data 221 from the extraction area 230 mainly extending in front of the terminal 100 in the space model data 220. Therefore, for example, the subspace model data from the extraction area 230 centered on the terminal 100. Compared with the case of extracting 221, the amount of data of the subspace model data 221 can be suppressed.

(Step S202) The server 200 extracts the spatial model data 220 included in the extraction area 230 set in step S201 as the subspace model data 221. The server 200 may extract all the spatial model data 220 in the height direction of the extraction region 230 as subspace model data 221. Alternatively, the server 200 may extract the spatial model data 220 up to a predetermined height of the extraction region 230 as the partial spatial model data 221.

As described above, the server 200 extracts the subspace model data 221 from the space model data 220 based on the terminal position / attitude information 121 transmitted from the terminal 100, and in the matching process of step S110, the subspace model data 221 is used. Is the matching target of the 3D scan data 120. As a result, the amount of data to be matched is smaller than when the spatial model data 200 is directly used as the matching target of the 3D scan data 120, so that the server 200 can shorten the time required for the matching process. Therefore, the server 200 can shorten the time required for the process of estimating the position of the terminal 100.

<Details of moving body removal processing>
FIG. 8 is a flowchart showing a detailed example of the moving body removal process. Next, with reference to FIG. 8, the moving body removing process in step S109 shown in FIG. 6 will be described in detail.

(Step S301) The server 200 removes the object data of the moving body 240 from the 3D scan data 120 received from the terminal 100, and generates the 3D scan data 120 after the moving body is removed. Examples of the moving body 240 include automobiles, motorcycles, bicycles, trains, pedestrians, animals and the like.

The server 200 removes the object data of the moving body 240 from the 3D scan data 120 by, for example, any of the following methods (A1), (A2), or (A3).

(A1) The server 200 holds 3D data of various mobile bodies 240 in advance. Then, the server 200 detects the segment data estimated to be the moving body 240 from the 3D scan data 120 (that is, the object data of the moving body 240) by pattern matching with the 3D data of the moving body 240. Then, the server 200 removes the detected object data of the moving body 240 from the 3D scan data 120.

(A2) The server 200 holds in advance a predetermined moving object detection AI (Artificial Intelligence) that detects the object data of the moving body 240 from the 3D scan data 120. Then, the server 200 detects the object data of the moving object from the 3D scan data 120 by using the moving object detecting AI. Then, the server 200 removes the detected object data of the moving body 240 from the 3D scan data 120. The moving object detection AI may be one in which the features of the object data of the moving object 240 are learned in advance by machine learning, neural network, deep learning, or the like.

(A3) The server 200 acquires the first 3D scan data 120 3D-scanned at time t1 and the second 3D scan data 120 3D-scanned at time t2 (> t1). Then, the server 200 detects the object data of the moving body 240 based on the change from the first 3D scan data 120 to the second 3D scan data 120. Typically, the amount of time change of the moving body 240 is large, and the amount of time change of a stationary body such as a building, a road, or a sign is small. Therefore, by the above processing, the server 200 detects the object data of the moving body 240. can. Then, the server 200 removes the detected object data of the moving body 240 from the first or second 3D scan data 120.

Typically, the spatial model data 220 does not include the moving body 240. Therefore, by removing the object data of the moving body 240 from the 3D scan data 120 in this way, the matching accuracy between the 3D scan data 120 after removing the moving body and the subspace model data 221 is improved in the matching process. At the same time, the robustness of matching is improved.

When the space model data 220 includes the moving body 240, the server 200 removes the moving body 240 from the subspace model data 221 extracted in step S104 in the moving body removing process, and removes the moving body. The subspace model data 221 may be used for the matching process in step S110.

<Details of matching process>
FIG. 9 is a flowchart showing a detailed example of the matching process. Next, with reference to FIG. 9, the matching process of step S110 shown in FIG. 6 will be described in detail.

(Step S401) The server 200 calculates the feature amount of each feature point of the subspace model data 221. The server 200 may calculate in advance the feature amount of each feature point of the subspace model data 221 and hold it in the memory 203 or the storage 204.

(Step S402) The server 200 calculates the feature amount of each feature point of the 3D scan data 120 after removing the moving object.

(Step S403) The server 200 detects the feature points of the subspace model data 221 that match the feature points of the 3D scan data 120 after the moving object is removed as the matching portion. For example, the server 200 detects the matching portion by detecting the feature points of the subspace model data 221 having the feature amounts similar to the feature amounts of the feature points of the 3D scan data 120 after the moving object is removed. The feature quantity may be expressed by a multidimensional vector. In this case, the fact that the two features are similar means that the inner product of the two multidimensional vectors is less than the predetermined threshold value, or the absolute value of the difference between the elements of the two multidimensional vectors is less than the predetermined threshold value. It may be there.

In this way, by matching the partial spatial model data 221 with a smaller amount of data than the entire spatial model data 220 and the 3D scan data 120 after removing the moving object, the entire spatial model data 220 and the 3D scan after removing the moving object are scanned. Compared with the case of matching with the data 120, the time required for the matching process can be shortened.

Further, since the spatial model data 220 typically does not include a moving object, the 3D scan data 120 that does not remove the moving object is matched by matching the 3D scan data 120 after the moving object is removed with the subspace model data 221. Compared with the case of matching the subspace model data 221 with the subspace model data 221, the probability that a matching error occurs is reduced. That is, the robustness of matching is improved.

When the moving body 240 is removed from the subspace model data 221 in the above moving body removing process, the server 200 may perform the matching process using the subspace model data 221 from which the moving body 240 is removed. ..

<Details of terminal position / orientation calculation processing in the model>
FIG. 10 is a flowchart showing a detailed example of the terminal position / orientation calculation process in the model. Next, with reference to FIG. 10, the terminal position / orientation calculation process in the model in step S111 shown in FIG. 6 will be described in detail.

(Step S501) When the server 200 arranges a virtual terminal 130 in the spatial model data 220 and performs a virtual 3D scan, the server 200 obtains a matching portion detected by the matching process from the spatial model data 220. The position, altitude, and attitude of the virtual terminal 130 in the spatial model data 220 that can be calculated are calculated.

As a result, the server 200 can generate high-precision terminal position / orientation information 122 indicating the position, altitude, and attitude of the high-precision terminal 100.

<Modification example>
In step S109, the server 200 detects the moving body 240 from the 3D scan data 120 instead of the process of removing the moving body 240 from the 3D scan data 120, and imparts moving body information to the detected moving body 240. You may. In this case, in step S110, the server 200 performs the matching process by excluding the portion to which the moving body information of the 3D scan data 120 is added from the matching process. This also improves the robustness of matching, as described above.

Further, the terminal 100 may perform at least a part of the processing performed by the server 200 in the above description. For example, the terminal 100 may perform the moving object removal process in step S109 and transmit the 3D scan data 120 after the moving object removal to the server 200. For example, the terminal 100 may receive the subspace model data 221 from the server 200 and use the subspace model data 221 to perform the matching process of step S110 and the terminal position / attitude calculation process in the model of step S111.

(Summary of this disclosure)
The present disclosure can be expressed as follows.

<Item 1>
The information processing system 1 according to the present disclosure includes a terminal 100 and a server 200. The terminal 100 measures the position of the terminal 100 in the real space, scans the real space in 3D, and includes the terminal position / orientation information 121 including the measurement result of the position of the terminal 100 and the 3D scan data 120 including the result of the 3D scan in the real space. To the server 200. The server 200 receives the terminal position / orientation information 121 and the 3D scan data 120 from the terminal 100, and based on the terminal position / orientation information 121, a part of the spatial model data 220 which is the data obtained by modeling the real space in 3D is a partial space. By extracting as model data 221 and detecting a portion matching with 3D scan data from the subspatial model data 221, the position of the terminal 100 in the spatial model data 220 is specified, and the position of the terminal 100 in the spatial model data 220. In-model terminal position / orientation information 222 indicating the above is transmitted to the terminal 100.
In this way, the server 200 extracts the subspace model data 221 from the space model data 200 based on the terminal position / attitude information 121 transmitted from the terminal 100, and matches the subspace model data 221 with the 3D scan data 120. set to target. As a result, the amount of data to be matched is smaller than when the spatial model data 200 is directly used as the matching target of the 3D scan data 120, so that the server 200 can shorten the time required for the matching process. Therefore, the server 200 can shorten the time required for the process of estimating the position of the terminal 100.

<Item 2>
In the information processing system 1 according to item 1, the terminal 100 may measure the posture of the terminal 100 in the real space, and the measurement result of the posture of the terminal 100 may be included in the terminal position / posture information 121.
As a result, the server 200 can recognize the posture of the terminal 100 based on the terminal position / posture information 121 transmitted from the terminal 100, so that the partial space model data 221 corresponding to the posture of the terminal 100 can be extracted.

<Item 3>
In the information processing system 1 according to item 2, the server 200 includes the position of the terminal 100 indicated by the terminal position / orientation information 121 in the spatial model data 220, and the direction of the attitude of the terminal 100 indicated by the terminal position / orientation information 121. The subspace model data 221 may be extracted from a predetermined area extending to.
As a result, the server 200 can extract the subspace model data 221 mainly from the area extending in front of the terminal 100 in the space model data 220, so that the subspace model data 221 can be extracted from the area centered on the terminal 100. The amount of data of the subspace model data 221 can be suppressed as compared with the above.

<Item 4>
In the information processing system 1 according to item 2 or 3, the terminal position / orientation information 121 includes a latitude and longitude indicating the position of the terminal 100 in the real space, and an azimuth angle and an elevation / depression angle indicating the attitude of the terminal 100, and is a model. The inner terminal position / orientation information 222 may include a longitude, latitude, and altitude indicating the position of the terminal 100 in the spatial model data 220, and an azimuth angle and an elevation / depression angle indicating the attitude of the terminal 100.
As a result, the server 200 can recognize the longitude and latitude at which the terminal 100 is located in the real space, and the azimuth and elevation / depression angles at which the terminal 100 is facing, based on the terminal position / attitude information 121 transmitted from the terminal 100. .. Further, the terminal 100 determines the longitude, latitude, and altitude at which the terminal 100 is located in the real space, and the azimuth and elevation / depression angles at which the terminal is facing, based on the terminal position / attitude information 222 in the model transmitted from the server 200. Can be recognized with high accuracy.

<Item 5>
In the information processing system 1 according to any one of items 1 to 4, the 3D scan data 120 and the spatial model data 220 may be composed of 3D point cloud data.
As a result, the server 200 can perform matching between the subspace model data 221 and the 3D scan data 120 by the matching process between the 3D point cloud data.

<Item 6>
In the information processing system 1 according to item 5, the server 200 matches the feature points of the 3D point cloud data included in the subspace model data 221 with the feature points of the 3D point cloud data included in the 3D scan data 120. By doing so, the position of the terminal 100 in the spatial model data 220 may be specified.
As a result, the server 200 can specify the position of the terminal 100 in the spatial model data 220 by the matching process of the feature points of the 3D point cloud data.

<Item 7>
In the information processing system 1 according to any one of items 1 to 6, the server 200 specifies a portion of the partial space model data 221 that matches the data excluding the moving body of the 3D scan data 120, thereby providing space. The position of the terminal 100 in the model data 220 may be specified.
As a result, in the matching process, the matching accuracy between the 3D scan data 120 after the moving object is removed and the subspace model data 221 is improved, and the robustness of the matching is improved.

<Item 8>
In the information processing system 1 according to any one of items 1 to 7, the terminal 100 further includes an image pickup unit 104 that captures an image in a direction common to 3D scanning, and images are captured based on the terminal position / orientation information 222 in the model. A predetermined virtual object image superimposition position on the captured image captured by the unit 104 may be specified, and an image in which the virtual object image is superimposed on the superimposition position of the captured image may be displayed on the predetermined display unit 106.
As a result, the terminal 100 can accurately superimpose the virtual object image on the captured image based on the terminal position / orientation information 222 in the model.

<Item 9>
In the information processing system 1 according to any one of items 1 to 7, the terminal 100 includes a display unit 106 capable of seeing through a direction common to 3D scanning, and displays based on the terminal position / orientation information 222 in the model. The display position of a predetermined virtual object image in the unit 106 may be specified, and the virtual object image may be displayed at the display position of the display unit 106.
As a result, the terminal 100 can accurately superimpose and display the virtual object image in the real space that can be seen through from the display unit 106 based on the terminal position / orientation information 222 in the model.

<Item 10>
The information processing method by the terminal 100 and the server 200 is as follows. The terminal 100 measures the position of the terminal 100 in the real space, scans the real space in 3D, and contains the terminal position / orientation information 121 including the measurement result of the position of the terminal 100 and the 3D scan data including the result of the 3D scan in the real space. 120 and 120 are transmitted to the server 200. , The server 200 receives the terminal position / orientation information 121 and the 3D scan data 120 from the terminal 100, and a part of the spatial model data 220 which is the data obtained by modeling the real space in 3D based on the terminal position / attitude information 121. By extracting as the spatial model data 221 and specifying the portion matching the 3D scan data from the partial spatial model data 221, the position of the terminal 100 in the spatial model data 220 is specified, and the position in the specified spatial model data 220 is specified. The terminal position / attitude information 222 in the model indicating the position of the terminal 100 is transmitted to the terminal 100.
In this way, the server 200 extracts the subspace model data 221 from the space model data 200 based on the terminal position / attitude information 121 transmitted from the terminal 100, and matches the subspace model data 221 with the 3D scan data 120. set to target. As a result, the amount of data to be matched is smaller than in the case where the spatial model data 200 is directly used as the matching target for the 3D scan data, so that the server 200 can shorten the time required for the matching process. Therefore, the server 200 can shorten the time required for the process of estimating the position of the terminal 100.

Each component of the above-described embodiment can be changed, modified, replaced, added, deleted, equalized, or the like without departing from the spirit of the invention. In addition, each component of the above-described embodiment can be arbitrarily combined as long as it does not deviate from the gist of the invention.

The technique of the present disclosure is useful for matching the data obtained by scanning the real space with the data imitating the real space, and can be used, for example, for highly accurate identification of the position and posture of the terminal in the real space. ..

1 Information processing system 3 Communication network 100 Terminal 101 Position measurement unit 102 Attitude measurement unit 103 3D scan unit 104 Imaging unit 105 Input unit 106 Display unit 107 Memory 108 Storage 109 Communication unit 110 Processor 120 3D scan data 121 Terminal position Attitude information 122 High Accuracy Terminal position Attitude information 131 In-model terminal position 200 Server 201 Input unit 202 Display unit 203 Memory 204 Storage 205 Communication unit 206 Processor 220 Spatial model data 221 Partial space model data 222 In-model terminal position / attitude information 230 Extraction area 240 Moving object

Claims (7)

An information processing system equipped with a terminal and a server.
The terminal is
The first position of the terminal in the real space is measured based on the GNSS signal .
Using an electronic compass, the first posture of the terminal in the real space is measured.
Using LiDAR, the real space is 3D-scanned as 3D point cloud data, and 3D scan data of the real space is generated .
The terminal position / orientation information including the measurement results of the first position and the first attitude of the terminal and the 3D scan data are transmitted to the server.
The server
The terminal position / orientation information and the 3D scan data are received from the terminal, and the terminal is received.
In the space model data which is the data obtained by 3D modeling the real space with 3D point group data, the terminal includes the first position of the terminal indicated by the terminal position / orientation information and the terminal position / orientation information indicates. A region extending by a predetermined distance in the direction of the first posture is extracted as subspace model data, and is extracted.
By detecting the portion matching with the 3D scan data from the subspace model data, the second position and the second posture of the terminal in the spatial model data are specified.
In-model terminal position / posture information indicating the second position and the second posture of the terminal in the spatial model data is transmitted to the terminal.
Information processing system. The first position is the latitude and longitude of the terminal measured based on the GNSS signal .
The first posture is the azimuth and elevation / depression angle of the terminal measured using the electronic compass .
The second position is the longitude, latitude and altitude of the terminal identified by matching the subspace model data with the 3D scan data .
The second posture is the azimuth and elevation / depression angles of the terminal specified by matching the subspace model data with the 3D scan data .
The information processing system according to claim 1 . The server
By matching the feature points of the 3D point cloud data included in the subspatial model data with the feature points of the 3D point cloud data included in the 3D scan data, the position of the terminal in the spatial model data is specified. do,
The information processing system according to claim 1 or 2 . The server
By specifying a portion of the subspace model data that matches the data excluding the moving body of the 3D scan data, the position of the terminal in the spatial model data is specified.
The information processing system according to any one of claims 1 to 3 . The terminal is
Further provided with an image pickup unit that captures images in the same direction as the 3D scan.
Based on the terminal position / orientation information in the model, the superimposed position of the predetermined virtual object image with respect to the captured image captured by the imaging unit is specified.
An image in which the virtual object image is superimposed on the superimposed position of the captured image is displayed on a predetermined display unit.
The information processing system according to any one of claims 1 to 4 . The terminal is
It is equipped with a display unit that can see through the same direction as the 3D scan.
Based on the terminal position / orientation information in the model, the display position of a predetermined virtual object image in the display unit is specified.
The virtual object image is displayed at the display position of the display unit.
The information processing system according to any one of claims 1 to 5 . Information processing method using terminals and servers
The terminal is
The first position of the terminal in the real space is measured based on the GNSS signal .
Using an electronic compass, the first posture of the terminal in the real space is measured.
Using LiDAR, the real space is 3D-scanned as 3D point cloud data, and 3D scan data of the real space is generated .
The terminal position / orientation information including the measurement results of the first position and the first attitude of the terminal and the 3D scan data are transmitted to the server.
The server
The terminal position / orientation information and the 3D scan data are received from the terminal, and the terminal is received.
In the space model data which is the data obtained by 3D modeling the real space with 3D point group data, the terminal includes the first position of the terminal indicated by the terminal position / orientation information and the terminal position / orientation information indicates. A region extending by a predetermined distance in the direction of the first posture is extracted as subspace model data, and is extracted.
By specifying the portion that matches the 3D scan data from the subspace model data, the second position and the second posture of the terminal in the spatial model data are specified.
In-model terminal position / posture information indicating the second position and the second posture of the terminal in the specified spatial model data is transmitted to the terminal.
Information processing method.

Images