WO2021156977A1

WO2021156977A1 - Space recognition system, space recognition method, information terminal, and server device

Info

Publication number: WO2021156977A1
Application number: PCT/JP2020/004388
Authority: WO
Inventors: 橋本　康宣; 尚久高見澤; 秋山　仁
Original assignee: マクセル株式会社
Priority date: 2020-02-05
Filing date: 2020-02-05
Publication date: 2021-08-12
Also published as: CN115053262A; US20230089061A1; JPWO2021156977A1

Abstract

Provided is a technique with which an information terminal can measure space to create and register spatial data, and can acquire and use the spatial data. This space recognition system comprises: an information terminal having a function of measuring space and a function of displaying a virtual image on a display surface, and having a terminal coordinate system; and an information processing device that performs a process based on a common coordinate system. The information terminal measures a relationship related to the position and orientation between the terminal coordinate system and the common coordinate system, and matches the terminal coordinate system and the common coordinate system, on the basis of data representing the measured relationship, and the information terminal and the information processing device share the recognition of space.

Description

Spatial recognition system, spatial recognition method, information terminal, and server device

The present invention relates to a technology such as a system for an information terminal to recognize a space.

Information terminals such as head-mounted displays (HMDs) and smartphones have a function of displaying images (sometimes referred to as virtual images) corresponding to virtual reality (VR), augmented reality (AR), etc. on the display surface. .. For example, the HMD worn by the user displays an AR image at a position matching the actual object such as a wall or a desk in a space such as a room.

As an example of the prior art related to the above technology, Japanese Patent Application Laid-Open No. 2014-514653 (Patent Document 1) can be mentioned. Patent Document 1 states that a plurality of terminals recognize the same object in real space, for example, a desk surface as an anchor surface based on a camera capture, and each terminal displays a virtual object on the anchor surface. So, the technique of displaying a virtual object at almost the same position is described.

Special Table 2014-514653

In order for the above information terminal to be able to suitably display a virtual image in a function such as AR, it is preferable to grasp the position, orientation, shape, etc. of a real object such as a wall or a desk in the space with as high accuracy as possible. The information terminal has a function of detecting and measuring the space around the own machine by using a camera or a sensor for grasping the information terminal. For example, the HMD can detect the reflection point when the light emitted from the sensor of the own machine hits a surrounding object and returns as a feature point, and can acquire a plurality of surrounding feature points as point cloud data. The HMD can use such point cloud data to form spatial data (in other words, data for an information terminal to recognize a space) representing the shape of a space or the like.

However, when the user's information terminal performs the above measurement on a vast space or a large number of spaces in the real world, there are problems in terms of efficiency, user convenience, workload, and the like. For example, when one user performs a work of measuring a space in a certain building with an information terminal, it may take a long time and a heavy load may occur.

In addition, when a user's information terminal measures and grasps a space such as a room once, uses it for AR image display, etc., and then uses the space again, it must be measured again. The efficiency is not good.

An object of the present invention is a technique in which an information terminal can measure a space and create / register spatial data, and the information terminal can acquire and use the spatial data, and the space among a plurality of information terminals of a plurality of users. It is to provide technology that can share data and corresponding spatial perceptions.

A typical embodiment of the present invention has the following configuration. The space recognition system of one embodiment includes an information terminal having a function of measuring space and a function of displaying a virtual image on a display surface and having a terminal coordinate system, and an information processing device that performs processing based on a common coordinate system. The information terminal measures the relationship regarding the position and orientation between the terminal coordinate system and the common coordinate system, and the information terminal measures the relationship between the terminal coordinate system and the common coordinate system, and based on the data representing the measured relationship, the information terminal measures the relationship between the terminal coordinate system and the common coordinate system. The system is adapted, and the information terminal and the information processing device share the recognition of the space.

According to a typical embodiment of the present invention, the information terminal can measure the space and create / register the spatial data, the information terminal can acquire and use the spatial data, and a plurality of information of a plurality of users. The spatial data and the corresponding spatial recognition can be shared between the terminals.

It is a figure which shows the structure of the space recognition system of Embodiment 1 of this invention. It is a figure which shows the structure of the space recognition method of Embodiment 1 of this invention. It is a figure which shows the composition example of the space in Embodiment 1. FIG. It is a figure which shows the example of the space sharing measurement in Embodiment 1. It is a figure which shows the use example of a space in Embodiment 1. FIG. It is a figure which shows the appearance composition example of the HMD which is an information terminal in Embodiment 1. It is a figure which shows the functional block composition example of the HMD which is an information terminal in Embodiment 1. It is explanatory drawing about the coordinate system pairing in Embodiment 1. FIG. 5 is an explanatory diagram for position transmission and the like in the first embodiment. FIG. 5 is a diagram showing a processing flow between information terminals in the first embodiment. It is a figure which shows the display example by an information terminal in Embodiment 1. FIG. In Embodiment 1, it is explanatory drawing about rotation of a coordinate system and the like. It is explanatory drawing about the coordinate system pairing and the like in the modification 2 of Embodiment 1. It is explanatory drawing about the conversion parameter in the modification 2 of Embodiment 1. It is explanatory drawing about the coordinate system pairing and the like in the modification 3 of Embodiment 1. It is a figure which shows the structure of the space recognition system of Embodiment 2 of this invention. It is explanatory drawing about the coordinate system pairing in Embodiment 2. It is explanatory drawing about the modification 4 of Embodiment 2. It is a figure which shows the structure of the space recognition system of Embodiment 3 of this invention. It is a figure which shows the structural example of the sign in Embodiment 3. FIG. It is a figure which shows the processing flow of an information terminal and a server in Embodiment 3. It is explanatory drawing about the modification 5 of Embodiment 3. It is a figure which shows the processing flow which concerns on the modification 6 of Embodiment 3. It is explanatory drawing which concerns on the modification 6 of Embodiment 3. It is a figure which shows the display example 1 by an information terminal in the space recognition system of Embodiment 4 of this invention. It is a figure which shows the example of the sharing of space in Embodiment 4. It is a figure which shows the display example 2 by the information terminal in Embodiment 4. It is a figure which shows the display example 3 by an information terminal in Embodiment 4. It is a figure which shows the display example 4 by an information terminal in Embodiment 4. It is a figure which shows the display example 5 by an information terminal in Embodiment 4. It is a figure which shows the basic structure of this invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In principle, the same parts are designated by the same reference numerals in all drawings, and repeated description will be omitted.

<Embodiment 1>
The space recognition system and method according to the first embodiment of the present invention will be described with reference to FIGS. 1 to 12 and the like. Hereinafter, the information terminal device may be referred to as a terminal.

Conventionally, with respect to terminals such as HMDs, the concept of creating / registering spatial data by measuring the space to be used by multiple terminals of multiple users in a spatially or temporally manner, and for that purpose. Insufficient studies have been conducted on suitable techniques. The spatial recognition system and method of the first embodiment provides suitable techniques such as spatial measurement by such sharing, creation of spatial data, sharing / reuse of spatial data, and a series of procedures for that purpose. .. This system and method realizes spatial measurement, spatial data creation, spatial data sharing / reuse, etc. efficiently, for example, at high speed.

First, FIG. 31 shows the basic configuration of the present invention. The basic configuration of the present invention processes an information terminal 1 having a function of measuring space 2 and a function of displaying a virtual image on a display surface and having a terminal coordinate system WT, and spatial data 6 described by a common coordinate system WS. It is composed of an information processing device 9 for processing. The information processing device 9 is an information terminal different from the information terminal 1, or a server 4 (FIG. 16 or the like) described later. The information terminal 1 measures the relationship regarding the position and orientation between the terminal coordinate system WT and the common coordinate system WS, and adapts the terminal coordinate system WT and the common coordinate system WS based on the data representing the measured relationship. .. This is called coordinate system pairing (described later). By this coordinate system pairing, the information terminal 1 and the information processing device 9 share the spatial recognition.

The sharing of the spatial data 6 is performed through the description of the spatial data 6 by the common coordinate system WS. For example, the information terminal 1 measures the space 2 and acquires the space data 6, and acquires the space data 6 described based on the common coordinate system WS from the information processing device 9. The information terminal 1 converts the spatial data 6 acquired from the information processing device 9 into the terminal coordinate system WT of its own machine, integrates it with the spatial data 6 measured by its own machine, and then uses the spatial data 6. Alternatively, the information terminal 1 converts the spatial data 6 measured by its own machine into a description by the common coordinate system WS and provides it to the information processing device 9. The information processing device 9 integrates the provided spatial data 6 and the retained spatial data 6 and uses the spatial data 6.

The space recognition system of the first embodiment shown in FIG. 1 and the like measures the space 2 by the division of the plurality of terminals 1 by a plurality of users, and creates the space data 6 based on the measurement data. In order to clarify the surface direction of the objects and the shielding relationship between the objects, the spatial data 6 may include the position of the terminal 1 at the time of measurement (measurement starting point). The spatial data 6 can be provided to each other, shared and used among a plurality of terminals 1 of a plurality of users, and recognition of a position, orientation, etc. in the same space 2 can be shared. As a result, the same virtual image 22 can be easily displayed at the same desired position 21 in the space 2 when the AR function or the like is used between the plurality of terminals 1, and work, communication, etc. are suitable. realizable. According to this system, more efficient work can be realized than when the work such as measurement is performed by the terminal of one user. In the first embodiment, the terminal 1 holds the spatial data 6. Further, the information processing device 9 in FIG. 31 is the terminal 1 in FIG. 1, and the common coordinate system WS is either terminal used as a reference for description when exchanging spatial data 6 between a plurality of terminals 1. It is a terminal coordinate system WT of 1.

The space recognition system and method of the first embodiment has a mechanism such as matching between coordinate systems related to the measurement and recognition of the space 2 in the above division. In general, the coordinate system of space (sometimes referred to as "spatial coordinate system") and the coordinate system of each terminal (sometimes referred to as "terminal coordinate system") are basically different coordinate systems. And at least initially they don't match. Therefore, in the embodiment, at the time of the above sharing, the operation of associating and matching the terminal coordinate systems of each terminal 1 with each other is performed as "coordinate system pairing". By this operation, the conversion parameter 7 for the coordinate system conversion is set in the terminal 1. In the state where this coordinate system pairing is established, the position, orientation, and the like can be mutually converted between the coordinate systems by the conversion parameter 7. As a result, the recognition of the position and the like with respect to the same space 2 can be shared between the terminals 1 that perform the sharing. Each terminal 1 creates partial spatial data 6 described in each terminal coordinate system by shared measurement. Then, the plurality of partial spatial data described in each terminal coordinate system is a space in units of space 2 described in a certain unified terminal coordinate system by conversion and integration using the conversion parameter 7. It can be configured as data 6.

[Spatial recognition system]
FIG. 1 shows the configuration of the spatial recognition system of the first embodiment. In this example, the case where the target space 2 used by the terminal 1 is set as one room in the building and the HMD is particularly used as the terminal 1 will be described. In the space recognition system of the first embodiment, a plurality of users, for example, users U1 and U2 carry or wear a plurality of terminals 1, for example, first terminal 1A and second terminal 1B, and measurement of each terminal 1. It has a space 2 to be used. Each terminal 1 creates and holds spatial data 6 and conversion parameters 7. The terminal 1 may be a device such as a

smartphone

1a, 1b or a tablet terminal. Each terminal 1 is connected to a communication network including the Internet, a mobile network, and the like through a wireless LAN access point 23 and the like, and can also communicate with an external device via the communication network.

Space 2 is an arbitrary space that is identified and managed separately, for example, one room in a building. In this example, the space 2 of this room is the object of creating the space data 6 by sharing, and is the object of recognition sharing by the plurality of terminals 1.

The plurality of terminals 1 have, for example, a first terminal 1A (= first HMD) of the first user U1 and a second terminal 1B (= second HMD) of the second user U2. The HMD, which is the terminal 1, is provided with a transmissive display surface 11, a camera 12, a distance measuring sensor 13, and the like in a housing, and has a function of displaying a virtual image of AR on the display surface 11. Similarly, the

smartphones

1a and 1b are provided with a display surface such as a touch panel, a camera, a distance measuring sensor, and the like, and have a function of displaying a virtual image of AR on the display surface. Even when a smartphone or the like is used as the terminal 1, functions and the like substantially the same as those in the case of the HMD can be realized. For example, the user sees a virtual image such as AR displayed on the display surface of a smartphone held in his hand.

Each terminal 1 has a function of performing coordinate system pairing between its own terminal and the other terminal 1. Each terminal 1 measures the relationship between its own terminal coordinate system (for example, the first terminal coordinate system WA) and the other party's terminal coordinate system (for example, the second terminal coordinate system WB), and the conversion parameter 7 is based on the relationship. Is generated and set to at least one of the own machine or the other party. The plurality of terminals 1 (1A, 1B) share and measure the target space 2, and create each partial spatial data 6 (may be described as "subspace data"). For example, the first terminal 1A creates the spatial data D1A, and the second terminal 1B creates the spatial data D1B. The plurality of terminals 1 can create spatial data 6 (for example, spatial data D1) in units of spatial 2 from the partial spatial data 6, and can share the recognition of spatial 2 using the spatial data 6. The terminal 1 has a function of measuring the space 2 using the camera 12, the distance measuring sensor 13, and the like, and creating the space data 6 based on the measurement data. The terminal 1 can perform conversion between coordinate systems related to the representation of measurement data and spatial data 6 by using the conversion parameter 7.

The relationship between the terminal coordinate systems (WA, WB) is roughly obtained as follows. First, the relationship of rotation between the coordinate systems is obtained based on the measurement of the representation in each terminal coordinate system (WA, WB) in two different specific directions in the real space. Next, the relationship of the origins between the terminal coordinate systems is obtained based on the measurement of the positional relationship between the terminals 1. The conversion parameter 7 can be composed of the rotation parameter and the origin parameter.

In the first embodiment, the terminal coordinate system of one of the terminals 1 (for example, the first terminal coordinate system WA) is used as a common coordinate system among the plurality of terminals 1, and the coordinate system pairing is performed for each pair of the two terminals 1. I do. As a result, at least one of the terminals 1 (for example, the first terminal 1A) creates and holds the conversion parameter 7. After that, each terminal 1 shares and measures the space 2, and creates each subspace data described in each terminal coordinate system. Each terminal 1 may exchange / exchange each subspace data with each other terminal 1 as data described based on a common coordinate system. The terminal 1 uses the conversion parameter 7 to convert the subspace data between the description based on the terminal coordinate system of the own machine and the description based on the common coordinate system. If the terminal coordinate system of the own machine is a common coordinate system, this conversion is not necessary. Then, the terminal 1 obtains the spatial data 6 in units of the space 2 by integration from the plurality of subspace data described in the unified terminal coordinate system. As a result, the plurality of terminals 1 can suitably display the same virtual image 22 at the same position 21 in the same space 2 by using the spatial data 6.

Even in the case of the terminal 1 provided with the non-transparent display, while the terminal coordinate system of the terminal 1 is fixed in the real space, the display position of the virtual image displayed on the display surface 11 is set to the other terminal 1. Can be shared with.

[Coordinate system]
In the first embodiment, the coordinate system that serves as a reference for designating the position, orientation, and the like in the real space in each terminal 1 and space 2 is called a world coordinate system. Each terminal 1 has a terminal coordinate system as its own world coordinate system. In FIG. 1, the first terminal 1A has a first terminal coordinate system WA, and the second terminal 1B has a second terminal coordinate system WB. Each terminal coordinate system is a coordinate system for recognizing and controlling the position and orientation (in other words, posture, rotation state), image display position, and the like of the terminal 1. Since these terminal coordinate systems are set for each terminal 1, they are basically different coordinate systems and do not match in the initial state. Further, the space 2 has a space coordinate system W1 as a world coordinate system representing the position and orientation of the space 2. In the first embodiment, the spatial coordinate system W1 is not used for the measurement and creation of the spatial data 6. In the first embodiment, the spatial data 6 is described in the terminal coordinate system. The first terminal coordinate system WA, the second terminal coordinate system WB, and the spatial coordinate system W1 are different coordinate systems. The origin and direction of each world coordinate system are fixed in the real space (earth, region, etc.).

The first terminal coordinate system WA has the origin _{O A,} the axis _{X A} as 3 orthogonal axes, the axis _{Y A,} and a shaft _{Z A.} The second terminal coordinate system WB, with the origin _{O B,} the axis _{X B} as 3 orthogonal axes, the axis _{Y B,} and a shaft _{Z B.} Space coordinate system W1 has, as an origin _{O 1,} the axis _{X 1} as 3 orthogonal axes, the axis _{Y 1,} and a shaft _{Z 1.} The origins O _A , _B and the origin O ₁ are fixed at predetermined positions in the real space, respectively. The position LA of the first terminal 1A in the first terminal coordinate system WA and the position LB of the second terminal 1B in the second terminal coordinate system WB are defined in advance as, for example, the housing center position (FIG. 8).

When the terminal 1 shares the recognition of the space 2, the terminal 1 performs coordinate system pairing with another terminal 1. For example, the sharing terminals 1 (1A, 1B) perform coordinate system pairing with each other. At the time of coordinate system pairing, each terminal 1 measures and acquires predetermined quantities from each other (FIG. 8), and obtains a relationship between the terminal coordinate systems (WA, WB) based on the quantities. Each terminal 1 calculates the conversion parameter 7 between the terminal coordinate systems (WA, WB) from the relationship. In the state where the coordinate system pairing is established, each terminal 1 can convert the position and the like with each other by using the conversion parameter 7. That is, each terminal 1 can convert the representation such as the position in the spatial data 6 created by the measurement of the space 2 into the representation in the common coordinate system. Each terminal 1 exchanges and exchanges spatial data 6 through spatial data 6 described with reference to a common coordinate system. As a result, each terminal 1 can integrate the subspace data measured by each terminal 1 and create the spatial data 6 described in the unified terminal coordinate system. After pairing the coordinate system, each terminal 1 can perform internal control of its own device not only based on its own terminal coordinate system but also based on the other party's terminal coordinate system. ..

[Spatial recognition method]
FIG. 2 shows an outline of the space recognition method of the first embodiment and a processing example. This method has steps S1 to S9 shown. In the example of FIG. 2, the first terminal 1A measures the area 2A and the second terminal 1B measures the area 2B as the division of the space 2 (FIG. 3 described later). Further, in the example of FIG. 2, the conversion parameter 7 is generated by the first terminal 1A, spatial data 6 (6A, 6B) with the space 2 as a unit are configured, and the first terminal 1A is the spatial data in the second terminal 1B. 6B is provided. Here, the second terminal 1B is the information processing device 9 in the basic configuration (FIG. 31), and the second terminal coordinate system WB corresponds to the common coordinate system WS.

In step S1, the first terminal 1A performs coordinate system pairing with the second terminal 1B (FIG. 8 described later), thereby converting the first terminal coordinate system WA and the second terminal coordinate system WB. The conversion parameter 7 of is generated and set in the own machine.

In step S2, the first terminal 1A measures the region 2A due to the division and creates the partial spatial data 6 (referred to as the partial spatial data D1A) described in the first terminal coordinate system WA. The * mark in the drawing indicates the coordinate system that describes the spatial data. On the other hand, in step S3, the second terminal 1B similarly measures the region 2B due to the division and creates the partial spatial data 6 (referred to as the subspace data D1B) described in the second terminal coordinate system WB. .. Steps S2 and S3 can be executed in parallel at the same time.

In step S4, the first terminal 1A receives and acquires the subspace data D1B from the second terminal 1B. In step S5, the first terminal 1A converts the subspace data D1B into the partial spatial data 6 (referred to as the subspace data D1BA) described in the first terminal coordinate system WA by using the conversion parameter 7. ..

In step S6, the first terminal 1A integrates the subspace data D1A and the subspace data D1BA into one, and the spatial data 6A (D1) having the space 2 described in the first terminal coordinate system WA as a unit. Get as. As a result, the first terminal 1A can obtain the spatial data 6A (D1) in units of the space 2 even if the measurement is performed only in the area 2A.

Furthermore, it has the following steps. In step S7, the first terminal 1A converts the subspace data D1A into the partial spatial data 6 (referred to as the subspace data D1AB) described in the second terminal coordinate system WB by using the conversion parameter 7. .. In step S8, the first terminal 1A integrates the subspace data D1B and the subspace data D1AB into one, and describes the spatial data 6B (D1) in the space 2 as a unit described in the second terminal coordinate system WB. ). In step S9, the first terminal 1A transmits the spatial data 6B (D1) to the second terminal 1B. As a result, the second terminal 1B can obtain the spatial data 6B (D1) in units of the space 2 even if only the area 2B is measured.

By the above method, for the same space 2, the first terminal 1A acquires the spatial data 6A (D1) described in the first terminal coordinate system WA, and the second terminal 1B is described in the second terminal coordinate system WB. The spatial data 6B (D1) is acquired. Therefore, the recognition of the space 2 can be shared between the terminals 1 (1A, 1B). For example, the first terminal 1A and the second terminal 1B can display the same virtual image 22 at the same position 21 in the space 2 (FIG. 5 described later). At that time, the first terminal 1A displays the virtual image 22 at the position 21 described in the first terminal coordinate system WA based on the spatial data 6A (D1). The second terminal 1B displays the virtual image 22 at the position 21 described in the second terminal coordinate system WB based on the spatial data 6B (D1).

The above method can be similarly applied to the case where the conversion parameter 7 is generated by the second terminal 1B and the spatial data 6 is configured.

[Spatial example]
FIG. 3 shows a configuration example of the space 2 and an example in which the space 2 is shared and measured by the terminals 1 of a plurality of users. This space 2 is, for example, one room in a building such as a company, for example, a seventh conference room. In the space 2, there are arrangements such as walls, floors, ceilings, doors 2d, and arrangements such as desks 2a, whiteboards 2b, and other devices. The arrangement is an arbitrary object that constitutes the space 2. The other space 2 may be a building or area such as a company or a store, or a public space or the like.

The spatial data 6 (particularly the spatial shape data described later) that describes the space 2 is, for example, data in an arbitrary format that represents the position and shape of the room. The spatial data 6 includes data representing the boundary of the space 2 and data of an arbitrary object arranged in the space 2. The data representing the boundary of the space 2 includes, for example, data of an arrangement such as a floor, a wall, a ceiling, and a door 2d constituting the room. There may be no placement at the boundary. The data of the object in the space 2 includes, for example, the data of the desk 2a and the whiteboard 2b arranged in the room. The spatial data 6 is, for example, data including at least point cloud data and having position coordinate information for each feature point in a certain terminal coordinate system. The spatial data 6 may be polygon data representing a line, a surface, or the like in the space.

In this example, the space 2 which is one room is shared and measured by the terminals 1 (1A and 1B) of the users U1 and U2 who are two users, and the space data 6 of the space 2 is created. .. The content of the division can be decided arbitrarily. For example, two users consult and share as shown in the figure. The division can be such that the target space 2 is spatially divided into a plurality of regions (in other words, subspaces) as in this example. In this example, the space 2 is divided into half area of the right and left with respect to the center in the lateral direction (axial Y ₁ direction) in FIG. The first terminal 1A is in charge of the left area 2A, and the second terminal 1B is in charge of the right area 2B.

[Example of shared measurement]
Figure 4 shows an example of measurement by sharing in the overhead space 2 room (e.g. _X 1 -Y ₁ side) as shown in FIG. 3, terminal 1 of the two users (U1, U2) (1A, 1B) Is shown. FIG. 4 shows an example of a state such as a measurement range (401, 402) at a certain position (L401, L402) and orientation (d401, d402) of the terminal 1 (1A, 1B) which is an HMD at a certain time. The measurement range is an example that depends on the function of the distance measuring sensor 13 and the like provided in the HMD. The measurement range 401 indicates a measurement range using, for example, a distance measuring sensor 13 at the position L401 and the direction d401 of the first terminal 1A. Similarly, the measurement range 402 indicates the measurement range at the position L402 and the orientation d402 of the second terminal 1B.

It is not necessary to cover 100% of the area when measuring the target space 2. A sufficient amount of area in the space 2 may be measured according to the function of AR or the like and if necessary. A part of the space 2 that is not measured may occur, or an area that is overlapped and measured may occur due to sharing. In the example of FIG. 4, the area 491 is an unmeasured area, and the area 492 is an overlapping measurement area. The measurement ratio (for example, 90%) of the space 2 or the shared area may be set in advance as a condition. For example, the first terminal 1A determines that the measurement is completed when the shared area 2A is measured at a ratio of the condition or more.

After pairing the coordinate system between the terminals 1 (1A, 1B), each terminal 1 measures each measurement range (401, 402) of the shared

areas

2A, 2B, and obtains each measurement data. The first terminal 1A measures the measurement range 401 of the region 2A and obtains the measurement data 411. The second terminal 1B measures the measurement range 402 of the region 2B and obtains the measurement data 412. The measurement data is, for example, point cloud data obtained by the distance measuring sensor 13. The point cloud data is data having a position, a direction, a distance, and the like for each point at a plurality of surrounding feature points. Each terminal 1 creates subspace data 420 from the measurement data. The first terminal 1A creates the subspace data D1A described in the first terminal coordinate system WA from the measurement data 411. The second terminal 1B creates the subspace data D1B described in the second terminal coordinate system WB from the measurement data 412.

In the example of FIG. 4, the first terminal 1A creates the spatial data 6A (D1) described in the first terminal coordinate system WA, and the second terminal 1B creates the spatial data 6B (D1) described in the second terminal coordinate system WB. The case of creating D1) is shown. Each terminal 1 transmits the subspace data 420 created by its own device to the other terminal 1. The first terminal 1A transmits the subspace data D1A to the second terminal 1B. The second terminal 1B transmits the subspace data D1B to the first terminal 1A.

Each terminal 1 converts the subspace data 430 obtained from the other terminal 1 into the subspace data 440 in the terminal coordinate system of its own device by using the conversion parameter 7. The first terminal 1A converts the subspace data D1B into the subspace data D1BA described in the first terminal coordinate system WA. The second terminal 1B converts the subspace data D1A into the subspace data D1AB described in the second terminal coordinate system WB.

Each terminal 1 converts the subspace data 420 obtained by its own machine and the subspace data 440 obtained from the other party into spatial data 6 (450) in one space 2 in a unified terminal coordinate system. Integrate. The first terminal 1A integrates the subspace data D1A and the subspace data D1BA into one, and obtains the subspace data D1 (6A) described in the first terminal coordinate system WA. The second terminal 1B integrates the subspace data D1B and the subspace data D1AB into one, and obtains the subspace data D1 (6B) described in the second terminal coordinate system WB. In the case of this example, which of the two terminals 1 is considered to correspond to the information processing device 9 having the basic configuration (FIG. 31) may be arbitrary. The terminal coordinate system used in the transfer of the spatial data 6 is the common coordinate system in the transfer of the spatial data 6.

According to the above method, the time related to the measurement and the acquisition of the spatial data can be shortened and efficiently realized as compared with the case where the space 2 is measured by the terminal 1 of one user.

In addition, in the space 2, when the other user is seen from the terminal 1 of one user and there is a space portion behind the other user, such a space portion is referred to by the terminal 1 of the other user. It is possible to measure by, and it is more efficient to measure by the terminal 1 of the other user.

At the time of measurement, the user and the corresponding terminal 1 can move the position appropriately to change the measurement range. The unmeasured area 491 shown in the figure can also be measured by separately including it in the measurement range from another position.

As a more advanced method for sharing, any terminal 1 may automatically determine and determine the sharing. For example, each terminal 1 determines the approximate position / orientation of its own device in the room, the presence / absence of reflection of another user / other device, its position / orientation, and the like based on a camera image or the like. For example, when the second user U2 and the second terminal 1B are not shown in the camera image, the first terminal 1A selects the area / range in the direction at that time as the area / range shared by the first terminal 1A. do.

[Usage example]
FIG. 5 shows an example of using the space 2 using the space data 6 between the terminals 1 (1A, 1B) of two users (U1, U2) who share the recognition of the space 2 as shown in FIG. The first terminal 1A and the second terminal 1B are the same in the space 2 by the AR function using the spatial data 6 in the state of the coordinate system pairing between the first terminal coordinate system WA and the second terminal coordinate system WB. The same virtual image 22 is displayed at the position 21. At that time, the first terminal 1A displays the virtual image 22 on the display surface 11 at the position 21 in the first terminal coordinate system WA, and the second terminal 1B is the position on the display surface 11 in the second terminal coordinate system WB. The virtual image 22 is displayed on the 21. One terminal 1, for example, the first terminal 1A specifies a position 21 and a virtual image 22 to be displayed, and transmits information such as the position 21 to the second terminal 1B. At that time, the first terminal 1A or the second coordinate system WB uses the conversion parameter 7 to convert the position 21 in the first terminal coordinate system WA to the position 21 in the second terminal coordinate system WB. Each terminal 1 can quickly and accurately display the virtual image 22 at a position 21 that matches the position, shape, and the like of the arrangement of the space 2 represented by the space data 6. For example, each terminal 1 can arrange and display the virtual image 22 according to the position 21 at the center of the upper surface of the desk 2a designated by the user U1. User U1 and user U2 can work and communicate while viewing the same virtual image 22.

[Information terminal device (HMD)]
FIG. 6 shows an example of the appearance configuration of the HMD as an example of the terminal 1. This HMD includes a display device including a display surface 11 in a spectacle-shaped housing 10. This display device is, for example, a transmissive display device, and a real image of the outside world is transmitted through the display surface 11, and the image is superimposed and displayed on the real image. A controller, a camera 12, a distance measuring sensor 13, another sensor unit 14, and the like are mounted on the housing 10.

The camera 12 has, for example, two cameras arranged on the left and right sides of the housing 10, and captures a range including the front of the HMD to acquire an image. The distance measuring sensor 13 is a sensor that measures the distance between the HMD and an object in the outside world. As the distance measuring sensor 13, a TOF (Time Of Flight) type sensor may be used, or a stereo camera or another type may be used. The sensor unit 14 includes a group of sensors for detecting the position and orientation of the HMD. On the left and right sides of the housing 10, an audio input device 18 including a microphone, an audio output device 19 including a speaker and an earphone terminal, and the like are provided.

The terminal 1 may be equipped with an operator such as a remote controller. In that case, the HMD performs, for example, short-range wireless communication with the controller. By manually operating the operating device, the user can input instructions related to the HMD function, move the cursor on the display surface 11, and the like. The HMD may communicate with an external smartphone, PC, or the like to cooperate. For example, the HMD may receive AR image data from a smartphone application.

The terminal 1 includes an application program or the like for displaying a virtual image such as AR on the display surface 11 for work support or entertainment. For example, the terminal 1 generates a virtual image 22 (FIG. 1) for work support by processing an application for work support, and a predetermined position on the display surface 11 near a work object in space 2. A virtual image 22 is arranged and displayed on the 21.

FIG. 7 shows an example of a functional block configuration of the terminal 1 (HMD) of FIG. The terminal 1 is a processor 101, a memory 102, a camera 12, a distance measuring sensor 13, a sensor unit 14, a display device 103 including a display surface 11, a communication device 104, a voice input device 18 including a microphone, a voice output device including a speaker, and the like. 19. The operation input unit 105, the battery 106, and the like are provided. These elements are connected to each other through a bus or the like.

The processor 101 is composed of a CPU, ROM, RAM, etc., and constitutes an HMD controller. The processor 101 realizes functions such as an OS, middleware, and applications, and other functions by executing processing according to the control program 31 and the application program 32 of the memory 102. The memory 102 is composed of a non-volatile storage device or the like, and stores various data and information handled by the processor 101 and the like. The memory 102 also stores images, detection information, and the like acquired by the camera 12 and the like as temporary information.

The camera 12 acquires an image by converting the light incident from the lens into an electric signal by the image sensor. When a TOF sensor is used, for example, the distance measuring sensor 13 calculates the distance to an object from the time until the light emitted to the outside world hits the object and returns. The sensor unit 14 includes, for example, an acceleration sensor 141, a gyro sensor (angular velocity sensor) 142, a geomagnetic sensor 143, and a GPS receiver 144. The sensor unit 14 detects a state such as the position, orientation, and movement of the HMD by using the detection information of these sensors. The HMD is not limited to this, and may include an illuminance sensor, a proximity sensor, a barometric pressure sensor, and the like.

The display device 103 includes a display drive circuit and a display surface 11, and displays a virtual image or the like on the display surface 11 based on the image data of the display information 34. The display device 103 is not limited to the transparent display device, and may be a non-transparent display device or the like.

The communication device 104 includes a communication processing circuit, an antenna, and the like corresponding to various predetermined communication interfaces. Examples of communication interfaces include mobile networks, Wi-Fi (registered trademark), Bluetooth (registered trademark), infrared rays and the like. The communication device 104 performs wireless communication processing and the like with another terminal 1 and the access point 23 (FIG. 1). The communication device 104 also performs short-range communication processing with the actuator.

The voice input device 18 converts the input voice from the microphone into voice data. The voice output device 19 outputs voice from a speaker or the like based on the voice data. The voice input device may include a voice recognition function. The voice output device may include a voice synthesis function. The operation input unit 105 is a part that receives operation inputs to the HMD, such as power on / off and volume adjustment, and is composed of a hardware button, a touch sensor, and the like. The battery 106 supplies electric power to each part.

The controller by the processor 101 has a communication control unit 101A, a display control unit 101B, a data processing unit 101C, and a data acquisition unit 101D as a configuration example of a functional block realized by processing.

The memory 102 stores the control program 31, the application program 32, the setting information 33, the display information 34, the coordinate system information 35, the spatial data information 36, and the like. The control program 31 is a program for realizing control including a spatial recognition function. The application program 32 is a program that realizes a function such as AR that uses the spatial data 6. The setting information 33 includes system setting information and user setting information related to each function. The display information 34 includes image data and position coordinate information for displaying an image such as a virtual image 22 on the display surface 11.

The coordinate system information 35 is management information related to the space recognition function. When the coordinate system information 35 is shared between two users as shown in FIG. 3, for example, the information of the first terminal coordinate system WA of the own machine, the information of the second terminal coordinate system WB of the other party, and the own machine side. The various amount data of the above, the various amount data of the other party (FIG. 8), and the conversion parameter 7 (FIG. 1 and the like) are included.

The spatial data information 36 is information corresponding to the spatial data 6 such as FIG. 1, and is information created and held by the terminal 1. The terminal 1 may hold the spatial data 6 relating to each space 2 as a library inside the terminal 1. The terminal 1 may acquire and hold the spatial data 6 from another terminal 1. The terminal 1 may acquire the spatial data 6 held and provided by an external server or the like as described later.

The communication control unit 101A controls communication processing using the communication device 104 when communicating with another terminal 1. The display control unit 101B uses the display information 34 to control the display of the virtual image 22 and the like on the display surface 11 of the display device 103.

The data processing unit 101C reads and writes the coordinate system information 35, performs processing for managing the terminal coordinate system of the own machine, processing of coordinate system pairing with the terminal coordinate system of the other party, and coordinates using the conversion parameter 7. Performs conversion processing between systems. At the time of coordinate system pairing, the data processing unit 101C performs a process of measuring various amount data on the own machine side, a process of acquiring various amount data on the other side, a process of generating conversion parameter 7, and the like.

The data acquisition unit 101D acquires each detection data from various sensors such as the camera 12, the distance measuring sensor 13, and the sensor unit 14. At the time of coordinate system pairing, the data acquisition unit 101D measures various quantity data on the own machine side according to the control from the data processing unit 101C.

[Coordinate system pairing]
Next, the details of the coordinate system pairing will be described. FIG. 8 shows an explanatory diagram in the case of performing coordinate system pairing between the first terminal coordinate system WA of the first terminal 1A and the second terminal coordinate system WB of the second terminal 1B of FIG. 1, and each coordinate system. And the relationship of various quantities. Hereinafter, spatial recognition sharing by coordinate system pairing between the first terminal coordinate system WA and the second terminal coordinate system WB between these two terminals 1 will be described.

In the example of FIG. 8, unlike the position of the origin O _A of the first terminal coordinate system WA and the position LA of the first terminal 1A, the position and the second terminal 1B of the origin O _B of the second terminal coordinate system WB It is different from the position LB, but is not limited to this. In some cases, their positions may match, and in that case, the same applies. In the following, the relationship between the coordinate systems will be described in a general case where the origin of the world coordinate system and the position of the terminal 1 do not match in this way.

For example, when the first terminal 1A shares the space recognition with the second terminal 1B, the first terminal 1A performs coordinate system pairing as an operation of sharing the world coordinate system information with each other as one pair. The two terminals 1 (1A, 1B) need only perform coordinate system pairing once. Even if there are three or more terminals 1, the coordinate system pairing may be performed for each pair in the same manner.

At the time of coordinate system pairing, each terminal 1 (1A, 1B) measures a predetermined amount in each terminal coordinate system (WA, WB), and exchanges various amount data with the other terminal 1. The first terminal 1A has, as the amount 801 to be measured by the own apparatus side, measuring the specific direction vector _{N A,} and the vector _{P BA} between terminals, and a coordinate value _{d A.} The first terminal 1A transmits the data of these quantities 801 to the second terminal 1B. The second terminal 1B has, as the amount 802 to be measured by the own apparatus side, measuring the specific direction vector _{N B,} and the vector _{P AB} between the terminals, and a coordinate value _{d B.} The second terminal 1B transmits the data of these quantities 802 to the first terminal 1A.

Each terminal 1 can obtain the relationship between the pair of terminal coordinate systems based on the various quantity data measured by the own machine and the various quantity data obtained from the other party, and from the relationship, the relationship between the terminal coordinate systems can be obtained. The conversion parameter 7 for conversion can be calculated. As a result, the world coordinate system information can be shared between the terminals 1 by associating each terminal coordinate system with the conversion parameter 7.

When only one terminal 1 in the coordinate system pairing, for example, the first terminal 1A performs conversion between the coordinate systems, only the first terminal 1A has the various quantities 801 on the own machine side and the other side. The various quantities 802 may be acquired to generate the conversion parameter 7. In this case, it is not necessary to transmit various quantities 801 from the first terminal 1A to the second terminal 1B. Further, the first terminal 1A may transmit the generated conversion parameter 7 to the second terminal 1B. Then, the conversion can be performed on the second terminal 1B side as well.

In the first embodiment, there is information on the following three elements as various quantities at the time of coordinate system pairing. The quantities have a specific direction vector as the first information, a terminal-to-terminal vector as the second information, and a world coordinate value as the third information.

(1) Specific direction vector: Each terminal 1 uses a specific direction vector as information regarding a specific direction in the real space in the world coordinate system. To determine the relationship between the rotational between coordinate systems, the two different specific direction vector _{_{_{(N A, N B, M}}} A, M B) used. Specific direction vector N _A is a representation of the first direction vector in the first terminal 1A, the unit direction vector and n _A. Specific direction vector N _B is the representation of the first direction vector in the second terminal 1B, the unit direction vector and n _B. Specific direction vector M _A is a representation of the second direction vector of the first terminal 1A, the unit direction vector and m _A. Specific direction vector M _B is a representation of the second direction vector of the second terminal 1B, the unit direction vector and m _B.

In the first embodiment, in particular, a vertical downward direction is used as one specific direction (first specific direction), and an inter-terminal vector described later is used as another specific direction (second specific direction). In the example of FIG. 8, a specific direction vector N _A of the vertically downward direction as a first specified _direction, using N _B. Specific direction vector N _A is the direction vector of the vertically downward direction of the first terminal 1A, the unit direction vector and n _A. Specific direction vector N _B is the direction vector of the vertically downward direction of the second terminal 1B, the unit direction vector and n _B.

The vertical downward direction can be measured as the direction of gravitational acceleration by using, for example, a three-axis acceleration sensor which is an acceleration sensor 141 (FIG. 7) provided in the terminal 1. Alternatively, in the setting of the world coordinate system (WA, WB), a vertically downward direction, Z-axis _{_(Z} A, Z _B) may be set as a negative direction. In any case, since the vertical downward direction, which is a specific direction, does not change in the world coordinate system, it is not necessary to measure each time the coordinate system pairing is performed.

(2) Regarding inter-terminal vector: Each terminal 1 has a terminal position (LA, for example) as information indicating a positional relationship from one terminal 1 (for example, the first terminal 1A) to the other terminal 1 (for example, the second terminal 1B). The information of the vector (that is, the direction and the distance) between LB) is used. This information is referred to as "terminal-to-terminal vector". In the example of FIG. 8, the inter-terminal vectors P _BA and P _AB are used. The inter-terminal vector P _BA is a vector representing the positional relationship in the direction from the position LA to the position LB of the second terminal 1B with reference to the first terminal 1A. The inter-terminal vector _PAB is a vector representing the positional relationship in the direction from the position LB to the position LA of the first terminal 1A with reference to the second terminal 1B. From the first terminal 1A to the second terminal 1B, the vector representation of the first terminal coordinate system WA is _{P BA,} from the second terminal 1B to the first terminal 1A, vector representation of the second terminal coordinate system WB Is _PAB .

The inter-terminal vector contains information about another specific direction (second specific direction) in real space for finding the orientation relationship between the world coordinate systems. Here, the specific direction vector _{_(M} A, M _B) and end-to-end vector _{_(P BA,} P _AB) for the corresponding relationship below.
P _BA = _{M A}
P _AB = -M _B

At the time of coordinate system pairing, each terminal 1 measures the vector between terminals up to the other terminal 1 by using, for example, the distance measuring sensor 13 shown in FIG. 1 or the stereo camera 12. The distance measurement of the positional relationship between the terminals 1 may be described in detail as follows. For example, the distance measuring sensor 13 of the first terminal 1A measures the distance to the second terminal 1B seen in front. At this time, the first terminal 1A may measure the shape of the housing of the second terminal 1B from the image of the camera 12 for the recognition of the second terminal 1B, or may be formed in the housing of the second terminal 1B. You may measure using a predetermined marker or the like as a feature point.

(3) World coordinate values: Each terminal 1 uses coordinate value information representing a position in the world coordinate system. In the example of FIG. 8, the world coordinate values, using the coordinate values d _B in the coordinate value d _A, and a second terminal coordinate system WB in the first terminal coordinate system WA. _{Let d A} = (x _A , y _A , z _A ) be the coordinate value in the first terminal coordinate system WA for the position LA of the first terminal 1A. The coordinate values in the second terminal coordinate system WB for the position LB of the second terminal _{_{_{1B, d B = (x B}}} , y B, z B) and. These coordinate values are determined according to the settings of the world coordinate system. Terminal position vector _{V A} is the vector from the origin _{O A} to the position LA. Terminal position vector _{V B} is the vector from the origin _{O B} to the position LB.

In FIG. 8, the vector F _A, corresponding to the terminal position information, a vector representing the position of the second terminal 1B of the first terminal coordinate system WA of the first terminal 1A, the coordinate values of the first terminal 1A It corresponds to a vector obtained by synthesizing d _A (vector _VA ) and the inter-terminal vector _PBA. Vector _{F B} is a vector representing the position of the first terminal 1A on the second terminal coordinate system WB of the second terminal 1B, the coordinate values of the second terminal 1B _{d B} (vector _{V B)} and the terminal between the vectors _{P AB} Corresponds to the vector obtained by synthesizing and. Position vector G _A is the vector of position 21 in the first terminal coordinate system WA, position coordinates r _A is the coordinate value of the position 21. Position vector G _B is the vector of position 21 in the second terminal coordinate system WB, the position coordinates r _B is the coordinate value of the position 21. Origin between the vectors _{o BA} is a vector from the origin _{O A} of the first terminal coordinate system WA to the origin _{O B} of the second terminal coordinate system WB, the origin between vectors _{o AB} is the origin O of the second terminal coordinate system WB is a vector from _B to the origin _{O a} of the first terminal coordinate system WA. Vector E _A is the vector of view the position 21 from the position LA corresponding to the viewpoint of the user U1. Vector E _B is the vector of view the position 21 from position LB corresponding to the viewpoint of the user U2.

[Conversion parameter]
By the above coordinate system pairing, the relationship of the world coordinate system (WA, WB) between the terminals 1 (1A, 1B) can be understood, and the position and orientation can be converted to each other. That is, a conversion for matching the second terminal coordinate system WB with the first terminal coordinate system WA, or vice versa is possible. The conversion between the world coordinate systems is represented by a predetermined conversion parameter 7. The conversion parameter 7 is a parameter for calculating the conversion of the direction of the coordinate system (in other words, rotation) and the difference between the origins of the coordinate system.

For example, when enabling the coordinate system conversion on the first terminal 1A, the first terminal 1A is located between the terminal coordinate system (WA, WB) from the various quantities 801 on the own machine side and the various quantities 802 on the other side. The relationship is calculated, the conversion parameter 7 is generated, and the conversion parameter 7 is set in the own machine. The same can be done on the second terminal 1B side. The conversion parameter 7 includes a conversion parameter 71 that converts a position or the like in the first terminal coordinate system WA into a position or the like in the second terminal coordinate system WB, and a conversion parameter 7 that converts a position or the like in the second terminal coordinate system WB into the position or the like in the first terminal coordinate system WA. There is a conversion parameter 72 that converts to the position in. Those transformations are inverse transformations of each other. At least one terminal 1 may hold the conversion parameter 7, and both terminals 1 may hold the same conversion parameter 7.

[Position conversion]
FIG. 9 shows a configuration example of position transmission and coordinate system conversion between the two terminals 1 (1A, 1B) after the coordinate system pairing. Four examples are shown as (A) to (D). The same position 21 (FIG. 8) in the space 2 can be designated and shared between the terminals 1 (1A, 1B) by using the conversion parameter 7. One terminal 1 transmits the designated information of the position 21 and the data of the virtual image 22 to be displayed to the other terminal 1. One or the other terminal 1 uses the conversion parameter 7 to convert the position 21 between the coordinate systems.

(A) shows the first example. _{The first terminal 1A uses the conversion parameter 71 to convert the position coordinate value r A} , which is the position in the first terminal coordinate system WA (for example, the position 21 of the display target of the virtual image 22), in the second terminal coordinate system WB. Is converted to the position (position coordinate value r _B ) of, and transmitted to the second terminal 1B.

(B) shows a second example. _{The first terminal 1A transmits the position coordinate value r A} , which is the position in the first terminal coordinate system WA, to the second terminal 1B, and the second terminal 1B _{converts the received position coordinate value r A} into a conversion parameter. 71 is used to convert to _{the position coordinate value r B} in the second terminal coordinate system WB.

(C) shows a third example. _{The second terminal 1B converts the position coordinate value r B} , which is the position in the second terminal coordinate system WB _{, into the position coordinate value r A} in the first terminal coordinate system WA by using the conversion parameter 72, and the first terminal 1B converts the position coordinate value r B into the position coordinate value r A in the first terminal coordinate system WA. Send to terminal 1A.

(D) shows a fourth example. _{The second terminal 1B transmits the position coordinate value r B} in the second terminal coordinate system WB to the first terminal 1A, and the first terminal 1A uses the received position coordinate value r _B with the conversion parameter 72. Then, it is converted into _{the position coordinate value r A} in the first terminal coordinate system WA.

As described above, for example, when transmitting the position from the first terminal 1A to the second terminal 1B, the conversion may be performed by the method (A) or (B), and the position is located from the second terminal 1B to the first terminal 1A. In the case of transmitting, the conversion may be performed by the method (C) or (D). In terms of correspondence with the basic configuration (FIG. 31), (A) and (D) are cases where the second terminal coordinate system is a common coordinate system, and (B) and (C) are cases where the first terminal coordinate system is used. This is the case when it becomes a common coordinate system.

A table configuration example of the conversion parameter 7 is shown at the lower side of FIG. Table 901 of the conversion parameter 71 has the conversion source terminal coordinate system, the conversion destination terminal coordinate system, the rotation, and the origin representation as items. The "conversion source terminal coordinate system" item stores the identification information of the conversion source terminal 1 (corresponding user in parentheses) and the identification information of the terminal coordinate system possessed by the terminal 1. The "conversion destination terminal coordinate system" item stores the identification information of the conversion destination terminal 1 (corresponding user in parentheses) and the identification information of the terminal coordinate system possessed by the terminal 1. The "Rotation" item stores information about the representation of rotation between those terminal coordinate systems. The "origin representation" item stores a representation of the difference in origin between those terminal coordinate systems. _{For example, the first row of the table 901 of the conversion parameter 71 includes rotation (q AB} ) for conversion from the first terminal coordinate system WA of the first terminal 1A to the second terminal coordinate system WB of the second terminal 1B. _{It has a representation (o BA} ) of the origin of the second terminal coordinate system WB as seen in the first terminal coordinate system WA.

[Processing flow]
FIG. 10 shows an example of a processing flow in the case where space 2 is shared and measured between two terminals 1 (1A, 1B) as shown in FIG. 3 and one space data 6 is obtained. FIG. 10 has a flow of the first terminal 1A (steps S1A to S12A) and a flow of the second terminal 1B (steps S1B to S12B).

In steps S1A and S1B, a wireless communication connection related to spatial recognition sharing is established between the first terminal 1A and the second terminal 1B through the processing of the communication device 107 of FIG.

In steps S2A and S2B, the user performs an input operation on the terminal 1 which is the HMD to start the measurement of the space 2. For example, the user U1 inputs a measurement start instruction to the first terminal 1A, and the user U2 inputs a measurement start instruction to the second terminal 1B. In addition, communication related to the start of measurement may be performed between the terminals 1 (1A, 1B). Further, for example, the terminal 1 may display a guide image related to the operation of starting or ending the measurement on the display surface 11. The user performs input operations for starting and ending the measurement accordingly. The input operation may be a hardware button operation, a voice recognition operation, or an operation by detecting a predetermined gesture such as moving a finger. Further, as another method, the terminal 1 may control the start and end of the measurement by a preset setting or an automatic determination.

Further, in steps S2A and S2B, the division of the space 2 region and the measurement range as shown in FIGS. 3 and 4 may be set. The terminal 1 may display an image related to the setting of sharing on the display surface 11. The user performs the setting operation according to the image. Based on steps S2A and S2B, each terminal 1 starts the following steps.

Steps S3A to S6A and S3B to S6B are steps for performing coordinate system pairing. The method of the first embodiment is a method of measuring the space 2 after pairing the coordinate system. Therefore, the measurement start instruction in steps S2A and S2B is, in other words, a coordinate system pairing start instruction.

In steps S3A and S3B, a coordinate system pairing request is transmitted from one terminal 1 to the other terminal 1. For example, the first terminal 1A transmits a coordinate system pairing request to the second terminal 1B. When the second terminal 1B receives the coordinate system pairing request and accepts it, the second terminal 1B transmits a coordinate system pairing response to the effect of accepting the request to the first terminal 1A. In steps S3A and S3B, each terminal 1 may display an image for guiding the coordinate system pairing (FIG. 11 described later) on the display surface 11.

In steps S4A and S4B, the first terminal 1A and the second terminal 1B measure various quantities (FIG. 8) for coordinate system pairing in time with each other. The first terminal 1A measures various quantities 801 and the second terminal 1B measures various quantities 802.

In steps S5A and S5B, the first terminal 1A and the second terminal 1B exchange various amount data by transmitting various amount data on the own machine side to each other. The first terminal 1A acquires various quantities 802 from the second terminal 1B, and the second terminal 1B acquires various quantities 801 from the first terminal 1A.

In steps S6A and S6B, the first terminal 1A and the second terminal 1B generate conversion parameters 7 and set them in their own machines. The first terminal 1A generates conversion parameters 7 (for example, both

conversion parameters

71 and 72 in FIG. 9) using the quantities 801 on the own machine side and the quantities 802 on the other side and sets them in the own machine. The second terminal 1B generates conversion parameters 7 (for example, both

conversion parameters

71 and 72 in FIG. 9) using the quantities 802 on the own machine side and the quantities 801 on the other side and sets them in the own machine. As a result, the state of coordinate system pairing is established.

Note that the flow may be such that the measurement start instruction input in steps S2A and S2B is performed after the coordinate system pairing is established.

After the coordinate system pairing is established up to steps S6A and S6B, in the loops after steps S7A and S7B, each terminal 1 measures the area of space 2 due to sharing (FIGS. 3 and 4). In step S7A, the first terminal 1A measures the area 2A using the distance measuring sensor 13 or the like to obtain the measurement data 411. In step S7B, the second terminal 1B measures the area 2B using the distance measuring sensor 13 or the like to obtain the measurement data 412.

In steps S8A and S8B, each terminal 1 configures subspace data based on the measurement data and transmits each other to the other terminal 1 (FIG. 4). The first terminal 1A obtains the subspace data D1A on the own machine side and the subspace data D1B from the other side. The second terminal 1B obtains the subspace data D1B on the own machine side and the subspace data D1A from the other side.

In steps S9A and S9B, each terminal 1 uses the conversion parameter 7 to convert the subspace data described in the terminal coordinate system of the other party into the portion described in the terminal coordinate system of the own machine, if necessary. Convert to spatial data (Fig. 4). For example, as shown in FIG. 9D, the first terminal 1A converts the subspace data D1B into the subspace data D1BA using the conversion parameter 72. As shown in FIG. 9B, the second terminal 1B converts the subspace data D1A into the subspace data D1AB using the conversion parameter 71. Further, each terminal 1 integrates the subspace data obtained from the own machine side and the subspace data obtained from the other side into one to obtain the spatial data 6 in units of the space 2 (FIG. 4). ). For example, the first terminal 1A obtains the spatial data 6A (D1) from the subspace data D1A and the subspace data D1BA. The second terminal 1B obtains the spatial data 6B (D1) from the subspace data D1B and the subspace data D1AB. In this example, the case where both terminals 1 create their respective spatial data 6 (6A, 6B) in parallel is shown, but the present invention is not limited to this, and the spatial data 6 created by one terminal 1 is used as the partner. It may be transmitted to the terminal 1 of.

In steps S10A and S10B, each terminal 1 determines whether or not to end the spatial measurement in the coordinate system pairing state. At this time, the user may input the measurement end instruction to the terminal 1, or the terminal 1 may end the measurement by automatic determination. For example, when the terminal 1 determines that a predetermined rate or more of the target space 2 or the shared area has been measured or created based on the measurement data or the spatial data, the terminal 1 has measured or created the target space 2 or the shared area. It may be determined that the measurement is completed. The rate is a variable set value. When the terminal 1 determines that the measurement is completed (Yes), the terminal 1 proceeds to the next step, and when it is determined that the measurement is not completed (No), the terminal 1 returns to steps S7A and S7B and repeats the same process.

In steps S11A and S11B, each terminal 1 uses the space 2 in a state where the recognition of the space 2 is shared with the other terminal 1 by using the space data 6 created above. If the purpose is to create the spatial data 6, steps S11A and S11B can be omitted. The use of the space 2 is typically performed by displaying the same virtual image 22 at a desired same position 21 in the space 2 and performing work or the like by using the AR function between the terminals 1 (1A, 1B). (Fig. 5).

In steps S12A and S12B, each terminal 1 releases the state of the coordinate system pairing. For example, when the use of the space 2 is temporary, each terminal 1 may delete the conversion parameter 7 or the space data 6. Not limited to this, each terminal 1 may maintain the state of coordinate system pairing thereafter. That is, each terminal 1 may continue to hold the conversion parameter 7 and the spatial data 6 thereafter. In that case, steps S12A and S12B can be omitted. For example, each terminal 1 holds the conversion parameter 7 and the spatial data 6 inside its own machine, so that when the same space 2 is reused thereafter, processing such as re-measurement can be omitted.

[Guide display example]
FIG. 11 shows an image of a graphical user interface (GUI) for a guide or the like on the display surface 11 of the terminal 1 when pairing the coordinate system between the terminals 1 (steps S3A, S3B, etc. in FIG. 10). An example to display is shown. The example of FIG. 11 is an example of the display surface 11 of the first terminal 1A of the user U1, and the second terminal 1B of the user U2 is visible. The first terminal 1A recognizes another user or the terminal 1 based on, for example, an image of the camera 12. For example, the first terminal 1A superimposes and displays the image 1101 according to the position where the second terminal 1B is recognized. The image 1101 is a virtual image such as a marker indicating the existence and position of the second terminal 1B. Further, the first terminal 1A displays an image 1102 for confirming whether or not to perform coordinate system pairing with the second terminal 1B of the recognized user U2. The image 1102 is a message image such as "Do you want to pair with the user U2? YES / NO". The user U1 performs a YES / NO selection operation on the image 1102, and accordingly, the first terminal 1A determines whether or not to execute the coordinate system pairing with the second terminal 1B, and controls the start.

Further, the first terminal 1A displays the image 1103 when measuring various quantities 801 in step S4A. Image 1103 is, for example, a message image such as "Pairing. Please do not move as much as possible." In the case of direct coordinate system pairing between terminals 1, various quantities can be measured with high accuracy by keeping them as stationary as possible. Therefore, the output of the image 1103 of such a guide is effective.

[Coordinate transformation]
The details of the coordinate transformation will be supplementarily described below. First, the notation for explaining the relationship of the coordinate system is summarized. In the embodiment, the coordinate system is unified to the right-handed system, and a normalized quaternion is used to represent the rotation of the coordinate system. A normalized quaternion is a quaternion with a norm of 1 and can represent rotation around an axis. The rotation of any coordinate system can be represented by such a normalized quaternion. The normalized quaternion q representing the rotation of the angle η with the unit vector (n _X , n _Y , n _Z ) as the rotation axis is given by the following equation 1. i, j, and k are units of quaternions. The clockwise rotation when facing the direction of the unit vector (n _X , n _Y , n _Z ) is the positive rotation direction of η.
Equation 1: q = cos (η / 2) + n _X sin (η / 2) i + n _Y sin (η / 2) j + n _Z sin (η / 2) k

The real part of the quaternion q is represented by Sc (q). Let q * be the conjugate quaternion of the quaternion q. An operator that normalizes the norm of the quaternion q to 1 is defined by [・]. Assuming that the quaternion q is an arbitrary quaternion, Equation 2 is the definition of [・]. The denominator on the right side of Equation 2 is the norm of the quaternion q.
Equation 2: [q] = q / (qq *) ^1/2

Next, the quaternion p representing the coordinate points or vectors (p _X , p _Y , p _Z ) is defined by Equation 3.
Equation 3: p = p _X i + p _Y j + p _Z k

In this specification, unless otherwise specified, the symbols representing coordinate points and vectors that are not component display are quaternion display. Further, it is assumed that the symbol representing rotation is a normalized quaternion.

_{Let PT} (n) be a vector projection operator on a plane perpendicular to the direction of the unit vector n. The projection of the vector p is represented by Equation 4.
Equation 4: _PT (n) p = p + nSc (np)

Assuming that the coordinate point or direction vector p ₁ _{is converted into the coordinate point or direction vector p 2} by the rotation operation of the origin center represented by the quaternion q, the direction vector p ₂ can be calculated by Equation 5.
Equation 5: p ₂ = qp ₁ q *

To overlap the unit vector n ₁ to a unit vector n _2, a normalized quaternion to rotate around an axis perpendicular to a plane including the unit vector n ₁ and the unit vector _{_{n 2 R (n 1, n}} 2) is, It becomes the following formula 6.
Equation 6: R (n ₁ , n ₂ ) = [1-n ₂ n ₁ ]

FIG. 12 shows an explanatory diagram of the coordinate system transformation. Figure (A) in 12, similarly to FIG. 8, between the first terminal coordinate system WA and the second terminal coordinate system WB, and representation of the same position 21 in the real space, the coordinate origin (O _A, O _B ) is shown as an expression of the difference. As an expression of the position 21, with a position vector _{G A,} the position coordinates _{r A,} the position vector _{G B} and the position coordinates _{r B.} As an expression of the difference between the coordinate origins, the inter-origin vectors o _BA and o _AB are provided. Origin between the vectors _{o BA} is a representation of the origin _{O B} of the second terminal coordinate system WB in the first terminal coordinate system WA. Origin between the vectors _{o AB} is a representation of the origin _{O A} of the first terminal coordinate system WA in the second terminal coordinate system WB.

Based on the various quantities of the aforementioned (FIG. 8), the terminal coordinate system for the two different specific directions in real space (vectors between corresponding specific direction vector and a terminal) (WA, WB) expressed in (N _A _{_{_{, N B, P BA, P}}} AB) is obtained. Then, the rotation operation between the coordinate systems that matches those expressions can be obtained by the above-mentioned operation using the normalized quaternion. Therefore, by combining the information with the information of the origin of each coordinate, it is possible to convert the position coordinates between the terminal coordinate systems.

The relationship of the terminal coordinate system (WA, WB) can be calculated as follows. In the following, the calculation for obtaining the rotation and the coordinate origin difference in the case of converting the representation of the coordinate value and the vector value in the second terminal coordinate system WB into the representation in the first terminal coordinate system WA will be described.

FIG. 12B shows a rotation operation for aligning the direction between the first terminal coordinate system WA and the second terminal coordinate system WB, for example, each axis (X _B , Y _{B) of the second terminal coordinate system WB.} shows each axis _(X _a of the direction of the _{Z B)} first terminal coordinate system _WA, Y a, the image of the rotary _{q AB} to match the direction of the _{Z a)} in a simple manner.

First, the rotation for matching the direction of the first terminal coordinate system WA and the direction of the second terminal coordinate system WB is obtained. Terminal between the vectors _P BA in the foregoing (FIG. _8), based on _{P AB,} a unit direction vector _m A between the terminals 1 and _{m B,} defined as follows. Unit vector m _A, m _B is the second specific direction, from the first terminal 1A on the real space for the unit vector in the direction toward the second terminal 1B, expressed in the first terminal coordinate system WA and the It is an expression in the two-terminal coordinate system WB.
m _A = _{[P BA]}
m _B = [-P _AB ]

First, in the rotation in the representation of the first terminal coordinate system WA, consider the rotation q _{T1 in} _{which the unit vector n A} in the first specific direction is superimposed on the unit vector n _B. Specifically, the rotation q _T1 is as follows.
q _T1 = R (n _A , n _B )

Next, a direction unit vector _n A in a particular direction, the _{m A} is rotated by the rotation _{q _T1,} and n _{A1, m A1.}
n _A1 = q _T1 n _A q _T1 * = n _B
m _A1 = q _T1 m _A q _T1 *

Since the angles are between the same directions in the real space, the angle formed by the direction n _A1 and the direction m _A1 is equal to the angle formed by the unit vector n _B and the unit direction vector m _B. Further, since the two specific directions are different directions as a premise, the angle formed _{by the unit vector n B} and the unit direction vector m _{B is not 0.} Therefore, it is possible to construct a rotation q _{T2 in} _{which the direction n A1,} that is, the unit vector n _B is used as an axis and the direction m _A1 is superimposed on the unit direction vector m _B. Specifically, the rotation q _T2 is given below.
q _T2 = R ([ _PT (n _B ) m _A1 ], [ _PT (n _B ) m _B ])

Since the direction _{n A1} is the rotation axis direction _{n B} in the same direction of rotation _{q T2,} invariant This rotation _{q T2.} Further, the direction m _A1 is rotated _{by the rotation q T2 in the} unit direction vector m _B.
n _B = q _T2 n _A1 q _T2 *
m _B = q _T2 m _A1 q _T2 *

Again, the rotation q _BA is defined below.
q _BA = q _T2 q _T1

This rotation _{q BA,} the unit vector _{n A} and the unit direction vector _{m A} is rotated to a unit vector _{n B} and the unit direction vector _{m B.}
n _B = q _BA n _A q _BA *
m _B = q _BA m _A q _BA

Since the unit vector n _A and the unit direction vector m _A is selected as two different directions, the rotary q _BA is, the direction represented by the first terminal coordinate system WA in the direction represented by the second terminal coordinate system WB The rotation to convert. Conversely, when the rotation of converting the direction represented by the second terminal coordinate system WB in the direction represented by the first terminal coordinate system WA and rotated q _AB, rotating q _AB is as follows similarly.
q _AB = q _BA *

Next, determine the conversion formula of coordinate values _d A, _{d B} (FIG. 8). Here coordinate values d _A, d _B of is quaternion number representation of the coordinate values defined by Equation 3. First, the coordinate value of the origin of the other coordinate system when viewed from one coordinate system is obtained. As Figure 12 of (A), expressed in the coordinate values of the origin _{O B} of the second terminal coordinate system WB in the first terminal coordinate system WA is _{o BA,} the origin _{O A} world coordinate system WA in the world coordinate system WB The representation of the coordinate values is o _AB . Knowing the coordinates _d A, _{d B} of the position of the terminal 1 in each coordinate system, the origin coordinate value representation _{_(o BA,} o _AB) is determined as shown in Equation A below.
Formula A:
o _BA = d _A + P _BA -q _AB d _B q _AB *
_{_{_{_{o AB = d B + P AB}}}} -q BA d A q BA *

In addition, as can be easily understood, there are the following relationships.
o _AB = -q _BA o _BA q _BA *

Finally, the coordinate values r _A of the first terminal coordinate system WA for any point in the real space (position 21), conversion formula between the coordinate values r _B in the second terminal coordinate system WB is the following Given.
r _B = q _BA (r _A- o _BA ) q _BA * = q _BA r _A q _BA * + o _AB
r _A = q _AB (r _B- o _AB ) q _AB * = q _AB r _B q _AB * + o _BA

As described above, for example, when it is desired to convert _{a specific position 21 (coordinate value r A} ) seen in the first terminal coordinate system WA to _{a position 21 (coordinate value r B} ) seen in the second terminal coordinate system WB. , Rotation q _BA , coordinate value r _A , and origin representation o _AB . The reverse transformation can be calculated in the same way. The conversion parameters 7 (71, 72) of FIGS. 8 and 9 described above can be configured by the parameters introduced in the above description. Since they can be easily converted to each other as described above, in the configuration and holding of the conversion parameter 7 _{, q BA} may be held instead of the _{rotation q AB} _{, and o AB} may be used instead of the _{origin expression o BA.} It may be retained or vice versa.

[Effects (1)]
As described above, according to the spatial recognition system and method of the first embodiment, the terminal 1 can measure the space 2 and create the spatial data 6, and the spatial data can be created between the plurality of terminals 1 of a plurality of users. 6 can be acquired and used, and the recognition of space 2 can be shared. According to this system and method, the above-mentioned functions and operations can be efficiently realized, the convenience of the user can be improved, and the workload can be reduced. According to this system and method, by using the spatial data 6, various application functions and services can be realized for the user.

The following is also possible as a modification of the first embodiment. In the modified example, the terminal 1 of each user may transmit the spatial data 6 described in the terminal coordinate system created by the user to an external device such as a PC or a server and register the data. The terminal 1 may transmit the generated conversion parameter 7 to an external device such as a PC or a server and register it.

[Modification 1]
In the first modification of the first embodiment, each terminal 1 measures the space 2 and creates the spatial data 6 described in the terminal coordinate system of the own machine before performing the coordinate system pairing. After that, the terminal 1 performs coordinate system pairing with another terminal 1. The terminal 1 uses the conversion parameter 7 to convert the spatial data 6 into the common terminal coordinate system, that is, the spatial data 6 described in the common coordinate system.

[Modification 2]
FIG. 13 shows an explanatory diagram of coordinate system pairing and the like in the case of sharing the spatial recognition by sharing the measurement of the space 2 among three or more terminals 1 in the modified example 2 of the first embodiment. In this example,

terminals

1A, 1B, 1C, and 1D are provided as four terminals 1 of four users (UA, UB, UC, UD). The terminal coordinate system of each terminal 1, and the terminal coordinate system WA, WB, WC, and WD, the respective origin origin _{_{_{O A, O B, O C}}} , and _{O D.} These terminals 1 are grouped together to perform measurement and recognition sharing regarding the same space 2. Based on the coordinate system pairing between the two terminals 1 described in the first embodiment, even in the case of a group consisting of three or more terminals 1, space recognition is performed by performing the coordinate system pairing between the terminals 1. Sharing can be realized.

For example, consider the terminal 1C of the user UC as a own machine. First, as in the first embodiment, it is assumed that the coordinate system pairing 1301 is established between the terminal 1A and the terminal 1B, for example. From this state, it is assumed that the coordinate system pairing 1302 is then performed between the terminal 1B and the terminal 1C. Thereby, the coordinate system pairing 1303 between the terminal 1C and the terminal 1A can be indirectly realized. This will be described below.

_{First, by the coordinate system pairing 1301, the terminal 1B obtains the rotation q BA} and the origin expression o _AB for conversion between the terminal coordinate system WA and the terminal coordinate system WB as the information 1321 of the conversion parameter. The rotation q _BA is a rotation in which the expression in the terminal coordinate system WA is changed to the expression in the terminal coordinate system WB. Origin representation _{o AB} are the coordinate values of the terminal coordinate system WB about the origin _{O A} terminal coordinate system WA. On the contrary, the terminal 1A obtains the _{rotation q AB} and the origin expression o _{BA as the conversion parameter information 1311.}

Next, by the coordinate system pairing 1302, the terminal 1C obtains the _{rotation q CB} and the origin expression o _{BC as the conversion parameter information 1331.} The rotation q _CB is a rotation in which the expression in the terminal coordinate system WB is changed to the expression in the terminal coordinate system WC. Origin representation _{o BC} are the coordinate values of the terminal coordinate system WC for the origin _{O B} of the terminal coordinate system WB. On the contrary, the terminal 1B obtains the _{rotation q BC} and the origin expression o _{CB as the conversion parameter information 1322.}

Here, the terminal 1C receives the conversion parameter information 1321 (rotation q _BA and origin expression o _AB ) from the terminal 1B, and holds it as the information 1332. As a result, the terminal 1C uses the conversion parameter information 1331 (q _CB , o _BC ) and information 1332 (q _BA , o _AB _{) to rotate q CA} with respect to the indirect coordinate system pairing 1303 with the terminal 1A. The origin expression o _AC can be calculated by the following formula. The rotation q _CA is a rotation in which the expression in the terminal coordinate system WA is changed to the expression in the terminal coordinate system WC. Origin representation _{o AC} are the coordinate values of the terminal coordinate system WC for the origin _{O A} terminal coordinate system WA.
q _CA = q _CB q _BA
o _AC = o _BC + q _CB o _AB q _CB *

Terminal 1C holds the obtained information 1333 (q _CA , o _AC ). Using this information 1333, the terminal 1C can convert the representation of the position 21 in the terminal coordinate system WA (r _A ) into the representation in the terminal coordinate system WC (r _C ) as in the following equation.
r _C = q _CA (r _A- o _CA ) q _CA * = q _CA r _A q _CA * + o _AC

Further, the terminal 1C _{transmits the information 1333 (q CA} , o _AC ) of the conversion parameter to the terminal 1A. Terminal 1A holds it as information 1312 (q _CA , o _AC ). Then, since there is generally the following relationship, the terminal coordinate system WA and the terminal coordinate system WC can be converted even in the terminal 1A. That is, the terminal 1A holds the information 1313 (q _AC , o _CA ) of the conversion parameter related to the reverse conversion. Further, since there is the following relationship, each terminal 1 may hold either q _IJ _{or q JI} , and may hold one of o _JI and o I _J.
q _IJ = q _JI *
o _JI = -q _IJ o _IJ q _IJ *

FIG. 14 shows a table 1401 of conversion parameters 7 held by the terminal 1A, a table 1402 of conversion parameters 7 held by the terminal 1B, and a table of conversion parameters 7 held by the terminal 1C in the coordinate system pairing of the group of FIG. 1403 is shown. Each terminal 1 in the group holds conversion parameter information with each other terminal 1 in the group in a table. Each table has a "counterpart" item and stores the identification information of the other party's terminal 1 and the terminal coordinate system of the coordinate system pairing (here, including the direct coordinate system pairing and the indirect coordinate system pairing). .. For example, the terminal 1C holds the conversion parameter information between each pair while exchanging information with each terminal 1 (1A, 1B). Specifically, for example, the table 1403 has information 1333 (q _CA , o _AC _{) of the conversion parameter with the terminal 1A and information 1331 (q CB} , o _BC ) of the conversion parameter with the terminal 1B.

As described above, in the modification 2, the spatial recognition can be shared within the group by sequentially performing coordinate system pairing with any two terminals 1 as a pair. A certain terminal 1C performs a coordinate system pairing between the terminal 1A and another terminal 1B that has already been paired with the coordinate system without performing a direct coordinate system pairing process with the certain terminal 1A. For example, indirect coordinate system pairing 1303 is possible. Similarly, even if there is a terminal 1D newly joining the group, the terminal 1D may perform the same procedure for one terminal 1 in the group, for example, the coordinate system pairing 1304 with the terminal 1C. The processing of coordinate system pairing with the terminal 1 is unnecessary. In the first embodiment and the second modification, since the conversion parameter 7 is held in each terminal 1, high-speed processing can be performed when displaying the virtual image 22 at the shared position 21.

[Modification 3]
FIG. 15 shows a configuration example relating to the coordinate system pairing and the conversion parameter 7 in the modification 3 of the first embodiment. In the third modification, one representative terminal 1 (described as “representative terminal”) is provided in a group consisting of a plurality of terminals 1 for sharing spatial recognition. The representative terminal holds the conversion parameter 7 with each terminal 1 of the group. Each terminal 1 other than the representative terminal holds a conversion parameter 7 with the representative terminal. For example, suppose there is a group similar to that in FIG. For example, terminal 1A is a representative terminal. The terminal 1A, which is a representative terminal, sequentially performs coordinate system pairing (1501, 1502, 1503) with each of the other terminals 1 (1B, 1C, 1D). In this group, the terminal coordinate system WA of the representative terminal is used as a reference. In the terminal coordinate system WA, a shared position 21 or the like is designated and transmitted between terminals 1.

Table 1511 of the conversion parameters 7 held by the terminal 1A has conversion parameter information with each terminal 1 (1B, 1C, 1D) as in the table 1401 of FIG. The table 1512 held by the terminal 1B has conversion parameter information (q _BA , o _AB ) with the representative terminal. The table 1513 held by the terminal 1C has conversion parameter information (q _CA , o _AC ) with the representative terminal. The table 1514 held by the terminal 1D has conversion parameter information (q _DA , o _AD ) with the representative terminal.

For example, if the terminal 1B is to specify the position 21 (FIG. 13) in the space 2, the terminal 1B, the expression of its position 21 on the terminal coordinate system WB to _{(r B),} using the table 1512, the representative terminal Is converted into the expression (r _A ) and transmitted to the representative terminal. The representative terminal uses the _{table 1511 to express the expression (r A} ) in each terminal coordinate system (WC, WD) of each other terminal 1 (1C, 1D) in the group (r _C , r _D ). Convert to. Then, the representative terminal transmits the position information (r _C , r _D ) to each of the other terminals 1 (1C, 1D).

As another modification, it is possible that only the representative terminal holds the conversion parameter 7 and performs each conversion. This modification corresponds to, for example, FIG. 15, in which

terminals

1B, 1C, and 1D do not hold tables 1512, 1513, and 1514 of conversion parameters 7. For example terminal 1B representing the position 21 of the terminal coordinate system WB to _{(r B),} and transmits the representative terminal. The representative terminal converts the expression (r _B _{) into the expression (r A} , r _C , r _D ) in each terminal coordinate system (WA, WC, WD) using the table 1511, and converts it into each terminal 1. introduce.

As another modification, the terminal coordinate system of the representative terminal may be fixed as a common coordinate system in the group, and the position may be transmitted between the terminals 1. The representative terminal does not hold the conversion parameter 7. Each terminal 1 other than the representative terminal holds a conversion parameter 7 for conversion of the representative terminal with the terminal coordinate system. This modification corresponds to, for example, in FIG. 15, a configuration in which terminal 1A, which is a representative terminal, does not hold the table 1511. _{For example, the terminal 1B converts the representation (r B} ) of the position 21 in the terminal coordinate system WB _{into the representation (r A} ) in the representative terminal using the table 1512, and transmits the representation to the representative terminal. The representative terminal _{transmits the expression (r A} ) to each of the other terminals 1 (1C, 1D) in the group. Each terminal 1 (1C, 1D) _{converts its representation (r A} _{) into a representation (r C} , r _D ) in its own terminal coordinate system using its own tables 1513, 1514.

Further, in this modification, the position transmission between the terminals 1 may be performed without going through the representative terminal. _{For example, the terminal 1B converts the representation (r B} ) of the position 21 in the terminal coordinate system WB _{into the representation (r A} ) in the representative terminal using the table 1512, and transmits the representation to the terminal 1C. The terminal 1C converts the _{expression (r A} _{) into the expression (r C} ) of its own machine using the table 1513.

As described above, according to each modification, the amount of data of the conversion parameter 7 held in the entire system can be reduced.

<Embodiment 2>
The space recognition system and the like according to the second embodiment of the present invention will be described with reference to FIGS. 16 to 17 and the like. Hereinafter, the components different from those of the first embodiment in the second embodiment and the like will be described. In the second embodiment shown in FIG. 16 and the like, a space coordinate system which is a world coordinate system describing the space 2 is used separately from each terminal coordinate system of the plurality of terminals 1 in the first embodiment. The second embodiment deals with coordinate system pairing between the terminal coordinate system and the spatial coordinate system, in other words, association and conversion between those coordinate systems. In this embodiment, the spatial coordinate system corresponds to the common coordinate system of the basic configuration (FIG. 31). Each terminal coordinate system of each terminal 1 that shares the measurement of space 2 is associated with each other via a common spatial coordinate system. Further, the spatial data 6 of the space 2 can be described in particular by using a common spatial coordinate system. The terminal 1 creates the spatial data 6 described in the spatial coordinate system. The recognition of the space 2 can be shared between the terminals 1 by using the space data 6.

In the second embodiment, at the time of coordinate system pairing, the terminal 1 measures the relationship with a predetermined feature (feature point or feature line) in the space 2 as various quantities. The terminal 1 obtains the relationship between the spatial coordinate system associated with the feature and the terminal coordinate system of the own machine based on the measured value, and calculates the conversion parameter 7 based on the relationship.

Further, in the second embodiment, the terminal 1 may register the created spatial data 6 in the DB 5 of the external server 4. In this case, the server 4 corresponds to the information processing device 9 having the basic configuration (FIG. 31). The second embodiment deals with the concept of registering the spatial data 6 from the terminal 1 to an external source such as the server 4. The server 4 holds and manages spatial data 6, which is external data, as an external source for the terminal 1. The spatial data 6 registered as a library in the DB 5 of the server 4 can be appropriately referred to and acquired from each terminal 1 (a terminal that does not perform spatial measurement may be used). The terminal 1 acquires the registered spatial data 6 regarding the space 2 to be used from the server 4, and uses the spatial data 6 to quickly obtain an image such as AR without requiring measurement of the space 2. It can be displayed with high accuracy. For example, a certain terminal 1 measures a certain space 2 once, creates spatial data 6, and registers it in the server 4. After that, when the space 2 is used again, the terminal 1 does not need to measure the space 2 again and can use the space data 6 acquired from the server 4. The business operator may use the server 4 to provide the management service of the spatial data 6.

[Spatial recognition system]
FIG. 16 shows the configuration of the space recognition system of the second embodiment, and in particular, illustrates the coordinate system pairing between the first terminal coordinate system WA of the first terminal 1A and the space coordinate system W1 of the space 2. show. In this example, as the terminal 1 that shares the measurement of the space 2, the first terminal 1A and the second terminal 1B are provided. The second terminal coordinate system WB and the like of the second terminal 1B are not shown.

In the second embodiment, the information of the space coordinate system W1 regarding the space 2 is defined in advance. In the space coordinate system W1, information such as the position of the space 2 and predetermined features (feature points and feature lines) is also defined. The spatial coordinate system W1 may be, for example, a local coordinate system peculiar to a building, or a coordinate system common to the earth, a region, or the like. Space coordinate system W1 has is fixed in the real space, the origin _{O 1,} the axis _{X 1} as 3 orthogonal axes, the axis _{Y 1,} and a shaft _{Z 1.} In the example of FIG. 16, the origin O ₁ of the spatial coordinate system W1 is located at a position distant from the space 2 such as a room, but the origin O ₁ may be in the space 2.

In the second embodiment, the coordinate system pairing between the terminal coordinate system (WA, WB) of each terminal 1 and the spatial coordinate system W1 of the space 2 is handled. Those terminals 1 (1A, 1B) share the recognition of the space 2 by using the spatial data 6 created by sharing. Each terminal 1 measures the shape and the like of the space 2 in the terminal coordinate system of its own machine and creates the space data 6 (particularly the space shape data) that describes the space 2. At that time, each terminal 1 performs coordinate system pairing with the space coordinate system W1 by using a predetermined feature in the space 2 as a clue. The feature points, feature lines, and the like, which are predetermined features in the space 2, are defined in advance. This feature may be, for example, a boundary line such as a wall or a ceiling, or a predetermined arrangement or the like. The feature points in the predetermined features of the space 2 have different meanings from the feature points of the point cloud data obtained by the distance measuring sensor 13 described above.

For example, the first terminal 1A recognizes a predetermined feature of the space 2 and measures various quantities to grasp the relationship between the first terminal coordinate system WA and the space coordinate system W1. Based on this relationship, the first terminal 1A generates a conversion parameter 7 between the first terminal coordinate system WA and the spatial coordinate system W1 and sets it in its own machine. Each terminal 1 measures the shared area in the space 2 in the state of the coordinate system pairing. For example, the first terminal 1A measures the region 2A and obtains the measurement data 1601 described in the first terminal coordinate system WA. The first terminal 1A constitutes the subspace data 1602 from the measurement data 1601. The first terminal 1A converts the subspace data 1602 into the subspace data described in the spatial coordinate system W1 by using the conversion parameter 7. Further, for example, the first terminal 1A acquires the subspace data created by the second terminal 1B from the second terminal 1B. Then, the first terminal 1A integrates the subspace data acquired on the own machine side and the subspace data acquired from the other party into one, so that the space 2 described in the spatial coordinate system W1 is used as a unit. Obtain the created spatial data 6. The second terminal 1B side can also obtain the spatial data 6 in the same manner as the first terminal 1A side.

In FIG. 16, the space recognition system of the second embodiment has a server 4 connected to a communication network. The server 4 is a server device managed by a business operator or the like, and is provided on, for example, a data center or a cloud computing system. The server 4 registers and holds the ID and the spatial data 6 as a library in the internal or external database (DB) 5. For example, ID = 101 is assigned to the illustrated space 2, and space data 6 (D101) identified by ID = 101 is registered in DB5. Spatial data 6 is similarly registered for each of the plurality of spaces 2. The server 4 may manage the spatial data 6 closed in a unit such as a company, or may manage a large number of spatial data 6 in a unit such as the earth or a region. For example, when the spatial data 6 is managed in units of a company building, each spatial data 6 relating to each space 2 in the building is registered in the server 4 of a computer system such as a corporate LAN.

In the second embodiment, in particular, the spatial data 6 relating to each space 2 in the real space is registered as a library in the DB 5 of the server 4 which is an external source. At first, before the space 2 is measured, the space shape data 61 of the space data 6 of the DB 5 is not registered. The spatial data 6 of the DB 5 includes the spatial shape data 61 and the feature data 62. The space shape data 61 is data representing the shape of the space 2 and the like described in the space coordinate system W1, and is a part created by the terminal 1. The feature data 62 includes data that defines various quantities of predetermined features (feature points, feature lines, etc.) in the space 2. The feature data 62 is referred to when the coordinate system is paired by the terminal 1.

The spatial data 6 of the DB 5 may be described in a unique spatial coordinate system corresponding to the space 2, or may be described in a common spatial coordinate system in a plurality of related spaces 2 (for example, a building). The common spatial coordinate system may be a common coordinate system within the earth or a region. For example, a coordinate system using latitude, longitude, and altitude in GPS or the like may be used.

Note that the configuration of this spatial data 6 is an example, and the details are not limited. As data different from the spatial data 6, there may be predetermined spatial coordinate system W1 and data related to features / quantities. Feature data 62 may be described as part of the spatial shape data 61. The feature data 62 may be held in the terminal 1 in advance. Various types of data may be held in different locations and associated with each other through identification information. The server 4 is not limited to one, and may be a plurality of servers 4, and may be, for example, a server 4 associated with each one or more spaces 2.

In particular, in the second embodiment, each terminal 1 can register the spatial data 6 created by the measurement of the space 2 in the DB 5 of the server 4. At that time, the spatial data 6 created by the terminal 1 is registered with respect to the spatial data 6 (particularly the spatial shape data 61) registered in the DB 5 in advance. In other words, the content of the spatial data 6 of the server 4 is appropriately updated according to the registration of the spatial data 6 from the terminal 1. Then, each terminal 1 can appropriately acquire and use the registered spatial data 6 from the DB 5 of the server 4. Each terminal 1 does not have to hold the spatial data 6 inside its own device.

The second embodiment is a case where the spatial data 6 of each space 2 is registered as a library in an external source such as a server 4, but the present invention is not limited to this, and the spatial data 6 is held in the terminal 1 as a library. May be good. Each terminal 1 that shares the recognition of the space 2 may only create, exchange, share, and hold the spatial data 6 between the terminals 1.

[Coordinate transformation]
FIG. 17 shows an explanatory diagram of the coordinate system pairing between the terminal coordinate system WA and the spatial coordinate system W1 in the second embodiment. In the second embodiment, as a predetermined feature (in other words, a feature) in the space 2, a feature point or a feature line in a predetermined object 1700 such as a wall or a ceiling is used. The terminal 1 uses the predetermined feature points and feature lines when pairing the coordinate system with the spatial coordinate system W1. In the example of FIG. 17, the points at the four corners of the rectangular surface of the object 1700 such as a wall are used. In the example of FIG. 17, three feature points and two feature lines corresponding to the left and top sides of the surface of the object 1700 are used in particular. The two feature lines correspond to two specific directions. The predetermined feature in the space 2 is defined by the feature data 62 (FIG. 16), and may be arbitrary as long as the terminal 1 can be recognized by a camera, a sensor, or the like. The predetermined feature is not limited to a wall or the like, and may be, for example, a predetermined object set by the user in the room.

In this example, the position of the origin O _A terminal coordinate system WA, unlike the first position LA of the terminal 1, also the position of the feature point in the position and space 2 of the origin O ₁ space coordinate system W1 It is different from L1 but is not limited to this. Hereinafter, a case where the origin of the terminal coordinate system and the position of the terminal 1 do not match and a case where the position of the origin of the spatial coordinate system and the position of the feature point in the space 2 do not match will be described.

_{Let d A} = (x _A , y _A , z _A ) be the coordinate value of the position LA of the terminal 1 in the terminal coordinate system WA. _{Let d 1} = (x ₁ , y ₁ , z ₁ ) be the coordinate value in the space coordinate system W1 for the position L1 of the feature point in space 2. These coordinate values are determined according to the settings of the world coordinate system. Terminal position vector _{V A} is the vector from the origin _{O A} to the position LA. The feature point position vector V ₁ is a vector from the origin O ₁ to the position L1.

At the time of coordinate system pairing, the terminal 1 acquires information on the spatial coordinate system W1 from the server 4 (or the reference terminal in the modified example). For example, the terminal 1 refers to the feature data 62 of the spatial data 6 from the server 4. The feature data 62 includes data of various quantities 1702 relating to features on the space 2 side (corresponding object 1700). The terminal 1 measures various quantities 1701 on the own machine side by using the distance measuring sensor 13 and the like. The terminal 1 obtains the relationship between the terminal coordinate system WA and the space coordinate system W1 based on the quantities 1702 on the space 2 side and the measured quantities 1701 on the own machine side. Based on the relationship, the terminal 1 calculates the conversion parameter 7 between those coordinate systems and sets it in its own machine.

As various quantities at the time of coordinate system pairing, it has information on the following three elements. The quantities have a specific direction vector as the first information, a world coordinate value as the second information, and a spatial position vector as the third information. In the example of FIG. 17, having a various amount 1701 of own device side, a first specific direction vector _{N A,} a second specific direction vector _{M A,} the coordinate value _{d A,} and the spatial position vector _{P 1A.} As various quantities on the space 2 side, it has a first specific direction vector N ₁ , a second specific direction vector M ₁ , and a coordinate value d ₁ .

(1) Regarding the specific direction vector: The terminal 1 uses the specific direction vector as information regarding the specific direction in the space 2 in the terminal coordinate system. This specific direction includes a direction measured by a sensor of the terminal 1, such as a vertically downward direction, and a direction corresponding to a feature line in space 2, for example, a direction corresponding to the left side or the upper side of the object 1700. The terminal 1 may use two different unit vectors in a specific direction from a plurality of candidates. The representation in the spatial coordinate system W1 for these unit vectors and _n _{1, m} 1, is a representation of the terminal coordinate system WA _n A, and _{m A.} Unit vector _n A in the terminal coordinate system WA, for _{m A,} measured by the terminal 1. _{The unit vectors n 1} and m ₁ in the spatial coordinate system W1 are defined in advance and can be obtained from the feature data 62 of the server 4.

When the vertical downward direction is used as one specific direction, the vertical downward direction can be measured as the direction of gravitational acceleration using an acceleration sensor as described above. Alternatively, in the setting of the world coordinate system (WA, W1), a vertically downward direction, Z-axis _{_(Z} A, Z ₁₎ may be set as a negative direction. In any case, since this vertical downward direction does not change in the world coordinate system, it is not necessary to measure each coordinate system pairing.

When, for example, the north direction of the geomagnetism is used as one specific direction, the north direction of the geomagnetism can be measured by using the geomagnetism sensor 143 (FIG. 7) provided in the terminal 1. Since the geomagnetism may be affected by the structure, it is preferable to measure each coordinate system pairing. If it is known that the influence of the structure is sufficiently small, the measurement may be omitted and the direction recognized as the north direction of the geomagnetism may be used.

When the directions of the predetermined feature lines in the space 2 are used as the specific directions, for example, when the directions of the two feature lines on the left side and the upper side of the object 1700 are used as the two specific directions, the measurement can be performed as follows. The terminal 1 measures the position coordinate values in the terminal coordinate system WA for each of the feature lines for two different feature points constituting the feature line. Terminal 1, the direction vector from the measured values determined (e.g. direction vector _N A corresponding to the left side _(n A), the direction vector _M A corresponding to the upper side _(m A)). This coordinate value can be measured by, for example, the distance measuring sensor 13 of the terminal 1.

(2) World coordinate values: The terminal 1 uses coordinate value information representing a position in the terminal coordinate system. In the example of FIG. 17, as the world coordinate values, the coordinate values d _A at the first terminal coordinate system _WA, and using the coordinate values d ₁ in the spatial coordinate system W1. In this example, as a feature of the object 1700, one feature point on the upper left is set to the position L1 (coordinate value d ₁ ).

(3) Spatial position vector: The spatial position vector (spatial position vector _P1A ) is a vector from the position LA of the terminal 1 to the position L1 of the feature point in the space 2. From this spatial position vector, information on the positional relationship between the two coordinate systems (WA, W1) can be obtained. This spatial position vector can be measured by, for example, the distance measuring sensor 13 of the terminal 1.

In Figure 17, the position vector _{G A} is the vector position 21 of the first terminal coordinate system WA, position coordinates _{r A} is the coordinate value of the position 21. The position vector G ₁ is a vector of the position 21 in the spatial coordinate system W1, and the position coordinate value r ₁ is a coordinate value of the position 21. The inter-origin vector o _1A is a vector from the origin O _A to the origin O ₁ , and is a representation _{of the origin O 1} in the first terminal coordinate system WA. Origin between the vectors _{o A1} is a vector from the origin _{O 1} to the origin _{O A,} is a representation of the origin _{O A} in the spatial coordinate system W1.

[transform]
Since the relationship between the first terminal coordinate system WA and the spatial coordinate system W1 can be known from the above-mentioned various amount data (1701,1702), the conversion between those world coordinate systems (WA, W1) can be calculated. That is, as the conversion parameter 7, the conversion parameter 73 for matching the spatial coordinate system W1 to the first terminal coordinate system WA, and vice versa, the conversion for matching the first terminal coordinate system WA to the spatial coordinate system W1. The conversion parameter 74 for this can be configured. This conversion parameter 7 can be defined by using rotation and the coordinate origin difference, as in the description in the first embodiment.

After pairing the coordinate system, any world coordinate system may be used for the recognition of the position in the space 2 by the terminal 1. The position in the spatial coordinate system W1 may be converted to the position in the first terminal coordinate system WA by the conversion parameter 73. The position in the first terminal coordinate system WA may be converted to the position in the spatial coordinate system W1 by the conversion parameter 74.

The table of transformation parameters 73 in the example of FIG. 17 has spatial coordinate systems, terminal coordinate systems, rotations, and origin representations as items. The "spatial coordinate system" item stores the identification information of the spatial coordinate system. The "terminal coordinate system" item stores the identification information of the terminal coordinate system or the corresponding identification information of the terminal 1 or the user. The "Rotation" item stores information on the representation of rotation between those spatial coordinate systems and the terminal coordinate system (eg q _A1 ). The "origin expression" item stores information on the expression of the difference between the origin of the spatial coordinate system and the origin of the terminal coordinate system (example: o _1A ).

Since the calculation method of the conversion parameter 7 in the second embodiment is the same as that in the first embodiment, only the calculation result will be described below. And coordinate values r _A of the terminal coordinate system WA for any point in space 2 (position 21), conversion formula between the coordinate values r ₁ in the spatial coordinate system W1 is given as follows.
r ₁ = q _1A (r _A- o _1A ) q _1A * = q _1A r _A q _1A * + o A ₁
r _A = q _A1 (r _1- o _A1 ) q _A1 * = q _A1 r ₁ q _A1 * + o _1A

However, the quantities in the above formula are given from the following.
q _T1 = R (n _A , n ₁ )
m _A1 = q _T1 m _A q _T1 *
q _T2 = R ([ _PT (n ₁ ) m _A1 ], [ _PT (n ₁ ) m ₁ ])
q _1A = q _T2 q _T1
q _A1 = q _1A *
o _1A = d _A + P _1A -q _A1 d ₁ q _A1 *
o _A1 = d _1- q _1A (d _A + P _1A ) q _1A *

As described above, for example, when it is desired to convert _{the position 21 (coordinate value r A} ) seen in the first terminal coordinate system WA to the position 21 (coordinate value r ₁ _{) seen in the spatial coordinate system W1, rotation q 1 A} , coordinates. It can be calculated using the value r _A and the origin representation (o _A1). The reverse transformation can be calculated in the same way. The conversion parameter 7 in the second embodiment can be configured by the parameters introduced in the above description. In the configuration and retention of the conversion parameter 7, as in the first embodiment, since they can be easily converted to each other, for example _{, q 1A} may be used instead of the _{rotation q A1.}

[Effects (2)]
As described above, according to the second embodiment, each terminal 1 can create spatial data 6 in accordance with the spatial coordinate system W1 of the space 2 as the common coordinate system and register it in the server 4, and can be registered by a plurality of users. The recognition of space 2 can be shared among a plurality of terminals 1.

The following is also possible as a modification of the second embodiment. In the modified example, the terminal 1 measures the space 2 and creates the spatial data 6 described in the terminal coordinate system of the own machine before performing the coordinate system pairing. After that, the terminal 1 performs coordinate system pairing with the spatial coordinate system W1 and uses the conversion parameter 7 to convert the spatial data 6 described in the terminal coordinate system into the spatial data 6 described in the spatial coordinate system W1. Convert.

As a modification of

Embodiments

1 and 2, the following is also possible. The information provided between the terminals 1 or between the terminals 1 and the server 4 may include data such as a virtual image (AR object) related to a function such as AR and arrangement position information of the virtual image. For example, in FIG. 16, such data may be exchanged between the server 4 and each terminal 1 through the spatial data 6. Data such as an AR object may be provided from the terminal 1 to the server 4 and registered in association with the spatial data 6. Data such as an AR object may be provided from the server 4 to the terminal 1 together with the spatial data 6. In the library of DB5, in the spatial data 6, the data of the AR object to be arranged and displayed in the space 2 and the arrangement position information and the like are registered in association with the spatial shape data 61 and the like. As a result, various services can be provided to the user through the terminal 1. For example, a store that sells products (corresponding space 2) can provide the terminal 1 with an AR object such as a product advertisement together with the space data 6.

[Modification example 4]
FIG. 18 shows the configuration of a modified example (referred to as modified example 4) of the second embodiment. Of a plurality of terminals 1 to be shared and shared, a specific terminal 1 is used as a reference (described as a "reference terminal"), and the terminal coordinate system of the reference terminal is used as a reference (referred to as a "reference coordinate system"). You may. In that case, the reference terminal measures and holds the features (directions of feature points and feature lines) of the space 2 as various quantity data 1800 in the reference coordinate system. The reference terminal performs coordinate system pairing 1801 with the space coordinate system W1 in space 2. Each terminal 1 other than the reference terminal, for example, the second terminal 1B, receives various quantity data 1800 from the reference terminal and performs coordinate system pairing 1802 with the reference terminal. This coordinate system pairing 1802 is the same as the coordinate system pairing described in the first embodiment. As a result, each terminal 1 that has undergone coordinate system pairing with the reference coordinate system realizes indirect coordinate system pairing with the spatial coordinate system W1 via the reference coordinate system.

<Embodiment 3>
The space recognition system and the like according to the third embodiment of the present invention will be described with reference to FIGS. 19 to 24 and the like. The third embodiment shown in FIG. 19 and the like is an advanced form of the second embodiment, and the point of handling the coordinate system pairing between the terminal coordinate system and the spatial coordinate system is the same. The unique features of the label 3 are used for measurement and the like. In the third embodiment, the terminal 1 uses the space coordinate system W1 related to the sign 3 to measure the space 2 and create the space data 6. Further, the terminal 1 may register / store the created spatial data 6 in the DB 5 of the server 4.

[Spatial recognition system and method]
FIG. 19 shows the configuration of the spatial recognition system and method of the third embodiment. The spatial recognition system of embodiment 3 has a marker 3. In the space 2, a sign 3 associated with the space 2 is installed. In the example of FIG. 19, a sign 3 is installed in the space 2 which is a room, for example, on the outer surface of the entrance wall 1901.

The sign 3 (in other words, a marker, a sign, etc.) has a special function for the terminal 1 in addition to a function as a general sign that allows the user to identify the space 2. The sign 3 gives the world coordinate system as a reference for the space 2 as the space coordinate system W1 (may be called a sign coordinate system). The sign 3 is a peculiar object that has predetermined features and can be used for measuring various quantities at the time of coordinate system pairing by the terminal 1. Further, the sign 3 has a function for enabling the terminal 1 to identify the space 2 (corresponding ID) and acquire the space data 6. The position, shape, and the like of the sign 3 are described in the same space coordinate system W1 as that of the space 2. The features in the space 2 in the second embodiment are feature points and feature lines as the features of the sign 3 in the third embodiment. The characteristics of this sign 3 are defined in advance as various quantities. For example, the indicator data 62 is registered in the spatial data 6 of the DB 5 of the server 4. The label data 62 includes various amount data of the label 3 and corresponds to the feature data 62 in the second embodiment.

The terminal 1, for example, the first terminal 1A measures the feature of the marker 3 as various quantity data on the own machine side, grasps the relationship between the first terminal coordinate system WA and the spatial coordinate system W1, and is based on the relationship. Then, the conversion parameter 7 between the first terminal coordinate system WA and the spatial coordinate system W1 is generated and set in the own machine.

[Sign]
FIG. 20 shows a configuration example of the sign 3. (A) is the first example, (B) is the second example, (C) is the third example, and (D) is the fourth example. In (A), the sign 3 is composed of a horizontally long rectangular plate or the like, and the surface of the plate or the like (sometimes referred to as a sign surface) indicates the name of the room which is the space 2. "7 meeting room" character string is described. In this example, the label surface is arranged in Y ₁ -Z ₁ side of the spatial coordinate system W1. Further, in this example, the ID 2001 of the space 2 and the sign 3 is directly described as a character string in one place on the sign surface. The terminal 1 can recognize the ID 2001 by the camera 12.

In this example, the feature points and feature lines in the spatial coordinate system W1 are defined in advance on the sign surface of the sign 3. On the marking surface, one feature point (point p1) representing the representative position L1 of the marking 3 is defined. In addition, two other feature points (points p2 and p3) are defined on the marking surface. Two feature lines (lines v1 and v2 corresponding to the vector) are defined by the three feature points (points p1 to p3). The point p1 is the upper left corner point of the marking surface, the point p2 is the lower left corner point, and the point p3 is the upper right corner point. The line v1 is the left side of the sign plane, and the line v2 is the upper side. These feature points and feature lines constitute the above-mentioned two specific directions. The various quantity data regarding the spatial coordinate system W1 of the marker 3 includes, for example, information of the above-mentioned one feature point (point p1) and two specific directions (lines v1, v2). For the sake of explanation, feature points such as point p1 and feature lines such as line v1 are shown, but they are not actually described. Alternatively, the feature points and feature points may be intentionally described as specific images on the sign surface so that the user and the terminal 1 can recognize them.

The terminal 1 measures the relationship with the marker 3 as various quantities at the time of coordinate system pairing. At that time, the terminal 1 measures these three feature points (points p1 to p3) using the distance measuring sensor 13 and the camera 12 based on the marking data 62. In other words, the terminal 1 measures two feature lines (lines v1, v2). When the positions of the three feature points in the terminal coordinate system WA can be grasped, it is the same as when the two feature lines corresponding to the two specific directions can be grasped.

_{The origin O 1} of the space coordinate system W1 may be outside the space 2, inside the space 2, or particularly set on the sign surface of the sign 3. For example, the origin O ₁ may be set to match the characteristic points of the marker 3 (point p1).

In (B), a predetermined code (code image) 2002 is written on the sign 3 at one of the same sign surfaces as in (A), for example, in the vicinity of the upper left point p1. This code 2002 is a code that describes predetermined information. As this code 2002, for example, a two-dimensional code such as a QR code (QR: Quick Response, registered trademark) may be used. The terminal 1 extracts the code 2002 from the image of the camera 12 and obtains predetermined information by decoding.

In (C), the sign 3 is configured as an image or medium of the code 2003. For example, the sign 3 may be a sticking medium on which a QR code is written. In this example, the character string of the room name is described on the code 2003 side. The terminal 1 may measure in the same manner, for example, using the three corner points of the code 2003 as feature points. Alternatively, the terminal 1 may measure three cutout symbols for recognizing the QR code as feature points.

In (D), the sign 3 is composed of a display image of a display device 2004 (for example, a wall-mounted display). The code 2005 is displayed on the screen of the display device 2004 and functions as a sign 3. In this case, the sign 3 can be easily changed.

The predetermined information described in the sign 3 may be information including the space 2 and the ID 2001 that identifies the sign 3, or as information including an address and a URL for accessing the space data 6 of the server 4 which is an external source. Alternatively, the configuration may be as follows.

The predetermined information may be information including various quantity data (label data 62 in FIG. 19) relating to the spatial coordinate system W1 of the marker 3 and spatial data transmission destination information. The spatial data transmission destination information is external source information, and is transmission destination identification information for spatial data 6 (particularly spatial shape data) measured and created by the terminal 1, for example, an address or URL of a server 4.

The predetermined information may be information including a predetermined ID and spatial data transmission destination information. Using this information, the terminal 1 can access the server 4 and acquire the spatial data 6 (particularly the sign data 62) associated with the sign 3. Then, the terminal 1 can acquire various quantity data from the indicator data 62.

[Spatial data registration]
In FIG. 19, as a space recognition method of the third embodiment, a processing flow example in which a plurality of terminals 1 measure the space 2 by sharing, create space data 6 in units of the space 2, and register the space data 6 in the server 6. Is as follows. First, in step S31, the terminal 1 (for example, the first terminal 1A) recognizes the sign 3 in the real space by the camera 12 or the like, and measures various quantities for the feature of the sign 3. The terminal 1 performs coordinate system pairing between the terminal coordinate system WA of its own machine and the spatial coordinate system W1 of its marker 3 by using various quantity data which are measured values. As a result, the terminal 1 sets the conversion parameter 7 between the terminal coordinate system WA and the spatial coordinate system W1 as the common coordinate system.

Next, in step S32, the terminal 1 measures the space 2 (the above-mentioned shared area) in the terminal coordinate system WA, and creates the space data 6 described in the space coordinate system W1 using the conversion parameter 7. .. The terminal 1 appropriately converts the position or the like in the terminal coordinate system WA in the measurement data or the subspace data into the position or the like in the spatial coordinate system W1. The details of the process in step S32 are the same as described above.

In step S33, the terminal 1 transmits the spatial data 6 described in the created spatial coordinate system W1 to the server 4 based on the predetermined information of the sign 3. The terminal 1 may attach identification information of the own device or the user, position information (measurement starting point), measurement date / time information (time stamp), and other related information to the spatial data 6 to be transmitted. When there is measurement date and time information, the server 4 can grasp the change of the spatial data 6 (state such as the shape of the space 2) on the time axis as data management.

The server 4 registers and stores the spatial data 6 (particularly the spatial shape data) received from the terminal 1 in the library of the DB 5. The server 4 registers the spatial data 6 (particularly the spatial shape data 61) in association with information such as the ID of the space 2. When the corresponding spatial data 6 (particularly the spatial shape data 61) has already been registered in the DB 5, the server 4 updates the contents of the spatial data 6. The server 4 manages the measurement date / time, registration date / time, update date / time, etc. of the spatial data 6.

Alternatively, the following may be used. In steps S32 to S33, the terminal 1 creates the spatial data 6 described in the terminal coordinate system WA of the own machine based on the measurement data. Then, the terminal 1 sets the spatial data 6 described in the terminal coordinate system WA and the conversion parameter 7 (conversion parameter capable of converting the terminal coordinate system WA to the spatial coordinate system W1) as a set of the server 4 Send to. The server 4 registers the data in the DB 5.

[Control flow]
FIG. 21 shows an example of a processing flow of an exchange regarding registration of spatial data 6 between the terminal 1 and the server 4 in the third embodiment. In this example, the communication connection is established between the terminal 1 and the server 4 based on the sign 3, and the coordinate system pairing is established. In that state, the terminal 1 measures the space, creates the space data 6, sends it to the server 4, and registers it. The flow is the same for a plurality of terminals 1 of a plurality of users who share the measurement of the space 2.

In step S301, the terminal 1 recognizes the sign 3 and reads predetermined information (for example, ID, spatial data transmission destination information), and establishes a communication connection with the server 4 based on the predetermined information. In step S301b, the server 4 establishes a communication connection with the terminal 1. At this time, the server 4 may authenticate the user and the terminal 1, confirm the authority regarding the space 2, and permit the terminal 1 for which the authority has been confirmed. As the authority, for example, measurement authority, registration / update authority of spatial data 6, acquisition / use authority of spatial data 6, and the like may be provided.

In step S302, the terminal 1 transmits a coordinate system pairing request to the server 4, and in step S302b, the server 4 transmits a coordinate system pairing response to the terminal 1.

In step S303, the terminal 1 transmits a request for various amounts of data related to the sign 3 to the server 4. In step S303b, the server 4 transmits the corresponding indicator data 62 to the terminal 1 in response to the quantity data relating to the indicator 3. The terminal 1 acquires various quantity data related to the sign 3.

In step S304, the terminal 1 measures predetermined features of the sign 3 (points p1 and lines v1 and v2 in FIG. 20) in the terminal coordinate system WA based on the acquired various amount data, and various items on the own machine side. Obtained as quantitative data. The measurement at this time is possible by the distance measuring sensor 13.

In step S305, the terminal 1 has the various quantities data described in the spatial coordinate system W1 on the marker side obtained in step S303 and the various quantities described in the terminal coordinate system WA on the own machine side obtained in step S304. Using the data, the conversion parameter 7 between the terminal coordinate system WA and the spatial coordinate system W1 is calculated and set in the own machine.

In step S306, the terminal 1 measures the space 2, obtains the measurement data, and creates the space data 6 (particularly the space shape data) described in the terminal coordinate system WA of the own machine. In detail, the spatial data 6 is subspace data by sharing.

In step S307, the terminal 1 converts the spatial data 6 created in step S306 into the spatial data 6 described in the spatial coordinate system W1 using the conversion parameter 7.

In step S308, the terminal 1 transmits the spatial data 6 obtained in step S307 to the server 4. In step S308b, the server 4 registers or updates the spatial data 6 received from the terminal 1 with the corresponding spatial data 6 (particularly the spatial shape data 61) in the DB 5.

In another method, instead of steps S307 and S308, the terminal 1 transmits the spatial data 6 described in the terminal coordinate system WA of the own machine and the conversion parameter 7 as a set to the server 4. The server 4 registers the spatial data 6 and the conversion parameter 7 in association with each other in the DB 5. In the case of this form, the server 4 may perform the coordinate conversion process using the conversion parameter 7 of the DB 5.

In steps S309 and S309b, the terminal 1 and the server 4 confirm whether or not to end the coordinate system pairing related to the spatial measurement, and if it ends (Yes), proceed to S310, and if it continues (No), proceed to S310. The process returns to step S306 and repeats in the same manner.

In steps S310 and S310b, the terminal 1 and the server 4 disconnect the communication connection related to the measurement of the space 2. The terminal 1 and the server 4 may explicitly cancel the coordinate system pairing state (for example, delete the conversion parameter 7), or may continue thereafter. The terminal 1 may be connected to the server 4 at all times via communication, or may be connected to the server 4 only when necessary. Basically, a method (client-server method) that does not hold data such as spatial data 6 may be used in the terminal 1.

In the above control flow example, the terminal 1 automatically transmits the created spatial data 6 to the server 4 and registers it. Not limited to this, the user may perform an operation for registering spatial data in the terminal 1 and register the spatial data 6 in the server 4 according to the operation. The terminal 1 displays a guide image related to spatial data registration on the display surface 11. The user performs the spatial data registration operation accordingly.

[Use of spatial data]
When the spatial data 6 (particularly the spatial shape data 61) of the space 2 is registered in the server 4 as described above, each terminal 1 acquires and uses the spatial data 6 by communication, particularly through the sign 3. be able to. The procedure at this time is as follows, for example.

The terminal 1 recognizes the corresponding sign 3 for the target space 2, acquires predetermined information (ID, etc.), and determines whether the coordinate system has been paired, whether the space data 6 has been registered, and the like. Check the status. For example, when the spatial data 6 is already registered, the terminal 1 acquires the spatial data 6 (particularly the spatial shape data 61) relating to the target space 2 from the server 4 by using the predetermined information. If the terminal 1 has not been paired with the coordinate system, the terminal 1 performs the coordinate system pairing with the space 2. When the conversion parameter 7 is already held in the terminal 1, the coordinate system pairing can be omitted.

When the terminal 1 recognizes the sign 3, the user is asked whether to measure the space 2 (create the corresponding space data 6), or to acquire and use the registered space data 6 or the like. An image for the options and guides may be displayed on the display surface 11, and the subsequent processing may be determined according to the operation of the user. For example, the terminal 1 transmits a spatial data request to the server 4 when using the spatial data 6 based on the user's operation. The server 4 searches the DB 5 in response to the request, and if there is a target spatial data 6 (particularly the spatial shape data 61), the server 4 transmits the spatial data 6 to the terminal 1 as a response.

The terminal 1 can suitably display the virtual image 22 in the space 2 at the position 21 according to the shape of the object in the space 2 by using the acquired space data 6, for example, by the AR function. The spatial data 6 (particularly the spatial shape data 61) can be used for various purposes other than the purpose of displaying the virtual image 22 by the AR function. For example, it can be used for grasping the positions of users and their own machines, and for searching and guiding routes to destinations. For example, the HMD, which is the terminal 1, uses the acquired spatial data 6 to display the shape of the space 2 on the display surface 11. At this time, the HMD may superimpose and display the shape of the space 2 in the actual size, for example, by a virtual image of a line drawing. Alternatively, the HMD may display the shape of the space 2 as a virtual image such as a three-dimensional map or a two-dimensional map in a size smaller than the actual one. In addition, the HMD may display a virtual image showing the current positions of the user and the own machine on the map. Further, the HMD may display the position of the user's destination or the route from the current position to the position of the destination as a virtual image on the map. Alternatively, the HMD may display a virtual image such as an arrow for route guidance according to the real thing.

[Effects (3)]
As described above, in the third embodiment, it is possible to efficiently pair the coordinate system and acquire the spatial data by using the marker 3 in particular. Further, in the second and third embodiments, the object or the sign 3 in the space 2 is fixed at the time of the coordinate system pairing between the terminal coordinate system WA of the terminal 1 and the space coordinate system W1 on the space 2 and the sign 3 side. There is. Therefore, at the time of this coordinate system pairing, it is sufficient to consider the stationary side of the terminal 1, high-precision measurement is possible, and the degree of freedom in practical use is increased.

As a modification of the third embodiment, regarding the coordinate system pairing between the terminal coordinate system of the terminal 1 and the spatial coordinate system of the marker 3, indirect as in the modification 4 (FIG. 18) of the second embodiment. The method of coordinate system pairing can also be applied. For example, when the second terminal 1B performs the coordinate system pairing with the spatial coordinate system W1 (FIG. 19) of the sign 3, the second terminal 1B performs the coordinate system pairing with the first terminal 1A for which the coordinate system pairing has already been completed. You may go instead. As a result, the second terminal coordinate system WB of the second terminal 1B can realize indirect coordinate system pairing with the spatial coordinate system W1 of the marker 3 via the first terminal coordinate system WA.

In the third embodiment, the terminal 1 relates to the coordinate system pairing (corresponding conversion parameter 7) with respect to the predetermined feature points and feature lines obtained by measuring in the space 2 after the coordinate system pairing with the marker 3. It may be used for calibration (adjustment). Further, a plurality of signs 3 or features may be provided in one space 2. The terminal 1 can use each of the markers 3 or features for coordinate system pairing or adjustment.

[Modification 5]
As a modification of the first to third embodiments (referred to as modification 5), the following is also possible. In the modified example 5, the division on the time axis in the case of measuring a certain space 2 and creating the space data 6 is dealt with. In this case, the number of users may be one or a plurality. Even if there is only one terminal 1 at the same time, sharing on the time axis is possible. In this case, each terminal 1 is in charge of each time in a plurality of times configured by time division.

FIG. 22 shows an example of sharing on the time axis in the modified example 5. Space 2 is, for example, a large building and has ID = 100. Although not shown, the space 2 may have a plurality of rooms, areas, and the like. For example, there are two users (U1, U2) as sharing users, and there are two corresponding terminals 1 (1A, 1B). The purpose of the work here is to create spatial data 6 (referred to as D100) in units of space 2 described in the spatial coordinate system W1.

(A) indicates the state at the first date and time. At the first date and time, the user U1 measures the area 2201 in the space 2 by the first terminal 1A, creates the subspace data D101 representing the shape of the area 2201, and registers it in the library of the DB 5 of the server 4. do. The area 2201 may be an area determined in advance by sharing, or may be an area arbitrarily measured by the user U1 at that time.

(B) indicates the state at the second date and time. At the second date and time, the user U2 measures the area 2202 in the space 2 by the second terminal 1B, creates the subspace data D102, and registers it in the library of the DB 5 of the server 4. The region 2202 is a region different from the region 2201 and may include an overlapping region (for example, an overlapping region 2212). The subspace data D102 includes at least data in a region that does not overlap with the subspace data D101.

(C) indicates the state at the third date and time. At the third date and time, the user U1 measures the area 2203 in the space 2 by the first terminal 1A, creates the subspace data D103, and registers it in the library of the DB 5 of the server 4. Region 2203 is a region different from regions 2201,202 and may include overlapping regions.

As described above, the spatial data 6 (particularly the spatial shape data 61) about the space 2 (ID = 100) is accumulated in the DB 5 of the server 4. The contents of the spatial data 6 are updated at any time on the time axis. For example, at the third date and time, the spatial data D100 is composed of subspace data D101, D102, and D103. In addition, each subspace data may have measurement date / time information, measurement user / terminal information, status information such as "measured", and the like. After that, similarly, by appropriately measuring an arbitrary area in the space 2 by one or more arbitrary users and an arbitrary terminal 1 on the time axis, a sufficient area in the space 2 can be measured. For example, spatial data 6 in units of space 2 can be created.

Further, each terminal 1 can grasp the measured area in the space 2 by referring to the spatial data 6 from the server 4 before the start of each measurement. Therefore, the terminal 1 can omit the measurement for the measured region and start the measurement for the unmeasured region. Further, when the measured area is measured again, the terminal 1 can update or correct the shape and the like of the area.

Regarding the handling of the overlapping area of measurement (for example, the overlapping area 2212), the following method is possible. As the first method, each subspace data is provided with the data of the overlapping area. For example, in the subspace data D101 and D102, the data of the overlapping region 2212 is held.

As the second method, each subspace data does not have the data of the overlapping area. For example, in the subspace data D101 or D102, there is no data of the overlapping region 2212. The terminal 1 or the server 4 determines whether or not a certain area in the space 2 has been measured. For example, the determination can be made from the state of the contents of the registered spatial data 6. For example, in the second terminal 1B and the server 4, since the subspace data D101 already has the data of the overlapping area 2212, the subspace data D102 does not have the data of the overlapping area 2212. Alternatively, as another method, the second terminal 1B and the server 4 overwrite and update the data of the overlapping area 2212 in the subspace data D101 with the data of the overlapping area 2212 of the subspace data D102.

Regarding the area in space 2, the state such as the shape in that area may change on the time axis. For example, an arrangement such as a desk may be moved. In this case, the terminal 1 and the server 4 can determine the change by observing the difference between the measurement data or the subspace data for each area on the time axis. Based on this determination, for example, when it is desired to reflect the latest state of the space 2, the terminal 1 and the server 4 may perform overwriting and updating using the subspace data at the newer measurement date and time. Further, the terminal 1 and the server 4 distinguish between a fixed arrangement (for example, a wall, a floor, a ceiling) constituting the space 2 and a position-variable arrangement (for example, a desk) based on such a judgment. You can also judge. Based on this, the terminal 1 and the server 4 may register the attribute information in the spatial data 6 by distinguishing between fixed and variable positions for each part. Further, the spatial data 6 can be configured so that the position-variable arrangement is not a component in the first place.

In the DB 5 of the server 4, only the spatial data of the latest measurement date and time may be retained as the spatial data 6 of the same space 2, but the spatial data for each measurement date and time may be retained as a history. In this case, the change in space 2 on the time axis can be grasped as a history. From the difference in the spatial data of each measurement date and time, it is possible to discriminate between the fixed arrangement and the position-variable arrangement.

[Modification 6]
As a modification of the first to third embodiments (referred to as modification 6), the following is also possible. In the modification 6, when the space 2 is measured and the space data 6 is created, each terminal 1 does not share the measurement in advance. In this case, the number of users may be one or a plurality. Each terminal 1 provides the measured spatial data to another terminal 1 or registers it in the server 4 if requested. Each terminal 1 searches for and acquires spatial data 6 not owned by its own device from the other terminals 1 and the server 4 and uses it.

The flow of the modified example 6 is shown in FIG. The information processing device 9 is a terminal 1 (for example, a second terminal 1B) or a server 4. Prior to the flow of FIG. 23, the terminal 1 measures and holds the spatial data 6, and the server 4 registers the measured spatial data 6 as in the flow of FIG. 21, for example. Further, the server 4 may hold the spatial data 6 as the design data of the wall of the building or the like.

In steps S331 and S331b, the first terminal 1A and the information processing device 9 establish communication. When the information processing device 9 is the server 4, the first terminal 1A selects the server 4 that manages the spatial data 6 to be acquired when establishing the communication. The selection can be made, for example, from the position information of the spatial data 6. Alternatively, the sign 3 may be recognized to read predetermined information (for example, ID, spatial data acquisition destination information), and a communication connection with the server 4 may be established based on the predetermined information. When the information processing device 9 is the second terminal 1B, the device can be specifically specified, so that communication can be established using the communication data held in advance.

In step S332, the first terminal 1A transmits a coordinate system pairing request to the information processing device 9, and in step S332b, the information processing device 9 transmits a coordinate system pairing response to the first terminal 1A.

In step S333, the terminal 1 transmits a request for various amounts of data to the information processing device 9. When the information processing device 9 is the second terminal 1B, the various amount data is various amount data related to the second terminal 1B. When the information processing device 9 is the server 4, the quantity data is the quantity data related to the marker 3. In step S333b, the information processing apparatus 9 transmits the requested amount of data to the first terminal 1A. The first terminal 1A acquires the various amount data. When the common coordinate system for acquiring the spatial data 6 from the second terminal 1B is the spatial coordinate system, the first terminal 1A has various quantities such as a marker 3 required for coordinate system pairing with the spatial coordinate system. The data is acquired from the second terminal 1B and others.

In step S334, the first terminal 1A has a predetermined feature of the second terminal 1B or the marker 3 (for example, the point p1 and the line v1 in FIG. 20) necessary for the coordinate system pairing based on the acquired various quantity data. v2) is measured by the terminal coordinate system WA and obtained as various quantity data on the own machine side. The measurement at this time is possible by the distance measuring sensor 13.

In step S335, the first terminal 1A has the various amount data described in the common coordinate system WS on the second terminal 1B or the sign 3 side obtained in step S333, and the terminal coordinate system on the own machine side obtained in step S334. Using the various amount data described in WA, the conversion parameter 7 between the terminal coordinate system WA and the common coordinate system WS is calculated and set in the own machine. As a result, the spatial recognition can be shared between the first terminal 1A and the information processing device 9.

In steps S336 and S336b, the first terminal 1A performs an inquiry for holding the spatial data 6, the information processing apparatus 9 performs an inquiry response, and the spatial data 6 is transmitted. First, the first terminal 1A transmits the position information described with reference to the common coordinate system of the region for which the spatial data 6 is to be acquired to the information processing device 9. The information processing device 9 replies with a list of spatial data 6 relating to the area for which the inquiry has been received. The region referred to here is a three-dimensional region surrounded by a rectangular parallelepiped defined by coordinate values, as shown in FIG. 24A, for example, and a partial region of space 2 is specified in detail. This may be specified by defining the spatial mesh in advance and specifying the ID of the spatial mesh. The spatial data 6 relating to the region is three-dimensional position information such as an object in which at least a part thereof exists in the region, feature points such as boundaries in the real space, feature lines, and polygon data. The list of spatial data 6 is, for example, as shown in FIG. 24 (B). The position of the area on the list is the range in which the position information of the object or the like exists, and does not necessarily match the area of the query. When the region is a rectangular parallelepiped whose sides are parallel to the coordinate axes, the region may be specified by the coordinate values at both ends of one diagonal line of the rectangular parallelepiped. If the region is an arbitrary polyhedron, specify it with all vertex coordinate values. The response by the information processing device 9 may include spatial data 6 in the vicinity of the area in which the inquiry was received. Upon receiving the response, the first terminal 1A selects the spatial data 6 to be acquired from the list, and receives the transmission of the spatial data 9 from the information processing device 9.

In step S337, the first terminal 1A converts the spatial data 6 acquired in step S336 into the spatial data 6 described in the terminal coordinate system WA of the own machine by using the conversion parameter 7, and uses it.

In another method, the conversion parameter 7 may be transmitted to the information terminal device 9, and the position information may be exchanged based on the terminal coordinate system WA.

In steps S338 and S338b, the first terminal 1A and the information processing device 9 confirm whether or not to end the coordinate system pairing related to the spatial data provision, and if so, proceed to steps S339 and S339b. When continuing (No), the process returns to steps S336 and S336b and is repeated in the same manner.

In steps S309 and S309b, the first terminal 1A and the information processing device 9 disconnect the communication connection related to the provision of spatial data. The first terminal 1A and the information processing device 9 may explicitly cancel the state of the coordinate system pairing (for example, delete the conversion parameter 7), or may continue thereafter. The first terminal 1A may be connected to the information processing device 9 at all times via communication, or may be connected to the information processing device 9 only when necessary. Basically, a method (client-server method) that does not hold data such as spatial data 6 may be used in the terminal 1. The server in this case is not the server 4 as the information processing device 9.

The terminal 1 may integrate the acquired spatial data 6 to create a new spatial data 6, or may simply use the spatial data 6 acquired from the information processing device for displaying an AR object or the like.

According to the modified example 6, the measurement of the spatial data 6 by the own machine can be omitted without the trouble of setting the measurement sharing in advance, and the work efficiency can be improved.

<Embodiment 4>
The space recognition system and the like according to the fourth embodiment will be described with reference to FIG. 25 and the like. The fourth embodiment is a modification of the first to third embodiments, and a function is added. The terminal 1 displays on the display surface 11 an image for guiding or supporting a measurement range or the like related to sharing to the user according to the position or orientation of the terminal 1. As for the position of the terminal 1, the position on the horizontal plane at the time of pairing the coordinate system is used.

[Display example 1]
FIG. 25 shows a display example of the display surface 11 of the terminal 1 according to the fourth embodiment. In this example, there is a user U1 who wears the HMD which is the first terminal 1A in the space 2 as shown in FIG. The user U1 is looking toward the wall 2301 where the whiteboard 2b is located through the display surface 11 of the HMD. It is assumed that the region related to the sharing has been set in advance in the first terminal 1A, for example, in step S2A of FIG. 10 described above. This setting may be set for the spatial data 6 of the DB 5 of the server 4. For example, the first terminal 1A of the user U1 is in charge of the area 2A as shown in FIGS. 3 and 4.

The first terminal 1A superimposes and displays an image 2300 representing an area 2A (that is, an area to be measured) shared by the own machine, a measurement range, and the like on the display surface 11. The image 2300 indicates that the region in the direction in which the image 2300 can be seen is a shared region. In this example, the image 2300 is an image representing the boundary surface of the

regions

2A and 2B in the space 2 (an image in which the back can be seen through), but the image 2300 is not limited to this and is an image representing a three-dimensional region 2A and the like. May be. In this example, the position of the first terminal 1A (corresponding user U1) in the spatial coordinate system W1 is outside the area 2A and faces the area 2A. Therefore, the boundary surface of the

regions

2A and 2B is displayed as the image 2300. Further, for example, when the position of the first terminal 1A is inside the area 2A and faces the arrangement of the area 2A, an image showing the state is displayed instead of the image 2300.

The user U1 can easily grasp the area 2A and easily measure the area 2A by looking at the image 2300. The user U1 may measure the direction in which the image 2300 can be seen. When the sensitivity region of the measurement sensor (for example, the distance measuring sensor 13) provided in the terminal 1 is in the front direction of the face of the user U1, the user U1 turns the image 2300 in the direction in which the face is turned for the measurement. It can be used as a guide. In other words, the user U1 may turn his / her face so that the line of sight points within the surface region of the image 2300 at the time of measurement.

As another example, the first terminal 1A represents an area (region 2B) shared by another terminal 1 (for example, the second terminal 1B) in addition to the image 2300, depending on the position and orientation. An image may be displayed.

Figure 26 is an example of a sharing process, an overview of a horizontal plane looking down the space 2 _(X ₁ -Y 1 side). This division shows an example of setting the measurement direction when the target space 2 is measured by a plurality of terminals 1 at the same time. In this case, in the horizontal plane, the measurement orientation of each terminal 1 is determined so that the plurality of terminals 1 cover all directions (corresponding regions) of the target space 2. The plurality of terminals 1 to be shared perform coordinate system pairing with each other. After that, the terminals 1 perform processing for sharing while appropriately communicating with each other.

In this example, each position and orientation (2401,402, 2403) when simultaneously measuring with terminals 1 (1A, 1B, 1C) of three users (U1, U2, U3) are shown. Between terminals 1, first, the sharing range is calculated in this state. An intersection (2411,241,2413) is taken between the perpendicular bisector of the line segment connecting the adjacent terminals 1 and the boundary line (four walls in this example) of the space 2. This intersection is defined as the boundary of the sharing range of space 2 (corresponding vertical line). Further, between the terminals 1, this sharing range may be used as an initial value, and further adjustment may be made so that the sharing is fair (for example, the same size) (for example, the intersection points are shifted in the horizontal direction). Further, in this example, the intersection is used as a boundary so that the sharing range does not overlap, but the present invention is not limited to this, and the sharing range may be overlapped at the boundary portion including the intersection. The shape of the wall or the like of the room can be measured by the shared measurement as shown in FIG.

[Display example 2]
FIG. 27 shows a display example on the first terminal 1A corresponding to the division of FIG. 26. Images 2501, 502 representing the measurement range boundary lines corresponding to the

intersections

2411 and 2413 are displayed on the display surface 11. Further, in this example, a plurality of images 2503 of horizontal line arrows are displayed within the measurement range represented by the images 2501, 502 of the measurement range boundary line. This horizontal line arrow serves as a guide for scanning when the sensor of the terminal 1 (for example, the distance measuring sensor 13) is moved so as to scan during measurement.

The user U1 can realize efficient and highly accurate measurement by moving the user U1 so as to change the direction of the face (corresponding image 2504) along the image 2503 of the horizontal line arrow. Image 2504 is a display example of an image such as a cursor indicating the orientation of the HMD, the sensor, or the like. The direction and spacing of the horizontal arrows in image 2503 are designed for efficient measurement. For example, the distance between two adjacent horizontal arrows is selected as a distance that minimizes measurement duplication without causing measurement omission.

[Display example 3]
FIG. 28 further shows another display example. The terminal 1 grasps whether or not the measurement (in other words, the acquisition of the measurement data) has been completed for each area in the space 2 or the shared area, so that the measured area and the unmeasured area can be easily understood by the user. An image that distinguishes and represents those areas is displayed on the display surface 11. Further, the terminal 1 may grasp whether or not the area is already registered as the spatial data 6 (particularly the spatial shape data 61) in the library of the DB 5 of the server 4, and display an image representing those areas separately. .. In this example, the image 2601 is a vertical line hatch-like image representing the measured range. Image 2602 is a diagonal hatch-like image representing an unmeasured range. The display state of these images is updated in real time. The image may be displayed with a character string, an icon, or the like indicating the types of "measured", "unmeasured", "registered", and "sharing range". Not only the image display but also the guide by audio output may be used.

[Display example 4]
FIG. 29 shows another display example. In this modification, the area of sharing is not determined between the plurality of terminals 1. Each user appropriately measures an arbitrary range with the terminal 1 and spontaneously measures the unmeasured range. When measuring the space 2, each terminal 1 measures a range according to the position and orientation of the user. Similar to FIG. 28, each terminal 1 displays an image so that the user can see the measured range and the unmeasured range by the own device. For example, the first terminal 1A displays images 2601,602. Between the terminals 1 to share, each time a measurement is performed, measurement data or information representing a measured area of the own machine is transmitted to another terminal 1. Each terminal 1 grasps a measured area and an unmeasured area by each terminal 1 in the space 2 based on the measurement data or information. Then, each terminal 1 displays an image showing the measured area when there is a measured area by another terminal 1 within the range of the display surface 11. For example, the first terminal 1A displays an image 2701 such as a horizontal line hatch indicating that the measurement has been performed by the second terminal 1B. The user U1 can easily determine the next measurement range by looking at the guide image. Not limited to the above example, two types of images, measured and unmeasured, may be collected by all the terminals 1 to be shared.

[Display example 5]
FIG. 30 shows a display example of another image. In the example of FIG. 25 and the like, a guide image representing a surface or the like is displayed in the real space, but the guide image is not limited to this and is displayed so as to match the surface of an object such as a wall or a desk in the space 2. You may. In this example, the object 2700 is arranged on the floor 2701 at a corner close to the

walls

2702 and 2703 in the space 2 such as a room. When the range including the object 2700 is measured, the terminal 1 displays an image 2710 such as a broken line indicating that the measurement has been completed. For example, the terminal 1 can display an image 2710 showing the shape of an object in the measured range according to the surface of the object represented by the point cloud based on the point cloud data acquired by the distance measuring sensor 13. The image 2710 may be a line image or a surface image.

Although the present invention has been specifically described above based on the embodiments, the present invention is not limited to the above-described embodiments and can be variously modified without departing from the gist. It is possible to add, delete, replace, and configure various combinations of the components of the embodiment. Some or all of the above-mentioned functions may be implemented by hardware, or may be implemented by software program processing. The programs and data constituting the functions and the like may be stored in a computer-readable storage medium, or may be stored in a device on a communication network.

1 ... Terminal (HMD), 1A ... 1st terminal, 1B ... 2nd terminal, 1a, 1b ... Smartphone, 2 ... Space, 4 ... Server, 6 ... Spatial data, 7 ... Conversion parameters, 9 ... Information processing device, 11 ... Display surface, 12 ... Camera, 13 ... Distance measuring sensor, U1 ... 1st user, U2 ... 2nd user, W1 ... Spatial coordinate system, WA ... 1st terminal coordinate system, WB ... 2nd terminal coordinate system, WS ... Common coordinate system, WT ... terminal coordinate system, 21 ... position, 22 ... image.

Claims

It is equipped with an information terminal having a function of measuring space and a function of displaying a virtual image on a display surface and having a terminal coordinate system, and an information processing device that performs processing based on a common coordinate system.
The information terminal measures the relationship regarding position and orientation between the terminal coordinate system and the common coordinate system, and adapts the terminal coordinate system and the common coordinate system based on data representing the measured relationship. Let me
The information terminal and the information processing device share the recognition of the space.
Spatial recognition system.
In the space recognition system according to claim 1,
A plurality of information terminals are provided as the information terminals.
Among the plurality of information terminals, the first information terminal having the first terminal coordinate system serving as the common coordinate system and also serving as the information processing device, and the second information terminal having the second terminal coordinate system are described above. When sharing the perception of space
The second information terminal measures the relationship regarding the position and orientation with the first information terminal, and based on the data representing the measured relationship, the first terminal coordinate system and the second terminal coordinate system And share the perception of the space,
Spatial recognition system.
In the space recognition system according to claim 1,
The spatial coordinate system of the space is the common coordinate system.
The information terminal measures the relationship regarding position and orientation with an object having a predetermined feature in the space, and based on the data representing the measured relationship, the terminal coordinate system and the space coordinate system are set. Adapt and share the recognition of the space with the information processing device.
Spatial recognition system.
In the space recognition system according to claim 1,
A plurality of information terminals are provided as the information terminals.
When the plurality of information terminals share and measure the space to obtain spatial data,
The information terminal or the information processing device integrates the subspatial data measured and obtained by each information terminal to create the spatial data in the spatial unit.
Spatial recognition system.
In the space recognition system according to claim 2,
When the first information terminal and the second information terminal share the space for measurement,
The first information terminal measures the first region of the space in the first terminal coordinate system, creates the first subspace data described in the first terminal coordinate system, and creates the first subspace data.
The second information terminal measures a second region of the space in the second terminal coordinate system, creates a second subspace data described in the second terminal coordinate system, and creates a second subspace data.
The first information terminal or the second information terminal converts the second subspace data into the subspace data described in the first terminal coordinate system, and the first subspace data and the first terminal coordinates. Create spatial data in the spatial unit by integrating with the subspatial data described in the system.
Spatial recognition system.
In the space recognition system according to claim 3,
A plurality of information terminals are provided as the information terminals, and among the plurality of information terminals, a first information terminal having a first terminal coordinate system and a second information terminal having a second terminal coordinate system share the space. When measuring
The first information terminal measures a first region of the space in the first terminal coordinate system, creates first subspace data described in the first terminal coordinate system, and uses the first subspace data. , Converted to the first subspace data described in the spatial coordinate system,
The second information terminal measures a second region of the space in the second terminal coordinate system, creates a second subspace data described in the second terminal coordinate system, and uses the second subspace data. , Converted to the second subspace data described in the spatial coordinate system,
The first information terminal or the second information terminal also serves as the information processing device, and obtains first subspace data described in the space coordinate system and second subspace data described in the space coordinate system. Integrate to create spatial data in the spatial unit,
Spatial recognition system.
In the space recognition system according to claim 5,
The second information terminal generates a conversion parameter for matching the first terminal coordinate system and the second terminal coordinate system, and sets the conversion parameter in the own machine.
Spatial recognition system.
In the space recognition system according to claim 6,
The first information terminal generates a conversion parameter for matching the first terminal coordinate system and the spatial coordinate system, sets it in its own machine, and sets it in its own machine.
The second information terminal generates a conversion parameter for matching the second terminal coordinate system and the spatial coordinate system, and sets the conversion parameter in the own machine.
Spatial recognition system.
In the space recognition system according to claim 4,
The information terminal uses the spatial data to display a virtual image on the display surface so as to match a position in the space.
Spatial recognition system.
In the space recognition system according to claim 1,
A server device serving as the information processing device for registering and holding spatial data in the space is provided.
The information terminal registers the created spatial data in the server device.
Spatial recognition system.
In the space recognition system according to claim 10,
The information terminal acquires and uses the spatial data from the server device.
Spatial recognition system.
In the space recognition system according to claim 11,
Data for displaying the virtual image in the space is registered in the server device in association with the space data.
Spatial recognition system.
In the space recognition system according to claim 3,
As the relationship, the information terminal has a quantity representing a relative orientation relationship between the terminal coordinate system and the spatial coordinate system, and a relative of the origin of the terminal coordinate system and the origin of the spatial coordinate system. Measure the amount that represents the relationship between the two positions,
Spatial recognition system.
In the space recognition system according to claim 3,
A sign is provided as the object provided in association with the space.
The information terminal measures the relationship regarding the position and orientation with the sign, and adapts the terminal coordinate system and the spatial coordinate system based on the data representing the measured relationship.
Spatial recognition system.
In the space recognition system according to claim 14,
The information terminal recognizes the sign, reads predetermined information described in the sign, and uses the predetermined information to identify spatial data associated with the space.
Spatial recognition system.
In the space recognition system according to claim 2,
Among the plurality of information terminals, the first information terminal having the first terminal coordinate system, the second information terminal having the second terminal coordinate system, and the third information terminal having the third terminal coordinate system. However, when the space is shared and measured,
The first information terminal adapts the first terminal coordinate system and the second terminal coordinate system to the second information terminal.
The second information terminal adapts the second terminal coordinate system and the third terminal coordinate system to the third information terminal.
The third information terminal uses the information acquired from the second information terminal to match the third terminal coordinate system with the first terminal coordinate system.
Spatial recognition system.
In the space recognition system according to claim 4,
The plurality of information terminals share time to measure the space.
The information terminal integrates a plurality of subspace data created by sharing and measuring the time to create the spatial data in the spatial unit.
Spatial recognition system.
In the space recognition system according to claim 3,
The spatial coordinate system of the space is a coordinate system common to a plurality of spaces in the real world.
Spatial recognition system.
In the space recognition system according to claim 4,
The information terminal displays an image representing the shared area or measurement range in the space on the display surface.
Spatial recognition system.
In the space recognition system according to claim 4,
The information terminal displays an image representing a measured area or range in the space and an image representing an unmeasured area or range on the display surface.
Spatial recognition system.
In the space recognition system according to claim 4,
The information terminal displays an image for guiding the measurement direction in the space on the display surface.
Spatial recognition system.
A space recognition method in a space recognition system including an information terminal having a function of measuring space and a function of displaying a virtual image on a display surface and having a terminal coordinate system, and an information processing device that performs processing based on a common coordinate system. And
The information terminal measures the relationship between the terminal coordinate system and the common coordinate system regarding the position and orientation, and matches the terminal coordinate system and the common coordinate system based on the data representing the measured relationship. Steps to make
A step in which the information terminal and the information processing device share recognition of the space,
A spatial recognition method that has.
In the space recognition method according to claim 22,
A step in which a plurality of information terminals share the space as the information terminal and measure the space to obtain spatial data.
A step of integrating the subspace data measured and obtained by each information terminal to create the spatial data in the spatial unit, and
A spatial recognition method that further has.
An information terminal having a function of measuring space and a function of displaying a virtual image on a display surface and having a terminal coordinate system, and an information processing device that performs processing based on a common coordinate system that provides spatial data of the space. The information terminal in the spatial recognition system provided.
The information terminal measures the relationship regarding position and orientation between the terminal coordinate system and the common coordinate system that describes the spatial data, and based on the data representing the measured relationship, the terminal coordinate system and the said. Align with the common coordinate system,
Utilizing the spatial data described in the common coordinate system,
Information terminal.
In the information terminal according to claim 24
When a plurality of information terminals share the space as the information terminal and measure the space to obtain spatial data.
The subspace data measured and obtained by each of the information terminals is integrated to create the spatial data in the spatial unit.
Information terminal.
It includes an information terminal having a function of measuring space and a function of displaying a virtual image on a display surface and having a terminal coordinate system, and a server device that performs processing based on a common coordinate system that provides spatial data of the space. The server device in the space recognition system.
The information terminal measures the relationship regarding position and orientation between the terminal coordinate system and the common coordinate system that describes the spatial data, and based on the data representing the measured relationship, the terminal coordinate system and the said. Align with the common coordinate system,
The information terminal registers the spatial data in the server device and registers the spatial data in the server device.
The server device registers and holds the spatial data.
Server device.
In the server device according to claim 26,
The information terminal measures the relationship regarding position and orientation with an object having a predetermined feature in the space, and based on the data representing the measured relationship, the terminal coordinate system and the common coordinate system are used. When adapting,
The server device provides the spatial data relating to an object having a predetermined feature in the space.
Server device.