JP2021086204A - Information processing device, information processing method, and program - Google Patents

Abstract

To provide an information processing device, an information processing method, and a program that can prevent the theft of a terminal device for AR.SOLUTION: An information processing device according to one embodiment includes a level setting unit and a content output unit. The level setting unit sets a monitoring level on the basis of the state of an AR display terminal used by a user and the state of an AR application operated by the AR display terminal. The content output unit outputs, on the basis of the monitoring level, instruction content encouraging the user to move to an action range corresponding to the state of the AR application.SELECTED DRAWING: Figure 7

Description

The present technology relates to an information processing device, an information processing method, and a program applicable to a terminal or the like that provides augmented reality (AR).

Conventionally, a service or the like that lends a terminal device to a user and provides a predetermined content is known, and a method for preventing the theft of such a terminal device has been devised. For example, Patent Document 1 describes a device that is lent to a user to provide a voice guide. This device identifies the position of the main body of the device by a GPS function or the like, and issues a warning by voice or the like when the device exceeds the range of use area, for example. As a result, the device can be reliably returned and collected, and the device can be prevented from being stolen (paragraphs [0021] [0023] [0027] of Patent Document 1 and the like).

In recent years, terminal devices that provide various contents using augmented reality (AR) have been developed. In this terminal device, virtual objects (AR objects, etc.) generated by the application are superimposed and displayed in the real space. Therefore, the user who uses the terminal device perceives the AR object or the like as if it exists in the real space. By renting out such a terminal device, it becomes possible to provide a service or the like that allows the AR experience to be easily enjoyed.

JP-A-2002-320295

If the terminal device for AR is rented out, there is a risk that the device will be stolen. On the other hand, since the position and behavior of the user who is experiencing AR changes every moment according to the position and movement of the AR object and the like, it is often difficult to determine the possibility of the device being stolen. Therefore, there is a demand for a technique for preventing the theft of AR terminal devices.

In view of the above circumstances, an object of the present technology is to provide an information processing device, an information processing method, and a program capable of preventing the theft of the terminal device for AR.

In order to achieve the above object, the information processing device according to one embodiment of the present technology includes a level setting unit and a content output unit.
The level setting unit sets the monitoring level based on the state of the AR display terminal used by the user and the state of the AR application operating on the AR display terminal.
Based on the monitoring level, the content output unit outputs instructional content that prompts the user to move to the action range according to the state of the AR application.

In this information processing device, the monitoring level is set based on the state of the AR display terminal and the state of the AR application. Based on this monitoring level, instruction content that prompts the user to move to the action range that matches the state of the AR application is output. As a result, it is possible to manage the range in which the user acts and to monitor suspicious behavior and the like, and it is possible to prevent the theft of the terminal device for AR.

The level setting unit may acquire condition information according to the state of the AR application, determine the state of the AR display terminal based on the condition information, and set the monitoring level.

The state of the AR application may include the progress state of the AR application. In this case, the level setting unit may acquire the condition information according to the progress state.

The level setting unit may acquire the condition information according to the role of the user in the AR application.

The condition information may include a plurality of determination conditions for determining the state of the AR display terminal.

The state of the AR display terminal may include at least one of the position, posture, or movement parameters of the AR display terminal.

Each of the plurality of determination conditions may be associated with the candidate value of the monitoring level. In this case, the level setting unit may determine the state of the AR display terminal for each of the plurality of determination conditions, and set the maximum candidate value represented by the determination result to the monitoring level.

The determination condition includes at least one of position determination information for determining the position of the AR display terminal with respect to a predetermined target and orientation determination information for determining the orientation of the AR display terminal with respect to the predetermined target. But it may be.

The predetermined target may be at least one point in real space, another user participating in the AR application, or a virtual object used in the AR application.

The determination condition may include the position determination information. In this case, the content output unit may set the action range so that the monitoring level becomes smaller based on the position determination information.

The content output unit may output the instruction content that prompts the approach to the predetermined target, which is the reference of the determination condition used for setting the monitoring level.

When the monitoring level is set to the first level, the content output unit outputs the first content that does not include a specific instruction of the moving direction for moving to the action range as the instruction content. May be good.

The first content may be content representing the position of the action range using at least one of a character object, an item object, character information, or voice information used in the AR application.

When the monitoring level is set to a second level higher than the first level, the content output unit includes a second movement direction specific instruction for moving to the action range as the instruction content. Content may be output.

The second content may be content that includes at least one of image information, character information, or sound information that indicates the moving direction.

When the monitoring level is set to a third level higher than the second level, the content output unit may stop the operation of the AR application and execute a monitoring mode for monitoring the user. ..

When the monitoring level of another user is relatively high, the content output unit may output the instruction content that prompts the other user to approach.

The content output unit may display an object to be searched by the user in the AR application in the vicinity of the other user.

The information processing method according to one form of the present technology is an information processing method executed by a computer system, and is based on the state of the AR display terminal used by the user and the state of the AR application operating on the AR display terminal. Including setting the monitoring level.
Based on the monitoring level, instruction content that prompts the user to move to the action range according to the state of the AR application is output.

A program according to a form of the present technology causes a computer system to perform the following steps.
A step of setting a monitoring level based on the state of an AR display terminal used by a user and the state of an AR application operating on the AR display terminal.
A step of outputting instruction content that prompts the user to move to an action range according to the state of the AR application based on the monitoring level.

It is a perspective view which shows the appearance of the head-mounted display which concerns on one Embodiment of this technique. It is a schematic diagram which shows the configuration example of the AR display system using a head-mounted display. It is a block diagram which shows the functional configuration example of a head-mounted display and a server device. It is a schematic diagram which shows an example of AR display. This is an example of a condition table for determining the state of the head-mounted display. It is a schematic diagram for demonstrating the index value of a head-mounted display. It is a flowchart which shows an example of a monitoring process. It is a flowchart which shows an example of the update process of a monitoring level. It is a flowchart which shows an example of the evaluation process of an index value. It is a schematic diagram for demonstrating the monitoring level. It is a schematic diagram which shows an example of the 1st content. It is a flowchart which shows an example of the output processing of the 1st content. It is a schematic diagram which shows an example of the 2nd content. It is a flowchart which shows an example of the output processing of the 2nd content. It is a schematic diagram which shows an example of the screen in a monitoring mode. It is a flowchart which shows an example of a monitoring mode. It is a schematic diagram which shows an example of the content display related to other users.

Hereinafter, embodiments relating to the present technology will be described with reference to the drawings.

<Configuration of information processing device>
FIG. 1 is a perspective view showing the appearance of a head-mounted display according to an embodiment of the present technology. FIG. 2 is a schematic view showing a configuration example of an AR display system using a head-mounted display. The AR display system 100 includes a plurality of head-mounted displays 10 (HMDs) and a server device 40 to which the plurality of head-mounted displays 10 are connected.

The head-mounted display 10 is a device capable of AR display. In the present disclosure, the AR display means that the virtual object is displayed so as to be perceived by the user as if it were a real object existing in the real space. The user who uses the head-mounted display 10 can enjoy the AR experience by perceiving the contents such as virtual objects and voices generated by the AR application.

Here, the AR application is an application program that realizes a predetermined service such as a game or a guide using AR. The AR application is composed of, for example, a terminal program mounted on an individual terminal device (head-mounted display 10) and a server program mounted on a server device 40 to which each terminal device is connected. In addition, a service (game, etc.) realized by an AR application may be referred to as an AR application.

The AR display system 100 is a system that provides a plurality of users with an AR experience service using an AR application. In the following, the head-mounted display 10 will be rented out and used by users who have participated in the service.

FIG. 3 is a block diagram showing a functional configuration example of the head-mounted display 10 and the server device 40. As shown in FIGS. 1 and 3, the head-mounted display 10 includes a head-mounted display main body 11, a display unit 20, an imaging unit 21, an inertial sensor 22, an operation unit 23, a speaker 24, and a communication unit 25. A control unit 26 and a storage unit 27 are provided.

The head-mounted display main body 11 is attached to the user's head and used. As shown in FIG. 1, the head-mounted display main body 11 includes a front portion 12, a right temple portion 13 provided on the right side of the front portion 12, a left temple portion 14 provided on the left side of the front portion 12, and a front portion. It has a glass portion 15 attached to the lower side of the portion 12.

The display unit 20 is a see-through type display element, and is provided on the surface of the glass unit 15. The display unit 20 performs AR display according to the control of the control unit 26 (content output unit 30). The display unit 20 may be a non-see-through type display element. In this case, a virtual object or the like used for AR display is displayed on the display unit 20 together with the image captured by the image pickup unit 21. As described above, the head-mounted display 10 is an example of a terminal device (AR display terminal) capable of AR display.

The image pickup unit 21 is, for example, a camera, and includes an image pickup element such as a CCD (Charge Coupled Device) sensor and a CMOS (Complemented Metal Oxide Semiconductor) sensor, and an optical system such as an imaging lens. The imaging unit 21 is provided outward on the outer surface of the front unit 12, images an object existing ahead of the user's line-of-sight direction, and outputs the image information obtained by the imaging to the control unit 26. Two image pickup units 21 are provided in the front unit 12 at predetermined intervals in the lateral direction. The location and number of imaging units 21 can be changed as appropriate.

The inertial sensor 22 includes a three-axis acceleration sensor that detects acceleration in the three-axis direction and an angular velocity sensor that detects an angular velocity around the three axes. The inertial sensor 22 outputs the acceleration in the three-axis direction obtained by the detection and the angular velocity around the three axes as inertial information to the control unit 26. In the present embodiment, the detection axes of the inertial sensor 22 are three axes, but the detection axes may be one axis or two axes. Further, in the present embodiment, two types of sensors are used as the inertial sensor 22, but one type or three or more types of sensors may be used as the inertial sensor 22. Other examples of the inertial sensor 22 include a speed sensor, an angle sensor, and the like.

The operation unit 23 is, for example, various types of operation units such as a pressing type and a contact type, and detects an operation by a user and outputs the operation to the control unit 26. In the example shown in FIG. 1, the operation unit 23 is provided on the front side of the left temple unit 14, but the position where the operation unit 23 is provided is any position as long as it is easy for the user to operate. May be good.

The speaker 24 receives the sound signal output from the control unit 26, converts the sound signal into voice, and outputs the sound signal. The communication unit 25 communicates directly or indirectly with the external device. Examples of the external device that communicates with the head-mounted display 10 include a server device 40 and the like, which will be described later. The head-mounted display 10 can also communicate with various PCs such as a desktop PC (Personal computer), a laptop PC, and a tablet PC, and any external device such as a mobile phone (including a smartphone).

The control unit 26 controls the operation of each block included in the head-mounted display 10. The control unit 26 functions as an information processing device according to the present embodiment, and has a hardware configuration necessary for a computer such as a CPU and a memory (RAM, ROM). Various processes are executed by the CPU loading the program according to the present embodiment stored in the storage unit 27 into the RAM and executing the program. As the control unit 26, for example, a PLD (Programmable Logic Device) such as an FPGA (Field Programmable Gate Array) or another device such as an ASIC (Application Specific Integrated Circuit) may be used.

In the present embodiment, when the CPU of the control unit 26 executes the program according to the present embodiment, the self-position estimation unit 28, the level setting unit 29, and the content output unit 30 are realized as functional blocks. Then, the information processing method according to the present embodiment is executed by these functional blocks. In order to realize each functional block, dedicated hardware such as an IC (integrated circuit) may be appropriately used.

The self-position estimation unit 28 estimates the self-position of the head-mounted display 10. Here, the self-position of the head-mounted display 10 includes, for example, the position and orientation of the head-mounted display 10 itself in a predetermined 3D coordinate system. The self-position estimation unit 28 executes initial position estimation (Relocation) of the head-mounted display 10 and subsequent self-position estimation. A self-position estimation technique such as SLAM (Simultaneous Localization and Mapping) is used for these estimation processes. In addition, any process capable of estimating the self-position of the head-mounted display 10 may be used.

The level setting unit 29 sets the monitoring level based on the state of the head-mounted display 10 used by the user and the state of the AR application operating on the head-mounted display 10. Here, the monitoring level is a level set for monitoring the theft or the like of the head-mounted display 10 rented out to the user. By using this monitoring level, it is possible to gradually indicate the possibility that the head-mounted display 10 is stolen, for example. The monitoring level is set for each user (head-mounted display 10) and is continuously updated. In the following, the higher the monitoring level, the higher the possibility of theft.

In the present embodiment, the level setting unit 29 acquires condition information according to the state of the AR application. The condition information is information in which a plurality of determination conditions used for setting the monitoring level are recorded. For example, condition information (condition table 32) according to the state of the AR application is set in advance and stored in the storage unit 27. Then, the corresponding condition information is read according to the state of the AR application. Alternatively, a condition value (threshold value or the like) included in the determination condition is calculated according to the state of the AR application. That is, the condition information according to the state of the AR application is newly calculated. In addition, the method of acquiring condition information is not limited.

Further, the level setting unit 29 determines the state of the head-mounted display 10 based on the acquired condition information and sets the monitoring level. Therefore, it can be said that the plurality of determination conditions included in the condition information are conditions for determining the state of the head-mounted display 10. In this way, in the monitoring level setting, the state of the head-mounted display 10 is determined based on the conditions set according to the state of the AR application.

As the state of the AR application, the progress state of the AR application is typically referred to. For example, suppose that an AR application is a type of program that progresses while sequentially generating a plurality of events according to a predetermined scenario. In this case, the progress of each event, the information indicating the progress of the entire scenario, and the like are read as the information (progress information) indicating the progress of the AR application. Alternatively, the presence / absence of an event, the elapsed time from the start of the AR application, and the like may be used as progress information. Conditional information corresponding to the progress state represented by such progress information is read.

As for the state of the head-mounted display 10, typically, the position and orientation of the head-mounted display 10 are subject to determination. In this case, the condition information includes a condition for determining the position and orientation of the head-mounted display 10. For example, in the current progress state, a determination condition indicating a position, a posture, or the like to be taken by the user (head-mounted display 10) is set as condition information. Further, the determination conditions relating to the position / posture include a condition for determining the absolute position and the absolute posture, and a condition for determining the relative position and the relative posture with reference to a predetermined target (see FIG. 5). The position and orientation of the head-mounted display 10 are estimated by the self-position estimation unit 28 described above.

Further, as the state of the head-mounted display 10, the motion parameter of the head-mounted display 10 may be used as a determination target. Here, the motion parameters are the velocity, acceleration, angular velocity, angular acceleration, etc. of the head-mounted display 10, and are calculated based on, for example, the output of the inertial sensor. In this case, the condition information includes a condition for determining the motion parameter of the head-mounted display 10. In addition, at least one of the states (position, posture, and motion parameters) of the head-mounted display 10 described above is used as a determination target for setting the monitoring level. The specific contents of the condition information will be described in detail later.

The content output unit 30 outputs various contents used in the AR application. In the present disclosure, the content refers to the content expressed by characters, images, sounds, vibrations, or a combination thereof output from the head-mounted display 10 so that the user can perceive it. The content output unit 30 generates, for example, video content (video data) including a character (virtual object) appearing in an AR application and displays it on the display unit 20. In addition, voice content (voice data) such as character voices and sound effects is generated and output from the speaker.

In the present embodiment, the content output unit 30 outputs instruction content that prompts the user to move to the action range according to the state of the AR application based on the monitoring level. Here, the action range is, for example, the range of the real space in which the user's action is allowed in the current state of the AR application. As will be described later, the range in which the user's actions are allowed may change depending on the progress of the AR application (see FIG. 4). The content output unit 30 generates and outputs content (instruction content) that moves the user to such an action range according to the monitoring level.

The storage unit 27 includes various programs required for processing of the control unit 26, a non-volatile memory for storing various data, and a volatile memory used as a work area of the control unit 26. The various programs may be read from a portable recording medium such as an optical disk or a semiconductor memory, or may be downloaded from a server device on a network.

As shown in FIG. 3, the storage unit 27 stores the map DB 31 (Data Base) and the condition table 32. Further, the storage unit 27 stores program data (terminal program) for the terminal of the AR application, various data transmitted from the server device 40, and the like (all are not shown).

The map DB 31 is a database composed of a plurality of key frames. The key frame includes, for example, a real-world image (key frame image) used for specifying a self-position, coordinate information of the key frame, depth information, position information of a feature point, and the like. The coordinates in the key frame are 3D coordinate systems having the same origin. For example, by comparing the keyframes stored in the map DB 31 with the image captured by the imaging unit 21, it is possible to specify the position and posture (orientation) of the user (head-mounted display 10) in the real world. is there.

The condition table 32 is the condition information used for determining the monitoring level. In this embodiment, a plurality of condition tables 32 according to the progress status of the AR application are stored. In addition, each of the plurality of condition tables 32 includes a plurality of determination conditions set according to the progress status of each. The condition table 32 will be described in detail later with reference to FIG.

The server device 40 is a device that controls and manages a plurality of head-mounted displays 10 rented out to a plurality of users. The server device 40 includes a communication unit 41, a display unit 42, a control unit 43, and a storage unit 44. The communication unit 41 communicates with the communication unit 25 of the plurality of head-mounted displays 10. The display unit 42 is a display for an administrator for monitoring the progress status of the AR application and the like.

The control unit 43 manages the progress of the entire AR application based on the program data (server program) for the server of the AR application. For example, progress processing for managing the occurrence of events, branching of scenarios, and the like is executed, and the processing results are transmitted to each head-mounted display 10. On the contrary, the progress process may be executed based on the information transmitted from each head-mounted display 10. Further, the control unit 43 manages the display position and the like of the virtual object and the like.

A server program (not shown) of an AR application is stored in the storage unit 44. Further, the storage unit stores application information 45 related to the AR application and terminal information 46 for each head-mounted display 10. The application information 45 includes, for example, progress information indicating the progress state of the AR application, a display position of a virtual object, and the like. Further, the terminal information 46 includes the position, posture, monitoring level, and the like of each head-mounted display 10. The application information 45 and the terminal information 46 are transmitted to the storage unit 27 of each head-mounted display 10 and shared between the terminals.

<Overview>
Hereinafter, the outline of the present technology will be described by taking as an example the case where the head-mounted display 10 which is an AR display terminal is rented out at the venue of LBE (Location Based Entertainment). Here, LBE is an event in which a game or the like using AR is performed at a venue set in a real space. In this technology, a monitoring process is executed to suppress the theft of taking out the head-mounted display 10 from the event venue without permission. In addition to LBE, this technology can be applied to arbitrary services such as tourist guides and educational support provided by renting AR display terminals.

FIG. 4 is a schematic diagram showing an example of AR display. 4A and 4B schematically show a user 1 wearing a head-mounted display 10 and a virtual object 2 displayed as the contents of an AR application. These virtual objects 2 are visible only to the user 1 who wears the head-mounted display 10. In FIGS. 4A and 4B, the progress status of the AR application is different. Further, it is assumed that the user 1a and the user 1b are set to play a role in the AR application.

Here, it is assumed that a plurality of head-mounted displays 10 (user 1) use the same map DB 31 and the initial position estimation of each terminal has been performed. This makes it possible to represent the position and orientation of each head-mounted display 10 or the position and orientation of the virtual object 2 and the like in a common 3D coordinate system. In the following, the position and posture of the head-mounted display 10 may be described as the position and posture of the user 1.

Further, each head-mounted display 10 transmits its own position to the server device 40 that manages the progress of the AR application, and receives the self-position (terminal information 46) of another head-mounted display 10. Therefore, each head-mounted display 10 stores information such as where each user 1 is in the real world. Note that these information are shared only between the head-mounted display 10 and the server device 40. Therefore, the position and the like of the other user 1 are not displayed on the display unit 20 and the like.

In FIG. 4A, an event occurs in which a bird-shaped character 2a is captured by using a basket-shaped item 2b. Both the character 2a and the item 2b are virtual objects 2. In this event, one user 1a plays a role of attracting the attention of the character 2a. The other user 1b is responsible for operating the basket-shaped item 2. In this case, the range in which the users 1a and 1b act is considered to be, for example, a range near the character 2a. Further, the line of sight of the user 1a is directed to the character 2a, and the line of sight of the user 1b is directed to the item 2b.

In FIG. 4B, an event for searching the virtual object 2 (bird-shaped character 2c) displayed so as to fly in the real space has occurred. In this event, users 1a and 1b both play a role in searching for character 2c. In this case, the range in which the users 1a and 1b act is considered to be a wider range than, for example, when the event in FIG. 4A occurs because the character 2c is searched. When the character 2c is found, it is considered that the eyes of the users 1a and 1c are both directed to the character 2c.

As described above, in the AR application, the display position of the virtual object 2 is updated according to the progress (progress state) of the game, and the location movement / posture change of the user 1 is generated accordingly. Therefore, the posture and the range of movement that the user can take differ depending on the progress status of the AR application, the role of each user 1 in the AR application, the display position of the virtual object 2, and the like. Therefore, for example, it is difficult to determine the possibility of the theft of the terminal only by the position of the user 1 (that is, the position of the head-mounted display 10).

In the present embodiment, the level setting unit 29 acquires condition information according to the progress status of the AR application. In this case, the condition information includes the determination condition set according to the current progress state. For example, in FIGS. 4A and 4B, since the type of event occurring (progress state) is different, the condition information to be read (condition table 32) is different. This makes it possible to properly determine the state of the head-mounted display 10 regardless of the progress state of the AR application.

Further, in the present embodiment, the level setting unit 29 acquires condition information according to the role of the user 1 in the AR application. In this case, the condition information includes the determination condition set according to the role of the user 1. For example, in FIG. 4A, the roles of user 1a and user 1b are different. Therefore, the head-mounted display 10 used by each of the users 1a and 1b reads the condition information suitable for the role of the user. This makes it possible to properly determine the state of the head-mounted display 10 regardless of the role of the user 1.

The method of referring to the role of user 1 is not limited. For example, when the role of the user 1 is set in advance, the condition information corresponding to the role is read. Further, for example, when the role of the user 1 is switched in the middle of the game, the condition information suitable for the current role of the user 1 is read. Therefore, for example, even if the same event occurs, if the roles are different, the condition information to be read is different.

These condition information includes determination conditions based on the display position, posture, and the like of the virtual object 2. Therefore, for example, even when the virtual object 2 moves, it is possible to determine the relative position and the relative posture of the head-mounted display 10 with respect to the virtual object 2. As a result, the state of the head-mounted display 10 can be appropriately determined regardless of the display position of the virtual object 2.

As described above, in the present technology, the condition information according to the current state of the AR application (scenario progress status, role of user 1, display position of virtual object 2, etc.) is read. The monitoring level is set based on this condition information. For the head-mounted display 10 of the user 1 who deviates from the conditions, monitoring contents such as instruction contents and warnings for guiding the user 1 are output according to the monitoring level. This makes it possible to determine the possibility of the theft of the head-mounted display 10 step by step and suppress the theft act itself.

<Condition table>
FIG. 5 is an example of the condition table 32 for determining the state of the head-mounted display 10. In the following, the user 1 wearing the head-mounted display 10 to be determined will be referred to as player A. Further, the player B and the player C are set as another user 1 who plays the game of the AR application together with the user 1. The condition table 32 shown in FIG. 5 is a condition table 32 relating to the player A.

For example, the storage unit 27 stores a plurality of condition tables 32 associated with the progress status of the AR application and the role in the AR application. From these condition tables 32, the condition table 32 corresponding to the current progress status of the AR application and the current role of player A is read. The condition table 32 is an example of condition information according to the state of the AR application.

The condition table 32 (condition information) includes a plurality of determination conditions 34 for determining the state of the head-mounted display 10. In the example shown in FIG. 5, each row of the condition table 32 corresponds to the determination condition 34. When setting the monitoring level, the state of the head-mounted display 10 is determined for all the determination conditions 34 included in the condition table 32.

In the present embodiment, an index value based on the index calculation target 35 is used as a value representing the state of the head-mounted display 10. As the index calculation target, for example, a point in the real space, another user 1 participating in the AR application, or a virtual object 2 used in the AR application is set.

A point in real space is a point that represents a specific position on a map. For example, the position of the user 1 when the event occurs, the position of the center of the area where the event occurs, and the like are set as points to be the index calculation target 35. In the example shown in FIG. 5, the point P represented by the coordinates (latitude x, longitude y, height z) corresponds to the point in the real space. The other user 1 who participates in the AR application is, for example, a player who participates in the game together with the user 1 (players B and C shown in FIG. 5). The virtual object 2 used in the AR application is a virtual object 2 (virtual object A shown in FIG. 5) displayed at the timing when the index value is calculated. At least one of these targets may be set as the index calculation target 35. In the present embodiment, the index calculation target 35 corresponds to a predetermined target.

FIG. 6 is a schematic diagram for explaining the index value of the head-mounted display 10. FIGS. 6A to 6C schematically show a view of the index calculation target 35 and the user 1 (player A) existing around the index calculation target 35 from above. The index calculation target 35 is shown as a gray person, and the user 1 is shown as a white person. The direction in which each person's nose (protruding part in the figure) faces is the front of the person.

In the present embodiment, as the index value indicating the state of the head-mounted display 10, the position difference (relative position) and the angle difference (relative orientation) of the head-mounted display 10 with respect to the index calculation target 35 are used. The position difference is the difference between the position of the index calculation target 35 and the position of the head-mounted display 10 (user 1). The angle difference is the amount of change from the angle when the angle facing the index calculation target 35 is 0 [deg], that is, the direction of the head-mounted display 10. The maximum value of the angle difference is 180 [deg].

For example, in FIG. 6A, the position difference is X [m]. Further, since the user 1 exists in front of the index calculation target 35, the angle difference is 0 [deg]. In FIG. 6B, the positional difference is X [m] as in FIG. 6A, but the user 1 exists directly behind the index calculation target 35. Therefore, the angle difference is 180 [deg]. In FIG. 6C, the distance between the user 1 and the index calculation target 35 is half that of (a) and (b), and the position difference is X / 2 [m]. Further, since the user 1 exists right next to the index calculation target 35, the angle difference is 90 [deg].

Returning to FIG. 5, a threshold value for the position difference is set in the item of "position difference" of each determination condition 34. The threshold value for the position difference is a threshold value for determining the position of the head-mounted display 10 (user 1) with respect to the index calculation target 35. By appropriately setting this threshold value, for example, it is possible to evaluate in stages how far the user 1 is from the index calculation target 35. It is also possible to define the range in which the user 1 acts by the threshold value for the position difference. This point will be described later. In the present embodiment, the threshold value for the position difference corresponds to the position determination information.

Further, a threshold value for the angle difference is set in the item of "angle difference" in each determination condition 34. The threshold value for the angle difference is a threshold value for determining the orientation of the head-mounted display 10 (user 1) with respect to the index calculation target 35. The threshold value for the angle difference can specify the direction in which the user 1 should exist as seen from the index calculation target. In the present embodiment, the threshold value for the angle difference corresponds to the directional determination information.

As described above, each determination condition 34 includes a threshold value for the position difference and the angle difference. As a result, for example, even when the index calculation target moves, the relative position and relative posture of the head-mounted display 10 can be appropriately determined, and flexible condition setting becomes possible. In each determination condition 34, only one of the threshold values of the position difference and the angle difference may be set.

Further, in the present embodiment, each of the plurality of determination conditions 34 is associated with the candidate value of the monitoring level. In the example shown in FIG. 5, a candidate value used as a candidate for the monitoring level is set in the item of "monitoring level". This candidate value is a value set as a candidate for the monitoring level when the index value of the head-mounted display 10 exceeds the threshold value for the position difference and the angle difference. By associating each determination condition 34 with the candidate value of the monitoring level, it is possible to intuitively set the condition and the like. Note that even if both the position difference and the angle difference exceed the threshold value under one determination condition 34, the candidate value is not set as it is at the monitoring level (see step 207 and the like in FIG. 8).

In this embodiment, four monitoring levels up to level 0, level 1, level 2, and level 3 are used. For each determination condition, these four levels are set as candidate values. Level 0 is the level at which the monitoring level is the lowest and the normal operation of the AR application is continued. At level 1, for example, instruction content suggesting a direction to move is output. Further, at level 2, for example, instruction content that clearly indicates the direction to be moved is output. At level 3, the operation of the AR application is stopped. The operation at each level will be described in detail later.

In this way, the condition table 32 is a table in which the index calculation target, the permissible index value (threshold value), and the candidate value of the monitoring level set when the index value exceeds the threshold value are associated with each other. ing. For example, in the judgment condition regarding the point P, when the position difference is 200 m or less, level 0 is set as the candidate value, and when it is larger than 200 m, the candidate value is raised to level 3. It can be said that it is allowed to move freely within a radius of 200 m from the point P. This makes it possible to easily detect the user 1 and the like who act far away from the point P.

Further, for example, in the determination condition 34 regarding the player B and the player C, when the position difference becomes larger than 5 m, 10 m, and 10 m, the candidate values are set to level 1, level 2, and level 3, respectively. This means that the permissible distance between players A, B, and C is set to 50 m. Therefore, for example, when the user 1 (player A) is more than 50 m away from either the player B or C, the level 3 is set as a candidate value for the monitoring level. This makes it possible to easily detect the user 1 and the like who act away from the other user 1.

In the determination condition 34 regarding the virtual object A, when the position difference is larger than 10 m and the angle difference is larger than 90 deg, the candidate value of the monitoring level is set to level 2. This means, for example, that the monitoring level of the user 1 who is sufficiently far from the virtual object A and exists behind the virtual object A is increased. This makes it possible to easily detect, for example, a user 1 or the like who acts while ignoring the virtual object A.

The type of index value used as the determination condition 34, the format of the condition table 32, and the like are not limited. For example, motion parameters such as velocity, acceleration, angular velocity, and angular acceleration of the head-mounted display 10 may be used as index values. The relative value (difference value) of the movement parameter with respect to the index calculation target 35 may be used as the index value, or the value as it is may be used as the index value. Further, the position difference and the difference of the motion parameters may be calculated for each axis and used as independent index values. In addition, the number of feature points obtained from the image captured by the imaging unit may be used as an index value. For example, when the number of feature points decreases, it is possible to determine that the user 1 is moving away from the range of the real space recorded as the map DB 31. When these index values are used, the items (columns) in the condition table 32 are added, and threshold values for determining each index value are appropriately set.

<Operation explanation>
FIG. 7 is a flowchart showing an example of the monitoring process. Hereinafter, the monitoring process executed by the head-mounted display 10 will be described with reference to the flowchart shown in FIG. 7. The process shown in FIG. 7 is executed at the same time as the start of the AR application, and is continuously executed until the end operation or the like is performed by the user 1.

[Initial position setting process]
The AR application is started, and the use of the map DB 31 is started (step 101). For example, the map DB 31 stored in the storage unit 27 or the server device 40 is read. Relocation based on the map DB 31 is executed by the self-position estimation unit 28, and the absolute position of the head-mounted display 10 is estimated (step 102). The estimated absolute position is set as the initial position of the head-mounted display 10.

In Relocalization, for example, the image information captured by the imaging unit 21 is compared with a plurality of keyframes constituting the map DB 31. Specifically, a matching process of feature points between image information and keyframes is executed, and the absolute position of the head-mounted display 10 is estimated based on the keyframes in which the feature points and the like are matched. In this case, the self-position estimation unit 28 functions as an arithmetic processing unit (Relocalizer) that performs Relocalization. In addition, the method of estimating the absolute position (initial position) is not limited.

If it is determined whether or not the absolute position estimation is completed (step 103) and it is determined that the absolute position estimation is not completed (No in step 103), step 102 is executed again. That is, the relocation is continued until the absolute position is estimated. When the estimation of the absolute position is completed (Yes in step 103), the AR application is started (step 104), and the drawing of the AR content is started. For example, the terminal program stored in the storage unit 27 is loaded, and the progress status of the AR application and the like are read from the server device 40. Then, the content output unit 30 generates image data for displaying AR content (virtual object 2) such as a character or an item according to the progress state of the AR application, and outputs the image data to the display unit 20. When the AR content is drawn, the monitoring level update process is started (step 106).

[Monitoring level update process]
FIG. 8 is a flowchart showing an example of the monitoring level update process. The process shown in FIG. 8 corresponds to the internal process of step 106 shown in FIG. 7. This update process is mainly executed by the level setting unit 29. In the update process, the current monitoring level is first set to level 0, and the monitoring level is initialized (step 201).

When the monitoring level is initialized, the condition table 32 corresponding to the current state is read (step 202). For example, the condition table 32 corresponding to the progress status of the AR application and the role of the user 1 at the timing when step 202 is executed is read from the storage unit 27. Then, the unevaluated determination condition 34 included in the condition table 32 is selected (step 203). If all the determination conditions 34 included in the condition table 32 have been evaluated, step 107 is executed. When the unevaluated determination condition 34 is selected, the process of evaluating the index value used in the determination condition 34 is executed (step 204).

FIG. 9 is a flowchart showing an example of the evaluation process of the index value. The process shown in FIG. 9 corresponds to the internal process of step 204 shown in FIG. Here, as the index values, the position difference and the angle difference described with reference to FIGS. 5 and 6 and the like are calculated, and these index values are evaluated. When another index value is used, a process for calculating the index value may be appropriately executed.

In the calculation process shown in FIG. 9, the position of the head-mounted display 10 is first read by the level setting unit 29 (step 301). For example, the self-position estimation unit 28 estimates the coordinates (absolute position) of the head-mounted display 10 in the map DB 31 using SLAM or the like. This estimation result is read into the level setting unit 29.

Next, the position / posture of the index calculation target 35 is read (step 302). For example, when the index calculation target 35 is another user 1, the position and orientation of the head-mounted display used by that user are read via the server device 40. When the index calculation target 35 is a point in the real space, the coordinate value of the point is read. When the index calculation target 35 is the virtual object 2, the current position and orientation of the virtual object 2 are read via the server device 40.

The current index value is calculated based on each position of the head-mounted display 10 and the index calculation target 35 (step 303). Here, the position difference and the angle difference for the head-mounted display 10 and the index calculation target 35 are calculated. The position difference is calculated as, for example, the Euclidean distance in the 3D coordinate system of the map DB31. The angle difference is calculated based on, for example, the horizontal position of the head-mounted display 10 with respect to the index calculation target 35. When the index calculation target 35 is a point in the real space, only the position difference is calculated.

When there are a plurality of determination conditions 34 for the same index calculation target 35, the calculated position difference and posture difference may be used for the determination of other determination conditions 34. Further, for the condition that the threshold value of the position difference (angle difference) is set to 0, the calculation process of the position difference (angle difference) may not be performed. In addition, the method and order of calculating the index value are not limited.

When the index value is calculated, the index value is evaluated based on the determination condition 34 (step 304). Specifically, the current index value and the threshold value of the determination condition 34 are compared, and an evaluation result indicating the magnitude relationship between the index value and the threshold value is calculated. For example, with respect to the position difference and the angle difference calculated in step 303, the magnitude relationship with the corresponding threshold value is calculated and recorded as an evaluation result.

Returning to FIG. 8, it is determined whether or not the current index value is larger than the threshold value of the determination condition (step 205). Here, based on the evaluation result calculated in step 304, the determination process for all the index values set in the determination condition 34 is executed. It can be said that this is a process of determining the state of the head-mounted display 10 represented by the index value based on the determination condition 34. When it is determined that all the index values are equal to or less than the threshold value (No in step 205), the processes after step 203 are executed again.

When it is determined that at least one index value among the index values set in the judgment condition 34 is larger than the threshold value (Yes in step 205), the candidate value of the monitoring level associated with the judgment condition 34 and the present Is compared with the monitoring level of (step 206). Based on this comparison result, it is determined whether or not the candidate value of the monitoring level of the determination condition is larger than the current monitoring level (step 207).

When the candidate value of the monitoring level is equal to or lower than the current monitoring level (No in step 207), it is assumed that the monitoring level does not need to be updated, and the processes after step 203 are executed again. When the candidate value of the monitoring level is larger than the current monitoring level (Yes in step 207), the candidate value of the monitoring level associated with the determination condition is assigned to the current monitoring level (step 208). That is, the monitoring level is updated to a higher level only when the value of the candidate value exceeds the current monitoring level.

When the monitoring level is updated, the index calculation target 35, which is the reference of the determination condition 34 used for the update, is stored (step 209). The index calculation target 35 is used for output processing of instruction content and the like, which will be described later. When the index calculation target 35 is stored, it is determined whether or not there is an unevaluated determination condition 34 (step 210). If the unevaluated determination condition 34 exists (Yes in step 210), the processes after step 203 are executed again. If there are no unrated determination conditions 34 (Yes in step 210), that is, if the evaluations for all the determination conditions 34 are completed, step 107 is executed.

FIG. 10 is a schematic diagram for explaining the monitoring level. FIG. 10 schematically illustrates the arrangement of the user 1a (player A), the user 1b (player B), the user 1c (player C), and the virtual object 2 (virtual object A), which are the index calculation target 35. Has been done. Further, the range of each index calculation target 35 to 5 m is illustrated by a dotted circle. Here, the monitoring level set in the arrangement shown in FIG. 10 will be described according to the condition table 32 shown in FIG. It is assumed that the user 1a exists within 200 m from the point P shown in the condition table 32.

As shown in FIG. 10, the user 1a exists within a range of 5 m from the user 1c and also exists within a range of 5 m from the virtual object 2. Therefore, under the determination condition shown in FIG. 5, the monitoring level (candidate value) of the user 1a for the player C and the virtual object A is 0. On the other hand, the user 1a exists at a position separated from the user 1b by 5 m or more. Therefore, from the determination conditions shown in FIG. 5, the monitoring level (candidate value) of the user 1a for the player B is 1. Therefore, when the condition table 32 shown in FIG. 5 is applied in the arrangement state shown in FIG. 10, the maximum candidate value, level 1, is set as the monitoring level.

As described above, in the monitoring level update process shown in FIG. 8, the level setting unit 29 determines the state (index value) of the head-mounted display 10 for each of the plurality of determination conditions 34, and is the largest candidate represented by the determination result. The value is set to the monitoring level. This makes it possible to appropriately set the monitoring level according to the state of the application.

Returning to FIG. 7, when the monitoring level is updated, the operation according to the monitoring level is executed (step 107). As described with reference to FIG. 5, in the present embodiment, four levels of monitoring from level 0 to level 3 are set. In step 107, display processing or the like suitable for these monitoring levels is executed.

It is determined whether the monitoring level is less than level 3 (step 108). As will be described later, at level 3, the monitoring mode is executed. Therefore, when the monitoring level is level 3 (No in step 108), the monitoring mode is executed as step 107. That is, when the monitoring level is set to level 3, the monitoring mode is continued thereafter. This makes it possible to continuously monitor the user 1 (user 1 whose monitoring level is set to level 3), which is determined to have a high possibility of stealing the head-mounted display 10.

If the monitoring level is less than level 3 (Yes in step 108), it is determined whether or not to terminate the AR application (step 109). For example, when an operation input or the like for instructing the termination of the AR application is accepted (Yes in step 109), the process of terminating the AR application is executed. If the AR application is not terminated (step 109), the processes after step 105 are executed again.

<Setting the range of action>
In the present embodiment, when the monitoring level is set to level 1 or level 2, the content output unit 30 outputs instruction content that guides the user to the action range according to the state of the AR application. Hereinafter, a method of setting the action range will be described with reference to FIG.

In the arrangement shown in FIG. 10, since the user 1a is separated from the user 1b by 5 m or more, the monitoring level is set to level 1. Therefore, the monitoring level can be lowered to level 0 by moving the user 1a within a range of 5 m centered on the user 1b. In this case, the range defined by the threshold value of the position difference set for the user 1b (player B) (the range having a radius of 5 m centered on the user 1b) becomes the action range 38 according to the state of the AR application.

As described above, in the present embodiment, the action range 38 to be moved by the user 1 is automatically set based on the determination condition used for setting the monitoring level. Further, apart from the determination condition, the action range 38 or the like may be set according to the state of the AR application. In this case, the action range 38 is set as a range in which the monitoring level is low. As described above, in the present embodiment, the content output unit 30 sets the action range 38 so that the monitoring level becomes smaller based on the threshold value of the position difference.

Further, the content output unit 30 outputs instruction content for moving the user 1 to the action range 38 set in this way. Specifically, guidance is performed so that the user 1a is brought closer to the index calculation target 35 (user 1b in FIG. 10), which is the target when the monitoring level is set. As the index calculation target 35, the target stored in step 209 of FIG. 8 is used.

For example, instruction content for moving the user 1a along the direction of the arrow shown in FIG. 10 is presented. As described above, in the present embodiment, the content output unit outputs instruction content for prompting the approach to the index calculation target 35, which is the reference of the determination condition 34 used for setting the monitoring level. This makes it possible to naturally lower the monitoring level.

<Operation according to the monitoring level>
The operations executed according to the monitoring level will be specifically described for each monitoring level. The operation described below is, for example, the operation executed in step 107 shown in FIG.

[Level 0: Normal operation]
If the monitoring level is set to level 0, no special processing is performed in step 107. As a result, when the end operation or the like is not performed, the process of drawing the AR content (step 105) or the like is continued. This is the behavior of a normal AR application. Further, even in the case of level 0, the monitoring level update process (step 106) is always executed. This makes it possible to constantly monitor the possibility that the head-mounted display 10 is stolen.

[Level 1: Implicit content output]
When the monitoring level is set to level 1, the first content that does not include a specific instruction of the moving direction for moving to the action range 38 is output as the instruction content. The first content is, for example, content configured so that the monitoring level returns to level 0 (the index value becomes smaller) when the presented instruction is followed, and a specific instruction or prohibition of the moving direction is displayed. Content that is not included. That is, the first content is also said to be implicit content that implies a moving direction. In this embodiment, level 1 corresponds to the first level.

FIG. 11 is a schematic diagram showing an example of the first content. 11A to 11D schematically show an example of the instruction content 50 (first content) drawn on the display unit 20 when the monitoring level is level 1.

In FIG. 11A, the instruction content 50 representing the position of the destination (action range 38 or index calculation target 35) in the moving direction in character information is displayed. In this example, the message "a bird is flying behind" can imply the position where the character "bird" (index calculation target 35) exists in the AR application. It is also possible to represent the position of the index calculation target 35 by using 3D sound or the like together with the character information.

In FIG. 11B, the instruction content 50 representing the position of the destination is displayed using the character object used in the AR application. Here, the moving direction is implied as a message from the cat-shaped character object (virtual object 2). This message may be voice, or the moving direction may be indicated by a gesture of a character object or the like.

In FIG. 11C, the instruction content 50 representing the position of the destination is displayed using the item object used in the AR application. Here, coin-shaped item objects (virtual objects 2) are arranged along the moving direction. In this case, the user 1 is guided to the destination by moving along the item object or collecting the item objects in order.

In FIG. 11D, the instruction content 50 indicating the position of the destination is displayed by moving the character object used in the AR application. Here, an animation or the like in which a bird-shaped character object (virtual object 2) moves is displayed. In this case, the user 1 is guided to the destination by moving to track the character object. In addition, the method of generating the first content is not limited, and for example, the first content may be configured by combining the above examples.

As described above, in the present embodiment, as the first content, the content representing the position of the action range 38 is used by using at least one of the character object, the item object, the character information, and the voice information used in the AR application. Be done. For example, if you are a beginner player, you can think of acting according to sounds, letters, and simple in-game rules. Therefore, it is possible to sufficiently avoid a situation in which the monitoring level becomes high by performing guidance according to the expression of the application. It is also possible to continue the monitoring process without compromising the quality of the AR experience.

FIG. 12 is a flowchart showing an example of the output processing of the first content. In FIG. 12, it is assumed that the content shown in FIG. 11A is output. First, the position of the head-mounted display 10 is read (step 401). Further, the position of the index calculation target 35 recorded in step 209 shown in FIG. 8 is read (step 402). Subsequently, the angle difference between the head-mounted display 10 and the index calculation target 35 is calculated (step 403). As a result, the orientation of the current position of the user 1 (head-mounted display 10) as seen from the index calculation target 35 is calculated.

The angle difference can be expressed as, for example, the amount of change X in the amount of rotation with respect to the vertical axis. Here, among the angles of view that the user 1 can see through the display unit 20 of the head-mounted display 10, the angle of view in the horizontal direction is described as F. For example, from the relationship between the angle of view F and the change X, it is possible to calculate the position of the index calculation target 35 with respect to the user 1 (for example, the positions of four patterns of front, back, left, and right). In the following, it is assumed that the counterclockwise rotation represents the positive value of the change X.

Based on the angle difference (change X), the wording to be displayed as the content is generated (step 404). For example, when the absolute value of the change X is within half of the horizontal angle of view F, the index calculation target 35 is likely to exist in the field of view of the user 1. Therefore, the wording is set to an empty string, and character information and the like are not displayed. If the change X is a value between (F) and (180-F), the wording is set to "left". If the change X is a value between (-F) and (F-180), the wording is set to "right". When the absolute value of the change X is a value between (180-F) and 180, the wording is set to "back", "back", or the like. The set wording is drawn as content (step 405). For example, when the word "behind" is generated, a message such as "a bird is flying behind" is drawn.

Further, the voice data for representing the position of the index calculation target 35 is read (step 406), and the read voice data is reproduced (step 407). Here, audio data suggesting the existence of a "bird" such as a bird's bark or a feather sound is used. When 3D sound or the like is available, the sound source is set to the current position or the like of the index calculation target 35 (“bird”). As a result, the position of the index calculation target 35 can be represented by sound information. When these processes are completed, the processes after step 108 are executed.

[Level 2: Explicit content output]
When the monitoring level is set to level 2, the second content including the specific instruction of the moving direction for moving to the action range is output as the instruction content 50. The second content is, for example, content configured so that the monitoring level returns to level 0 (the index value becomes smaller) when the presented instruction is followed, and a specific instruction or prohibition of the moving direction is displayed. Content that includes. That is, the second content is also said to be explicit content in which the moving direction is specified. In this embodiment, level 2 corresponds to a second level higher than the first level.

FIG. 13 is a schematic diagram showing an example of the second content. 13A to 13C schematically show an example of the instruction content 50 (second content) drawn on the display unit 20 when the monitoring level is level 2.

In FIG. 13A, the instruction content 50 including the arrow 51 indicating the moving direction and the message is displayed. In this example, the direction of movement is clearly indicated by the arrow 51. In addition, the message "Area exit warning / Please return" clearly indicates that forward movement is prohibited. A warning sound such as a beep sound may be reproduced together with the arrow 51 and the message.

In FIG. 13B, the fence-shaped virtual object 2 is displayed as the instruction content 50. In this way, the virtual object 2 may be displayed so as to remind the user 1 that the visible range cannot be passed. This makes it possible to control the behavior of the user 1 so as not to go out of the area.

In FIG. 13C, the map-like virtual object 2 is displayed as the instruction content 50. In the instruction content 50, a map on which the icon 52 indicating the current position of the user 1 and the arrow 53 indicating the moving direction are superimposed is displayed. This makes it possible for the user 1 to specify the direction in which the user 1 should move. In addition, the method for generating the second content is not limited, and for example, the second content may be configured by combining the above examples.

As described above, in the present embodiment, as the second content, content including at least one of image information (arrow 51, map, etc.) indicating the moving direction, character information, or sound information is used. By clearly indicating the moving direction in this way, for example, a malicious player can know that the position of the head-mounted display 10 is being monitored, which leads to deterrence of theft. Further, if the player is a beginner, the instruction including the specific moving direction is easier to understand than the level 1, so that the possibility of following the instruction is higher.

FIG. 14 is a flowchart showing an example of the output processing of the second content. In FIG. 14, it is assumed that the content shown in FIG. 13A is output. First, the position of the head-mounted display 10 is read (step 501). Further, the position of the index calculation target 35 recorded in step 209 shown in FIG. 8 is read (step 502).

Subsequently, the direction of the arrow 51 is calculated from the positions of the head-mounted display 10 and the index calculation target 35 (step 503). For example, a vector passing through each position of the head-mounted display 10 and the index calculation target 35 is calculated. Then, the direction from the user 1 (head-mounted display 10) toward the index calculation target 35 is calculated as the direction of the arrow 51. A virtual object 2 (arrow 51) representing the calculated direction is drawn (step 504).

Further, the wording of the message displayed together with the arrow 51 is drawn (step 505). Further, the voice data used as the warning sound is read (step 506), and the voice data is generated (step 507). As a result, it is possible to output the second content that clearly indicates the direction in which the user 1 should move by the arrow 51, the message, and the warning sound. The other second content can also be appropriately generated based on the positions of the head-mounted display 10 and the index calculation target 35 and the like.

[Level 3: Monitoring mode]
When the monitoring level is set to level 3, the operation of the AR application is stopped and the monitoring mode for monitoring the user is executed. Therefore, at level 3, various processes for preventing theft are executed as a monitoring mode without performing operations according to the AR application. In this embodiment, level 3 corresponds to a third level, which is higher than the second level.

FIG. 15 is a schematic diagram showing an example of a screen in the monitoring mode. FIG. 16 is a flowchart showing an example of the monitoring mode. As shown in FIG. 15, in the monitoring mode, the display of the AR application itself is stopped, and only the message 55 indicating that the AR application is stopped is displayed. At this time, for example, the operation input to the head-mounted display 10 via the operation unit 23 or the like is completely prohibited. That is, in the monitoring mode, the head-mounted display 10 does not respond to the operation of the user 1.

As shown in FIG. 16, in the monitoring mode, the position of the head-mounted display 10 is first read (step 601). Subsequently, the output of each sensor mounted on the head-mounted display 10 is read (step 602). When the monitoring level becomes level 3 and the monitoring mode is executed, the user 1 is likely to be a malicious person. Therefore, for example, the iris information of the user 1 is detected by using an internal camera for eye tracking or the like. Further, an image of the external environment (image of the stolen route) is detected by using an external camera (imaging unit 21). The position of the head-mounted display 10 read in steps 601 and 602 and the output of the sensor (iris information and the like) are transmitted to the server device 40 (step 603). This makes it possible to notify the event administrator of the information of the user 1.

Further, voice data such as a warning sound is read (step 604), and the read voice data is played back (step 605). The audio data is reproduced at a volume that can be heard outside the device, for example. As a result, it is possible to attract attention to the user 1 and suppress the theft. As described with reference to FIG. 7, at level 3, the above-mentioned monitoring mode processing is repeatedly executed.

[Operation when there is another user with a high monitoring level]
FIG. 17 is a schematic diagram showing an example of content display related to other users. In the following, with reference to FIG. 17, the operation when a relatively high monitoring level is set for a user other than the user 1 (target user) who is using the head-mounted display 10 will be described. ..

In FIG. 17, user 1a is the target user. User 1b is another user whose monitoring level is set higher than that of user 1a. User 1c is another user who has the same or lower monitoring level as user 1a. Of these, the user 1b is likely to be a malicious player or a beginner player because the monitoring level is relatively high.

In the present embodiment, when the monitoring level of another user is relatively high, the instruction content for urging the other user to approach is output. For example, the head-mounted display 10 of the user 1a outputs instruction content that encourages the user 1b to approach the user 1b. For example, by using the first content or the like described with reference to FIG. 11 or the like, it is possible to guide the user 1a to the vicinity of the user 1b.

As a result, the non-malicious player (user 1a) approaches the malicious player or the beginner player (user 1b), and as a result, the index value is expected to decrease. Further, as the distance between the players becomes closer, it becomes possible to monitor each other, and it is possible to sufficiently reduce the possibility of theft.

Further, the object to be searched by the user 1 in the AR application may be displayed in the vicinity of another user whose monitoring level is relatively high. For example, as shown in FIG. 17, a bird-shaped virtual object 2 to be searched by the user 1a is arranged in the vicinity of the user 1b. As a result, the user 1a sees the user 1b having a high monitoring level in the process of searching for the virtual object 2.

For example, when the virtual object 2 to be searched is displayed, the line of sight of a non-malicious player (user 1a or user 1c) is directed to the virtual object 2, or the non-malicious player moves closer to the virtual object 2. It is expected to do. Therefore, by controlling the position of the virtual object 2 and displaying it in the vicinity of the malicious player (user 1b), the malicious player can be seen in the camera of the non-malicious player, which is effective in deterring theft. Can be demonstrated.

As described above, in the control unit 26 according to the present embodiment, the monitoring level is set based on the state of the head-mounted display 10 and the state of the AR application. Based on this monitoring level, the instruction content 50 prompting the user 1 to move to the action range 38 suitable for the state of the AR application is output. As a result, it is possible to manage the range in which the user 1 acts, monitor suspicious behavior, and prevent the theft of the terminal device for AR.

When renting an AR terminal at an event venue where LBE etc. is held, there is a risk that the rented terminal will be stolen by a malicious participant. If the AR terminal is stolen, in addition to financial loss, there is a risk that the number of participants in the event will decrease and sensitive information such as personal information and content will be leaked.

Conventional anti-theft measures include measures such as the operation staff monitoring near the participants or installing a gate at the entrance and exit of the event venue. These measures increase human costs and physical costs for gates and substrates. In addition, if a gate is installed, security will be vulnerable to entry and exit from other than the gate, making it difficult to hold an event in an open space such as a park. In addition, the method of installing a surveillance camera requires a high cost for introducing a base material. In addition, it is difficult to cover the entire venue because there are places (toilet, etc.) where it is difficult to install surveillance cameras. Further, in an AR event, since the user can freely perform various actions, it is difficult to judge the possibility of theft only from the camera image.

In the present embodiment, the monitoring level is set by determining the state such as the position and posture of the head-mounted display 10 by using the condition information (condition table 32) according to the state of the AR application. As a result, even if the type of event in the AR application, the progress of the scenario, or the like changes, the monitoring level can be set appropriately.

Further, these processes are automatically executed by using the head-mounted display 10 and the server device 40. Therefore, no additional cost or the like for monitoring theft is required. Therefore, it is possible to hold an AR event such as LBE with peace of mind even in an open space such as a park where a gate or the like cannot be installed. Moreover, since it is different from surveillance by a surveillance camera or the like, it is possible to determine the possibility of theft (surveillance level) regardless of the position of the user.

Further, according to the monitoring level, instruction content for urging the user 1 to move to an action range where the action is allowed is output. This action range 38 is set based on the condition table 32 according to the state of the AR application. As a result, the user 1 can be naturally guided to the range where the user 1 should be. As a result, even if the non-malicious user 1 deviates from the action range 38, it is possible to return to an appropriate position without destroying the world view. Further, it can be seen that the position of the head-mounted display 10 is monitored even for the malicious user 1, and it is possible to prevent theft.

<Other Embodiments>
The present technology is not limited to the embodiments described above, and various other embodiments can be realized.

The position difference described with reference to FIG. 6 and the like was mainly a position difference on the horizontal plane. At an AR event, it is possible to use an area where the height changes, such as a building or underground, as a venue. Therefore, for example, a three-dimensional position difference (position difference for each axis) may be determined. Thereby, for example, it is possible to appropriately set the monitoring level even for users who are at different heights while being at the same latitude and longitude as the index calculation target.

In the above, as the determination condition, the threshold value for the position difference and the angle difference is mainly determined. The content of the judgment condition is not limited. For example, conditions for determining the position information, posture information, etc. of the head-mounted display as they are may be used. When the position difference was used as the index value, the distance between the index calculation target and the head-mounted display was set as the threshold value. Not limited to this, for example, an area (range) arbitrarily set in the venue may be set as a determination condition as it is. In this case, for example, an area to be level 1 or an area to be level 2 may be set according to the state of the AR application.

When the motion parameter of the head-mounted display is used as an index value, a threshold value or the like related to the motion parameter is set. For example, by determining the speed, acceleration, or the like, it is possible to detect an operation that the user does not normally perform (for example, a state in which the top and bottom are inverted for a long time). This makes it possible to prevent, for example, the act of putting the device in a bag or the like and trying to take it away. In addition, it is possible to easily detect a state that is not expected in normal use, such as a case where the person moves at a speed higher than the moving speed of a human being or a case where there is no movement for a certain period of time or longer.

In the above, condition information (condition table) according to the progress status of the AR application was mainly used. For example, condition information according to the number of users participating in the AR application, the number of beginners included in the participants, the type of operation expected in the AR application, and the like may be appropriately set and used. In addition, conditional information suitable for various situations assumed when the AR application is operated may be used.

In the above, as an embodiment of the information processing device according to the present technology, a head-mounted device such as a head-mounted display has been given as an example. However, the information processing device according to the present technology may be realized by an arbitrary computer that is configured separately from the head-mounted device and is connected to the head-mounted device via a wired or wireless device. For example, an external device that controls a head-mounted device such as a PC or game machine connected to the HMD or a smartphone connected to the headphones may be used as the information processing device according to the present technology. Further, for example, the information processing method according to the present technology may be executed by a cloud server (server device or the like). Alternatively, the information processing method according to the present technology may be executed in conjunction with the HMD or the like and another computer.

That is, the information processing method and the program according to the present technology can be executed not only in a computer system composed of a single computer but also in a computer system in which a plurality of computers operate in conjunction with each other. In the present disclosure, the system means a set of a plurality of components (devices, modules (parts), etc.), and it does not matter whether or not all the components are in the same housing. Therefore, a plurality of devices housed in separate housings and connected via a network, and one device in which a plurality of modules are housed in one housing are both systems.

The information processing method and program execution related to this technology by a computer system are, for example, when the monitoring level is set, the instruction content is output, etc. are executed by a single computer, or when each process is executed by a different computer. Including both. Further, the execution of each process by a predetermined computer includes causing another computer to execute a part or all of the process and acquire the result.

That is, the information processing method and program according to the present technology can be applied to a cloud computing configuration in which one function is shared by a plurality of devices via a network and jointly processed.

It is also possible to combine at least two feature parts among the feature parts according to the present technology described above. That is, the various feature portions described in each embodiment may be arbitrarily combined without distinction between the respective embodiments. Further, the various effects described above are merely examples and are not limited, and other effects may be exhibited.

In the present disclosure, "same", "equal", "orthogonal", etc. are concepts including "substantially the same", "substantially equal", "substantially orthogonal", and the like. For example, a state included in a predetermined range (for example, a range of ± 10%) based on "exactly the same", "exactly equal", "exactly orthogonal", etc. is also included.

The present technology can also adopt the following configurations.
(1) A level setting unit that sets a monitoring level based on the state of the AR display terminal used by the user and the state of the AR application operating on the AR display terminal.
An information processing device including a content output unit that outputs instruction content that prompts the user to move to an action range according to the state of the AR application based on the monitoring level.
(2) The information processing device according to (1).
The level setting unit is an information processing device that acquires condition information according to the state of the AR application, determines the state of the AR display terminal based on the condition information, and sets the monitoring level.
(3) The information processing device according to (2).
The state of the AR application includes the progress state of the AR application, and the level setting unit is an information processing device that acquires the condition information according to the progress state.
(4) The information processing device according to (2) or (3).
The level setting unit is an information processing device that acquires the condition information according to the role of the user in the AR application.
(5) The information processing device according to any one of (2) to (4).
The condition information is an information processing device including a plurality of determination conditions for determining the state of the AR display terminal.
(6) The information processing device according to (5).
The state of the AR display terminal is an information processing device including at least one of the position, posture, or movement parameters of the AR display terminal.
(7) The information processing device according to (5) or (6).
Each of the plurality of determination conditions is associated with the candidate value of the monitoring level.
The level setting unit is an information processing device that determines the state of the AR display terminal for each of the plurality of determination conditions and sets the maximum candidate value represented by the determination result to the monitoring level.
(8) The information processing device according to any one of (5) to (7).
The determination condition includes at least one of position determination information for determining the position of the AR display terminal with respect to a predetermined target and orientation determination information for determining the orientation of the AR display terminal with respect to the predetermined target. Information processing device.
(9) The information processing apparatus according to (8).
The predetermined target is an information processing device that is at least one of a point in real space, another user participating in the AR application, or a virtual object used in the AR application.
(10) The information processing device according to (8) or (9).
The determination condition includes the position determination information.
The content output unit is an information processing device that sets the action range so that the monitoring level becomes smaller based on the position determination information.
(11) The information processing apparatus according to any one of (8) to (10).
The content output unit is an information processing device that outputs the instruction content that prompts the approach to the predetermined target that is the reference of the determination condition used for setting the monitoring level.
(12) The information processing apparatus according to any one of (1) to (11).
When the monitoring level is set to the first level, the content output unit outputs the first content that does not include a specific instruction of the moving direction for moving to the action range as the instruction content. Processing equipment.
(13) The information processing apparatus according to (12).
The first content is an information processing device that represents the position of the action range using at least one of a character object, an item object, character information, or voice information used in the AR application.
(14) The information processing device according to (12) or (13).
When the monitoring level is set to a second level higher than the first level, the content output unit includes a second movement direction specific instruction for moving to the action range as the instruction content. Information processing device that outputs the contents of.
(15) The information processing apparatus according to (14).
The second content is an information processing device that includes at least one of image information, character information, or sound information that indicates the moving direction.
(16) The information processing apparatus according to (14) or (15).
When the monitoring level is set to a third level higher than the second level, the content output unit is an information processing device that stops the operation of the AR application and executes a monitoring mode for monitoring the user. ..
(17) The information processing apparatus according to any one of (1) to (16).
The content output unit is an information processing device that outputs the instruction content that prompts the other user to approach the other user when the monitoring level of the other user is relatively high.
(18) The information processing apparatus according to (17).
The content output unit is an information processing device that displays an object to be searched for by the user in the vicinity of the other user in the AR application.
(19) The monitoring level is set based on the state of the AR display terminal used by the user and the state of the AR application operating on the AR display terminal.
An information processing method in which a computer system executes instruction content that prompts the user to move to an action range according to the state of the AR application based on the monitoring level.
(20) A step of setting a monitoring level based on the state of the AR display terminal used by the user and the state of the AR application operating on the AR display terminal.
A program that causes a computer system to execute a step of outputting instruction content that prompts the user to move to an action range according to the state of the AR application based on the monitoring level.

1, 1a to 1c ... User 2 ... Virtual object 10 ... Head-mounted display 20 ... Display unit 21 ... Imaging unit 26 ... Control unit 27 ... Storage unit 28 ... Self-position estimation unit 29 ... Level setting unit 30 ... Content output unit 31 ... Map DB
32 ... Condition table 34 ... Judgment condition 35 ... Index calculation target 38 ... Action range 40 ... Server device 100 ... AR display system

Claims (20)

A level setting unit that sets the monitoring level based on the state of the AR display terminal used by the user and the state of the AR application operating on the AR display terminal.
An information processing device including a content output unit that outputs instruction content that prompts the user to move to an action range according to the state of the AR application based on the monitoring level. The information processing device according to claim 1.
The level setting unit is an information processing device that acquires condition information according to the state of the AR application, determines the state of the AR display terminal based on the condition information, and sets the monitoring level. The information processing device according to claim 2.
The state of the AR application includes the progress state of the AR application.
The level setting unit is an information processing device that acquires the condition information according to the progress state. The information processing device according to claim 2.
The level setting unit is an information processing device that acquires the condition information according to the role of the user in the AR application. The information processing device according to claim 2.
The condition information is an information processing device including a plurality of determination conditions for determining the state of the AR display terminal. The information processing device according to claim 5.
The state of the AR display terminal is an information processing device including at least one of the position, posture, or movement parameters of the AR display terminal. The information processing device according to claim 5.
Each of the plurality of determination conditions is associated with the candidate value of the monitoring level.
The level setting unit is an information processing device that determines the state of the AR display terminal for each of the plurality of determination conditions and sets the maximum candidate value represented by the determination result to the monitoring level. The information processing device according to claim 5.
The determination condition includes at least one of position determination information for determining the position of the AR display terminal with respect to a predetermined target and orientation determination information for determining the orientation of the AR display terminal with respect to the predetermined target. Information processing device. The information processing device according to claim 8.
The predetermined target is an information processing device that is at least one of a point in real space, another user participating in the AR application, or a virtual object used in the AR application. The information processing device according to claim 8.
The determination condition includes the position determination information.
The content output unit is an information processing device that sets the action range so that the monitoring level becomes smaller based on the position determination information. The information processing device according to claim 8.
The content output unit is an information processing device that outputs the instruction content that prompts the approach to the predetermined target that is the reference of the determination condition used for setting the monitoring level. The information processing device according to claim 1.
When the monitoring level is set to the first level, the content output unit outputs the first content that does not include a specific instruction of the moving direction for moving to the action range as the instruction content. Processing equipment. The information processing device according to claim 12.
The first content is an information processing device that represents the position of the action range using at least one of a character object, an item object, character information, or voice information used in the AR application. The information processing device according to claim 12.
When the monitoring level is set to a second level higher than the first level, the content output unit includes a second movement direction specific instruction for moving to the action range as the instruction content. Information processing device that outputs the contents of. The information processing apparatus according to claim 14.
The second content is an information processing device that includes at least one of image information, character information, or sound information that indicates the moving direction. The information processing apparatus according to claim 14.
When the monitoring level is set to a third level higher than the second level, the content output unit is an information processing device that stops the operation of the AR application and executes a monitoring mode for monitoring the user. .. The information processing device according to claim 1.
The content output unit is an information processing device that outputs the instruction content that prompts the other user to approach the other user when the monitoring level of the other user is relatively high. The information processing apparatus according to claim 17.
The content output unit is an information processing device that displays an object to be searched for by the user in the vicinity of the other user in the AR application. The monitoring level is set based on the state of the AR display terminal used by the user and the state of the AR application operating on the AR display terminal.
An information processing method in which a computer system executes instruction content that prompts the user to move to an action range according to the state of the AR application based on the monitoring level. A step of setting the monitoring level based on the state of the AR display terminal used by the user and the state of the AR application operating on the AR display terminal, and
A program that causes a computer system to execute a step of outputting instruction content that prompts the user to move to an action range according to the state of the AR application based on the monitoring level.

Images