JP2024049400A - Information Processing System - Google Patents

Abstract

To provide an information processing system which can present, to a user, information which cannot be visually recognized by the user.SOLUTION: An information processing system 1 comprises: a storage unit 51 which stores spatial structure data; an acquisition unit 52 which acquires a position of a user, and a visibility state of the user; n identification unit 53 which identifies the spatial structure data corresponding to the position of the user from the storage unit 51; a first estimation unit 54 which estimates an ideal visibility state in which the user can visually recognize all objects which could be included in a field of view of the user on the basis of the position of the user in the spatial structure data identified by the identification unit 53; a second estimation unit 55 which compares the ideal visibility state estimated by the first estimation unit 54 with a visibility state of the user and estimates an object which the user cannot visually recognize from a difference between the ideal visibility state and the visibility state of the user; and an output unit 56 which outputs the object estimated that the user cannot visually recognize by the second estimation unit 55 to allow the user to visually recognize it.SELECTED DRAWING: Figure 4

Description

The present invention relates to an information processing system.

Literature 1 describes a terminal device that uses AR (Augmented Reality) to present information that is not included in the user's field of view in real space to the user. With such a device, the user can obtain information outside of his or her field of view, improving user convenience.

Patent Publication No. 2013-518275

Here, even if the information is in the user's line of sight (information that exists within the user's field of vision), if there is an object between the information and the user that physically blocks the user's line of sight, the user may not be able to see the information. In such a case, the above-mentioned conventional device may not be able to present the above-mentioned information to the user, which may impair the convenience of the user who uses the information and the profits of the individual or organization that provides the information. For example, if the user's line of sight is blocked by a car or a roadside tree, etc., and the user cannot see a traffic sign, the conventional device may not be able to present the traffic sign to the user while driving, which may impair the user's safety. Also, for example, if a corporate advertisement on a train is blocked by a passenger or a strap, the conventional device may not be able to present the advertisement to the user, which may result in the user losing an opportunity to see the advertisement and impairing the profits of the company that places the advertisement.

The present invention has been made in consideration of the above-mentioned circumstances, and aims to provide an information processing system that can appropriately present information to a user that the user cannot visually recognize.

An information processing system according to one aspect of the present invention includes a storage unit that stores spatial structure data that represents an object in a real space in a three-dimensional virtual space and that represents the shape of the object at a position in the virtual space that corresponds to the position of the object in the real space; an acquisition unit that acquires a user's position and the user's viewing state; an identification unit that identifies the spatial structure data that corresponds to the user's position from the storage unit; a first estimation unit that estimates an ideal viewing state in which the user can view one or more objects that may be included in the user's field of view based on the user's position in the spatial structure data identified by the identification unit; a second estimation unit that compares the ideal viewing state estimated by the first estimation unit with the user's viewing state and estimates an object that the user is unable to view from the difference between the ideal viewing state and the user's viewing state; and an output unit that outputs the object that the second estimation unit estimates the user is unable to view so that the object can be viewed by the user.

In an information processing system according to one aspect of the present invention, the user's position and the user's viewing state are acquired, spatial structure data corresponding to the user's position is identified, and in the identified spatial structure data, an ideal viewing state is estimated based on the user's position, the ideal viewing state is compared with the user's viewing state, and an object that the user is unable to view is estimated from the difference between the ideal viewing state and the user's viewing state, and the object that is estimated to be unable to be viewed by the user is output so that the user can view it. Here, in real space, information about the user's surroundings is physically blocked, and the user is unable to view the information, which may impair the convenience of the user who uses the information and the profits of individuals or groups who provide the information. For example, the user's line of sight is blocked by cars, roadside trees, etc., and the user cannot view traffic signs while driving or walking, which impairs the safety of the user. In addition, when a company's advertisement on a train is blocked by passengers, hanging straps, etc., the user loses the opportunity to view the advertisement, which impairs the company's profits. In this regard, in an information processing system according to one aspect of the present invention, objects that the user cannot see in real space are displayed on the user's terminal, such as an AR display device. This makes it possible to appropriately present information that the user cannot see to the user.

The present invention provides an information processing system that can appropriately present information to a user that the user cannot see.

FIG. 1 is a diagram illustrating an overview of an information processing system according to an embodiment of the present invention. FIG. 1 is a diagram illustrating an overview of an information processing system according to an embodiment of the present invention. FIG. 1 is a diagram illustrating an overview of an information processing system according to an embodiment of the present invention. 1 is a block diagram showing a functional configuration of an information processing system according to an embodiment of the present invention. 11 is a diagram illustrating an example of information that a communication terminal transmits to a server. FIG. FIG. 11 is a diagram illustrating an example of information processing in a server. FIG. 11 is a diagram illustrating an example of a method for presenting an object that the user cannot visually recognize to the user. FIG. 11 is a diagram illustrating an example of a method for presenting an object that the user cannot visually recognize to the user. 4 is a flowchart showing a process executed by the information processing system according to the present embodiment. 4 is a flowchart showing a process executed by the information processing system according to the present embodiment. FIG. 11 is a diagram illustrating an example of a method for presenting an object that the user cannot visually recognize to the user. 2 is a diagram showing the hardware configurations of a communication terminal, a positioning server, and a spatial structure server included in the information processing system according to the present embodiment. FIG.

Embodiments of the present invention will now be described in detail with reference to the accompanying drawings. In the description of the drawings, the same or equivalent elements will be designated by the same reference numerals, and duplicate descriptions will be omitted.

The information processing system 1 shown in FIG. 1, FIG. 2, and FIG. 4 is a system that presents to a user information that is in the user's line of sight but that the user cannot see. More specifically, the information processing system 1 is a system that acquires information such as the position of the communication terminal 10 and the user's viewing state, identifies spatial structure data from the position of the communication terminal 10, estimates an ideal viewing state from the identified spatial structure data and the position of the communication terminal 10, compares the ideal viewing state with the user's viewing state, estimates information that the user cannot see from the difference between the ideal viewing state and the user's viewing state, and presents the information to the user. Here, the user's viewing state is information that at least indicates a state that is estimated to be actually viewed by the user in real space (details will be described later), and the ideal viewing state is information that indicates a state in which the user can see all objects that can be included in the user's field of view in real space (ideal viewing state) (details will be described later). By comparing the ideal viewing state with the user's viewing state, the information processing system 1 identifies objects that the user is unable to view despite being within the user's field of view, and presents the objects to the user. The information processing system 1 also updates objects in the three-dimensional virtual space based on the acquired information such as the position of the communication terminal 10 and the user's viewing state. The information processing system 1 includes a communication terminal 10, a positioning server 30, and a spatial structure server 50. The communication terminal 10 is a terminal carried by the user or a terminal placed around the user. First, an overview of the processing performed by the information processing system 1 will be described.

In the information processing system 1, for example, an image captured by the communication terminal 10 is transmitted to the positioning server 30. In the example shown in FIG. 1, an image P1 captured by the communication terminal 10 is shown. An obstacle D1 and an object X3 indicating a "pedestrian crossing traffic sign" are shown in the captured image P1. The communication terminal 10 transmits the captured image P1 to the positioning server 30. The positioning server 30 acquires global position information based on the captured image P1 captured by the communication terminal 10, and transmits the global position information to the communication terminal 10. The global position information is position information (absolute position information) indicated in a common coordinate system that can be used by any device. The global position information includes, for example, information on position, direction, and tilt. Details of the global position information will be described later. Note that the acquisition of the global position information is not limited to a method using a captured image. For example, the global position information may be acquired in the positioning server 30 based on information acquired using GPS, RTK, geomagnetism, Wi-Fi, Bluetooth (registered trademark), etc.

The communication terminal 10 estimates the position of the communication terminal 10 by acquiring global position information from the positioning server 30. The communication terminal 10 then transmits information including at least the acquired global position information and the captured image to the spatial structure server 50. In the example shown in FIG. 1 and FIG. 2, the communication terminals 10 carried by the users A and B (hereinafter referred to as "user's communication terminals 10"), or the communication terminals 10 with imaging functions placed around the users (hereinafter referred to as "surrounding communication terminals 10") transmit the captured image R captured by each communication terminal 10 to the positioning server 30. The user's communication terminal 10 and the surrounding communication terminals 10 receive the global position information of each terminal from the positioning server. Each communication terminal 10 transmits the global position information (the user's position and the terminal's position in FIG. 2) and the captured image to the spatial structure server 50.

The spatial structure server 50 stores spatial structure data (details will be described later) that is data that represents objects in real space in a three-dimensional virtual space. The spatial structure server 50 receives information including at least global position information and a captured image from the communication terminal 10. The spatial structure server 50 identifies spatial structure data corresponding to the global position information. The spatial structure server 50 also estimates the user's visual state from the captured image. In the example shown in FIG. 1, the spatial structure data D (see the lower right diagram of FIG. 1) includes an object X1 indicating a "traffic light" and an object X2 indicating a "stop sign" in addition to an object X3 shown in the captured image P1 (see the lower left diagram of FIG. 1). The objects X1, X2, and X3 are objects in real space. The spatial structure server 50 identifies the spatial structure data D based on the global position information acquired from the captured image P1 by the positioning server 30. The identified spatial structure data D represents the shapes of the objects X1, X2, and X3 represented in the three-dimensional virtual space. Furthermore, the spatial structure server 50 estimates from the captured image P1 that "the user is viewing the object X3 in real space." An example of the estimation of the user's viewing state by the spatial structure server 50 will be described later. Note that in this embodiment, the spatial structure server 50 performs the estimation process of the user's viewing state, but as described in the modified example, this is not limited to this. In other words, the estimation process of the user's viewing state may be performed in a server other than the spatial structure server 50. For example, the estimation process may be performed by the communication terminal 10.

The spatial structure server 50 also updates the information of objects in the spatial structure data based on the objects in real space included in the user's visual state. In the example shown in Figures 2 and 3, the spatial structure server 50 receives global position information and captured images from communication terminals 10 carried by users A and B, or communication terminals 10 such as surveillance cameras. The spatial structure server 50 obtains the latest states of objects X1, X2, and X3 in real space from the received global position information and captured images. The spatial structure server 50 updates objects X1, X2, and X3 to their latest states in the spatial structure data.

The spatial structure server 50 estimates an ideal viewing state (described later) for the identified spatial structure data based on global position information. The spatial structure server 50 compares the ideal viewing state with the user's viewing state, and estimates objects that the user is unable to view from the difference between the ideal viewing state and the user's viewing state. The spatial structure server 50 transmits the objects that the user is unable to view to the communication terminal 10. When transmitting the object to the communication terminal 10, the spatial structure server 50 changes the position information of the object so that the object can be viewed by the user. The spatial structure server 50 transmits the object and the changed position information to the communication terminal 10. The communication terminal 10 displays the object on a screen or the like on the communication terminal 10 based on the received position information.

In the example shown in FIG. 2, the spatial structure server 50 estimates that the state in which objects X1, X2, and X3 are visible to user A is the ideal viewing state of user A based on the global position information received from user A. The spatial structure server 50 estimates (acquires) the state in which only object X3 is visible as the viewing state of user A based on the captured image received from user A. The spatial structure server 50 compares the ideal viewing state of user A with the viewing state of user A. The spatial structure server 50 estimates that the objects that user A cannot view are objects X1 and X2 from the difference between the ideal viewing state of user A and the viewing state of user A. The spatial structure server 50 changes the position information of objects X1 and X2 so that user A can view them, and transmits the changed position information and objects X1 and X2 to the communication terminal 10 of user A. The communication terminal 10 of user A receives objects X1 and X2 from the spatial structure server 50. User A's communication terminal 10 displays the received objects X1 and X2 on a screen or the like on the terminal that corresponds to the updated location information.

3, the spatial structure server 50 acquires the captured image P2 including the objects X1 and X2 and the position information of the user B from the user B who can see the objects X1 and X2. The spatial structure server 50 acquires the latest state of the objects X1 and X2 from the acquired captured image P2 and the position information. The spatial structure server 50 updates the objects X1 and X2 in advance in the three-dimensional virtual space based on the acquired latest state. Here, when the spatial structure server 50 estimates that the objects that the user A cannot see are the objects X1 and X2, the spatial structure server 50 changes the position information of the objects X1 and X2 so that the user A can see them. The spatial structure server 50 transmits the changed position information and the objects X1 and X2 to the screen G of the communication terminal 10 of the user A. The communication terminal 10 displays the shape of the object X1 in the changed position information, and sets the displayed object as the object Y1 (see the left diagram in FIG. 3). The communication terminal 10 performs a similar display process on object X2, and the displayed object becomes object Y2 (see the left diagram in Figure 3).

Next, referring to FIG. 4, the functional components of the communication terminal 10, the positioning server 30, and the spatial structure server 50 will be described. The positioning server 30 has a memory unit 31 and a positioning unit 32 as functional components.

The storage unit 31 stores map data 300. In the map data 300, the feature amount (e.g., luminance direction vector) of a feature point included in a captured image acquired in advance is associated with global position information, which is absolute position information associated with the feature point. The map data 300 is, for example, a 3D point cloud. The map data 300 is generated based on a large amount of captured images captured in advance by a stereo camera (not shown) that can capture an object from multiple different directions at the same time. A feature point is a point that is prominently detected in an image, and is, for example, a point with a high (or low) luminance (intensity) compared to other areas. The global position information of a feature point is global position information set in association with the feature point, and is global position information in the real world for the area indicated by the feature point in the image. Note that the association of global position information with each feature point can be performed by a conventionally known method.

The storage unit 31 stores three-dimensional global position information as the global position information of the feature points of the map data 300. The storage unit 31 stores, for example, the latitude, longitude, and altitude of the feature points as the three-dimensional global position information of the feature points. Note that the storage unit 11 may store multiple divided map data in which the map data 300 is divided into certain regions according to the global position information.

The positioning unit 32 estimates global position information (three-dimensional position information) of the communication terminal 10 at the time of capturing an image on the communication terminal 10, based on the captured image captured on the communication terminal 10 and the map data 100 stored in the storage unit 31. Specifically, the positioning unit 32 matches the feature points of the map data 100 with the feature points of the captured image captured on the communication terminal 10, and identifies the area of the map data 300 that corresponds to the captured image. The positioning unit 32 then estimates the capturing position of the captured image (i.e., the global position information of the communication terminal 10 at the time of capturing an image) based on the global position information associated with the feature points of the map data 300 related to the identified area. The positioning unit 32 transmits the positioning result to the communication terminal 10.

The positioning result includes information on the direction (direction in the three-dimensional coordinate system of roll, pitch, and yaw) estimated from the captured image in addition to the global position information. The positioning unit 32 may obtain the global position information based on captured images captured at regular intervals by the communication terminal 10, or may obtain the global position information based on captured images captured by the communication terminal 10 at the timing when an instruction from the user is received.

The communication terminal 10 is a terminal carried by a user or a terminal placed around the user. The communication terminal 10 is, for example, a terminal configured to perform wireless communication. The communication terminal 10 is, for example, a smartphone, a tablet terminal, a PC, a goggle-type wearable device, etc. The communication terminal 10 may also be an imaging device with a communication function, such as a surveillance camera or a fixed-point camera. When an application is executed, for example, the communication terminal 10 performs imaging using an installed camera. Then, the communication terminal 10 transmits the captured image to the positioning server 30 and acquires a positioning result corresponding to the captured image from the positioning server 30. The communication terminal 10 acquires global position information and the user's viewing direction from the positioning result. The communication terminal 10 acquires user's objective information (details will be described later) from manual input by the user or from the search history of the browser of the communication terminal 10. The communication terminal 10 transmits the captured image, the acquired global position information, the user's viewing direction, and the user's objective information to the spatial structure server 50. In the example shown in FIG. 1, the user's viewing direction is information identified from the positioning result acquired by the communication terminal 10 from the positioning server 30. As an example, the user's viewing direction is a direction toward the center of the captured image P1. The user's intent information is information indicating the user's intent. The user's intent information is identified from manual input by the user, the search history of the communication terminal 10's browser, an app currently being used by the user, and the like. The user's intent information is, for example, information indicating that the user is currently driving, location information of a destination set by the user in a map app or the like, and information on the user's preferred foods, and the like.

The communication terminal 10 receives from the spatial structure server 50 "objects that the user cannot see" estimated by the spatial structure server 50 based on the information transmitted to the spatial structure server 50, and outputs the objects to a screen or the like on the communication terminal 10. Details will be described later, but in the examples shown in Figures 3, 7, and 8, the communication terminal 10 of user A changes the position information so that objects X1 and X2 that user A cannot see can be seen by user A. The communication terminal 10 displays object Y1, which is displayed based on the position information in which the shape of object X1 has been changed. The communication terminal 10 similarly performs display processing on object X2, and displays object Y2.

The spatial structure server 50 has, as its functional components, a memory unit 51, an acquisition unit 52, an identification unit 53, a first estimation unit 54, a second estimation unit 55, an output unit 56, and an update unit 57.

The storage unit 51 stores spatial structure data. The storage unit 51 also stores the type of object included in the spatial structure data. The spatial structure data is data that represents an object in a real space in a three-dimensional virtual space, and is data that represents the shape of the object at a position in the virtual space corresponding to the position of the object in the real space. The storage unit 51 stores data 500 in which global position information and spatial structure data are associated. Here, the spatial structure data will be described in detail. The spatial structure data is data that represents an object in a real space in a three-dimensional virtual space, and is data that represents the shape of the object at a position in the virtual space corresponding to the position of the object in the real space. For example, assume that multiple buildings (multiple objects) exist in a certain outdoor location in the real space. In that case, the structural data of the virtual space corresponding to the outdoor location (real space) represents an object on the ground and multiple building objects arranged at the same position as the outdoor location. Also, assume that a chair exists in a certain indoor location in the real space. In this case, the structural data of the virtual space corresponding to the indoor location (real space) represents wall objects, floor objects, ceiling objects, and chair objects that are located at the same positions as the indoor location. In other words, the objects in the spatial structural data are linked to objects in the real space. The spatial structural data may be data representing the shapes of only static objects (objects that do not basically move). The objects in the spatial structural data may also include objects (virtual objects) that are not linked to objects in the real space. The storage unit 51 also stores the captured images acquired by the acquisition unit 52, the global position information acquired from the communication terminal 10, the user's viewing direction, the user's purpose information, and the user's movement information (described later).

The acquisition unit 52 acquires the position of the user. Specifically, the acquisition unit 52 acquires global position information from the communication terminal 10. The acquisition unit 52 also acquires the viewing state of the user. Specifically, the acquisition unit 52 acquires the viewing state of the user based on information captured by the communication terminal 10 carried by the user or the communication terminal 10 arranged around the user. The viewing state of the user is information including at least a state in which the user is actually viewing, at least a part of the real space, the user's movement information, the user's viewing direction, and the user's purpose information. The acquisition unit 52 estimates objects actually included in the user's field of view in the real space from the captured image, and acquires a state in which the user is actually viewing in the real space. The acquisition unit 52 acquires at least a part of the object in the real space from the captured image. The acquisition unit 52 generates (acquires) the user's movement information from the global position information acquired over time. Then, the acquisition unit 52 acquires the user's viewing direction and the user's purpose information from the communication terminal 10. Note that at least a portion of an object in real space means, for example, the shape of an object viewed from a certain direction.

In the example shown in FIG. 5 and FIG. 6, the acquisition unit 52 acquires from the communication terminal 10 a captured image P1 captured by the communication terminal 10 of user A and global position information of user A. Based on the acquired captured image P1, the acquisition unit 52 estimates a state in which user A can visually recognize object X3 (user A's visual recognition state). Based on the acquired captured image P1, the acquisition unit 52 estimates a shape of object X3 as viewed from the imaging direction (at least a part of the object in real space). The acquisition unit 52 acquires user A's movement history (movement information) from the change over time in the global position information acquired continuously. In addition, the acquisition unit 52 acquires user A's viewing direction and user A's destination information from the communication terminal 10.

The identification unit 53 identifies spatial structure data corresponding to the user's position from the storage unit 51. Specifically, the identification unit 53 identifies spatial structure data corresponding to the global position information of the communication terminal 10 based on the data 500 stored in the storage unit 51 and the global position information acquired by the acquisition unit 52. For example, the identification unit 53 identifies a cylindrical three-dimensional virtual space with the position of user A as the center of the bottom circle as the spatial structure data from the global position information received from the communication terminal 10 of user A. The identification unit 53 may also identify the spatial structure data based on the user's viewing direction included in the user's viewing state. Specifically, the identification unit 53 may identify a three-dimensional virtual space that falls within the range of the viewing angle of user A as the spatial structure data from the three-dimensional virtual space identified based on the position of user A, based on the viewing direction of user A. In the example shown in FIG. 6, the identification unit 53 identifies the spatial structure data D from the global position information of user A and the viewing direction of user A.

The first estimation unit 54 estimates an ideal viewing state in which the user can visually recognize one or more objects that may be included in the user's field of view based on the user's position in the spatial structure data identified by the identification unit 53. Specifically, the first estimation unit 54 estimates one or more objects included in the spatial structure data identified by the identification unit 53 based on the position of user A, and sets the state in which the objects can be visually recognized as the ideal viewing state. Alternatively, the first estimation unit 54 estimates one or more objects included in the spatial structure data identified by the identification unit 53 based on the position of user A and the viewing direction of user A, and sets the state in which the objects can be visually recognized as the ideal viewing state. In the example shown in FIG. 6, the first estimation unit 54 estimates that the state in which objects X1, X2, and X3 included in the identified spatial structure data D can be visually recognized is the ideal viewing state.

The first estimation unit 54 may also estimate the user's future position from the user's movement information included in the user's viewing state, and estimate an ideal viewing state based on the user's future position in the identified spatial structure data. Specifically, the first estimation unit 54 estimates one or more positions to which the user may move from a change over time in the user's global position information. Then, the first estimation unit 54 estimates the ideal viewing state for each estimated position. For example, the first estimation unit 54 estimates the user's destination as the user's future position from the user's global position information every minute. Then, the first estimation unit 54 estimates the ideal viewing state at each future position.

The first estimation unit 54 may identify one or more objects from all objects that may be included in the user's field of view, taking into consideration the degree of association between the user's purpose information included in the user's viewing state and the type of object, and estimate an ideal viewing state in which the user can view the identified object. Specifically, the first estimation unit 54 identifies an object of a type that is highly related to the user's purpose information acquired by the acquisition unit 52 in one or more objects included in the identified spatial structure data, and sets the state in which the user can view the identified object as the ideal viewing state. For example, when the user's purpose information includes information that the user is driving, the first estimation unit 54 identifies an object having a type of "traffic sign" in the estimated ideal state as an object of a type that is highly related to the information that the user is driving. Also, for example, when the user's purpose information includes information that the user likes ramen, the first estimation unit 54 identifies an object having a type of "ramen", such as a signboard of a ramen shop or an advertisement for instant ramen, as an object having a type that is highly related to the information that the user likes ramen. The first estimation unit 54 then determines the ideal viewing state as a state in which the user can view the identified object.

The second estimation unit 55 compares the ideal viewing state estimated by the first estimation unit 54 with the user's viewing state, and estimates an object that the user cannot view from the difference between the ideal viewing state and the user's viewing state. Specifically, the second estimation unit 55 identifies an object that is not included in the user's viewing state as a difference object among the objects included in the ideal viewing state. The second estimation unit 55 regards the identified difference object as an object that the user cannot view. In the example shown in FIG. 6, the second estimation unit 55 compares a state in which the objects X1, X2, and X3 included in the identified spatial structure data D can be viewed (ideal viewing state) with a state in which the user A views the object X3 shown in the captured image P1 (user A's viewing state). The second estimation unit 55 identifies the objects X1 and X2, which are included in the ideal viewing state but not included in the user's viewing state, as difference objects, and estimates that the objects X1 and X2 identified as difference objects are objects that the user cannot view.

The output unit 56 outputs the object that is estimated by the second estimation unit 55 as being invisible to the user so that the user can see it. Specifically, the output unit 56 changes the distance between the user's position and the position of the object that is estimated to be invisible to the user so that the user can see the object that the user cannot see, and outputs the object that is estimated to be invisible to the user. As an example, the output unit 56 connects the user's position and the position of the object with a straight line, and changes the distance, etc. so that the object is closer to the user on the straight line. The output unit 56 transmits the object whose display position has been changed to the communication terminal 10 carried by the user. In the example shown in FIG. 7 and FIG. 8, the output unit 56 changes the distance between the objects X1 and X2 so that the objects X1 and X2 that the user A cannot see are displayed in front of the obstacle D1 as seen from the user A. The output unit 56 transmits the changed distance and the objects X1 and X2 to the communication terminal 10 of the user A. User A's communication terminal 10 receives the changed separation distance and objects X1 and X2 from the spatial structure server 50. The communication terminal 10 displays object Y1, which shows the shape of object X1 at the changed separation distance. The communication terminal 10 similarly performs display processing on object X2, and displays object Y2.

The update unit 57 updates the information of the object in the spatial structure data based on at least a part of the object in the real space included in the visual state of the user acquired by the acquisition unit 52. Specifically, the update unit 57 updates the information of the object in the spatial structure data stored in the storage unit 51 to the latest state based on at least a part of the object in the real space estimated from the captured image by the acquisition unit 52. In the example shown in FIG. 1, the user's communication terminal 10 and communication terminals 10 arranged in the surroundings capture images of objects X1 and X2 and transmit the captured images, global position information, and imaging direction (user's visual direction) to the spatial structure server 50 in real time. The acquisition unit 52 estimates the position and/or shape of the objects X1 and X2 viewed from a certain direction from the captured image and imaging direction. The update unit 57 updates the position and/or shape of the objects X1 and X2 in the spatial structure data stored in the storage unit 51 in real time based on the information of the objects X1 and X2 viewed from a certain direction estimated (acquired) by the acquisition unit 52.

Next, the process performed by the information processing system 1 according to this embodiment will be described with reference to FIG. 9, specifically, the process of acquiring information such as the position of the communication terminal 10 carried by the user and the user's viewing state, estimating an ideal viewing state from the position of the communication terminal 10, etc., comparing the ideal viewing state with the user's viewing state, estimating information that is in the user's line of sight but that the user is unable to view from the difference between the ideal viewing state and the user's viewing state, and presenting the information to the user. Also, the process of updating an object in the three-dimensional virtual space based on the acquired information will be described with reference to FIG. 10. FIGS. 9 and 10 are flowcharts showing the process performed by the information processing system 1.

As shown in FIG. 9, in the information processing system 1, the acquisition unit 52 acquires the user's position and the user's viewing state (step S101). Specifically, the acquisition unit 52 may acquire global position information and at the same time acquire the user's viewing state including at least one of the state that is estimated to be actually viewed by the user in real space, at least a part of an object in real space, the user's movement information, the user's viewing direction, and the user's purpose information. Next, the identification unit 53 identifies spatial structure data corresponding to the user's position (step S102). The identification unit 53 may also identify the spatial structure data based on the user's viewing direction included in the user's viewing state.

Next, the first estimation unit 54 estimates an ideal viewing state based on the user's position in the spatial structure data identified by the identification unit 53 (step S103). The first estimation unit 54 may also estimate the ideal viewing state based on the user's viewing direction. In addition, the first estimation unit 54 may estimate the user's future position from the user's movement information. The first estimation unit 54 may estimate the ideal viewing state based on the user's future position in the identified spatial structure data. Furthermore, the first estimation unit 54 may take into account the degree of association between the user's purpose information and the type of object, and identify one or more objects from all objects that may be included in the user's field of view. Then, the first estimation unit 54 estimates an ideal viewing state in which the user can view all of the identified objects.

Then, the second estimation unit 55 compares the ideal viewing state estimated by the first estimation unit 54 with the user's viewing state. Then, the second estimation unit 55 estimates an object that the user is unable to view from the difference between the ideal viewing state and the user's viewing state (step S104). Next, the output unit 56 outputs the object that is estimated to be unable to be viewed by the user so that the user can view the object (step S105). Specifically, the output unit 56 changes the distance between the user's position and the position of the object that the user is unable to view. Then, the output unit 56 outputs the object.

10, in the information processing system 1, the acquisition unit 52 acquires the user's position and the user's viewing state (step S201), similar to step S101. Specifically, the acquisition unit 52 estimates what objects are included in the image captured by the communication terminal 10, for the terminal carried by the user or a communication terminal 10 such as a surveillance camera placed around the user, and acquires at least a portion of the object in real space and the imaging direction (user's viewing direction).

Next, the update unit 57 updates the objects in the spatial structure data based on at least a portion of the objects in real space included in the user's visual state acquired by the acquisition unit 52 (step S202).

Next, we will explain the effects of the information processing system 1 according to this embodiment.

The information processing system 1 according to this embodiment includes a storage unit 51 that stores spatial structure data that represents an object in a real space in a three-dimensional virtual space and that represents the shape of the object at a position in the virtual space that corresponds to the position of the object in the real space; an acquisition unit 52 that acquires the user's position and the user's viewing state; a determination unit 53 that identifies the spatial structure data that corresponds to the user's position from the storage unit 51; a first estimation unit 54 that estimates an ideal viewing state in which the user can view one or more objects that may be included in the user's field of view based on the user's position in the spatial structure data identified by the determination unit 53; a second estimation unit 55 that compares the ideal viewing state estimated by the first estimation unit 54 with the user's viewing state and estimates an object that the user cannot view from the difference between the ideal viewing state and the user's viewing state; and an output unit 56 that outputs the object that the second estimation unit 55 estimates to be unable to view the user so that the user can view it.

In the information processing system 1 according to the present embodiment, the user's position and the user's visual state are acquired, spatial structure data corresponding to the user's position is identified, and in the identified spatial structure data, an ideal visual state is estimated based on the user's position, the ideal visual state is compared with the user's visual state, and an object that the user is unable to visually recognize is estimated from the difference between the ideal visual state and the user's visual state, and the object that is estimated to be unable to visually recognize is output so that the user can visually recognize the object. Here, in the real space, information about the user's surroundings is physically blocked, and the user is unable to visually recognize the information, which may impair the convenience of the user who uses the information and the profits of individuals or groups who provide the information. For example, traffic signs are blocked by cars, roadside trees, etc., which impairs the safety of users while driving or walking. In addition, when a company's advertisement on a train is blocked by passengers, hanging straps, etc., the user loses the opportunity to view the advertisement, which impairs the company's profits. In this regard, in an information processing system according to one aspect of the present invention, objects that the user cannot see in real space are displayed on the user's terminal, such as an AR display device. This makes it possible to properly present information that the user cannot see to the user. As a result, it is possible to ensure the convenience of the user who uses the information, and the profits of the individual or organization that provides the information.

In the example shown in FIG. 11, when a user is driving, a traffic light is blocked by a truck traveling in front of the vehicle driven by the user. Here, for example, if the user cannot see the traffic light when the traffic light is about to change from yellow to red, there is a risk that the vehicle driven by the user will drive through the intersection even though the light is red. In other words, the safety of the user while driving is compromised because the traffic light cannot be seen. In this regard, in an information processing system according to one aspect of the present invention, the traffic light is displayed as an object Z on the user's communication terminal 10, such as an AR display device. This makes it possible to appropriately present the traffic light, which is information that the user cannot see, to the user.

The output unit 56 changes the distance between the user's position and the position of the object that is estimated to be invisible to the user so that the user can see the object that the user cannot see, and outputs the object that is estimated to be invisible to the user. In this configuration, the object that the user cannot see is moved to a position that is visible to the user. This makes it possible to appropriately present information that the user cannot see to the user.

The acquisition unit 52 may acquire the user's viewing state based on information captured by a communication terminal 10 carried by the user or a communication terminal 10 placed around the user. In such a configuration, the user's viewing state is acquired from various terminals. This makes it possible to more appropriately present information that the user is unable to view to the user. For example, when there is an object that is not visible to the user, the user's viewing state is acquired by the communication terminal 10 carried by the user or a surveillance camera (communication terminal 10) placed on the street. This makes it possible to appropriately present to the user objects that the user is unable to view.

The information processing system 1 according to this embodiment may further include an update unit 57 that updates the information of an object in the spatial structure data based on the object in real space included in the user's visual state acquired by the acquisition unit 52. With this configuration, for example, the object in the spatial structure data is updated to the latest state by a terminal carried by the user or a surveillance camera placed around the user, and the object information transmitted from the spatial structure server 50 to the communication terminal 10 is updated to the latest state. This makes it possible to more appropriately present information that the user is unable to visually recognize to the user.

The acquisition unit 52 acquires the user's viewing state including the user's viewing direction, and the identification unit 53 identifies the spatial structure data based on the user's viewing direction and the user's position included in the user's viewing state. With this configuration, the spatial structure data is identified and the ideal viewing state is estimated only in the direction in which the user views, thereby reducing the amount of processing by the information processing system 1 in the above identification and estimation. This makes it possible to reduce the load on the information processing system 1 without compromising the convenience of the user.

The acquisition unit 52 acquires the user's visual state including the user's movement information, and the first estimation unit 54 estimates the user's future position from the user's movement information and estimates an ideal visual state based on the user's future position in the identified spatial structure data. With this configuration, an ideal visual state is estimated at a position to which the user may move in the future. This makes it possible to present information that the user is unable to visually recognize to the user while minimizing delays. Specifically, for example, when the user's movement information includes the user's movement history and purpose of movement, the user's destination is predicted and information that the user is unable to visually recognize in advance is estimated. This makes it possible to present information that the user is unable to visually recognize to the user while minimizing delays, even when the user is moving quickly in a passenger car or the like.

The storage unit 51 further stores the type of object, the acquisition unit 52 acquires the user's viewing state including the user's purpose information, and the first estimation unit 54 identifies one or more objects from all objects that can be included in the user's field of view, taking into account the degree of association between the user's purpose information included in the user's viewing state and the type, and estimates an ideal viewing state in which the user can view the identified object. According to this configuration, an object related to the user's purpose information is estimated as an object that the user cannot view, and the object is presented to the user. This can improve the user's convenience and the user's quality of experience. Specifically, when the user is driving, only traffic signs, which are information related to driving, are estimated as objects that the user cannot view, and the object is presented to the user. This can prevent information unnecessary for the user while driving from being presented, while reliably presenting only necessary information, thereby improving the user's convenience while driving and ensuring safety. In addition, according to the above configuration, objects that do not meet the user's purpose among the objects included in the ideal viewing state are removed from the ideal viewing state, and the number of objects included in the ideal viewing state is reduced. This improves user convenience while reducing the load on the information processing system 1.

Next, the hardware configuration of the communication terminal 10, the positioning server 30, and the spatial structure server 50 included in the information processing system 1 will be described with reference to FIG. 12. The communication terminal 10, the positioning server 30, and the spatial structure server 50 described above may be physically configured as a computer device including a processor 1001, a memory 1002, a storage 1003, a communication device 1004, an input device 1005, an output device 1006, a bus 1007, etc.

In the following description, the term "apparatus" may be interpreted as a circuit, device, unit, etc. The hardware configurations of the communication terminal 10, the positioning server 30, and the spatial structure server 50 may be configured to include one or more of the devices shown in FIG. 12, or may be configured to exclude some of the devices.

The functions of the communication terminal 10, the positioning server 30, and the spatial structure server 50 are realized by loading specific software (programs) onto hardware such as the processor 1001 and memory 1002, causing the processor 1001 to perform calculations and control communication by the communication device 1004 and the reading and/or writing of data in the memory 1002 and storage 1003.

The processor 1001, for example, operates an operating system to control the entire computer. The processor 1001 may be configured as a central processing unit (CPU) including an interface with peripheral devices, a control device, an arithmetic unit, a register, etc. For example, the control functions of the positioning unit 32 of the positioning server 30, etc. may be realized by the processor 1001.

The processor 1001 also reads out programs (program codes), software modules, and data from the storage 1003 and/or the communication device 1004 into the memory 1002, and executes various processes according to these. The programs used are those that cause a computer to execute at least some of the operations described in the above embodiments.

For example, the control functions of the positioning unit 32 of the positioning server 30 may be realized by a control program stored in the memory 1002 and running on the processor 1001, and the other functional blocks may be realized in a similar manner. Although the above-mentioned various processes have been described as being executed by one processor 1001, they may be executed simultaneously or sequentially by two or more processors 1001. The processor 1001 may be implemented in one or more chips. The program may be transmitted from a network via a telecommunications line.

The memory 1002 is a computer-readable recording medium, and may be composed of at least one of, for example, a ROM (Read Only Memory), an EPROM (Erasable Programmable ROM), an EEPROM (Electrically Erasable Programmable ROM), a RAM (Random Access Memory), etc. The memory 1002 may also be called a register, a cache, a main memory (primary storage device), etc. The memory 1002 can store executable programs (program codes), software modules, etc. for implementing a wireless communication method according to one embodiment of the present invention.

Storage 1003 is a computer-readable recording medium, and may be, for example, at least one of an optical disk such as a CD-ROM (Compact Disc ROM), a hard disk drive, a flexible disk, a magneto-optical disk (e.g., a compact disk, a digital versatile disk, a Blu-ray (registered trademark) disk), a smart card, a flash memory (e.g., a card, a stick, a key drive), a floppy (registered trademark) disk, a magnetic strip, and the like. Storage 1003 may also be referred to as an auxiliary storage device. The above-mentioned storage medium may be, for example, a database, a server, or other suitable medium including memory 1002 and/or storage 1003.

The communication device 1004 is hardware (transmitting/receiving device) for communicating between computers via a wired and/or wireless network, and is also referred to as, for example, a network device, a network controller, a network card, a communication module, etc.

The input device 1005 is an input device (e.g., a keyboard, a mouse, a microphone, a switch, a button, a sensor, etc.) that accepts input from the outside. The output device 1006 is an output device (e.g., a display, a speaker, an LED lamp, etc.) that performs output to the outside. Note that the input device 1005 and the output device 1006 may be integrated into one configuration (e.g., a touch panel).

In addition, each device, such as the processor 1001 and memory 1002, is connected by a bus 1007 for communicating information. The bus 1007 may be configured as a single bus, or may be configured as different buses between the devices.

The communication terminal 10, the positioning server 30, and the spatial structure server 50 may be configured to include hardware such as a microprocessor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a programmable logic device (PLD), or a field programmable gate array (FPGA), and some or all of the functional blocks may be realized by the hardware. For example, the processor 1001 may be implemented by at least one of these pieces of hardware.

Although the present embodiment has been described in detail above, it is clear to those skilled in the art that the present embodiment is not limited to the embodiment described in this specification. The present embodiment can be implemented in modified and altered forms without departing from the spirit and scope of the present invention as defined by the claims. Therefore, the description in this specification is intended as an illustrative example and does not have any restrictive meaning with respect to the present embodiment.

For example, the information processing system 1 has been described as including a communication terminal 10, a positioning server 30, and a spatial structure server 50, but is not limited to this, and each function of the information processing system 1 may be realized by the spatial structure server 50.

In addition, in this embodiment, the spatial structure server 50 is exemplified as having a storage unit 51, an acquisition unit 52, an identification unit 53, a first estimation unit 54, a second estimation unit 55, an output unit 56, and an update unit 57. However, in the information processing system 1, other servers or communication terminals 10 may have some or all of these functional components.

Each aspect/embodiment described herein may be applied to systems utilizing LTE (Long Term Evolution), LTE-Advanced (LTE-A), SUPER 3G, IMT-Advanced, 4G, 5G, FRA (Future Radio Access), W-CDMA (registered trademark), GSM (registered trademark), CDMA2000, UMB (Ultra Mobile Broadband), IEEE 802.11 (Wi-Fi), IEEE 802.16 (WiMAX), IEEE 802.20, UWB (Ultra-Wide Band), Bluetooth (registered trademark), or other suitable systems and/or next generation systems based on and enhanced thereon.

The steps, sequences, flow charts, etc. of each aspect/embodiment described herein may be reordered unless inconsistent. For example, the methods described herein present elements of various steps in an example order and are not limited to the particular order presented.

The input and output information may be stored in a specific location (e.g., memory) or may be managed in a management table. The input and output information may be overwritten, updated, or added to. The output information may be deleted. The input information may be sent to another device.

The determination may be based on a value represented by one bit (0 or 1), a Boolean (true or false) value, or a numerical comparison (e.g., with a predetermined value).

Each aspect/embodiment described in this specification may be used alone, in combination, or switched depending on the execution. In addition, notification of specific information (e.g., notification that "X is the case") is not limited to being done explicitly, but may be done implicitly (e.g., not notifying the specific information).

Software shall be construed broadly to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, threads of execution, procedures, functions, etc., whether referred to as software, firmware, middleware, microcode, hardware description language, or otherwise.

Software, instructions, etc. may also be transmitted and received over a transmission medium. For example, if the software is transmitted from a website, server, or other remote source using wired technologies, such as coaxial cable, fiber optic cable, twisted pair, and digital subscriber line (DSL), and/or wireless technologies, such as infrared, radio, and microwave, these wired and/or wireless technologies are included within the definition of a transmission medium.

The information, signals, etc. described herein may be represented using any one of a variety of different technologies. For example, data, instructions, commands, information, signals, bits, symbols, chips, etc. that may be referred to throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or magnetic particles, optical fields or photons, or any combination thereof.

Note that terms explained in this specification and/or terms necessary for understanding this specification may be replaced with terms having the same or similar meanings.

In addition, the information, parameters, etc. described in this specification may be expressed as absolute values, as relative values from a predetermined value, or as corresponding other information.

A communications terminal may also be referred to by those skilled in the art as a mobile communications terminal, subscriber station, mobile unit, subscriber unit, wireless unit, remote unit, mobile device, wireless device, wireless communications device, remote device, mobile subscriber station, access terminal, mobile terminal, wireless terminal, remote terminal, handset, user agent, mobile client, client, or some other suitable terminology.

As used herein, the phrase "based on" does not mean "based only on," unless expressly stated otherwise. In other words, the phrase "based on" means both "based only on" and "based at least on."

When designations such as "first," "second," and the like are used herein, any reference to that element is not intended to generally limit the quantity or order of those elements. These designations may be used herein as a convenient way to distinguish between two or more elements. Thus, a reference to a first and a second element does not imply that only two elements may be employed therein or that the first element must precede the second element in some way.

To the extent that the terms "include," "including," and variations thereof are used herein in the specification or claims, these terms are intended to be inclusive, similar to the term "comprising." Further, the term "or" as used herein is not intended to be an exclusive or.

In this specification, multiple devices are included unless the context or technical basis clearly indicates that only one device exists.

Throughout this disclosure, the plural is intended to be included unless the context clearly indicates the singular.

1...information processing system, 51...storage unit, 52...acquisition unit, 53...identification unit, 54...first estimation unit, 55...second estimation unit, 56...output unit, 57...update unit.

Claims (7)

a storage unit that stores spatial structure data representing a real-space object in a three-dimensional virtual space, the spatial structure data representing a shape of the object at a position in the virtual space corresponding to a position of the object in the real space;
An acquisition unit that acquires a user's position and a visual recognition state of the user;
an identification unit that identifies the spatial structure data corresponding to the position of the user from the storage unit;
A first estimation unit that estimates an ideal visual recognition state in which the user can visually recognize one or more objects that may be included in the user's visual field based on the position of the user in the spatial structure data identified by the identification unit;
a second estimation unit that compares the ideal viewing state estimated by the first estimation unit with a viewing state of the user, and estimates the object that the user is unable to view from a difference between the ideal viewing state and the viewing state of the user;
an output unit that outputs the object that is estimated by the second estimation unit to be not visually recognized by the user so that the object can be visually recognized by the user;
An information processing system comprising: The information processing system according to claim 1, wherein the output unit changes the distance between the position of the user and the position of an object that is estimated to be invisible to the user so that the user can see the object that is invisible to the user, and outputs the object that is invisible to the user. The information processing system according to claim 1 or 2, wherein the acquisition unit acquires the visibility state of the user based on information captured by a terminal carried by the user or a terminal placed around the user. the acquisition unit acquires a visual recognition state of the user including at least a portion of an object in the real space;
an update unit that updates information on the object in the space structure data based on at least a part of the object in the real space included in the visual recognition state of the user acquired by the acquisition unit;
The information processing system according to any one of claims 1 to 3. The acquisition unit acquires a viewing state of the user including a viewing direction of the user,
The identification unit identifies the spatial structure data based on a viewing direction of the user and a position of the user included in the viewing state of the user.
The information processing system according to any one of claims 1 to 4. The acquisition unit acquires a visual recognition state of the user including movement information of the user,
The first estimation unit estimates a future position of the user from movement information of the user included in the visual state of the user, and estimates the ideal visual state based on the future position of the user in the specified spatial structure data.
The information processing system according to any one of claims 1 to 5. The storage unit further stores a type of the object,
The acquisition unit acquires a visual recognition state of the user including user purpose information which is information indicating a purpose of the user's action,
The first estimation unit identifies one or more objects from all objects that may be included in the user's visual field, taking into consideration a degree of association between the user's purpose information included in the user's visual state and the type, and estimates the ideal visual state in which the user can visually recognize the identified objects.
The information processing system according to any one of claims 1 to 6.

Images