JP2014164374A - Information display system, information terminal, server device, control method of information terminal, control method of server device, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To specify an observed person transparently viewed by an observer via display means and display information which is associated with the observed person and is to be displayed.SOLUTION: In a system including an information terminal and a server device comprising database which controls feature point information present in a physical space in which each user moves, a relative distance is calculated to a person image detected in image data, using the image data imaged by a user whose position is not estimated in the physical space and a physical amount which changes according to user's behavior, and a user viewed by the user whose position is not estimated in the physical space is specified by the calculated relative distance and an estimated position of each user to be present in the physical space. Further, information which is to be displayed in association with the specified user is output to an information terminal associated with the user whose position is not estimated in the physical space.

Description

  The present invention relates to an information display system, an information terminal, a server device, a control method for an information terminal, a control method for a server device, and a program that process captured image information and display corresponding information. It is.

  There are information display systems aimed at estimating a person's own position in a physical space. The information display system disclosed in Patent Document 1 estimates the approximate self-position of the person X by the movement amount detection means, and captures the position of the object in advance from the map information captured by the person X's camera. Compute markers for objects that will be included in the image.

  And the self-position of the person X is estimated by comparing with the marker extracted from actual imaging. By applying this self-position estimation to each person, the self-positions of all persons existing in the physical space can be estimated.

  By combining Patent Document 1 and a technology for grasping the orientation of the camera of the person X, such as a geomagnetic sensor, the system can be appropriately selected depending on who is included in the imaging of the camera of the person X. It becomes possible to generate content.

JP-A-2-259912

However, in Patent Document 1, when the system fails to compare the marker derived from the map information with the marker derived from the actual imaging regarding the user A, the system cannot accurately estimate the position of the user A. As a result, the system does not know who is included in the image captured by the user A's camera, and cannot generate content with appropriate contents. Here, in this system, it is conceivable to use estimation of a person by face authentication, but it cannot be applied when the other party is standing with his back facing himself / herself.
The present invention has been made in order to solve the above-described problem, and the object of the present invention is when the accurate position of a specific person in the physical space cannot be estimated on the physical space shared by the information display system. However, it is to provide a mechanism that allows the observer to identify the observer who is viewing through the display means and to display the information to be displayed associated with the observer.

The information display system of the present invention that achieves the above object has the following configuration.
An information display system in which a server device and a plurality of information terminals moving in a physical space communicate with each other, wherein each information terminal is capable of imaging an object that enters a user's field of view, and is capable of transmitting the object Display means for displaying information, detection means for detecting a physical quantity that changes in association with the movement of the user moving in the physical space, and imaging data obtained by photographing the physical space in which each user exists, or Transmitting means for transmitting the detected changing physical quantity in association with each user information to the server device; and information to be displayed associated with other users who are viewing through the display means. Control means for receiving from the apparatus and displaying on the display means, wherein the server apparatus manages a database for managing feature point information existing in a physical space in which each user moves. An acquisition means for acquiring the changing physical quantity detected by the detection means of each information terminal and the imaging data captured by the imaging means; a feature point of the physical space detected from the imaging data acquired from each information terminal; Image estimation data obtained from position estimation means for estimating the position where each user should exist in physical space from feature points managed in the database, and information terminals associated with users whose position is not estimated in physical space The person image detected from the imaging data using detection means for detecting a person image from the image, imaging data captured by a user whose position is not estimated in physical space, and a physical quantity that changes according to the user's operation From the calculation means for calculating the relative distance up to, the calculated relative distance, and the position of each user that should be present in the estimated physical space. The identification means for identifying the user viewed by the user whose position is not estimated in the above and the information to be displayed in association with the identified user are associated with the user whose position is not estimated in the physical space Output means for outputting to the information terminal.

  According to the present invention, even when the exact position of a specific person in the physical space cannot be estimated on the physical space shared by the information display system, the observer sees through the display means. It is possible to specify an observer and display information to be displayed associated with the observer.

It is a figure which shows the external appearance of HMD used with an information display system. It is a block diagram which shows the structure of the hardware of HMD. It is a block diagram which shows the structure of the hardware of a processing server. It is a figure which shows the structure of an information display system. It is a figure which shows an example of the screen displayed on a display. It is a figure explaining how the display position of person specific information is calculated. It is a flowchart explaining the control method of an information display system. It is a figure which shows the concept of a person's movement in a physical space, and a person's presence position candidate. It is a figure which shows an example of the database which a processing server manages. It is a flowchart explaining the control method of an information display system. It is a figure which shows the person imaged with the camera with which HMD is incorporated. It is a figure which shows the relationship between the distance and angle of an observer and a to-be-observed person. It is a flowchart explaining the control method of an information display system. It is a figure which shows the concept of a specific person's presence position candidate. It is a figure which shows the example of the map information of physical space. FIG. 16 is a diagram in which a candidate position of a specific person is superimposed on the map shown in FIG. 15. It is the figure which showed a mode that a to-be-observed person moved. It is a flowchart explaining the control method of an information display system.

Next, the best mode for carrying out the present invention will be described with reference to the drawings.
<Description of system configuration>
[First Embodiment]
<Head-mounted display (HMD)>
FIG. 1 is a diagram illustrating an appearance of a head mounted display (HMD) 100 used in the information display system according to the present embodiment. In the present embodiment, an HMD is used as an example of an information terminal that is positioned in accordance with a user's visual recognition position, and the server device is configured to be able to communicate with a plurality of information terminals.

  In FIG. 1, the HMD 100 is similar to a commercially available one and is attached to the head with a frame 201. In addition, the structure is such that a transmissive display 101 is attached to one eye or both eyes, and the user can simultaneously see an artificial image (display image) displayed on the display 101 and a real image at the tip of the display 101. . A camera 102 is also attached to the HMD 100, and it is possible to take a picture from a viewpoint close to the eyes of the user.

  FIG. 2 is a block diagram showing a hardware configuration of the HMD 100 shown in FIG. The HMD 100 includes a controller 120, a transmissive display 101, and a camera 102. Here, the camera 102 images a target that enters the user's field of view. In addition, the display 101 is configured to be able to display information to be displayed output from the server device in a state where the object entering the field of view can be transmitted. Hereinafter, the configuration of the controller 120 of the HMD 100 will be further described.

  In FIG. 2, the internal bus 112 transmits and receives electrical signals so that information can be transmitted between the memory controller 107 and the I / O controller 109. The memory controller 107 controls all memory accesses to the main memory 108. The I / O controller 109 transmits and receives information to and from the processing unit connected via the ROM 110, the LAN port 103, the transmissive display 101, the MEMS sensor 111, and the internal bus 112. The LAN port 103 transmits / receives information to / from the I / O controller 109 with other devices such as the processing server 300 connected via the access point 400 (FIG. 4) and / or the network 9000.

The ROM 110 stores a boot loader program and an HMD control program. Since the main memory 108 is a volatile memory and can be accessed at high speed, information stored in the ROM 110 and information used temporarily are stored here. When the power of the HMD 100 is turned on, the CPU 104 reads and executes the boot loader program, extracts the HMD control program stored in the ROM 110, and stores it in the main memory 108. Then, the CPU 104 executes the HMD control program stored in the main memory 108 and executes each function of the HMD 100. Further, when the CPU 104 executes this HMD control program, processing performed by the HMD 100 such as data communication with the processing server 300 and information display on the transmissive display 101 is executed.
The above is the description of the configuration of the controller 120.

The HMD 100 includes a transmissive display 101 that displays information. The graphic controller 106 controls display on the transmissive display 101. The HMD 100 displays information on the transmissive display 101 when presenting information to the user. The HMD 100 includes a camera 102. The camera 102 digitally converts the captured image taken. The camera 102 transmits / receives information to / from the I / O controller 109. The HMD 100 includes a MEMS sensor 111.
The MEMS sensor 111 detects an amount of change that changes in association with the action of the user moving in the physical space. Specifically, the MEMS sensor 111 can measure the direction of east, west, south, and north, and the inclination and acceleration in a three-dimensional direction, and converts the measured information into digital information. The MEMS sensor 111 transmits and receives information to and from the I / O controller 109. The I / O controller 109 changes the image data obtained by photographing the physical space where the user wearing the HMD 100 is present to the processing server 300 on the network via the LAN port 103 or the change detected by the MEMS sensor 111. Each physical quantity is transmitted in association with user information. The transmission process by the I / O controller 109 is executed at predetermined intervals.
With the MEMS sensor 111, it is possible to grasp in which direction the user wearing the HMD 100 is facing in the physical space. The battery 105 supplies power to the entire HMD 100.
<Processing server>
FIG. 3 is a block diagram illustrating a hardware configuration of a processing server used in the information display system according to the present embodiment. Note that the processing server includes a database that manages user information and feature information existing in the physical space, and compares the analysis result of the imaging data captured by the HMD 100 with the feature information, thereby allowing the user's physical space to be compared. The process of specifying the position of is performed. Here, the characteristic information includes information indicating the characteristics of the structure of the room where the user moves and information indicating the characteristics such as the arrangement information of the accessories.

In FIG. 3, reference numeral 300 denotes a processing server. The processing server 300 includes a controller 320, a display 301, a keyboard 302, and a mouse 303. Hereinafter, the configuration of the controller 320 will be further described.
The internal bus 311 transmits and receives electrical signals so that information can be transmitted between the memory controller 307 and the I / O controller 309.

  The memory controller 307 controls overall memory access to the main memory 308. The I / O controller 309 transmits / receives information to / from processing units connected via the HDD 310, the LAN port 304, the keyboard 302, the mouse 303, and the internal bus 311. The LAN port 304 transmits / receives information to / from other devices connected via the network 9000 or the wireless access point 400 and the I / O controller 309.

The HDD 310 stores a boot loader program and a processing server control program. Since the main memory 308 is a volatile memory and can be accessed at high speed, information stored in the HDD 310 and information used temporarily are stored here. When the processing server 300 is powered on, the CPU 305 reads and executes the boot loader program, retrieves the processing server control program stored in the HDD 310, and stores it in the main memory 308. Then, the CPU 305 executes the processing server control program stored in the main memory 308, and executes each function of the processing server. CPU 3
When 05 executes this processing server control program, the processing performed by the processing server shown in FIGS. 7, 10, 13 and 18 is executed. The above is the description of the configuration of the controller 320.
The processing server 300 includes a display 301 that displays information. The graphic controller 306 controls display on the display 301.

The processing server 300 displays information on the display 301 when presenting information to the user, and the user can input information to the processing server 300 by operating the keyboard 302 or the mouse 303.
<System configuration>

  FIG. 4 is a diagram illustrating the configuration of the information display system according to the present embodiment. The information display system of this example includes an HMD (Head Mounted Display) 100 worn by three users A, B, and C, a wireless access point 400, a processing server 300, and a network 9000. Each HMD 100 and the wireless access point 400 are connected by radio, and the processing server 300 and the wireless access point 400 are connected by a network 9000.

  In this information display system, imaging by the camera built in the HMD worn by each user and azimuth, acceleration, and tilt information by the MEMS sensor are transmitted to the processing server 300 via the wireless access point 400 and the network 9000 one by one. The The processing server 300 estimates the presence position of each user in the physical space based on the information. There are existing technologies represented by SLAM (Simultaneous Localization and Mapping) for position estimation technology from camera imaging. In addition, regarding the position estimation technique from the MEMS sensor, there is an existing technique represented by PDR (Pedestrian Dead Reckoning), which is not a core part of the present proposal, and therefore description thereof is omitted. It is possible to specify who is the person shown in the camera built in each user's HMD from the presence position information of each user and the direction information of the MEMS sensor.

  If the orientation of the camera incorporated in the HMD and the display of the HMD is fixedly determined, it is possible to specify who is in which direction of the user's visual field. The HMD 100 acquires the location information of each user in the physical space from the processing server 300 via the network 9000 and the wireless access point 400. Based on the position information, the HMD 100 displays information specific to the person on the HMD translucent display regarding the person in the field of view of the user, so that the user can associate the person-specific information with the real image of the person. You can see the video.

FIG. 5 is a diagram showing an example of a screen displayed on the display 301 shown in FIG. This example shows an example of an image obtained when the user views another person through the HMD translucent display. For example, when Mr. A in FIG. 4 sees Mr. B through the HMD translucent display, Mr. A sees an image as shown in FIG. In addition, when Mr. A in FIG. 4 sees Mr. C through the HMD translucent display, Mr. A will see an image as shown in FIG.
<Display position of person-specific information in HMD>

  FIG. 6 is a diagram for explaining how the processing server 300 illustrated in FIG. 3 calculates the display position of the person-specific information in the HMD. Hereinafter, with reference to FIG. 6, a configuration will be described in which an HMD wearer can view a video in which information unique to a person is associated with a real image of the person in the field of view.

In FIG. 6, the controller 320 of the processing server 300 grasps the physical position information of all the persons wearing the HMD and the orientation information that is facing as described above. For example, consider a case where there are two persons, an observer wearing an HMD and an observer.
In the physical position information, when a specific place in the physical space is the origin, the position coordinates of the observer are (Xo, Yo), and the position coordinates of the observer are (Xt, Yt). In addition, the orientation that the observer is facing (hereinafter referred to as the normal 601 of the HMD 100) is θ. Here, θ is in radians, but direction information is in degrees and can be converted to each other. Based on these pieces of information, the controller 320 internally generates a map as shown in FIG. The map in FIG. 6 shows a state in which the positional relationship between the observer and the observer is viewed from above.

  Next, the controller 320 of the processing server 300 obtains the angle γ shown in FIG. The angle γ is an angle in the direction in which the non-observer is located with respect to the normal 0601 of the HMD 100 worn by the observer. γ is obtained by the following equation.

Then, the controller 320 compares the viewing angle α and the angle γ covered by the transmissive display 101 of the HMD 100 to determine whether the subject is in the field of view of the observer. The term “observed person” as used herein refers to a person who can be seen through the transmissive display 101 of the HMD 100 worn by the observer. If | γ | <α / 2, the controller 320 determines that the subject is in the field of view of the observer. If the controller 320 determines that | γ | <α / 2 is not satisfied, it is determined that the person being observed is out of the field of view of the observer.
<Person absolute position estimation processing flow>

  FIG. 7 is a flowchart for explaining a control method of the information display system according to the present embodiment. This example is a human absolute position estimation process example by the processing server 300 shown in FIG. This process is periodically executed for all persons in the physical space. Based on the imaging of the camera and the azimuth, acceleration, and tilt information obtained by the MEMS sensor transmitted from the HMD 100 worn by each user via the wireless access point 400 and the network 9000, the processing server 300 operates on the physical space of each user. Is estimated. Note that all of the processing from S1301 to S1310 is realized by the CPU 305 of the controller 320 of the processing server 300 executing the control program.

  In step S <b> 1301, the CPU 305 calculates the movement direction and movement amount of the person wearing the HMD based on the azimuth, acceleration, and tilt information obtained by the MEMS sensor. At that time, measurement errors in the direction and amount of movement of the person are inevitable. It is possible to grasp how much this measurement error is empirically. In S1302, the CPU 305 calculates the accumulated amount of error accumulated over time. In step S <b> 1303, the CPU 305 calculates the presence position of the person wearing the HMD from the movement direction and the movement amount, and calculates an area serving as a person's existence position candidate considering the accumulated amount of measurement error. Hereinafter, the region serving as the existence position candidate will be described with reference to FIG.

FIG. 8 is a diagram showing the concept of the movement of a person in the physical space and the person's existence position candidate grasped by the system. Note that the method of calculating the presence position of the person based on the MEMS sensor information is the calculation of the presence position with respect to the place where the person was in front. Therefore, measurement errors accumulate with time.
When a person moves in accordance with the locus indicated by the arrow in FIG. 8A, an error as shown in the gray circled area in FIG. 8B occurs in the position of the person calculated by the system. It will be. The fact that measurement errors build up over time means that this area grows over time. If the error accumulates to some extent, the system cannot accurately calculate the positional relationship between the persons, and thus cannot determine who is in the observer's field of view. In order to prevent this situation, the system needs to wipe out the error periodically.

Therefore, in S1304 shown in FIG. 7, the CPU 305 analyzes the camera image picked up in the HMD 100 and detects the feature point of the physical space. Although there is a technique such as SIFT (Scale-Invariant Feature Transform) for detecting feature points in the physical space, the description is omitted because it is not directly related to the description of the present embodiment.
In S1305, the CPU 305 determines whether or not the feature point of the physical space has been detected. If the CPU 305 determines that it is TRUE, the process proceeds to S1306. If the CPU 305 determines that it is FALSE, the process proceeds to S1309.

In step S1306, the CPU 305 calculates the presence position of the person based on the feature point information in the physical space. There is a self-position estimation technique such as SLAM (Simultaneous Localization and Mapping) as a method for calculating the presence position based on the feature point information, but the description is omitted because it is not directly related to the present embodiment.
As described above, by periodically applying the location information calculated based on the feature point information of the physical space, it is possible to eliminate errors accumulated at the location calculated based on the MEMS information. Accordingly, in S1307, the accumulated amount of measurement error is reset.
In step S1308, the CPU 305 sets a lost flag indicating whether the system can accurately grasp the position of the person to FALSE, and the process advances to step S1301.

In S1309, the process is performed when the CPU 305 cannot detect a feature point in the physical space. This means that the position of the person cannot be calculated based on the feature point information in the physical space. In other words, if a feature point cannot be taken due to some factor from the camera's imaging, the measurement error of the person's location calculated based on the information of the MEMS sensor cannot be reset, and the measurement error accumulates with time. Become.
Therefore, in S1309, the CPU 305 determines whether or not the accumulated amount of measurement error is greater than or equal to the threshold. If the CPU 305 determines that the measured error is greater than or equal to the threshold, the process proceeds to S1310. .

  In S1310, the accumulated amount of measurement error increases, indicating a situation where the system cannot accurately calculate the positional relationship between persons. At this time, it is impossible to determine who is in the observer's field of view. In step S1310, the CPU 305 sets the lost flag of the person to TRUE, and the process advances to step S1301. The processing server 300 manages presence position information for each person, information on MEMS sensors, lost flag information, and measurement error accumulation amount information on a database as shown in FIG.

(Method 1 for estimating person X)
The information display system according to the present embodiment estimates the person X in a situation where the positional relationship between the persons cannot be accurately calculated and the person X appearing in the observer's field of view cannot be determined. How to do it.
The processing of the present embodiment is performed for a person whose lost flag is TRUE in the absolute position estimation processing flow of the person shown in FIG. In this embodiment, it is assumed that Mr. A's lost flag is TRUE.

  FIG. 10 is a flowchart for explaining a control method of the information display system according to the present embodiment. This example is an example of relative position estimation processing between persons by the processing server 300 shown in FIG. This process is executed in parallel with the absolute position estimation process of the person shown in FIG. Based on the camera image and the azimuth, acceleration, and tilt information obtained by the MEMS sensor transmitted from the HMD 100 worn by each user via the wireless access point 400 and the network 9000, the processing server 300 determines the observer and the observer. The relative position of is estimated. Also, the processes from S1401 to S1405 are all realized by the CPU 305 of the controller 320 provided in the processing server 300 executing the control program.

In step S1401, the CPU 305 acquires and analyzes the captured image of the camera 102 built in the HMD 100, and detects a human body included in the captured image. An image of a human body included in the imaging of the camera built in the HMD 100 is as shown in FIG. As a method for detecting a human body in imaging, a detection method using a detector using a statistical learning method based on HOG (Histograms of Oriented Gradients) features is generally used. The explanation is omitted because it is not directly related.
In step S1402, the CPU 305 calculates the orientation of the face of the person (observer) wearing the HMD, that is, the angle to the foot of the human body during camera imaging with respect to the normal 601 of the HMD. Hereinafter, the angle calculation process will be described more specifically with reference to FIG.

FIG. 12 is a diagram illustrating the relationship between the distance and angle between the observer and the person to be observed.
In FIG. 12, h is the height of the observer's HMD from the ground, and this value may be registered in advance as static information in the system. L1 is the normal 601 of the HMD. L2 is a horizontal line of HMD height (h). L3 is a line connecting the HMD of the observer and the feet of the observer.
In S1402, the CPU 305 calculates an angle of L3 with respect to L1, that is, φ, as shown in FIG. In step S <b> 1403, the CPU 305 calculates the face orientation of the person (observer) wearing the HMD, that is, the orientation of the HMD normal 601 based on the tilt information from the MEMS sensor built in the HMD 100. In the example of FIG. 12, the angle β formed by L1 and L2 is calculated.
In step S <b> 1404, the CPU 305 calculates the distance between the height of the person (observer) wearing the HMD 100 and the angle information between the person being photographed by the camera. In the example of FIG. 12, the dist is calculated from the height h of the observer and the angle α = φ−β formed by L3 and the ground.

In step S <b> 1405, the CPU 305 wears the HMD for the human body during camera imaging based on the direction information of the MEMS sensor, that is, the face orientation of the person wearing the HMD (observer) and the relative distance dist between the persons. The position of the person is calculated.
As a result, it is possible to know in which direction and at what distance the observer is present with respect to the observed person in the physical space. In the present embodiment, the distance between the two is derived from the camera imaging and the MEMS sensor information, but may be derived by other methods such as measurement by short-range communication.

  FIG. 13 is a flowchart for explaining a control method of the information display system according to the present embodiment. This example is an example of person estimation processing during camera imaging. This process is executed after the relative position estimation process between persons shown in FIG. The processing server 300 estimates the presence position of the observer. The processing from S1501 to S1509 is realized by the CPU 305 of the controller 320 provided in the processing server 300 executing the control program. Hereinafter, a person image is detected from image data acquired from an information terminal associated with a user whose position is not estimated in the physical space in the process shown in FIG. 7, and is acquired from the information terminal of the user whose position is not estimated. Processing for calculating a relative distance to a human image detected using a conversion amount that changes according to the user's operation will be described.

In step S <b> 1501, the CPU 305 determines the presence position information of someone X in the physical space and the presence of a person (observer) wearing an HMD with respect to the human body that is being captured by the camera in the relative position estimation process flow between the persons. From the position, a candidate for the position of the person (observer) wearing the HMD when the person X is assumed to be reflected in the HMD imaging is calculated.
In step S1502, the CPU 305 determines whether the processing in step S1501 has been performed on all persons existing in the physical space. If it is determined that the process of S1501 has been performed for all persons existing in the physical space (true), the process proceeds to S1503.
On the other hand, if it is determined that the process of S1501 has not been performed on all persons existing in the physical space (false), the process returns to S1501. Hereinafter, a more specific description will be given with reference to FIG.

In FIG. 14, it is assumed that there are three persons in the physical space, Mr. A, Mr. B, and Mr. C, as shown in FIG. 4, and Mr. A is an observer. As illustrated in FIG. 14, when the person who is capturing the image of Mr. A is Mr. B and Mr. C, the position candidate of Mr. A is derived for each.
In step S <b> 1503, the CPU 305 reads physical space map information including information on the possibility of human existence for each location from the HDD 310. Hereinafter, a more specific description will be given with reference to FIG.
FIG. 15 is a diagram illustrating an example of physical space map information including human presence possibility information for each place. For example, on the floor map of an office, there are three high, medium, and low human existence probabilities for each location where there are walls, pillars, and fixtures in the office (equipment / equipment for displaying products and catalogs). It is assigned in stages, and this existence probability is linked. In the example of FIG. 15, it is assumed that there is a fixture on the upper right of the map, and the fixture basically lowers the possibility of human existence, but it can exist because the human being may sit on the edge of the fixture. Sex is medium. The map information is registered in advance in the HDD 310 of the processing server 300 as static information.
In step S <b> 1504, the CPU 305 calculates the possibility of existence of each person (observer) wearing the HMD in each existence position candidate, and assigns each person (observed person candidate) as a basis for deriving each existence position candidate. The possibility of existence of a person (observer) wearing the HMD is assigned. This will be described more specifically with reference to FIG.

FIG. 16 is a diagram in which Mr. A's location candidate is superimposed on the map shown in FIG.
In FIG. 16, it is assumed that Mr. A's existence position candidate derived based on the assumption that Mr. A is taking a picture of Mr. A's camera is low, and that the person being shot is Mr. C. It can be seen that the possibility of Mr. A's location candidate is high. Based on this calculation result, information that Mr. A's existence is low is associated with Mr. B, and information that Mr. A's existence is high is associated with Mr. C.
In step S <b> 1505, the CPU 305 employs the lowest value among the values calculated so far regarding the possibility of existence of a person (observer) wearing the HMD assigned to the candidate for the observer. In step S <b> 1506, the CPU 305 removes from the candidates candidates to be observed that the possibility value of the person (observer) wearing the HMD is equal to or less than a predetermined value. By doing so, the candidate for the observer is narrowed down as time passes and the person moves around.
In step S1507, the CPU 305 determines whether the predetermined time has elapsed. If the CPU 305 determines that the predetermined time has elapsed (true), the process advances to step S1508 to determine that the predetermined time has not elapsed (false). If yes, the process returns to S1501.
In step S <b> 1508, the CPU 305 determines the candidate to be observed who has the highest possibility of existence of the person (observer) wearing the HMD as a person who is capturing an image of the person (observer) wearing the HMD. Then, this process ends. In the example of FIG. 16, it is determined that the person who is capturing the image of Mr. A is Mr. C.
[Second Embodiment]

  In the present embodiment, a method for increasing the possibility of estimation of the method for estimating a person who is capturing an image of a person (observer) wearing the HMD described in the first embodiment will be described. In the present embodiment, in addition to the processing executed in the first embodiment, the person estimation assisting process during camera imaging shown in FIG. 18 is executed in parallel with the person estimation flow during camera imaging shown in FIG. Realize.

  Similar to the first embodiment, the process of the present embodiment is performed for a person whose lost flag is TRUE in the person absolute position estimation process flow shown in FIG. Assume that Mr. A's lost flag is TRUE.

  As described in the first embodiment, after the processing of S1502 shown in FIG. 13 becomes TRUE, as shown in FIG. In this case, Mr. A's location candidate is derived for each. Here, consider the case where Mr. A is stationary on the spot. Whether or not a person is stationary can be grasped by detecting acceleration with a MEMS sensor built in the HMD.

FIG. 17 is a diagram illustrating a state in which the observed person moves with time. For example, Mr. B moves in the direction of Mr. A's right hand, and Mr. C indicates that he is not moving on the spot.
In this information display system, the existence position relationship between Mr. A and Mr. B, Mr. A and Mr. C is grasped by the relative position estimation process between persons shown in FIG. In addition, the system grasps the positions of Mr. B and Mr. C in the physical space by the absolute position estimation process of the person shown in FIG. In the present embodiment, it is possible to estimate a person who is taking an image of a person wearing an HMD (observer) by removing the candidate for the observer whose results of the relative position estimation process and the absolute position estimation process are different from the candidates. Increase sex.

FIG. 18 is a flowchart for explaining a control method of the information display system according to the present embodiment. This example is an example of person estimation assistance processing during camera imaging. This processing is executed in parallel with the person estimation flow during camera imaging shown in FIG. Note that the processing from S1701 to S1703 is all realized by the CPU 305 of the controller 320 included in the processing server 300 executing the control program.
In S1701, the CPU 305 determines whether or not the movement amount of the person wearing the HMD 100 is “0”. If the movement amount of the person is “0” (true), the process proceeds to S1702. If it is determined that the amount of movement of the person is not “0” (false), the process ends.
In step S <b> 1702, the CPU 305 calculates the movement direction of the human body in the physical space from the face direction of the person (observer) wearing the HMD 100 and the movement direction of the human body during camera imaging. In step S <b> 1703, the CPU 305 excludes the person that does not match the movement direction detected in the absolute position estimation process flow shown in FIG. 7 from the person estimation flow process during camera imaging, and ends this process.

  Each process of the present invention can also be realized by executing software (program) acquired via a network or various storage media by a processing device (CPU, processor) such as a personal computer (computer).

  The present invention is not limited to the above embodiment, and various modifications (including organic combinations of the embodiments) are possible based on the spirit of the present invention, and these are excluded from the scope of the present invention. is not.

100 HMD
300 processing server

Claims (10)

An information display system in which a server device communicates with a plurality of information terminals moving in physical space,
Each information terminal
Imaging means for imaging an object entering the user's field of view;
Display means for displaying information in a state where the object can be transmitted;
Detecting means for detecting a physical quantity that changes in association with an action of a user moving on the physical space;
Transmitting means for transmitting imaging data obtained by imaging a physical space in which each user exists or detected detected physical quantity to the server device in association with each user information;
Control means for receiving from the server device information to be displayed and displayed on the display means associated with another user who is viewing through the display means; and
With
The server device
A database for managing feature point information existing in the physical space to which each user moves;
Obtaining means for obtaining the changing physical quantity detected by the detecting means of each information terminal and the imaging data imaged by the imaging means;
Position estimation means for estimating a position where each user should exist in the physical space from the feature points of the physical space detected from the imaging data acquired from each information terminal and the feature points managed in the database;
Detecting means for detecting a person image from imaging data acquired from an information terminal associated with a user whose position is not estimated in physical space;
Calculating means for calculating a relative distance from the imaged data to a human image detected from the imaged data captured by a user whose position is not estimated in a physical space and a physical quantity that changes in accordance with the user's operation; ,
A specifying means for specifying a user who is viewed by a user whose position is not estimated in the physical space from the calculated relative distance and the position of each user who should be present in the estimated physical space;
Output means for outputting information to be displayed in association with the identified user to an information terminal associated with the user whose position is not estimated in the physical space;
An information display system comprising:   The information display system according to claim 1, wherein the physical quantity that changes in association with the user's movement includes an azimuth, an acceleration, and an inclination. An information terminal that communicates with user information and a server device that manages feature information existing in physical space,
Imaging means for imaging an object entering the user's field of view;
Display means for displaying information in a state where the object can be transmitted;
Detecting means for detecting a physical quantity that changes in association with the action of a user moving in the physical space;
Transmitting means for transmitting imaging data obtained by imaging a physical space in which each user exists or detected detected physical quantity to the server device in association with each user information;
Control means for receiving from the server device information to be displayed and displayed on the display means associated with another user who is viewing through the display means; and
An information terminal comprising:   The information terminal according to claim 3, wherein the physical quantity that changes in association with the user's movement includes azimuth, acceleration, and inclination.   The information terminal according to claim 4, wherein the information terminal is a head mounted display that is positioned in accordance with a user's visual recognition position. A server device that communicates with a plurality of information terminals moving in physical space,
A database for managing feature point information existing in the physical space to which each user moves;
An acquisition means for acquiring a changing physical quantity and captured image data detected by each information terminal;
Position estimation means for estimating a position where each user should exist in the physical space from the feature points of the physical space detected from the imaging data acquired from each information terminal and the feature points managed in the database;
Detecting means for detecting a person image from imaging data acquired from an information terminal associated with a user whose position is not estimated in physical space;
Calculating means for calculating a relative distance from the imaged data to a human image detected from the imaged data captured by a user whose position is not estimated in a physical space and a physical quantity that changes in accordance with the user's operation; ,
A specifying means for specifying a user who is viewed by a user whose position is not estimated in the physical space from the calculated relative distance and the position of each user who should be present in the estimated physical space;
Output means for outputting information to be displayed in association with the identified user to an information terminal associated with the user whose position is not estimated in the physical space;
A server device comprising: A control method for an information terminal that communicates with user information and a server device that manages feature information existing in a physical space,
An imaging process for imaging an object entering the user's field of view;
A display step of displaying information on the display means in a state where the object can be transmitted;
A detection step of detecting a physical quantity that changes in association with the action of a user moving on the physical space;
A transmission step of transmitting imaging data obtained by imaging a physical space in which each user exists or a detected changing physical quantity to the server device in association with each user information;
A control step of receiving, from the server device, information to be displayed associated with another user who is viewing through the display means and displaying the information on the display means;
An information terminal control method comprising: A method for controlling a server device that communicates with a plurality of information terminals moving on a physical space,
A database for managing feature point information existing in the physical space in which each user moves,
An acquisition step of acquiring the changing physical quantity detected by each information terminal and the captured imaging data;
A position estimation step of estimating a position where each user should exist in the physical space from the feature points of the physical space detected from the imaging data acquired from each information terminal and the feature points managed in the database;
A detection step of detecting a human image from imaging data acquired from an information terminal associated with a user whose position is not estimated in physical space;
A calculation step of calculating a relative distance from the imaging data to a human image detected from the imaging data captured by a user whose position is not estimated in a physical space and a physical quantity that changes according to the user's operation; ,
A specific step of identifying a user who is viewed by a user whose position is not estimated in physical space from the calculated relative distance and the position of each user who should be present in the estimated physical space;
An output step of outputting information to be displayed in association with the identified user to an information terminal associated with a user whose position is not estimated in the physical space;
A control method for a server device, comprising:   A program for causing a computer to execute the method for controlling an information terminal according to claim 7.   A program for causing a computer to execute the method for controlling a server device according to claim 8.

Images