JP6361714B2 - Information processing apparatus, information processing system, control method thereof, and program - Google Patents

Description

  The present invention relates to an information processing apparatus, an information processing system, a control method thereof, and a program.

  In recent years, technologies of virtual reality (Virtual Reality / hereinafter referred to as VR), augmented reality (Augmented Reality), and mixed reality (Mixed Reality / hereinafter referred to as MR) have become widespread. Using MR technology, for a user wearing a head-mounted display (hereinafter, HMD), an MR space (a mixed reality space in which a real space and a virtual space are superimposed) in which a real object and a CG model are arranged in the real space. Can provide a simulated experience. Also in the AR, it is possible to provide an image by superimposing a CG model on the real space to the user wearing the HMD.

  Patent Document 1 and Cited Document 2 describe a mechanism for specifying a position of an HMD using a sensor such as a magnetic sensor or an optical sensor, and generating and displaying a mixed reality image in which a real image and a CG model are superimposed. Yes.

JP-A-11-88913 JP 2002-229730 A JP 2004-38470 A

  For example, when creating a booth or the like, there are cases where a place for placing and displaying a product to be noticed is considered. By using MR technology, a virtual booth displaying CG products can be created and displayed, and the examination can be performed without preparing a real shelf or product sample. As a method of selecting a CG when moving the CG, there is a method of specifying and selecting a CG in the line-of-sight direction of the HMD as described in Patent Document 3.

  Opinion on whether the display position of the product you want to have the third party (booth experience person) walk through the booth where the product was displayed and pay attention to the customers who entered the booth The product position is adjusted. However, the opinions of the experiencers are based on their own feelings of “whether or not they paid attention to the product”, and it was difficult to judge whether the senses and opinions of these experiences were really correct. .

  It is an object of the present invention to provide a mechanism that can easily identify whether an experienced person has focused on an object in a three-dimensional space.

The present invention, information including a user can communicate with the display device to be mounted, an acquisition unit configured to acquire information of the line of sight of the user the display device is mounted, and a storage means for storing the objects of the three-dimensional space, the a processing apparatus, by using the information of the line of sight, on the three-dimensional space, and focuses the object specifying means for the user to identify the object that has been focused, the previous SL user identified by the focused object specifying means A degree-of-interest specifying means for specifying the degree of attention indicating how much the object that has been noticed has received attention based on the information on the line-of-sight transition, and an image indicating the degree of attention specified by the degree-of-interest specifying means characterized by comprising output means for outputting to be displayed on the position corresponding to the body thing that it has been noted, the.

  ADVANTAGE OF THE INVENTION According to this invention, the mechanism which can specify easily whether the experience person was paying attention to the target object in three-dimensional space can be provided.

It is a figure which shows an example of a system configuration | structure in embodiment of this invention. It is a figure which shows an example of the hardware constitutions of various apparatuses in embodiment of this invention. It is a figure which shows an example of a function structure of various apparatuses in embodiment of this invention. It is a flowchart which shows the flow of the memory | storage process of the position and orientation of HMD in embodiment of this invention. It is a flowchart which shows the flow of the analysis process based on the position and orientation of HMD in embodiment of this invention. It is a flowchart which shows the flow of the output process of an analysis result in embodiment of this invention. It is a figure which shows an example of a structure of various data in embodiment of this invention. It is an image figure of MR space and an image figure of an analysis result display in an embodiment of the present invention. 5 is a flowchart showing a flow of MR image generation / output processing in the embodiment of the present invention. It is a flowchart which shows the flow of the log display process of the position replacement of a virtual object and HMD position and orientation in embodiment of this invention. It is a figure which shows an example of the display image of the selection part of the log display of the position replacement of a virtual object and the HMD position and orientation in embodiment of this invention. It is a figure which shows an example of the log display of HMD position and orientation in embodiment of this invention. It is a flowchart which shows the flow of the specific process of the positional relationship of a real object and HMD in the 2nd Embodiment of this invention. It is a flowchart which shows the flow of the analysis process of the log concerning a real object in the 2nd Embodiment of this invention. It is a figure which shows an example of the various data structure in the 2nd Embodiment of this invention. It is a figure which shows the mode of the object recognition in the 2nd Embodiment of this invention. It is a flowchart which shows the flow of the memory | storage process of the relative position and orientation of objects in the 3rd Embodiment of this invention. It is a flowchart which shows the flow of the output process of the analysis result in the 3rd Embodiment of this invention. It is a figure which shows an example of the various data structure in the 3rd Embodiment of this invention. It is a figure which shows an example of the analysis result object in embodiment of this invention. It is a figure which shows the mode of a user's eyes | visual_axis movement in embodiment of this invention.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

<First Embodiment>
FIG. 1 is a diagram illustrating an example of a system configuration in an embodiment of the present invention.

  The system of the present invention includes a server 200, a head mounted display 101 (hereinafter referred to as HMD101 / wearable display device), an optical sensor 102, and the like. In FIG. 1, various devices are assumed to be communicably connected via a network 150. It does not matter whether the network is wired or wireless.

  The server 200 stores a three-dimensional model (CG model / virtual object / virtual object) to be superimposed on a real image captured by the HMD 101. The information on the virtual object is, for example, as shown in the model information 720 in FIG. Details of the model information 720 will be described later with reference to FIG.

  The server 200 acquires a real image from the HMD 101 managed by the server 200, generates an MR image (a mixed reality image obtained by superimposing a real image and a virtual space image) on which the three-dimensional model is superimposed on the real image, and generates the HMD 101. The MR image is transmitted to the HMD 101 for display on the display. The HMD 101 displays the received MR image on the display.

  The optical sensor 102 detects the optical marker installed in the HMD 101 to identify the position and orientation of the HMD 101 in the real space (position = XYZ coordinates, orientation = rotation angle and direction of the XYZ axes), and The posture value is transmitted to the server 200. The server 200 can specify the position and orientation of the HMD 101 by receiving information on the position and orientation from the optical sensor 102. A known technique is used as a method for acquiring and specifying the position and orientation of the HMD 101 using the sensor. For example, the position and orientation of the HMD 101 are specified using a method described in Patent Document 1 / Japanese Patent Laid-Open No. 11-88913, Patent Document 2 / Japanese Patent Laid-Open No. 2002-229730, and the like. The above is the description of FIG.

  Next, with reference to FIG. 2, the hardware configuration of various apparatuses in the embodiment of the present invention will be described.

  The CPU 201 comprehensively controls each device and controller connected to the system bus 204.

  The ROM 202 stores various programs necessary for realizing functions executed by various devices such as a BIOS (Basic Input / Output System), an operating system (OS), and the like, which are control programs of the CPU 201.

  The RAM 203 functions as a main memory, work area, and the like for the CPU 201. The CPU 201 implements various operations by loading a program necessary for execution of processing into the RAM 203 and executing the program.

Various programs used by the client PC 100 and the server 200 of the present invention to execute various processes to be described later are recorded in the external memory 211 and executed by the CPU 201 by being loaded into the RAM 203 as necessary. It is. Furthermore, definition files and various information tables used by the program according to the present invention are stored in the external memory 211.
An input controller (input C) 205 controls input from a pointing device (input device 210) such as a keyboard and a mouse.

  A video controller (VC) 206 controls display on a display device such as the right eye / left eye display 222 provided in the HMD 101. For example, an external output terminal (for example, Digital Visual Interface) is used for output to the right eye / left eye display 222. The right-eye / left-eye display 222 includes a right-eye display and a left-eye display.

  The memory controller (MC) 207 is an adapter to a hard disk (HD), flexible disk (FD) or PCMCIA card slot for storing boot programs, browser software, various applications, font data, user files, editing files, various data, and the like. Controls access to an external memory 211 such as a card-type memory connected via the.

  A communication I / F controller (communication I / FC) 208 is connected to and communicates with an external device via a network, and executes communication control processing in the network. For example, Internet communication using TCP / IP is possible. The communication I / F controller 208 also controls communication with the optical sensor 102 (optical sensor) through Gigabit Ethernet (registered trademark) or the like.

  The general-purpose bus 209 is used for capturing video from the right-eye / left-eye video camera 221 of the HMD 101. Input from the right-eye / left-eye video camera 221 is performed using an external input terminal (for example, an IEEE 1394 terminal). The right-eye / left-eye video camera 221 includes a right-eye video camera and a left-eye video camera.

  Note that the CPU 201 enables display on a display by executing outline font rasterization processing on a display information area in the RAM 203, for example. In addition, the CPU 201 allows a user instruction with a mouse cursor (not shown) on the display. The above is the description of FIG.

  Next, with reference to FIG. 3, an example of a functional configuration of various apparatuses in the embodiment of the present invention will be described.

  The imaging unit 311 is a camera of the HMD 101 and is an imaging unit that captures a real image. The captured image transmission unit 312 is a transmission unit that transmits a captured image (real image) to a server corresponding to the own device.

  The MR image receiving unit 313 is a receiving unit that receives MR images generated by the server 200. The MR image display unit 314 is a display unit that displays a received MR image on a display screen.

  The captured image reception unit 321 of the server 200 receives a real image from the HMD 101. The HMD position / posture specifying unit 322 is a specifying unit that specifies the position / posture of the HMD 101 by receiving the position / posture value of the HMD 101 from the optical sensor 102.

  The HMD position / orientation storage unit 323 is a storage unit that stores information on the position and orientation specified by the HMD position / orientation specifying unit 322 in an external memory. The HMD position / orientation storage unit 323 is the current position / orientation of the HMD 101, and is an MR image (superimposed image / mixed reality obtained by superimposing an image of a virtual object placed in a virtual space, as viewed from the position and orientation of the HMD, on a real image). Information on the position and orientation of the HMD 101 (also referred to as an image) (HMD information 700 in FIG. 7) and the current and past positions and orientations of the HMD 101 are continuously logged (HMD position and orientation log 710 / history in FIG. 7). As a storage means) in an external memory.

  The MR image generation / storage unit 324 is a storage unit that generates an MR image in which a virtual object is superimposed on a real image based on the current HMD position / orientation specified by the HMD position / orientation specifying unit 322 and stores the MR image in an external memory. is there. The MR image transmission unit 325 is a transmission unit that transmits the MR image to the HMD 101.

  The virtual object storage unit 326 stores information on virtual objects to be arranged in the virtual space (arranged in the MR space by aligning the virtual space and the real space). Specifically, information as shown in the model information 720 in FIG. 7 is stored in the external memory of the server 200.

  The model information 720 includes a model name 721 which is identification information of a model (virtual object), a position 722 in a virtual space where the model is arranged / displayed, a posture 723, and the model (virtual object). Attention flag 724 (corresponding to the target object storage means) indicating whether or not to what extent the attention is focused on, and a virtual object when the line of sight of the HMD 101 passes through which space It is comprised from the attention area 725 etc. which show the area | region of determining that it was paying attention to. For example, when a straight line indicating the line-of-sight direction of the HMD 101 specified from the position and orientation of the HMD (the direction in which the HMD 101 faces) and the outer surface of the area indicated by the area of interest 725 correspond to the area of interest 725 It is determined that the user of the HMD 101 was viewing the virtual object to be viewed (the user was paying attention). The virtual object with the attention flag 724 = 1 is a target object for determining whether or not the user has focused on the object. The virtual object with the attention flag 724 = 0 is not a determination target of whether or not the user has focused on the object. It is assumed that the value of the attention flag 724 is stored for each virtual object in advance. The attention area 725 is a three-dimensional area occupied by the smallest cube surrounding the virtual object when the virtual object indicated by the model name 721 is arranged in the virtual space at the position 722 and the posture 723, and each vertex of the cube Coordinates are stored. The coordinates of the intersection point at which the vector of the line-of-sight direction and the outer surface of the cube representing the area intersect are the coordinates of the point (point of interest) that the user has focused on in the virtual object corresponding to the area of interest. Note that the method of determining whether or not the user has focused on the object is not limited to this. For example, when the gaze direction vector intersects the outer surface of the virtual object itself, it is determined that the user is paying attention to the object, and the user determines the coordinates of the intersection point where the gaze direction vector intersects the outer surface of the virtual object itself. It is good also as the coordinate of the focus point which paid its attention to the object which is an object.

  The analysis execution unit 327 analyzes and analyzes the position / orientation log of the HMD 101, and determines whether or not the user has focused on the target virtual object, and how much the user has focused on the target virtual object Part.

  The analysis result information generation unit 328 generates analysis result information so that the user can confirm the analysis result of the analysis performed by the analysis execution unit 327. For example, in order to make the user confirm the analysis result in the MR space, as shown at 810 in FIG. 8, the virtual object is arranged in the virtual space in association with each virtual object and indicates how much the user has seen each virtual object. The virtual object is generated, and the virtual object of the analysis result (the balloon in 810 in FIG. 8) is arranged at a predetermined position as viewed from the respective virtual object to be analyzed (CG of the displayed product). The position and orientation of the analysis result virtual object are determined and stored in the external memory in association with the analysis result virtual object. The above is the description of FIG.

  Next, with reference to FIG. 4, the flow of the storage processing of the position and orientation of the HMD in the embodiment of the present invention will be described.

  When the CPU 201 of the server 200 receives an instruction to start measurement / specification processing of the position and orientation of the HMD 101 (for example, a user operation), the CPU 201 acquires the position and orientation information of the HMD 101 from the optical sensor (step S401) and stores it in the external memory (step S401). Step S402). Specifically, the CPU 201 of the server 200 stores the identification information of the HMD 101 in the HMDID 701 of the HMD information 700 in FIG. 7 and the position and orientation information in the position 702 and the orientation 703. The HMD information 700 indicates the current position and orientation of the HMD 101. The information on the attitude of the HMD in the present embodiment is information indicating the rotation angle of the HMD and the direction in which the HMD is facing.

  In addition, the CPU 201 of the server 200 adds the latest data string to the HMD position / orientation log 710 to identify the identification information of the HMD 101, the position / orientation acquired in step S401, and the date / time when the position / orientation information is acquired (specify the position / orientation). Are inserted into HMDID 711, position 713, posture 714, and date 712, respectively, and stored. The HMD position / orientation log 710 is an HMD position / orientation measurement history stored in an external memory of the server 200. The log ID 715 is identification information given to the log when the log is output as a file in step S403 of FIG. The posture 714 is history information indicating a change in the viewing direction of the user of the HMD 101.

  The CPU 201 of the server 200 repeatedly executes the processes of steps S401 and S402 until an instruction to end position / orientation measurement (for example, an operation from the user) is received. When receiving an instruction to end the position / orientation measurement, the CPU 201 of the server 200 uses the HMD position / orientation log 710 stored from the start to the end of the position / orientation measurement as a single file in the predetermined memory of the external memory 211 of the server 200. Output / store in the area (step S403). Note that a new log ID 715 is created and embedded in the file at the time of output. Log ID = identification information of the log output in step S403. The above is the description of FIG.

  Next, the flow of MR image generation processing in the embodiment of the present invention will be described with reference to FIG.

  The HMD 101 captures a real image with a video camera installed in the device (step S901), and transmits the captured real image to the server 200 (step S902).

  The server 200 acquires a virtual object (model information 720 in FIG. 7), and arranges each virtual object in the virtual space with the position and orientation of the position 722 and the orientation 723 (step S903).

  The CPU 201 of the server 200 receives the real image and stores it in the external memory (step S904). It is assumed that the actual image includes information on the imaging date and time. Further, the CPU 201 of the server 200 is assumed to store in association with the ID of the HMD 101 that captured the real image. Then, the current position and orientation of the HMD 101 shown in the HMD information 700 of FIG. 7 is read out, and according to the value of the position and orientation (the virtual camera is placed in the virtual space and the virtual space image (virtual object image / virtual An MR image is generated and stored in the memory (by capturing the image) and superimposing it on the real image acquired from the HMD 101 (step S905).

  The CPU 201 of the server 200 transmits the MR image to the HMD 101 (step S906), and the HMD 101 receives the MR image (step S907) and displays it on the display screen (step S908). The above is the description of FIG.

  Next, the flow of analysis processing based on the position and orientation of the HMD in the embodiment of the present invention will be described with reference to FIG.

  The CPU 201 of the server 200 receives a log analysis instruction (for example, user operation) (step S501). For example, the log analysis operation accepts the file selection and log analysis operations output in step S403, and accepts pressing of an analysis start button on a display screen (not shown) of the server 200, thereby selecting the selected HMD position. It is executed on the posture log file.

  The CPU 201 of the server 200 acquires the model information 720 (step S502), and acquires the virtual object with the focus flag = 1 from the virtual objects of the model information 720 (step S503). In other words, in the log acquired in step S502, a virtual push object to be analyzed for identifying whether or not the user of the HMD 101 has been focused is identified, and a list of objects to be analyzed is generated on the external memory. And remember. For example, the ranking information 730 in FIG. 7 is generated and stored. The ranking information 730 is a table that stores how much time (focused time 733 / focused degree) the user has focused on each virtual object (model name 732).

  The CPU 201 of the server 200 acquires one file log (information 711 to 714 shown in the HMD position / orientation log 710 in FIG. 7) acquired in step S501 (step S504), and the viewpoint of the HMD 101 indicated by the log The line-of-sight direction of the HMD 101 is specified from the (position) and posture (step S505). That is, it is specified from which position the user of the HMD 101 is viewing which direction.

  The CPU 201 of the server 200 determines whether there is a virtual object to be analyzed generated in step S503 ahead of the identified line-of-sight direction. Specifically, it is determined whether or not the straight line indicating the line-of-sight direction identified in step S505 and the attention area 725 of the virtual object with the attention flag 724 = 1 are passed. If it has passed, it is determined that there is a virtual object corresponding to the area of interest 725 ahead of the line of sight. Among the attention areas 725 of the virtual object with the attention flag 724 = 1, when no straight line indicating the line-of-sight direction passes through any area 725, it is determined that there is no virtual object corresponding to the attention area 725 beyond the line of sight. .

  If there is no virtual object at the end of the line of sight, the process proceeds to step S509. When there is a virtual object at the end of the line of sight, for example, among the virtual objects at the end of the line of sight, the virtual object closest to the HMD position (nearest) is identified as the virtual object that the user of the HMD 101 has focused on (Step S507 / corresponding to object-of-interest specifying means). However, when the transparency of the virtual object has reached the threshold value, the virtual object located next to the HMD next to the virtual object is identified as the virtual object that the user has focused on among the virtual objects ahead of the line-of-sight direction. To do. For example, a virtual object with a transmittance of 70% or more is ignored, and a virtual object with a transmittance of 30% or less at the next position close to the HMD is set as a target virtual object.

  The CPU 201 of the server 200 acquires the date and time 712 of the next log after the log acquired in step S504, and the user pays attention to the time from the date and time 712 of the log acquired in step S504 to the date and time 712 of the next log. It adds to the attention time 733 corresponding to the model name 732 of the virtual object that has been (corresponding to step S508 / attention level specifying means).

  The CPU 201 of the server 200 determines whether or not the processing in steps S504 to S508 has been executed for all the HMD position / orientation logs in the log file acquired in step S501, and if not, returns the processing to step S504 to perform unprocessed processing. Get the log.

  If it has been executed, the virtual objects in the ranking information 730 are ranked in descending order of the attention time 733, and the ranking (degree of attention degree) is written in the ranking value 731 (value indicating the degree of attention) (step S510 / degree of attention) Applicable to specific means). Then, the ranking information 730 is stored as information of an analysis result (analysis result) in association with a log file (step S512). Specifically, the ranking information 730 for which the ranking value has been written is associated with the log ID of the log file acquired in step S501 (log ID 734), and the ranking information 730 is stored in a predetermined area of the external memory 211. Output and store as The above is the description of FIG.

  According to the processing of FIG. 4 and FIG. 5, it can be easily specified whether the experience person has focused on the object in MR.

  Next, with reference to FIG. 6, the flow of output processing of analysis results in the embodiment of the present invention will be described.

  The CPU 201 of the server 200 receives a log analysis result display instruction (for example, user operation) (step S601). For example, the selection of the analysis result file stored in the external memory of the server 200 is accepted, and by pressing the display of the analysis result display button on the display screen (not shown) of the server 200, the display instruction of the log analysis result is given. It shall be accepted.

  The CPU 201 of the server 200 reads the selected analysis result file (the file having the information shown in the ranking information 730) onto the memory (step S602), and acquires one piece of the read analysis result information (unprocessed ranking). Information is acquired in descending order of ranking (step S603).

  The CPU 201 of the server 200 generates a virtual object indicating information of the acquired analysis result (corresponding to step S604 / attention level object generation means), determines the position and orientation of the virtual object (step S605), and stores it in the external memory. Remember. Further, as shown in the analysis result model information 740 of FIG. 7, the analysis result model information 740 having the log ID 741 corresponding to the analysis result selected in step S601 is generated, and the analysis result model name 742 is generated. The file name of the virtual object indicating the analysis result is inserted, the model name 732 to be analyzed is inserted into the model name 745, and a predetermined position from the virtual object of the model name 745 (from the position of the virtual object of the model name 745 to a predetermined position) XYZ coordinates to which the offset value is applied) and a predetermined posture value are inserted into the position 743 and posture 744 and stored.

  The CPU 201 of the server 200 determines whether or not the processing in steps S603 to S605 has been executed for all analysis results (all models in the ranking information 730) in the file acquired in step S601 (step S606), and has not been executed. In this case, the process returns to step S603 to obtain an unprocessed analysis result. If the process has been executed, the process proceeds to step S607, and the MR image generation / output process (identification display process of the target virtual object) shown in FIG. 9 is executed. The above is the description of FIG.

  In the MR image generation process in step S607, the CPU 201 of the server 200 acquires the analysis result model information 740 in addition to the model information 720 in the virtual object acquisition / placement process in step S903. A process of arranging a model (virtual object) in a virtual space is performed. Further, the CPU 201 of the server 200 reads and arranges the analysis result model information 740, and displays a virtual object (the analysis result model information of the analysis result model information 740) until an analysis result display end instruction (for example, user operation) is received. It is assumed that generation and output of an MR image using an image of a virtual space in which 740 virtual objects are arranged are continued. On the HMD 101, an MR image in which the virtual object of the analysis result model information 740 can be seen (the virtual object on which the user has focused) is displayed.

  An example of the MR space in which the virtual object of the model information 720 is arranged is shown at 800 in FIG. An example of the MR space in which the virtual object of the analysis result model information 740 is arranged in addition to the model information 720 is shown at 810 in FIG.

  Next, with reference to FIG. 10, the flow of the virtual object position replacement and the HMD position / posture log display process according to the embodiment of the present invention will be described.

  When the CPU 201 of the server 200 receives the selection of the virtual object (step (S1001)), whether the analysis result of the virtual object is being displayed (whether the MR image is being generated / displayed in step S607 / analysis result display instruction is received) (Step S1002) A virtual object selection method is performed using an existing technique (for example, a virtual object that contacts a selection virtual object is selected). ).

  If the analysis result of the virtual object is not being displayed, the processing is terminated. If the analysis result of the virtual object is being displayed, the processing is performed at 1101 to 1103 in FIG. 11 at a position away from the selected virtual object by a predetermined distance. Create and place a virtual object for the selection button. That is, the position and orientation of the virtual object of the user operation unit, which are selection buttons 1101 to 1103 in FIG. Then, an MR image on which the selection buttons 1101 to 1103 are superimposed is generated according to the position and orientation of the HMD 101 and the position and orientation of the virtual object of the selection button in the virtual space, and is output and displayed on the HMD 101. (Display control).

  If the CPU 201 of the server 200 determines in step S1004 that the replacement target virtual object has been selected, the process proceeds to step S1010, where the model is used to determine the position of the virtual object selected as the replacement target and the selected virtual object. A process of replacing in the information 720 is performed (step S1010), and the process proceeds to step S1009. That is, the MR image generation / output process of FIG. 9 is performed in order to generate / output an MR image in which virtual objects are exchanged according to an instruction (step S1009).

  When it is determined in step S1004 that the replacement target virtual object has been selected, specifically, the button 1101 is selected, and the server 200 generates the replacement target virtual object candidate selection buttons 1111 and 1112. This is a case where the placement / drawing is performed as indicated by 1110 in FIG. 11 and the selection button 1111 or 1112 is selected.

  Here, the virtual object to be displayed on the selection buttons 1111 and 1112 is set as a higher ranking object than the virtual object being selected. As a result, it is possible to prevent the display screen from becoming complicated due to a large number of replacement candidates being displayed.

  In order to prevent the display screen from becoming complicated due to a large number of replacement candidates being displayed, a virtual object with a ranking lower than the virtual object being selected may be displayed as a replacement candidate.

  Further, when there are no virtual objects as the display screen becomes complicated, all the virtual objects being displayed in the analysis result may be displayed on the selection buttons. Further, all virtual objects that have been analyzed may be displayed on the selection button.

  In accordance with the selection of the virtual object, the position of the virtual object designated by the selection of the selection button can be exchanged on the MR space.

  The CPU 201 of the server 200 determines whether or not the sort button 1102 has been selected in step S1005. When the sort button 1102 is clicked, for example, information on the first to nth positions in the order that you want to pay attention, which is stored in advance in the server 200 in association with the virtual object, is acquired. The position of the virtual object that is set to the first place in the desired order is changed to the position where the virtual object that is actually ranked first in the ranking is placed (step S1011). Thereafter, the process proceeds to step 1009, and the MR image generation / output process of FIG. 9 is performed in order to generate / output the MR image in which the positions of the virtual objects are rearranged according to the instruction (step S1009). Accordingly, the virtual objects can be collectively arranged at the virtual object arrangement positions that are actually noticed in the order in which the user desires attention.

  In step S1006, the CPU 201 of the server 200 determines whether the HMD position / orientation log display instruction button 1103 is selected, and when the HMD position / orientation log display instruction button 1103 is selected, pays attention to the model being selected. The HMD position / orientation log at the time of being specified is specified (step S1007). The identification method of which HMD user of which log has been paying attention to which virtual object from when to when (from which time to which time has been noted) is as described in FIG. For example, every time an object (virtual object or real object) focused on by the user is specified, the ID of the HMD 101 of the focused user, the date and time at which the user focused, and the position of the HMD 101 for each object By storing the association information in which the posture is associated in the external memory (corresponding to the association storage unit / posture association storage unit), it is possible to identify which HMD user has been paying attention to which virtual object from when to when. It is. In step S1007, the CPU 201 of the server 200 acquires a log (HMD position and orientation log or the like) of the log ID 741 of the analysis result being displayed, refers to the date and time 712, and from the first date and time to the last date and time of the log. In the meantime, by acquiring the association information of the HMD indicated by the HMDID 711, the position and orientation of the HMD 101 focused on the selected object can be specified for each date and time.

  The CPU 201 of the server 200 generates a virtual object (1202 in FIG. 12) indicating the position of the HMD at the time when the user focused on the selected virtual object identified in step S1007 (corresponding to the position object generating unit). / Corresponding to the posture object generating means), as shown in FIG. 12, the processing of arranging the posture of the HMD 101 at the position of the HMD 101 indicated by each log is performed (corresponding to step S1008 / position and posture determining means). As indicated by 1210 in FIG. 12, a straight line (arrow) indicating the line-of-sight direction of the HMD when the user focuses on the virtual object being selected, which is specified from the position of the HMD and the position and orientation of the HMD. May be generated and placed in the virtual space. The arrow starts from the position of the virtual object indicated by 1202 (= the position of the HMD 101 at a certain point in time while the user of the HMD 101 focuses on a certain virtual object), and the virtual object is focused from that position. This is an arrow-shaped virtual object extended to the point of interest (end point) on the virtual object. 1210 in FIG. 12 is an overhead view of the MR space as seen from above.

  The CPU 201 of the server 200 generates and arranges the MR image of FIG. 9 to display on the HMD 101 the virtual object indicating the position of the HMD at the time when the user focused on the selected virtual object generated and arranged in step S1007. An output process is performed (step S1009).

  Thereby, it is possible to easily confirm the position of the user who has focused on the virtual object.

  The CPU 201 of the server 200 superimposes the placement of the log of the HMD position and orientation in the virtual space and the virtual object of the log of the position and orientation of the HMD until receiving an instruction to end the log display of the position and orientation of the HMD. The generation of the MR image is continued. When the end instruction is received, the virtual object indicating the position and orientation log of the HMD is deleted.

  Note that only 1202 in FIG. 12 may be generated and arranged, or the position 1202 (the position of the HMD at the time when the user focused on the selected virtual object) as indicated by 1201 in FIG. By generating the smallest sphere or cuboid to be included and placing it in the virtual space, it becomes possible to confirm the position of the user of the HMD 101 during the period when the user has focused on the selected virtual object.

  As described above, according to the present invention, it is possible to provide a mechanism capable of easily specifying whether an experienced person has focused on an object in MR.

  In addition, for example, by comparing the opinions of the experience and the actions in the booth (which products were focused / focused on), the consistency of the opinions and actions of the experience was confirmed, and this consistency was used for the experience. It may be possible to decide whether or not to adopt the opinion of the person. In order to monitor the behavior of the experience person, there is a method in which a camera is arranged around the booth, the action of the experience person is photographed and recorded, and the recorded video is confirmed later. A method of arranging a plurality of cameras around the booth and visually checking a plurality of recorded moving images on a PC monitor in order to accurately monitor the behavior of the experience person can be considered, but it is troublesome to check the plurality of moving images. In addition, it is possible to check the recorded video by placing the camera so that you can look down at the booth, but since the behavior of the experience that appears small in the video is visually confirmed, which product the experience is focused on by the experience Difficult to check.

  On the other hand, in the present invention, the degree to which the experience person has focused on the object is specified according to the history of the position and orientation of the HMD and the position and orientation of the virtual object, and the action of the experience person is easily confirmed in the MR space. be able to.

  That is, in MR, it is possible to provide a mechanism that can easily identify whether an experienced person has focused on an object.

  In the above-described embodiment, the selection of one HMD position / orientation log file is accepted, and the processing of FIG. 5 is performed on the one log file. You may select and perform the process of FIG. 5 with respect to the said several log file. By analyzing the degree of focus on a plurality of logs (for example, logs of positions and orientations of different users), the accuracy and reliability of the analysis result can be improved.

  Specifically, the CPU 201 of the server 200 receives selection of a plurality of log files in step S501, and acquires the plurality of log files. And the process of step S504-S508 is performed with respect to all the logs of a some file. In this case, the CPU 201 of the server 200 stores and outputs the analysis results of all the logs selected in step S501 as one ranking information 730 file in step S510, and logs all the selected logs in the log ID 734. Insert ID. The CPU 201 of the server 200 generates an MR image based on the analysis results of the plurality of logs and displays the MR image on the HMD 101 by receiving selection and display instructions for the analysis results of the plurality of logs in step S601.

  Further, in the above-described embodiment, the virtual object with the flag of interest 724 = 1 is set as the analysis target in FIG. 5, but for example, all virtual objects stored in the model information 720 may be set as the processing target in FIG. . Specifically, the CPU 201 of the server 200 executes steps S505 to S509 for all the virtual objects acquired in step S502 without executing the process of step S503.

  In the above-described embodiment, the processes in steps S603 to S605 in FIG. 6 are executed for all the virtual objects of the analysis result acquired in step S601. The processes in steps S603 to S605 may be executed only for the virtual object.

  Specifically, in step S606, the CPU 201 of the server 200 determines whether steps S603 to S605 have been executed for the virtual objects up to a predetermined number in the ranking. If not, the process returns to step S603. If it has already been executed, the process proceeds to step S607. Thereby, even when the number of virtual objects arranged in the virtual space is large (for example, when there are many virtual objects of interest flag = 1), it is possible to prevent the display of the analysis result information from becoming complicated. The predetermined number of values is stored in the external memory of the server 200, and the setting can be freely changed by a user operation.

  In the above-described embodiment, the virtual object of the analysis result model = the ranking value 731 and the time of interest 733 are both generated as a virtual object embedded (pasted as a texture) (810 in FIG. 8). For example, a virtual object in which either the ranking value 731 or the time of interest 733 is embedded may be generated and arranged as a virtual object of the analysis result model.

  In the above-described embodiment, the server 200 generates a virtual object of an analysis result model, arranges it in a virtual space, and generates an MR image, thereby displaying the attention level (analysis result) of each virtual object on the HMD. However, the method of identifying and displaying the degree of attention is not limited to this. For example, in step S604, instead of generating the virtual object of the analysis result model, a process of changing the color of the virtual object that has been focused according to the ranking of interest may be performed. For example, the virtual objects in the top five rankings may be processed so as to have the same color, or the colors corresponding to the rankings are stored in the external memory of the server 200 in advance, and the colors corresponding to the ranking values are stored. Alternatively, the color of each virtual object may be changed.

  Further, if attention is paid for a certain period of time (for example, 1 minute or more), the virtual object may be identified and displayed. Specifically, the value of the predetermined time is stored in advance in the external memory of the server 200, and the CPU 201 of the server 200 receives the time of interest (FIG. 7) of the analysis result obtained in step S603 after step S603 in FIG. 730 and 740 (which can be specified from the log ID and model name) has reached the predetermined time. If the predetermined time has been reached, the process proceeds to step S604, and the time of interest has reached the predetermined time. If not, the process proceeds to step S606. Thereby, it is possible to easily identify and confirm whether the virtual object has been focused for a predetermined time.

  In the above-described embodiment, the position and orientation of the HMD are specified using the sensor. However, since only the position and orientation of the HMD can be specified, it is not always necessary to use the sensor. For example, a plurality of markers may be attached to the room, and the position and orientation of the HMD 101 may be specified by capturing the markers with the HMD 101 camera, and an MR image may be generated.

  When the position and orientation of the HMD is specified using the marker and the virtual object is superimposed on the real image, the shape of the marker is not limited, but is, for example, all squares and the same size. Further, it is assumed that a unique marker number is embedded in each marker. It is assumed that each marker can be identified when imaged by a camera provided in the HMD 101, and the marker number can be obtained when decoded. The type of marker (the type of role played by the marker) includes a position detection marker for determining the position and orientation of the HMD 101, and a virtual object arrangement used to draw a virtual object at a location defined by the marker There are two types of markers. And about a position detection marker, the some marker shall be affixed on the known position previously.

  The principle of detecting the position and orientation of the HMD 101 will be described. From the size of each of the three position detection markers (its positions are known) in the captured image, the distance from the HMD 101 to each position detection marker is obtained. Conversely, the position where the three distances obtained from the three position detection markers overlap is determined as the position of the HMD 101. Further, the attitude of the HMD 101 may be obtained from the arrangement of three position detection markers in the captured image.

  When the viewpoint is in the normal direction of the marker, the marker image looks square. When the viewpoint deviates from the normal direction, the square appears to be distorted depending on the degree of the deviation. That is, the orientation of the plane defined by the marker with respect to the axis of the viewpoint is found from this distortion, the distance between the viewpoint and the marker can be detected from the size of the marker, and the plane on which the marker is attached can be defined. In the embodiment, the marker is provided with two marks that can be distinguished from each other. The center position of the marker is the origin, and two vectors from the origin to the mark on the plane defined by the marker are 2 It is assumed that three axes defining local three-dimensional coordinates are defined as one axis in the normal direction from the center position of the axis and marker.

  Further, by comparing the size and shape of one position detection marker in the captured image and the position on the image with the size of the marker stored in the server 200 in advance, the relative relationship between the HMD 101 and the marker can be determined. It is possible to specify the position and orientation. As a relative position / posture specifying method, for example, a known technique described in Japanese Patent Application Laid-Open No. 2013-61870 is used.

  The virtual object placement marker is a marker used to specify the position where the virtual object is displayed. When the virtual object placement marker is detected, the CPU 201 of the server 200 and the client PC 100 detects the position where the virtual object placement marker is present (exactly offset from the center of gravity position of the virtual object placement marker as will be described later). To display a virtual object at a position having a). For example, a process of arranging the three-dimensional model at a position in the virtual space indicated by the offset is performed.

  A virtual object is a three-dimensional model (also called three-dimensional CAD data, 3D model, CG, etc.) arranged in a virtual space. Note that the data of the 3D model is associated with position and orientation information indicating in what position in the virtual space the 3D model is placed in the virtual space on the external memory of the server 200 and the client PC 100. Is remembered.

The coordinates of the center position of the virtual object placement marker are {Xv, Yv, Zv}, and the reference coordinates in the data of the three-dimensional model are {Xi, Yi, Zi} (where i = 0, 1, 2,. When defined as ..), the distance L is obtained by the following equation.
L = {(Xv-Xi) 2+ (Yv-Yi) 2+ (Zv-Zi) 2} 1/2

  Here, the minimum distance L when i is changed is the distance between the virtual object placement marker and the three-dimensional model placed according to the position of the virtual object placement marker. In the above, when calculating the distance, the square root is finally obtained. However, if it is sufficient to determine the size, it is not necessary to obtain the parallel root, and the sum of squares of the coordinate difference may be calculated. .

  In the above formula, the center position of the virtual object placement marker = the center position of the virtual object. As described above, the position at which the virtual object is arranged is given an offset indicated by “position” with respect to the center position of the virtual object arrangement marker.

  Depending on the number of bits that can be embedded in the marker, a marker that serves both as a position detection marker and a virtual object placement marker may be defined.

<Second Embodiment>
A second embodiment of the present invention will be described with reference to FIGS. Note that description and description of processes common to the first embodiment are omitted.

  In the second embodiment, it is an object to provide a mechanism that can easily identify whether an experienced person has focused on an object that is a real object.

  Note that in the second embodiment, it is possible to easily confirm in which position the real object is focused on in the three-dimensional space.

  Further, the position and orientation of the HMD when paying attention to the real object can be easily displayed in the three-dimensional space.

  With reference to FIG. 13, description will be given of a process for specifying the positional relationship between a real object and an HMD in the second embodiment of the present invention.

  The CPU 201 of the server 200 executes in parallel with the processing in FIG. 9 and the logging of the position and orientation of the HMD 101 in FIG. For example, the processing of FIG. 13 is started and executed by receiving a start instruction (for example, user operation) of the position / orientation measurement / specification processing of the HMD 101.

  The CPU 201 of the server 200 acquires the latest real image (captured image) acquired in step S904 from the memory.

  The CPU 201 of the server 200 analyzes the real image and performs object recognition processing. Specifically, a subject (object) shown in the image is recognized, and information on the recognized object is stored in an external memory. Specifically, the information shown in the object information 1500 in FIG. 15 is stored.

  The object recognition processing here refers to recognition of an object on the image, recognition of the contour of the object on the image, recognition of coordinates (region surrounded by the contour) on the image, It refers to a process of recognizing and identifying a certain (what name the object is).

  Object recognition is performed using a known technique. For example, the image is divided into a plurality of parts using the Objectness method, and scoring is performed based on the saliency, the edge, the color, and the place, and the part with the high score is recognized as the object region. That is, recognition of the outline of the object on the image (area surrounding the object) and recognition of the coordinates where the object exists on the image are performed. Thereafter, the CPU 201 of the server 200 uses the dictionary data in which the object name stored in advance in the external memory and the feature amount of the object are associated to identify what the object in the recognized area (what name of the object )).

  Note that the above-described object recognition method is merely an example, and any method may be used as long as the outline of the object, the region on the image where the object exists, and what the object is can be recognized. For example, a CPMC (Constrained Parametric Min-Cuts) method may be used. Further, R-CNN (Regions with Convolutional Neural Network) technology may be used.

  The object information 1501 in FIG. 15 is stored for each date and time 1502 when the image of the HMD ID 1501 of the HMD 101 that is the transmission source of the real image is captured. The object ID 1503 is a unique value assigned by the CPU 201 of the server 200 for each recognized object, and is object identification information. The object name 1504 is a name of an object that is estimated and stored by the CPU 201 of the server 200 using dictionary data in which the object name stored in advance in the server 200 is associated with the feature amount of the object. An area 1505 is information indicating an area on the image where the object exists. Specifically, it is a set of vertex coordinates of the outline that indicates the region.

  A relative position 1506 and a relative orientation 1507 are relative positions and orientations of the position and orientation of the HMD 101 and the object with the object ID 1503 at the date and time 1502.

  The object position 1508 and the object posture 1509 are absolute positions in the three-dimensional space of the object with the object ID 1503 obtained from the position and posture of the HMD 101 and the relative position 1506 and the relative posture 1507 at the date 1502.

  Since the object ID is assigned to each object, for example, when a newly captured real image includes a real object that has been captured in the past, the object ID is the same. .

  The CPU 201 of the server 200 stores the object name of the real object in the newly captured real image and the image of the object stored in the external memory, and the object name of the real object that has been captured in the past and the external memory. When the similarity with the image of the object is equal to or greater than a threshold value (for example, 90%), it is determined and determined that these real objects are the same.

  A state of object recognition is shown by 1600 in FIG. Reference numerals 1600, 1610, and 1620 denote real images. Reference numeral 1601 denotes a marker. A marker is also a real object. Reference numeral 1602 denotes an area (1505) where the marker is detected. Reference numeral 1603 denotes a desk. Reference numeral 1604 denotes an area (1505) where a desk which is a real object is detected.

  The CPU 201 of the server 200 acquires the contour line of each real object on the real image being processed, and specifies the region surrounded by the contour line as the region where the real object is captured. Further, the XY coordinates on the image of the area are stored in the area 1505 (step S1301).

  Specifically, by storing the XY coordinates of a plurality of points (vertex in the case of a rectangle) on the contour line of the region, a region that can be connected by connecting the plurality of adjacent points is a region where a real object is captured. To do.

  The CPU 201 of the server 200 specifies the distance from the HMD 101 to each real object recognized. Specifically, the distance between each recognized real object and the HMD 101 is measured using a depth sensor (for example, an infrared depth sensor) (not shown) installed in the HMD 101 (step S1302).

  For example, by using a high-speed infrared light source and a camera (CMOS image sensor) for acquiring distance image data, the time for which the irradiated light returns to hit the target object is measured in real time for each pixel of the image. The distance to each object can be measured.

  The measurement process is a process for specifying how far each real object is separated from the HMD 101 in the three-dimensional space in the XYZ-axis directions. The relative positional relationship between each real object and the HMD 101 is specified to be relative. This is a process of storing in the position 1506.

  Note that information on the relative position between each camera device mounted on the HMD 101 and the depth sensor is stored in advance in an external memory by calibration. The position serving as a reference for the distance to be measured is the position of the HMD 101 set in advance by calibration and the center position of each real object region 1505.

  Note that the method of measuring the distance to the object does not necessarily have to be based on the depth sensor. For example, triangulation may be performed using two cameras installed in the HMD 101, and the distance to the object and the positional relationship with the object may be specified.

  The CPU 201 of the server 200 uses the absolute position and orientation of the HMD 101 obtained by the optical sensor and information on the relative positions of the HMD 101 and each real object measured by the depth sensor in the three-dimensional space. Specify the absolute position of the object. Information on the absolute position is stored in the object position 1508.

  By setting a coordinate system indicating the posture of each real object at a position in the same virtual space as the absolute position of each identified real object, the posture of each real object is determined. As the posture information, a predetermined rotation angle of each of the XYZ axes is adopted and stored in the object posture 1509 as an absolute posture of the real object in the three-dimensional space.

  The CPU 201 of the server 200 obtains a relative posture between the HMD 101 and each real object using the posture of the HMD 101 and the value of the object posture 1509 and stores the relative posture 1507.

  When the real object is a marker, the relative position / relative orientation between the marker and the HMD 101 is specified using the method for measuring the position and orientation of the HMD 101 using the position detection marker described above. . That is, the position and orientation of the marker are specified when the position of the HMD is the origin (XYZ coordinates = 0, 0, 0) and the rotation angle is 0 (rotation angle in the XYZ axis direction = 0, 0, 0). Remember.

  The CPU 201 of the server 200 identifies an object ahead of the line-of-sight direction (step S1303). Here, since the image used for the analysis is a real image, the object ahead of the line-of-sight direction is a real object.

  For example, as shown in FIG. 16, the HMD 101 display (image displayed on the display) is displayed at the center position 1611 (cursor 1611 indicating the point of interest) and the area 1505 (1602 and 1604 in FIG. 16). The real object is identified, and the real object is identified as an object ahead of the line-of-sight direction.

  In the example 1610, a marker is specified as a real object ahead of the line-of-sight direction. In the example 1620, a desk is specified as a real object ahead of the line of sight.

  The CPU 201 of the server 200 generates the attention history 1510 on the memory and specifies the object information 1500 of the real object that the user has focused on (at the time of the date 1502) identified in steps S1301 to S1303. Copy and store it in the attention history 1510 (step S1304).

  The attention history 1510 is a list of object information of objects that have been noticed, and is a history (log) that indicates when and which object has been noticed by the user. Note that each item of the attention history 1510 and the information stored therein are the same as the object information 1500, and thus the description thereof is omitted.

  The CPU 201 of the server 200 repeatedly executes the processing of steps S1301 to S1304 until receiving an instruction to end the position / orientation measurement described with reference to FIG. 4 (for example, an operation from the user).

  When an instruction to end position / orientation measurement is received, the CPU 201 of the server 200 stores the attention history 1510 stored from the start to the end of the measurement in the log file output in step S403 of FIG. . The above is the description of FIG.

  Next, with reference to FIG. 14, the flow of log analysis processing for a real object in the second embodiment of the present invention will be described.

  In step S504 of FIG. 5, the CPU 201 of the server 200 has one record of the HMD position / orientation log 710 in the log file selected and read in step S501, and the date / time 1512 having the same date / time as the date / time 712 of the record. The attention history 1510 is read out.

  The CPU 201 of the server 200 starts the process of FIG. 14 when it is determined in step S506 of FIG. 5 that the analysis target object is not beyond the line of sight.

  The CPU 201 of the server 200 refers to the attention history 1510 read in step S504, and determines whether there is a real object that the user has focused on (step S1401). If there is no real object focused on by the user, the process proceeds to step S509 in FIG.

  If there is a real object that the user has focused on, the real object is determined as a target object (object) of interest (corresponding to step S1402 / target object specifying means). Here, processing for inserting and storing the object ID of the real object in the model name 732 of the ranking information 730 is performed. The attention time 733 at this time is 0.

  Further, the record immediately after the record currently being processed in the HMD position / orientation log 710 acquired in step S504 is acquired, the date / time 712 of the record is acquired, and the date / time of the next record from the date / time 712 of the record being processed is acquired. The time up to 712 is added to 733 as the attention time that the user has focused on the real object (step S1403). Then, the process proceeds to step S509. The above is the description of FIG.

  The CPU 201 of the server 200 generates a virtual object as an analysis result relating to the real object and the virtual object by executing the process of step S604 in FIG. For the position and orientation of the virtual object indicating the analysis result information corresponding to the real object, in step S605, a position that is a predetermined distance away from the object position 1508 of the real object and a predetermined posture (predetermined from the position of the real object). It is assumed that the offset value is applied to XYZ coordinates / posture) and stored in the position 743 and the posture 744. The model name 745 is inserted with the object ID of the real object.

  The CPU 201 of the server 200 can display the virtual object (for example, 1202 in FIG. 12) of the analysis result relating to the real object and the virtual object according to the position and orientation of the HMD 101 by executing the process of step S607.

  Further, the CPU 201 of the server 200 displays the analysis result display virtual object (analysis result model) displayed in association with the real object (the object having the object ID stored in the model name 745) in step S1001 of FIG. By accepting the selection of the model / analysis result object (name 742), a selection button for the real object is displayed in step S1003. The analysis result object is a degree-of-interest object that indicates the degree of attention of the user to the object.

  In step S1007, all records of the real object (object ID) corresponding to the selected virtual object are extracted from the attention history 1510 in the log file.

  In step S1008, the position 713 and posture 714 of the HMD 101 at each date and time 712 focused on the real object are read, and the virtual position indicating the position of the HMD at the time when the user focused on the real object is read for each date and time 712. An object (1202 in FIG. 12) is generated.

  Then, the position of each virtual object 1202 (analysis result object 1202) is determined and stored as the position 713 of the HMD 101 at each date and time. That is, as shown in FIG. 12, a process of arranging a plurality of virtual objects 1202 at the positions indicated by the date and time 712 of each record is performed (step S1008).

  Note that an arrow-shaped virtual object is generated, and the line-of-sight direction of the HMD at the time when the user focused on the real object, starting from the position 713 of each date and time 712 of the HMD 101, as indicated by 1210 in FIG. The shape may be changed and the position may be determined so as to extend to the point of interest (end point) on the real object shown.

  As described above, according to the second embodiment of the present invention, it is possible to provide a mechanism that allows the user to easily identify whether or not the user has focused on a target object that is a real object.

  In addition, it is possible to easily confirm on the three-dimensional space which position the real object has been focused on.

  Further, the position and orientation of the HMD when paying attention to the real object can be easily displayed in the three-dimensional space.

  <Third Embodiment>

  A third embodiment of the present invention will be described with reference to FIGS. Note that descriptions and descriptions of processes common to the first embodiment and the second embodiment are omitted.

  In the third embodiment, even if the absolute position / posture in the three-dimensional space of the HMD cannot be specified, it is possible to provide a mechanism that allows the user to easily specify whether the user has focused on the object. .

  First, with reference to FIG. 17, the flow of the storage processing of the relative position and orientation of objects in the third embodiment of the present invention will be described.

  The CPU 201 of the server 200 starts the process of FIG. 17 by accepting an augmented reality display process start instruction (for example, a user operation on the server 200), and continues until the augmented reality display process end instruction (for example, a user operation) is accepted. The processing from S1701 to S1720 is repeatedly executed.

  The CPU 201 of the server 200 reads out the latest image received from the HMD 101 onto the RAM (step S1701), analyzes the real image, recognizes the object appearing in the image, and stores the information of the recognized object. This is performed (step S1702).

  Specifically, using the method described above with reference to FIG. 13, the recognition of the contour of the object (subject) on the image, the recognition of the coordinates (region) where the object exists on the image, Recognition object). Further, as described in step S1304 in FIG. 13, the recognized information is associated and stored in 1500 in FIG.

  The CPU 201 of the server 200 determines whether the augmented reality virtual object display registration image (for example, a virtual object placement marker) has been detected in the image read in step S1701 (step S1703). The registered image is stored in advance in the registered image information 1900, for example.

  The registered image information 1900 is stored in advance in the external memory 211 of the server 200. The image ID 1901 is registered image identification information. A model name 1902 indicates a virtual object corresponding to the registered image.

  The virtual object of the model name 1902 uses the augmented reality technique to determine the center position of the registered image based on the size, position on the image, and orientation of the registered image included in the real image captured by the camera. Is disposed in a position indicated by a position 1904 at a position distant in the XYZ directions by the position 1903 from the first position.

  If a registered image is detected, the process proceeds to step S1704, and the relative position and relative attitude between the registered image and the HMD 101 are calculated using the method described above in the description of step S1302 in FIG. (Step S1704).

  The CPU 201 of the server 200 generates a virtual object image having a model name 1902 corresponding to the registered image based on the shape, size, and direction of the registered image on the real image (step S1705). The region (coordinates) on the real image to be displayed (superimposed) is specified and stored (step S1706).

  For example, the ID assigned to the generated virtual object image and the coordinates of the display area are stored in the image ID 1911 and the area 1912 in the virtual object image information 1910 of FIG.

  The area 1912 stores a plurality of point coordinates (XY coordinates) on the outline of the outline of the image with the image ID 1911. A region formed by connecting a plurality of adjacent points is set as a region on which the image of the virtual object indicated by the image ID 1911 is superimposed.

  The model name 1913 stores the name of the virtual object so that the virtual object image with the image ID 1911 is the virtual object image indicated by the model name 1913.

  The CPU 201 of the server 200 generates an AR image (superimposed image / augmented reality image) obtained by superimposing the virtual object image on the real image, and transmits the generated AR image to the HMD 101 to display the augmented reality image on the HMD 101. (Step S1707).

  The CPU 201 of the server 200 identifies the viewing direction of the user of the HMD 101 (step S1708). As a method for specifying the line-of-sight direction, the method described in step S1303 in FIG. 13 is used.

  The CPU 201 of the server 200 determines whether the user is paying attention to the virtual object displayed in step S1707 from the line-of-sight direction (step S1709). Here, since the direction in which the HMD is facing = the line-of-sight direction, for example, when the virtual object region 1912 exists at the center position of the AR image, it is determined that the virtual object displayed in the region is focused. To do.

  If it is determined that the virtual object is focused, the process proceeds to step S1710. If it is determined that the virtual object is not focused, the process proceeds to step S1713.

  In step S1710, the CPU 201 of the server 200 identifies a point on the object that the user has focused on (three-dimensional focused point / XYZ coordinates) (step S1710).

  The CPU 201 of the server 200 captures the HMD that captured the real image, the date and time of image capture, the model name of the virtual object that the user focused on in the real image, the point of interest on the virtual object, the HMD 101 and the registered image at the time of focus. Are identified and stored in the virtual object attention history 1920 shown in FIG. 19 (step S1711 / step S1712).

  Specifically, the virtual object attention history 1920 is generated on the memory. Then, the HMD ID of the HMD 101 that is the transmission source of the image obtained by capturing the image read in step S1701 is stored in the HMD ID 1921. The imaging date / time of the real image read out in step S1701 is stored in the date / time 1922.

  The model name 1923 stores the model name of the virtual object of interest, and the relative position 1924 and relative orientation 1925 store information on the relative position and relative orientation between the HMD 101 and the virtual object stored in the model name 1923, respectively. .

  Further, information on the point of interest (XYZ coordinates) on the virtual object is stored in the point of interest 1916.

  In step S1713, the CPU 201 of the server 200 determines whether the user is paying attention to the registered image from the line-of-sight direction (step S1713). That is, it is determined whether there is a registered image ahead of the line of sight.

  If it is determined that the registered image is focused, the process proceeds to step S1714. In step S1714, the CPU 201 of the server 200 identifies a point of interest (XYZ coordinates) on the registered image in the three-dimensional space. For the pixel at the center point coordinates of the display, the distance to the current object specified using the depth sensor can be acquired, and the positional relationship with the HMD 101 can be specified.

  The CPU 201 of the server 200 captures the HMD that captured the real image, the imaging date and time, the image ID of the registered image that the user has focused on in the real image, the point of interest on the registered image, the HMD 101 and the registered image at the point of interest. The relative position / relative posture is identified, and a real object attention history 1930 shown in FIG. 19 is generated and stored (step S1714).

  Specifically, the HMD ID of the HMD 101 that has captured the image read out in step S1701 and that is the transmission source of the image is stored in the HMD ID 1931. The imaging date / time of the real image read out in step S1701 is stored in the date / time 1932.

  The object ID 1933 stores the image ID of the registered image of interest, and the object name 1934 stores the name of the registered image. Here, “marker” is stored assuming that the registered image = marker.

  The object image 1935 stores the registered image (image image). The HMD relative position 1936 and the HMD relative attitude 1937 store information on the relative position and relative attitude between the HMD 101 and the registered image stored in the object ID 1933, respectively.

  Further, information on the point of interest (XYZ coordinates) on the virtual object is stored in the point of interest 1938.

  If it is determined in step S1713 that the registered image is not focused, the process proceeds to step S1716. In step S1716, it is determined whether the user is paying attention to a real object other than the registered image. That is, it is determined whether there is a real object other than the registered image ahead of the line-of-sight direction (step S1716).

  When it is determined that a real object other than the registered image is focused, the relative position / posture between the HMD 101 and the real object is specified (step S1717).

  In addition, an object ID is assigned to the real object ahead of the line of sight, and is stored in 1933 of the real object attention history 1930 as a history of the focused real object. Also, the relative position and orientation relative to the HMD are stored in 1936 and 1937.

  An object name 1934 is estimated and stored. In addition, an image of the real object is cut out and stored in the object image 1935. The HMD ID 1931 is an ID of the HMD that captured the real image read in step S1701. The date and time 1932 is the imaging date and time of the real image.

  Further, the largest registered image on the real image is specified (step S1718), and the relative position and orientation relationship between the registered image and the real object is specified (step S1719).

  The relative position and orientation between the registered image and the real object are calculated using information on the relative position and orientation between the registered image and the HMD 101 and information on the relative position and orientation between the real object and the HMD 101.

  The ID of the registered image is stored in the registered image ID 1939, and the relative position (distance in each of the XYZ axes directions) between the registered image and the real object of interest and the value of the relative posture are registered image relative position 1940, registered image. The relative posture 1941 is stored (step S1720).

  When the CPU 201 of the server 200 receives an augmented reality image display processing end instruction, the CPU 201 stores information on the virtual object attention history 1920 and the real object attention history 1930 stored from the start to the end of the augmented reality image display processing. In addition, the log file is output and stored in the external memory 211 (S1721). The above is the description of FIG.

  According to the description of FIG. 17, the relative position and orientation between the HMD and the object (virtual object or real object) can be specified and stored.

  Further, it is possible to specify and store the relative positions and orientations of real objects (registered image and real objects other than registered images).

  The CPU 201 of the server 200 executes the log analysis processing of FIGS. In step S504, one unprocessed record is acquired from the records of the virtual object attention history 1920 and the real object attention history 1930 in the log file.

  In step S509, it is determined whether all records of the virtual object attention history 1920 and the real object attention history 1930 in the log file have been processed. If there are unprocessed records, the process returns to step S504. If all records have been processed, the process proceeds to step S510.

  Next, with reference to FIG. 18, the flow of output processing of analysis results in the third embodiment of the present invention will be described.

  The CPU 201 of the server 200 receives a log analysis result display instruction (for example, a user operation) (step S1801). For example, the selection of the analysis result file stored in the external memory of the server 200 is accepted, and pressing of the analysis result display button on the display screen (not shown) of the server 200 is accepted, whereby the analysis result indicated by the analysis result file is displayed. The display instruction is accepted. The analysis result file here is the ranking information 730 shown in FIG.

  The CPU 201 of the server 200 acquires the latest real image captured and received / stored by the camera of the HMD 101 (step S1802), and determines whether or not a registered image is included in the real image (step S1803). That is, it is determined whether the HMD 101 is capturing a registered image.

  When the registered image is included in the real image, the CPU 201 of the server 200 reads the analysis result file (the file having the information shown in the ranking information 730) selected in step S1801 on the memory (step S1804). Then, one piece of information of the read analysis result is acquired (records of unprocessed ranking information 730 are acquired in descending order of ranking / step S1805).

  The CPU 201 of the server 200 generates a virtual object indicating the acquired analysis result information (step S1806), determines the position and orientation of the virtual object (step S1807), and stores the virtual object in the external memory 211.

  When the model name 732 corresponding to the virtual object indicates the virtual object (that is, in the case of the virtual object indicating the degree of attention of the virtual object), the position and orientation of the virtual object indicating the analysis result information Based on the position and orientation of the virtual object shown, the position and orientation separated by a predetermined distance are determined.

  When the model name 732 indicates a real object (that is, in the case of a virtual object corresponding to a marker or other real object), a position separated by a predetermined distance on the basis of the position and orientation of the real object indicated by the model name 732 And determine the posture.

  When the real object is a marker (registered image), the reference position and orientation of the real object is a relative position and orientation from the HMD 101 that can be specified from the size and angle of the marker in the image captured by the HMD 101. Shall.

  If the real object is a real object other than the marker, the registered image relative posture 1941 is away from the position of the registered image stored in association with the real object in step S1720 by the registered image relative position 1940. The posture shown is the position / posture of the real object.

  Also, as shown in the analysis result model information 740 of FIG. 7, the analysis result model information having the log ID 741 corresponding to the analysis result selected in step S1801 is generated.

  Then, the file name of the virtual object indicating the generated analysis result is inserted into the analysis result model name 742, and the model name of the analysis target focused by the user, the image ID of the registered image, or the object of the real object is inserted into the model name 745 The ID is inserted, and based on the position / posture of the virtual object or the real object of the model name 745, the position / posture predetermined position / posture value is inserted into the position 743, the posture 744 and stored (step S1807). .

  The CPU 201 of the server 200 determines whether or not the processing in steps S1805 to S1807 has been executed for all analysis results (all models in the ranking information 730) in the file acquired in step S1804 (step S1808).

  If not executed, the process returns to step S1804 to acquire an unprocessed analysis result. If it has been executed, the process moves to step S1809.

  The CPU 201 of the server 200 executes the processing described in steps S1704 to S1707 in FIG. 17 for the registration included in the latest image read in step S1802, thereby performing the AR image of the virtual object associated with the registered image. (Augmented reality image / superimposed image) is generated and output / displayed on the HMD 101. That is, identification display processing of the target object is performed. The above is the description of FIG.

  Further, in step S1007 in FIG. 10, a record of the position and orientation of the HMD 101 being focused on the virtual object selected in step S1001 is acquired from the log file corresponding to the analysis result object being displayed. That is, information on the relative position and orientation of the selected virtual object and the HMD 101 is acquired.

  Then, for each date and time 1922, a virtual object (1202 in FIG. 12) indicating the position of the HMD at the time when the user focused on the virtual object is generated with respect to the position and orientation of the selected virtual object.

  Specifically, the position and orientation of the analysis result object are separated by a predetermined distance on the basis of the position and orientation of the virtual object (the positions indicated by the relative position 1924 and the relative orientation 1925 when using the position and orientation of the registered image as a reference). The determined position and orientation are determined and stored (step S1008).

  Further, in step S1001 in FIG. 10, by accepting selection of the analysis result display virtual object displayed in association with the real object, the selection button for the real object is displayed in step S1003. In step S1007, an analysis result object corresponding to the real object is generated for each date 1932 of the real object attention history 1930.

  Then, the position and orientation of each analysis result display object are determined as the position and orientation of the HMD 101 for each date 1932. The position and orientation of the HMD 101 are specified from the HMD relative position and the HMD relative orientation corresponding to the real object in each record of the real object attention history 1930.

  Note that, as indicated by 1210 in FIG. 12, a straight line (arrow) indicating the line-of-sight direction of the HMD when the user focuses on the virtual object or the real object, which is specified from the HMD relative position and the HMD relative posture. A virtual object may be generated and positioned to be placed in the virtual space.

  As described above, according to the third embodiment of the present invention, it is determined whether the experience person has focused on the object in a state where the absolute position and orientation of the HMD in the three-dimensional space cannot be specified. A mechanism that can be easily identified above can be provided.

  Note that the above embodiment is an example according to the present invention. For example, in order to detect the position and orientation of the HMD 101, in the above embodiment, it is assumed that there are three position detection markers in the field of view imaged, and a large number of position detection markers are attached in the room. However, the present invention is not limited by these. The position detection marker has a mark that defines the direction. If not only the coordinates of the position detection marker but also the shape and dimensions are known, the position detection distortion, size, and position of the mark are 1 The position and orientation of the HMD 101 can be specified from only one position detection marker.

  Further, the method for specifying the position and orientation of the real object is not necessarily limited to the method described above. For example, as with the model information 720, the object ID, the position and orientation, the focus flag, and the focus area are associated with each other and stored in advance in the external memory as real object information. When the object ID is in the line-of-sight direction of the HMD 101, that is, when the vector indicated by the line-of-sight direction intersects the area of interest, the CPU 201 of the server 200 has a real object corresponding to the area of interest ahead of the line-of-sight direction. And the real object can be specified as the real object of interest.

  In addition, when two or more registered images having the same posture are detected from the captured real images, the planes that connect the vertices at the farthest positions of the two registered images with line segments and use them as diagonal lines are actually It may be recognized as an object. For example, it is assumed that a marker is pasted on the corner of a wall or desk.

  In this case, the center point of the surface is determined as the position of the real object indicated by the plane. The posture of the real object indicated by the plane is assumed to be the posture of the registered image.

  Also, in the above embodiment, the focus ranking is determined based on the length of time of focus. For example, the number of times of focus is stored for each target object (virtual object and real object), and the number of times of focus is increased. The degree of attention may be ranked higher. The number of times of attention is counted and stored as having been noticed once, from when the user starts paying attention to an object until it stops paying attention to that object.

  The HMD 101 may be a see-through type device. In the case of the see-through type, since the display is translucent and there is no need to display a real image, for example, a real image is assumed to have a transmittance of 100%, and a virtual object having a transmittance of 0% is placed on a real image having a transmittance of 100%. The superimposed image on which the image is superimposed is transmitted to the HMD 101 for display.

  In the above embodiment, the object positioned at the center of the HMD display = the object of interest (that is, the object ahead of the direction in which the HMD 101 is directed is regarded as the object of interest). The method for identifying the object is not limited to this.

  For example, as described in JP 2010-199789 A or JP 2006-127487 A, a non-visible light irradiation device or a half mirror that irradiates the HMD with invisible light toward the user's eyes is provided. Install a camera that captures the invisible light reflected by the user's eyes and capture the reflected light to see where the user's eyes are on the display (on the real image displayed on the display screen) A known technique for identifying whether or not Thereby, what the user of the HMD 101 is paying attention to on the screen can be specified.

  For example, as indicated by 2100 to 2120 in FIG. 21, the cursor 1611 can be moved according to where the user's eyes are looking on the display. That is, the position of the cursor 1611 = the point of interest on the user's image. Reference numeral 2101 denotes an area where a virtual object is displayed / displayed. Reference numeral 2120 denotes a state in which the user is paying attention to the virtual object.

  In this case, it is assumed that the line-of-sight direction = the direction from the HMD 101 to the point of interest in the three-dimensional space of the object that the user is paying attention to. Note that in the three-dimensional space, which position on the outer surface of the real object (the point indicated by the XYZ coordinates) the user is paying attention to is the size, orientation, shape, and shape of the object on the image displayed on the display. The calculation can be performed by using a point (XY coordinates on the image) on which the user focuses on the object and the reference size, orientation, and shape of the object stored in the external memory in advance.

  Further, the analysis result virtual object 742 is not limited to the shape and display form indicated by 810 in FIG. For example, as shown in FIG. 20, a virtual object (analysis result object) as shown in 2002 is generated for each target point focused on the target object 2001, and the position of the virtual object is set to the position of each target point. It may be determined.

  The user can confirm that an object on which more virtual objects shown in 2002 are arranged is an object that has attracted much attention. That is, the degree of attention can be presented and confirmed by the number of virtual objects shown in 2002.

  Further, the points of interest are classified into a plurality of clusters by clustering (grouped into a plurality of groups), and, as indicated by 2010, according to the number of points of interest in the cluster, for example, a cluster is displayed to be displayed like a heat map. The color of the point of interest may be changed. The user can confirm an object that has been focused more on the basis of the color of a virtual object such as 2111. That is, the degree of attention of each object can be presented and confirmed.

  Further, the position and orientation of the virtual object indicating the position and orientation of the HMD 101 determined in step S1008 in FIG. 10 are clustered, and a virtual object indicating the cluster including the position in the cluster is generated, You may make it arrange | position to the center position of a cluster. Further, the color of the point of interest indicated by the cluster may be changed according to the number of HMD positions in the cluster. As a result, the user can easily confirm from which position the object of interest is often viewed.

  The clustering of the position of the HMD 101 (virtual object indicating the position and orientation of the HMD 101) is performed when the analysis results generated by reading the log files of the plurality of HMDs 101 in the process of FIG. Since the (position of HMD101) is grouped, a more accurate result can be confirmed.

  In the above-described embodiment, the analysis result object 1202 has a spherical shape, but a virtual object having the same shape as the HMD 101 may be used instead of the 1202 in order to make it easy to confirm the rotation state of the HMD 101 when the object is focused. .

  Although MR and AR have been described in the above embodiment, it goes without saying that the present invention is applicable in the field of VR in which the position and orientation of the camera in the virtual space can be changed in conjunction with the movement of the HMD 101. When the present invention is implemented with VR, the degree of attention to a virtual object is specified and displayed.

  For example, the analysis result information can be displayed as text information on a display connected to the server 200 by a predetermined application, and can be browsed by the user.

  In the above embodiment, the case of displaying an object such as a booth has been taken as an example. However, in the design verification of a car, for example, in the interior and exterior of a newly designed car, the user pays attention to where, how much and from where. It can also be used for verification. Rather than presenting in the coordinate list of the point of interest or the coordinate list of the component of interest the object that has a complex shape with many components (objects) combined, If the attention level is displayed in association with the focused object in space, the user can easily check the attention level for each object.

  Note that the HMD 101 and the server 200 may be integrated, and a case in which the HMD 101 and the server 200 are integrated may be configured to implement the present invention by performing all the processes in each flowchart.

  By supplying a recording medium recording a program for realizing the functions of the above-described embodiments to a system or apparatus, and reading and executing the program stored in the recording medium by a computer (or CPU or MPU) of the system or apparatus However, it goes without saying that the object of the present invention is achieved.

  In this case, the program itself read from the recording medium realizes the novel function of the present invention, and the recording medium storing the program constitutes the present invention.

  As a recording medium for supplying the program, for example, a flexible disk, hard disk, optical disk, magneto-optical disk, CD-ROM, CD-R, DVD-ROM, magnetic tape, nonvolatile memory card, ROM, EEPROM A silicon disk, a solid state drive, or the like can be used.

  Further, by executing the program read by the computer, not only the functions of the above-described embodiments are realized, but also an OS (operating system) operating on the computer based on an instruction of the program is actually It goes without saying that a case where the function of the above-described embodiment is realized by performing part or all of the processing and the processing is included.

  Furthermore, after the program read from the recording medium is written to the memory provided in the function expansion board inserted into the computer or the function expansion unit connected to the computer, the function expansion board is based on the instructions of the program code. It goes without saying that the case where the CPU or the like provided in the function expansion unit performs part or all of the actual processing and the functions of the above-described embodiments are realized by the processing.

  Furthermore, by downloading and reading a program for achieving the present invention from a server, database, etc. on a network using a communication program, the system or apparatus can enjoy the effects of the present invention. In addition, all the structures which combined each embodiment mentioned above and its modification are also included in this invention.

101A HMD
101B HMD
101C HMD
102 optical sensor 150 network 200 server

Claims (17)

User can communicate with the display device to be mounted, a information processing apparatus including an acquisition unit that acquires information of the line of sight of the user the display device is mounted, and a storage means for storing the objects of the three-dimensional space, the And
Using the information of the line of sight, on the three-dimensional space, and focuses the object specifying means for specifying an object the user has focused,
A degree of attention specifying unit that is pre-SL user identified the interest degree indicating whether attention is paid how much they were objects of interest, identified based on the transition information of the sight line by the focusing object specifying means,
And output means for an image showing the degree of attention identified, and outputs to be displayed on the position corresponding to the body material which has been focused by the focusing degree specifying means,
An information processing apparatus comprising: The information processing apparatus is an information processing apparatus that further stores a position and orientation of the display device in a three-dimensional space and a position of an object in the three-dimensional space,
The object-of-interest specifying means specifies an object that the user has focused on from the position of the display device in a three-dimensional space, the user's line of sight, and the position of the object in the three-dimensional space. The information processing apparatus according to 1. In order to display the attention level in association with the object that the user has focused on, the position of the attention level display corresponds to each of the focused objects specified by the object specifying unit. Determining means for determining the position;
Image generating means for generating an image for actually superimposing the display of the degree of interest according to the current position and orientation of the display device and the position of the degree of interest displayed determined by the determining means;
With
3. The information processing apparatus according to claim 1, wherein the output unit outputs the image generated by the image generation unit including the display of the degree of interest so as to be displayed on the display device. A degree-of-interest object generating means for generating a degree-of-interest object that is a virtual object indicating the degree of attention;
With
The determining means determines the position of the attention level object generated by the attention level object generating means to be a position corresponding to each of the objects that have been specified, specified by the target object specifying means,
Said image generating means, depending on the current position and orientation and the position of the degree of attention object of the display device, according to claim 3, characterized in that for generating an image to be displayed superimposed image of the degree of attention object actually The information processing apparatus described in 1. The degree-of-interest specifying means specifies the length of time that the user has focused on the object identified by the degree-of-focus identifying means and / or the number of times that the user has focused.
The information processing apparatus according to claim 1, wherein the output unit outputs an image representing the length of time and / or the number of times. The degree-of- interest specifying means specifies the length of time that the user has focused on the object identified by the degree-of-interest identifying means and / or the ranking of the number of times the user has focused on the object ,
And the output means, before the information processing apparatus according to any one of claims 1 to 5, characterized in that outputs an image indicating the Kira trunking. And the output means, depending on the ranking, the information processing apparatus according to claim 6, characterized in that the output in order to display an image that identifies displaying the object. A history storage unit that is the user specified for each object that has been focused, to memorize the history of the position of the display device of the user that has been focused on the object by the focused object specifying means,
When receiving the designation of the object, based on the Ki履 history before being memorize the history storage unit, a virtual object that indicates the position of the display device while it was focused on the object which has received the designation Position object generating means for generating a position object;
Between the position of the three-dimensional space of the position object generated by the position object generation means, a position on the three-dimensional space indicated by the respective position objects, which has been focused on the object which has received the designation Position determining means for determining the position of the display device;
The information processing apparatus according to any one of claims 1 to 7, further comprising: Posture correlation storage that stores the history of the posture of the display device of the user who has focused on the object in association with each other so that the user identified by the target object specifying means can be identified for each object Means,
When the designation of the object is received, the display device while paying attention to the object that has received the designation based on the history of the posture of the object and the display device stored by the posture association storage unit Posture object generation means for generating a posture object which is a virtual object indicating the posture of
Position / orientation determining means for determining the position / orientation of the attitude object generated by the attitude object generating means in the three-dimensional space to be the position / orientation of the display device while paying attention to the specified object. When,
The information processing apparatus according to any one of claims 1 to 8, characterized in that it comprises a. Display control means for displaying and controlling an operation unit for replacing the object that has received the designation with the position of another object identified by the object-of-interest identifying means when the designation of the object is accepted. The information processing apparatus according to any one of claims 1 to 9 , wherein the information processing apparatus is characterized in that: And the target object storage means for storing the target object specified as an object which the user has focused by the focusing object specifying means,
With
The noted object identifying means, wherein among the objects stored have that Target in the object storage means, any one of claims 1 to 10, wherein the identifying the object to the user has focused The information processing apparatus described in 1. The degree-of-interest specifying means uses the information on the line-of-sight transition of the plurality of users of the plurality of display devices, and how much the object that the plurality of users have focused on is noted by the plurality of users The information processing apparatus according to any one of claims 1 to 11, wherein a degree of attention indicating ??? is specified. User can communicate with the display device to be mounted, the control of the information processing apparatus including an acquisition unit that acquires information of the line of sight of the user the display device is mounted, and a storage means for storing the objects of the three-dimensional space, the A method,
A target object specifying step of specifying an object on which the user has focused on the three-dimensional space using the information on the line of sight ;
The degree of attention indicating whether the noted object before SL user identified by a particular process has been focused degree have been object of interest, the degree of attention specifying step of specifying, based on the transition information of the line of sight,
An image showing the degree of attention identified by the degree of attention specifying step, and an output step of outputting to be displayed on the position corresponding to the body thing that has been focused,
That control how to be characterized in that it comprises a. The user can communicate with the display device to be mounted, control and acquisition means for acquiring information of the user's line of sight with the display device is mounted, and a storage means for storing the objects of the three-dimensional space, the information processing apparatus including a A program for
The information processing apparatus;
Using the information of the line of sight, on the three-dimensional space, and focuses the object specifying means for specifying an object the user has focused,
A degree of attention specifying unit that is pre-SL user identified the interest degree indicating whether attention is paid how much they were objects of interest, identified based on the transition information of the sight line by the focusing object specifying means,
Program for operating an image showing the degree of attention identified by the degree of attention specifying means, an output means for outputting to be displayed on the position corresponding to the body thing that has been focused. An information processing system including a display device worn by the user, an acquisition unit configured to acquire information of the line of sight of the user who wears the display device, storage means for storing the objects of the three-dimensional space, the information processing apparatus including a Because
Using the information of the line of sight, on the three-dimensional space, and focuses the object specifying means for specifying an object the user has focused,
A degree of attention specifying unit that is pre-SL user identified the interest degree indicating whether attention is paid how much they were objects of interest, identified based on the transition information of the sight line by the focusing object specifying means,
And output means for an image showing the degree of attention identified, and outputs to be displayed on the position corresponding to the body material which has been focused by the focusing degree specifying means,
An information processing system comprising: An information processing system including a display device worn by the user, an acquisition unit configured to acquire information of the line of sight of the user who wears the display device, storage means for storing the objects of the three-dimensional space, the information processing apparatus including a Control method,
A target object specifying step of specifying an object on which the user has focused on the three-dimensional space using the information on the line of sight ;
The degree of attention indicating whether the noted object before SL user identified by a particular process has been focused degree have been object of interest, the degree of attention specifying step of specifying, based on the transition information of the line of sight,
An image showing the degree of attention identified by the degree of attention specifying step, and an output step of outputting to be displayed on the position corresponding to the body thing that has been focused,
That control how to be characterized in that it comprises a. An information processing system including a display device worn by the user, an acquisition unit configured to acquire information of the line of sight of the user who wears the display device, storage means for storing the objects of the three-dimensional space, the information processing apparatus including a A program for controlling
The information processing system;
Using the information of the line of sight, on the three-dimensional space, and focuses the object specifying means for specifying an object the user has focused,
A degree of attention specifying unit that is pre-SL user identified the interest degree indicating whether attention is paid how much they were objects of interest, identified based on the transition information of the sight line by the focusing object specifying means,
Program for operating an image showing the degree of attention identified by the degree of attention specifying means, an output means for outputting to be displayed on the position corresponding to the body thing that has been focused.

Images