JP2016035742A - Information processing system, control method of the same, and program, and information processing apparatus, control method of the same, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To specify a desired position on a first virtual space with a line of sight from a first head-mounted display so as to display the position on a second head-mounted display so that the position can be identified.SOLUTION: There is provided an information processing system comprising a plurality of information processing apparatuses communicably connected to head-mounted displays that each include display means for displaying an image of a virtual space having information on a virtual object in common among the plurality of information processing apparatuses. The information processing system acquires positions and directions of the head-mounted displays in the real space and specifies positions in the virtual space on the basis of the position and direction information obtained. It then implements control so as to display arrows from the head-mounted displays to the specified positions.SELECTED DRAWING: Figure 11

Description

  The present invention relates to mixed reality technology, and in particular, a desired position in a first virtual space can be identified by a second head-mounted display by specifying the line of sight of the first head-mounted display. The present invention relates to an information processing system that can be displayed on a computer, a control method and program thereof, and an information processing device, and a control method and program thereof.

  2. Description of the Related Art Conventionally, there is a technique for superimposing a real space image obtained by photographing with a camera and a virtual space image (virtual object) generated by a computer and displaying them on a head mounted display (HMD) or a smartphone. For example, there are mixed reality (hereinafter referred to as MR) technology and augmented reality (hereinafter referred to as AR) technology.

  In particular, MR technology superimposes the real space image and the virtual space image on the HMD worn by the user and displays them in real time, so that it appears as if the virtual space image is displayed in the real space. The presence is very high. For this reason, various utilization using MR technology is considered.

  On the other hand, if the virtual object that the user is looking through the HMD is wearing the HMD, a plurality of people can see (share) the virtual object at each position. And while sharing, for example, when one user gives an instruction to move the position by 1 m with respect to the virtual object, control is performed so that the virtual object viewed by the other user is also displayed as if moved by 1 m. be able to.

  In Patent Document 1 below, when a position to which explanation information is given is specified for a free viewpoint image composed of a plurality of image data with different viewpoints, an annotation is given to the specified location, and a new free viewpoint image is created. A mechanism is disclosed in which the explanation information is displayed at a designated location even when switching.

JP 2012-170989 A

  In addition, MR space is shared between remote locations such as between Tokyo (point A) and Osaka (point B). As a result, the arrangement of the same virtual object can be shared even in a remote place. For example, a conference or the like can be performed while sharing a showroom made up of virtual objects between remote places.

  Even in such a situation, when the virtual object is moved at the point A, the virtual object is controlled to be displayed at the point B so as to move in synchronization with the point A.

  At this time, only the coordinate information of the virtual object is shared between the remote locations, and the image data of the real space is not shared. Therefore, the user of the B point wants to move the virtual object with the finger etc. Even if an instruction is given to the user, the user at the point A cannot see the user at the point B, so that the position designated by the user at the point B is not known.

  Therefore, a method of designating a position by giving an annotation using the method of the patent document described above can be considered. In other words, it is necessary for the user at point B to add an annotation to the position where the virtual object is to be moved. It is necessary to perform an operation of removing the part and designating a position in the virtual space on the PC screen. Although it is conceivable to operate the PC with the HMD attached, since the field of view of the HMD is not wide, the operation of the PC is difficult.

  It is an object of the present invention to display a desired position in the first virtual space so as to be identifiable on the second head mounted display by specifying the line of sight of the first head mounted display.

  In order to achieve the above object, an information processing system of the present invention manages a virtual space of a first head mounted display and a virtual space of a second head mounted display, which are common virtual spaces. An acquisition unit that acquires line-of-sight information, which is information indicating the line of sight of the first head-mounted display in real space, and a virtual of the first head-mounted display based on the line-of-sight information acquired by the acquisition unit. Specifying means for specifying position coordinates in space; and display control means for displaying the position coordinates corresponding to the position coordinates specified by the specifying means in an identifiable manner on the virtual space of the second head mounted display. It is characterized by providing.

  According to the present invention, by specifying the desired position in the first virtual space by the line of sight of the first head mounted display, it is possible to display the second head mounted display in an identifiable manner. Play.

It is a figure which shows an example of the system configuration | structure of the information processing system in embodiment of this invention. 2 is a diagram illustrating an example of a hardware configuration of an information processing apparatus 200. FIG. 2 is a diagram illustrating an example of a functional configuration of an information processing device 200. FIG. It is a flowchart which shows an example of the flow of a series of processes in embodiment of this invention. It is a flowchart which shows the detail of the designated point model display process shown in FIG. 5 is a flowchart showing details of a virtual object movement process shown in FIG. 4. It is a flowchart which shows the detail of the virtual object movement process to the designated point model shown in FIG. 5 is a configuration diagram showing an example of an HD information table 800 that stores coordinate information in the virtual space of the HMD 100 and vector information. FIG. It is a block diagram which shows an example of the object table 900 which stores the various data regarding the virtual object in virtual space. 3 is a schematic diagram illustrating an example of a selection device 105. FIG. A designated point model image 1100 which is an example of display of a designated point model 1102 created by the information processing apparatus 200-A and a designated point model 1104 which can be displayed by the HMD 100-B connected to the information processing apparatus 200-B. It is a schematic diagram shown. It is a schematic diagram which shows the distance display image 1200 which is an example of the distance display to the designated point model displayed by HMD100-B connected with information processing apparatus 200-B. It is a schematic diagram which shows the arrow display image 1300 which is an example of the arrow display to the designated point model displayed by HMD100-B connected with information processing apparatus 200-B. It is a mimetic diagram showing virtual object movement image 1400 which is an example of an image which moves a virtual object to the position of a specified point model in MR space created by information processor 200-B. FIG. 10 is a schematic diagram showing an example of a designated point model display instruction receiving screen 1500 showing an example of a dialog for receiving an instruction as to whether or not to display a designated point model based on a line of sight from a user. 5 is a schematic diagram showing an example of a threshold table 1600 stored in an external memory 211. FIG. It is a flowchart which shows the detail of the designated point model display process in the 2nd Embodiment of this invention. 5 is a diagram showing an example of an identification display image 1800. FIG. It is a block diagram which shows an example of the object table 900 in the 2nd Embodiment of this invention.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  FIG. 1 illustrates an information processing system according to the present invention, which includes an information processing apparatus 200, an HMD 100 (head mounted display), an optical sensor 104, a selection device 105, and the like, and includes a plurality of information processing systems having the same configuration. It is. The information processing apparatus 200, the HMD 100, the optical sensor 104, and the selection device 105 are connected to each other so as to be able to perform data communication with each other via the LAN 101 shown in FIG. 1 or a USB (Universal Serial Bus). In this embodiment, the information processing apparatus 200 is installed at each of the bases A and B. However, a plurality of information processing apparatuses 200 may be installed at one base A. It is also possible to implement the present invention by installing one information processing apparatus 200 at the site A, connecting a plurality of HMDs 100, and building each MR space. When implemented with one information processing apparatus 200, the information processing apparatus 200 needs to include a plurality of graphic boards.

  The information processing apparatus 200 is a virtual object that is arranged and displayed in the MR space in the external memory 211, and stores virtual objects that are common to a plurality of information processing apparatuses 200. The information processing apparatus 200 generates a mixed reality image (MR image) obtained by superimposing a virtual object on the real image received from the HMD 100, and transmits the mixed reality image to the HMD. Details of the mechanism of the MR technique are as described above. The HMD may be a head-mounted type or a glasses type.

  The information processing apparatus 200 includes a secondary information processing apparatus 200-A and a primary information processing apparatus 200-B. The information processing apparatus 200-B receives the change in the position coordinates of the virtual object received from the information processing apparatus 200-A (for example, a movement instruction of the virtual object based on the selection device 105), and sends the position to the other information processing apparatus 200. Plays a role in notifying changes in coordinates. Thereby, even if MR space is shared between remote locations with different locations, for example, the sharing user can visually recognize an image in which a common virtual space and each real space are superimposed through the HMD 100. It becomes possible. The functions of the information processing device 200-A and the information processing device 200-B are assumed to be the same. In this embodiment, the information processing device 200-B is primary for convenience, but both information processing devices 200 have a primary role. Is possible. Therefore, the respective uses of the information processing device 200-A and the information processing device 200-B are not limited.

  The HMD 100 transmits image data in the real space obtained by photographing with the right-eye / left-eye video camera 221 to the information processing apparatus 200. The mixed reality image received from the information processing apparatus 200 is displayed on the right eye / left eye display 222.

  The selection device 105 is a device for selecting and moving a virtual object arranged in the MR space. For example, the selection device 105 shown in FIG. The selection device 105 includes a selection button 1001 and an optical marker 103. When the selection button 1001 is pressed, the information processing apparatus 200 synchronizes the position coordinates of the virtual object within a certain range (within the range 1002 shown in FIG. 10) from the position of the selection device 105 with the position coordinates of the selection device 105. . Thus, the user can move a virtual object at a certain distance of the selection device 105 by pressing the selection button 1001. When it is detected that pressing of the selection button 1001 is not detected, the synchronization of the position coordinates of the virtual object and the selection device 105 is stopped.

  An optical marker 103 is installed in each of the HMD 100 and the selection device 105, and the optical sensor 104 captures (detects / detects) the optical marker 103, whereby the orientation (coordinates) of the HMD 100 and the selection device 105. X, Y, Z) can be specified and transmitted to the information processing apparatus 200.

  Note that the alignment (calibration) between the real space and the MR space (virtual space in which the virtual object is arranged with the origin coordinates X, Y, Z = 0, 0, 0 as the origin) has already been performed. Hereinafter, embodiments of the present invention will be described.

  FIG. 2 is a diagram illustrating a hardware configuration of the information processing apparatus 200. Note that the hardware configuration of the information processing apparatus 200 in FIG. 2 is merely an example, and there are various configuration examples depending on applications and purposes.

  The CPU 201 comprehensively controls each device and controller connected to the system bus 204.

  The ROM 202 or the external memory 211 stores a BIOS (Basic Input / Output System) that is a control program of the CPU 201, an operating system, and various programs described later that are necessary for realizing the functions executed by various devices. Yes. The RAM 203 functions as a main memory, work area, and the like for the CPU 201.

  The CPU 201 implements various operations by loading a program necessary for execution of processing into the RAM 203 and executing the program.

  An input controller (input C) 205 controls input from a pointing device (input device 209) such as a keyboard and a mouse.

  A video controller (VC) 206 controls display on a display device such as the right eye / left eye display 222 provided in the HMD 100. For example, an external output terminal (for example, Digital Visual Interface) is used for output to the right eye / left eye display 222. The right-eye / left-eye display 222 includes a right-eye display and a left-eye display.

  The memory controller (MC) 207 is an adapter to a hard disk (HD), flexible disk (FD) or PCMCIA card slot for storing boot programs, browser software, various applications, font data, user files, editing files, various data, and the like. Controls access to an external memory 211 such as a card-type memory connected via the.

  A communication I / F controller (communication I / FC) 208 is connected to and communicates with an external device via a network, and executes communication control processing in the network. For example, Internet communication using TCP / IP is possible. The communication I / F controller 208 also controls communication with the optical sensor 104 through Gigabit Ethernet (registered trademark) or the like.

  The general-purpose bus 212 is used for capturing video from the right-eye / left-eye video camera 221 of the HMD 100. Input from the right-eye / left-eye video camera 221 is performed using an external input terminal (for example, an IEEE 1394 terminal). The right-eye / left-eye video camera 221 includes a right-eye video camera and a left-eye video camera.

  Note that the CPU 201 enables display on a display by executing outline font rasterization processing on a display information area in the RAM 203, for example. In addition, the CPU 201 allows a user instruction with a mouse cursor (not shown) on the display.

  Various programs used by the information processing apparatus 200 of the present invention to execute various processes described later are recorded in the external memory 211, and are executed by the CPU 201 by being loaded into the RAM 203 as necessary. is there. Furthermore, definition files and various information tables used by the program according to the present invention are stored in the external memory 211.

  Next, the functional configuration of the information processing apparatus 200 will be described with reference to FIG.

  The information processing apparatus 200 includes a communication control unit 301, a virtual object storage unit 302, an HMD information acquisition unit 303, a real space image acquisition unit 304, an HMD line-of-sight specification unit 305, a distance specification unit 306, a mixed reality image generation unit 307, a virtual space A management unit 308, a display control unit 309, and an HMD view determination unit 310 are provided.

  The communication control unit 301 is a functional unit that can control communication with the HMD 100, the optical sensor 104, and the other information processing apparatus 200 that are communicably connected. The communication control unit 301 transmits / receives information to / from these apparatuses through the video controller 206, the communication I / F controller 208, the general-purpose bus 212, and the like described above.

  The virtual object storage unit 302 is a functional unit that stores a virtual object to be superimposed on a real image by the HMD 100.

  The HMD information acquisition unit 303 is a functional unit that acquires information indicating the position and orientation of the HMD 100 in the real space from the optical sensor 104. The real space image acquisition unit 304 is a functional unit that acquires a real space image from the right-eye / left-eye video camera 221 of the HMD 100.

  The HMD line-of-sight identifying unit 305 is a functional unit that identifies the line of sight of the user wearing the HMD 100 from the position and orientation of the HMD 100 acquired in the HMD information acquisition unit 303 in the real space.

  The distance specifying unit 306 is a functional unit that specifies by calculating the distance between the HMD 100 and the specified point model, or calculates the distance between the virtual object that is the movement target and the specified point model. Note that the distance is calculated by using a conventional technique for calculating using the coordinates between two points in the virtual space.

  The mixed reality image generation unit 307 generates an image in which a real space image transmitted from the right-eye / left-eye video camera 221 of the HMD 100 and a virtual space image based on the position and orientation of the HMD 100 acquired from the HMD information acquisition unit 303 are superimposed. It is a functional part to do.

  The virtual space management unit 308 is a functional unit that manages coordinate information of a virtual object in the virtual space. If the information processing device 200 is the primary information processing device 200, the communication control unit 301 is used to transmit the coordinate information of the common virtual object transmitted from the secondary information processing device 200 to the other secondary information processing devices 200. Use it to control it. If the information processing apparatus 200 is a secondary information processing apparatus 200, when a virtual object movement instruction is detected from a user, coordinate information of the virtual object based on the movement instruction is transmitted to the primary information processing apparatus 200.

  The display control unit 309 is a functional unit that controls display on the right-eye / left-eye display 222 of the HMD 100.

  Next, a series of processes performed by the information processing apparatus 200-B that is the primary machine and the information processing apparatus 200-A that is the secondary machine in the embodiment of the present invention will be described with reference to the flowchart shown in FIG.

  In step S401, the CPU 201 of the information processing apparatus 200-A activates the MR application by receiving an MR application activation instruction from the user. After starting the MR application, the process proceeds to step S402.

  In step S402, the CPU 201 of the information processing device 200-A transmits a connection request to the information processing device 200-A that is another information processing device.

  In step S403, the CPU 201 of the information processing apparatus 200-B receives the connection request transmitted from the information processing apparatus 200-A in step S402.

  In step S404, the CPU 201 of the information processing device 200-B transmits all of the virtual object data to the information processing device 200-A that is the transmission source of the connection request. For example, the object table 900 shown in FIG. 9 and the file shown in the file path 905 are transmitted. This makes it possible to perform real-time display control of the virtual object only by exchanging coordinate information of the virtual object.

  The object table 900 shown in FIG. 9 stores an object ID 901, an attribute 902, coordinates 903, a size 904, a file path 905, a selection flag 906, and a selection prohibition flag 907.

  The object ID 901 is identification information of data of each virtual object. The attribute 902 indicates whether each virtual object is a virtual object corresponding to the selected device 105 (selected device), a virtual object to be selected (contacted) selected by touching the virtual object of the selected device 105, an HMD This is information indicating whether the designated point is set at the intersection of the line of sight and the virtual object.

  A coordinate 903 indicates an arrangement position of each virtual object in the MR space (XYZ coordinates with the origin at XYZ = 0.000.000.000). According to FIG. 9, the coordinates 903 of the object ID = M111 = XX1. YY1. Because of ZZ1, the virtual object with the object ID = M111 is arranged at the coordinates indicated by the coordinates in the MR space. The origin is set at the time of the calibration described above. The coordinate value of each coordinate 903 is the coordinate of the center point of each virtual object.

  Note that the coordinates 903 of the selection device 105 are supplemented by an optical sensor. For example, when the user moves the selection device 105, the coordinates 903 are updated in real time. Information on real-time coordinates 903 of the selection device 105 used for the update is transmitted from the optical sensor 104 to the information processing apparatus 200, and information on coordinates received by the information processing apparatus 200 from the optical sensor 104 is used as the selection device. Updating is performed by overwriting the coordinates 903 of 105.

  A size (scale) 904 indicates the size of each virtual object. A file path 905 indicates an address on a storage area in which a file of each virtual object is stored.

  The selection flag 906 is a flag indicating whether or not the selected virtual object is selected by the selection device 105. When the selection flag 906 = 0, the selection is not performed. When the selection flag 906 = 1, the selection is in progress.

  The selection prohibition flag 907 is a flag for controlling the selected virtual object so as not to accept selection by the selection device 105. When the selection prohibition flag 907 = 0, the selection of the selection device 105 can be accepted. When the selection prohibition flag 907 = 1, the selection device 105 is controlled not to accept the selection. Accordingly, even if the selection device 105 performs a selection action, there is an effect that it is impossible to move the selected virtual object.

  In step S405, the CPU 201 of the information processing apparatus 200-B permits connection to the information processing apparatus 200-A that is the transmission source of the connection request received in step S403. As a result, the information processing apparatus 200-A and the information processing apparatus 200-B display a virtual object that is common to the information processing apparatus 200-B, and the information processing apparatus 200-B is in real time even if an instruction to move the virtual object is given by the selection device 105. The movement of the virtual object can be displayed by the information processing apparatus-A.

  In step S406, the CPU 201 of the information processing device 200-A receives the object table 900 transmitted from the information processing device 200-B in step S404 and the file indicated by the file path 905.

  In step S407, the CPU 201 of the information processing device 200-A detects the optical marker attached to the HMD 100-A using the optical sensor 104, and the position and orientation (position information and position information) of the HMD 100-A in the MR space. (Direction information) is acquired (acquisition means). The acquired position and orientation of the HMD 100-A are stored in the HMD information table 800. For example, the HMD information table 800 shown in FIG. The HMD information table 800 stores the HMD position / direction 801. The HMD position / orientation 801 stores coordinate information in the MR space of the HMD and orientation information (orientation). The direction information (orientation) stores information indicating the same direction as the user's line-of-sight direction (line-of-sight information).

  In step S408, the CPU 201 of the information processing apparatus 200-A acquires the real space image transmitted from the right-eye / left-eye video camera 221 of the HMD 100-A and stores it in the RAM 203 or the like. Since there are two right-eye / left-eye video cameras 221, that is, a video camera corresponding to the user's right eye and a video camera corresponding to the left eye, the real-eye images for the right eye and the left eye are acquired.

  In step S409, the CPU 201 of the information processing apparatus 200-A acquires a virtual space image (virtual space image) and stores it in the RAM 203 or the like. More specifically, the position / orientation 801 of the HMD 100-A stored in step S407 is read, and the position and orientation of the HMD 100-A in the virtual space corresponding to the HMD position / orientation are specified. Then, a virtual camera (viewpoint) is installed on the virtual space at the specified position and orientation on the virtual space, and the virtual space is imaged by the camera. Thereby, a virtual space image is generated. Similar to the above-described real space image, two images of the right eye virtual space image and the left eye virtual space image are acquired for display on the right eye / left eye display 222 of the HMD 100-A.

  In step S410, the CPU 201 of the information processing apparatus 200-A reads the real space image acquired in step S408 and the virtual space image acquired in step S409 from the RAM 203 or the like, and uses the mixed reality image generation unit 307 to read the real space. The virtual space image is superimposed on the image to generate a mixed reality image. The generated mixed reality image is stored in the RAM 203 or the like. As described above, two images for the right eye and for the left eye are stored in the RAM 203 or the like in the real space image and the virtual space image, respectively. Then, by superimposing the right-eye virtual space image on the right-eye real space image and superposing the left-eye virtual space image on the left-eye real space image, the user can obtain a more immersive vision. .

  In step S411, the CPU 201 of the information processing apparatus 200-A reads the right-eye mixed reality image and the left-eye mixed reality image generated in step S410 from the RAM 203 or the like and displays them on the right-eye / left-eye display 222 of the HMD 100-A. To control. As described above, the right-eye mixed reality image is controlled to be displayed on the right-eye display of the right-eye / left-eye display 222, and the left-eye mixed reality image is controlled to be displayed on the left-eye display of the right-eye / left-eye display 222. .

  In step S412, whether the CPU 201 of the information processing device 200-A has received an instruction to display the designated point model according to whether or not an instruction on whether to display the designated point model based on the line of sight is received from the user in advance. Determine whether. For example, when a screen such as the designated point model display instruction acceptance screen 1500 shown in FIG. 15 is displayed on the display 210 and the pressing of the YES button 1501 is accepted, it is determined that the designated point model display instruction is accepted, and the NO button 1502 Is received, it is determined that an instruction to hide the specified point model has been received. If it is determined that a display instruction for the designated point model has been received, the process proceeds to step S413. If it is determined that a display instruction for the specified point model has not been received, the process proceeds to step S415.

  In step S413, the CPU 201 of the information processing device 200-A displays the right-eye display and the left-eye display of each of the HMD 100-A connected to the information processing device 200-A and the HMD 100-B connected to the information processing device 200-B. A designated point model display process, which is a process for displaying the designated point model, is performed. Details of the designated point model display processing will be described in detail later with reference to FIG. After performing the designated point model display process, the process proceeds to step S415, and the information processing apparatus 200-B performs the process of step S414.

  In step S414, the CPU 201 of the information processing device 200-B performs a virtual object movement process that is a process of moving the virtual object to the position of the designated point model displayed in step S413. Details of the virtual object movement processing will be described in detail later with reference to FIG. After performing the virtual object movement process, the process proceeds to step S415.

  In step S415, the CPU 201 of the information processing apparatus 200-A determines whether an MR application termination instruction has been received from the user. For example, when it is determined that an application end instruction from the user has been received, such as when an application end button is pressed, the process ends. When it is determined that an application end instruction from the user has not been received, the process is performed in step S407. Return to.

  As described above, for example, where the user A is looking in the virtual space by the line of sight of the user A wearing the HMD 100-A even in a state where other users such as a user in a remote place cannot be visually recognized. It brings about the effect that it can grasp. Further, since the user B wearing the HMD 100-B can visually recognize the designated point model based on the line of sight of the user A wearing the HMD 100-A, it is easy to move the virtual object to the designated point model. It provides a mechanism that can be used.

  Next, details of the designated point model display process shown in FIG. 4 will be described using the flowchart shown in FIG.

  In step S501, the CPU 201 of the information processing apparatus 200-A reads the line of sight of the HMD 100-A acquired in step S407 and stored in the RAM 203 or the like. For example, it is information indicating the orientation stored in the HMD position / orientation 801 shown in FIG.

  In step S502, the CPU 201 of the information processing device 200-A specifies a virtual object arranged in the virtual space based on the line of sight of the HMD 100-A read in step S501. The object ID 901 of the virtual object with which the line of sight extending from the HMD 100-A first hit (crossed) is specified.

  In step S503, the CPU 201 of the information processing device 200-A calculates and acquires the coordinates (position coordinates in the virtual space) of the intersection point with the virtual object with which the line of sight extending from the HMD 100-A first hits (identifying means). The acquired coordinates of the intersection are stored in the RAM 203 or the like.

  It should be noted that the virtual object hitting determination area may be defined to be slightly larger than the actual virtual object size so that the virtual object can be easily identified.

  In step S504, the CPU 201 of the information processing apparatus 200-A sends the information on the HMD position / orientation 801 of the HMD 100-A read in step S501 and the coordinates of the intersection acquired in step S503 to the information processing apparatus 200-B. Send. By transmitting these pieces of information to the information processing apparatus 200-B, the information processing apparatus 200-B can display the designated point model.

  In step S505, the CPU 201 of the information processing device 200-A displays the information on the HMD position / orientation 801 of the HMD 100-A read in step S501 and the coordinates of the intersection acquired in step S503 on the virtual space image. Displays the specified point model. The display is performed by superimposing the virtual space image on the real space image as in step S410. As the designated point model, a designated point model of object ID 901 = M444 in the object table 900 is used. By storing the coordinates of the intersection obtained in step S503 in the coordinates 903, the designated point model is displayed with respect to the coordinates of the intersection obtained in step S503. It is assumed that control is performed so that the specified point model is displayed in the direction of the line of sight of the HMD 100-A acquired in step S407 and stored in the RAM 203 or the like. Note that the display position of the designated point model may be specified by detecting a physical object provided with the optical marker 103 by the optical sensor 104.

  For example, the designated point model image 1100 shown in FIG. In the designated point model image 1100, images of an MR space (base A) generated by the information processing apparatus 200-A and an MR space (base B) generated by the information processing apparatus 200-B are drawn. A designated point model displayed in the direction of the line of sight of the HMD 100-A with respect to the intersection 1103 is a designated point model 1102 shown in FIG. By displaying the direction of the line of sight, it is possible to easily notify which part of the virtual object the other user is viewing from which direction.

  In step S506, the CPU 201 of the information processing device 200-B receives the information on the HMD position / orientation 801 of the HMD 100-A and the coordinates of the intersection transmitted from the information processing device 200-A in step S504. After receiving, the process proceeds to step S507.

  In step S507, the CPU 201 of the information processing device 200-B distributes the HMD position / orientation 801 received in step S506 and the coordinates of the intersection to the information processing device 200 of the secondary machine other than the information processing device 200-A. To do. As a result, the information processing devices 200 of a plurality of secondary machines can share the position of the designated point model in the information processing device 200-A.

  In step S508, the CPU 201 of the information processing device 200-B superimposes the specified point model on the virtual space image based on the information of the HMD position / orientation 801 of the HMD 100-A received in step S506 and the coordinates of the intersection. Display (display control means). The virtual object is displayed in an identifiable manner by superimposing it. Since the display method of the designated point model is the same as that in step S505, it is omitted. The figure shown in the base B of the designated point model image 1100 shown in FIG. 11 is a display image of the designated point model. A designated point model 1104 is displayed at the base B so as to point to the virtual object. Displaying the designated point model 1104 also has an effect of enabling the user to visually recognize the virtual object pointed to by the designated point model 1104. In the present embodiment, the designated point model 1104 has an arrow shape, but the method for enabling the virtual object to be identified is not limited to this. Various methods are included, such as changing the color of a virtual object having an intersection with the line of sight, or making other than the virtual object semi-transparent.

  The designated point model is an arrow-shaped virtual object indicating a vector from the HMD 100 to the virtual object, but may be configured to identify and display the intersection between the line of sight of the HMD 100 and the virtual object. Details will be described later in a second embodiment.

  The present invention is configured to display a specified point model in real time by constantly acquiring the user's line of sight, but detects that the user's hand has contacted the virtual object, and the user's You may make it the structure which displays a designated point model based on a eyes | visual_axis.

  According to the above processing, the position in the virtual space is specified based on the line of sight of the user A wearing the HMD 100-A, and the HMD 100-B is attached where the user A is looking based on the specified position. There is an effect that the user B can grasp.

  Next, details of the virtual object movement processing shown in FIG. 4 will be described using the flowchart shown in FIG.

  In step S601, the CPU 201 of the information processing apparatus 200-B receives from the selection device 105 a movement operation signal that is a virtual object movement instruction from the selection device 105 (movement instruction reception unit). Specifically, the movement operation signal is received by accepting pressing of the selection button 1001 of the selection device 105 shown in FIG.

  In step S602, the CPU 201 of the information processing apparatus 200-B acquires coordinate information on the real space of the selection device 105 when the movement operation signal is received in step S601. The acquired coordinate information on the real space of the selection device 105 is stored in the RAM 203. As described above, the optical sensor 104 is used to acquire the coordinate information of the selection device 105.

  In step S603, the CPU 201 of the information processing apparatus 200-B, based on the coordinate information in the real space of the selection device 105 acquired in step S602, the virtual object in a predetermined area on the virtual space corresponding to the coordinate information. It is determined whether or not there is. Specifically, the predetermined area is an area indicated by the size 904 of the object table 900, and a range indicated by the size 904 with reference to the coordinates 903 of the selection device 105 is defined as the predetermined area. If it is determined that the virtual object is in contact with the predetermined area, the process proceeds to step S604. If it is determined that the virtual object is not in contact with the predetermined area, the process is performed at step S601. Return to.

  In step S604, the CPU 201 of the information processing apparatus 200-B specifies the object ID 901 of the virtual object determined to be in contact in step S603, and changes the selection flag 906 corresponding to the object ID 901 from 0 to 1. .

  In step S605, the CPU 201 of the information processing apparatus 200-B synchronizes the value of the coordinate 903 corresponding to the object ID 901 specified in step S604 with the coordinate 903 of the selection device 105. For example, when the object ID 901 = M222 is specified, the coordinates 903 corresponding to M222 are overwritten as the coordinates 903 (XX4, YY4, ZZ4) of M444. Accordingly, since the coordinates of the selection device 105 and the virtual object to be moved match, it is possible to display the HMD 100-B so that the virtual object is moving according to the behavior of the selection device 105 of the user.

  When the virtual device is moved by the selection device 105, the coordinate 903 of the moved virtual object is transmitted to the information processing apparatus 200 of the primary machine, and the information processing apparatus 200 of the primary machine sends the information to the information processing apparatus 200 of the secondary machine. Deliver to As a result, even if an instruction to move a virtual object is issued by one information processing apparatus 200, the position coordinates of the virtual object can be synchronized between the plurality of information processing apparatuses 200.

  In step S606, the CPU 201 of the information processing device 200-B acquires the coordinates of the HMD 100-B. Specifically, the optical marker attached to the HMD 100-B is detected using the optical sensor 104 as in step S407, and the position and orientation of the HMD 100-B in the MR space are acquired. The acquired information is stored in the HMD position / orientation 801 of the HMD information table 800.

  In step S607, the CPU 201 of the information processing device 200-B determines the distance between the two points indicated by the two coordinate information based on the coordinate information of the HMD 100-B acquired in step S606 and the coordinate information of the specified point model received in step S506. Distance (a) is calculated (distance specifying means). The calculated distance (a) is stored in the RAM 203 or the like.

  In step S608, the CPU 201 of the information processing device 200-B determines whether the distance (a) acquired in step S607 is within a threshold value. Whether or not the threshold value is within the threshold value is determined based on the threshold value 1601 of the HMD in the threshold value table 1600 set in advance shown in FIG. Since the value of 5 meters is set in the threshold value 1601 of the HMD in FIG. 16, it is determined whether the distance indicated by the coordinates of the HMD 100-B and the designated point model is within 5 meters. If it is determined that the distance threshold indicated by the coordinates of the HMD 100-B and the designated point model is exceeded, the process proceeds to step S609. If it is determined that the coordinates are within the threshold, the process proceeds to step S610. Thereby, it becomes possible to aim at the space | interval with a designated point model not by an arrow but by a distance, when a user approaches a designated point model.

  In step S609, the CPU 201 of the information processing device 200-B displays the value of the distance (a) on the HMD 100-B. The display is performed by superimposing the virtual space image on the real space image as in step S410. For example, a distance display image 1200 shown in FIG. Since the user can visually recognize the designated point model within the range within the threshold value, the user can grasp the distance to the designated point model by displaying the distance to the designated point model.

  In step S610, the CPU 201 of the information processing device 200-B specifies a vector composed of two points, the coordinate information of the HMD 100-B acquired in step S606 and the coordinate information of the designated point model received in step S506 (vector). Specific means).

  In step S611, the CPU 201 of the information processing apparatus 200-B displays an arrow based on the vector specified in step S610. The display is performed by superimposing the virtual space image on the real space image as in step S410. It is assumed that an arrow object that is a virtual object stored in the object table 900 is used for the arrow. Control is performed so that the arrow object is oriented in the vector direction specified in step S610 and further superimposed as a virtual space image to generate a mixed reality image and displayed on the HMD 100-B as in step S410. For example, an arrow display image 1300 shown in FIG. An arrow 1301 is displayed in the arrow display image 1300 as illustrated. Thereby, even if the user is far away from the designated point model, the user can reach the designated point model by moving along the arrow.

  In step S612, the CPU 201 of the information processing device 200-B performs a process of moving the virtual object to the designated point model. Details will be described with reference to a flowchart shown in FIG.

  By the above processing, it is possible to perform appropriate guidance for the user by switching the display of the arrow or the distance to the designated point model according to the predetermined distance.

  Next, details of the process of reaching the designated point shown in FIG. 6 will be described with reference to FIG.

  In step S701, the CPU 201 of the information processing device 200-B determines the two points indicated by the two coordinate information based on the coordinate information of the selection device 105 acquired in step S602 and the coordinate information of the designated point model received in step S506. The distance (b) is calculated (distance specifying means). After calculating the distance (b), the process proceeds to step S702.

  In step S <b> 702, the CPU 201 of the information processing device 200 -B compares the value (b) calculated in step S <b> 701 with the threshold 1602 of the virtual object in the threshold table 1600 stored in the external memory 211. If it is determined that (b) is within the threshold 1602 of the virtual object in the threshold table 1600 stored in the external memory 211, the process proceeds to step S705. If it is determined that (b) is not within the threshold 1602 of the virtual object in the threshold table 1600 stored in the external memory 211, the process proceeds to step S703. For example, a description will be given using a virtual object moving image 1400 shown in FIG. When the moving virtual object selected by the selection device 105 is in contact with the predetermined area 1201, the synchronization with the selection device 105 is canceled, and the virtual object is placed at a position where the designated point model exists.

  In step S <b> 703, the CPU 201 of the information processing apparatus 200 -B determines whether a release signal is received from the selection device 105. The release signal is received when, for example, the pressing of the selection button 1001 is not detected. If it is determined that a release signal has been received from the selected device 105, the process proceeds to step S704. If it is determined that a release signal has not been received from the selected device 105, the process returns to step S606.

  In step S704, the CPU 201 of the information processing device 200-B releases the synchronization with the selection device 105 at the coordinate 903 of the virtual object whose coordinates are synchronized with the selection device 105 (the virtual object whose selection flag 906 is 1). .

  In step S705, the CPU 201 of the information processing apparatus 200-B synchronizes the coordinates 903 of the virtual object whose selection flag 906 is “1” with the coordinates 903 of the designated point model. Thereby, the virtual object can be fixed to the coordinates of the designated point model.

  In step S706, the CPU 201 of the information processing apparatus 200-B sets the selection prohibition flag 907 corresponding to the object ID 901 of the virtual object synchronized with the coordinates 903 of the specified point model in step S705, and sets the selection flag 906 from 1. 0. (Movement control means)

  As described above, selection by the selection device 105 is not accepted while the selection prohibition flag 907 is set.

  In step S707, the CPU 201 of the information processing apparatus 200-B determines whether or not a predetermined time (predetermined time) set in advance has elapsed. If it is determined that a predetermined time set in advance has elapsed, the process proceeds to step S708. If it is determined that the predetermined time set in advance has not elapsed, the process returns to step S707.

  In step S708, the CPU 201 of the information processing apparatus 200-B returns the selection prohibition flag 907 set in step S706 to 0.

  This is the end of the description of the first embodiment of the present invention. Next, the description of the second embodiment of the present invention is started.

  In the first embodiment, a position in the virtual space is specified based on the line of sight of the user A wearing the HMD 100-A, and an arrow-like shape is determined based on two points: the specified position and the position of the user A's HMD 100-A. Displayed virtual objects. Thereby, the other user can grasp from which direction the user A is looking into the specified position.

  The features of the second embodiment are as follows. In the second embodiment, a virtual object (marked virtual object) having no concept of direction is superimposed and displayed at a position on the virtual space specified based on the line of sight of the user A wearing the HMD 100-A. It is characterized by. Further, the HMD 100 that has read the line-of-sight direction is characterized in that the visibility of the HMD 100 is improved by not displaying the designated point model indicating the line of sight. Further, the display of the designated point model is controlled by determining whether or not the virtual object closest to the line-of-sight direction of the HMD 100 is a virtual object that should display the designated point model.

In the second embodiment, the system configuration, hardware configuration, functional configuration, screen example, and table are the same as those in the first embodiment except for the following changes. As changes, FIG. 5 is changed to FIG. 17, FIG. 9 is changed to FIG. 19, and FIG. 11 is changed to FIG.
17 and FIG. 18 will now be described.

  FIG. 17 is a flowchart for explaining the detailed processing flow of the designated point model display processing according to the second embodiment of the present invention.

  Steps S1701 to S1702 are the same as steps S501 to S502 shown in FIG.

  In the flowchart of FIG. 17, the process corresponding to step S505 of FIG. 5 is deleted. Thus, the designated point model is not displayed in the user's field of view by not displaying the designated point model on the HMD 100 of the user at the site A (the user A wearing the HMD 100-A in this embodiment). It is possible to do that. In the HMD 100-B worn by the user at the site B, the designated point model specified from the line of sight of the user at the site A is displayed.

  In the second embodiment, the designated point model is a mark, but if it can be identified and displayed, an arrow-shaped one may be used as in the first embodiment.

  In step S1703, the CPU 201 of the information processing apparatus 200-A refers to the object table 1900 in FIG. The object table 1900 is obtained by adding a line-of-sight display flag 1908 to the object table 900 shown in FIG. The line-of-sight display flag 1908 is set in advance. If the line-of-sight display flag 1908 is “0”, it indicates that the virtual object does not display the designated point model, and if it is “1”, the designated point is displayed. It is a virtual object that displays a model.

  In step S1704, the CPU 201 of the information processing apparatus 200-A checks the line-of-sight display flag 1908 of the virtual object specified in step S1702. If the line-of-sight display flag 1908 is “0”, the process returns to step S1702 to display the line of sight. If the flag 1908 is “1”, the process proceeds to step S1705.

  As described above, by providing the setting for each virtual object, there is an effect that the specified point model may not be displayed depending on the virtual object.

  Since the processing from step S1705 to step S1708 is the same as that of steps S503 to 504 and steps S506 to 507, description thereof will be omitted. As described above, step S505 is deleted.

  In step S1709, the CPU 201 of the information processing apparatus 200-B superimposes and displays a predetermined designated point model (virtual object) on the coordinates of the intersection received in step S1705 (coordinates on the virtual object that intersects the viewpoint).

For example, the image shown in FIG. FIG. 18 shows an identification display image 1800. The specified point model 1801 is not displayed on the HMD 100-A in which the position coordinates of the virtual object are specified based on the line of sight (notifying the target being viewed by another HMD), and the HMD 100 is an HMD 100 different from the HMD 100-A. In -B, the designated point model 1801 is displayed. By displaying the designated point model 1801, the virtual object can be displayed in an identifiable manner. The designated point model 1801 is not limited to the image in FIG. 18, and is not limited to this as long as the intersection between the virtual object and the line of sight can be identified. In addition, there is an effect that the virtual object can be identified.
Thereby, there exists an effect which brings about the visibility improvement of the user who mounts HMD100-A.

  In the second embodiment, the display form other than the arrow-shaped designated point model is described. However, the above-described processing may be performed using the arrow-shaped designated point model.

  With the above processing, when the virtual object selected and moved by the selection device 105 within a predetermined range of the designated point model approaches, the user can easily place the virtual object at a certain coordinate of the designated point model. To do. Further, by prohibiting selection for a predetermined time, the user can place a virtual object at a certain coordinate of the designated point model without performing a selection cancellation instruction by the selection device 105.

  As described above, according to the present embodiment, the desired position in the first virtual space is specified by the line of sight of the first head mounted display, so that the second head mounted display can be displayed in an identifiable manner. Play.

  The present invention can be implemented as a system, apparatus, method, program, storage medium, or the like, and can be applied to a system including a plurality of devices. You may apply to the apparatus which consists of one apparatus.

  Note that the present invention includes a software program that implements the functions of the above-described embodiments directly or remotely from a system or apparatus. The present invention also includes a case where the system or the computer of the apparatus is achieved by reading and executing the supplied program code.

  Accordingly, since the functions of the present invention are implemented by computer, the program code installed in the computer also implements the present invention. In other words, the present invention includes a computer program itself for realizing the functional processing of the present invention.

  In that case, as long as it has the function of a program, it may be in the form of object code, a program executed by an interpreter, script data supplied to the OS, and the like.

  Examples of the recording medium for supplying the program include a flexible disk, hard disk, optical disk, magneto-optical disk, MO, CD-ROM, CD-R, and CD-RW. In addition, there are magnetic tape, nonvolatile memory card, ROM, DVD (DVD-ROM, DVD-R), and the like.

  As another program supply method, a browser on a client computer is used to connect to an Internet home page. The computer program itself of the present invention or a compressed file including an automatic installation function can be downloaded from the homepage by downloading it to a recording medium such as a hard disk.

  It can also be realized by dividing the program code constituting the program of the present invention into a plurality of files and downloading each file from a different homepage. That is, a WWW server that allows a plurality of users to download a program file for realizing the functional processing of the present invention on a computer is also included in the present invention.

  In addition, the program of the present invention is encrypted, stored in a storage medium such as a CD-ROM, distributed to users, and key information for decryption is downloaded from a homepage via the Internet to users who have cleared predetermined conditions. Let The program can also be executed by executing the encrypted program using the downloaded key information and installing it on the computer.

  Further, the functions of the above-described embodiments are realized by the computer executing the read program. In addition, based on the instructions of the program, an OS or the like running on the computer performs part or all of the actual processing, and the functions of the above-described embodiments can also be realized by the processing.

  Further, the program read from the recording medium is written in a memory provided in a function expansion board inserted into the computer or a function expansion unit connected to the computer. Thereafter, the CPU of the function expansion board or function expansion unit performs part or all of the actual processing based on the instructions of the program, and the functions of the above-described embodiments are realized by the processing.

  Note that the above-described embodiments are merely examples of implementation in carrying out the present invention, and the technical scope of the present invention should not be construed in a limited manner. That is, the present invention can be implemented in various forms without departing from the technical idea or the main features thereof.

100 HMD
101 LAN
103 optical marker 104 optical sensor 105 selection device 200 information processing apparatus 201 CPU
202 ROM
203 RAM
204 System bus 205 Input controller 206 Video controller 207 Memory controller 208 Communication I / F controller 209 Input device 210 Display 211 External memory 212 General-purpose bus 221 Right-eye / left-eye video camera 222 Right-eye / left-eye display

The present invention, a desired position on the virtual space, by identifying the line of sight of the first head-mounted display, and an object thereof to identifiably displayed in the second head-mounted display.

According to the present invention, the desired position on the virtual space, by identifying the line of sight of the first head-mounted display, exhibits the possible effect of be identifiably displayed in the second head-mounted display.

JP 2014-032443 A

Claims (14)

An information processing system for managing a virtual space of a first head mounted display and a virtual space of a second head mounted display, which are common virtual spaces,
Acquisition means for acquiring line-of-sight information, which is information indicating the line of sight of the first head-mounted display in real space;
Based on the line-of-sight information acquired by the acquiring means, specifying means for specifying position coordinates in a virtual space of the first head mounted display;
An information processing system comprising: display control means for displaying, in an identifiable manner, position coordinates corresponding to the position coordinates specified by the specifying means on a virtual space of a second head mounted display.   The information processing system according to claim 1, wherein the line-of-sight information includes position information and orientation of a head-mounted display in a real space. The position coordinate is a position coordinate of an intersection of a line of sight indicated by the line-of-sight information acquired by the acquisition unit and a virtual object in a virtual space of the first head-mounted display. 2. The information processing system according to 2.   4. The display control unit according to claim 1, wherein the display control unit displays the position coordinates specified by the specifying unit in an identifiable manner on a virtual space of the first head mounted display. 5. Information processing system. 5. The information according to claim 1, wherein the display control unit displays a virtual object including a position coordinate corresponding to the position coordinate specified by the specifying unit in an identifiable manner. Processing system. Storage means for storing a flag for each virtual object;
A determination means for determining whether or not the display control means displays the position coordinates in an identifiable manner by the flag stored in the storage means with the virtual object specified by the specifying means; and
The display control means, when it is determined that the determination means should identify and display, the position coordinates corresponding to the position coordinates specified by the specifying means are displayed in an identifiable manner. The information processing system according to any one of 1 to 5. A movement instruction receiving means for receiving a movement instruction for the virtual object;
Distance specifying means for specifying the distance between the position coordinates indicated by the position information of the virtual object received by the movement instruction receiving means and the position coordinates on the virtual space specified by the specifying means;
The movement control means for controlling the virtual object not to move when the distance specified by the distance specifying means is smaller than a predetermined value. 7. Information processing system described in 1. The distance specifying means further specifies the distance from the position of the head mounted display to the position coordinates acquired by the acquiring means,
The display control means includes
The information indicating the distance is displayed when the distance from the position of the head mounted display specified by the distance specifying unit to the position coordinates acquired by the acquiring unit is within a predetermined value. 8. The information processing system according to 7. A vector specifying means for specifying a vector from the position coordinates of the virtual object to the position coordinates specified by the specifying means;
The display control means includes
When the distance from the position of the head mounted display specified by the distance specifying unit to the position coordinates acquired by the acquiring unit is larger than a predetermined value, the vector specified by the vector specifying unit is displayed. The information processing system according to claim 7 to 8. A control method of an information processing system for managing a virtual space of a first head mounted display and a virtual space of a second head mounted display, which are common virtual spaces,
An acquisition step in which the acquisition unit of the information processing system acquires line-of-sight information that is information indicating the line of sight of the first head-mounted display in real space;
A specifying step of specifying a position coordinate in a virtual space of the first head mounted display based on the line-of-sight information acquired in the acquiring step;
A display control step of causing the display control means of the information processing system to display the position coordinates corresponding to the position coordinates specified in the specifying step in an identifiable manner on the virtual space of the second head mounted display. An information processing system control method characterized by the above. A program that can be executed by an information processing system that manages a virtual space of a first head mounted display and a virtual space of a second head mounted display, which is a common virtual space,
The information processing system;
Acquisition means for acquiring line-of-sight information, which is information indicating the line of sight of the first head-mounted display in real space;
Based on the line-of-sight information acquired by the acquiring means, specifying means for specifying position coordinates in a virtual space of the first head mounted display;
A program that functions as display control means for displaying, in an identifiable manner, position coordinates corresponding to the position coordinates specified by the specifying means on a virtual space of a second head-mounted display. A virtual space in which the virtual space of the first head-mounted display and the virtual space of the second head-mounted display are common, and the first space is capable of communicating with an information processing device that manages the virtual space of the second head-mounted display. An information processing apparatus for managing the virtual space of the head-mounted display of
Acquisition means for acquiring line-of-sight information, which is information indicating the line of sight of the first head-mounted display in real space;
Based on the line-of-sight information acquired by the acquiring means, specifying means for specifying position coordinates in a virtual space of the first head mounted display;
An information processing apparatus comprising: transmission means for transmitting the position coordinates specified by the specifying means to an information processing apparatus that manages a virtual space of a second head mounted display. A virtual space in which the virtual space of the first head-mounted display and the virtual space of the second head-mounted display are common, and the first space is capable of communicating with an information processing device that manages the virtual space of the second head-mounted display. A method for controlling an information processing apparatus for managing a virtual space of a head-mounted display of
An acquisition step in which the acquisition unit of the information processing apparatus acquires line-of-sight information, which is information indicating the line of sight of the first head-mounted display in real space;
A specifying step of specifying a position coordinate in a virtual space of the first head-mounted display based on the line-of-sight information acquired in the acquiring step;
A transmission step of transmitting the position coordinates specified by the specifying means to an information processing apparatus managing a virtual space of a second head mounted display. A virtual space in which the virtual space of the first head-mounted display and the virtual space of the second head-mounted display are common, and the first space is capable of communicating with an information processing device that manages the virtual space of the second head-mounted display. A program that can be executed by an information processing device that manages the virtual space of the head-mounted display of
The information processing apparatus;
Acquisition means for acquiring line-of-sight information, which is information indicating the line of sight of the first head-mounted display in real space;
Based on the line-of-sight information acquired by the acquiring means, specifying means for specifying position coordinates in a virtual space of the first head mounted display;
A program that causes the position coordinates specified by the specifying means to function as a transmitting means for transmitting to an information processing device that manages a virtual space of a second head mounted display.

Images