JPWO2017191701A1 - Display control apparatus, display control method, and program - Google Patents

Abstract

  Provided are a display control device, a display control method, and a program capable of displaying an image of a virtual space in which a pointer is arranged at an appropriate position. The virtual space management unit (126) places a pointer at a position in the virtual space according to the direction of the head mounted display. The frame image display control unit (130) displays an image of the virtual space where the pointer is arranged.

Description

  The present invention relates to a display control device, a display control method, and a program.

  There is known a head mounted display (HMD) that displays a view of the viewing direction from a viewpoint arranged in a virtual space. Among such HMDs, there is one in which the line-of-sight direction set in the virtual space changes according to the change in the attitude of the HMD.

  For the purpose of transmitting the position in the virtual space to other users, specifying the operation target, etc., for example, an appropriate position according to the attitude of the HMD, such as the position in the virtual space corresponding to the center of the image displayed on the HMD It would be convenient if the pointer could be placed at the position. However, conventionally, the pointer could not be placed at an appropriate position according to the attitude of the HMD.

  The present invention has been made in view of the above circumstances, and one of its purposes is to provide a display control device, a display control method, and a program capable of displaying an image of a virtual space in which a pointer is arranged at an appropriate position. There is.

  In order to solve the above problems, a display control device according to the present invention includes a pointer placement unit that places a pointer at a position in a virtual space according to the orientation of a head mounted display, and an image of the virtual space in which the pointer is placed. And a display control unit for displaying.

  In one aspect of the present invention, the pointer arrangement unit changes a position where the pointer is arranged according to an operation of a user wearing the head mounted display.

  In this aspect, the pointer arrangement unit may perform control so that a position where the pointer is arranged is limited to a range indicating a position in the virtual object arranged in the virtual space.

  Alternatively, the pointer placement unit is a virtual where the position where the pointer is placed intersects a line along a direction corresponding to the direction of the head mounted display among the plurality of virtual objects placed in the virtual space. You may control so that it may be restrict | limited to the range which shows the position in an object.

  In the aspect of the invention, the display control unit displays the image on a head mounted display different from the head mounted display.

  Further, the display control method according to the present invention includes a step of arranging a pointer at a position in the virtual space according to the orientation of the head mounted display, and a step of displaying an image of the virtual space in which the pointer is arranged. .

  Further, the program according to the present invention causes a computer to execute a procedure for placing a pointer at a position in a virtual space according to the orientation of the head mounted display and a procedure for displaying an image of the virtual space in which the pointer is placed.

It is a figure which shows an example of the whole structure of the virtual space sharing system which concerns on one Embodiment of this invention. It is a figure which shows an example of a structure of the user system which concerns on one Embodiment of this invention. It is a figure which shows an example of a structure of the head mounted display which concerns on one Embodiment of this invention. It is a figure which shows an example of a structure of the entertainment apparatus which concerns on one Embodiment of this invention. It is a figure which shows an example of a mode that the virtual space is shared by several user systems. It is explanatory drawing explaining an example of the coordinate value of the position in a texture image. It is a figure which shows an example of viewpoint data. It is a figure which shows an example of viewpoint data. It is a figure which shows an example of a virtual space image. It is a figure which shows an example of virtual space. It is a figure which shows an example of a virtual space image. It is a figure which shows an example of virtual space. It is a figure which shows an example of indicator data. It is a figure which shows an example of indicator data. It is a figure which shows an example of virtual space. It is a figure which shows an example of a virtual space image. It is a figure which shows an example of virtual space. It is a figure which shows an example of a virtual space image. It is a figure which shows an example of pointer data. It is a figure which shows an example of virtual space. It is a figure which shows an example of a virtual space image. It is a functional block diagram which shows an example of the function mounted with the entertainment apparatus which concerns on one Embodiment of this invention. It is a flowchart which shows an example of the flow of the process performed in the entertainment apparatus which concerns on one Embodiment of this invention. It is a flowchart which shows an example of the flow of the process performed in the entertainment apparatus which concerns on one Embodiment of this invention. It is a flowchart which shows an example of the flow of the process performed in the entertainment apparatus which concerns on one Embodiment of this invention. It is a figure which shows an example of virtual space. It is a figure which shows an example of a virtual space image. It is a figure which shows an example of a virtual space image. It is a figure which shows an example of a virtual space image. It is a figure which shows an example of a virtual space image.

  Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

  FIG. 1 is a diagram showing an example of the overall configuration of a virtual space sharing system 1 according to an embodiment of the present invention. As shown in FIG. 1, the virtual space sharing system 1 according to the present embodiment includes a plurality of user systems 2 (2-1, 2-2,..., 2-n) and a management system 3. Yes. The user system 2 and the management system 3 are connected to a computer network 4 such as the Internet, and the plurality of user systems 2 and the management system 3 can communicate with each other.

  FIG. 2 is a diagram showing an example of the configuration of the user system 2 according to the embodiment of the present invention. FIG. 3A is a diagram illustrating an example of the configuration of the head mounted display (HMD) 12 according to the present embodiment. FIG. 3B is a diagram illustrating an example of the configuration of the entertainment device 14 according to the present embodiment.

  As shown in FIG. 2, the user system 2 according to the present embodiment includes an HMD 12, an entertainment device 14, a relay device 16, a display 18, a camera microphone unit 20, and a controller 22.

  For example, as shown in FIG. 3A, the HMD 12 according to the present embodiment includes a control unit 30, a storage unit 32, a communication unit 34, an input / output unit 36, a display unit 38, a sensor unit 40, and an audio output unit 42.

  The control unit 30 is a program control device such as a microprocessor that operates according to a program installed in the HMD 12, for example.

  The storage unit 32 is, for example, a storage element such as a ROM or a RAM. The storage unit 32 stores a program executed by the control unit 30.

  The communication unit 34 is a communication interface such as a wireless LAN module.

  The input / output unit 36 is an input / output port such as an HDMI (registered trademark) (High-Definition Multimedia Interface) port, a USB port, an AUX port, or the like.

  The display unit 38 is a display such as a liquid crystal display or an organic EL display disposed on the front side of the HMD 12 and displays an image generated by the entertainment device 14. The display unit 38 is housed in the housing of the HMD 12. For example, the display unit 38 may receive a video signal output from the entertainment device 14 and relayed by the relay device 16 and output a video represented by the video signal. The display unit 38 according to the present embodiment can display a three-dimensional image by displaying a left-eye image and a right-eye image, for example. The display unit 38 may not be able to display a three-dimensional image and can only display a two-dimensional image.

  The sensor unit 40 is a sensor such as an acceleration sensor or a motion sensor. The sensor unit 40 outputs motion data indicating measurement results such as the rotation amount and movement amount of the HMD 12 to the control unit 30 at a predetermined frame rate.

  The sound output unit 42 is, for example, a headphone or a speaker, and outputs sound represented by sound data generated by the entertainment device 14. The audio output unit 42 receives an audio signal output from the entertainment device 14 and relayed by the relay device 16, for example, and outputs the audio represented by the audio signal.

  The entertainment device 14 according to the present embodiment is a computer such as a game console, a DVD player, a Blu-ray (registered trademark) player, or the like. The entertainment device 14 according to the present embodiment generates video and audio by, for example, executing a stored game program or reproducing content recorded on an optical disc. Then, the entertainment device 14 according to the present embodiment outputs a video signal representing the generated video and an audio signal representing the generated audio to the HMD 12 and the display 18 via the relay device 16.

  The entertainment apparatus 14 according to the present embodiment includes a control unit 50, a storage unit 52, a communication unit 54, and an input / output unit 56, for example, as shown in FIG. 3B.

  The control unit 50 is a program control device such as a CPU that operates according to a program installed in the entertainment device 14, for example. The control unit 50 according to the present embodiment also includes a GPU (Graphics Processing Unit) that draws an image in a frame buffer based on graphics commands and data supplied from the CPU.

  The storage unit 52 is, for example, a storage element such as a ROM or a RAM, a hard disk drive, or the like. The storage unit 52 stores a program executed by the control unit 50. In the storage unit 52 according to the present embodiment, a frame buffer area in which an image is rendered by the GPU is secured.

  The communication unit 54 is a communication interface such as a wireless LAN module.

  The input / output unit 56 is an input / output port such as an HDMI (registered trademark) (High-Definition Multimedia Interface) port or a USB port.

  The relay device 16 according to the present embodiment is a computer that relays video signals and audio signals output from the entertainment device 14 and outputs them to the HMD 12 and the display 18.

  The display 18 according to the present embodiment is a liquid crystal display or the like, for example, and displays a video or the like represented by a video signal output from the entertainment device 14.

  The camera microphone unit 20 according to the present embodiment includes, for example, a camera 20a that outputs an image obtained by imaging a subject to the entertainment device 14, and a microphone that acquires surrounding audio, converts the audio to audio data, and outputs the audio data to the entertainment device 14. 20b is included. The camera 20a according to this embodiment is a stereo camera.

  For example, the HMD 12 and the relay device 16 can mutually transmit and receive data by wireless communication. The entertainment device 14 and the relay device 16 are connected via, for example, an HDMI cable or a USB cable, and data can be transmitted and received between them. The relay device 16 and the display 18 are connected via, for example, an HDMI cable. The entertainment apparatus 14 and the camera microphone unit 20 are connected via, for example, an AUX cable.

  The controller 22 according to the present embodiment is an operation input device for performing operation input to the entertainment device 14. The user can perform various operation inputs using the controller 22 by pressing a direction key or button provided in the controller 22 or tilting an operation stick. In this embodiment, the controller 22 outputs input data associated with the operation input to the entertainment device 14. The controller 22 according to the present embodiment includes a USB port. And the controller 22 can output input data to the entertainment apparatus 14 with a wire by connecting with the entertainment apparatus 14 with a USB cable. The controller 22 according to the present embodiment includes a wireless communication module and the like, and can output input data to the entertainment apparatus 14 wirelessly.

  The management system 3 is a computer system composed of a server and the like. In the present embodiment, a three-dimensional virtual space is shared by a plurality of user systems 2 via the management system 3.

  FIG. 4 is a diagram illustrating an example of a state in which the virtual space 60 is shared between the user system 2-1 and the user system 2-2.

  In the virtual space 60 illustrated in FIG. 4, there are three hanging object 62 (a hanging object 62a, a hanging object 62b, and a hanging object 62c), each of which is a plate-like three-dimensional virtual object, to be viewed by the user. Has been placed. These hanging shaft objects 62 are virtual objects that are visible to the user wearing the HMD 12 included in the user system 2 for each of the plurality of user systems 2. The hanging object 62 is composed of a plurality of polygons, and a texture image, which is a two-dimensional image, such as a hanging image on which a drawing is drawn, is mapped on the front surface. In FIG. 4, the direction from the left to the right when viewed from the front of the hanging object 62 is the X-axis positive direction. A direction from the front to the back when viewed from the front of the hanging shaft object 62 is defined as a positive Y-axis direction. Further, the direction from the bottom to the top as viewed from the front of the hanging object 62 is taken as the positive Z-axis direction.

  For example, the management system 3 transmits virtual space data representing the virtual space 60 illustrated in FIG. 4 to the user system 2 in response to a request from the user system 2. Virtual space data representing the virtual space to be shared is stored in each user system 2. The virtual space data includes, for example, data indicating the position (for example, coordinate value) of the hanging object 62 in the virtual space 60 and a texture image.

  In the virtual space 60, individual viewpoints 64 and line-of-sight directions 66 are set for each user system 2. Then, on the display unit 38 of the HMD 12 included in the user system 2, an image representing a state in which the line-of-sight direction 66 is viewed from the viewpoint 64 individually set in the user system 2 is displayed. For example, on the display unit 38 of the HMD 12 included in the user system 2-1, an image representing a state where the viewing direction 66 a is viewed from the viewpoint 64 a is displayed. In addition, the display unit 38 of the HMD 12 included in the user system 2-2 displays an image representing a state in which the viewing direction 66b is viewed from the viewpoint 64b.

  In the example of FIG. 4, the line-of-sight direction 66a and the line-of-sight direction 66b are both directed in the positive Y-axis direction. A line passing through the viewpoint 64a along the line-of-sight direction 66a and a line passing through the viewpoint 64b along the line-of-sight direction 66b intersect with the hanging object 62b. Here, the position of the intersection point between the line passing through the viewpoint 64a along the line-of-sight direction 66a and the hanging object 62b is P1, and the position of the line passing through the viewpoint 64b along the line-of-sight direction 66b and the hanging line object 62b is P2. And

  In the X1Y1 two-dimensional coordinate system in the texture image mapped on the surface of the hanging object 62b shown in FIG. 5, the coordinate value of the position P1 is (p1x, p1y), and the coordinate value of the position P2 is (p2x, p2y). ). As shown in FIG. 5, in the X1Y1 two-dimensional coordinate system, the direction from the top to the bottom of the texture image is the X1 axis positive direction, and the direction from the left to the right of the texture image is the Y1 axis positive direction. The X1Y1 two-dimensional coordinate system may be a texture coordinate system.

  In the present embodiment, the position and orientation of the HMD 12 can be specified based on an image including an image of the HMD 12 captured by the camera 20a at a predetermined sampling rate and a measurement result by the sensor unit 40. In the present embodiment, the position of the viewpoint 64 and the line-of-sight direction 66 in the user system 2 are set according to the position and orientation of the HMD 12 included in the user system 2. For example, when the user wearing the HMD 12 turns the HMD 12 in the right direction, the line-of-sight direction 66 set for the user system 2 including the HMD 12 turns in the right direction. Here, for example, the amount of change in the line-of-sight direction 66 may be proportional to the amount of change in the direction of the HMD 12. For example, when the user wearing the HMD 12 translates the HMD 12 in the right direction, the position of the viewpoint 64 set for the user system 2 including the HMD 12 translates in the right direction. Here, for example, the change amount of the position of the viewpoint 64 may be proportional to the change amount of the position of the HMD 12. In this way, in the present embodiment, the user of the user system 2 changes the position and orientation of the HMD 12 included in the user system 2 to change the position of the viewpoint 64 and the line-of-sight direction 66 set in the virtual space 60. Can be changed.

  In the present embodiment, each user system 2 receives viewpoint data indicating the viewpoint 64 and the line-of-sight direction 66 set for the user system 2 from the other user systems 2 included in the virtual space sharing system 1.

  6A and 6B are diagrams illustrating an example of viewpoint data according to the present embodiment. As shown in FIGS. 6A and 6B, the viewpoint data according to the present embodiment includes a user system ID, viewpoint position data, and line-of-sight direction vector data. The user system ID included in the viewpoint data is an identifier of the user system 2, and the viewpoint data indicates the viewpoint 64 and the line-of-sight direction 66 of the user system 2 identified by the user system ID. In the present embodiment, each user system 2 included in the virtual space sharing system 1 is assigned a unique user system ID in advance. For example, the user system ID of the user system 2-1 is 1, and the user system ID of the user system 2-2 is 2. The viewpoint position data is data indicating the position of the viewpoint 64. The line-of-sight vector data is data indicating a vector in the line-of-sight direction 66.

  The viewpoint data shown in FIG. 6A shows the viewpoint 64a and the line-of-sight direction 66a set for the user system 2-1. For example, the three-dimensional coordinate values (xa, ya, za) indicated by the viewpoint position data included in the viewpoint data shown in FIG. 6A indicate the position of the viewpoint 64a. For example, the three-dimensional coordinate values (Vxa, Vya, Vza) indicated by the line-of-sight direction vector data included in the viewpoint data shown in FIG. 6A indicate unit vectors along the line-of-sight direction 66a. For example, as shown in FIG. 4, when the line-of-sight direction 66a faces the Y-axis positive direction, the value of (Vxa, Vya, Vza) is (0, 1, 0).

  In the viewpoint data shown in FIG. 6B, a viewpoint 64b and a line-of-sight direction 66b set for the user system 2-2 are shown. For example, the three-dimensional coordinate values (xb, yb, zb) indicated by the viewpoint position data included in the viewpoint data shown in FIG. 6B indicate the position of the viewpoint 64b. For example, the three-dimensional coordinate values (Vxb, Vyb, Vzb) indicated by the line-of-sight direction vector data included in the viewpoint data shown in FIG. 6B indicate unit vectors along the line-of-sight direction 66b. For example, as shown in FIG. 4, when the line-of-sight direction 66b faces the positive Y-axis direction, the value of (Vxb, Vyb, Vzb) is (0, 1, 0).

  The viewpoint data shown in FIG. 6A is transmitted from the user system 2-1 to the management system 3. Then, the management system 3 transmits the viewpoint data to the user system 2 other than the user system 2-1 included in the virtual space sharing system 1 such as the user system 2-2. The viewpoint data shown in FIG. 6B is transmitted from the user system 2-2 to the management system 3. Then, the management system 3 transmits the viewpoint data to the user system 2 other than the user system 2-2 included in the virtual space sharing system 1, such as the user system 2-1. In this way, in the present embodiment, data indicating the position of the viewpoint 64 and the line-of-sight direction 66 set for each user system 2 can be shared by a plurality of user systems 2. That is, each user system 2 can specify the position of the viewpoint 64 and the line-of-sight direction 66 set for the other user systems 2.

  The user system 2 according to the present embodiment can arrange the viewpoint object indicating the position of the viewpoint 64 set in the other user system 2 and the gazing point object indicating the position of the gazing point in the shared virtual space. It has become. Hereinafter, the viewpoint object according to the present embodiment is referred to as a viewpoint indicator 68, and the gazing point object according to the present embodiment is referred to as a gazing point indicator 70. In the present embodiment, the user of the user system 2 can know from where the user of the other user system 2 is looking from the viewpoint indicator 68 or the gazing point indicator 70 displayed. Yes.

  Hereinafter, the display of the viewpoint indicator 68 and the gazing point indicator 70 will be further described.

  FIG. 7 is a diagram illustrating an example of the virtual space image 72a that is displayed on the display unit 38 of the HMD 12 of the user system 2-1, and is an image representing the viewing direction 66a from the viewpoint 64a. FIG. 8 shows an example of the virtual space 74 in which the viewpoint 64a and the line-of-sight direction 66a are set when the virtual space image 72a shown in FIG. 7 is displayed on the HMD 12 of the user system 2-1.

  FIG. 9 is a diagram illustrating an example of a virtual space image 72b that is an image representing the appearance of the line-of-sight direction 66b viewed from the viewpoint 64b, which is displayed on the display unit 38 of the HMD 12 of the user system 2-2. FIG. 10 shows an example of the virtual space 76 in which the viewpoint 64b and the line-of-sight direction 66b are set when the virtual space image 72b shown in FIG. 9 is displayed on the HMD 12 of the user system 2-2.

  The virtual space image 72a and the virtual space image 72b may be a three-dimensional image or a two-dimensional image.

  In the virtual space 74 shown in FIG. 8, a viewpoint indicator 68b is arranged at the position of the viewpoint 64b associated with the user system 2-2. Further, in the virtual space 74 shown in FIG. 8, a gazing point indicator 70b is arranged at the intersection (position P2 shown in FIG. 4) of the intersection of the line passing the viewpoint 64b along the line-of-sight direction 66b and the hanging object 62b. Has been. In the example of FIG. 8, the viewpoint indicator 68b is disposed on the opposite side of the line-of-sight direction 66a when viewed from the viewpoint 64a. That is, the Y coordinate value of the viewpoint indicator 68b is smaller than the Y coordinate value of the viewpoint 64a. Therefore, the virtual space image 72a shown in FIG. 7 does not include the image of the viewpoint indicator 68b.

  In the virtual space 76 shown in FIG. 10, a viewpoint indicator 68a, which is a virtual object representing the viewpoint, is arranged at the position of the viewpoint 64a associated with the user system 2-1. Also, in the virtual space 76 shown in FIG. 10, a gazing point indicator 70a is arranged at the position of the intersection (position P1 shown in FIG. 4) between the line passing the viewpoint 64a along the line-of-sight direction 66a and the hanging object 62b. Has been.

  Here, the viewpoint indicator 68b illustrated in FIG. 8 and the viewpoint indicator 68a illustrated in FIG. 10 are, for example, spherical virtual three-dimensional objects. The gazing point indicator 70b illustrated in FIG. 8 and the gazing point indicator 70a illustrated in FIG. 10 are, for example, cross-shaped two-dimensional images superimposed on the texture image mapped to the hanging scroll object 62b.

  FIG. 11A is a diagram illustrating an example of indicator data associated with the viewpoint indicator 68a and the gazing point indicator 70a. FIG. 11B is a diagram illustrating an example of indicator data associated with the viewpoint indicator 68b and the gaze point indicator 70b.

  As shown in FIGS. 11A and 11B, the indicator data according to the present embodiment includes a user system ID, viewpoint indicator position data, texture ID, and gazing point indicator position data. The user system ID included in the indicator data is an identifier of the user system 2. The indicator data indicates the viewpoint indicator 68 associated with the viewpoint 64 of the user system 2 identified by the user system ID and the gaze point indicator 70 associated with the line-of-sight direction 66. The viewpoint indicator position data is data indicating the position of the viewpoint indicator 68. The texture ID is an identifier of the texture image in which the gazing point indicator 70 is arranged. Here, for example, it is assumed that the texture ID of the texture image mapped to the hanging scroll object 62b is 2. The gazing point indicator position data is data indicating the position of the gazing point indicator 70. In the gazing point indicator position data shown in FIGS. 11A and 11B, the position of the gazing point indicator 70 is expressed by coordinate values in the X1Y1 two-dimensional coordinate system set to the texture image identified by the texture ID.

  For example, the three-dimensional coordinate values (x1a, y1a, z1a) indicated by the viewpoint indicator position data included in the indicator data shown in FIG. 11A indicate the position of the viewpoint indicator 68a. Here, the three-dimensional coordinate values (x1a, y1a, z1a) are three-dimensional coordinate values in the XYZ three-dimensional coordinate system. Further, for example, the two-dimensional coordinate values (x2a, y2a) indicated by the gazing point indicator position data included in the indicator data illustrated in FIG. 11A indicate the position of the gazing point indicator 70a. Here, the two-dimensional coordinate values (x2a, y2a) are, for example, two-dimensional coordinate values in the X1Y1 two-dimensional coordinate system set in the texture image mapped to the multiplying object 62b.

  In addition, for example, the three-dimensional coordinate values (x1b, y1b, z1b) indicated by the viewpoint indicator position data included in the indicator data shown in FIG. 11B indicate the position of the viewpoint indicator 68b. Here, the three-dimensional coordinate values (x1b, y1b, z1b) are three-dimensional coordinate values in the XYZ three-dimensional coordinate system. Further, for example, the two-dimensional coordinate values (x2b, y2b) indicated by the gazing point indicator position data included in the indicator data illustrated in FIG. 11B indicate the position of the gazing point indicator 70b. Here, the two-dimensional coordinate values (x2b, y2b) are, for example, two-dimensional coordinate values in the X1Y1 two-dimensional coordinate system set in the texture image mapped to the multiplying object 62b.

  In the present embodiment, the user system 2-1 generates indicator data illustrated in FIG. 11A based on the viewpoint data illustrated in FIG. 6A. Further, the user system 2-2 generates the indicator data illustrated in FIG. 11B based on the viewpoint data illustrated in FIG. 6B. Then, the viewpoint indicator 68 and the gazing point indicator 70 are arranged in the virtual space at the position indicated by the generated indicator data. In this way, in the present embodiment, the user of the user system 2 can know where the user of the other user system 2 is looking from where in the shared virtual space.

  In the present embodiment, the audio data representing the audio input to the microphone 20 b included in the user system 2 is transmitted to the other user system 2. Then, the voice represented by the voice data is output from the voice output unit 42 included in the HMD 12 of the other user system 2. In this way, in the present embodiment, voice communication can be performed between users of the user system 2. In the present embodiment, smooth communication between users can be achieved using such sounds and images such as the viewpoint indicator 68 and the gazing point indicator 70 as clues.

  Note that the positions and shapes of the viewpoint indicator 68 and the gazing point indicator 70 are not limited to those described above. For example, as shown in FIG. 12, the gaze point indicator 70a arranged in the virtual space 76 may be a three-dimensional cube. The gazing point indicator 70a illustrated in FIG. 12 is disposed, for example, at the intersection of a line passing through the viewpoint 64a along the line-of-sight direction 66a and a spherical surface with a predetermined length around the viewpoint 64a. In this case, for example, the virtual space image 72b illustrated in FIG. 13 is displayed on the display unit 38 of the HMD 12 of the user system 2-2.

  Further, for example, the position of the gazing point indicator 70 may be expressed by a three-dimensional coordinate value instead of a two-dimensional coordinate value. In this case, the indicator ID may not include the texture ID. For example, when the gazing point indicator 70a shown in FIG. 12 is arranged in the virtual space 76, the texture data is not included in the indicator data shown in FIG. 11A, and the gazing point indicator position data is a three-dimensional coordinate value in the XYZ three-dimensional coordinate system. It may be expressed as

  Further, for example, the color or shape of the object that is the viewpoint indicator 68 or the gazing point indicator 70 may be a color or shape corresponding to the user system 2 associated with the object. The user system 2 associated with the object may be identifiable. In this way, when the plurality of viewpoint indicators 68 are displayed, the user of the user system 2 can know which viewpoint indicator 68 represents the position of the viewpoint of the user of which user system 2. . Further, for example, when a plurality of gazing point indicators 70 are displayed, the user of the user system 2 can know which gazing point indicator 70 represents the user's gaze direction of which user system 2. Become.

  In the present embodiment, the user of the user system 2 performs a predetermined operation on the controller 22, for example, whether or not the viewpoint indicator 68 is displayed and whether or not the gazing point indicator 70 is displayed. Can be switched.

  A three-dimensional object representing the user's silhouette may be generated based on the user's image taken by the camera 20a. Then, the three-dimensional object may be arranged in the virtual space as an object indicating the user's viewpoint 64 or line-of-sight direction 66 instead of the viewpoint indicator 68 or the gazing point indicator 70. In this case, whether or not a three-dimensional object representing the user's silhouette is displayed may be switched by performing a predetermined operation on the controller 22, for example.

  Further, for example, the user system 2-1 may generate indicator data illustrated in FIG. 11A and transmit the indicator data to the user system 2-2. For example, the user system 2-2 may generate indicator data illustrated in FIG. 11B and transmit the indicator data to the user system 2-1.

  Further, the user system 2 according to the present embodiment can arrange a pointer in the virtual space in accordance with a predetermined operation performed by the user. Hereinafter, the display of the pointer will be further described.

  FIG. 14 is a diagram illustrating an example of the virtual space 78. In the virtual space 78 shown in FIG. 14, when the pointer image 80 is arranged in the virtual space 76 shown in FIG. 10 where the pointer image 80 is not arranged according to the reset operation performed by the user of the user system 2-2. Indicates the state. In the present embodiment, in response to the reset operation, the pointer image 80 is arranged at the position P2 of the intersection of the line passing the viewpoint 64b along the line-of-sight direction 66b and the hanging object 62b. The reset operation includes, for example, a pressing operation of a predetermined button provided in the controller 22, a click operation on a mouse that can communicate with the entertainment device 14, a tap operation on a touch pad that can communicate with the entertainment device 14, and the like. .

  FIG. 15 is a virtual space image that is displayed on the display unit 38 of the HMD 12 of the user system 2-2 and is an image showing a view direction 66 b from the viewpoint 64 b arranged in the virtual space 78 shown in FIG. 14. It is a figure which shows an example of 72b. The virtual space image 72b shown in FIG. 15 includes a pointer image 80 in addition to the viewpoint indicator 68a and the gazing point indicator 70a. The pointer image 80 according to the present embodiment is, for example, an arrow-shaped two-dimensional image superimposed on a texture image mapped to the hanging scroll object 62b. As described above, according to the present embodiment, it is possible to display an image of the virtual space in which the pointer is arranged at an appropriate position according to the reset operation by the user. Here, for example, the pointer is placed at the position P2 of the intersection of the line passing through the viewpoint 64b along the line-of-sight direction 66b and the hanging object 62b.

  FIG. 16 is a diagram illustrating an example of pointer data associated with the pointer image 80. As shown in FIG. 16, the pointer data according to the present embodiment includes a user system ID, a texture ID, and pointer position data. The user system ID included in the pointer data is an identifier of the user system 2. The pointer data indicates the pointer image 80 arranged by the user of the user system 2 identified by the user system ID. The texture ID is an identifier of the texture image to which the pointer image 80 is mapped. The pointer position data is data indicating the position of the pointer image 80.

  In the pointer position data shown in FIG. 16, the position of the pointer image 80 is expressed by coordinate values in the X1Y1 two-dimensional coordinate system set for the texture image identified by the texture ID. The two-dimensional coordinate value (x3, y3) indicated by the pointer position data included in the pointer data shown in FIG. 16 indicates the position of the pointer image 80 in the X1Y1 two-dimensional coordinate system set in the texture image mapped to the multiplying object 62b. Show. In the initial state when the reset operation is performed, since the pointer image 80 is arranged at the position P2, the two-dimensional coordinate values (x3, y3) indicated by the pointer position data included in the pointer data shown in FIG. 16 are (p2x, p2y).

  In this embodiment, the pointer data shown in FIG. 16 is transmitted from the user system 2-2 to the management system 3. Then, the management system 3 transmits the pointer data to the user system 2 other than the user system 2-2 included in the virtual space sharing system 1, such as the user system 2-1. In this way, in the present embodiment, data indicating the position of the pointer image 80 set for each user system 2 can be shared by a plurality of user systems 2. That is, each user system 2 can specify the position of the pointer image 80 set for the other user system 2.

  FIG. 17 is a diagram illustrating an example of the virtual space 82 in which the pointer image 80 is arranged based on the pointer data transmitted to the user system 2-1. In the virtual space 82 shown in FIG. 17, unlike the virtual space 74 shown in FIG. 8, the viewpoint indicator 68b and the gazing point indicator 70b are not arranged. FIG. 18 is a virtual space image that is displayed on the display unit 38 of the HMD 12 of the user system 2-1 and is an image representing a view direction 66 a from the viewpoint 64 a arranged in the virtual space 82 shown in FIG. 17. It is a figure which shows an example of 72a.

  In the present embodiment, the user of the user system 2 can change the position of the pointer image 80 arranged in the user system 2 by performing a predetermined movement operation. Examples of the moving operation include a pressing operation of a direction key provided in the controller 22, a drag operation to a mouse that can communicate with the entertainment device 14, and a sliding operation to a touch pad that can communicate with the entertainment device 14. Here, for example, the position of the pointer image 80 may be changed in a direction associated with the pressed direction key. Further, for example, the position of the pointer image 80 may be changed in the direction in which the mouse is dragged or the direction in which the touch pad is slid.

  Further, the user of the user system 2 can erase the pointer image 80 arranged in the user system 2 by performing a predetermined erasing operation. Here, the erasing operation includes, for example, a pressing operation of a predetermined button provided in the controller 22, a double-clicking operation on a mouse that can communicate with the entertainment device 14, a double-tapping operation on a touch pad that can communicate with the entertainment device 14, and the like. Can be mentioned. In this embodiment, in this case, 0 is set to the texture ID included in the pointer data, and the value of the pointer position data is set to (0, 0). In this embodiment, when the texture ID included in the pointer data is 0 and the value of the pointer position data is (0, 0), the pointer image 80 is controlled not to be placed in the virtual space. . In addition, when the user system 2 receives pointer data having a texture ID of 0 and a pointer position data value of (0, 0), control is performed so that the pointer image 80 is not arranged in the virtual space in the user system 2. Will be. When an erasing operation is performed, an instruction to erase the pointer image 80 may be transmitted instead of the pointer data described above.

  Further, using a known peak detection technique, whether or not the user's fingertip exists in the detection area in the real space may be specified based on the image captured by the camera 20a. For example, an operation in which the user's fingertip enters the detection area may be handled as the above-described reset operation. Further, for example, a change in the position of the user's fingertip in the detection area may be handled as the above-described movement operation. Here, for example, the position of the pointer image 80 may be changed in the direction in which the user's fingertip is moved. Further, for example, an operation of bringing the user's fingertip existing in the detection area out of the detection area may be handled as the above-described erasing operation.

  Further, the position where the pointer image 80 is arranged may be controlled so as to be limited to a range indicating the position on the hanging scroll object 62b. More specifically, for example, the movement range of the pointer image 80 may be limited on the texture image of the multiplying object 62b on which the pointer image 80 is superimposed. In this case, for example, when the pointer image 80 is arranged at the right end of the texture image, for example, the position of the pointer image 80 is controlled not to change even if a moving operation for moving the pointer image 80 to the right is performed. The

  Further, for example, the pointer data may not be transmitted when the user performs a reset operation, and the pointer data may be transmitted when the user performs a predetermined transmission operation thereafter. Then, after a predetermined transmission operation is performed, the pointer image 80 may be arranged in the virtual space in another user system 2.

  The pointer need not be a two-dimensional image, and may be a three-dimensional virtual object, for example. Further, for example, the position of the pointer may be expressed by a three-dimensional coordinate value instead of a two-dimensional coordinate value. In this case, the texture data may not be included in the pointer data. For example, the texture data may not be included in the pointer data shown in FIG. 16, and the pointer position data may be expressed as a three-dimensional coordinate value in the XYZ three-dimensional coordinate system.

  Hereinafter, functions of the entertainment device 14 according to the present embodiment and processing executed by the entertainment device 14 according to the present embodiment will be further described.

  FIG. 19 is a functional block diagram illustrating an example of functions implemented by the entertainment device 14 according to the present embodiment. Note that the entertainment apparatus 14 according to the present embodiment does not have to include all the functions shown in FIG. 19, and may have functions other than the functions shown in FIG.

  As shown in FIG. 19, the entertainment device 14 according to the present embodiment functionally includes, for example, a virtual space data storage unit 90, a self-viewpoint data storage unit 92, another viewpoint data storage unit 94, an indicator data storage unit 96, Self-pointer data storage unit 98, other pointer data storage unit 100, captured image reception unit 102, position / orientation specifying unit 104, self-viewpoint data management unit 106, viewpoint data transmission unit 108, viewpoint data reception unit 110, other viewpoint data management unit 112, indicator data management unit 114, pointer operation specifying unit 116, own pointer data management unit 118, pointer data transmission unit 120, pointer data reception unit 122, other pointer data management unit 124, virtual space management unit 126, frame image generation unit 128 and a frame image display control unit 130.

  The virtual space data storage unit 90, the own viewpoint data storage unit 92, the other viewpoint data storage unit 94, the indicator data storage unit 96, the own pointer data storage unit 98, and the other pointer data storage unit 100 are mainly implemented by the storage unit 52. . The captured image receiving unit 102 and the frame image display control unit 130 are mainly implemented by the input / output unit 56. Position / attitude specifying unit 104, own viewpoint data management unit 106, other viewpoint data management unit 112, indicator data management unit 114, own pointer data management unit 118, other pointer data management unit 124, virtual space management unit 126, frame image generation unit 128 is mainly implemented by the control unit 50. The viewpoint data transmission unit 108, the viewpoint data reception unit 110, the pointer data transmission unit 120, and the pointer data reception unit 122 are mainly implemented by the communication unit 54. The pointer operation specifying unit 116 is mainly implemented by the communication unit 54 or the input / output unit 56.

  The above functions may be implemented by causing the control unit 50 to execute a program that is installed in the entertainment device 14 that is a computer and that includes instructions corresponding to the above functions. This program may be supplied to the entertainment apparatus 14 via a computer-readable information storage medium such as an optical disk, a magnetic disk, a magnetic tape, a magneto-optical disk, or a flash memory, or via the Internet.

  In this embodiment, the virtual space data storage unit 90 includes, for example, the virtual space 60 shown in FIG. 4, the virtual space 74 shown in FIG. 8, the virtual space 76 shown in FIGS. 10 and 12, and the virtual space 78 shown in FIG. 14. Data indicating the position (for example, coordinate value) of each virtual object is stored. The virtual space data storage unit 90 stores, for example, data indicating the positions of representative points of the hanging object 62, the viewpoint 64 and the line-of-sight direction 66, the viewpoint indicator 68, the gazing point indicator 70, and the pointer image 80. Further, the virtual space data storage unit 90 may store, for example, data indicating the shape and size of a virtual object, a texture image, a virtual object identifier to which a texture image is mapped, and the like.

  In this embodiment, the own viewpoint data storage unit 92 stores, for example, viewpoint data indicating the position and line-of-sight direction of the user's viewpoint 64 of the user system 2 including the own viewpoint data storage unit 92. As shown in FIGS. 6A and 6B, the viewpoint data includes, for example, a user system ID, viewpoint position data, and line-of-sight direction vector data. For example, the viewpoint data illustrated in FIG. 6A is stored in the own viewpoint data storage unit 92 of the user system 2-1. For example, the viewpoint data illustrated in FIG. 6B is stored in the own viewpoint data storage unit 92 of the user system 2-2. In the present embodiment, for example, the user system ID assigned to the user system 2 is set as the user system ID of the viewpoint data stored in the own viewpoint data storage unit 92 of the user system 2.

  In this embodiment, for example, the other viewpoint data storage unit 94 stores viewpoint data indicating the position of the user's viewpoint 64 and the line-of-sight direction 66 of the user system 2 different from the user system 2 including the own viewpoint data storage unit 92. To do. As shown in FIGS. 6A and 6B, the viewpoint data includes, for example, a user system ID, viewpoint position data, and line-of-sight direction vector data.

  The other viewpoint data storage unit 94 may store a plurality of viewpoint data having different user system IDs. For example, the other viewpoint data storage unit 94 of the user system 2-1 may store a plurality of viewpoint data respectively associated with the user systems 2 other than the user system 2-1, which are included in the virtual space sharing system 1. In this case, for example, the viewpoint data illustrated in FIG. 6B is one of a plurality of viewpoint data stored in the other viewpoint data storage unit 94 of the user system 2-1. Further, for example, the other viewpoint data storage unit 94 of the user system 2-2 may store a plurality of viewpoint data respectively associated with the user systems 2 other than the user system 2-2 included in the virtual space sharing system 1. . In this case, for example, the viewpoint data illustrated in FIG. 6A is one of a plurality of viewpoint data stored in the other viewpoint data storage unit 94 of the user system 2-2.

  In this embodiment, for example, the indicator data storage unit 96 stores indicator data associated with the position of the viewpoint 64 and the line-of-sight direction 66 of the user of the other user system 2. As shown in FIGS. 11A and 11B, the indicator data includes, for example, a user system ID, viewpoint indicator position data, texture ID, and gazing point indicator position data.

  The indicator data storage unit 96 may store a plurality of indicator data having different user system IDs. For example, the indicator data storage unit 96 of the user system 2-1 may store a plurality of indicator data respectively associated with the user systems 2 other than the user system 2-1, which are included in the virtual space sharing system 1. In this case, for example, the indicator data illustrated in FIG. 11B is one of a plurality of indicator data stored in the indicator data storage unit 96 of the user system 2-1. Further, for example, the indicator data storage unit 96 of the user system 2-2 may store a plurality of indicator data respectively associated with the user systems 2 other than the user system 2-2 included in the virtual space sharing system 1. In this case, for example, the viewpoint data illustrated in FIG. 11A is one of a plurality of indicator data stored in the indicator data storage unit 96 of the user system 2-2.

  In the present embodiment, the own pointer data storage unit 98 stores pointer data indicating a pointer set by the user of the user system 2 including the own pointer data storage unit 98, for example. As shown in FIG. 16, the pointer data includes, for example, a user system ID, a texture ID, and pointer position data. For example, the pointer data illustrated in FIG. 16 is stored in the own pointer data storage unit 98 of the user system 2-2. In this embodiment, for example, the user system ID assigned to the user system 2 is set as the user system ID of the pointer data stored in the own pointer data storage unit 98 of the user system 2.

  As described above, when the pointer of the user system 2 is not arranged in the virtual space, 0 is set as the texture ID included in the pointer data, and (0, 0) is set as the value of the pointer position data. The

  In this embodiment, the other pointer data storage unit 100 stores pointer data indicating a pointer set by a user of the user system 2 different from the user system 2 in which the other pointer data storage unit 100 is included.

  The other pointer data storage unit 100 may store a plurality of pointer data having different user system IDs. For example, the other pointer data storage unit 100 of the user system 2-1 may store a plurality of pointer data respectively associated with the user systems 2 other than the user system 2-1, which are included in the virtual space sharing system 1. In this case, for example, the pointer data illustrated in FIG. 16 is one of a plurality of pointer data stored in the other pointer data storage unit 100 of the user system 2-1. Further, for example, the other pointer data storage unit 100 of the user system 2-2 may store a plurality of pointer data respectively associated with the user systems 2 other than the user system 2-2 included in the virtual space sharing system 1. .

  In the present embodiment, for example, the captured image reception unit 102 receives an image including an image of the HMD 12 that is captured by the camera 20a at a predetermined sampling rate.

  In this embodiment, for example, the position / orientation specifying unit 104 specifies the position and orientation of the HMD 12 based on the image received by the captured image receiving unit 102. Note that the position / orientation specifying unit 104 may specify the position and orientation of the HMD 12 based on motion data representing measurement results such as the rotation amount and movement amount of the HMD 12 measured by the sensor unit 40 of the HMD 12. In this case, for example, a position obtained by moving the position specified last time by the movement amount represented by the motion data may be specified as the position of the new HMD 12. Further, for example, a posture obtained by rotating the previously specified posture by the rotation amount represented by the motion data may be specified as the new posture of the HMD 12.

  In this embodiment, the own viewpoint data management unit 106 updates the value of the viewpoint data stored in the own viewpoint data storage unit 92 based on the position and orientation of the HMD 12 specified by the position / orientation specifying unit 104, for example. Here, for example, the value of the viewpoint position data is updated so that the change amount of the three-dimensional coordinate value indicated by the viewpoint position data included in the viewpoint data is proportional to the change amount of the position of the HMD 12 specified by the position / orientation specifying unit 104. You may be made to do. Also, for example, the value of the line-of-sight vector data is updated so that the amount of change of the vector indicated by the line-of-sight vector data included in the viewpoint data is proportional to the rotation amount of the HMD 12 specified by the position / orientation specifying unit 104. May be.

  In this embodiment, the viewpoint data transmission unit 108 transmits the viewpoint data stored in the own viewpoint data storage unit 92 to the management system 3, for example. As described above, the management system 3 relays the viewpoint data transmitted in this way and transmits it to the other user systems 2. Here, for example, whenever the viewpoint data stored in the own viewpoint data storage unit 92 is updated, the updated viewpoint data is transmitted to the other user system 2 via the management system 3 in accordance with the update. May be. Further, for example, the viewpoint data stored in the own viewpoint data storage unit 92 may be transmitted to another user system 2 via the management system 3 at a predetermined time interval (for example, every 5 seconds).

  In this embodiment, the viewpoint data receiving unit 110 receives viewpoint data transmitted from the other user system 2 and transmitted from the management system 3, for example.

  In this embodiment, the other viewpoint data management unit 112 updates the value of the viewpoint data stored in the other viewpoint data storage unit 94 based on the viewpoint data received by the viewpoint data reception unit 110, for example. Here, the other viewpoint data management unit 112 may update the value of the viewpoint data stored in the other viewpoint data storage unit 94 in accordance with the reception of the viewpoint data by the viewpoint data reception unit 110.

  In this embodiment, for example, the indicator data management unit 114 generates indicator data associated with the viewpoint data based on the viewpoint data stored in the other viewpoint data storage unit 94. In the present embodiment, for example, based on the value of the viewpoint data stored in the other viewpoint data storage unit 94, the value of the indicator data having the same user system ID as the viewpoint data is updated.

  For example, the value of the viewpoint position data of the viewpoint data stored in the other viewpoint data storage unit 94 is set as the value of the viewpoint indicator position data of the indicator data having the same user system ID as the viewpoint data.

  In the present embodiment, for example, a texture image that intersects a line passing through a position represented by the viewpoint position data of the viewpoint data along a vector represented by the line-of-sight direction vector data of the viewpoint data stored in the other viewpoint data storage unit 94. Is identified. Further, in the X1Y1 two-dimensional coordinate system set for the texture image, the intersection of the texture image and a line passing through the position represented by the viewpoint position data of the viewpoint data along the vector represented by the line-of-sight direction vector data of the viewpoint data A two-dimensional coordinate value of the position is specified. For example, the identifier of the specified texture image is set as the texture ID of the indicator data having the same viewpoint data and user system ID. Also, for example, the specified two-dimensional coordinate value is set as the value of the gazing point indicator position data of the indicator data having the same viewpoint data and the user system ID.

  Note that, as in the example of FIG. 12, a spherical surface having a predetermined length around the position represented by the viewpoint position data of the viewpoint data stored in the other viewpoint data storage unit 94 may be specified. Then, the intersection of the specified spherical surface and the line passing through the position represented by the viewpoint position data of the viewpoint data along the vector represented by the line-of-sight direction vector data of the viewpoint data stored in the other viewpoint data storage unit 94 is It may be specified. Then, the three-dimensional coordinate value of the specified intersection may be set as the value of the gazing point indicator position data of the indicator data having the same viewpoint data and user system ID. In this case, the indicator data may not include the texture ID.

  In the present embodiment, the pointer operation specifying unit 116 specifies an operation related to a pointer placed in the virtual space, such as the above-described reset operation, move operation, and erase operation. Here, for example, when a user operation on the controller 22, mouse, or touch pad as described above is detected, it is specified that an operation related to the pointer image 80 corresponding to the operation has been performed.

  In this embodiment, for example, when the pointer operation specifying unit 116 specifies an operation related to the pointer image 80, the own pointer data management unit 118 stores the pointer data stored in the own pointer data storage unit 98 according to the operation. Update.

  For example, when the reset operation is specified, a line passing through the position indicated by the viewpoint position data of the viewpoint data along the vector indicated by the line-of-sight direction vector data of the viewpoint data stored in the pointer data storage unit 98 is An intersecting texture image is identified. In addition, the position of the intersection of the texture image and the line passing through the position indicated by the viewpoint position data of the viewpoint data along the vector indicated by the line-of-sight direction vector data of the viewpoint data stored in the own pointer data storage unit 98. A dimension coordinate value is also specified. The two-dimensional coordinate value is, for example, a two-dimensional coordinate value in the X1Y1 two-dimensional coordinate system set for the texture image.

  For example, the identifier of the specified texture image is set as the texture ID included in the pointer data stored in the own pointer data storage unit 98. For example, the specified two-dimensional coordinate value is set as the value of the pointer position data included in the pointer data.

  When the movement operation is specified, for example, the value of the pointer position data included in the pointer data stored in the own pointer data storage unit 98 is updated by a value corresponding to the movement operation. Here, the own pointer data management unit 118 may perform control so that the value of the pointer position data is limited to the range of the two-dimensional coordinate value indicating the position in the multiplying-axis object 62b.

  When the erasing operation is specified, for example, the value of the texture ID is updated to 0, and the value of the pointer position data is updated to (0, 0).

  In this embodiment, for example, the pointer data transmission unit 120 transmits the pointer data stored in the own pointer data storage unit 98 to the management system 3. As described above, the management system 3 relays the pointer data transmitted in this way and transmits it to the other user system 2. Here, for example, whenever the pointer data stored in the own pointer data storage unit 98 is updated, the updated viewpoint data of the value is transmitted to the other user system 2 via the management system 3 in accordance with the update. May be. Further, for example, the pointer data stored in the own pointer data storage unit 98 may be transmitted to another user system 2 via the management system 3 at a predetermined time interval (for example, every 5 seconds).

  In this embodiment, for example, the pointer data receiving unit 122 receives pointer data transmitted from the other user system 2 and transmitted from the management system 3.

  In this embodiment, the other pointer data management unit 124 updates the value of the pointer data stored in the other pointer data storage unit 100 based on the pointer data received by the pointer data reception unit 122, for example. Here, the other pointer data management unit 124 may update the value of the pointer data stored in the other pointer data storage unit 100 in response to the reception of the pointer data by the pointer data reception unit 122.

  In this embodiment, the virtual space management unit 126 arranges various virtual objects in the virtual space to be displayed. In the present embodiment, for example, the virtual space management unit 126 sets various virtual objects in the virtual space by setting values of various data stored in the virtual space data storage unit 90. For example, the viewpoint 64 and the line-of-sight direction 66 are arranged based on the viewpoint data stored in the own viewpoint data storage unit 92. Further, for example, the viewpoint indicator 68 and the gazing point indicator 70 are arranged based on the indicator data stored in the indicator data storage unit 96. Further, for example, the pointer is arranged based on the pointer data stored in the own pointer data storage unit 98 and the pointer data stored in the other pointer data storage unit 100.

  Here, for example, the virtual space management unit 126 of the user system 2-1 sets various data values corresponding to the virtual space 74 illustrated in FIG. For example, in the virtual space 74, a viewpoint 64a and a line-of-sight direction 66a corresponding to the viewpoint data stored in the own viewpoint data storage unit 92 of the user system 2-1 are arranged. For example, in the virtual space 74, a viewpoint indicator 68b and a gazing point indicator 70b corresponding to the indicator data stored in the indicator data storage unit 96 of the user system 2-1 are arranged. In this case, the virtual space management unit 126 arranges the gazing point indicator 70b at a position away from the viewpoint 64b along the line-of-sight direction 66b corresponding to the posture of the HMD 12 included in the user system 2-2 that passes through the viewpoint 64b. Will be.

  Similarly, for example, the virtual space management unit 126 of the user system 2-1 sets various data values corresponding to the virtual space 82 illustrated in FIG. 17, so that various objects are arranged in the virtual space 82. For example, in the virtual space 82, a viewpoint 64a and a line-of-sight direction 66a corresponding to the viewpoint data stored in the own viewpoint data storage unit 92 of the user system 2-1 are arranged. Further, for example, a pointer image 80 corresponding to the pointer data stored in the other pointer data storage unit 100 of the user system 2-1 is arranged in the virtual space 82. In this case, the virtual space management unit 126 of the user system 2-1 places a pointer at a position in the virtual space corresponding to the direction of the HMD 12 included in the user system 2-2.

  In addition, for example, the virtual space management unit 126 of the user system 2-2 sets various data values corresponding to the virtual space 76 illustrated in FIG. 10 or FIG. For example, in the virtual space 76, a viewpoint 64b and a line-of-sight direction 66b corresponding to the viewpoint data stored in the own viewpoint data storage unit 92 of the user system 2-2 are arranged. In the virtual space 76, a viewpoint indicator 68a and a gazing point indicator 70a corresponding to the indicator data stored in the indicator data storage unit 96 of the user system 2-2 are arranged. In this case, the virtual space management unit 126 arranges the gaze point indicator 70a at a position away from the viewpoint 64a along the line-of-sight direction 66a according to the attitude of the HMD 12 included in the user system 2-1 that passes through the viewpoint 64a. Will be.

  Similarly, for example, the virtual space management unit 126 of the user system 2-2 sets various data values corresponding to the virtual space 78 illustrated in FIG. 14, so that various objects are arranged in the virtual space 78. For example, in the virtual space 78, a viewpoint 64b and a line-of-sight direction 66b corresponding to the viewpoint data stored in the own viewpoint data storage unit 92 of the user system 2-2 are arranged. In the virtual space 78, a viewpoint indicator 68a and a gazing point indicator 70a corresponding to the indicator data stored in the indicator data storage unit 96 of the user system 2-2 are arranged. A pointer image 80 corresponding to the pointer data stored in the own pointer data storage unit 98 of the user system 2-2 is arranged in the virtual space 78. In this case, the virtual space management unit 126 of the user system 2-2 places a pointer at a position in the virtual space corresponding to the direction of the HMD 12 included in the user system 2-2.

  The virtual space management unit 126 may hold a flag indicating whether or not each of the viewpoint indicator 68 and the gazing point indicator 70 is arranged. And the value of the said flag may be changeable according to operation by a user. Whether or not each of the viewpoint indicator 68 and the gazing point indicator 70 is arranged may be controlled according to the value of the flag. For example, it is assumed that a value indicating that the viewpoint indicator 68 and the gaze point indicator 70 are arranged is set as the value of the flag by the user's operation of the user system 2-1. In this case, as shown in FIG. 8, the viewpoint indicator 68b and the gazing point indicator 70b may be arranged in the virtual space. Further, for example, it is assumed that a value indicating that the viewpoint indicator 68 and the gazing point indicator 70 are not arranged is set as the value of the flag by a user operation of the user system 2-1. In this case, as shown in FIG. 17, the viewpoint indicator 68b and the gazing point indicator 70b may not be arranged in the virtual space.

  In addition, the virtual space management unit 126 may arrange the viewpoint indicator 68 and the gazing point indicator 70 that can identify the associated user system 2, that is, the associated HMD 12, in the virtual space. For example, a viewpoint indicator 68 or a gazing point indicator 70 having a color or shape corresponding to the user system ID included in the indicator data may be arranged in the virtual space. In this case, each of the plurality of viewpoint indicators 68 arranged in the virtual space is an object that can identify the associated user system 2, that is, the associated HMD 12. Each of the plurality of gazing point indicators 70 arranged in the virtual space is an object that can identify the associated user system 2, that is, the associated HMD 12.

  Further, as described above, when a movement operation is performed by the user wearing the HMD 12, the pointer position data included in the pointer data stored in the own pointer data storage unit 98 is a value corresponding to the movement operation. The value of is updated. Then, the virtual space management unit 126 changes the position where the pointer image 80 is arranged in accordance with the update of the value of the pointer position data. As described above, the virtual space management unit 126 according to the present embodiment changes the position where the pointer image 80 is arranged according to the moving operation by the user wearing the HMD 12.

  Further, as described above, the value of the pointer position data may be controlled so as to be limited to the range of the two-dimensional coordinate value indicating the position in the hanging scroll object 62b. In this case, the virtual space management unit 126 according to the present embodiment performs control so that the position where the pointer is arranged is limited to a range indicating the position in the virtual object arranged in the virtual space. In addition, as described above, the virtual space management unit 126 is a virtual where the position where the pointer is arranged intersects the line along the direction corresponding to the direction of the HMD 12 among the plurality of virtual objects arranged in the virtual space. You may control so that it may restrict | limit to the range which shows the position in an object.

  In the present embodiment, for example, the frame image generation unit 128 generates an image representing a state in which the viewing direction 66 is viewed from the position of the viewpoint 64 arranged in the virtual space by the virtual space management unit 126 at a predetermined frame rate. For example, the frame image generation unit 128 of the user system 2-1 generates the virtual space image 72a shown in FIGS. 7 and 18 as a frame image. In this case, for example, the frame image generation unit 128 viewed the line-of-sight direction 66a corresponding to the posture of the HMD 12 included in the user system 2-1 from the viewpoint 64a arranged in the virtual space where the gazing point indicator 70b is arranged. An image representing the state is generated. Further, for example, the frame image generation unit 128 of the user system 2-2 generates the virtual space image 72b shown in FIGS. 9, 13, and 15 as a frame image. In this case, for example, the frame image generation unit 128 looks at the line-of-sight direction 66b corresponding to the attitude of the HMD 12 included in the user system 2-2 from the viewpoint 64b arranged in the virtual space where the gazing point indicator 70a is arranged. An image representing the appearance is generated.

  In the present embodiment, for example, the frame image display control unit 130 displays a virtual space image 72 that is an image of a virtual space in which a pointer is arranged. For example, the frame image display control unit 130 displays the frame image generated by the frame image generation unit 128 on the display unit 38 of the HMD 12 at a predetermined frame rate. In the present embodiment, for example, the frame image display control unit 130 of the user system 2-1 transfers the virtual space image 72a generated by the frame image generation unit 128 of the user system 2-1 to the HMD 12 included in the user system 2-1. Send. The HMD 12 that has received the frame image causes the display unit 38 of the HMD 12 to display the frame image. For example, the frame image display control unit 130 of the user system 2-2 transmits the virtual space image 72b generated by the frame image generation unit 128 of the user system 2-2 to the HMD 12 included in the user system 2-2. The HMD 12 that has received the frame image causes the display unit 38 of the HMD 12 to display the frame image.

  As illustrated in FIG. 18, the frame image display control unit 130 of the user system 2-1 displays the virtual space image 72 a on which a pointer corresponding to the direction of the HMD 12 included in the user system 2-2 is arranged. -1 may be displayed on the HMD 12 included.

  Hereinafter, an example of a flow of processing performed in the entertainment apparatus 14 according to the present embodiment in response to reception of viewpoint data will be described with reference to a flowchart illustrated in FIG.

  First, the viewpoint data receiving unit 110 receives viewpoint data (S101). Then, the other viewpoint data management unit 112 specifies viewpoint data including the same user system ID as the viewpoint data received in the process shown in S101, which is stored in the other viewpoint data storage unit 94 (S102).

  Then, the other viewpoint data management unit 112 updates the viewpoint data identified in the process shown in S102 to the viewpoint data received in the process shown in S101 (S103).

  And the indicator data management part 114 produces | generates indicator data based on the viewpoint data updated by the process shown to S103 as mentioned above (S104). The value of the viewpoint indicator position data included in the indicator data generated here can be specified based on the value of the viewpoint position data included in the viewpoint data updated in the process shown in S103 as described above. Further, the texture ID and the value of the gazing point indicator position data included in the generated indicator data are the value of the viewpoint position data and the line-of-sight direction vector included in the viewpoint data updated in the process shown in S103 as described above. It can be identified based on the value of the data.

  And the indicator data management part 114 specifies the indicator data containing the same user system ID as the indicator data produced | generated by the process shown to S104 memorize | stored in the indicator data memory | storage part 96 (S105).

  Then, the other-viewpoint data management unit 112 updates the indicator data identified in the process shown in S105 to the indicator data received in the process shown in S104 (S106), and the process shown in this process example ends.

  In the present embodiment, for example, the processing shown in S101 to S106 described above is executed every time viewpoint data is received.

  Next, an example of the flow of processing performed in the entertainment device 14 according to the present embodiment in response to reception of pointer data will be described with reference to the flowchart illustrated in FIG.

  First, the pointer data receiving unit 122 receives pointer data (S201). Then, the other pointer data management unit 124 specifies pointer data that is stored in the other pointer data storage unit 100 and includes the same user system ID as the pointer data received in the process shown in S201 (S202).

  Then, the other pointer data management unit 124 updates the pointer data specified in the process shown in S202 to the pointer data received in the process shown in S201 (S203).

  In the present embodiment, for example, each time the pointer data is received, the processes shown in S201 to S203 are executed.

  Next, an example of the flow of the virtual space display process performed in the entertainment device 14 according to the present embodiment will be described with reference to the flowchart illustrated in FIG.

  First, the captured image reception unit 102 receives an image including an image of the HMD 12 captured by the camera 20a from the camera 20a (S301). Then, the position / orientation specifying unit 104 specifies the position and orientation of the HMD 12 based on the image received in the process shown in S301 (S302).

  Based on the position and orientation of the HMD 12 specified in the process shown in S302, the own viewpoint data management unit 106 includes viewpoint position data and gaze direction vector data included in the viewpoint data stored in the own viewpoint data storage unit 92. Is updated (S303).

  Then, the viewpoint data transmission unit 108 transmits the viewpoint data updated in the process shown in S303 to the management system 3 (S304). The viewpoint data transmitted in this way is transmitted to the user system 2 included in the virtual space sharing system 1 other than the user system 2 (see S101 in FIG. 20).

  Then, the pointer operation specifying unit 116 checks whether or not an operation related to the pointer such as the above-described reset operation, movement operation, or deletion operation has been performed (S305).

  Here, it is assumed that an operation related to the pointer is confirmed (S305: Y). In this case, the own pointer data management unit 118 updates the values of the texture ID and the pointer position data included in the pointer data stored in the own pointer data storage unit 98 to values according to the operation (S306). Then, the pointer data transmission unit 120 transmits the updated pointer data to the management system 3 (S307). The pointer data transmitted in this way is transmitted to the user system 2 included in the virtual space sharing system 1 other than the user system 2 (see S201 in FIG. 21).

  If it is not confirmed in the process shown in S305 that an operation related to the pointer has been performed (S305: N), or the process shown in S307 is completed. In this case, the virtual space management unit 126 identifies the viewpoint indicator 68 and the gazing point indicator 70 to be arranged based on the value of the flag being held (S308). Then, the virtual space management unit 126 acquires indicator data associated with the viewpoint indicator 68 and the gazing point indicator 70 identified in the process shown in S307 from the indicator data storage unit 96 (S309).

  Then, the virtual space management unit 126 acquires the pointer data associated with the pointer image 80 to be arranged from the own pointer data storage unit 98 and the other pointer data storage unit 100 (S310). Here, for example, pointer data whose texture ID is 0 and whose pointer position data value is (0, 0) is excluded from acquisition targets.

  Then, the virtual space management unit 126 arranges the viewpoint indicator 68, the gazing point indicator 70, and the pointer image 80 in the virtual space (S311). Here, for example, the viewpoint indicator 68 and the gazing point indicator 70 corresponding to the value of the indicator data acquired in the process shown in S308 are arranged. A pointer image 80 corresponding to the value of the pointer data acquired in the process shown in S309 is arranged. Here, the position of the object already arranged may be changed. When pointer data associated with the pointer image 80 arranged in the virtual space has not been acquired, the pointer image 80 is deleted from the virtual space.

  Then, the virtual space management unit 126 acquires the viewpoint data stored in the own viewpoint data storage unit 92 (S312).

  Then, the virtual space management unit 126 sets the position of the viewpoint 64 and the line-of-sight direction 66 in the virtual space based on the viewpoint data acquired in the process shown in S312 (S313). Here, for example, the position of the viewpoint 64 and the line-of-sight direction 66 already set in the virtual space are changed.

  Then, the frame image generation unit 128 generates an image representing a state in which the line-of-sight direction 66 is viewed from the position of the viewpoint 64 set in the process shown in S313 (S314). In the process shown in S314, a two-dimensional image or a three-dimensional image may be generated.

  Then, the frame image display control unit 130 transmits the frame image generated by the process shown in S314 to the HMD 12 (S315). The HMD 12 that has received the frame image causes the display unit 38 to display the frame image.

  In this embodiment, the processes shown in S301 to S315 are repeatedly executed at predetermined time intervals (for example, 1/60 second intervals).

  In the above example, the pointer data is transmitted every time an operation related to the pointer is performed. Instead, for example, the pointer data is transmitted at a predetermined time interval (for example, 1/60 second interval or 5 second interval). It doesn't matter.

  In the above example, the viewpoint data is transmitted at a predetermined time interval (for example, 1/60 second interval). Instead, for example, the viewpoint data is transmitted at a time interval different from the time interval (for example, 5 second interval). May be sent.

  In the above example, the user system 2 that has received the viewpoint data generates indicator data associated with the viewpoint data. Alternatively, for example, the user system 2 may generate indicator data associated with the viewpoint data. Good. The user system 2 may transmit the indicator to another user system 2.

  In the above example, the pointer data is shared between the user systems 2. However, the pointer data may not be shared between the user systems 2 instead.

  Further, when the user of the user system 2 performs a predetermined operation while the pointer image 80 associated with the user system 2 is displayed, the user system 2 moves to the position where the pointer image 80 is displayed. A corresponding process may be executed. For example, a process of enlarging and displaying the position where the pointer image 80 is arranged in accordance with a predetermined operation may be executed.

  The present embodiment has been described above using the virtual space in which the hanging object 62 that is an example of the virtual space according to the present embodiment is arranged. However, the virtual space according to the present embodiment is not limited to the virtual space in which the hanging object 62 is arranged. For example, the virtual space according to the present embodiment may be the virtual space 140 shown in FIG.

  FIG. 23 is a diagram illustrating an example of a state in which the virtual space 140 is shared between the user system 2-1 and the user system 2-2.

  In the virtual space 140 illustrated in FIG. 23, a screen object 142 that is a virtual object representing a screen, which is a flat three-dimensional object, is arranged. In the virtual space 140, a background object 144, which is a spherical virtual object, is arranged. The radius of the background object 144 may be set to infinity, for example.

  A texture image taken by an omnidirectional camera is mapped inside the background object 144 arranged in the virtual space 140 illustrated in FIG. The texture image may be a moving image recorded in the user system 2 or a moving image that is streamed to the user system 2.

  The screen object 142 is mapped with a texture image taken by a camera different from the above-mentioned omnidirectional camera. This texture image may also be a moving image recorded in the user system 2, a moving image that is streamed to the user system 2, or the like.

  Here, for example, it is assumed that the moving image mapped to the background object 144 is a video of the entire view of the stadium where a soccer game is being performed. The moving image mapped to the screen object 142 is a relay video of the soccer game that is being broadcast on television.

  In the virtual space 140 illustrated in FIG. 23, an image representing a state in which the line-of-sight direction 148a is viewed from the viewpoint 146a arranged inside the background object 144 is displayed on the display unit 38 of the HMD 12 of the user system 2-1. . In addition, on the display unit 38 of the HMD 12 of the user system 2-2, an image representing a state in which the line-of-sight direction 148 b is viewed from the viewpoint 146 b arranged inside the background object 144 is displayed.

  Here, it is assumed that a line passing through the viewpoint 146a along the line-of-sight direction 148a intersects the texture image mapped to the background object 144. A line passing through the viewpoint 146b along the line-of-sight direction 148b also intersects with the texture image mapped to the background object 144.

  In the virtual space 140 shown in FIG. 23, a viewpoint indicator and a gazing point indicator can be arranged in the same manner as described above.

  FIG. 24 shows an example of a virtual space image 150a that is displayed on the display unit 38 of the HMD 12 of the user system 2-1, and is an image representing a view of the line-of-sight direction 148a from the viewpoint 146a arranged in the virtual space 140. FIG. The virtual space image 150a illustrated in FIG. 24 includes a gazing point indicator 152b that indicates the position of the gazing point of the user of the user system 2-2. In the example of FIG. 24, a gazing point indicator 152b is superimposed on the texture image mapped to the background object 144. This is because the line passing through the viewpoint 146b along the line-of-sight direction 148b intersects the texture image mapped to the background object 144.

  FIG. 25 shows an example of a virtual space image 150b that is an image showing a view of the line-of-sight direction 148b from the viewpoint 146b arranged in the virtual space 140 displayed on the display unit 38 of the HMD 12 of the user system 2-2. FIG. A virtual space image 150b illustrated in FIG. 25 includes a gazing point indicator 152a indicating the position of the gazing point of the user of the user system 2-1. In the example of FIG. 25, the gazing point indicator 152a is superimposed on the texture image mapped to the background object 144. This is because the line passing through the viewpoint 146a along the line-of-sight direction 148a intersects the texture image mapped to the background object 144.

  Also in the virtual space 140 shown in FIG. 23, pointers can be arranged in the same manner as described above.

  FIG. 26 shows another example of the virtual space image 150a displayed on the display unit 38 of the HMD 12 of the user system 2-1, which is an image representing a view of the line-of-sight direction 148a from the viewpoint 146a arranged in the virtual space 140. FIG.

  FIG. 27 shows another example of the virtual space image 150b displayed on the display unit 38 of the HMD 12 of the user system 2-2, which is an image showing a view direction 148b from the viewpoint 146b arranged in the virtual space 140. FIG.

  The virtual space image 150a illustrated in FIG. 26 and the virtual space image 150b illustrated in FIG. 27 include a pointer image 154a disposed by the user of the user system 2-1 and a pointer image disposed by the user of the user system 2-2. 154b is included. Moreover, the colors of the pointer image 154a and the pointer image 154b are different, and the user system 2 associated with each pointer image 154 can be identified.

  In the example of FIGS. 26 and 27, the pointer image 154a and the pointer image 154b are superimposed on the texture image mapped to the background object 144. This is because the line passing through the viewpoint 146a along the line-of-sight direction 148a and the line passing through the viewpoint 146b along the line-of-sight direction 148b intersect with the texture image mapped to the background object 144.

  Here, the movement range of the pointer image 154a and the movement range of the pointer image 154b may be limited on the texture image mapped to the background object 144.

  For example, the pointer image 154 may be arranged on the texture image mapped to the screen object 142. In this case, similarly, the movement range of the pointer image 154 may be limited on the texture image mapped to the screen object 142.

  Note that the position of the screen object 142 may be moved in accordance with a user operation. In addition, each user system 2 may be able to move the position of the screen object 142 arranged in the virtual space configured in the user system 2. And the screen object 142 in the virtual space comprised in the said user system 2 may differ for every user system 2. FIG.

  In addition, a plurality of screen objects 142 on which different images are displayed in the virtual space may be arranged.

  In the present embodiment, the moving images displayed as texture images in the user system 2 included in the virtual space sharing system 1 are synchronized, and the user system 2 can enjoy the same video at the same timing. ing.

  The plurality of omnidirectional cameras may each shoot images from different stadium positions. In accordance with a predetermined operation by a user of a certain user system 2, an all-sky camera (shooting point) that captures a video that is a texture image mapped to the background object 144 may be switched. At this time, a user may prompt another user to change the shooting point by voice. Here, the possibility of the user getting drunk can be reduced if the shooting points are not switched continuously but switched continuously.

  In the present embodiment, the position of the viewpoint 146 and the line-of-sight direction 148 set by the user of the user system 2 are not affected by the movement of the HMD 12 of another user. Thus, in this embodiment, the user who wears the HMD 12 is less likely to get drunk.

  The moving images mapped to the screen object 142 and the background object 144 are not limited to those described above. For example, the moving image mapped to the background object 144 may be a starry sky video. In this case, the moving image mapped to the screen object 142 may be an environmental video.

  In the present embodiment, an image having a higher resolution than other regions may be arranged in the region specified based on the position or orientation of the HMD 12. For example, a texture image having a resolution higher than that of other regions may be arranged in a region having a predetermined size centered on a position in the texture image that intersects with a line passing through the viewpoint 146 along the line-of-sight direction 148. For example, a texture image having a higher resolution than other texture images may be mapped to an object that intersects with a line passing through the viewpoint 146 along the line-of-sight direction 148.

  For example, the texture image mapped to the screen object 142 may have a higher resolution than the texture image mapped to the background object 144.

  In addition, this invention is not limited to the above-mentioned embodiment.

  For example, transmission / reception of viewpoint data and pointer data need not be relayed by the management system 3. For example, transmission / reception of viewpoint data and pointer data may be performed directly between the user systems 2.

  Further, the present invention can be applied to the case where the HMD 12 and the entertainment device 14 communicate with each other by wire or wirelessly.

  The specific character strings and numerical values described above and the specific character strings and numerical values in the drawings are examples, and are not limited to these character strings and numerical values.

Claims (7)

A pointer placement unit for placing a pointer at a position in the virtual space according to the orientation of the head mounted display;
A display control unit for displaying an image of a virtual space in which the pointer is arranged;
A display control apparatus comprising: The pointer arrangement unit changes a position where the pointer is arranged according to an operation of a user wearing the head mounted display.
The display control apparatus according to claim 1. The pointer arrangement unit controls the position where the pointer is arranged to be limited to a range indicating a position in a virtual object arranged in the virtual space;
The display control apparatus according to claim 2. The pointer placement unit is a virtual object in which a position at which the pointer is placed intersects a line along a direction according to the direction of the head mounted display among the plurality of virtual objects placed in the virtual space. Control to be limited to the range indicating the position,
The display control apparatus according to claim 2. The display control unit displays the image on a head mounted display different from the head mounted display;
The display control device according to claim 1, wherein the display control device is a display control device. Placing a pointer at a position in the virtual space according to the orientation of the head mounted display;
Displaying an image of a virtual space in which the pointer is arranged;
A display control method comprising: Procedure to place the pointer at a position in the virtual space according to the orientation of the head mounted display,
A procedure for displaying an image of a virtual space in which the pointer is arranged;
A program that causes a computer to execute.

Images