JP2018207517A - Method to be executed by computer for controlling display in head-mounted device, program for causing computer to execute the same method, and information processing device - Google Patents

Abstract

To provide a technology for displaying comments without losing a feeling of immersion into a virtual space.SOLUTION: A method to be executed by a computer for controlling display in a head-mounted device which provides a virtual space includes: a step for transmitting, to the head-mounted device, a video signal for displaying a video corresponding to a user's operation of the head-mounted device; a step for transmitting the video signal to one or more display terminals communicably connected to the computer; a step for receiving comments 103 to 106 given to a video displayed by one or more display terminals on the basis of the video signal from one or more display terminals; and a step for causing a monitor 112 to present the comments 103 to 106 in a state that the comments 103 to 106 are maintained at a fixed place even while an image of the virtual space transits.SELECTED DRAWING: Figure 1

Description

  The present disclosure relates to a technique for providing a virtual space using a head-mounted device, and more specifically to a technique for presenting a comment in a virtual space.

  A computer that executes moving image distribution, a game program, and the like generates a video signal corresponding to a play by a live player. The computer (or another device that has received the video signal) transmits the live video to the distribution server. The distribution server transmits the live video to one or more other terminals that are connected to the distribution server and logged in. The distribution server receives an input of a comment from one of the terminals, and transmits the received comment to another terminal as needed. Thereby, both the live viewer and the viewer can enjoy the interaction (interaction) in which the live broadcaster plays live while referring to the comments from the live viewer.

  For example, International Publication No. 2016/039156 (Patent Document 1) relates to a technique for displaying a user's comment in the distribution of a moving image. The technology that can reduce the user's trouble when communicating information related to the moving image managed by the moving image distribution system that distributes the moving image is disclosed.

International Publication No. 2016/039156

  However, when a video or other video is provided in a virtual reality space (also referred to as “virtual space”), a live wearer wearing a head mounted device (HMD) sees the video presented on the HMD. On the other hand, when the actual situation is performed, the actual person cannot refer to the comment of the viewer. In addition, when a moving image is distributed using the HMD, simply displaying a comment may impair the immersive feeling in the virtual space. Therefore, there is a need for a technique in which a comment input by a viewer of a content is presented to the user of the content without impairing the immersive feeling in the virtual space.

  The present disclosure has been made to solve the above-described problems, and an object in one aspect is to provide a technique in which comments are presented in a virtual space without impairing the sense of immersion.

  According to an embodiment, a computer-implemented method is provided for controlling display on a head mounted device that provides virtual space. The method includes a step of transmitting a video signal for displaying a video according to a user operation of the head mounted device to the head mounted device, and a video signal to one or more display terminals connected to be communicable with the computer. Transmitting a viewer response given to the video displayed on the one or more display terminals based on the video signal from the one or more display terminals, and a viewer response to the head mounted device Presenting.

According to an embodiment, the comment is presented in the virtual space without impairing the immersive feeling.
The above and other objects, features, aspects and advantages of the present invention will become apparent from the following detailed description of the present invention taken in conjunction with the accompanying drawings.

It is a figure showing transition of the screen displayed on monitor 112 which provides virtual space. It is a figure showing the outline of the structure for adding a comment with respect to the image shown in virtual space. It is a figure showing the outline of a structure of the HMD system 100 according to a certain embodiment. It is a block diagram showing an example of the hardware constitutions of the computer 200 according to one situation. It is a figure which represents notionally the uvw visual field coordinate system set to HMD110 according to a certain embodiment. It is a figure which represents notionally the one aspect | mode which represents the virtual space 2 according to a certain embodiment. It is the figure showing the head of user 190 wearing HMD110 according to a certain embodiment from the top. 3 is a diagram illustrating a YZ cross section of a visual field region 23 viewed from the X direction in a virtual space 2. FIG. 3 is a diagram illustrating an XZ cross section of a visual field region 23 viewed from a Y direction in a virtual space 2. FIG. It is a figure showing schematic structure of the controller 160 according to a certain embodiment. It is a figure which shows an example of the hand object 810 arrange | positioned corresponding to the right hand of the user 190 who holds the right controller 800 in virtual space. FIG. 3 is a block diagram representing a computer 200 according to an embodiment as a module configuration. It is a block diagram showing the structure of the function implement | achieved by the server computer 20 according to a certain situation. FIG. 11 is a diagram conceptually illustrating one aspect of data storage in a storage unit 1310. 4 is a flowchart showing a part of processing executed when server computer 20 is realized by computer 200. 4 is a flowchart showing a part of processing executed when server computer 20 is realized by server 150. It is a flowchart showing a part of process performed by the user terminal 201A of another user. It is a figure showing an example of the screen displayed on the monitor 212 of 201 A of user terminals. It is a figure which represents notionally the flow of the production | generation of the data for displaying an image on the monitor 112. FIG. It is a figure showing the one aspect | mode of the screen by which the comment input by the user 190 is displayed on the monitor 212 of the user terminal of another user.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the following description, the same parts are denoted by the same reference numerals. Their names and functions are also the same. Therefore, detailed description thereof will not be repeated.

[Technology]
The outline of the technical idea disclosed here is as follows. Here, a case will be described in which comments of viewers of content distributed live are presented on the monitor of the HMD worn by the content distributor (actual person).

  (A) A computer that generates a video signal to be presented to the HMD of the live broadcaster according to the play of the live broadcaster transmits the video signal to the content distribution server.

  (B) The distribution server distributes a video signal (for example, a video signal corresponding to the result of the operation or the operation of the controller) to other connected terminals (logged in). At this time, a computer (for example, the computer or the distribution server) that generates a view image obtained by photographing the virtual space controls the placement of the virtual camera placed in the virtual space and updates the view image.

  (C) The terminal connected to the distribution server receives a video signal related to live distribution from the distribution server and reproduces the video, and accepts an input of a comment by a user (viewer) of the terminal. In response, the comment is transmitted.

  (D) The distribution server specifies a live video distributor (actual person) device (for example, a computer to which the HMD is connected), and transmits a comment input from the terminal to the actual person device.

  (E) The HMD monitor worn by the live commentator displays the comment received from the distribution server. At this time, the comment can be displayed in a place where the video is not obstructed, for example, at the end of the display area, beyond the object presented in the virtual space. Or the character which displays a comment may be displayed by a transparent character. In addition, the comment can be temporarily displayed. For example, the comment is displayed at the timing when the view field image is switched or is not displayed.

  With reference to FIG. 1, an implementation example of the technical idea according to the present disclosure will be described. FIG. 1 is a diagram illustrating transition of a screen displayed on the monitor 112 that provides a virtual space. The monitor 112 is provided in the HMD, for example, or attached to the HMD. The monitor 112 presents an object and a comment based on a program executed by a computer connected to the HMD or a computer built in the HMD. Hereinafter, a case where the program presents a scene on the monitor 112 where a horse running in the virtual space is caught by a rope will be described.

  [State A] In a certain situation, the monitor 112 displays a horse object 101 and a rope object 102 for catching the horse object 101. Further, the monitor 112 displays the comment 103 (“Now!”). The comment 103 is input from a computer used by a user other than the HMD user. A computer used by another user can display the scene as either a two-dimensional image or an image in a virtual space.

  [State B] Thereafter, when the user of the monitor 112 performs an operation and throws the rope object 102 to the horse object 101, the monitor 112 displays another comment 104 (“delivery!”) Input by another user. . Note that the user who inputs the comment 103 and the user who inputs the comment 104 may be the same or different. In one aspect, the comment 103 is displayed until the scene displayed on the monitor 112 is switched or for a predetermined time. When the scene switches to the next scene or when a predetermined time elapses, the display of the comment 103 ends.

  [State C] When the user of the monitor 112 can successfully hook the rope object 102 on the horse object 101, the monitor 112 displays the comment 105 (“Yes!”) And the comment 106 (“Success!”). To do. Also in this case, the user who input the comment 105 and the user who input the comment 106 may be the same or different. Also in this case, as in the case of the state B, when a predetermined condition including the above-described condition is satisfied, the comment display is ended.

  When a plurality of comments are displayed, the first comment can be displayed at a predetermined location, and the next comment can be displayed at a position shifted from the first comment by a preset interval. Thereby, the user can confirm each comment clearly. In another aspect, each comment may be displayed sequentially. For example, when the end of the content is displayed, each comment can be displayed sequentially. In this case, since the display of the content is finished, the problem of immersive feeling can be suppressed. In yet another aspect, the computer equipped with the HMD may receive an input of a comment for the comment displayed on the monitor 112 from the user, and transmit the comment to another viewer. Since comments are exchanged between the HMD user and the viewer, mutual interaction can be further promoted.

  As is clear from the comments 103, 104, and 105, each comment is displayed at the same place even if the horse object 101 presented in the virtual space moves. Thereby, the visibility of a comment is maintained, without impairing the immersive feeling in virtual space. Each comment can be deleted after being displayed for a predetermined time. Alternatively, each comment may be switched to another comment when the video scene is switched.

  With reference to FIG. 2, an apparatus configuration that realizes the technical idea according to the present disclosure will be described. FIG. 2 is a diagram showing an outline of a configuration for adding a comment to an image presented in the virtual space. The virtual space is provided by the HMD 110. The HMD 110 is communicably connected to a server computer 20 having a known configuration. The server computer 20 is communicably connected to one or more user terminals 201A, 201B, 201N (referred to collectively as user terminals 201). The user terminal 201 includes a processor 211 and a monitor 212. In one aspect, the user terminal 201 is, for example, a computer, a tablet terminal, a smartphone, or other information communication terminal having a known configuration. In another aspect, similarly to the HMD 110, the user terminal 201 may be realized by the user's computer and an HMD connected to the computer.

  The HMD 110 displays a video on the monitor 112 based on the video signal sent from the server computer 20. When the user operates the controller, the HMD 110 transmits information representing the operation to the server computer 20. The server computer 20 executes processing such as moving an object included in the image presented on the HMD 110 based on the information, and the HMD 110 and a logged-in user for displaying a video signal for displaying the processed image Displayed on the terminals 201A, 201B, and 201N. The format of the video signal transmitted to the HMD 110 and the format of the video signal transmitted to the user terminals 201A, 201B, 201N may be the same or different. The video signal transmitted to the HMD 110 is three-dimensional data for use in a virtual space while maintaining an immersive feeling. On the other hand, the video signals transmitted to the user terminals 201A, 201B, and 201C may be data for displaying an image in two dimensions when an immersive feeling is not required.

  Each user of the user terminals 201A, 201B, and 201N can visually recognize the same image as the image that the user of the HMD 110 is viewing. At this time, each user can input a comment on the image using a touch panel, a mouse, a voice input device, or other input devices. The input comment is sent to the server computer 20. For example, the server computer 20 displays the comment so as not to interfere with the image presented on the HMD 110 so as to overlap the edge of the viewing area. Alternatively, the server computer 20 displays the comment in a transparent color. The user wearing the HMD 110 can visually recognize the comment while confirming the image presented in the virtual space.

[Configuration of HMD system]
The configuration of an HMD (Head Mounted Device) system 100 will be described with reference to FIG. FIG. 3 is a diagram representing a schematic configuration of HMD system 100 according to an embodiment. In one aspect, the HMD system 100 is provided as a home system or a business system. In the present embodiment, the HMD may include both a so-called head mounted display including a monitor and a head mounted device on which a terminal having a monitor such as a smartphone can be mounted.

  The HMD system 100 includes an HMD 110, an HMD sensor 120, a controller 160, and a computer 200. The HMD 110 includes a monitor 112 and a gaze sensor 140. The controller 160 can include a motion sensor 130.

  In one aspect, the computer 200 can be connected to the Internet and other networks 19, and can communicate with the server 150 and other computers connected to the network 19. User terminals 201 </ b> A to 201 </ b> N are connected to the network 19.

  In another aspect, instead of the HMD system 100 including the HMD sensor 120, the HMD 110 may include the sensor 114.

  The server 150 includes a processor 151, a memory 152, and a communication interface 153. The server 150 is realized by a computer having a known configuration.

  The HMD 110 may be worn on the user's head and provide a virtual space to the user during operation. More specifically, the HMD 110 displays a right-eye image and a left-eye image on the monitor 112, respectively. When each eye of the user visually recognizes each image, the user can recognize the image as a three-dimensional image based on the parallax of both eyes.

  The monitor 112 is realized as, for example, a non-transmissive display device. In one aspect, the monitor 112 is disposed on the main body of the HMD 110 so as to be positioned in front of both eyes of the user. Therefore, when the user visually recognizes the three-dimensional image displayed on the monitor 112, the user can be immersed in the virtual space. In one embodiment, the virtual space includes, for example, a background, an object that can be operated by the user, and an image of a menu that can be selected by the user. In an embodiment, the monitor 112 may be realized as a liquid crystal monitor or an organic EL (Electro Luminescence) monitor provided in a so-called smartphone or other information display terminal.

  In one aspect, the monitor 112 may include a sub-monitor for displaying an image for the right eye and a sub-monitor for displaying an image for the left eye. In another aspect, the monitor 112 may be configured to display a right-eye image and a left-eye image together. In this case, the monitor 112 includes a high-speed shutter. The high-speed shutter operates so that an image for the right eye and an image for the left eye can be displayed alternately so that the image is recognized only by one of the eyes.

  The HMD sensor 120 includes a plurality of light sources (not shown). Each light source is realized by, for example, an LED (Light Emitting Diode) that emits infrared rays. The HMD sensor 120 has a position tracking function for detecting the movement of the HMD 110. Using this function, the HMD sensor 120 detects the position and inclination of the HMD 110 in the real space.

  In another aspect, HMD sensor 120 may be realized by a camera. In this case, the HMD sensor 120 can detect the position and inclination of the HMD 110 by executing image analysis processing using image information of the HMD 110 output from the camera.

  In another aspect, the HMD 110 may include a sensor 114 instead of the HMD sensor 120 as a position detector. The HMD 110 can detect the position and inclination of the HMD 110 itself using the sensor 114. For example, when the sensor 114 is an angular velocity sensor, a geomagnetic sensor, an acceleration sensor, a gyro sensor, or the like, the HMD 110 detects its own position and inclination using any one of these sensors instead of the HMD sensor 120. Can do. As an example, when the sensor 114 is an angular velocity sensor, the angular velocity sensor detects angular velocities around the three axes of the HMD 110 in real space over time. The HMD 110 calculates a temporal change in the angle around the three axes of the HMD 110 based on each angular velocity, and further calculates an inclination of the HMD 110 based on the temporal change in the angle. The HMD 110 may include a transmissive display device. In this case, the transmissive display device may be temporarily configured as a non-transmissive display device by adjusting the transmittance. Further, the visual field image may include a configuration for presenting the real space in a part of the image configuring the virtual space. For example, an image captured by a camera mounted on the HMD 110 may be displayed so as to be superimposed on a part of the field-of-view image, or by setting the transmittance of a part of the transmissive display device to be high. Real space may be visible from a part.

  The gaze sensor 140 detects a direction (gaze direction) in which the gaze of the right eye and the left eye of the user 190 is directed. The detection of the direction is realized by, for example, a known eye tracking function. The gaze sensor 140 is realized by a sensor having the eye tracking function. In one aspect, the gaze sensor 140 preferably includes a right eye sensor and a left eye sensor. The gaze sensor 140 may be, for example, a sensor that irradiates the right eye and the left eye of the user 190 with infrared light and detects the rotation angle of each eyeball by receiving reflected light from the cornea and iris with respect to the irradiated light. . The gaze sensor 140 can detect the line-of-sight direction of the user 190 based on each detected rotation angle.

  Server 150 may send a program to computer 200. In another aspect, the server 150 may communicate with other computers 200 for providing virtual reality to HMDs used by other users. For example, when a plurality of users play a participatory game in an amusement facility, each computer 200 communicates a signal based on each user's operation with another computer 200, and a plurality of users are common in the same virtual space. Allows you to enjoy the game.

  The controller 160 receives input of commands from the user 190 to the computer 200. In one aspect, the controller 160 is configured to be gripped by the user 190. In another aspect, the controller 160 is configured to be attachable to the body of the user 190 or a part of clothing. In another aspect, the controller 160 may be configured to output at least one of vibration, sound, and light based on a signal sent from the computer 200. In another aspect, the controller 160 accepts an operation given by the user 190 to control the position and movement of an object arranged in a space that provides virtual reality.

  In one aspect, the motion sensor 130 is attached to the user's hand and detects the movement of the user's hand. For example, the motion sensor 130 detects the rotation speed, rotation speed, etc. of the hand. The detected signal is sent to the computer 200. The motion sensor 130 is provided in a glove-type controller 160, for example. In some embodiments, for safety in real space, it is desirable that the controller 160 be mounted on something that does not fly easily by being mounted on the hand of the user 190, such as a glove shape. In another aspect, a sensor that is not worn by the user 190 may detect the hand movement of the user 190. For example, a signal from a camera that captures the user 190 may be input to the computer 200 as a signal representing the operation of the user 190. The motion sensor 130 and the computer 200 are connected to each other by wire or wirelessly. In the case of wireless communication, the communication form is not particularly limited, and for example, Bluetooth (registered trademark) or other known communication methods are used.

[Hardware configuration]
A computer 200 according to the present embodiment will be described with reference to FIG. FIG. 4 is a block diagram illustrating an example of a hardware configuration of computer 200 according to one aspect. The computer 200 includes a processor 10, a memory 11, a storage 12, an input / output interface 13, and a communication interface 14 as main components. Each component is connected to the bus 15.

  The processor 10 executes a series of instructions included in the program stored in the memory 11 or the storage 12 based on a signal given to the computer 200 or based on the establishment of a predetermined condition. In one aspect, the processor 10 is realized as a CPU (Central Processing Unit), an MPU (Micro Processor Unit), an FPGA (Field-Programmable Gate Array), or other device.

  The memory 11 temporarily stores programs and data. The program is loaded from the storage 12, for example. The data includes data input to the computer 200 and data generated by the processor 10. In one aspect, the memory 11 is realized as a RAM (Random Access Memory) or other volatile memory.

  The storage 12 holds programs and data permanently. The storage 12 is realized as, for example, a ROM (Read-Only Memory), a hard disk device, a flash memory, and other nonvolatile storage devices. The programs stored in the storage 12 include a program for providing a virtual space in the HMD system 100, a simulation program, a game program, a user authentication program, and a program for realizing communication with another computer 200. The data stored in the storage 12 includes data and objects for defining the virtual space.

  In another aspect, the storage 12 may be realized as a removable storage device such as a memory card. In still another aspect, a configuration using a program and data stored in an external storage device may be used instead of the storage 12 built in the computer 200. According to such a configuration, for example, in a scene where a plurality of HMD systems 100 are used as in an amusement facility, it is possible to update programs and data collectively.

  In some embodiments, the input / output interface 13 communicates signals with the HMD 110, HMD sensor 120, or motion sensor 130. In one aspect, the input / output interface 13 is realized using a USB (Universal Serial Bus) interface, a DVI (Digital Visual Interface), an HDMI (registered trademark) (High-Definition Multimedia Interface), or other terminals. The input / output interface 13 is not limited to that described above.

  In certain embodiments, the input / output interface 13 may further communicate with the controller 160. For example, the input / output interface 13 receives a signal output from the motion sensor 130. In another aspect, the input / output interface 13 sends the instruction output from the processor 10 to the controller 160. The command instructs the controller 160 to vibrate, output sound, emit light, and the like. When the controller 160 receives the command, the controller 160 executes vibration, sound output, or light emission according to the command.

  The communication interface 14 is connected to the network 19 and communicates with other computers (for example, the server 150) connected to the network 19. In one aspect, the communication interface 14 is realized as, for example, a local area network (LAN) or other wired communication interface, or a wireless communication interface such as WiFi (Wireless Fidelity), Bluetooth (registered trademark), NFC (Near Field Communication), or the like. Is done. The communication interface 14 is not limited to the above.

  In one aspect, the processor 10 accesses the storage 12, loads one or more programs stored in the storage 12 into the memory 11, and executes a series of instructions included in the program. The one or more programs may include an operating system of the computer 200, an application program for providing a virtual space, game software that can be executed in the virtual space using the controller 160, and the like. The processor 10 sends a signal for providing a virtual space to the HMD 110 via the input / output interface 13. The HMD 110 displays an image on the monitor 112 based on the signal.

  In the example illustrated in FIG. 4, the configuration in which the computer 200 is provided outside the HMD 110 is illustrated. However, in another aspect, the computer 200 may be incorporated in the HMD 110. As an example, a portable information communication terminal (for example, a smartphone) including the monitor 112 may function as the computer 200.

  Further, the computer 200 may be configured to be used in common for a plurality of HMDs 110. According to such a configuration, for example, the same virtual space can be provided to a plurality of users, so that each user can enjoy the same application as other users in the same virtual space.

  In an embodiment, in the HMD system 100, a global coordinate system is set in advance. The global coordinate system has three reference directions (axes) parallel to the vertical direction in the real space, the horizontal direction orthogonal to the vertical direction, and the front-rear direction orthogonal to both the vertical direction and the horizontal direction. In the present embodiment, the global coordinate system is one of the viewpoint coordinate systems. Therefore, the horizontal direction, the vertical direction (up-down direction), and the front-rear direction in the global coordinate system are defined as an x axis, a y axis, and a z axis, respectively. More specifically, in the global coordinate system, the x axis is parallel to the horizontal direction of the real space. The y axis is parallel to the vertical direction of the real space. The z axis is parallel to the front-rear direction of the real space.

  In one aspect, HMD sensor 120 includes an infrared sensor. When the infrared sensor detects the infrared rays emitted from each light source of the HMD 110, the presence of the HMD 110 is detected. The HMD sensor 120 further detects the position and inclination of the HMD 110 in the real space according to the movement of the user 190 wearing the HMD 110 based on the value of each point (each coordinate value in the global coordinate system). More specifically, the HMD sensor 120 can detect temporal changes in the position and inclination of the HMD 110 using each value detected over time.

  The global coordinate system is parallel to the real space coordinate system. Therefore, each inclination of the HMD 110 detected by the HMD sensor 120 corresponds to each inclination around the three axes of the HMD 110 in the global coordinate system. The HMD sensor 120 sets the uvw visual field coordinate system to the HMD 110 based on the inclination of the HMD 110 in the global coordinate system. The uvw visual field coordinate system set in the HMD 110 corresponds to a viewpoint coordinate system when the user 190 wearing the HMD 110 views an object in the virtual space.

[Uvw visual field coordinate system]
The uvw visual field coordinate system will be described with reference to FIG. FIG. 5 is a diagram conceptually showing a uvw visual field coordinate system set in HMD 110 according to an embodiment. The HMD sensor 120 detects the position and inclination of the HMD 110 in the global coordinate system when the HMD 110 is activated. The processor 10 sets the uvw visual field coordinate system to the HMD 110 based on the detected value.

  As shown in FIG. 5, the HMD 110 sets a three-dimensional uvw visual field coordinate system with the head (origin) of the user wearing the HMD 110 as the center (origin). More specifically, the HMD 110 includes a horizontal direction, a vertical direction, and a front-rear direction (x-axis, y-axis, z-axis) that define the global coordinate system by an inclination around each axis of the HMD 110 in the global coordinate system. Three directions newly obtained by tilting around the axis are set as the pitch direction (u-axis), yaw direction (v-axis), and roll direction (w-axis) of the uvw visual field coordinate system in the HMD 110.

  In a certain situation, when the user 190 wearing the HMD 110 stands upright and is viewing the front, the processor 10 sets the uvw visual field coordinate system parallel to the global coordinate system to the HMD 110. In this case, the horizontal direction (x-axis), vertical direction (y-axis), and front-back direction (z-axis) in the global coordinate system are the pitch direction (u-axis) and yaw direction (v-axis) of the uvw visual field coordinate system in the HMD 110. , And the roll direction (w axis).

  After the uvw visual field coordinate system is set to the HMD 110, the HMD sensor 120 can detect the inclination (the amount of change in inclination) of the HMD 110 in the set uvw visual field coordinate system based on the movement of the HMD 110. In this case, the HMD sensor 120 detects the pitch angle (θu), yaw angle (θv), and roll angle (θw) of the HMD 110 in the uvw visual field coordinate system as the inclination of the HMD 110. The pitch angle (θu) represents the inclination angle of the HMD 110 around the pitch direction in the uvw visual field coordinate system. The yaw angle (θv) represents the inclination angle of the HMD 110 around the yaw direction in the uvw visual field coordinate system. The roll angle (θw) represents the inclination angle of the HMD 110 around the roll direction in the uvw visual field coordinate system.

  The HMD sensor 120 sets the uvw visual field coordinate system in the HMD 110 after the HMD 110 has moved to the HMD 110 based on the detected tilt angle of the HMD 110. The relationship between the HMD 110 and the uvw visual field coordinate system of the HMD 110 is always constant regardless of the position and inclination of the HMD 110. When the position and inclination of the HMD 110 change, the position and inclination of the uvw visual field coordinate system of the HMD 110 in the global coordinate system change in conjunction with the change of the position and inclination.

  In one aspect, the HMD sensor 120 is based on the infrared light intensity acquired based on the output from the infrared sensor and the relative positional relationship between a plurality of points (for example, the distance between the points). The position in the real space may be specified as a relative position to the HMD sensor 120. Further, the processor 10 may determine the origin of the uvw visual field coordinate system of the HMD 110 in the real space (global coordinate system) based on the specified relative position.

[Virtual space]
The virtual space will be further described with reference to FIG. FIG. 6 is a diagram conceptually showing one aspect of expressing virtual space 2 according to an embodiment. The virtual space 2 has a spherical structure that covers the entire 360 ° direction of the center 21. In FIG. 6, the upper half of the celestial sphere in the virtual space 2 is illustrated for the sake of simplicity. In the virtual space 2, each mesh is defined. The position of each mesh is defined in advance as coordinate values in the XYZ coordinate system defined in the virtual space 2. The computer 200 associates each partial image constituting content (still image, moving image, etc.) that can be developed in the virtual space 2 with each corresponding mesh in the virtual space 2, and the virtual space image 22 that can be visually recognized by the user. Is provided to the user.

  In one aspect, the virtual space 2 defines an XYZ coordinate system with the center 21 as the origin. The XYZ coordinate system is, for example, parallel to the global coordinate system. Since the XYZ coordinate system is a kind of viewpoint coordinate system, the horizontal direction, vertical direction (vertical direction), and front-rear direction in the XYZ coordinate system are defined as an X axis, a Y axis, and a Z axis, respectively. Therefore, the X axis (horizontal direction) of the XYZ coordinate system is parallel to the x axis of the global coordinate system, the Y axis (vertical direction) of the XYZ coordinate system is parallel to the y axis of the global coordinate system, and The Z axis (front-rear direction) is parallel to the z axis of the global coordinate system.

  When the HMD 110 is activated, that is, in the initial state of the HMD 110, the virtual camera 1 is disposed at the center 21 of the virtual space 2. The virtual camera 1 similarly moves in the virtual space 2 in conjunction with the movement of the HMD 110 in the real space. Thereby, changes in the position and orientation of the HMD 110 in the real space are similarly reproduced in the virtual space 2.

  As with the HMD 110, the uvw visual field coordinate system is defined for the virtual camera 1. The uvw visual field coordinate system of the virtual camera 1 in the virtual space 2 is defined so as to be linked to the uvw visual field coordinate system of the HMD 110 in the real space (global coordinate system). Therefore, when the inclination of the HMD 110 changes, the inclination of the virtual camera 1 also changes accordingly. The virtual camera 1 can also move in the virtual space 2 in conjunction with the movement of the user wearing the HMD 110 in the real space.

  Since the orientation of the virtual camera 1 is determined according to the position and inclination of the virtual camera 1, the reference line of sight (reference line of sight 5) when the user visually recognizes the virtual space image 22 depends on the orientation of the virtual camera 1. Determined. The processor 10 of the computer 200 defines the visual field region 23 in the virtual space 2 based on the reference line of sight 5. The visual field area 23 corresponds to the visual field of the user wearing the HMD 110 in the virtual space 2.

  The gaze direction of the user 190 detected by the gaze sensor 140 is a direction in the viewpoint coordinate system when the user 190 visually recognizes the object. The uvw visual field coordinate system of the HMD 110 is equal to the viewpoint coordinate system when the user 190 visually recognizes the monitor 112. Further, the uvw visual field coordinate system of the virtual camera 1 is linked to the uvw visual field coordinate system of the HMD 110. Therefore, the HMD system 100 according to a certain aspect can regard the line-of-sight direction of the user 190 detected by the gaze sensor 140 as the line-of-sight direction of the user in the uvw visual field coordinate system of the virtual camera 1.

[User's line of sight]
With reference to FIG. 7, the determination of the user's line-of-sight direction will be described. FIG. 7 is a top view of the head of user 190 wearing HMD 110 according to an embodiment.

  In one aspect, gaze sensor 140 detects each line of sight of user 190's right eye and left eye. In a certain aspect, when the user 190 is looking near, the gaze sensor 140 detects the lines of sight R1 and L1. In another aspect, when the user 190 is looking far away, the gaze sensor 140 detects the lines of sight R2 and L2. In this case, the angle formed by the lines of sight R2 and L2 with respect to the roll direction w is smaller than the angle formed by the lines of sight R1 and L1 with respect to the roll direction w. The gaze sensor 140 transmits the detection result to the computer 200.

  When the computer 200 receives the detection values of the lines of sight R1 and L1 from the gaze sensor 140 as the line-of-sight detection result, the computer 200 identifies the point of sight N1 that is the intersection of the lines of sight R1 and L1 based on the detection value. On the other hand, when the detected values of the lines of sight R2 and L2 are received from the gaze sensor 140, the computer 200 specifies the intersection of the lines of sight R2 and L2 as the point of sight N1. The computer 200 specifies the line-of-sight direction N0 of the user 190 based on the specified position of the gazing point N1. For example, the computer 200 detects the direction in which the straight line passing through the midpoint of the straight line connecting the right eye R and the left eye L of the user 190 and the gazing point N1 extends as the line-of-sight direction N0. The line-of-sight direction N0 is a direction in which the user 190 is actually pointing the line of sight with both eyes. The line-of-sight direction N0 corresponds to the direction in which the user 190 actually directs his / her line of sight with respect to the field-of-view area 23.

  In another aspect, the HMD system 100 may include a microphone and a speaker in any part constituting the HMD system 100. The user can give a voice instruction to the virtual space 2 by speaking to the microphone.

  In another aspect, HMD system 100 may include a television broadcast receiving tuner. According to such a configuration, the HMD system 100 can display a television program in the virtual space 2.

  In still another aspect, the HMD system 100 may include a communication circuit for connecting to the Internet or a call function for connecting to a telephone line.

[Visibility area]
The field-of-view area 23 will be described with reference to FIGS. FIG. 8 is a diagram illustrating a YZ cross section of the visual field region 23 viewed from the X direction in the virtual space 2. FIG. 9 is a diagram illustrating an XZ section of the visual field region 23 viewed from the Y direction in the virtual space 2.

  As shown in FIG. 8, the visual field region 23 in the YZ cross section includes a region 24. The region 24 is defined by the reference line of sight 5 of the virtual camera 1 and the YZ cross section of the virtual space 2. The processor 10 defines a range including the polar angle α around the reference line of sight 5 in the virtual space as the region 24.

  As shown in FIG. 9, the visual field region 23 in the XZ cross section includes a region 25. The region 25 is defined by the reference line of sight 5 and the XZ cross section of the virtual space 2. The processor 10 defines a range including the azimuth angle β around the reference line of sight 5 in the virtual space 2 as a region 25.

  In one aspect, the HMD system 100 provides the virtual space to the user 190 by displaying the view image 26 on the monitor 112 based on a signal from the computer 200. The view image 26 corresponds to a portion of the virtual space image 22 that is superimposed on the view region 23. When the user 190 moves the HMD 110 worn on the head, the virtual camera 1 also moves in conjunction with the movement. As a result, the position of the visual field area 23 in the virtual space 2 changes. As a result, the view image 26 displayed on the monitor 112 is updated to an image that is superimposed on the view region 23 in the direction in which the user faces in the virtual space 2 in the virtual space image 22. The user can visually recognize a desired direction in the virtual space 2.

  While wearing the HMD 110, the user 190 can visually recognize only the virtual space image 22 developed in the virtual space 2 without visually recognizing the real world. Therefore, the HMD system 100 can give the user a high sense of immersion in the virtual space 2.

  In one aspect, the processor 10 can move the virtual camera 1 in the virtual space 2 in conjunction with the movement of the user 190 wearing the HMD 110 in the real space. In this case, the processor 10 specifies an image region (that is, a view field region 23 in the virtual space 2) projected on the monitor 112 of the HMD 110 based on the position and orientation of the virtual camera 1 in the virtual space 2.

  According to an embodiment, the virtual camera 1 preferably includes two virtual cameras, that is, a virtual camera for providing an image for the right eye and a virtual camera for providing an image for the left eye. Moreover, it is preferable that appropriate parallax is set in the two virtual cameras so that the user 190 can recognize the three-dimensional virtual space 2. In the present embodiment, the virtual camera 1 includes two virtual cameras, and the roll direction (w) generated by combining the roll directions of the two virtual cameras is adapted to the roll direction (w) of the HMD 110. The technical idea concerning this indication is illustrated as what is constituted.

[controller]
An example of the controller 160 will be described with reference to FIG. FIG. 10 is a diagram showing a schematic configuration of controller 160 according to an embodiment. In certain aspects, the controller 160 may include a right controller 800 and a left controller. The right controller 800 is operated with the right hand of the user 190. The left controller is operated with the left hand of the user 190. In one aspect, the right controller 800 and the left controller are configured symmetrically as separate devices. Therefore, the user 190 can freely move the right hand holding the right controller 800 and the left hand holding the left controller. In another aspect, the controller 160 may be an integrated controller that receives operations of both hands. Hereinafter, the right controller 800 will be described.

  The right controller 800 includes a grip 30, a frame 31, and a top surface 32. The grip 30 is configured to be held by the right hand of the user 190. For example, the grip 30 can be held by the palm of the right hand of the user 190 and three fingers (middle finger, ring finger, little finger).

  The grip 30 includes buttons 33 and 34 and a motion sensor 130. The button 33 is disposed on the side surface of the grip 30 and receives an operation with the middle finger of the right hand. The button 34 is disposed in front of the grip 30 and accepts an operation with the index finger of the right hand. In one aspect, the buttons 33 and 34 are configured as trigger buttons. The motion sensor 130 is built in the housing of the grip 30. Note that when the operation of the user 190 can be detected from around the user 190 by a camera or other device, the grip 30 may not include the motion sensor 130.

  The frame 31 includes a plurality of infrared LEDs 35 arranged along the circumferential direction. The infrared LED 35 emits infrared light in accordance with the progress of the program during the execution of the program using the controller 160. The infrared rays emitted from the infrared LED 35 can be used to detect the positions and postures (tilt and orientation) of the right controller 800 and the left controller (not shown). In the example shown in FIG. 8, infrared LEDs 35 arranged in two rows are shown, but the number of arrays is not limited to that shown in FIG. An array of one or more columns may be used.

  The top surface 32 includes buttons 36 and 37 and an analog stick 38. The buttons 36 and 37 are configured as push buttons. The buttons 36 and 37 receive an operation with the thumb of the right hand of the user 190. In one aspect, the analog stick 38 accepts an operation in an arbitrary direction of 360 degrees from the initial position (neutral position). The operation includes, for example, an operation for moving an object arranged in the virtual space 2.

  In one aspect, the right controller 800 and the left controller include a battery for driving the infrared LED 35 and other members. The battery includes, but is not limited to, a rechargeable type, a button type, a dry battery type, and the like. In another aspect, the right controller 800 and the left controller may be connected to a USB interface of the computer 200, for example. In this case, the right controller 800 and the left controller do not require batteries.

  FIG. 11 shows an example of a hand object 810 arranged in the virtual space corresponding to the right hand of the user 190 holding the right controller 800. For example, the yaw, roll, and pitch directions are defined for the hand object 810 corresponding to the right hand of the user 190. For example, when the input operation is performed on the button 34 of the right controller 800, the hand object 810 is held in the index finger, and when the input operation is not performed on the button 34, the hand object 810 is performed. The index finger can be extended. For example, when the thumb and index finger are extended in the hand object 810, the direction in which the thumb extends is the yaw direction, and the direction in which the index finger extends is perpendicular to the plane defined by the roll direction, the yaw direction axis, and the roll direction axis. The direction is defined in the hand object 810 as a pitch direction.

[Control device of HMD110]
With reference to FIG. 12, the control apparatus of HMD110 is demonstrated. In one embodiment, the control device is realized by a computer 200 having a known configuration. FIG. 12 is a block diagram showing a computer 200 according to an embodiment as a module configuration. In the following, a case where the comment adding function by the server computer 20 is realized by the computer 200 will be described. In another aspect, this additional function may be realized by the server 150.

  As shown in FIG. 12, the computer 200 includes a display control module 220, a virtual space control module 230, a memory module 240, and a communication control module 250. The display control module 220 includes a virtual camera control module 221, a visual field region determination module 222, a visual field image generation module 223, a reference visual line identification module 224, and a comment addition module 225 as submodules. The virtual space control module 230 includes a virtual space definition module 231, a virtual object generation module 232, and a controller management module 233 as submodules.

  In an embodiment, the display control module 220 and the virtual space control module 230 are realized by the processor 10. In another embodiment, multiple processors 10 may operate as the display control module 220 and the virtual space control module 230. The memory module 240 is realized by the memory 11 or the storage 12. The communication control module 250 is realized by the communication interface 14.

  In one aspect, the display control module 220 controls image display on the monitor 112 of the HMD 110. The virtual camera control module 221 arranges the virtual camera 1 in the virtual space 2 and controls the behavior, orientation, and the like of the virtual camera 1. The view area determination module 222 defines the view area 23 according to the direction of the head of the user wearing the HMD 110. The view image generation module 223 generates a view image 26 to be displayed on the monitor 112 based on the determined view area 23.

  The reference line-of-sight identifying module 224 identifies the line of sight of the user 190 based on the signal from the gaze sensor 140. The comment adding module 225 overlaps the comment received via the server 150 with the view image generated by the view image generating module 223.

  The virtual space control module 230 controls the virtual space 2 provided to the user 190. The virtual space definition module 231 defines the virtual space 2 in the HMD system 100 by generating virtual space data representing the virtual space 2. The virtual object generation module 232 generates an object placed in the virtual space 2. The object is, for example, an object constituting a background such as a mountain, a tree, an animal object (for example, a horse object 101) presented according to a story constituting a program realized by the computer 200, or an object for capturing an animal object. Includes objects (eg, rope object 102).

  The controller management module 233 receives the operation of the user 190 in the virtual space 2 and controls the controller object according to the operation. A controller object according to an embodiment functions as a controller that gives instructions to other objects arranged in the virtual space 2. In one aspect, the controller management module 233 generates data for placing a controller object that receives control in the virtual space 2 in the virtual space 2. When the HMD 110 receives this data, the monitor 112 may display the controller object.

  The memory module 240 holds data used for the computer 200 to provide the virtual space 2 to the user 190. In one aspect, the memory module 240 holds space information 241, user information 242, content 243, and comments 244.

  The space information 241 holds one or more templates defined for providing the virtual space 2. The user information 242 includes identification information of the user 190 of the HMD 110, authority associated with the user 190, and the like. The authority includes, for example, account information (user ID, password) for accessing a website that provides the application. The content 243 includes content presented by the HMD 110, for example. The comment 244 is a comment input by any of the user terminals 201A to 201N.

  The communication control module 250 can communicate with the server 150 and other information communication devices via the network 19. The communication control module 250 may be either wireless or wired.

  In an aspect, the display control module 220 and the virtual space control module 230 may be realized using, for example, Unity (registered trademark) provided by Unity Technologies. In another aspect, the display control module 220 and the virtual space control module 230 can also be realized as a combination of circuit elements that realize each process.

  Processing in the computer 200 is realized by hardware and software executed by the processor 10. Such software may be stored in advance in a memory module 240 such as a hard disk. The software may be stored in a CD-ROM or other non-volatile computer-readable data recording medium and distributed as a program product. Alternatively, the software may be provided as a program product that can be downloaded by an information provider connected to the Internet or other networks. Such software is read from a data recording medium by an optical disk drive or other data reader, or downloaded from the server 150 or other computer via the communication control module 250 and then temporarily stored in the storage module. . The software is read from the storage module by the processor 10 and stored in the RAM in the form of an executable program. The processor 10 executes the program.

  The hardware that constitutes the computer 200 is general. Therefore, it can be said that the most essential part according to the present embodiment is a program stored in the computer 200. Since the hardware operation of computer 200 is well known, detailed description will not be repeated.

  The data recording medium is not limited to a CD-ROM, FD (Flexible Disk), and hard disk, but is a magnetic tape, cassette tape, optical disk (MO (Magnetic Optical Disc) / MD (Mini Disc) / DVD (Digital Versatile Disc)). ), IC (Integrated Circuit) card (including memory card), optical card, mask ROM, EPROM (Electronically Programmable Read-Only Memory), EEPROM (Electronically Erasable Programmable Read-Only Memory), flash ROM, etc. It may be a non-volatile data recording medium that carries a fixed program.

  The program here may include not only a program directly executable by the processor 10, but also a program in a source program format, a compressed program, an encrypted program, and the like.

[Configuration of server computer]
The configuration of the server computer 20 will be further described with reference to FIG. FIG. 13 is a block diagram showing a configuration of functions implemented by server computer 20 according to a certain aspect. The server computer 20 includes a storage unit 1310, a signal reception unit 1320, a video signal processing unit 1330, a video signal transmission unit 1340, a comment reception unit 1350, an image generation unit 1360, and an image transmission unit 1370.

  The storage unit 1310 holds data and programs input to the server computer 20. For example, the storage unit 1310 holds data transmitted from the HMD 110, comments input by any of the user terminals 201A, 201B, and 201N, and distribution destinations of content displayed on the HMD 110. The storage unit 1310 can be realized as the memory module 240 by, for example, a flash memory, a hard disk device or other nonvolatile memory, and a volatile memory such as a RAM.

  The signal receiving unit 1320 receives a video signal transmitted from the HMD 110 or the computer 200. The signal receiving unit 1320 is realized as the communication control module 250.

  The video signal processing unit 1330 converts the format of the video signal received by the signal receiving unit 1320 into a signal in a format suitable for display on the user terminals 201A, 201B, and 201N. For example, a video signal for displaying a three-dimensional image on the HMD 110 is converted into a video signal for displaying a two-dimensional image on each user terminal 201. The video signal processing unit 1330 is realized by the display control module 220.

  The video signal transmission unit 1340 transmits the video signal generated by the video signal processing unit 1330 to the user terminal 201 registered in the storage unit 1310 as a distribution destination. When the user terminal 201 displays an image based on the video signal, the user of the user terminal 201 can recognize the image that the user 190 wearing the HMD 110 is viewing. When the user of the user terminal 201 inputs a comment for an image and performs a transmission operation, the comment is transmitted to the server computer 20. At this time, position information specifying a place where the comment is displayed is also transmitted to the server computer 20. The video signal transmission unit 1340 is realized by the communication control module 250.

  The comment receiving unit 1350 receives the comment and position information transmitted from the user terminal 201. The comment receiving unit 1350 is realized by the communication control module 250, for example. The received comment and position information are stored in the storage unit 1310.

  The image generation unit 1360 generates a comment image to be presented to the HMD 110 using the comments and position information stored in the storage unit 1310. For example, the image generation unit 1360 generates a comment image so that the comment is displayed at the same place in the field of view image regardless of the orientation and position of the HMD 110. The image generation unit 1360 can be realized as the view image generation module 223 and the comment addition module 225 by the display control module 220, for example.

  The image transmission unit 1370 transmits the comment image generated by the image generation unit 1360 to the HMD 110. The HMD 110 can visually recognize an image in which a content image and a comment image overlap. At this time, the position of the comment image is determined at a predetermined fixed position regardless of the position and orientation of the HMD 110.

  The data structure of the server computer 20 will be described with reference to FIG. FIG. 14 is a diagram conceptually illustrating one mode of data storage in the storage unit 1310. The storage unit 1310 holds tables 1410, 1420, and 1430. The table 1410 includes a user ID 1411, a user name 1412, a password 1413, and a last login date 1414. The table 1420 includes a user ID 1421 and a status 1422. The table 1430 includes a user ID 1431, a comment date 1432, a position 1433, a comment 1434, and a frame number 1435.

  The table 1410 manages information on users who share certain contents. More specifically, the user ID 1411 represents user identification data registered in the distribution site. The user name 1412 represents the name of the user. A password 1413 represents a password necessary for login. The last login date 1414 represents the date and time when the user logged in last time.

  The table 1420 manages the user status for the content. More specifically, the user ID 1421 represents identification data of the currently logged-in user. The status 1422 represents the current state of the user, for example, whether the content is being played or whether the content is simply viewed.

  The table 1430 manages comments input by each user. More specifically, the user ID 1431 identifies the user who wrote the comment. The comment date 1432 represents the date and time when the comment was written. A position 1433 represents a position where the comment is displayed. The comment 1434 represents the content of the comment. The frame number 1435 is a number of a frame constituting the content. Frame number 1435 represents position information in the content in which the comment is written. In another aspect, instead of the frame number 1435, time information for specifying the playback position of the content may be stored as the position information. When the reproduction of the same content is designated at another date and time after the reproduction of the content is finished, the processor of the server computer 20 refers to the table 1430 to change the comment written on the content. It can be read and comments can be presented to the viewer along with the content.

[Control structure]
A control structure of the server computer 20 will be described with reference to FIG. FIG. 15 is a flowchart showing a part of processing executed when server computer 20 is realized by computer 200.

  In step S <b> 1510, the processor 10 accesses the server 150 that provides a moving image distribution site via the communication control module 250, and transmits login information to the server 150.

  In step S1515, the processor 10 transmits a distribution request for the content (for example, 360-degree moving image, game application, chat application) selected by the user 190 to the server 150 via the communication control module 250. The server 150 transmits the content to 20 in response to the request.

  In step S1520, processor 10 receives a video signal for displaying a moving image from server 150 via communication control module 250.

  In step S1525, the processor 10 generates data (content image data) for presenting a visual field image based on the video signal in the virtual space as the visual field image generation module 223.

  In step S1530, processor 10 transmits the content image data to HMD 110.

  In step S1535, the processor 10 detects that a signal corresponding to the operation of the user 190 has been received based on the signal from the controller 160.

  In step S1540, the processor 10 generates content image data corresponding to the operation as the view image generation module 223, and the generated content image data is transmitted to the other user's user terminal (for example, the user terminal 201A, 201B... 201N) or the like.

  In step S <b> 1545, the processor 10 receives, from the server 150, a comment input by another user and position information for displaying the comment via the communication control module 250.

  In step S1550, processor 10 generates data (comment image data) for displaying a comment on HMD 110. In one aspect, the processor 10 generates data for displaying the comment on the HMD 110 separately from the image data based on the application for displaying the virtual space presented on the HMD 110.

  In step S1555, the processor 10 uses the content image data and the comment image data as the comment addition module 225 to generate view field image data in which the content and the comment are overlapped.

  In step S1560, processor 10 outputs the view image data to HMD 110. The user 190 can recognize the comment displayed on the monitor 112.

  With reference to FIG. 16, the control structure of server computer 20 in another aspect will be described. FIG. 16 is a flowchart showing a part of processing executed when server computer 20 is realized by server 150.

  In step S1610, the processor 151 of the server 150 detects the login of the user 190 wearing the HMD 110 based on the signal sent from the computer 200, and previously stores the user ID and password included in the received signal as registration information. The authentication process is executed using the held user ID and password.

  In step S1620, processor 151 of server 150 loads the application program designated by user 190 from the storage device to the RAM. The processor 151 executes the loaded application program and transmits a video signal based on the execution to the computer 200. The HMD 110 displays content in the virtual space 2 based on the video signal. The user 190 can operate the controller 160 while viewing the content. The investigation information is sent from the controller 160 to the computer 200, and the display of content changes.

  In step S1630, processor 151 of server 150 receives a video signal for displaying an image from computer 200 based on the operation of the user.

  In step S1640, processor 151 of server 150 detects login by another user based on a signal received from user terminal 201A or the like, and executes an authentication process using the user ID and password. When it is confirmed that the login request is issued by an authorized user registered in advance, the server 150 accepts access from the user terminal 201A or the like.

  In step S1650, processor 151 of server 150 converts an image viewed as a three-dimensional image into an image for two-dimensional display, and transmits the video signal generated by the conversion to user terminal 201A of another user or the like. . Since the user terminal 201A displays the same content as the content that the user 190 is viewing on the monitor, the other user can also enjoy the content. Other users can input comments for the user 190 using the input device while viewing the content. The input comment is transmitted to the server 150 together with the position information.

  In step S1660, processor 151 of server 150 receives a comment from any of user terminals 201A, 201B,.

  In step S1670, processor 151 of server 150 generates a video signal for displaying the comment as a view field image, and transmits the generated video signal to computer 200 of user 190 wearing HMD 110. The monitor 112 of the HMD 110 may display a comment overlapped with the content image.

  With reference to FIG. 17, the control structure of another user's terminal is demonstrated. FIG. 17 is a flowchart showing a part of the processing executed by user terminal 201A of another user.

  In step S1710, processor 211 logs in to the distribution site displayed on monitor 212 based on the operation of another user. In step S1720, processor 211 accepts selection of content desired to be distributed based on the operation of another user. For example, the other user can select the content that the user 190 is viewing. In step S1730, processor 211 transmits a content distribution request to server 150.

  In step S1740, processor 211 receives a video signal from server 150 and displays video based on the signal on monitor 212. In step S1750, processor 211 accepts input of a comment on the video via an input interface such as a mouse, a keyboard, or a touch panel.

  In step S1760, processor 211 transmits a comment to server 150. When the server 150 transmits this comment to the computer 200 to which the HMD 110 is connected, the comment is displayed on the monitor 112.

  With reference to FIG. 18, the display mode of the screen in user terminal 201A will be described. FIG. 18 is a diagram illustrating an example of a screen displayed on the monitor 212 of the user terminal 201A. In one aspect, the monitor 212 displays the same content as the content displayed on the monitor 112 of the HMD 110 worn by the user 190. Furthermore, the monitor 212 displays an image 1810 indicating that the user who is playing is the user 190 (= user 0001), and an image 1820 prompting the transmission of the comment. The image 1810 and the image 1820 are, for example, pop-up images.

  For example, when the user terminal 201A starts displaying the content displayed on the monitor 112, the monitor 212 displays the image 1810 for a predetermined time. In another aspect, when the user 190 operates the controller 160, an image 1810 is displayed in response to the operation. In this case, another user using the user terminal 201A can know the user who is playing when the scene changes in response to the operation.

  The image 1820 is displayed every predetermined time in a certain situation. In another aspect, image 1820 is displayed in response to the timing when user 190 performs some operation, similar to image 1810. In this case, since it becomes difficult for other users to miss the comment input timing for the operation of the user 190, the comments of the other users are also quickly displayed on the monitor 112 as shown in FIG. As a result, the user 190 can easily view timely comments.

  The display of content and comments on the monitor 112 will be described with reference to FIG. FIG. 19 is a diagram conceptually showing a flow of generating data for displaying an image on the monitor 112.

  As shown in state A, in one aspect, the memory 11 holds drawing data 1910 for displaying content on the monitor 112 in a part of the work area. The content includes a horse object 101 and a rope object 102.

  As shown in state B, the memory 11 holds drawing data 1920 for displaying a comment 1921 input by another user in another work area different from the work area. The drawing data 1920 is generated after the comment 1921 is received by the computer 200. Note that the place where the comment 1921 is displayed is maintained at a certain place on the monitor 112 regardless of the orientation and orientation of the HMD 110. Therefore, for example, even when the line-of-sight direction (the direction of the virtual camera 1) moves due to the horse object 101 moving in the virtual space 2, the comment 1921 is displayed at a fixed position on the monitor 112. Become. Such a positional relationship for display is maintained, for example, by defining the display area of the monitor 112 as a coordinate value of the absolute position and defining the display location of the comment 1921 in association with the coordinate value of the absolute position.

  As shown in the state C, the memory 11 holds the drawing data 1930 in yet another work area. The drawing data 1930 is generated by configuring drawing data 1910 and drawing data 1920. When the processor 10 transmits the drawing data 1930 to the HMD 110, the monitor 112 displays an image as shown in the state A of FIG. Similarly, comments are displayed for other images (state B and state C in FIG. 1).

  With reference to FIG. 20, the display mode of the screen in the user terminal 201 is demonstrated. FIG. 20 is a diagram illustrating an aspect of a screen in which a comment input by user 190 is displayed on monitor 212 of user terminal 201 of another user. The monitor 212 displays an image 2010 and an image 2020 in addition to the virtual space image displayed on the monitor 112 of the HMD 110. The image 2010 represents the user 190 who is playing. An image 2020 represents a comment input by the user 190. For example, when the user 190 inputs a response comment after enjoying the content, the response comment is transmitted to the other user who sent the comment to the user 190. As described above, the comments between the other user and the user 190 are sent to each other, so that the interaction between the user 190 and the other user can be promoted.

As described above, the technical features according to the present disclosure can be summarized as follows.
(1st structure) The method performed in the server computer 20 in order to control the display in HMD110 which provides the virtual space 2 transmits the video signal for displaying the image | video according to operation of the user 190 of HMD110 to HMD110. A step of transmitting a video signal to one or more user terminals 201A, 201B, 201N and the like that are communicably connected to the server computer 20, and one or more user terminals 201A, 201B based on the video signal , 201N, etc., and receiving a viewer response (for example, comments 103, 104, 105, 106, etc.) from one or more user terminals 201A, 201B, 201N, etc. And causing the monitor 112 to present a viewer response.

  (2nd structure) In addition to said structure, the step which transmits a video signal to one or more user terminal 201A, 201B, 201N etc. is a two-dimensional image to one or more user terminal 201A, 201B, 201N etc. Transmitting a signal for displaying.

  (3rd structure) In addition to said structure, the step which receives a viewer response from one or more user terminal 201A, 201B, 201N etc. is a position which designates the place where a viewer response is shown in the virtual space 2. Receiving the information.

  (Fourth Configuration) In addition to the above-described configuration, the step of receiving a viewer response from one or more user terminals 201A, 201B, 201N, etc. includes a viewer response from each of the plurality of user terminals 201A, 201B, 201N, etc. Receiving. The step of receiving position information includes receiving position information of each of the user terminals 201A, 201B, 201N, etc. The step of causing the monitor 112 to present the viewer response includes causing each viewer response to be presented in a shifted manner when the position information indicates the same position.

  (Fifth Configuration) In addition to any of the configurations described above, the step of receiving a viewer response from one or more user terminals 201A, 201B, 201N, etc. includes a plurality of viewer responses as one or more user terminals 201A. , 201B, 201N, etc. Presenting the viewer response includes overlaying any viewer response of the plurality of viewer responses on the other viewer responses.

  (Sixth Configuration) In addition to any of the above configurations, the method receives a user response input by the user from the server computer 20 in response to a viewer response presented in the virtual space 2, and the user And a step of transmitting a signal for displaying the response to one or more user terminals 201A, 201B, 201N, and the like.

  (Seventh Configuration) In addition to any of the above configurations, the method further includes a step of deleting the viewer response presented in the virtual space 2.

  (8th structure) In addition to one of said structures, the step which erase | eliminates a viewer response includes the step which erase | eliminates a comment according to the change of the image shown in the virtual space 2. FIG.

  (Ninth Configuration) In addition to any of the above configurations, the method further includes a step of storing the video signal and the viewer response in a storage device in association with each other.

  (Tenth Configuration) In addition to the ninth configuration described above, the method includes a step of playing a video based on a video signal, and a viewer response associated with the video signal according to the playback of the video. Further presenting.

  (Eleventh Configuration) In addition to any of the configurations described above, the method includes a step of detecting the movement of the user 190 of the HMD 110 and a step of causing the monitor 112 to present a visual field image of the virtual space 2 in conjunction with the movement. Further included. Presenting the viewer response includes presenting the viewer response without being linked to the movement.

  As described above, according to the technology according to the present disclosure, when a user (viewer) of the user terminal 201 inputs a comment on the video provided to the other user terminal 201 by the user 190 who is a live commentator. The comment is displayed in the video displayed by the HMD 110 of the user 190. For example, it is displayed at the end of the display area or beyond the object presented in the virtual space 2. Alternatively, the comment can be popped up. In another aspect, the comment may be overlaid on a game or other content screen. When the comment is displayed in this way, the user 190 can confirm the comment without being disturbed by the immersive feeling.

  The embodiment disclosed this time should be considered as illustrative in all points and not restrictive. The scope of the present invention is defined by the terms of the claims, rather than the description above, and is intended to include any modifications within the scope and meaning equivalent to the terms of the claims.

  The technical features according to the present disclosure can be applied to a technology that provides a virtual space using an HMD.

  1 virtual camera, 2 virtual space, 5 reference line of sight, 10, 151, 211 processor, 11, 152 memory, 12 storage, 13 input / output interface, 14, 153 communication interface, 15 bus, 19 network, 20 server computer, 21 center , 22 virtual space image, 23 field of view, 24, 25 region, 26 field of view image, 30 grip, 31 frame, 32 top surface, 33, 34, 36, 37 buttons, 38 analog stick, 100 system, 101 horse object, 102 Rope object, 103, 104, 105, 106, 244, 1434, 1921 Comment, 112, 212 Monitor, 114, 120 Sensor, 130 Motion sensor, 140 Gaze sensor, 150 Server, 160 Controller, 200 computer, 201,201A, 201B, 201N user terminal.

Claims (13)

A computer-implemented method for controlling display in a head-mounted device that provides virtual space,
Transmitting a video signal for displaying a video according to an operation of a user of the head mounted device to the head mounted device;
Transmitting the video signal to one or more display terminals communicably connected to the computer;
Receiving, from the one or more display terminals, a viewer response given to the video displayed on the one or more display terminals based on the video signal;
Causing the head mounted device to present the viewer response.   The method of claim 1, wherein transmitting the video signal to the one or more display terminals includes transmitting a signal for displaying a two-dimensional image to the one or more display terminals.   The step of receiving the viewer response from the one or more display terminals includes receiving position information specifying a place where the viewer response is presented in the virtual space. the method of. Receiving the viewer response from the one or more display terminals includes receiving a viewer response from each of a plurality of display terminals;
Receiving the position information includes receiving position information of each of the display terminals;
4. The method of claim 3, wherein causing the monitor to present the viewer response includes causing the viewer responses to be presented with a shift relative to each other when the location information indicates the same location. Receiving the viewer response from the one or more display terminals includes receiving a plurality of the viewer responses from the one or more display terminals;
5. The step of presenting the viewer response includes displaying the viewer response of any one of the plurality of viewer responses as an overlay on another viewer response. Method. Receiving from the computer a user response input by the user in response to a viewer response presented in the virtual space;
The method according to claim 1, further comprising: transmitting a signal for displaying the user response to the one or more display terminals.   The method according to claim 1, further comprising the step of erasing a viewer response presented in the virtual space.   The method of claim 7, wherein deleting the viewer response includes deleting a comment in response to a change in an image presented in the virtual space.   The method according to claim 1, further comprising the step of associating the video signal and the viewer response with a storage device. Playing a video based on the video signal;
10. The method of claim 9, further comprising: causing a viewer response associated with the video signal to be presented in the virtual space in response to playback of the video. Detecting a movement of a user of the head mounted device;
Further comprising the step of causing the monitor to present a visual field image of the virtual space in conjunction with the movement,
The method according to claim 1, wherein the step of presenting the viewer response includes causing the viewer response to be presented without being interlocked with the movement.   The program which makes a computer perform the method in any one of Claims 1-11. A memory storing the program according to claim 12;
An information processing apparatus comprising: a processor for executing the program.

Images